Has anyone tried REST API Put Or Post on AWS S3 ?

Hi

We are experimenting with maintaining my image files outside of Cache as it will scale out pretty soon and obviously as its not Object based.

For this purpose we have decided to go with AWS S3 File Storage

Now S3 does support REST Operations and Ensemble does it too

However I am not able to find any example in Intersystems Documentation which demonstrates any Put Operation

Only documentation I could find is this explaining the GET operation

http://docs.intersystems.com/latest/csp/docbook/DocBook.UI.Page.cls?KEY=...

Can someone help me with an example of Put / Post operation code via REST to any host

If I could get an AWS S3 example, that will be a blessing

Thanx in advance

  • 0
  • 0
  • 829
  • 11
  • 3

Answers

Here's a small example:

Class Production.Operation.PostOperation Extends Ens.BusinessOperation
{
Parameter ADAPTER = "EnsLib.HTTP.OutboundAdapter";

Property Adapter As EnsLib.HTTP.OutboundAdapter;

Method OnMessage(request As Ens.Request, Output response As Ens.StringResponse) As %Status
{
    #dim sc As %Statis = $$$OK
    set input = "body"
    set sc = ..Adapter.Post(.httpResponse,,input)
    quit:$$$ISERR(sc) sc
    set response = ##class(Ens.StringResponse).%New(httpResponse.Read($$$MaxStringLength))
    quit sc
}
}

everything is pretty much the same as with GET requests.

Thank You Eduard,

Yes this is straight forward HTTPOperation and the example is also in the documentation.
I was hoping if I could get some insight from someone who has already worked with S3 read & write and the complexities they had to face for the same.

I'd go with AWS Java lib, but write your own wrapper lib and import only your lib into Ensemble.

Ok so I should not try to use simple Put via Rest Operation as even S3 documentation says its not that straightforward.

Ok so coming to next question. 

- Do I need to install the Java SDK / JRE on my servers and they also should be running along with Ensemble ? 
Though I do think Ensemble must have an inbuilt JRE and we may not have to run a separate JRE for the same. Am I right

Is there any rough example or step by step guide for doing the java import into ensemble and making this work?

FYI - Sent you a FB request

If you have a library you'll only need JRE. Check that java is installed with:

java --version

I recommend JRE 1.8 from Oracle.

Here's a tutorial on using Java Gateway.

If you want to develop your own java wrapper library (my recommended approach) you'll need JDK and Java IDE (i.e. IntelliJ IDEA: Community). But you can develop on your own machine, build the Jar there and then only install JRE on a production server.

Some advice.

1. Test and debug Java logic on Java side. There are several IDEs available. Debug initial errors with Java debugger - it's way better and easier to do.
2. Handle all exceptions on the Java side. Outermost method (the one you call from Ensemble should not throw exceptions. Note: speed may be faster without exception handling on java side (only on Ens side).
3. Minimize number of Ens<>Java calls. It is very easy to return several values in one java call using one of these 2  approaches:

  • $lb() - String (presumably faster, but I didn't run tests on it)
  • %ListOfDatatype - String[]

so one of the results in the list would be an actual status (which would contain serialized exception information, just a status or some success value).
To use $lb, there are com.intersys.jdbc.CacheListReader and com.intersys.jdbc.CacheListBuilder classes available in our jdbc jar.

4. Always wrap calls to jgw from in try blocks. %objlasterror contains exception info. Here's an example (from EnsLib.SAP.Operation):

try {
  Kill %objlasterror
  // Call JGW
} catch (e) {
  If e.Name="<ZJGTW>",$Data(%objlasterror) Set tSC=%objlasterror Quit
  Set tSC=e.AsStatus()
}

5. If possible test outside of Ensemble scope directly from a terminal. Here's initial method to get gateway object by name of a system object gateway:

ClassMethod Connect(gatewayName As %String, path As %String, Output sc
As %Status) As %Net.Remote.Gateway
{
  Set gateway = ""
  Set sc = ##class(%Net.Remote.Service).OpenGateway(gatewayName,.gatewayConfig)
  Quit:$$$ISERR(sc) gateway
  Set sc = ##class(%Net.Remote.Service).ConnectGateway(gatewayConfig,.gateway, path, $$$YES)
  Quit gateway
}


My workflow when developing a new Ensemble-Java integration looks like this:

  1. Write Java code.
  2. Test Java code in Java IDE.
  3. Import Java code.
  4. Test it from our Terminal.
  5. Write operation/service
  6. Test operation/service from Ensemble.

Addendum.

Do not use $lb() and com.intersys.jdbc.CacheListReader /com.intersys.jdbc.CacheListBuilder classes - use $c(1) delimited strings instead. Works way better and faster.

Neerav,

Looks like you got what you needed from the provided example put..

But I think what you will find is it is not as straight forward with an HTTP PUT to S3... and should consider wrapping up the java SDK in a class operation that extends the Java Gateway ( EnsLib.JavaGateway.AbstractOperation) and use the AWS SDK for the PUT.

You need to build a request and response object to pass around the production to send to the operation it for the things the S3 PUT needs, but you get the encryption you need, credentials for the call,  proper encryption (--sse AES256) etc...

Also consider an IAM instance profile on the box and bypass housing the credentials at all anywhere on the box/container/instance/production item.

This is an example below of what you are up against (not complete, apologies) but just to give you an idea before you go down the HTTP Client route.  Its difficult to get a working example exported for your use with the gateway in the mix, but I will see what I can do.

#Dim tCredentials As com.amazonaws.auth.AWSCredentials

    Set tBucketName = pRequest.BucketName

    Set tSC = ..CreateTemporaryFile(pRequest,.tTempFilename) Quit:$$$ISERR(tSC) tSC

    Set tSC = ..GetConnection(.tJavaGateway) Quit:$$$ISERR(tSC) tSC

    // Handle credentials (from OS ENVIRONMENT) OR DONT DO THIS AT ALL AND APPLY AN INSTANCE PROFILE

    Set tCredentialsProviderObj = ##class(com.amazonaws.auth.EnvironmentVariableCredentialsProvider).%New(tJavaGateway)

    Set tCredentials = tCredentialsProviderObj.getCredentials()

    // Create the S3 client Java Object

    Set tS3Client = ##class(com.amazonaws.services.s3.AmazonS3Client).%New(tJavaGateway,tCredentialsProviderObj)

    // Create the FileInputStream that the Java SDK requires

    Set tFileInputStream = ##class(java.io.FileInputStream).%New(tJavaGateway,tTempFilename)

    Set tJavaFile = ##class(java.io.File).%New(tJavaGateway,tTempFilename)

    // Create metadata for the bucket object

    Set tObjectMetadata = ##class(com.amazonaws.services.s3.model.ObjectMetadata).%New(tJavaGateway)

    Set tDocumentClassName = pRequest.DocumentClassName  // more stuff needed from request

    Do tObjectMetadata.setContentType("text/plain")

    Set tAesAlgorithm = ##class(com.amazonaws.services.s3.model.ObjectMetadata).%GetParameter("AESu256uSERVERuSIDEuENCRYPTION")

    Do tObjectMetadata.setSSEAlgorithm(tAesAlgorithm)

    // Create a PutObjectRequest, at the root or in a key?

    Set prefixKey = pRequest.Key //from the request

    If (pRequest.Prefix '= "") {

        Set prefixKey = pRequest.Prefix _ pRequest.Key

    }

    Set tPutObjectRequest = ##class(com.amazonaws.services.s3.model.PutObjectRequest).%New(tJavaGateway, tBucketName, prefixKey, tJavaFile)

    Do tPutObjectRequest.setMetadata(tObjectMetadata)

    // Put the object to S3

    #Dim tPutObjectResult As com.amazonaws.services.s3.model.PutObjectResult

    Set tPutObjectResult = tS3Client.putObject(tPutObjectRequest)

    Kill tS3Client

    Quit ..CleanupTemporaryFile(tTempFilename)

Is that a AWS java library import? Or did you write your own Java wrapper for AWS library?

Any reasons you prefer this approach to writing Java library wrapper and importing that?

Thanx Ron, 

I am not so experienced with AWS SDK Or Java SDK for Cache.  I do have experience in Java but not with Cache imports etc. And have just recently learnt about AWS S3. will def study the AWS SDK

Appreciate your code example. However on the ensemble /cache front I do not know what does this code is doing or where to put it or from where to start. 
- #dim isn't this how you declare variables in VB?

- Is the above example using REST at all?

Pls guide. 

This is how its done.  Finally got it resolved with the help of Intersystems itself.

Goes without saying that people who use it have to put their own credentials

Note : We made a root user on AWS with Admin access 

<Routine name="MMM.TestS3" type="MAC" languagemode="0" timestamp="63022,83214.668057"><![CDATA[

GET(tFilename)

              set tBucket = "3mpoc-intersystems"

              set tServer = tBucket_".s3.amazonaws.com"

              set tPort = 80

              set tAccessKeyId = "AKIAIBOM7V5HFWYJ742A"

              set tSecretAccessKey = "XSw/x2SGMVEovlpYC31q2QTXI1BcEOpYJ5fOsCav"

 

              set tStart = $ZH

              kill tRequest

              set tRequest = ##class(%Net.HttpRequest).%New()                         

              set tRequest.Server = tServer                                       

              set tRequest.Port = tPort

 

              set tDateH = $ZTS

              set tWeekDay = $P($ZDT(tDateH,11)," ",1)

              set tDate = tWeekDay_", "_$ZDT(tDateH,2,1)_" GMT"

              //w !,"Date: "_tDate

              set tRequest.Date = tDate

 

              set tContentMD5 = ""

              set tContentType = ""

 

              set tStringToSign = "GET"_$C(10)_tContentMD5_$C(10)_tContentType_$C(10)_tDate_$C(10)_"/"_tBucket_"/"_tFilename

              //w !,"StringToSign (before UTF-8 conversion): "_tStringToSign

              set tStringToSignUTF8 = $ZCONVERT(tStringToSign,"O","UTF8")

              //w !,"StringToSign (after UTF-8 conversion): "_tStringToSign

              set tSignature = ##class(%SYSTEM.Encryption).HMACSHA1(tStringToSignUTF8,tSecretAccessKey)

              //w !,"Signature (before Base64 encoding): "_tSignature

              set tSignatureBase64 = ##class(%SYSTEM.Encryption).Base64Encode(tSignature)

              //w !,"Signature (after Base64 encoding): "_tSignature

              set tAuthorization = "AWS "_tAccessKeyId_":"_tSignatureBase64

              //w !,"Authorization: "_tAuthorization

              set tRequest.Authorization = tAuthorization

 

              set tURL =  "/"_tFilename

              w !,"GET"

              w !,tURL

              w !

              w !

              set tSC = tRequest.Get(tURL)

              //do tRequest.HttpResponse.OutputHeaders()

              do tRequest.HttpResponse.OutputToDevice()

              set tStop = $ZH

              set tElapsed = (tStop - tStart)

              w !,"Elapsed time: "_tElapsed_" seconds"

              w !

 

              quit

 

PUT(tFilename)

              set tBucket = "3mpoc-intersystems"

              set tServer = tBucket_".s3.amazonaws.com"

              set tPort = 80

              set tAccessKeyId = "AKIAIBOM7V5HFWYJ742A"

              set tSecretAccessKey = "XSw/x2SGMVEovlpYC31q2QTXI1BcEOpYJ5fOsCav"

 

              set tStart = $ZH

              kill tRequest

              set tRequest = ##class(%Net.HttpRequest).%New()                         

              set tRequest.Server = tServer                                       

              set tRequest.Port = tPort

 

              set tDateH = $ZTS

              set tWeekDay = $P($ZDT(tDateH,11)," ",1)

              set tDate = tWeekDay_", "_$ZDT(tDateH,2,1)_" GMT"

              //w !,"Date: "_tDate

              do tRequest.SetHeader("Date",tDate)

 

              set tContentMD5 = ""

              set tContentType = "text/xml; charset=UTF-8"

              do tRequest.SetHeader("Content-Type",tContentType)

 

              set tStringToSign = "PUT"_$C(10)_tContentMD5_$C(10)_tContentType_$C(10)_tDate_$C(10)_"/"_tBucket_"/"_tFilename

              //w !,"StringToSign (before UTF-8 conversion): "_tStringToSign

              set tStringToSignUTF8 = $ZCONVERT(tStringToSign,"O","UTF8")

              //w !,"StringToSign (after UTF-8 conversion): "_tStringToSign

              set tSignature = ##class(%SYSTEM.Encryption).HMACSHA1(tStringToSignUTF8,tSecretAccessKey)

              //w !,"Signature (before Base64 encoding): "_tSignature

              set tSignatureBase64 = ##class(%SYSTEM.Encryption).Base64Encode(tSignature)

              //w !,"Signature (after Base64 encoding): "_tSignature

              set tAuthorization = "AWS "_tAccessKeyId_":"_tSignatureBase64

              //w !,"Authorization: "_tAuthorization

              set tRequest.Authorization = tAuthorization

 

              set tXMLStream = ##class(%FileCharacterStream).%New()

              set tXMLStream.Filename = "C:\Users\moulcker\Documents\InterSystems\Customers\3M\EPRS\essay.txt"

              do tXMLStream.Rewind()

              set tSC = tRequest.EntityBody.CopyFrom(tXMLStream)

 

              set tURL =  "/"_tFilename

              w !,"PUT"

              w !,tURL

              w !

              w !

              set tSC = tRequest.Put(tURL)

              do tRequest.HttpResponse.OutputToDevice()

              set tStop = $ZH

              set tElapsed = (tStop - tStart)

              w !,"Elapsed time: "_tElapsed_" seconds"

              w !

 

              quit

 

DELETE(tFilename)

              set tBucket = "3mpoc-intersystems"

              set tServer = tBucket_".s3.amazonaws.com"

              set tPort = 80

              set tAccessKeyId = "AKIAIBOM7V5HFWYJ742A"

              set tSecretAccessKey = "XSw/x2SGMVEovlpYC31q2QTXI1BcEOpYJ5fOsCav"

 

              set tStart = $ZH

              kill tRequest

              set tRequest = ##class(%Net.HttpRequest).%New()                         

              set tRequest.Server = tServer                                       

              set tRequest.Port = tPort

 

              set tDateH = $ZTS

              set tWeekDay = $P($ZDT(tDateH,11)," ",1)

              set tDate = tWeekDay_", "_$ZDT(tDateH,2,1)_" GMT"

              //w !,"Date: "_tDate

              set tRequest.Date = tDate

 

              set tContentMD5 = ""

              set tContentType = ""

 

              set tStringToSign = "DELETE"_$C(10)_tContentMD5_$C(10)_tContentType_$C(10)_tDate_$C(10)_"/"_tBucket_"/"_tFilename

              //w !,"StringToSign (before UTF-8 conversion): "_tStringToSign

              set tStringToSignUTF8 = $ZCONVERT(tStringToSign,"O","UTF8")

              //w !,"StringToSign (after UTF-8 conversion): "_tStringToSign

              set tSignature = ##class(%SYSTEM.Encryption).HMACSHA1(tStringToSignUTF8,tSecretAccessKey)

              //w !,"Signature (before Base64 encoding): "_tSignature

              set tSignatureBase64 = ##class(%SYSTEM.Encryption).Base64Encode(tSignature)

              //w !,"Signature (after Base64 encoding): "_tSignature

              set tAuthorization = "AWS "_tAccessKeyId_":"_tSignatureBase64

              //w !,"Authorization: "_tAuthorization

              set tRequest.Authorization = tAuthorization

 

              set tURL =  "/"_tFilename

              w !,"DELETE"

              w !,tURL

              w !

              w !

              set tSC = tRequest.Send("DELETE",tURL)

              //do tRequest.HttpResponse.OutputHeaders()

              //do tRequest.HttpResponse.OutputToDevice()

              set tStop = $ZH

              set tElapsed = (tStop - tStart)

              w !,"Elapsed time: "_tElapsed_" seconds"

              w !

 

              quit

 

LIST

              set tBucket = "3mpoc-intersystems"

              set tServer = tBucket_".s3.amazonaws.com"

              set tPort = 80

              set tAccessKeyId = "AKIAIBOM7V5HFWYJ742A"

              set tSecretAccessKey = "XSw/x2SGMVEovlpYC31q2QTXI1BcEOpYJ5fOsCav"

 

              set tStart = $ZH

              kill tRequest

              set tRequest = ##class(%Net.HttpRequest).%New()                         

              set tRequest.Server = tServer                                       

              set tRequest.Port = tPort

 

              set tDateH = $ZTS

              set tWeekDay = $P($ZDT(tDateH,11)," ",1)

              set tDate = tWeekDay_", "_$ZDT(tDateH,2,1)_" GMT"

              w !,"Date: "_tDate

              set tRequest.Date = tDate

 

              set tContentMD5 = ""

              set tContentType = ""

 

              set tStringToSign = "GET"_$C(10)_tContentMD5_$C(10)_tContentType_$C(10)_tDate_$C(10)_"/"_tBucket_"/"

              //w !,"StringToSign (before UTF-8 conversion): "_tStringToSign

              set tStringToSignUTF8 = $ZCONVERT(tStringToSign,"O","UTF8")

              //w !,"StringToSign (after UTF-8 conversion): "_tStringToSign

              set tSignature = ##class(%SYSTEM.Encryption).HMACSHA1(tStringToSignUTF8,tSecretAccessKey)

              //w !,"Signature (before Base64 encoding): "_tSignature

              set tSignatureBase64 = ##class(%SYSTEM.Encryption).Base64Encode(tSignature)

              //w !,"Signature (after Base64 encoding): "_tSignature

              set tAuthorization = "AWS "_tAccessKeyId_":"_tSignatureBase64

              //w !,"Authorization: "_tAuthorization

              set tRequest.Authorization = tAuthorization

 

              set tURL =  "/"

              w !,"GET"

              w !,tURL

              w !

              w !

              set tSC = tRequest.Get(tURL)

              //do tRequest.HttpResponse.OutputHeaders()

              do tRequest.HttpResponse.OutputToDevice()

              set tStop = $ZH

              set tElapsed = (tStop - tStart)

              w !,"Elapsed time: "_tElapsed_" seconds"

              w !

 

              quit

 

]]></Routine>

As long ago as October 2007, we released an S3 Client for Cache and announced it in the ISC Google Group.

It's still available and should probably still work: http://gradvs1.mgateway.com/download/s3Client.zip

BTW I hope that's not your actual AWS credentials that you've posted in your code example!

No obviously those are just test credentials which are no use to anyone