AWS API calls

Primary tabs

REST API, Caché

This question is about calling AWS REST APIs. Based on:

http://docs.aws.amazon.com/general/latest/gr/sigv4_signing.html

AWS requires REST clients to call their APIs using Signature Version 4 which in case you don't know what I am talking about is a pain in the neck.  Here comes the question:

Has anybody, by any chance implemented the v4 signing alg. in COS? If yes, would she or he have the kind heart to share?

Thanks,

Chris

 

 

  • 0
  • 0
  • 421
  • 9
  • 2

Answers

Chris
 
I've not looked at the V4 Signature rules in detail, but the V1 & V2 rules were implemented in this SimpleDB Client that I created years ago for use with both AWS SimpleDB and my M/DB clone:

https://s3.amazonaws.com/rtweeddata/simpleDBClient.xml

Not sure if that helps, but feel free to adapt as needed if it provides a useful starting point - start around line 426 which is the createHTTPRequest() function that creates a signed HTTP request

Rob

Thanks Rob. Will have a look.

Chris,

Did you ever get anywhere with this. I had mine working until I tried uploading a character stream with content-type of text/csv and now it won't sign properly

Hey Dave,

I encountered similar issues with 'text/*' ( text/plain, text/csv, etc) Content-Type header values when I included them in the v4 signed canonical request, and found (in my case) that this is due to Caché appending the charset based on the instance's localization settings (string "; charset=UTF-8") for this header value, eg "Content-Type=text/csv; charset=UTF-8". That likely means you're passing "text/csv" into your signing algorithm to derive your signature but the actual request you're sending has "text/csv; charset=UTF-8", hence the signature failure on the AWS end.

As per this doc,

  • The ContentCharset property controls the desired character set for any content of the request if the content is of type text (text/html or text/xml for example). If you do not specify this property, Caché uses the default encoding of the Caché server.

    Note:

    If you set this property, you must first set the ContentType property.

  • The NoDefaultContentCharset property controls whether to include an explicit character set for content of type text if you have not set the ContentCharset property. By default, this property is false.

Since this isn't one of the headers that needs to be included in AWS v4 signing for the authorisation header,  I just removed this from the list to sign (while still passing it in the actual HTTP request) so I could retain the default charset without too much fuss (I'm not too worried about an attack here, since I'm signing payloads), but the other options (as far as I can tell) would be to pass the same full string to your signing function (including the charset), or prevent charset from being included in Content-Type via that NoDefaultContentCharset property.

Cheers

Joe

Hi Joe,

would you mind sharing some of your code (minus API key values :-) ) for signing AWS REST calls? I have almost scratched my head off trying to find out why things still aren't working when my StringToSign and SigningKey appear to be correct, but the hash I create from them isn't. I can even reproduce (aka "make the same mistake") using the sample Python code AWS provides.

Relevant but not working (and therefore less relevant) code:

Property AWSAccessKeyId As %String [ InitialExpression = "AKIDEXAMPLE" ];

Property AWSSecretAccessKey As %String [ InitialExpression = "wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY" ];

Property Region As %String [ InitialExpression = "us-east-1" ];

Property Service As %String [ InitialExpression = "iam" ];

Method BuildAuthorizationHeader(pHttpRequest As %Net.HttpRequest, pOperation As %String = "", pURL As %String = "", Output pAuthorizationHeader As %String, pVerbose As %Boolean = 0) As %Status
{
set tSC = $$$OK
try {
if ..AWSAccessKeyId="" {
set tSC = $$$ERROR($$$GeneralError, "No AWS Access Key ID provided")
quit
}
if ..AWSSecretAccessKey="" {
set tSC = $$$ERROR($$$GeneralError, "No AWS Secret Access Key provided")
quit
}

set tAMZDateTime = $tr($zdatetime($h,8,7),":") // 20190319T151009Z
//set tAMZDateTime = "20150830T123600Z" // for AWS samples
set tAMZDate = $e(tAMZDateTime,1,8) // 20190319
set tLineBreak = $c(10)

set pOperation = $$$UPPER(pOperation)

// ensure the right date is set
do pHttpRequest.SetHeader("X-Amz-Date", tAMZDateTime)


// ************* TASK 1: CREATE A CANONICAL REQUEST *************
// http://docs.aws.amazon.com/general/latest/gr/sigv4-create-canonical-request.html

// Step 1 is to define the verb (GET, POST, etc.) -- inferred from pOperation

// Step 2: Create canonical URI--the part of the URI from domain to query 
// string (use '/' if no path)
set tCanonicalURL = $s($e(pURL,1)="/":pURL, $e(pURL,1)'="":"/"_pURL, 1:"/"_pHttpRequest.Location)


// Step 3: Create the canonical query string. In this example (a GET request),
// request parameters are in the query string. Query string values must
// be URL-encoded (space=%20). The parameters must be sorted by name.
// For this example, the query string is pre-formatted in the request_parameters variable.
set tQueryString = $piece(tCanonicalURL,"?",2,*)
set tCanonicalURL = $piece(tCanonicalURL,"?",1)

// TODO: append pHttpRequest.Params content?
// TODO: sort params!

// Step 4: Create the canonical headers and signed headers. Header names
// must be trimmed and lowercase, and sorted in code point order from
// low to high. Note that there is a trailing \n.
set tCanonicalHeaders = "content-type:" _ pHttpRequest.ContentType _ tLineBreak
_ "host:" _ pHttpRequest.Server _ tLineBreak
_ "x-amz-date:" _ tAMZDateTime _ tLineBreak

// Step 5: Create the list of signed headers. This lists the headers
// in the canonical_headers list, delimited with ";" and in alpha order.
// Note: The request can include any headers; canonical_headers and
// signed_headers lists those that you want to be included in the 
// hash of the request. "Host" and "x-amz-date" are always required.
set tSignedHeaders = "content-type;host;x-amz-date"

// Step 6: Create payload hash (hash of the request body content). For GET
// requests, the payload is an empty string ("").
if (pOperation = "GET") {
set tPayload = ""
else {
// TODO
set tPayload = ""
}
set tPayloadHash = ..Hex($SYSTEM.Encryption.SHAHash(256,$zconvert("","O","UTF8")))


// Step 7: Combine elements to create canonical request
set tCanonicalRequest = pOperation _ tLineBreak
_ tCanonicalURL _ tLineBreak
_ tQueryString _ tLineBreak
_ tCanonicalHeaders _ tLineBreak 
_ tSignedHeaders _ tLineBreak
_ tPayloadHash
set tCanonicalRequestHash = ..Hex($SYSTEM.Encryption.SHAHash(256, tCanonicalRequest))

w:pVerbose !!,"Canonical request:",!,$replace(tCanonicalRequest,tLineBreak,"<"_$c(13,10)),!!,"Hash: ",tCanonicalRequestHash,!

// ************* TASK 2: CREATE THE STRING TO SIGN*************
// Match the algorithm to the hashing algorithm you use, either SHA-1 or
// SHA-256 (recommended)
set tAlgorithm = "AWS4-HMAC-SHA256"
set tCredentialScope = tAMZDate _ "/" _ ..Region _ "/" _ ..Service _ "/" _ "aws4_request"
set tStringToSign = tAlgorithm _ tLineBreak 
_ tAMZDateTime _ tLineBreak 
_ tCredentialScope _ tLineBreak
_ tCanonicalRequestHash
w:pVerbose !!,"String to sign:",!,$replace(tStringToSign,tLineBreak,$c(13,10)),!

// ************* TASK 3: CALCULATE THE SIGNATURE *************
// Create the signing key using the function defined above.
// def getSignatureKey(key, dateStamp, regionName, serviceName):
     set tSigningKey = ..GenerateSigningKey(tAMZDate)
     w:pVerbose !!,"Signing key:",!,..Hex(tSigningKey),!

// Sign the string_to_sign using the signing_key
set tSignature = ..Hex($SYSTEM.Encryption.HMACSHA(256, tStringToSign, tSigningKey))


// ************* TASK 4: ADD SIGNING INFORMATION TO THE REQUEST *************
// The signing information can be either in a query string value or in 
// a header named Authorization. This code shows how to use a header.
// Create authorization header and add to request headers
set pAuthorizationHeader = tAlgorithm _ " Credential=" _ ..AWSAccessKeyId _ "/" _ tCredentialScope _ ", SignedHeaders=" _ tSignedHeaders _ ", Signature=" _ tSignature
w:pVerbose !!,"Authorization header:",!,pAuthorizationHeader,!!
b
catch (ex) {
set tSC = ex.AsStatus()
}
quit tSC
}

Method GenerateSigningKey(pDate As %String) As %String
{
set kDate = $SYSTEM.Encryption.HMACSHA(256, pDate, $zconvert("AWS4" _ ..AWSSecretAccessKey,"O","UTF8"))
    //w !,"kDate: ",..Hex(kDate)
    set kRegion = $SYSTEM.Encryption.HMACSHA(256, ..Region, kDate)
    //w !,"kRegion: ",..Hex(kRegion)
    set kService = $SYSTEM.Encryption.HMACSHA(256, ..Service, kRegion)
    //w !,"kService: ",..Hex(kService)
    set tSigningKey = $SYSTEM.Encryption.HMACSHA(256, "aws4_request", kService)
    //w !,"kSigning: ",..Hex(tSigningKey),! 
quit tSigningKey
}

ClassMethod Hex(pRaw As %String) As %String [ Internal ]
{
set out="", l=$l(pRaw)
for = 1:1:{
set out=out_$zhex($ascii(pRaw,i))
}
quit $$$LOWER(out)
}

ClassMethod SimpleTest() As %Status
{
set tSC = $$$OK
try {
set tAdapter = ..%New()
set tAdapter.AWSAccessKeyId = "use yours"
set tAdapter.AWSSecretAccessKey = "not mine"

set tAdapter.Region = "us-east-1", tAdapter.Service = "iam"

set tRequest = ##class(%Net.HttpRequest).%New()
set tRequest.ContentType = "application/x-www-form-urlencoded"
set tRequest.ContentCharset = "utf-8"
set tRequest.Https = 1
set tRequest.SSLConfiguration = "SSL client" // simple empty SSL config
set tRequest.Server = "iam.amazonaws.com"

set tURL = "/?Action=ListUsers&Version=2010-05-08"

set tSC = tAdapter.BuildAuthorizationHeader(tRequest, "GET", tURL, .tAuthorization, 1)
quit:$$$ISERR(tSC)
set tRequest.Authorization = tAuthorization

set tSC = tRequest.Get(tURL)
quit:$$$ISERR(tSC)

Do tRequest.HttpResponse.OutputToDevice()

catch (ex) {
set tSC = ex.AsStatus()
}
write:$$$ISERR(tSC) !!,$system.Status.GetErrorText(tSC),!
quit tSC
}

I am attempting the same thing currently. At first I used the same approach for converting to hex: 

ClassMethod Hex(pRaw As %String) As %String [ Internal ]
{
set out="", l=$l(pRaw)
for = 1:1:{
set out=out_$zhex($ascii(pRaw,i))
}
quit $$$LOWER(out)
}

But have discovered that the Hex function above is not correct. Instead use the system function,  ##class(%xsd.hexBinary).LogicalToXSD.

##class(%xsd.hexBinary).LogicalToXSD

The conversion to hex using the custom method was removing zeroes in the string. That had a cascading effect, the canonical request hash will be wrong, and consequently so will the string to sign and then ultimately the signature. Hope this helps anyone else that might be struggling. 

Here is some documentation specifically for S3:

https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-header-based-auth.html

Here's a (very simple) working example of uploading to an AWS S3 bucket. I used essentially the same approach that Cliff originally used to generate the hex. I didn't experience the same problem, but I may have just been lucky and not had any zeroes in the value I'm converting.

<?xml version="1.0" encoding="UTF-8"?>
<Export generator="Cache" version="25" zv="Cache for Windows (x86-64) 2017.2.1 (Build 801_3U)" ts="2018-10-04 08:14:32">
<Class name="Demo.Cloud.Storage.Util">
<Super>%RegisteredObject</Super>
<TimeChanged>64925,29407.276303</TimeChanged>
<TimeCreated>64924,56047.115234</TimeCreated>

<Method name="TestS3Upload">
<ClassMethod>1</ClassMethod>
<Implementation><![CDATA[
    set tAccessKeyId="[Shh... it's a secret]"
    set tSecretAccessKey="[Shh... it's a secret]"
    set tAWSRegion="us-east-1"
    set tAWSService="s3"
    set tHost="s3.us-east-1.amazonaws.com"
    set tPort="443"
    set tBucket="MyBucket12345"
    set tContentType="text/plain; charset=UTF-8"

    set tSSLConfig="AWS"
    set tUseHTTPS=1
    
    set tContentStream=##class(%Stream.TmpCharacter).%New()
    do tContentStream.Write("this is some test content oh yeah!")
    do tContentStream.Rewind()

    set tHorolog=$h
    set tDateKey=$System.Encryption.HMACSHA(256,$ZD(tHorolog,8),"AWS4"_tSecretAccessKey)
    set tDateRegionKey=$System.Encryption.HMACSHA(256,tAWSRegion,tDateKey)
    set tDateRegionServiceKey=$System.Encryption.HMACSHA(256,tAWSService,tDateRegionKey)
    set tSigningKey=$System.Encryption.HMACSHA(256,"aws4_request",tDateRegionServiceKey)
        
    set tHashedPayload=..BytesToHex($System.Encryption.SHAHashStream(256,tContentStream,.tSC))
    
    set tTmpTime=$ZDT(tHorolog,2,7)
    set tHeaderDate=$ZD(tHorolog,11)_", "_$E(tTmpTime,1,11)_" "_$E(tTmpTime,13,*-1)_" GMT"
    set tAmzDate=$TR($zdt(tHorolog,8,7),":","")

    set tKeyName="test_"_tHorolog_".txt"

    set tCanonicalURI="/"_tBucket_"/"_tKeyName
    set tCanonicalURI=$zconvert(tCanonicalURI,"O","URL")

    set tHttpVerb="PUT"
    set tCanonicalHeaders="content-length:"_tContentStream.Size_$c(10)_"content-type:"_tContentType_$c(10)_"host:"_tHost_$c(10)_"x-amz-content-sha256:"_tHashedPayload_$c(10)_"x-amz-date:"_tAmzDate_$c(10)
    set tSignedHeaders="content-length;content-type;host;x-amz-content-sha256;x-amz-date"
    set tCanonicalRequest=tHttpVerb_$c(10)_tCanonicalURI_$c(10,10)_tCanonicalHeaders_$c(10)_tSignedHeaders_$c(10)_tHashedPayload
    set tStringToSign="AWS4-HMAC-SHA256"_$c(10)_$TR($zdt(tHorolog,8,7),":","")_$c(10)_$zd(tHorolog,8)_"/"_tAWSRegion_"/"_tAWSService_"/aws4_request"_$c(10)_..BytesToHex($System.Encryption.SHAHash(256,tCanonicalRequest))
    set tSignature=..BytesToHex($System.Encryption.HMACSHA(256,tStringToSign,tSigningKey))    
    set tAuthHeader="AWS4-HMAC-SHA256 Credential="_tAccessKeyId_"/"_$zd(tHorolog,8)_"/"_tAWSRegion_"/"_tAWSService_"/aws4_request,SignedHeaders="_tSignedHeaders_",Signature="_tSignature

    set tHReq=##class(%Net.HttpRequest).%New()
    set tHReq.Server=tHost
    set tHReq.Port=tPort
    set tHReq.SSLConfiguration=tSSLConfig
    set tHReq.Https=tUseHTTPS

    do tHReq.SetHeader("Authorization",tAuthHeader)
    do tHReq.SetHeader("Content-Type","text/plain")
    do tHReq.SetHeader("Content-Length",tContentStream.Size)
    do tHReq.SetHeader("Host",tHost)
    do tHReq.SetHeader("x-amz-content-sha256",tHashedPayload)
    do tHReq.SetHeader("x-amz-date",tAmzDate)

    do tHReq.EntityBody.CopyFrom(tContentStream)

    //do tHReq.OutputHeaders()

    s tSC=tHReq.Put(tCanonicalURI)
    do $System.OBJ.DisplayError(tSC)
    
    write "Status Code:",tHReq.HttpResponse.StatusCode,!
    
    //Do tHReq.HttpResponse.OutputToDevice()
    
    if $isobject(tHReq.HttpResponse.Data) {
        s ^zresp=tHReq.HttpResponse.Data.Read(1000000)
    } else {
        s ^zresp=tHReq.HttpResponse.Data
    }
]]></Implementation>
</Method>

<Method name="BytesList">
<ClassMethod>1</ClassMethod>
<FormalSpec>pBytes:%String</FormalSpec>
<ReturnType>%String</ReturnType>
<Implementation><![CDATA[
    s l=$LISTFROMSTRING(pBytes," ")
    f i=1:1:$LL(l)  s st=$G(st)_$CHAR($ZHEX($LISTGET(l,i)))
    return st
]]></Implementation>
</Method>

<Method name="BytesToHex">
<ClassMethod>1</ClassMethod>
<FormalSpec>pBytes:%String</FormalSpec>
<ReturnType>%String</ReturnType>
<Implementation><![CDATA[
    set tHex=""
    
    for i=1:1:$L(pBytes) {
        set tHexByte=$ZHEX($ASCII($E(pBytes,i)))
        if $L(tHexByte)=1 s tHexByte="0"_tHexByte
        set tHex=tHex_tHexByte
    }
    
    return $ZCONVERT(tHex,"L")
]]></Implementation>
</Method>
</Class>
</Export>

Or it looks like I was handling the zero issue manually:

if $L(tHexByte)=1 s tHexByte="0"_tHexByte

Comments

Just wondering if anyone has put together similar examples for Azure -- preferably for uploading Azure Blobs.

Thanks Anton. Maybe I didn't dig deep enough in that github link but it looks like that is for AWS, not Azure?