Can you try the code I posted above, substituting appropriate paths/filenames in the calls to the LinkToFile() methods? Any file will do for the in stream, as long as file/directory permissions permit. This at least would tell us whether the issue is with the key file or the JWT you're attempting to encrypt.

I've tried this on I4H 2023.1 and Health Connect 2021.1.2 and the RSASHASign() method has not failed to generate a signature unless the key was passphrase-protected or not readable (due to file ownership/permissions) by the process opening it.

I just tried RSASHASign() myself and a signature was returned, using your call:
 

Set in=##class(%Stream.FileCharacter).%New()
Do in.LinkToFile("/home/jeff/sample.jwt")
Set token=in.Read()
Set key=##class(%Stream.FileCharacter).%New()
Do key.LinkToFile("/home/jeff/sample.pem")
Set secret=key.Read()
Set sig=##class(%SYSTEM.Encryption).RSASHASign(512,token,secret)

The return value will be the signature only, though, not a signed JWT. If you want the latter, see the ObjectToJWT() method in  %OAuth2.JWT - InterSystems IRIS for Health 2021.1.

The reason this 2-year-old thread floated to the top is because I found it researching an issue I had encountered with EnsLib.HTTP.GenericService.. I wanted to "pass through" a status code I had set as a property of a response message in a business process. While a solution was buried in the thread, the OP claimed it did not work in his case. I'm not sure why; I used a slight variation of that solution with success and simply felt the variation was worth sharing.

Just in case someone stumbles into this thread looking for an answer (as I did) ...

Assuming the vanilla, un-extended EnsLib.HTTP.GenericService is the service handling the request, any response it receives from a business process needs to be an EnsLib.HTTP.GenericMessage. %Net.HttpResponse is not needed, nor is a CSP layer required.

The service requires that a stream is attached to the message; the stream doesn't need to contain anything.

The response message is composed in the BP something like this:

Set rstream = ##class(%Stream.GlobalCharacter).%New()
// Optional body content
Do rstream.Write("<HTML><HEAD>Uh oh.</HEAD><BODY><BR><STRONG>Error: Invalid Patient ID</STRONG></BODY></HTML>")
// Provide a stream object, empty is fine
Set response = ##class(EnsLib.HTTP.GenericMessage).%New(rstream)
// This works as expected in I4H 2023.1
Do response.HTTPHeaders.SetAt("HTTP/1.1 400 Bad Request","StatusLine")
// if you're providing a payload ...
Do response.HTTPHeaders.SetAt("text/html; charset=utf-8","Content-Type")

I've verified that it works as coded above, using curl:

< HTTP/1.1 400 Bad Request
< Content-Type: text/html; charset=utf-8
< Content-Length: 99
<
* Connection #0 to host iristest.local left intact
<HTML><HEAD>Uh oh.</HEAD><BODY><BR><STRONG>Error: Invalid Patient ID</STRONG></BODY></HTML>

With #2 (at least for me anyway), the issue seems to be related to running iris session when using the Windows version of ssh.exe (called from VS Code, configured in settings under node terminal.integrated.profiles.windows). Home and End work normally at the Linux shell prompt, but when running iris session the effect is that either key produces the same result as pressing the Enter key. The current command is executed and a new IRIS prompt is generated.

It doesn't seem to be a VS Code problem so much as an ISC problem, at least on Windows.

This should work (no looping required):

I'm using the parenthesis syntax with the Matches() function to locate a pattern of any number of punctuation characters (.P) followed by 8 numeric characters (8N) followed by any number of any character (.E).

The parenthesis syntax returns the repeating values in the form "<><><20230512191543><>" where <> represents an empty iteration of the repeating field (and fortunately qualifies as a punctuation character).

According to a StackOverflow thread I just read, the connection url below is purported to work on Linux and authenticate with the MS JDBC driver:

jdbc:sqlserver://[server]:[port];database=[db];trustServerCertificate=true;integratedSecurity=true;user=[user without domain];password=[pw];authenticationScheme=NTLM;domain=[domain];authentication=NotSpecified

Web services normally use an HTTP status code; for example, an ACK would be 200 OK for REST/HTTP and would be available through the %Net.HttpResponse Object in the StatusCode/StatusLine properties. SOAP usually provides some sort of payload along with the status code, and that would be found in the Data property. The type of response would likely be identified in the source/target system's WSDL for the SOAP interface.

This is something I wrote a long time ago; it extracts all business hosts and their settings. I've learned some things since I wrote it and would probably do a few things differently these days. It should be enough to give you some ideas, though ...

ClassMethod GetConfigs(pProduction As %String = {$G(^Ens.Runtime("Name"),$G(^Ens.Suspended,$G(^Ens.Configuration("csp","LastProduction"))))}, pFile As %String = {$System.Util.GetEnviron("HOME")_"/"_$NAMESPACE_"_hostconfigs.csv"}) As %Status
{
    Set tPrd = ##class(Ens.Config.Production).%OpenId(pProduction)
    Set tOut = ##class(%File).%New()
    Set tOut.Name = pFile
    Set tSC = tOut.Open("RWN")
    if '$$$ISERR(tSC)
    {
        Set tSC = vOut.WriteLine("""Type"",""Name"",""ClassName"",""Adapter"",""Enabled"",""ConfigName"",""ConfigValue""")
    }
    Quit:$$$ISERR(tSC) tSC
    If $ISOBJECT(tPrd)
    {
        For i=1:1:tPrd.Items.Count()
        {
            Set tHost = tPrd.Items.GetAt(i)
            Set tName = tHost.Name
            Set tClassName = tHost.ClassName
            Set tType = $CASE(tHost.BusinessType(),0:"Unknown",1:"Service",2:"Process",3:"Operation",4:"Actor",:"Huh?")
            Set tAdapter = $CLASSMETHOD(tClassName,"%GetParameter","ADAPTER")
            Set tEnabled = tHost.Enabled
            Set tCategory = tHost.Category
            Set tLine = """"_tType_""","""_tName_""","""_tClassName_""","""_tAdapter_""","""_tEnabled_""","""
            Do tOut.WriteLine(tLine_"Category"","""_tCategory_"""")
            For l=1:1:tHost.Settings.Count()
            {
                Set tCfg = tHost.Settings.GetAt(l)
                Set tCfgName = tCfg.Name
                Set tCfgVal = tCfg.Value
                Set tSC = vOut.WriteLine(tLine_tCfgName_""","""_tCfgVal_"""")
                Return:$$$ISERR(tSC) tSC
            }
        }
        Do tOut.Close()
    }
    Else
    {
        Return $$$ERROR(0,"Production Not Found in this namespace")
    }
    Return $$$OK
}