Using %CSP.REST class to get value will work. It's a bit weird that %REST.Impl does not have this out of the box though. My guess there's probably an expectation to have an IAM in front of this (e.g. Kong or MS Azure)

Hi Rich. Sorry for the delay. Thanks for your input. In answer to some of your questions:

Re: Use of messages/tracing.

The interface would just produce too much data. For instance, if I had to call an API and retrieve a full set of data every night and then paginate through it, then every http response would have to be passed back to process or another operation and get attached to the session and become part of massive view trace that takes an age to render and causes the data namespace messages to expand too much.

What happens to the data if the process aborts in the middle?

As all our adapters have AlertGroups, this will just use the Ens.Alert an email will be generated for someone to look at so we have some visibility.

Would missing any of the data being pulled have a negative effect on the business?

In this instance no because we are making a copy of some reporting data and the business can just check the source if needed. As mentioned, at minimum we catch any exceptions and alert a technical contact. Obviously with an API pull to SQL, its possible to get non-2XX status code that ruins a whole run. We would probably use the http adapter settings here to manage that but because we are not getting a delta unfortunately, only the latest run is ever important.

Is there any concern over data lineage for security or external auditing? 

Generally no, because data from the http response is just passed into a SQL table. The actual transformation of the data happens after it reaches its destination via reporting engine external to IRIS.

Re: Temporary holding table.

I previously did this to produce 5GB of json for a MongoDB parser, it works well but you have to compact the namespace after kill the globals or just live with reserving an amount of space for subsequent runs in the data namespace.
Yes, a global mapping to IRISTEMP could work to get around the journalling. My only issue is it looks like I'd have to add this mapping manually to my namespace in every environment before I promote my code. Is there are way of do this via method so I can script it?

Re: Batching, this would only run once a day, however I'm generating a run identifier so should be unique enough.

Hi Dmitry

Ahhh. I can see why my loop with the find wasn't working as I expected. Thank you so much for providing this example.

Dan

Hi. Thanks for your tips! Though it has to be Mod N version which is an extension that supports non-numerical strings. 

I guess I can use a proxy class to get the result but it would be nice to do this natively in Cache/MUMPS. What I will say is this: the need to do this it is going to be very important for others I expect also. A bit more info about why it is needed is below, with the character set needed to create codepoints from.

Link

Ok I got your method working now also. I can see the benefit of having the shell logs captured. It's great to have two options! Thanks.

So it turns out my original code and your suggested version worked after all, just needed a server reboot for the PATH to take affect! I just need to sort out user permissions around the account that can use $ZF(-100) and I think I'm all good!

Anyway, I thought i'd just write up my a way of doing this in case it's helpful to anyone.

Below is a REST class that takes a list of files in array and it will convert them to Basic64 using a temp path. Example payload in the code, there's few parameters you can mess around with.

In my actual code I get paths from a SOAP interfaces and need to convert the files back to website. Therefore I am attempting to do this at the last point so it doesn't produce audits/messages. I also don't want exceptions to ever go back to the client so I will always force 1x1 image to rendered but I've added the errors back into this example.

Include %occInclude /// <PRE style="font-size:0.8em;">  
/// ************************************************************************************************************************************
/// Class Name  : Demo.REST.Server
/// Description : Demo REST Service   
/// ************************************************************************************************************************************
/// </PRE>
/// 
/// Please note there are some limitations as string max is 3,641,144 characters. This is addressed in newer IRIS versions, see links below:
/// https://community.intersystems.com/post/size-limitation-tojson-large-stream
/// https://community.intersystems.com/post/cache-how-change-image-file-size-when-saving-cache-database-table
/// https://community.intersystems.com/post/how-resize-image-classmethod
/// 
Class Demo.REST.Server Extends %CSP.REST
{ Parameter CONVERTINPUTSTREAM = 1; Parameter CHARSET = "utf-8"; Parameter CONTENTTYPE = "application/json"; XData UrlMap
{
<Routes>
<Route Url="/api/v2/published/images/toBase64" Method="POST" Call="PostConvertImages"/>
</Routes>
} /// Example:
/// {
///    "Images": [
///        "D:\\PATH\1221701739_20190716134351965_1_0.jpg"
///    ],
///    "TempOutputDir": "D:\\PATH\\TEMP\\",
///    "UseResize": true,
///    "RemoveTempFiles": true,
///    "Dimensions":
///    {
///        "x": "640",
///        "y": "480"
///    }
/// }
/// 
ClassMethod PostConvertImages() As %Status
{
    set tSC=$$$OK
    try
    {
        //read input from REST    
        set sourceJson = {}.%FromJSON(%request.Content)         //setup response
        set tmpStream = ##class(%Stream.GlobalCharacter).%New()
        
        //initial output
        set targetJson = {}
        set targetJson.Images = []         //check tokens
        if (sourceJson.Dimensions.x'="")
        {
            set x = sourceJson.Dimensions.x
        }
        else
        {
            set x = "640"
        }         if (sourceJson.Dimensions.y'="")
        {
            set y = sourceJson.Dimensions.y
        }
        else
        {
            set y = "480"
        }
        
        set dimensions = x_"x"_y         if (sourceJson.TempOutputDir'="")
        {
            set tempDirectory = sourceJson.TempOutputDir
        }
        else
        {
            set tempDirectory = ""
        }         if (sourceJson.UseResize="")
        {
            set useResize = 0
        }
        else
        {
            set useResize = sourceJson.UseResize
        }         if (sourceJson.RemoveTempFiles="")
        {
            set removeTempFiles = 1
        }
        else
        {
            set removeTempFiles = sourceJson.RemoveTempFiles
        }         //Check for null
        if (sourceJson.Images '= "")
        {
            //setup image array 
            set imageArray = []             //iterate through images
            set iterArray = sourceJson.Images.%GetIterator()
            while iterArray.%GetNext(.key, .value)
            {
                try
                {
                    //get file details
                    set fileExtension = $PIECE(value,".",2) 
                    set imageFile = $REPLACE(value,"\\","\") //For escaped paths                     //get file size
                    set checkFile = ##class(%File).%New(imageFile)
                    set sc = checkFile.Open("R") 
                    $$$ThrowOnError(sc)
                    set fileSize = checkFile.SizeGet()
                    do checkFile.Close()                     if (fileSize > 0)
                    {
                        try
                        {
                            //work out what will be in base64 - if over certain number of bytes force resize regardless
                            if ((4 * fileSize / 3) > 300000)
                            {
                                set useResize = 1
                            }                             //check whether to rencode image
                            if ((useResize=1)&&(tempDirectory'=""))
                            {
                                //update flag
                                set tempFile = tempDirectory_$System.Util.CreateGUID()_fileExtension
                                set sc = $ZF(-100,"/STDIN="""_imageFile_""" /STDOUT="""_tempFile_"""","magick","fd:0","-resize",dimensions,"fd:1")
        
                                set convertedImageStream=##class(%Stream.FileBinary).%New()
                                set sc=convertedImageStream.LinkToFile(tempFile)                                 if ((sc = 1)&&(convertedImageStream.Size > 0))
                                {
                                    //encode to base64
                                    do ..Base64EncodeStream(convertedImageStream,.encodedStream)
                                    set line = ""
                                    do encodedStream.Rewind()
                                    while (encodedStream.AtEnd = 0) {
                                        set len = 5700
                                        set line = line_encodedStream.Read(.len)
                                    }
                                    set image = {}
                                    set image.src ="<img src=""data:image/"_$ZSTRIP(fileExtension,"*P")_";charset=utf-8;base64,"_line_""">"
                                }
                                else
                                {
                                    //create smallest jpg image 1x1
                                    set image = {}
                                    set image.src ="<img src=""data:image/jpg;charset=utf-8;base64,/9j/4AAQSkZJRgABAQEASABIAAD/2wBDAP//////////////////////////////////////////////////////////////////////////////////////wgALCAABAAEBAREA/8QAFBABAAAAAAAAAAAAAAAAAAAAAP/aAAgBAQABPxA="">"    
                                }
                                
                                //delete from disk
                                if (removeTempFiles=1)
                                {
                                    set deleteTmpFile = ##class(%File).%New(tempFile)
                                    set sc = deleteTmpFile.Delete(tempFile)
                                }
                            }
                            else
                            {
                                //encode to base64
                                set stream=##class(%Stream.FileBinary).%New()
                                set sc=stream.LinkToFile(imageFile)
                                do ..Base64EncodeStream(stream,.encodedStream)
                                set line = ""
                                do encodedStream.Rewind()
                                while (encodedStream.AtEnd = 0) {
                                    set len = 5700
                                    set line = line_encodedStream.Read(.len)
                                }
                                set image = {}
                                set image.src ="<img src=""data:image/"_$ZSTRIP(fileExtension,"*P")_";charset=utf-8;base64,"_line_""">"
                            }
                        }
                        catch ex
                        {
                            //ignore exception, create smallest jpg image 1x
                            set image = {}
                            set image.error = ex.DisplayString()
                            set image.src ="<img src=""data:image/jpg;charset=utf-8;base64,/9j/4AAQSkZJRgABAQEASABIAAD/2wBDAP//////////////////////////////////////////////////////////////////////////////////////wgALCAABAAEBAREA/8QAFBABAAAAAAAAAAAAAAAAAAAAAP/aAAgBAQABPxA="">"
                        }
                    }
                    else
                    {
                        //create smallest jpg image 1x1
                        set image = {}
                        set image.error = ex.DisplayString()
                        set image.src ="<img src=""data:image/jpg;charset=utf-8;base64,/9j/4AAQSkZJRgABAQEASABIAAD/2wBDAP//////////////////////////////////////////////////////////////////////////////////////wgALCAABAAEBAREA/8QAFBABAAAAAAAAAAAAAAAAAAAAAP/aAAgBAQABPxA="">"
                    }
                    do imageArray.%Push(image)
                }
                catch ex
                {
                    set image = {}
                    set image.error = ex.DisplayString()
                    do imageArray.%Push(image)
                }
                //Write images to string
                set targetJson.Images = imageArray
            }
        }           //write to stream
        do targetJson.%ToJSON(tmpStream)
        set ns = ##class(%Stream.GlobalCharacter).%New()
        do ns.CopyFrom(tmpStream)
        $$$ThrowOnError(tSC)
        set %response.ContentType = "application/json"
        set %response.Status = 200
        do ns.Rewind()
        return ns.OutputToDevice()
    }
    catch ex 
    {    
        set tSC = ex.AsStatus()
    }
    return tSC
} /// Encode a stream as BASE64, (based off intersystems exampk but with CRLF off)
ClassMethod Base64EncodeStream(pStream As %Stream, Output pEncoded As %Stream) As %Status
{
  s tSC=$$$OK
  try {
    s tSC=pStream.Rewind()
    q:$$$ISERR(tSC)     s pEncoded=##class(%Stream.TmpCharacter).%New()
    while 'pStream.AtEnd {
      s tLen=5700 
      s tSC=pEncoded.Write($system.Encryption.Base64Encode(pStream.Read(.tLen),1))
      q:$$$ISERR(tSC)
    }
    q:$$$ISERR(tSC)
        
    s tSC=pEncoded.Rewind()
  } catch (e) {
    s tSC=e.AsStatus()
  }
  q tSC
}

Thanks very much for replying with some suggestions. It's really helpful.

@Timothy Leavitt 
Thanks for the advice. I did try separate arguments in a previous version of my code, so it's nice to see I was on the right track originally so I will go back that way. With the $ZF(-100) approach I am getting 0k output file which suggest the STDOUT is being creating something (if somewhat unusable), so it could be something on the Windows side that is throwing the NOTOPEN.

@Danny Wijnschenk 
I've not thought of using OPEN. It's interesting option. I was exploring this briefly just now. What would you usually expect to see in the output normally. I've just been getting a 2.