Replies:

Hi Rich. Sorry for the delay. Thanks for your input. In answer to some of your questions:

Re: Use of messages/tracing.

The interface would just produce too much data. For instance, if I had to call an API and retrieve a full set of data every night and then paginate through it, then every http response would have to be passed back to process or another operation and get attached to the session and become part of massive view trace that takes an age to render and causes the data namespace messages to expand too much.

What happens to the data if the process aborts in the middle?

As all our adapters have AlertGroups, this will just use the Ens.Alert an email will be generated for someone to look at so we have some visibility.

Would missing any of the data being pulled have a negative effect on the business?

In this instance no because we are making a copy of some reporting data and the business can just check the source if needed. As mentioned, at minimum we catch any exceptions and alert a technical contact. Obviously with an API pull to SQL, its possible to get non-2XX status code that ruins a whole run. We would probably use the http adapter settings here to manage that but because we are not getting a delta unfortunately, only the latest run is ever important.

Is there any concern over data lineage for security or external auditing? 

Generally no, because data from the http response is just passed into a SQL table. The actual transformation of the data happens after it reaches its destination via reporting engine external to IRIS.

Re: Temporary holding table.

I previously did this to produce 5GB of json for a MongoDB parser, it works well but you have to compact the namespace after kill the globals or just live with reserving an amount of space for subsequent runs in the data namespace.
Yes, a global mapping to IRISTEMP could work to get around the journalling. My only issue is it looks like I'd have to add this mapping manually to my namespace in every environment before I promote my code. Is there are way of do this via method so I can script it?

Re: Batching, this would only run once a day, however I'm generating a run identifier so should be unique enough.

Hi Dmitry

Ahhh. I can see why my loop with the find wasn't working as I expected. Thank you so much for providing this example.

Dan

Hi. Thanks for your tips! Though it has to be Mod N version which is an extension that supports non-numerical strings. 

I guess I can use a proxy class to get the result but it would be nice to do this natively in Cache/MUMPS. What I will say is this: the need to do this it is going to be very important for others I expect also. A bit more info about why it is needed is below, with the character set needed to create codepoints from.

Link

Followers:
Following:
Global Masters badges:
Daniel has no Global Masters badges yet.