Data transformation and <store> error

Primary tabs

We have a data transformation where source is object collection (populated from a json file)  and target is EnsLib.EDI.XML.Document.

If source file is large enough, transformation fails and we get <store> error and I quickly found this:

https://docs.intersystems.com/irislatest/csp/docbook/DocBook.UI.Page.cls?KEY=EBPLR_DTL_foreach which at the bottom in "Avoiding <STORE> Errors with Large Messages" section tells to: 

"As you loop over segments in a message or object collections, they are brought into memory. If these objects consume all the memory assigned to the current process, you may get unexpected errors.

To avoid this, remove the objects from memory after you no longer need them. For example, if you are processing many segments in a <foreach> loop, you can call the commitSegmentByPath method on both the source and target as the last step in the loop. Similarly, for object collections, use the %UnSwizzleAt method."

 

Adding that %UnSwizzleAt on the source as the last step in the loop did indeed help - a bit. Now it's possible to process somewhat larger files but any tips on how to increase the max size even more? I'm mostly confused with "on both the source and target" part. Is there a way to do something similar with my target of type EnsLib.EDI.XML.Document?

 

 

Replies

Hi Jack,

We had used commitSegmentByPath for Huge HL7 related transformations. 

Can you try doing an intermediate %Save on the target object in the DTL using the <code> block. You can do this save multiple times, it will save as the same object. This might help you with some memory issues.

The other method you might consider is the one below, found in the EnsLib.EDI.XML.Document (IRIS 2019.2).

/// Switch from keeping changes in memory to keeping them in storage. Allows for unlimited changes to a document without running out of memory. This mode is automatically in effect when the object is created by loading from a stored Id
Method CommittedMode(pMode As %Boolean = 1) As %Status
 

There might be other efficient ways. Please let me know what you find out.

Thanks,

Arun