Published on InterSystems Developer Community (https://community.intersystems.com)

Home > Data transformation and error

Question
Jack Smith · Oct 27, 2019

Data transformation and <store> error

We have a data transformation where source is object collection (populated from a json file)  and target is EnsLib.EDI.XML.Document.

If source file is large enough, transformation fails and we get <store> error and I quickly found this:

https://docs.intersystems.com/irislatest/csp/docbook/DocBook.UI.Page.cls?KEY=EBPLR_DTL_foreach which at the bottom in "Avoiding <STORE> Errors with Large Messages" section tells to: 

"As you loop over segments in a message or object collections, they are brought into memory. If these objects consume all the memory assigned to the current process, you may get unexpected errors.

To avoid this, remove the objects from memory after you no longer need them. For example, if you are processing many segments in a <foreach> loop, you can call the commitSegmentByPath method on both the source and target as the last step in the loop. Similarly, for object collections, use the %UnSwizzleAt method."

 

Adding that %UnSwizzleAt on the source as the last step in the loop did indeed help - a bit. Now it's possible to process somewhat larger files but any tips on how to increase the max size even more? I'm mostly confused with "on both the source and target" part. Is there a way to do something similar with my target of type EnsLib.EDI.XML.Document?

#DTL #Error Handling #ObjectScript #XML #InterSystems IRIS

Source URL:https://community.intersystems.com/post/data-transformation-and-error