The %FromJSONFile(file) method does not take a %File object as an argument but instead takes a string containing the name of file. You need to replace
set newObject = {}.%FromJSONFile(file)
with one of
set newObject = {}.%FromJSONFile("/tmp/longObjectFile.txt")
set newObject = {}.%FromJSON(file)
Originally, %FromJSON(input) would accept the 'input' argument as a %File object, a %Stream object, a JSON text string or a filename text string. However, accepting an argument string as either JSON or a filename was considered to be security issue because a malicious user could be requested to enter a json-string but, being malicious, the user would instead enter a filename-string, or vice-versa. As a result %FromJSON no longer accepts filename strings while %FromJSONFile only accepts filename strings.
No longer an issue, but it would be possible that legal VMS filename syntax could also be legal JSON text string syntax and such an argument string might be ambiguous on a VMS system. But IRIS no longer supports VMS and Caché for VMS did not support Dynamic Objects.
%DynamicAbstractObject-s provides the fastest for reading JSON from an external location and once in memory it provides excellent random access reading of data components. Large JSON arrays/objects are supported. I have done %FromJSON on a 20GB JSON file on a system with 16GB of physical memory—required paging of memory but still acceptable performance. You don’t require random access performance and it is best to avoid using more memory than you need. Maybe someday InterSystems can provide methods for reading JSON subobjects while remember how that object was nested in its parent objects.
Assuming your JSON is in a file or stream and the outer array contains only elements which are arrays or objects you can try the following. 1. First read the leading ‘[‘; 2. Then use %FromJSON to read the array element and process that JSON element (the read will stop on the closing ‘]’ or ‘}’); 3. Read single chars from your stream/file skipping white space and if that char is a ‘,’ then go back to step 2 but if that char is ‘]’ then you are done.
We need to see more of your code to see what is truly happening. The routine method call Do obj.%Set("data",pStream,"stream>base64") should have no problem providing your system has enough memory to construct entire 'obj' Dynamic Object in your process memory. Your system will want to avoid having many large Dynamic Objects in memory at the same time. If the system runs out memory for all the active processes then the operating system will usually start aborting processes randomly until the operating system decides it now has enough free memory to continue execution.
Calling %Set(,,) to read JSON text from a very large %Stream should only use moderate size buffers on the string stack. If a call on %Set(,,) writing from a %Stream generates a <STRINGSTACK> signal then that should be reported to InterSystems as a bug.
However, your reply mentions the %ToJSON() function method call. Calling %ToJSON as a function will place the result on the String Stack so it is possible such a function call could signal a <STRINGSTACK> error although if the %ToJSON() function call appears in a very simple expression then I would expect a <MAXSTRING> signal instead of a <STRINGSTACK> signal.
Calling %ToJSON() or %ToJSON(outstrm) as a routine method call in a DO statement should not get either a <STRINGSTACK> or a <MAXSTRING> signal since the result is not going to an ObjectScript string value.