Implemented Ideas: Global->JSON->Global converter

JSON is a lightweight data interchange format for representing and exchanging data between a server and a web application. Its popularity has led to its widespread use in applications based on InterSystems technology, as well as demand for a converter that transforms globals to JSON and back. Therefore, @Evgeny Shvarov suggested developing Global->JSON->Global converter. Implementing such a converter is important to ensure interoperability, simplify data exchange, support web services, and provide a standardized approach to data representation across different software ecosystems. This article, the second in the "Implemented Ideas" series, focuses on several projects created by the legend of InterSystems Developer Community @Robert Cemper that deal with this task:

• GlobalToJSON-Compact
• GlobalToJSON-Efficient
• GlobalToJSON-Academic
• GlobalToJSON-embeddedPython
• GlobalToJSON-ePython-pure
• GlobalToJSON-XLA
• JSONfile-to-Global.

Many thanks to @Robert Cemper, who kindly answered the questions about the reasons behind the implementation of this idea. Below, you can read a short review of all 7 variants based on his answers and his articles dedicated to these implementations.

First, the demand for the Global ->JSON utility rose in 2021 in a post by @Fabian Haupt. Then, @Evgeny Shvarov added this idea to the Ideas Portal. While researching the problem, @Robert Cemper realized that there are two main obstacles:

• the expectation was rather unclear of how a global written in JSON should look like
• just converting a global to JSON was not enough: doing the reverse operation became the implicit requirement

Therefore, not just Global->JSON conversion but also JSON->Global was needed. That's why Cemper implemented the first solution called "Compact." This solution is easy to read and write but full of redundancy, almost just a text scan exercise. Globals are included with data for a fast data load, but the related code is also quite compact.

For most solutions, the developer wrote articles with a functionality description: 

• GlobalToJSON-Compact
• GlobalToJSON-Efficient
• GlobalToJSON-Academic
• GlobalToJSON-embeddedPython
• GlobalToJSON-ePython-pure
• JSONfile-to-Global #1
• JSONfile-to-Global #2

In these articles, Cemper uses the same example of a global (see the picture below) to convert and describes how this global is converted using different variants of application. Then, he shows that each variant of the application that creates the same global after we use the Loader for JSON.

The second solution is called "Efficient" and it was created to make it more JSON-like. This version is looking nice but still has enough redundancy. Only globals nodes containing data are presented in the generated JSON object.

The third version is called "Academic". Each logical node of the global is represented separately with all its descendants, even if they don’t contain any stored data. This solution uses the tree structure implicit to JSON, it is not redundant anymore, but the result is large and almost unreadable. Therefore, Cemper believes that "Academic" version is not very useful for practical work.

Several months later, Cemper created two new Global->JSON->Global converter solutions using a mix of ePy and ObjectScript with improved navigation. These variants are called "embeddedPython" and "pure ePython". These variants are similar to the "Efficient" version from the perspective of the data they convert. They convert only globals nodes containing data, but unlike the previous version, instead of using pure ObjectScript, they use ePy instead. "Pure ePython" version doesn't use ObjectScript at all, while in "embeddedPython" variant ePy did only the JSON work, because global access from ePy wasn't yet complete.

As a development of "Academic" version, Cemper developed "GlobaltoJSON-XLA" variant to handle big globals (^oddDEF ~2 million nodes) with read/write directly to file. In a previous "Academic" example, this was all processed in memory.  

The 7th variant, "JSONfile-to-Global," helps developers load JSON objects into globals because navigation inside large JSON objects frequently becomes a challenge. 

Based on his experience of developing these convertors, Cemper highlights, that examples for individual cases with appropriate adjustment would be better than a generic all-size-fit solution. Understanding these examples well should give a clear picture of where to use JSON to transport globals and the issues around them. Cemper believes that, despite the initial fear, ePy may completely replace ObjectScript in business applications. Developers can use SQL for data access, not fiddle with globals. He resumes that the mix of system and application code is in the past. Now it is possible to separate these 2 worlds more clearly from each other.

Seven solutions developed to implement the Global->JSON->Global converter idea aroused great interest from Developer Community members. They were downloaded 118 times, and many users wrote positive reviews of these applications. For example, @Yuri Marx wrote in his review, "Excellent application to write your global data into JSON."

Once again, we thank our legend @Robert Cemper for the excellent solutions and detailed answers to the questions that formed the basis of this article.

Looking forward to your feedback in comments about highlighted ideas and solutions implementing this idea.

Post your ideas on the Ideas Portal, promote them to other Developers to collect more votes, and implement Community Opportunity ideas!