go to post Steven Hobbs · Oct 25, 2021 You should not follow the recommendation to modify the $ZTIMEZONE system variable. See this warning in the $ZTIMEZONE documentation: Note:Changing the $ZTIMEZONE special variable is a feature designed for some special situations. Changing $ZTIMEZONE is not a consistent way to change the time zone that Caché uses for local date/time operations. The $ZTIMEZONE special variable should not be changed except by those programs that are prepared to handle all the inconsistencies that result. On some platforms there may be a better way to change time zones than changing the $ZTIMEZONE special variable. If the platform has a process-specific time zone setting (for example, the TZ environment variable on POSIX systems) then making an external system call to change the process-specific time zone may work better than changing $ZTIMEZONE. Changing the process-specific time zone at the operating system level will change both the local time offset from UTC and apply the corresponding algorithm that determines when local time variants are applied. This is especially important if the default system time zone is in the Northern Hemisphere, while the desired process time zone is in the Southern Hemisphere. Changing $ZTIMEZONE changes the local time to a new time zone offset from UTC, but the algorithm that determines when local time variants are applied remains unchanged If you change $ZTIMEZONE then the local time will change but local changes in timezone rules (e.g., entering/leaving Daylight Saving Time, DST) will not be changed. If the system is in the Northern Hemisphere but you want a local time in the Southern Hemisphere (or vice-versa) then DST changes will be backwards. Also, DST rules near the equator can be very different from the DST rules at latitudes closer to the poles. If the system local timezone and the modified local timezone are in different countries then the national date/time rules may be incorrect for the modified local timezone. Using the $SYSTEM.Process.TimeZone(...) method suggested by Jon Willeke is the best way to modify the local timezone used by a Caché/IRIS process. However, the 'TZ' environment variable modified by the $SYSTEM.Process.TimeZone(...) method requires an argument string that is specific to the Operating System under which Caché/IRIS is running. Generally the Windows Operating System wants the TZ variable to contain a POSIX format timezone string while Unix/Linux systems want the TZ variable to contain an Olson format timezone string (sometimes called the IANA or ICU format timezone string.) If you need dates/times using rules from the past then generally the Olson format will work much better than the POSIX format.
go to post Steven Hobbs · Sep 15, 2021 Aside from several arithmetic differences, the biggest difference between $INCREMENT AND $SEQUENCE occurs when "SET index=$INCREMENT(^global)" versus "SET index=$SEQUENCE(^global)" are being executed by multiple processes. All the processes evaluating $INCREMENT(^global) on the same ^global variable will see a sequence of increasing integers. No two processes will see the same integer returned. The integers are given out in strict increasing time order and no integer value is skipped. All the processes evaluating $SEQUENCE(^global) on the same ^global variable will see a sequence of increasing integers. No two processes will see the same integer returned. Because blocks of integers are assigned to processes, it is possible for one process to receive a larger integer in the sequence before some other process receives a smaller integer in the sequence. If some process decides to stop processing integer values at some point then the larger integer values assigned to that process will be skipped and not returned as part of the sequence. The $SEQUENCE function can have less multi-process overhead because integers in the sequence are assigned in blocks but $SEQUENCE does not guarantee a sequence of numbered tasks is assigned to processes in strictly increasing order. Every process must continue processing numbered tasks until that particular process has been assigned a sequence number larger than the highest assigned task. Just because one process has finished the highest assigned task does not mean that other processes are done with the earlier tasks (or have even started earlier tasks) and the sequence is only complete when every process has received a sequence number larger that the number of the final task.
go to post Steven Hobbs · Jul 22, 2021 The JSON classes in Ens.Util.JSON, %ZEN.Auxiliary.json* and %JSON.* all contain methods that convert JSON representation to/from ObjectScript classes. Once you have an ordinary ObjectScript Class then you are using ObjectScript data types for Property values. The JSON null is usually converted to "" (null string). Also, ordinary Property variables of an ObjectScript Class are never undefined but are automatically initialized to "" (the null string). [[ An exception is [MULTIDIMENSIONAL] Properties which can be undefined but by default such Property variables do not participate in %Save() and JSON/XML export/import operations. ]] SQL operations involving Class properties treat "" (the null string) as the SQL NULL value and SQL assumes a Class Property containing the ObjectScript string $C(0) is the empty string. [[ Although the original question involved Caché and not IRIS, IRIS has signifcantly more complete support for the %DynamicAbstractObject classes so my examples will use IRIS. If possible, I recommend upgrading to IRIS. ]] There is the class %Library.DynamicAbstractObject and its subclasses %DynamicArray and %DynamicObject that can contain elements which can either be JSON values or ObjectScript values. The ObjectScript statement: USER>SET x={"a":null,"b":"","e":1.0,"f":"1.0","g":(00.1),"h":($c(0))} makes x be a %DynamicObject oref where element a is a JSON null and where element "g" is the ObjectScript number .1 and element "h" is the ObjectScript string $c(0). Note that if an ObjectScript expression is a %DynamicObject constructor then ObjectScript parses the elements of that constructor using JSON syntax except for the extension to constructor syntax where an element inside round parentheses is parsed as an ObjectScript run-time expression generating an ObjectScript value. You can convert a %DynamicObject to JSON string representation and any ObjectScript valued element will be converted to JSON representation. [[ Note that JSON does not support certain ObjectScript values: $double("NAN"), $double("INFINITY") and orefs that are not a subclass of %DynamicAbstractObject. A %DynamicAbstractObject containing such an ObjectScript element cannot be converted to JSON. ]] USER>WRITE x.%ToJSON(){"a":null,"b":"","e":1.0,"f":"1.0","g":0.1,"h":"\u0000"} Evaluating a %DynamicObject element in an ObjectScript expression converts that element to an ObjectScript value. USER>ZWRITE x.a, x.b, x.c, x.e, x.f, x.g, x.h""""""1"1.0".1$c(0) Notice that the undefined element x.c is converted to the ObjectScript null string. You never get an error evaluating x.%Get(key) for any value of the expression key as every undefined element in a %DynamicObject has the value of the null string. Also, x.a, which contains a JSON null, is converted to the ObjectScript null string. The JSON treatment of undefined elements and the ObjectScript treatment of undefined Properties means that when we convert an ordinary ObjectScript class to either XML or JSON then we can skip converting a Property with the null string value as converting JSON or XML back to an ordinary class object will result in all unrepresented properties getting the value of the null string. If you need to know if a %DynamicObject element is JSON null, null string or undefined then evaluating the %GetTypeOf(key) will tell you that. USER>ZWRITE x.%GetTypeOf("a"),x.%GetTypeOf("b"),x.%GetTypeOf("c"),x.%GetTypeOf("e")"null""string""unassigned""number" The %FromJSON(stream)/%ToJSON(stream) methods will let you read/write JSON representation from/to a %Stream. [[ Things that only work in IRIS follows. ]] The size of the %DynamicArray/%DynamicObject class objects is limited only by the amount of memory available to your process. A string valued %DynamicObject element can be significantly longer than the maximum length supported by ObjectScript string values. If you have such a long string element then you will have convert that element to an ObjectScript %Stream in order to manipulate it in ObjectScript. USER>SET stream=x.%Get(key,,"stream") ;; Note 3 arguments with 2nd argument missing will generate an in-memory, readonly %Stream that can be copied to a Global or File %Stream or can be examined by reading that string in small pieces. In recent IRIS releases you can do USER>SET binarystream=x.%Get(key,,"stream<base64") which will convert a base64 encoded element into a readonly binary %Stream. You can do the reverse conversion by evaluating x.%Set(key,binarystream,"stream>base64"). See the the Class Reference documentation pages for the %Library.%DyanmicAbstractObject class and its %DynamicArray and %DynamicObject subclasses for more details.
go to post Steven Hobbs · Jun 23, 2021 By the looks of it, the original question applied $DATA to an element of the %DynamicAbstractObject classes. The $DATA and $GET functions can only be applied to Multidimensional variables. All ObjectScript local and global variables are Multidimensional. By default a property variable of a class object is not multidimensional unless the property default is over ridden with the [ Multidimensional ] attribute. The %DynamicAbstractObject classes provide no way of supporting elements which are Multidimensional. If dynobj is an object reference to a %DynamicObject then dynobj.%Get(key) is defined for all possible string values of 'key'. If dynobj.%GetTypeOf(key) returns "unassigned", or "null", then dynobj.%Get(key) will return the empty string value. Currently, $DATA(dynobj.keyname) signals <PROPERTY DOES NOT EXIST> which is what is signaled for by all classes if the Property 'keyname' does not exist. In a future IRIS release it will report <INVALID CLASS> *Class '%Library.DynamicObject' does not support MultiDimensional operations The correct way to see if keyname "test" exists in a %DynamicObject is to evaluate (dynobj.%GetTypeOf("test") '= "undefined") .
go to post Steven Hobbs · Jun 2, 2021 The %DynamicAbstractObject classes (%DynamicArray and %DynamicObject) are not %Persistent as they cannot be directly written into a database of class objects. However, most %DynamicAbstractObject objects can be serialized with the %GetSerial method which produces a JSON string value or with the %GetSwizzleObject method which builds an OID containing the JSON string value. And a Serial object can be the value of a Property internal to a %Persistent object. The %DynamicAbstractObject objects can contain elements which are either JSON values or ObjectScript values. Any element which can be translated into a JSON value can be serialized with the %GetSerial/%GetSwizzleObject methods. However, if a %Dynamic object contains ObjectScript values that are not representable in JSON then that %Dynamic object cannot be serialialized. Examples of ObjectScript values that cannot be translated into JSON include $double("nan"), $double("infinity"), $double("-infinity") and any oref not a member of the %DynamicArray or %DynamicObject classes. Also, the %DynamicArray and %DynamicObject classes can contain string element values much longer than the %String values supported by ObjectScript. If the JSON translation of a %DynamicArray or a %DynamicObject results in a string longer than the ObjectScript maximum string length then the serialization of the object will fail. However, the JSON translation of a large %Dynamic object can be stored into a class of the %Stream package that is abled to be swizzled. Such a swizzled object can be a Property in a %Persistent object. However, the %Stream property must be processed by the %FromJSON(stream) method in order to turn the %Stream into a %DynamicAbstractObject object.
go to post Steven Hobbs · May 3, 2021 Since the JSON content of 'members' property is a JSON array (aka the ObjectScript class %DynamicArray) and that array contains a list of JSON objects (aka the ObjectScript class %DynamicObject), I assume you don't really want a members array containing less that 1 element to be the null JSON value but instead you want that members value to be an empty array, I.e. set ObjectScriptObject.members=[ ] Making this choice would mean ObjectScriptObject.members is always a %Dynamic.Array containing 0 or more elements. A *property* of a class always contains an ObjectScript value that can be a string, a canonical decimal number, a $DOUBLE(...) IEEE 64-bit floating-point value or an oref. An oref is a reference to a Class type object which includes the %DynamicArray and %DynamicObject classes. The %DynamicArray/Object classes are special in that they can contain zero or more elements where each element can either be a JSON value or an ObjectScript value. Where appropriate, conversions will be performed between JSON values and ObjectScript values. ObjectScript variables and [MULTIDIMENSIONAL] properties can contain ObjectScript arrays of ObjectScript values. These ObjectScript arrays are hierarchical, or tree like. Internal and leaf elements can have a value or can be undefined. A non-multidimensional property and an ObjectScript valued element in a %DynamicArray/Object object are not hierarchical and not undefined. Non-multidimensional properties and elements that have not been initialized will have the empty string, "", as their value the first time they are accessed.
go to post Steven Hobbs · Apr 28, 2021 You are using a 5 year old version of Ensemble that is using an experimental release of JSON support with classes named %Library.AbstractObject, %Library.Object and %Library.Array. When you upgrade to any version after 2016.1 you will find the JSON support classes are now named %Library.DynamicAbstractObject, %Library.DynamicObject and %Library.DynamicArray. All of the method names will change because support for system methods with names that start with $ has been removed. Method names like $getIterator, $fromJSON and $getNext will become %GetIterator, %FromJSON and %GetNext. Other than the extensive name changes, most of features of the experimental JSON classes has remained unchanged. There are additional new features added during future releases. For example, the ObjectScript language will accept JSON object and array constructor syntax as part of the ObjectScript expression syntax. If an element which is part of an object/array constructor inside an ObectScript program is enclosed in round parentheses then that element is evaluated using ObjectScript expression syntax instead using JSON literal syntax. There will also be some additional ways to control the conversions between JSON types and ObjectScript types.
go to post Steven Hobbs · Dec 17, 2020 Standard class object properties have a limited string length but a property can contain a %Stream.GlobalCharacter oref which %JSON.Adaptor can export as a JSON string. Another option is to create a %DynamicObject (%Library.DynamicObject) class object or a %DynamicArray (%Library.DynamicArray) class object instead of using a subclass of %JSON.Adaptor. You can create a %DynamicObject/%DynamicArray is ObjectScript by using a JSON object/array literal as an ObjectScript literal. The %FromJSON class method will create a %DynamicObject/%DynamicArray by importing JSON from a %Stream, file or device and the %ToJSON class method will export a %DynamicObject/%DynamicArray to a %Stream, file or device. On IRIS, the %Set and %Get methods in %DynamicObject-s/%DynamicArray-s have been extended to take type keywords of the form "stream", "stream>base64", "stream<base64" which can transfer %DynamicObject string elements between unlimited length %Stream-s and include the ability to encode or decode Base64 during the transfer. There are also type key words "string", "string>base64", "string<base64" which can set/get ObjectScript string values into/from %DynamicObject elements but ObjectScript strings are currently limited to a length of 3,641,144 characters. So on IRIS you can do something like: Set DynObj = {"ID":23, "Name":"John Doe", "BirthDay":"1974-12-15"} Do DynObj.%Set("DataFile",DataOref,"stream>base64") Do DynObj.%ToJSON(OutputOref)where DataOref is a class reference to a %Stream.FileBinary referencing the binary file containing data related to John Doe and where OutputOref is a %Stream.FileCharacter referencing the file that will contain JSON text.
go to post Steven Hobbs · Aug 4, 2020 Although the replies have already answered the question about creating JSON on Caché, I would look to add some discussion about additional features that support the %Library.DynamicArray and %Library.DynamicObject classes. I am running my examples on IRIS Version 2020.2 (Build 199U) so a few of the features will not be available in older Caché versions. You asked how does a programmer generate the following JSON: { "MyProperty":"1"}. If you are writing an ObjectScript program then just execute the statement: SET x = { "MyProperty":"1"} Whenever an ObjectScript expression contains language syntax which is enclosed either in nested curly brackets, { ... }, or in nested square brackets, [ ... ], then the ObjectScript compiler will parse the contents of the bracketed expression using the rules for a JSON object or a JSON array respectively. The strings and numbers and identifiers inside the bracketed syntax will be JSON literal values with one exception. If a value inside the bracketed expression is enclosed in round parentheses, ( ... ) then the contents of the parenthesized value will have the syntax of an ObjectScript expression which will be evaluated at run time. The %DynamicArray and %DynamicObject class objects can contain property values using JSON syntax for their representation as well as containing property values which are ObjectScript values. Details of all the methods that can be applied to a %DynamicArray/%DynamicObject class object can be found in Class Reference web pages for the %Libraray.DynamicArray, %Library.DynamicObject and %Library.DynamicAbstractObject classes. When you use the %ToJSON( ) method on a %DynamicArray/%DynamicObject then all the properties containing ObjectScript values will be translated to the appropriate JSON literal representation. Certain ObjectScript values, such as an oref or $DOUBLE("NaN"), do not have a JSON representation and their occurrence will cause %ToJSON to generate an <ILLEGAL VALUE> signal. When you use the %Get(key) method to evaluate a property containing JSON representation then that JSON representation will first be converted into an ObjectScript value. When you use the %Set(key,value) to modify a property then value is computed as a run-time ObjectScript expression and that ObjectScript computation becomes the value of the property. The %Get(key,default,type) method call has two optional parameters. The value of the default argument is returned (without any type conversions) if there is no property with the specified key argument. If there is no default argument then an unassigned key argument will return the ObjectScript empty string. The type argument contains a string that specifies what type conversion should be applied to the element with the specified key. An empty string type argument or a missing type argument does the normal default conversion to ObjectScript. The %Set(key,value,type) takes an optional type argument which is a string which specifies how the ObjectScript value argument is converted to a property element. Th supported type arguments have changed over time so check the Class Reference web pages for documentation on what the possible type strings mean. Consider: USER>set x=["0.1230",12.30E-2,(.12300),("0.1230")] ;; JSON string, JSON number, ObjScr number, ObjScr string USER>write x.%ToJSON() ["0.1230",12.30E-2,0.123,"0.1230"] ;; Everything converts to JSON representation (strings are the same) USER>write x.%Get(0),",",x.%Get(1),",",x.%Get(2),",",x.%Get(3),"," ;; Everything converts to ObjectScript0.1230,.123,.123,0.1230, ;; Newly added "json" type argument.;; The 6-char string 0.1230 becomes 8 chars with leading/trailing double-quotes in JSON literal representation;; The JSON number is not changed;; The ObjectScript number picks up the leading zero required by JSON representationUSER>write x.%Get(0,,"json"),",",x.%Get(1,,"json"),",",x.%Get(2,,"json"),",""0.1230",12.30E-2,0.123,USER>set y=["null",null,""] ;; JSON 4-char string null, JSON null value, JSON empty string USER>write y.%ToJSON( ) ;; JSON array printed as a %String ["null",null,""]USER>write y.%Get(0),",",y.%Get(1),",",y.%Get(2),",",y.%Get(3),"," ;; Convert to ObjectScript 4 elements of 3-element array null,,,, ;; Note that null, "" and unassigned all convert to empty string in ObjectScriptUSER>write y.%Get(0,,"json"),",",y.%Get(1,,"json"),",",y.%Get(2,,"json"),",",y.%Get(3,,"json"),"," ;; Convert 4 elements to JSON"null",null,"",,;; Note: 4-char string null now is 6 chars with 2 "-chars, null identifier, empty string with only 2 "-chars,;; and ObjectScript default null string for unassigned value (there is no legal JSON representation) Unfortunately %Set does not yet support the "json" type argument but maybe that will happen in a future IRIS release.
go to post Steven Hobbs · May 21, 2020 There are actually two (maybe more) levels of ObjectScript. There is basic ObjectScript, which is an very extended version of the ANSI M language. [ footnote [ANSI M is the successor of the ANSI MUMPS language and that language, without the large number of extensions supported by basic ObjectScript, could be considered to be a third language level although no modern programmer would restrict their code to this much older language definition.] ] And there is the Class Language ObjectScript, which includes things like type-name classes: %Library.String (can be abbreviated %String), %Library.Integer (can be abbreviated %Integer), etc. It also includes Class Methods (with syntax like ##class(Class.Name).ClassMethodName(arg1,arg2)) and there are Object Methods (which look like oref.ObjectMethodName(arg1,arg2), where oref must contains an extended ObjectScript object reference) and there are Object Properties (which look like oref.PropName, where oref must contain an extended ObjectScript object reference.) An example of a basic ObjectScript statement is SET var1=42,var2="42" Almost every operation in basic ObjectScript thinks var1 and var2 contain identical values. So the string equality operation var1=var2 returns 1 because var1 is converted to a string and "42" equals "42". The numeric comparison operations var1<var2 and var1'<var2 return 0 and 1 because var2 is converted to a numeric and 42<42 is false while 42'<42 is true. We could also evaluate var1+var2 and the result will be 84 (or is that result "84"--who can tell?). In Class Language ObjectScript you can declare property LimitedInt : %Integer(MAXVAL=10);and if you (directly or indirectly) call the %ValidateObject() method on a class object containing the property LimitedInt then the contents of LimitedInt may be checked to make sure they look like an integer with a value not greater than 10. However, only Class Language methods like %SerializeObject, %ValidatObject, %Save, etc. make these checks on the value of LimitedInt. If someone executes SET oref.LimitedInt=20.95in basic ObjectScript, the basic ObjectScript execution will not signal an error despite the fact that 20.95 is larger than the MAXVAL and despite the fact that 20.95 is NOT a %Integer. Only executing an appropriate Class Language method will detect that oref.LimitedInt does not contain a valid value. The purpose of %Library.DataType subclasses is to make it possible that a %Save() method does not save invalid property values into a data base. Certain ObjectScript conversions may change an ObjectScript value. This can happen when changing a well-formed numeric string to be a decimal number because the decimal arithmetic implemented in ObjectScript mathematics supports no more than 19 decimal digits of precision. Consider, USER>set a="12345678901234567890123",b="12345678901234567890124",c="123456789012345678901230"USER>write b>a," ",a>b," ",+a," ",+b0 0 12345678901234567890000 12345678901234567890000 Because the > and < operators are arithmetic-comparison operators the string operands are converted to a numeric value with only 19 significant digits of precision and the resulting numeric values for a and b end up being equal so neither b>a nor a>b are true. However, the sorts-after operator, ]], orders the canonical numeric strings before any non-empty string that does not have canonical numeric syntax. The canonical numeric strings are sorted in numeric order while strings that do not have canonical numeric syntax are sorted in textual string order. This is also the default rule for ordering subscript values in ObjectScript. Operands of ]] are converted to strings before doing this subscript ordering. The values a, b and c are all canonical numeric strings and ObjectScript is perfectly capable of doing the sorts-after string comparisons on very long canonical numeric strings. Consider, USER>write a]]b," ",b]]a," ",b]]c," ",c]]b0 1 0 1 Now a and b are both canonical numeric strings with 23 digits and their first 22 digits are equal but the b has a 23rd digit larger than the 23rd digit of a, so b sorts-after a. But b does not sorts-after c because c has more significant digits so the canonical numeric value of c is greater than the canonical numeric value of b. The basic ObjectScript language has a very small class of built-in types. (1) There are the ObjectScript string values, which include the subclass of canonical numeric strings. (2) There are the default decimal floating-point values which do not have more 19 digits of decimal precision (and may sometimes have less than 19 digits of precision because the implemented accuracy of ObjectScript decimal arithmetic operations is approximately 18.96 decimal digits.) Every decimal floating-point value can be converted exactly to a canonical numeric string (but canonical numeric string values with more than 18 digit characters cannot always be converted exactly to a decimal floating-point numeric result but must instead be a decimal numeric value that is an approximation. (3) Basic ObjectScript supports a third set of "$DOUBLE values" which contains the *binary* floating-point values specified by the IEEE 64-bit binary floating-type type. The 64-bit binary floating-point arithmetic specified by the IEEE standard has approximately 15.95 decimal digits of precision but since the representation is binary, and not decimal, an exact conversion of a 64-bit IEEE binary floating-point value to a decimal string can have over 1000 digits. Now every $DOUBLE binary floating-point value could be exactly converted to a canonical numeric string but it is not reasonable to have such a conversion produce such long strings. The default conversion of a $DOUBLE value to a canonical numeric string will have no more than 20 significant digits, and the approximated 20th significant digit will never be a 5 or 0 (unless that 20th digit results in an exact conversion.) Using a default canonical numeric string with 20 significant digits for $DOUBLE conversions means the ]], sorts-after, operator will correctly order a $DOUBLE binary floating-point value in-between the adjacent 19-digit decimal floating-point values of the ObjetScript default decimal numeric type. (4) There are also the oref values which are the basic ObjectScript values created by Class Language ObjectScript. Basic ObjectScript can do property evaluation, property assignment and method calls using the basic ObjectScript oref type. Basic ObjectScript can convert oref values to the ObjectScript string type and the ObjectScript decimal arithmetic type but neither of these conversions is particularly useful unless you are debugging.
go to post Steven Hobbs · Apr 13, 2020 My comment applies generally to this entire discussion and I know your comment was not talking about Unicode character conversion. However, your comment did include actual code with a loop. I wanted to make it easy for readers to see your Base64 encoding which is looping over a %Stream while I discussed the issues involved with adding calls to $ZCONVERT(UnicodeText,"O","UTF8",handle) to convert 16-bit UTF-16 characters into 8-bit UTF-8 bytes which could then be Base64 encoded.
go to post Steven Hobbs · Apr 9, 2020 I should point out that if you have a Unicode character with an encoding larger than 65535 (i.e., encoded in UTF-16 using a surrogate pair of two adjacent 16-bit characters) then the statement SET BinaryText=$ZCONVERT(UnicodeText,"O","UTF8") in the encoding loop will also need a fourth "handle" argument to handle the case where the UnicodeText substring ends with the leading half of a surrogate pair. Characters with Unicode encoding greater than 65535 consist mostly of less frequently used Chinese, Korean and Japanese ideograms but also many of the emojis.
go to post Steven Hobbs · Apr 9, 2020 Base64 encoding only works on strings of 8-bit bytes. If you have a Unicode string with a character with an encoded value greater than 255 then direct Base64 encoding is not possible. The following documentation excerpt from the %SYSTEM.Encryption.Base64Encode Class Reference page will describe a first step at turning Unicode into a UTF-8 byte string and then applying the Base64 encoding to that byte string (but if your encoded stream exceeds the string limit in size then you will need to do more.) Note: Base 64 encoding is not able to encode a string which contains unicode (2 byte) characters. If you need to Base 64 encode an unicode string, you should first translate the string to UTF8 format, then encode it. s BinaryText=$ZCONVERT(UnicodeText,"O","UTF8")s Base64Encoded=$system.Encryption.Base64Encode(BinaryText)Now to Decode it:s BinaryText=$system.Encryption.Base64Decode(Base64Encoded)s UnicodeText=$ZCONVERT(BinaryText,"I","UTF8") Now if your Base64Encoded, UTF-8 encoded string cannot be longer than 3,641,144 bytes then you can ignore the rest of this reply. Just execute the code excerpt from the %SYSTEM.Encryption.Base64Encode Class Reference documentation. If you have a %Stream that is too long then you will need to loop over substrings of the %Stream and you should read on. If you read from a long %Stream a sequence of substrings then it might not be possible to simply call $system.Encryption.Base64Encode(BinaryText,...) and $ZConvert(BinaryText,"I","UTF8") and because the substrings can be broken between binary bytes that must be combined in order to do the conversion. So you will need to workaround some issues in the code that was part of the %SYSTEM.Encryption.Base64Encode Class Reference documentation. Issue one: the SET BinaryText=$ZCONVERT(UnicodeText,"O","UTF8") statement can return more characters of BinaryText than there are characters of UnicodeText. The new string length may not be a perfect multiple of 3 characters long and $system.Encryption.Base64Encode converts sequences of 3 bytes into 4 bytes so when the Base64Encode(substring) method is called with a substring that has a non-multiple of 3 byte length then the extra 1 or 2 bytes at the end of the substring must be saved and then concatenated onto the beginning of the next substring to be passed to the Base64Encode(substring) method. Only the very last call on the Base64Encode(substring) method can have a substring byte length which is not a multiple of 3. Issue two: the "SET UnicodeText=$ZCONVERT(BinaryText,"I","UTF8")" statement might be given a BinaryText substring that ends with an incomplete UTF-8 sequence of characters. That incomplete sequence must be concatenated onto the beginning of the next BinaryText substring. Fortunately, the $ZCONVERT function takes an optional fourth argument which is a local variable name. Evaluating $ZCONVERT(UnicodeText,"O","UTF8",handle) will do its conversion on an input value containing the concatenation of handle with UnicodeText. When $ZCONVERT is done then the new value of handle will either be the empty string or handle will contain the unconverted substring at the end of the input value. So before your loop which decodes the %Stream containing Base64 encoded UTF-8 bytes you should execute SET handle="" and inside your decoding loop you should call $ZCONVERT using handle as a forth argument variable. If at the exit of your decoding loop the variable handle does not contain the empty string then your input stream was ill-formed. Note that when your decoding loop reads a Base64Encoded substring from the %Stream then I am assuming you read a perfect multiple of 4 bytes from the %Stream before you execute SET BinaryText=$system.Encryption.Base64Decode(Base64Encoded). [[ If the Base64 encoded %Stream was not generated using the code described above then I am also assuming the Base64 encoded BinaryText does not contain any additional white-space characters, or if it does contain white space then those white-space characters were removed before building a substring that contains a perfect multiple of 4 bytes. ]] Each sequence of 4 bytes in Base64Encoded will be turned into 3 bytes in BinaryText.
go to post Steven Hobbs · Feb 10, 2020 Extension of the %Set and %Get methods of the %DynamicAbstractObject classes to support type parameters (such as using "stream" as a type) is an IRIS feature and not a Caché feature. In Caché the %ToJSON and %FromJSON methods can accept %File-s and %Stream-s. These methods are generally limited to Dynamic Objects/Arrays involving sizes related to the largest 32-bit signed integer. But in Caché the %ToJSON/%FromJSON methods work only on entire Dynamic Objects/Arrays. Using %Set/%Get in Caché to modify an element of a Dynamic Object/Array is limited by the length of an ObjectScript %String (currently 3,641,144 characters in Caché.) In recent and future IRIS releases the sizes of a %DynamicAbstractObject will be limited by the amount of virtual memory available to the process. (Please consider avoiding the activation of many multi-gigabyte Dynamic objects/arrays at the same time.) The future IRIS versions of the %Get/%Set methods will support type keywords involving streams which will allow a %Get/%Set to access Dynamic Object/Array elements involving any size that can be fetched from (or stored into) a %Stream. Future versions of these methods will also encode and decode Base64 representation while transferring byte data between a %Stream and an element of a Dynamic Object/Array.
go to post Steven Hobbs · Apr 30, 2019 For Set MyObj.JSON=JSONString.%ToJSON() to signal <INVALID OREF> then either local variable MyObj is not an oref or local variable JSONString is not an oref. Make sure that JSONString is really a %DynamicObject by doing a WRITE JSONString before doing the Set command. The WRITE statement should write some like 99@%Library.DynamicObject .
go to post Steven Hobbs · Apr 27, 2019 I agree that comments here will have little effect on developers other than InterSystems developers or InterSystems customers. However, processing of JSON is controlled by the European standard "ECMA-404 The JSON Data Interchange Format". If a JSON decoder deliberately ignores this standard specification then it is not a *JSON* decoder. If any customer discovers that InterSystems's %FromJSON method is violating the ECMA-404 standard then I am sure InterSystems will fix this situation. I suspect that third-party developers would also fix such a violation of ECMA-404. The ECMA-404 document actually contains examples of 4 different strings that are identical to "/". ECMA-404 does not require that JSON encoders generate the "\/" string syntax but this standard does require that all decoders accept this string syntax.
go to post Steven Hobbs · Apr 25, 2019 There are some issues with above C code for manipulating a string of $LIST elements.I do not believe the above code will work on a big-endian platform, such as PowerPC running the AIX operating system. Conversions from $LIST representation to numeric representation will order the bytes backwards on big-endian hardware.Decimal floating-point values in IRIS and Caché have almost 19 decimal digits of precision while the above code translates these numbers to IEEE binary double-precision floating-point values which have less than 16 decimal digits of precision. This means that $LISTBUILD(0.3) will be converted by the above code into the value 0.29999999999999998889... . The above code also introduces a double-round error so that decoding $listbuild(x) and $listbuild($double(x)) will not always be equal because the $double(x) function in IRIS/Caché will do the conversion from decimal to binary without a double round.The above code is inconsistent in resulting type of integer values. Consider,USER>set L5A=$LISTBUILD(5),L5B=$LISTBUILD(50/10)USER>WRITE $LISTSAME(L5A,L5B)1The $LIST elements in L5A and L5B contain the same value, 5. However, the above code will convert L5A to the C int64_t type while it will convert L5B to the C double type. If the value of the integer is greater than 2**53 then the above C code can convert the identical integer values into different integer values.Also, the above code does not correctly handle all the special cases when (type==FLOAT) is true.The IRIS/Caché $LIST representation is not as simple as most people think. There are some unusual rules that must be followed if you want to get the same results as the $LISTxxx functions get in IRIS/Caché.
go to post Steven Hobbs · Apr 25, 2019 Agreed that there is nothing wrong with escaping a solidus. However, saying "most decoders" will unescape the solidus correctly" is not quite correct. "*ALL* decoders will unescape the solidus correctly." If a decoder does not correctly unescape the solidus then it is *NOT* a JSON decoder. The %FromJSON method in the %DynamicObject class in IRIS and Caché versions after 2016.1 will do the following:USER>write x{"FileStatus":"P","Path":"\/somepath\/test\/test123\/filename.txt","InterchangeOID":"100458"}USER>set y={}.%FromJSON(x)USER>write y.Path/somepath/test/test123/filename.txtSo %FromJSON is the method that breaks JSON apart and removes all escaping from JSON string values. (I assume that $fromJSON will do the same in Caché version 2016.1. The JSON manipulation added to version 2016.1 was changed significantly in all future releases. Most JSON manipulation programs written in version 2016.1 will require rewriting when porting to a later release.)
go to post Steven Hobbs · Jan 14, 2019 The ClassMethod IsNumeric(...) seems to contain three SET statements that do nothing more than copy data around. The following is equivalent (without data copies)ClassMethod IsNumeric(value As %String) As %Boolean { QUIT $ISVALIDNUM(value) }And this implementation of IsNumeric is equivalent to the first answer, ClassMethod IsValidNumber(...) by Eduard Lebedyuk except the IsNumeric ClassMethod does not have the [ FINAL ] attribute.
go to post Steven Hobbs · Nov 28, 2018 Let us assume that you are using the %DynamicAbstractObject class and its subclasses, which include %DynamicArray and %DynamicObject (instead of using the legacy support for JSON in the %ZEN package.) If you have an element of a JSON array or JSON object that contains more characters than are supported by an ObjectScript string then the way to manipulate that element is to send it to a %File or a %Stream. In the present releases of Caché and IRIS Data Platform the only way to do that is to use the %ToJSON(outstream) method call. If the element in question is in a nested sequence of JSON elements then you can first use a sequence of %Get method calls to access the closest containing %DynamicArray or %DynamicObject. You can then use %ToJSON(outstream) method call to create a %Stream containing the textual JSON of that entire object. Then process the contents of that %Stream using the Read(len,...)/Readline(len,...) methods, with the 'len' parameter chosen to provide blocking that will prevent exceeding ObjectScript string length limits. Some JSON parsing will be necessary to find the beginning and the ending of the JSON formatted string containing the element in question so that the required element can be copied as the only contents of a new %Stream.If you can control how the JSON is formatted then you can choose to enclose the long JSON string element inside a JSON array containing just that one element--the long string value requiring additional manipulation. Parsing such an %DynamicArray element would be easy--send the Array to a %Stream using %ToJSON and then ignore the first two characters, [", and the last two characters, "], of the %Stream. What remains is contents of the string element with a few special characters encoded using \-sequences. If you cannot control how the JSON is formatted then it still might be possible to use knowledge of the application that is using JSON representation to short cut the parsing issues.In a future InterSystems product release there will certainly be an extension to the %Get method that will allow any %DynamicAbstractObject element (including string and numeric elements) to be sent to a %Stream using either Raw String representation or using JSON representation. You will also be able to copy such an element from one JSON array/object into another JSON array/object without needing to copy the characters to a %Stream that is using an external representation.