My comment applies generally to this entire discussion and I know your comment was not talking about Unicode character conversion.  However, your comment did include actual code with a loop.  I wanted to make it easy for readers to see your Base64 encoding which is looping over a %Stream while I discussed the issues involved with adding calls to $ZCONVERT(UnicodeText,"O","UTF8",handle) to convert 16-bit UTF-16 characters into 8-bit UTF-8 bytes which could then be Base64 encoded.

I should point out that if you have a Unicode character with an encoding larger than 65535 (i.e., encoded in UTF-16 using a surrogate pair of two adjacent 16-bit characters) then the statement SET BinaryText=$ZCONVERT(UnicodeText,"O","UTF8") in the encoding loop will also need a fourth "handle" argument to handle the case where the UnicodeText substring ends with the leading half of a surrogate pair.  Characters with Unicode encoding greater than 65535 consist mostly of less frequently used Chinese, Korean and Japanese ideograms but also many of the emojis.  

Base64 encoding only works on strings of 8-bit bytes.  If you have a Unicode string with a character with an encoded value greater than 255 then direct Base64 encoding is not possible.  The following documentation excerpt from the %SYSTEM.Encryption.Base64Encode Class Reference page will describe a first step at turning Unicode into a UTF-8 byte string and then applying the Base64 encoding to that byte string (but if your encoded stream exceeds the string limit in size then you will need to do more.)

Note: Base 64 encoding is not able to encode a string which contains unicode (2 byte) characters. If you need to Base 64 encode an unicode string, you should first translate the string to UTF8 format, then encode it.

s BinaryText=$ZCONVERT(UnicodeText,"O","UTF8")
s Base64Encoded=$system.Encryption.Base64Encode(BinaryText)
Now to Decode it:
s BinaryText=$system.Encryption.Base64Decode(Base64Encoded)
s UnicodeText=$ZCONVERT(BinaryText,"I","UTF8")

Now if your Base64Encoded, UTF-8 encoded string cannot be longer than 3,641,144 bytes then you can ignore the rest of this reply.  Just execute the code excerpt from the %SYSTEM.Encryption.Base64Encode Class Reference documentation.  If you have a %Stream that is too long then you will need to loop over substrings of the %Stream and you should read on.

If you read from a long %Stream a sequence of substrings then it might not be possible to simply call $system.Encryption.Base64Encode(BinaryText,...) and $ZConvert(BinaryText,"I","UTF8") and because the substrings can be broken between binary bytes that must be combined in order to do the conversion.  So you will need to workaround some issues in the code that was part of the %SYSTEM.Encryption.Base64Encode Class Reference documentation.

Issue one: the SET BinaryText=$ZCONVERT(UnicodeText,"O","UTF8") statement can return more characters of BinaryText than there are characters of UnicodeText.  The new string length may not be a perfect multiple of 3 characters long and $system.Encryption.Base64Encode converts sequences of 3 bytes into 4 bytes so when the Base64Encode(substring) method is called with a substring that has a non-multiple of 3 byte length then the extra 1 or 2 bytes at the end of the substring must be saved and then concatenated onto the beginning of the next substring to be passed to the Base64Encode(substring) method.  Only the very last call on the Base64Encode(substring) method can have a substring byte length which is not a multiple of 3.

Issue two: the "SET UnicodeText=$ZCONVERT(BinaryText,"I","UTF8")" statement might be given a BinaryText substring that ends with an incomplete UTF-8 sequence of characters.  That incomplete sequence must be concatenated onto the beginning of the next BinaryText substring.  Fortunately, the $ZCONVERT function takes an optional fourth argument which is a local variable name.  Evaluating $ZCONVERT(UnicodeText,"O","UTF8",handle) will do its conversion on an input value containing the concatenation of handle with UnicodeText.  When $ZCONVERT is done then the new value of handle will either be the empty string or handle will contain the unconverted substring at the end of the input value.  So before your loop which decodes the %Stream containing Base64 encoded UTF-8 bytes you should execute SET handle="" and inside your decoding loop you should call $ZCONVERT using handle as a forth argument variable.  If at the exit of your decoding loop the variable handle does not contain the empty string then your input stream was ill-formed.

Note that when your decoding loop reads a Base64Encoded substring from the %Stream then I am assuming you read a perfect multiple of 4 bytes from the %Stream before you execute SET BinaryText=$system.Encryption.Base64Decode(Base64Encoded).  [[ If the Base64 encoded %Stream was not generated using the code described above then I am also assuming the Base64 encoded BinaryText does not contain any additional white-space characters, or if it does contain white space then those white-space characters were removed before building a substring that contains a perfect multiple of 4 bytes. ]]  Each sequence of 4 bytes in Base64Encoded will be turned into 3 bytes in BinaryText.

Extension of the %Set and %Get methods of the %DynamicAbstractObject classes to support type parameters (such as using "stream" as a type) is an IRIS feature and not a Caché feature.  In Caché the %ToJSON and %FromJSON methods can accept %File-s and %Stream-s.  These methods are generally limited to Dynamic Objects/Arrays involving sizes related to the largest 32-bit signed integer.  But in Caché the %ToJSON/%FromJSON methods work only on entire Dynamic Objects/Arrays.  Using %Set/%Get in Caché to modify an element of a Dynamic Object/Array is limited by the length of an ObjectScript %String (currently 3,641,144 characters in Caché.)

In recent and future IRIS releases the sizes of a %DynamicAbstractObject will be limited by the amount of virtual memory available to the process.  (Please consider avoiding the activation of many multi-gigabyte Dynamic objects/arrays at the same time.)  The future IRIS versions of the %Get/%Set methods will support type keywords involving streams which will allow a %Get/%Set to access Dynamic Object/Array elements involving any size that can be fetched from (or stored into) a %Stream.  Future versions of these methods will also encode and decode Base64 representation while transferring byte data between a %Stream and an element of a Dynamic Object/Array.

For Set MyObj.JSON=JSONString.%ToJSON() to signal <INVALID OREF> then either local variable MyObj is not an oref or local variable JSONString is not an oref.  Make sure that JSONString is really a %DynamicObject by doing a WRITE JSONString before doing the Set command.  The WRITE statement should write some like 99@%Library.DynamicObject .
 

I agree that comments here will have little effect on developers other than InterSystems developers or InterSystems customers.  However, processing of JSON is controlled by the European standard "ECMA-404 The JSON Data Interchange Format".  If a JSON decoder deliberately ignores this standard specification then it is not a *JSON* decoder.  If any customer discovers that InterSystems's %FromJSON method is violating the ECMA-404 standard then I am sure InterSystems will fix this situation.  I suspect that third-party developers would also fix such a violation of ECMA-404.  The ECMA-404 document actually contains examples of 4 different strings that are identical to "/".  ECMA-404 does not require that JSON encoders generate the "\/" string syntax but this standard does require that all decoders accept this string syntax.

There are some issues with above C code for manipulating a string of $LIST elements.

I do not believe the above code will work on a big-endian platform, such as PowerPC running the AIX operating system.  Conversions from $LIST representation to numeric representation will order the bytes backwards on big-endian hardware.

Decimal floating-point values in IRIS and Caché have almost 19 decimal digits of precision while the above code translates these numbers to IEEE binary double-precision floating-point values which have less than 16 decimal digits of precision.  This means that $LISTBUILD(0.3) will be converted by the above code into the value 0.29999999999999998889... .  The above code also introduces a double-round error so that decoding $listbuild(x) and $listbuild($double(x)) will not always be equal because the $double(x) function in IRIS/Caché will do the conversion from decimal to binary without a double round.

The above code is inconsistent in resulting type of integer values.  Consider,
USER>set L5A=$LISTBUILD(5),L5B=$LISTBUILD(50/10)
USER>WRITE $LISTSAME(L5A,L5B)
1

The $LIST elements in L5A and L5B contain the same value, 5.  However, the above code will convert L5A to the C int64_t type while it will convert L5B to the C double type.  If the value of the integer is greater than 2**53 then the above C code can convert the identical integer values into different integer values.

Also, the above code does not correctly handle all the special cases when (type==FLOAT) is true.

The IRIS/Caché $LIST representation is not as simple as most people think.  There are some unusual rules that must be followed if you want to get the same results as the $LISTxxx functions get in IRIS/Caché.
 

Agreed that there is nothing wrong with escaping a solidus.  However, saying "most decoders" will unescape the solidus correctly" is not quite correct.  "*ALL* decoders will unescape the solidus correctly."  If a decoder does not correctly unescape the solidus then it is *NOT* a JSON decoder.  The %FromJSON method in the %DynamicObject class in IRIS and Caché versions after 2016.1 will do the following:

USER>write x
{"FileStatus":"P","Path":"\/somepath\/test\/test123\/filename.txt","InterchangeOID":"100458"}
USER>set y={}.%FromJSON(x)

USER>write y.Path
/somepath/test/test123/filename.txt

So %FromJSON is the method that breaks JSON apart and removes all escaping from JSON string values.  (I assume that $fromJSON will do the same in Caché version 2016.1.  The JSON manipulation added to version 2016.1 was changed significantly in all future releases.  Most JSON manipulation programs written in version 2016.1 will require rewriting when porting to a later release.)
 

Let us assume that you are using the %DynamicAbstractObject class and its subclasses, which include %DynamicArray and %DynamicObject (instead of using the legacy support for JSON in the %ZEN package.)  

If you have an element of a JSON array or JSON object that contains more characters than are supported by an ObjectScript string then the way to manipulate that element is to send it to a %File or a %Stream.  In the present releases of Caché and IRIS Data Platform the only way to do that is to use the %ToJSON(outstream) method call.  If the element in question is in a nested sequence of JSON elements then you can first use a sequence of %Get method calls to access the closest containing %DynamicArray or %DynamicObject.  You can then use %ToJSON(outstream) method call to create a %Stream containing the textual JSON of that entire object.  Then process the contents of that %Stream using the Read(len,...)/Readline(len,...) methods, with the 'len' parameter chosen to provide blocking that will prevent exceeding ObjectScript string length limits.  Some JSON parsing will be necessary to find the beginning and the ending of the JSON formatted string containing the element in question so that the required element can be copied as the only contents of a new %Stream.

If you can control how the JSON is formatted then you can choose to enclose the long JSON string element inside a JSON array containing just that one element--the long string value requiring additional manipulation.  Parsing such an %DynamicArray element would be easy--send the Array to a %Stream using %ToJSON and then ignore the first two characters, [", and the last two characters, "], of the %Stream.  What remains is contents of the string element with a few special characters encoded using \-sequences.  If you cannot control how the JSON is formatted then it still might be possible to use knowledge of the application that is using JSON representation to short cut the parsing issues.

In a future InterSystems product release there will certainly be an extension to the %Get method that will allow any %DynamicAbstractObject element (including string and numeric elements) to be sent to a %Stream using either Raw String representation or using JSON representation.  You will also be able to copy such an element from one JSON array/object into another JSON array/object without needing to copy the characters to a %Stream that is using an external representation.

Essay on Types and their Representations within InterSystems Object Script

Backwards compatibility is a reason why some things that looked normal in 1977 (date of first ANSI MUMPS standard) now look unusual in 2017.  InterSystems Object Script is based on the ANSI MUMPS language (more recently called ANSI M) but Object Script has undergone quite a bit of extension beyond that standard (and not all those extensions were designed by InterSystems so there are some inconsistencies, see $ZHEX example below.)

The original MUMPS standard said that a subscript string containing the canonical numeric character representation of a numeric value was identical to that number so the original standard allowed an implementation where the only supported data type was the character string.  When used as global subscripts, the canonical numeric subset of string values were sorted in numeric order before the strings that did not contain the canonical character representation of a number.  Those non-numeric subscript strings sorted in textual order.  Thus when used as a subscript, the string "2", a canonical number, is sorted before "1.0", a text string different from the canonical number "1".  The canonical numeric string "1" does sort before "2".

The original implementation of Caché could use several different internal representations for a numeric value besides also supporting a character string representation.  These additional internal representations helped improve performance when executing Object Script programs. These initial numeric representations included an integer representation and a decimal floating-point representation.

The $LIST family of functions are Object Script extensions that provide a way to encode a list of multiple Object Script values in a single string value.  Internally a $LIST string need not store "identical" values using identical internal $LIST representations.  Avoiding conversions between different representations is done for performance reasons while building a $LIST.  Thus, the following lists are represented by different packed strings $lb("230"), $lb(230), $lb(23e1), $lb(2300e-1) but the $LISTSAME function assumes that all four of these lists are identical.  E.g., so the Object Script statement:

   WRITE $lb(230) = $lb(2300e-1) will write 0 while WRITE $LISTSAME($lb(230),$lb(2300e-1)) will write 1.

There are about 21 strings different from $LB(23) that $LISTSAME will assume are identical to the string value $LB(23).

The ZZDUMP command dumps the internal representation of a list and it can expose the some of the different internal representations that are used for the same value.  Copying a value using string representation will always use string representation as you move it in and out of a $LIST. Copying a value using numeric representation in and out of a $LIST will not change its numeric value but you might get different internal numeric representations on different Caché instances.

InterSystems Object Script avoids conversions that change between internal numeric representations and the equivalent string representations because we have inherited features from other vendors of extended MUMPS implementations that treat a numeric value differently from the corresponding canonical string value.  E.g.,  the function calls $ZHEX(10) and $ZHEX("10") give very different answers.

USER>WRITE $zhex("10"),!,$zhex(10)
16
A

Generally you can apply the unary-plus operator, +, to an Object Script string expression to change it from string representation into an Object Script numeric representation.  The unary-plus operator is a conversion operator so it works on strings that do not contain canonical numeric representation.  (E.g., +"7.0", +"+700E-2" and +"7Dwarves" all convert the canonical numeric value 7.)  However, applying the unary-plus operator to a canonical numeric string will sometimes involve a conversion that changes the value because the various internal numeric representations have a more limited range and a more limited accuracy than that supported by the canonical numeric strings that can be sorted using numeric ordering.

There are other extensions added to InterSystems Object Script that extended the set of values supported by the original MUMPS standard.

InterSystems now supports two different string representations.  There is the original 8-bit character strings and there is also a representation using the 16-bit UTF-16 Unicode encoding.

Consider the Object Script expression ##class(%DynamicArray).%New().  It returns an oref value (different from a numeric or a stsring value) which is a reference to an object data structure defined by the %DynamicArray class.  This particular oref value references structured data that is similar to the JSON array constructor [ ].  The set of Class language defined oref values is the largest extension that InterSystems has made to the original standardized set of MUMPS values.

Also consider $double(230), which returns an IEEE 64-bit binary floating-point value which is equal to the decimal floating-point value 230.  The $DOUBLE( ) extension is useful for applications using scientific data encoded using the representation defined by the IEEE binary floating-point standard.  However, binary floating-point arithmetic gives quite different results than decimal floating-point arithmetic.  E.g.

USER>WRITE $double(230),!,230   ;; Gives identical looking answers
230
230
USER>WRITE $double(230)/100,!,230/100   ;; but this shows different computational results.
2.2999999999999998223
2.3

There are also recent extensions to the Object Script language to support JSON objects.  When a JSON constructor is written using standard JSON constant syntax then the values are stored internally as JSON values.  Retrieving a value from a JSON object or JSON array by using the %Get method in an Object Script expression will need to convert the JSON value to a compatible Object Script value.  When coding Object Script statements and expressions, the Object Script language supports an extended the JSON constructor syntax where a JSON element value in a JSON object or array can be replaced with a parenthesized Object Script expression.  These parenthesized expressions are evaluated using InterSystems Object Script semantics.

E.g.,

USER>SET x=["230",230,23e1,2300E-1]  ;; JSON standard syntax

USER>SET y=[("230"),(230),(23e1),(2300E-1)] ;; extended expression syntax with parentheses

USER>WRITE x.%ToJSON(),!,y.%ToJSON()
["230",230,23e1,2300E-1]
["230",230,230,230]

Note that the %DynamicArray constructor value stored in variable x contains JSON numeric syntax while the %DynamicArray constructor value stored in variable y is using Object Script representation although the string representation has been kept separate from the numeric representations.

E.g., We can use the %Get( ) method to convert a JSON value to an Object Script value:

When the Object Script %DynamicArray method %Get is applied to variable x then the resulting value will be converted to Object Script representation because the %Get method call is part of an Object script expression.  E.g.,

USER>WRITE x.%Get(0),!,x.%Get(3)
230
230

Note:  The two output lines containing "230" look identical but internally the first output line is the result of a string write and the second output line is the result of a numeric write.  Also, note that numbers supported by JSON can exceed the capacity of the internal representations supported by Object Script.  Rounding or overflow can occur when converting a JSON numeric element for use in an Object Script expression.  E.g.,

USER>SET z=[3E400]  ;; No error placing a large JSON number into a %DynamicArray

USER>WRITE z.%Get(0)  ;; But converting the JSON number gets a <MAXNUMBER> signal

WRITE z.%Get(0)
^
<MAXNUMBER>

Yes, in my original posting I meant to use the "sorts after" operator, ]], when I incorrectly used the "follows" operator, ].  I have corrected the original post.

Comment on the statement that decimal numbers with more than 18 significant digits are strings.  This is often true but it does not affect the accuracy of collating canonical numeric strings.

(1) Strings containing *canonical* representations of decimal numbers collate according to their numeric value and *NOT* according to their string value.  (Every non-repeating decimal value has a unique canonical representation in COS.  See http://docs.intersystems.com/latest/csp/docbook/DocBook.UI.Page.cls?KEY=... for a description of the canonical numeric representation used by Caché.)  This collation rule applies even for canonical numeric strings with many more that 19 significant digits.

(2) When Caché does arithmetic on decimal values it generates results with approximately 18.96 decimal digits of precision.  I.e. all the integers between 1 and 9223372036854775807 can be represented as an arithmetic result.  All the integers between 1000000000000000000 and 9223372036854775807 are integers with 19 digits of precision that can be the result of a Caché arithmetic computation.  But the next larger decimal numbers that can result from a Caché arithmetic computation are 9223372036854775810, 9223372036854775820, 9223372036854775830 and so on up to 9999999999999999990.  This range of decimal arithmetic will only have 18 digits of precision even though they represent 19 digit integer values. Then come 20 digit integers which go back to having 19 digits of precision starting with 10000000000000000000, followed by 10000000000000000010, 10000000000000000020, 10000000000000000030 and so on up to 92233720368547758070, which is largest 20 digit decimal integer with 19 digits of precision,  the next possible arithmetic result is 92233720368547758100, which is a 20 digit number with only 18 digits of precision and which is followed by 92233720368547758200, etc.  [[You can ignore these details unless you really need more than 18 digits of precision when doing arithmetic computations in the COS language.]]

Now for an example using canonical numeric strings with 20 digits with a mixture of 20, 19 and 18 significant digits.  In this range of values Caché supports 19 digits of precision.  We also have two non-canonical numeric strings which do not collate in numeric order but instead collate in character value order after all the canonical numeric strings have collated.

USER>set ^a("09")="noncanonical index"
USER>set ^a("01234567890123456780")="noncanonical index"
USER>set ^a("12345678901234567800")="numeric index"
USER>set ^a("12345678901234567870")="numeric index"
USER>set ^a("12345678901234567874")="string index" 
USER>set ^a("12345678901234567876")="string index"
USER>set ^a("12345678901234567880")="numeric index"
USER>set ^a("12345678901234567890")="numeric index"
USER>set ^a("12345678901234567900")="numeric index"
USER>zw ^a                            
^a(12345678901234567800)="numeric index"
^a(12345678901234567870)="numeric index"
^a("12345678901234567874")="string index"
^a("12345678901234567876")="string index"
^a(12345678901234567880)="numeric index"
^a(12345678901234567890)="numeric index"
^a(12345678901234567900)="numeric index"
^a("01234567890123456780")="noncanonical index"
^a("09")="noncanonical index"

Note that strings containing canonical numbers with 20 significant digits (i.e.,  "12345678901234567874" and "12345678901234567876") have quotes around them because Caché cannot convert them to numeric format without rounding the low order significant digit into a zero digit.  Note also that strings containing non-canonical numeric representation (i.e., "01234567890123456780" and "09") appear last with quotes around the subscript values because these string indices collate as string values.  If you do arithmetic on the 20 digit canonical arithmetic strings (e.g.,  +"12345678901234567874" and +"12345678901234567876") then they will be treated as having the value of nearest decimal number with 19 significant digits (i.e.,  values 12345678901234567870 and 12345678901234567880 respectively.) 

Note: given two canonical numeric strings then the COS "sorts after" operator, ]], returns 1 if the first canonical numeric string operand is greater-than than the second canonical numeric string operand.  (I.e., the "]]"-operator returns 1 if the first string operand collates after the second string operand.) This greater-than collation when applied to canonical numeric strings is correctly computed regardless of the number of significant digits in the canonical numeric strings.  However, using the COS numeric "greater-than" operator,  >,  will first convert the operands to numeric representation with only 18.96 digits of precision before doing the numeric comparison.

Examples:
USER>write "12345678901234567874" ]] "12345678901234567870"
1
USER>write "12345678901234567874" > "12345678901234567870"
0
The second write statement returns 0 because the numeric ">" operator rounds it first operand to 19 significant digits which will compare equal-to the numeric value 12345678901234567870.

Summary:  When it comes to collating canonical numeric strings, the number of significant digits supported is limited only by the string length.  When it comes to doing decimal arithmetic in Caché, the computational results are rounded so they have less than 19 decimal digits of precision.