go to post Rubens Silva · Jun 2, 2017 %DynamicAbstractObject instances have a method %Set(key, value) that you can use to insert new or update existing values.set o = { "foo": "bar" }write o.%ToJSON() // {"foo": "bar"}do o.%Set("foo", "bars")write o.%ToJSON() // {"foo": "bars"}
go to post Rubens Silva · Jun 2, 2017 By default, Base64Decode and Base64Encode are functions used to decode and encode datatypes, or best saying... STRING. Since you want to encode a stream the Decoder must understand that it should continue from the last chunk position instead of assuming a new string, otherwise you'll get a corrupted result. Here's how XML Writer outputs an encoded binary. /// <method>WriteBase64</method> encodes the specified binary bytes as base64 and writes out the resulting text. /// This method is used to write element content.<br> /// Argument:<br> /// - <var>binary</var> The binary data to output. Type of %Binary or %BinaryStream. Method WriteBase64(binary) As %Status { If '..InRootElement Quit $$$ERROR($$$XMLNotInRootElement) If ..OutputDestination'="device" { Set io=$io Use ..OutputFilename:(/NOXY) } If ..InTag Write ">" Set ..InTag=0 If $isObject(binary) { Do binary.Rewind() Set len=12000 While 'binary.AtEnd { Write $system.Encryption.Base64Encode(binary.Read(.len),'..Base64LineBreaks) } } Else { Write $system.Encryption.Base64Encode(binary,'..Base64LineBreaks) } If ..OutputDestination'="device" { Use io } Set ..IndentNext=0 Quit $$$OK }
go to post Rubens Silva · Jun 1, 2017 I see, is there some kind of security implemented or planned? Like an authentication mechanism.Also, one single license. I'm starting to see how things work now, if I understood correctly most of the heavy-lifting is done using NodeJS.Similar to sending a payload with multiple requests from multiple client connections, limited by a threshould, let's say... about 1000 requests. So, Caché simply resolves all of them and return them at once. NodeJS looks what's relevant for each client and forwards the filtered response.But that is only possible if you host an intermediate server, do you use something like Express?
go to post Rubens Silva · Jun 1, 2017 Thanks for posting about it's usage.It's looks pretty straightforward and with a low learning curve.You mentioned earlier about using TCP over CSP, does that means you implemented a messaging protocol over TCP layer instead of using HTTP? Is that lib restricted for NodeJS usage or it could be webpack'ed for example? Nevermind, I just re-read the part where you mentioned about client API.How do you deal with license usage? How much does it escalates with a fair amount of users and how do you manage all of that?
go to post Rubens Silva · Jun 1, 2017 for i=1:1:collection.Count() { set item = collection.GetAt(i) // From this point it depends of the type of item you're storing. // You must know to fetch each property values. // Ex: set value = item.property (%ZEN.proxyObject) // set value = item.GetAt("property") (%ArrayOfDataTypes) } Change the values and reassign it to each related property. do item.SetAt("property", newValue) set item.property = newValue
go to post Rubens Silva · Jun 1, 2017 So do you need to iterate over a Caché %Collection or a Javascript array?Which one?If you need to iterate over Javascript arrays WHILE transforming each item, it's just like Dmitry said.
go to post Rubens Silva · May 31, 2017 And here's the fixtures used for this test.https://mega.nz/#!USAlUZBS!zhmdlL-ovp0mq8ACic2BBqia_HxJWqmBwXFkiMkbmgQ
go to post Rubens Silva · May 31, 2017 Here's a snapshot of my last test run.It's portuguese though. But I think you can figure the essentials. There's a few things to notice (and some to fix).1 - The biggest file is MG.json, it's a JSON containing all coordinates to render the biggest state from here. You can see it takes about 6.5 minutes to parse all data. But it's a special case, not something common to be parsed. Btw, I said 5~8 MB, but eh... it's about 3 MB (3.810.312 bytes), my mistake. 2 - blns.json.txt is a nasty sure-fire JSON file. I took it from this repo and trying to figure how to solve a parsing issue. If you want something to risk breaking your parser, really, try this file.3 -Some chars below might not be shown correctly, that is due to unicode support. That's because of assertions related to unescaping paired unicode sequences like: \uD83D\uDE02 that generates a smile. tests begins ... UnitTest.JQX.JSON.Parser begins ... TestArrayMiscParsing() begins ... AssertEquals:Array precisa conter 6 valores (passed) AssertEquals:Posição 1 do array deve ser um objeto (passed) AssertEquals:Posição 1 do array deve ser um proxy (passed) AssertEquals:Proxy deve conter propriedade 'foo' definido com valor 'bar' (passed) AssertEquals:Proxy deve conter propriedade 'foo' definida como outro proxy (passed) AssertEquals:Posição 2 do array deve um %ListOfDataTypes (passed) AssertEquals:Deve conter 5 valores (passed) AssertEquals:Posição 1 deve ser ter valor 1 (passed) AssertEquals:Posição 2 deve ser ter valor 2 (passed) AssertEquals:Posição 3 deve ser ter valor 3 (passed) AssertEquals:Posição 4 deve ser um %ListOfDataTypes (passed) AssertEquals:O array dessa posição deve ter um item (passed) AssertEquals:Com o valor 15 (passed) AssertEquals:Posição 5 deve ser um proxy (passed) AssertEquals:O proxy deve ter uma propriedade 'well' com valor 'done' (passed) AssertEquals:De volta ao array principal, a posição 3 deve ser null (passed) AssertEquals:De volta ao array principal, a posição 4 deve ser 'null' equivalendo null (passed) AssertEquals:De volta ao array principal, a posição 5 deve ser true (passed) AssertEquals:De volta ao array principal, a posição 6 deve ser 'true' equivalendo true (passed) LogMessage:Duration of execution: .269543 sec. TestArrayMiscParsing passed TestArrayNumberParsing() begins ... AssertEquals:Array precisa conter 3 valores (passed) AssertEquals:Posição 1 do array deve ser 1 (passed) AssertEquals:Posição 2 do array deve ser 2 (passed) AssertEquals:Posição 3 do array deve ser 2.50 (passed) LogMessage:Duration of execution: .045573 sec. TestArrayNumberParsing passed TestArrayReservedParsing() begins ... AssertEquals:Array precisa conter 3 valores (passed) AssertEquals:Posição 1 do array deve ser true (passed) AssertEquals:Posição 2 do array deve ser false (passed) AssertEquals:Posição 3 do array deve ter null (passed) LogMessage:Duration of execution: .047348 sec. TestArrayReservedParsing passed TestArrayStringParsing() begins ... AssertEquals:Array precisa conter 3 valores (passed) AssertEquals:Posição 1 do array deve ter 'quite' (passed) AssertEquals:Posição 2 do array deve ter 'some' (passed) AssertEquals:Posição 3 do array deve ter 'string' (passed) LogMessage:Duration of execution: .052526 sec. TestArrayStringParsing passed TestEscapedStringParsing() begins ... AssertEquals:Deve retornar um sorriso �� (passed) AssertEquals:'a' deve ter valor 1 (passed) AssertEquals:'c' deve ter conter 3 itens (passed) AssertEquals:A primeira posição deve conter o sorriso de novo �� (passed) AssertEquals:Deve ter a seguinte mensagem: Isso é um "teste" �� (passed) AssertEquals:Deve conter um objeto vazio (passed) LogMessage:Duration of execution: .104971 sec. TestEscapedStringParsing passed TestMultilineParsing() begins ... LogMessage: LogMessage:******************************************************* LogMessage:*** <!> Testando arquivo blns.json.txt <!> *** LogMessage:******************************************************* LogMessage: ERRO #5001: Syntax Error: Expected value separator or array ending, but found ASCII code 92 at offset 207 LogMessage: LogMessage:******************************************************* LogMessage:*** <!> Testando arquivo example.json <!> *** LogMessage:******************************************************* LogMessage: LogMessage:Duração em segundos: .07764 LogMessage:Início do parsing: 05/31/2017 13:43:38 LogMessage:Fim do parsing: 05/31/2017 13:43:38 AssertEquals:Deve retornar um objeto a partir do arquivo example.json (passed) AssertEquals:id deve ter valor igual a string 0001 (passed) AssertEquals:type deve ter valor igual a string donut (passed) AssertEquals:name deve ter valor igual a string Cake (passed) AssertEquals:ppu deve ter valor numerico igual .55 (passed) AssertEquals:A propriedade batter deve ter 4 itens (passed) AssertEquals:id de batter deve ter valor igual 1001 (passed) AssertEquals:type de batter deve ter valor igual Regular (passed) AssertEquals:id de batter deve ter valor igual 1002 (passed) AssertEquals:type de batter deve ter valor igual Chocolate (passed) AssertEquals:id de batter deve ter valor igual 1003 (passed) AssertEquals:type de batter deve ter valor igual Blueberry (passed) AssertEquals:id de batter deve ter valor igual 1004 (passed) AssertEquals:type de batter deve ter valor igual Devil's Food (passed) AssertEquals:id de topping deve ter valor igual 5001 (passed) AssertEquals:type de topping deve ter valor igual None (passed) AssertEquals:id de topping deve ter valor igual 5002 (passed) AssertEquals:type de topping deve ter valor igual Glazed (passed) AssertEquals:id de topping deve ter valor igual 5005 (passed) AssertEquals:type de topping deve ter valor igual Sugar (passed) AssertEquals:id de topping deve ter valor igual 5007 (passed) AssertEquals:type de topping deve ter valor igual Powdered Sugar (passed) AssertEquals:id de topping deve ter valor igual 5006 (passed) AssertEquals:type de topping deve ter valor igual Chocolate with Sprinkles (passed) AssertEquals:id de topping deve ter valor igual 5003 (passed) AssertEquals:type de topping deve ter valor igual Chocolate (passed) AssertEquals:id de topping deve ter valor igual 5004 (passed) AssertEquals:type de topping deve ter valor igual Maple (passed) AssertEquals:id deve ter valor igual a string 0002 (passed) AssertEquals:type deve ter valor igual a string donut (passed) AssertEquals:name deve ter valor igual a string Raised (passed) AssertEquals:ppu deve ter valor numerico igual .55 (passed) AssertEquals:A propriedade batter deve ter 1 itens (passed) AssertEquals:id de batter deve ter valor igual 1001 (passed) AssertEquals:type de batter deve ter valor igual Regular (passed) AssertEquals:id de topping deve ter valor igual 5001 (passed) AssertEquals:type de topping deve ter valor igual None (passed) AssertEquals:id de topping deve ter valor igual 5002 (passed) AssertEquals:type de topping deve ter valor igual Glazed (passed) AssertEquals:id de topping deve ter valor igual 5005 (passed) AssertEquals:type de topping deve ter valor igual Sugar (passed) AssertEquals:id de topping deve ter valor igual 5003 (passed) AssertEquals:type de topping deve ter valor igual Chocolate (passed) AssertEquals:id de topping deve ter valor igual 5004 (passed) AssertEquals:type de topping deve ter valor igual Maple (passed) AssertEquals:id deve ter valor igual a string 0003 (passed) AssertEquals:type deve ter valor igual a string donut (passed) AssertEquals:name deve ter valor igual a string Old Fashioned (passed) AssertEquals:ppu deve ter valor numerico igual .55 (passed) AssertEquals:A propriedade batter deve ter 2 itens (passed) AssertEquals:id de batter deve ter valor igual 1001 (passed) AssertEquals:type de batter deve ter valor igual Regular (passed) AssertEquals:id de batter deve ter valor igual 1002 (passed) AssertEquals:type de batter deve ter valor igual Chocolate (passed) AssertEquals:id de topping deve ter valor igual 5001 (passed) AssertEquals:type de topping deve ter valor igual None (passed) AssertEquals:id de topping deve ter valor igual 5002 (passed) AssertEquals:type de topping deve ter valor igual Glazed (passed) AssertEquals:id de topping deve ter valor igual 5003 (passed) AssertEquals:type de topping deve ter valor igual Chocolate (passed) AssertEquals:id de topping deve ter valor igual 5004 (passed) AssertEquals:type de topping deve ter valor igual Maple (passed) LogMessage: LogMessage:******************************************************* LogMessage:*** <!> Testando arquivo MG.json <!> *** LogMessage:******************************************************* LogMessage: LogMessage:Duração em segundos: 395.318218 LogMessage:Início do parsing: 05/31/2017 13:43:39 LogMessage:Fim do parsing: 05/31/2017 13:50:14 AssertEquals:Deve retornar um objeto a partir do arquivo MG.json (passed) LogMessage: LogMessage:******************************************************* LogMessage:*** <!> Testando arquivo municipios.json <!> *** LogMessage:******************************************************* LogMessage: LogMessage:Duração em segundos: 10.035081 LogMessage:Início do parsing: 05/31/2017 13:50:15 LogMessage:Fim do parsing: 05/31/2017 13:50:25 AssertEquals:Deve retornar um objeto a partir do arquivo municipios.json (passed) LogMessage: LogMessage:******************************************************* LogMessage:*** <!> Testando arquivo package.json <!> *** LogMessage:******************************************************* LogMessage: LogMessage:Duração em segundos: .057244 LogMessage:Início do parsing: 05/31/2017 13:50:25 LogMessage:Fim do parsing: 05/31/2017 13:50:25 AssertEquals:Deve retornar um objeto a partir do arquivo package.json (passed) LogMessage: LogMessage:******************************************************* LogMessage:*** <!> Testando arquivo random.json <!> *** LogMessage:******************************************************* LogMessage: LogMessage:Duração em segundos: .518465 LogMessage:Início do parsing: 05/31/2017 13:50:25 LogMessage:Fim do parsing: 05/31/2017 13:50:25 AssertEquals:Deve retornar um objeto a partir do arquivo random.json (passed) LogMessage: LogMessage:******************************************************* LogMessage:*** <!> Testando arquivo random_bigger.json <!> *** LogMessage:******************************************************* LogMessage: LogMessage:Duração em segundos: 1.920557 LogMessage:Início do parsing: 05/31/2017 13:50:25 LogMessage:Fim do parsing: 05/31/2017 13:50:27 AssertEquals:Deve retornar um objeto a partir do arquivo random_bigger.json (passed) LogMessage: LogMessage:******************************************************* LogMessage:*** <!> Testando arquivo schema.json <!> *** LogMessage:******************************************************* LogMessage: LogMessage:Duração em segundos: .03808 LogMessage:Início do parsing: 05/31/2017 13:50:27 LogMessage:Fim do parsing: 05/31/2017 13:50:27 AssertEquals:Deve retornar um objeto a partir do arquivo schema.json (passed) AssertEquals:Deve conter uma propriedade $schema com o valor http://json-schema.org/draft-04/schema# (passed) AssertEquals:Deve conter um objeto chamado 'definitions' (passed) AssertEquals:Deve conter um objeto chamado 'address' (passed) AssertEquals:Deve conter uma propriedade 'type' em 'address' com o valor object (passed) AssertEquals:Deve conter um objeto chamado 'properties' em 'address' (passed) AssertEquals:Deve conter um objeto chamado 'street_address' em 'properties' (passed) AssertEquals:Deve conter uma propriedade 'type' em 'street_address' com o valor string (passed) AssertEquals:Deve conter um objeto chamado 'city' em 'properties' (passed) AssertEquals:Deve conter uma propriedade 'type' em 'city' com o valor string (passed) AssertEquals:Deve conter um objeto chamado 'state' em 'properties' (passed) AssertEquals:Deve conter uma propriedade 'type' em 'state' com o valor string (passed) AssertEquals:Deve conter uma coleção em 'address' chamada 'required' (passed) AssertEquals:Deve conter 3 valores na coleção mencionada (passed) AssertEquals:Posicao 1 deve ter 'street_address' (passed) AssertEquals:Posicao 2 deve ter 'city' (passed) AssertEquals:Posicao 3 deve ter 'state' (passed) LogMessage: LogMessage:******************************************************* LogMessage:*** <!> Testando arquivo youtube.json <!> *** LogMessage:******************************************************* LogMessage: LogMessage:Duração em segundos: .092909 LogMessage:Início do parsing: 05/31/2017 13:50:27 LogMessage:Fim do parsing: 05/31/2017 13:50:28 AssertEquals:Deve retornar um objeto a partir do arquivo youtube.json (passed) LogMessage:Duration of execution: 409.506008 sec. TestMultilineParsing passed TestObjectKeyArrayValueParsing() begins ... AssertEquals:Objeto deve ser um proxy (passed) AssertEquals:Objeto precisa conter propriedade 'a' com valor de um array (passed) LogMessage:Duration of execution: .012107 sec. TestObjectKeyArrayValueParsing passed TestObjectKeyFalseValueParsing() begins ... AssertEquals:Objeto deve ser um proxy (passed) AssertEquals:Objeto precisa conter propriedade 'a' com valor true (passed) LogMessage:Duration of execution: .012377 sec. TestObjectKeyFalseValueParsing passed TestObjectKeyNullValueParsing() begins ... AssertEquals:Objeto deve ser um proxy (passed) AssertEquals:Objeto precisa conter propriedade 'a' com valor null (passed) LogMessage:Duration of execution: .015294 sec. TestObjectKeyNullValueParsing passed TestObjectKeyObjectValueParsing() begins ... AssertEquals:Objeto deve ser um proxy (passed) AssertEquals:Objeto precisa conter propriedade 'a' com valor de um proxy (passed) LogMessage:Duration of execution: .01224 sec. TestObjectKeyObjectValueParsing passed TestObjectKeyStringValueParsing() begins ... AssertEquals:Objeto deve ser um proxy (passed) AssertEquals:Objeto precisa conter propriedade 'a' com valor 'string' (passed) LogMessage:Duration of execution: .012922 sec. TestObjectKeyStringValueParsing passed TestObjectKeyTrueValueParsing() begins ... AssertEquals:Objeto deve ser um proxy (passed) AssertEquals:Objeto precisa conter propriedade 'a' com valor true (passed) LogMessage:Duration of execution: .012159 sec. TestObjectKeyTrueValueParsing passed TestObjectMultiKeyObjectValueParsing() begins ... AssertEquals:Objeto deve ser um proxy (passed) AssertEquals:Objeto precisa conter propriedade 'a' como sendo um proxy (passed) AssertEquals:Objeto precisa conter propriedade 'b' como sendo um proxy (passed) AssertEquals:'b' deve ter foo definido com valor 'bar (passed) LogMessage:Duration of execution: .024768 sec. TestObjectMultiKeyObjectValueParsing passed TestObjectMultiKeyStringValueParsing() begins ... AssertEquals:Objeto deve ser um proxy (passed) AssertEquals:Objeto precisa conter propriedade 'a' com valor 'string' (passed) AssertEquals:Objeto precisa conter propriedade 'b' com valor 'foo' (passed) LogMessage:Duration of execution: .019991 sec. TestObjectMultiKeyStringValueParsing passed UnitTest.JQX.JSON.Parser passed Skipping deleting classes tests passed
go to post Rubens Silva · May 31, 2017 Interesting...Can you provide some example of usage for that node package?
go to post Rubens Silva · May 31, 2017 I do suppose it's easier to extract parameters from ReturnTypeParams than parsing then through property documentations. Especially if there's a case with multiple parameters. So I think that's more a technical debt situation than an end-user preference.
go to post Rubens Silva · May 31, 2017 Yeah, this approach forces the user to specify the return type.
go to post Rubens Silva · May 31, 2017 Here's how we did.It works for 2010, in that case I expose the controller methods and using &p1=id ... &pN=idN to retrieve a JSON collection of SamplePerson serialized instances.Notice PUBLIC, methods that have this parameter are available to be called over HTTP. Method getUsers(args... As Sample.Person) As %ListOfObjects(PUBLIC=1){ set persons = ##class(%ListOfObjects).%New() set sumEq = "" set sum = 0 for i=1:1:args { do persons.Insert(args(i)) } quit persons}I'll check if I can remove the proprietary code and make it available as well.
go to post Rubens Silva · May 31, 2017 It does (EDIT: nope, it doesn't, because it's easier). I use it for publishing (or making RPC-aware methods).And it doesn't even requires PropertyClass, instead of properties.Yeah, I also had to do something along these lines.Though I can't publish the source code for now, since it does contains business related authentication code.
go to post Rubens Silva · May 31, 2017 90 and 100 lines for serializers is quite the achievement.Oh, I see, so that's how you introduced decorators.Now about JSON parsing...In my case I had to provide out-of-the-box support for streams and keep the parsing process the faster as I could, since we intended to use it for parsing files that weighted around 5~8 MBs to populate the database. CSV or XML wasn't an option at that time.I can provide the code for that.About NodeJS ORM:Also, how are you connecting NodeJS to Caché? Is it using the native driver provided by InterSystems? If so, are you dealing with globals directly or did you implemented an adapter for NoSQL?
go to post Rubens Silva · May 31, 2017 Outstanding!I'll keep a close eye on that thread.Since you're using something close to decorators/annotations, does that means you made a COS syntax parser, lexer and AST?I'm also very interested about how fast and what strategies you used for parsing JSONs.Here is a techical background:So far, when realising performance tests I had to opt-out from using the default %Stream classes's MoveTo method, since it always rewinds the position and moves the pointer , this caused me great performance headaches. So I had to implement some sort of algorithm that lazy loads and calculates the position.I'm wondering if you had to implement something focused on older Caché versions and faced such issues, because that was my case. I had to implement a JSON parser for the 2010 version, due to our client and partner restrictions.
go to post Rubens Silva · May 29, 2017 If you need faster IO but taking care with data size, I would recommend you to use local variables.Memory IO is always faster than disk based storage.That being said, 1GB is quite expensive to be allocated for a single process. Thus, for your case I would recommend using process private globals.
go to post Rubens Silva · May 26, 2017 Here.If you need an explanation, the method I used for example is part of an algorithm that keeps the repository in-sync with the the project.You're right, your approach is the best, even though it's not cross-platform. Such issue could be solved by using $$$isUNIXand $$$isVMS though.Something like:if $$$isUNIX set command = "find %1"
go to post Rubens Silva · May 26, 2017 Vitaliy is faster that's probably because Caché is delegating the control to the OS's native API.That is indeed the best approach, and could be made cross-platform by using $$$isUNIX, $$is$WINDOWS and $$$isVMS.Now I gotta say, I'm impressed by these results. Ex: Using find -iname %1 instead of dir.Then, what's the advantage of using %SQL.Statement over %ResultSet? The only reason I can think about is for using it's metadata now.EDIT: Ahh, there's a detail. %FileSet is not persisted, neither is using SQL. It's a custom query being populated by $zsearch internally.Maybe for SQL based queries %SQL.Statement would be better.
go to post Rubens Silva · May 26, 2017 What the ... ? wow.I'll have to update my code if that's true for every case. Could you measure the execution time for both approaches?
