Dmitry Maslennikov · Jul 26, 2023 go to post

If your enterprise still uses mechanical disks, that's still a factor that has to be considered

All the blocks required to be read to get through the list can be placed all around the disk/s. And it takes time. On a live system, with many changes, and when a lot of different data is stored, the next block can be far from the previous one. So, on mechanical discs, defragmentation is matter, and may slow the speed.

I don't know the character of your data, but if you have a lot of data, stored in the globals, it will require to read much more blocks, to even just count the items.

And most probably the easiest way to solve it, is just to use bitmap index.

Dmitry Maslennikov · Jul 26, 2023 go to post

If you wish to have a conditional filter, then you can use it this way

SELECT * FROM SomeTable
WHERE (:FilterName   = '' OR Name   LIKE :FilterName)
  AND (:FilterValue1 = '' OR Value1 LIKE :FilterValue1)
  AND (:FilterValue2 = '' OR Value2 LIKE :FilterValue2)
Dmitry Maslennikov · Jul 26, 2023 go to post

As I mentioned above, it's a complicated question, and to be sure what's really happening, and why the speed is so slow, it requires a look deeply into the database file and into hardware.

Dmitry Maslennikov · Jul 26, 2023 go to post

110 minutes, it seems impossible, or, it's something way too wrong

USER>set ts = $zh f i=1:1:65000000 { set ^YYY(i)="somedata" } write !,"elapsed: ", $zh-ts 

elapsed: 11.126631

USER>s sub="", count = 0, ts = $zh for { set sub = $Order(^YYY(sub)) quit:sub=""  set count = count + 1 } write !,"elapsed: ", $zh-ts 

elapsed: 9.549079

Here result in seconds

Yes, for sure, my example is too simple, and too far from any real situation. 

And there are multiple issues that may happen with your data, it can be how it is stored, where it is stored, how much data in values. And it's difficult to suggest how to check it

Have a look at my series of articles about globals in the database, just for information, it may help understand something, what may go wrong

In any case, there is a right way to count objects, without counting all of them this way. Is using bitmap index, which you can use even if you have own storage, and do not use objects yet. You still able to build own bitmap index, and count items by this index will be at least 64000 times faster, whereas 64000 is just chunk size for bitmap, and speed will vary depends if you don't have much empty spaces between id's, which needs to be numeric

Dmitry Maslennikov · Jul 18, 2023 go to post

What is the actual concern, then?

From my experience on different systems with highload, I did not see effects, that could point to think about it. 

Dmitry Maslennikov · Jul 18, 2023 go to post

Python may help

Class dc.Demo
{

ClassMethod ValidateJSON(data As %String = "") As %Status [ Language = python ]
{
import iris
import json
from json import JSONDecodeError

try:
    json.loads(data)
    return iris.system.Status.OK()
except JSONDecodeError as ex:
    return iris.system.Status.Error(5001, f"{ex.msg}: line {ex.lineno} column {ex.colno} (char {ex.pos})")
except Exception as ex:
    return iris.system.Status.Error(5001, repr(ex))
}

}

And result

USER>set status = ##class(dc.Demo).ValidateJSON("{""aa"":123 ""name"": ""value""}") do:'status $system.OBJ.DisplayError(status) 
ERROR #5001: Expecting ',' delimiter: line 1 column 11 (char 10)

USER>set status = ##class(dc.Demo).ValidateJSON("{""aa"": true, ") do:'status $system.OBJ.DisplayError(status) 
ERROR #5001: Expecting property name enclosed in double quotes: line 1 column 14 (char 13)

USER>set status = ##class(dc.Demo).ValidateJSON("{""aa"": wrong ") do:'status $system.OBJ.DisplayError(status) 
ERROR #5001: Expecting value: line 1 column 8 (char 7)

USER>set status = ##class(dc.Demo).ValidateJSON("{""aa"": true}") do:'status $system.OBJ.DisplayError(status)
Dmitry Maslennikov · Jul 17, 2023 go to post

try this

set objectContext = ##class(Some.Class).%OpenId(123)
write ##class(Ens.Rule.Definition).EvaluateRules("Rule.ClassName",,objectContext)
Dmitry Maslennikov · Jul 14, 2023 go to post

The error on screen shot saying Access Denied, why do you think it's Timeout?

Check the amount of connection available for you, and how much connection you do

Dmitry Maslennikov · Jul 13, 2023 go to post

Not so much magic in it, if you already know how to read it, the content of XData is just a Stream, which can be used both ways. Just Write to it, and %Save it. And you will probably need to compile the class

Class Demo.Test
{

XData OpenAPI [ MimeType = application/json ]
{
}

ClassMethod Update()
{
    set xdata = ##class(%Dictionary.XDataDefinition).IDKEYOpen($ClassName(), "OpenAPI")
    do xdata.Data.Write("{}")
    do xdata.%Save()
}

}

In VSCode, the content of XData will not appear after that, because it happened on the server, and VSCode will not see the changes, you'll need to export it manually

I've published online demo, where you could play with two executions of FHIR Load for two different patients.

This uncovers some interesting discoveries, like, about 40% of the time is spent on IndexResource

And about 20% to ValidateResource 

Of course, this can't be very accurate, but can help to understand what actually happens during the process, and at what moment. Including finding places, where it's not just reads of globals, but sets or even kills. Something like, 17% of GloKills are happening during CommitTransactionBundle

34

ClassMethod Check(w As %String) As %Boolean [ Language = python ]
{
return len(w)==len(set(w.lower()))
}

You can try to install Haproxy, and configure it for http2 or http3, but define backend as http1. So, it will look like HTTP/2/3 from the browser but still work as HTTP/1 in Caché/IRIS. And only if you have a lot of static files, which you can process independently from IRIS, it may help.

While the requests is ended up in IRIS, than IRIS is responsible for the response, and it should support responses in HTTP/2 which is very different from HTTP/1. In HTTP/1 it does not matter how many connections are, queries will be processed one by one, and responses will go accordingly one by one. But in HTTP/2 queries processed simultaneously and response will go to the client as soon as it's done, no matter where it's started and it goes through the same connection, while in HTTP/1 connection per request. And IRIS and for sure Caché does not have support for it, the only way it's working is to process one request per connection and response as soon as the result is ready.

So, even if you manage to mimic HTTP/2 somehow, it will not help almost at all, the queries in the connection still process synchronously.

So, if your application is using CSP files, the only way is to completely rewrite it with some external framework, Python could be a solution, most of the popular frameworks can work this way, just select one you like more, Django, Flask, FastAPI or whatever you find. Even if you have just only REST, it will not be easy to implement. 

I see no reasons to do it on ObjectScript, it will be quite difficult to achieve it.

Well, the issue is that Caché, or even IRIS, still uses one Job to process requests for the same session. So, anything higher than 1.1 will not help at all. So, the only solution is to make sure that as much more possible static files are processed without Caché/IRIS or WebGateway, through a webserver configured for HTTP/2/3. And only API requests which require data would go to Caché/IRIS, and best case if it will be session-less queries, meaning that your requests are not tied to the session on the server, and those queries could be processed in parallel, and everything needed can be reconstructed from the query, e.g. username to check permissions should go from Authorization header.

Moving from HTTP/1.1 to HTTP/2/3 is not as simple as you expect, differences between protocols are significant. And requires a lot of work on the application.

Yes, I forgot to mention, that it will work only from terminal, real one (with any webterminal will not work too).

It's the limitation of ZBreak, which is used in this project

So, the only case is to wrap needed in classmethod which will be possible to call from terminal

So, I think the best is to create an issue, describing what's wrong with inspectdb, what's missing in the models, and how you solve it, manually, and I could have a look and probably solve it on Django side.

Dmitry Maslennikov · Jun 30, 2023 go to post

Well the current version of IPM allows to build and publish to some private registry deployed packages. 

So, for companies that would like to distribute their packages through IPM, they can install own registry provide authentication for users who bought their product, so they will be able to connect to the registry and install the package.

Dmitry Maslennikov · Jun 29, 2023 go to post

You can't publish manually to the public repository, you can do it only through OpenExchange, check IPM during application creation, and with the next release, it will publish your project for you

If you want to publish to your repo, you just have to specify login and password, with command like this

repo -r -n myrepo -url https://server/registry/ -user "test" -pass "test"
Dmitry Maslennikov · Jun 29, 2023 go to post

I'm sure that the best way to test REST API would be to use some external tools, such as Postman/newman

You can look at my example project, which contains running tests against REST using newman

But it also contains some tests for REST on the IRIS side

Dmitry Maslennikov · Jun 26, 2023 go to post

dispatch class, which just named disp next to impl class, is generated and generated classes are hidden by default and you can show them with a flag in the Server Explorer. But, this class always regenerates, when you compile your spec file.

have a look at this example, of how I deal with errors

Dmitry Maslennikov · Jun 26, 2023 go to post

Another solution is to use getattr

classname="%SYSTEM.SYS"
methodname="ProcessID"
pid = getattr(iris.cls(classname), 'ProcessID')()

myerror=iris.cls("%Exception.PythonException")._New("MyOops",123,"def+123^XYZ","SomeData")
for propertyname in ["Name","Code","Data","Location"]:
    print(getattr(myerror, propertyname))