Stefan Cronje · Oct 25, 2024 go to post

My view on this solution started as that is will not work practically, but the more I thought about the scenarios, the more feasible it looks. My thinking was as follows.

When is temp table built, when the page is opened? Because that will cause a waiting time for the first load, especially if no filters have been applied yet. But in this scenario, you can just select like the top 200 or something else, as long as it is not all the rows.

Then also, every time a filter changes or the page size selection changes, you will have to rebuild the temp table. This means your temp table's lookup needs to include filters and page size to determine that it has changed for the session. It is not in the solution above, but not difficult to implement.


If you hit the indexes correct and get a small result this may be useful.

What about REST APIs, which are usually built to end the session after each request.
This will not work for REST APIs requiring pagination. There can be worked around this by letting the front-end pass in a value for the session which is more related to the front-end session.

You will also need to build some cleanup mechanism to purge rows after a session has ended. The front-end can send you some instruction on a logout, but not if the browser is just closed. It will have to be a task that runs at night or some other random time and truncate it.

Stefan Cronje · Jul 19, 2024 go to post

Tried it out, but id does not indicate when a method call or class is deprecated.

Stefan Cronje · Jul 19, 2024 go to post

Thank you.

I will check it out. Doing one by one is a bit impractical as I have around 5200 classes in the project.

Stefan Cronje · Apr 5, 2024 go to post

I have also been playing around now with SQL inserts and updates to see how the DB reacts to uncommitted data.

If I insert a record after a tstart and it has not been committed yet, then in another session I set IsolationMode to 1 and do a select on the table for that specific record.

It does not give me an SQL 100 as I would have expected. It is a new record and hasn't been committed yet.
What it did give is -114 (Unable to get Shared Lock), BUT the values were still put into the binded variables I selected into.

With an update the same thing. I got a SQLCODE of -114, but the new values were actually put into the binded variables.

I guess there is no real uncommitted data functionality then.

Stefan Cronje · Apr 5, 2024 go to post

I do want it in the Transaction. If the commit of the records related to this event being published fails, this event should not be rolled back too.

Problem is that it may have been picked up by the other process already.

I like the idea of using the PPG and merge after the commit. This does however add more complexity to the code in "remembering" to do the merge after the commit.

Stefan Cronje · Jan 12, 2024 go to post

Here is an example of this concept working. On Ensemble, but should be the same for IRIS.
Note that I configured password authentication on the web application and set up Authentication on Postman.

Code:

Class Tests.RESTFormData Extends%CSP.REST
{

ClassMethod Test() As%Status
{
	w"Posting: ",$Get(%request.Data("Posting",1))
	q1
}

XData UrlMap
{
<Routes>
	<Route Url="/" Method="POST" Call="Test" />
	</Routes>
}

}

Web Application Config:

 

Postman:

 

Stefan Cronje · Jan 12, 2024 go to post

Can you post the raw request message of Postman here please?
Or the complete request sent, including the headers?

Stefan Cronje · Jan 11, 2024 go to post

I want the class route.

I did a lot of abstraction for the UI, like the page header, sidebar (menu) and footer. Re-usable classes for each of these page sections, as well as being able to create a "template" page to extend from, then let each specific page add its content into it. I also needed security on menu items and resource access control across pages, so abstraction made sense.

I did it a bit like you would with Django templates.

I might post an example of how I did this at some point. I used the Material Dashboard Pro UI framework, so I can't share that specific content due to copyright issues on the CSS and JS included.

Stefan Cronje · Jan 11, 2024 go to post

Can you provide the full class? Including what it extends and so forth? Also the code of where you construct and send the request.

The following is a working example of uploading a file using a CSP Page.

Class Test.FileUpload Extends%CSP.Page
{

ClassMethod OnPage() As%Status
{
	&html<<html><head></head><body>>If%request.Method = "POST" {
		Set tFile = ##class(%Stream.FileBinary).%New()
		Set tSC = tFile.LinkToFile("C:\Tmp\" _ %request.MimeData("FileStream",1).FileName)
		If 'tSC {
			Write !,"Upload Failed"
		} Else {
			Do tFile.CopyFrom(%request.MimeData("FileStream",1))
			Set tSC = tFile.%Save()
			If 'tSC {
			Write !,"Upload Failed"Do tFile.%Close()
			} Else {	
				Write !,"File Posted",!
			}
		}
	}
	&html<
<h2>Upload a File</h2><formenctype="multipart/form-data"method="post"action="">
    Enter a file to upload here: <inputtype=filesize=30name=FileStream><hr /><ul><inputtype="submit"value="Upload file"></ul></form></body></html>>Quit$$$OK
}

}
Stefan Cronje · Dec 22, 2023 go to post

Thank you. I will try this.

I will dig a bit deeper. All the processes run under one of the built-in system users. So if I can get the contending PID, then I can try and get that PID's info to log.

Stefan Cronje · Dec 21, 2023 go to post

I haven't considered it.

It is not a deadlock. An update fails in a process, I log the exception, and the process moves on to the next record to process. This sometimes happens when batch runs occur between 00:00 and 05:00 AM.

When that exception occurs, I need it to programmatically find the contending process' information and the node locked, to log that information so that I can investigate it at a later time.

Stefan Cronje · Nov 16, 2023 go to post

Hi,

I am not sure I know what the issue is exactly. Is it speed, is it DB writes, or memory (FRAMESTACK) errors.

In the meantime, some things we've done to help with large files.

We have BPL's that process CSV files with 100k rows, and these things helped.

Stefan Cronje · Nov 6, 2023 go to post

Agree. Use the default settings.

They can be inserted with SQL statements.

Also, if you want to add Configuration Items to the Production post deployment, without having to recompile or change the production manually, use a session script, or create a classmethod or routine that can import a CSV containing the relevant information.

E.g.

&sql( INSERT INTO Ens_Config.DefaultSettings (Deployable, Description, HostClassName, ItemName, ProductionName, SettingName, SettingValue) 
VALUES (1, 'My setting description', 'My.Operation', 'My Nifty Operation', 'My.Production', 'TheSetting', 'SettingValue'))

and for the Production:

Set tProduction = ##class(Ens.Config.Production).%OpenId("My.Production")
Set tObj = ##class(Ens.Config.Item).%New()
Set tObj.Name = "My Nifty Operation"
Set tObj.ClassName = "My.Operation"
Set tObj.PoolSize = 1
Set tObj.Enabled = 1
Set tObj.LogTraceEvents = 0
Set tSC = tProduction.Items.Insert(tObj)
Set tSC = tProduction.%Save()
w $System.Status.GetErrorText(tSC)
Set tSC = ##class(Ens.Director).UpdateProduction()
w $System.Status.GetErrorText(tSC)

You can also open existing confiritems in order to update the Pool Size or any of those other settings that are not configurable in the "System Default Settings" for some reason.

Stefan Cronje · Feb 23, 2023 go to post

That is helpful, thank you.

I just find it strange that it is quite non-standard compared to other compilers/interpreters

Stefan Cronje · Feb 18, 2023 go to post

Hi,

There is an item on the InterSystems Ideas portal for this. You can go vote for it if you want to.

But currently, it is not a built-in feature.

Stefan Cronje · Feb 14, 2023 go to post

Not currently that I am aware of.

There is the Installer Manifest you can use to automate a lot of this during deployment, but there is no export to create such a manifest.

Maybe add this as an Idea on InterSystems Ideas...

Stefan Cronje · Feb 12, 2023 go to post

If you want to use the group by, then you should probably do the count where the group by is being done, and use distinct by as you had it.

select distinct by(serial,hq) hq, count(1)
from thetable
group by hq

If you want it per that grouping.

There are no filters in it, so it is going to do a full table scan and compare and calculate values for each row. Taking the amount of time it takes, it is actually fast.

Maybe look into Bitslice indexes. It might help but at a cost of performance on insert and update:
InterSystems Documentation

Stefan Cronje · Feb 12, 2023 go to post

For a start:

select distinct by (VISIT_SERIAL_NO,HQ_ORG_CODE) VISIT_SERIAL_NO,HQ_ORG_CODE
can be changed to
select distinct VISIT_SERIAL_NO,HQ_ORG_CODE

It will do the same.

Secondly:
Will you please remove the %parallel and click on "Show Plan". Post that plan here. It will help to determine where the query is slow. It might be using the wrong index. There are many.

Lastly:
Have you tuned the table and checked the result after that?

Stefan Cronje · Feb 11, 2023 go to post

Thank you for the clarification.

If rollback and commit is supported, then verifying the results is just the step of doing a select before committing in order to verify the update/insert was correct and as expected.

Nothing special to it or automated in any way.

This is great. Thank you.

Stefan Cronje · Feb 11, 2023 go to post

I updated it like that last week. Apparently, I did not send it for approval, which I thought i did.

Stefan Cronje · Feb 10, 2023 go to post

This is a great tool.

I am wondering if it will work for everyone. In the world of finance, you do not get SSH access to servers.
Most of the times the super-server port is also closed off for everything except the web gateway.

If the web version can be run on it, it is great - but in banking environment, not everyone is on the "containerised" buzz yet, so this will not be allowed.

Sure, I can probably install and configure the package and set up the web application.

Now there are two things left I want to raise:

  1. Multi-line SQL without having SSH access. Also do not have SCP or SFTP access.
    1. If this is present, and I have missed it, I apologise.
  2. Database transactions.
    1. I have a SQL shell I built a long time ago, which worked with db transactions.
    2. Doing DML, you may want to verify the results before committing it to the DB and have to option to rollback.
      1. This will be really great if the app can handle it.
Stefan Cronje · Feb 10, 2023 go to post

Thank you for your response on this. I see what you are saying regarding the different problems being solved.

This is basically the agenda I am pushing - let's talk about packages and what should be and should not be packaged together, what we need, etc. BUT without adding "red tape" that will demotivate community members from contributing.

I seem to be giving a lot of 2c today. wink

Stefan Cronje · Feb 10, 2023 go to post

I understand completely. As a side comment, which is a bit off topic, I like what you've done and think I will contribute to it when time allows. I created the old Dynamic Object Adapter pacakge for Ensemble. There are things in there we can add to the OpenAPI suite if needed, for example the reverse of API first. Existing class definitions to swagger type thing. 

Stefan Cronje · Feb 10, 2023 go to post

Great idea this.

May I suggest one change. Use a different tag for the unit test code.

The <example> tag is used to display nicely formatted code in the documatic for a dev - like a one liner on how to use it or something like that, or a just a block of code for context.

Now if you put in an example, it is going to be mixing with the unit tests.

Stefan Cronje · Feb 10, 2023 go to post

Thank you for the information and the proposal to have a brainstorming session as a community.