New post

#Database Transaction Processing

6

Databases and other data stores which treat the integrity of data as paramount often include the ability to handle transactions to maintain the integrity of data. A single transaction consists of one or more independent units of work, each reading and/or writing information to a database or other data store.

Documentation.

Hi all,

I have a situation where I am writing to global from different processes - let's call it an Event Queue.
There is then a different process that uses $order to read through the data - the PubSub processing BP.

Sometimes the writing of this global will form part of a transaction that also insert data into other tables, using tstart and tcommit.
If there is some issue, it will roll back, and the entry written to the global will also roll back.

0 6
0 74

Work Queue Manager (WQM) is a feature of InterSystems IRIS that enables you to improve performance by distributing work to multiple concurrent processes programmatically. The idea is that you split the work into chunks, and WQM distributes the chunks across worker processes and can provide the info that the work is done.

However, recently, I got an interesting question: there's a large logical transaction composed of ~1,000,000 individual objects and SQL inserts and updates. Some updates are CPU-intensive, so the original idea was to use WQM to split an update into chunks to speed things up.

But, here's a catch: if one of the individual 1,000,000 changes fails (there's a variety of application-level checks so that it can fail, and that's not even that abnormal a behavior), the entire transaction must be rolled back. That creates a problem: each chunk must report success before committing their individual transactions, and someone must get all these reports and decide if we are committing or not.

Unfortunately, it looks like WQM does not have a bidirectional communication between workers and manager, so I suggested an approach using events:

  1. Start jobs.
  2. Wait for all jobs to report success using $System.Event.WaitMsg().
  3. Send Commit or Rollback using the $System.Event.Signal().
2 9
1 173

Hi,

The system I am working on processes large numbers of records. Inserts, updates and so forth. There are multiple processes that can potentially work on the same table at the same time.

It is an almost impossible task to try and time these processes to not run at the same time, due to volumes, SLAs, etc.

Every now and then there are a few locking issues, Not many considering the volumes. It is just a bit of a nuisance as we need to reprocess these records later.

0 6
0 151