Question
· Jan 7

Optimizing performance of production using Pool Size>1

#Performance #Ensemble #InterSystems IRIS for Health

Hi Everyone!

We have an existing process (running in FIFO order), all business hosts having pool size=1.

Currently we are reading records from a file (one record at a time) then that record goes to business process for further processing and finally through the business operation. As of now we are using synchronous call in our existing code. Before processing the last record we are using hang of 50 seconds because we need to initiate a batch once the processing of last record is finished.

This process is taking too much time for a file having 0.1 million records (max records in a file).

  
Now we want to increase the performance of this process, so that processing can be faster by modifying the synchronous call with asynchronous call and removing the hang for last record. 

While modifying the code, we have met with a condition which counts the processed records and save it in a class using below code

&sql (Select ID into :RowID from Data.A where MetadataID=:MetadataID)

set obj=##class(Data.A).%OpenId(RowID)

set obj.ProcessedCount=obj.ProcessedCount + 1

When the pool size=1, the count was updating correctly but when I tested with pool size=2 or pool size=5, the counts are not getting updated correctly.

I further tried using SQL update but no luck & even tried with %NOLOCK condition if SQLCODE'=0 but that is also not working.

&sql(Update Data.A set ProcessedCount = ProcessedCount + 1 Where ID =:RowID)
if SQLCODE'=0 {
    &sql(Update Data.A set ProcessedCount = ProcessedCount + 1 Where ID =:RowID)
}    

I believe we cannot use a lock condition before updating ProcessedCount, because it will again process them one by one taking the similar amount of time even with pool size >1

What can be the ideal solution for this issue. Please suggest.

Thanks in advance

Product version: IRIS 2024.1
$ZV: IRIS for UNIX (Red Hat Enterprise Linux 9 for x86-64) 2024.1 (Build 263U) Wed Mar 13 2024 15:23:29 EDT
Discussion (4)3
Log in or sign up to continue
DC AI Bot · Jan 7

To optimize the performance of a production system using Pool Size > 1, consider the following suggestions:

  1. Parallel Processing using Pool Size:
    Setting a pool size greater than 1 enables multiple concurrent jobs to process messages. This approach is useful if the ordering of messages does not matter. When pool size is increased, each additional pool job adds an OS-level process, which incurs memory and CPU usage based on the work being done. Start with small pool size increments (e.g., 2 or 4) and monitor the impact on queues and system performance before increasing further [1][2].

  2. Message Ordering Concerns:
    If message order is important, asynchronous calls with larger pool sizes can lead to inconsistencies. To address this:

    • Use an intermediate process configured with Pool Size = 1 to ensure first-in-first-out (FIFO) processing [3].
    • Alternatively, apply a mechanism such as message routing rules by metadata (e.g., splitting queues by certain keys like patient IDs) to maintain order for related messages across threads [3].

  3. Concurrent Updates (e.g., ProcessedCount):
    Concurrent processing can cause record updates to fail or become inconsistent. If synchronization issues arise:

    • Avoid direct SQL updates using %NOLOCK when multiple threads are involved. Locks are necessary to ensure the atomicity of operations but may reduce concurrency.
    • Implement robust mechanisms for updating shared counters, such as lock-based synchronization or queuing updates within dedicated threads to avoid SQL collisions [1][3].

  4. Dynamic Pool Size Adjustments:
    Dynamically scaling the pool size based on production load can enhance performance further. This can be achieved through scripts or services that periodically review workload metrics (queue length, CPU, memory) to adjust pool sizes. However, excessive resizing can introduce overhead [4].

  5. Asynchronous Call Improvements:
    Ensure asynchronous calls are properly implemented to maximize throughput while avoiding bottlenecks in subsequent stages (e.g., business processes or operations). Consider whether dependent components also require pooling adjustments [1].

  6. Performance Monitoring:
    Regularly monitor the queues and system resources. Long or stuck queues can indicate performance issues, potentially highlighting when to scale pool size or address bottlenecks (e.g., external system delays) [5].

Try testing these ideas and adjusting settings incrementally for better insight into the optimal configuration for your workload requirements [1][2][3][5].

Sources:

0 0
Enrico Parisi · Jan 7

Unless your post is missing crucial code/info, the SQL update you posted is atomic and should work even with concurrent processes updating the same row. No need of %NOLOCK or retry (is that second update a retry?).
It will not be atomic if you use %NOLOCK.

If you want to use object code to update ProcessedCount, then:

Set obj=##class(Data.A).%OpenId(RowID,4)
Set obj.ProcessedCount=obj.ProcessedCount + 1
Set sc=obj.%Save()
; handle error here
Kill obj

I feel your post/code misses some info.

Also, what do you mean with "not working"?

 

0 0