Autoscaling Business Service to consume available CPU

Question

Question

Eduard Lebedyuk · Jan 19, 2023

#Business Service #Interoperability #InterSystems IRIS

I have a production with one Business Host - a Business Service which I need to scale automatically to consume ~80% of CPU time.
Business Service pulls data from a (non-FIFO) queue so that I can adjust pool size without any issues.

So far, I'm planning a different BS running every X seconds and sampling CPU with $system.Process.GetCPUTime() and scaling the pool size of the main BS up/down based on that metric.

Has anyone tried something similar? Any advice/code samples would be appreciated.

Product version: IRIS 2022.1

Discussion (6)3

Log in or sign up to continue

score 0 · Answer 1 · 2023-01-19T04:41:18-05:00

Standard queues provide at-least-once delivery, which means that each message is delivered at least once. FIFO queues provide exactly-once processing, which means that each message is delivered once and remains available until a consumer processes it and deletes it. Duplicates are not introduced into the queue.

..so I guess that you mean standard queue with several worker processes dequeuing items from the queue. In this case the CPU utilization would likely depend on the number of workers, wouldn't it?

score 0 · Answer 2 · 2023-01-19T10:23:01-05:00

CPU utilization would likely depend on the number of workers, wouldn't it?

It sure does. Question is how to scale workers to optimize queue consumption

score 0 · Answer 3 · 2023-01-19T06:23:52-05:00

How is the BS pulling the data? From files? From SQL? Or?

I think you need to parallelize the dequeuing, but I don't think a BS can have more than one job. You can move the dequeuing to a Business Operation, in that case you can assign as many jobs (pool size) as required.

Enrico

score 0 · Answer 4 · 2023-01-19T07:42:32-05:00

It is possible for a BS to have more than one job - the issue becomes how to control the multi process access to the inbound feed in an orderly manner. Hence as you asked it is important to know how the BS is pulling data.

Eduard, Is the rate at which one BS can process known or is it variable based on data unit to be processed? Similarly is the rate of arrival known or possible to detect? If possible to know these 2 rates they could be used to control BS count rather than CPU (a tight looping BS looking for work could skew CPU test comes to mind)

Is the design to have all the processing of a data unit in the BS rather than pass to a BP/BO?

James

NB: there is a variation on pool size meaning for a BS using TCP based listening adapter where there can only be one listener but when Job Per Connection is true pool size setting is used as the maximum allowed number of concurrent connections created by the listener.

score 0 · Answer 5 · 2023-01-19T10:33:30-05:00

Eduard, Is the rate at which one BS can process known or is it variable based on data unit to be processed?

It is variable based on data unit to be processed. Data unit (file) size varies between 1Kb and 100Mb.

Similarly is the rate of arrival known or possible to detect?

IRIS BS pulls messages, so as soon as BS job is done with the message, next message is pulled from the external queue (AWS SQS).

Is the design to have all the processing of a data unit in the BS rather than pass to a BP/BO?

Yes, it's a stateless app, so I need to process message and report success/error immediately since container can be reprovisioned at any time.

score 0 · Answer 6 · 2023-01-20T15:25:08-05:00

How about using Work Queue Manager framework in conjunction with a BS?

Pick up a message but do not delete before we start processing since we are not going to record anywhere permanently that we are working on it.
Check if message queued in current lifetime temp list.
If already queued skip.
If not already queued:
   mark as in progress for duration of this container lifetime
   queue up for Work Queue Manager to process.
   use callbacks to say finished and delete the message.perhaps also give back answer as well.
We then also have auto scaling using the WQM framework if I am not mistaken.