Using multiple processes to process a single file

Question

Question

Stella Ticker · May 12, 2017

It is taking several hours to read a large text file because the while loop uses ReadLine() ! Is there some way in Cache to process a single file using multiple processes. Something comparable to this:-

http://stackoverflow.com/questions/11196367/processing-single-file-from-...

Discussion (7)3

Log in or sign up to continue

Sean Connelly · May 12, 2017

Please see previous comment to same question...

https://community.intersystems.com/post/multi-threading-improve-performance

You previously mentioned your file contains 500,000 records. How big are the records?

If each record is just a few k in size then reading and writing each record from the file to a global will take under 5 minutes.

In which case you have a serious bottleneck going on such as a poor referential integrity check. If this is not indexed or tuned then you will have some serious IO thrashing going on, and no matter how many cores you throw at the problem, you will not get any overall performance gains.

1 0

Sean Connelly · May 12, 2017

> What is your definition of a few k? Each line is about 25000 KB.

Do you mean 25,000 characters (25K)?

> Previous comment recommended processing the file as a single message stream. That ended up slowing the message viewer so much for these large messages that that it is impossible to view the message at all.

You can override the content display method on your Ens.Request class so that it doesn't display the entire message. You can replace this with a small summary about the file, size, no of records etc.

Creating 500,000 Ensemble messages is going to generate a lot of IO that you probably don't need.

I would still recommend processing them as one file.

2 0

score 1 · Answer 1 · 2017-05-12T11:34:50-04:00

Hi,

it depends a little bit what exactly the bottleneck is. If it's the processing, you could possibly split the reading from the processing by inserting a scheduling mechanism and just have the read process queue up things and several worker threads for the processing.

If the bottleneck is the reading, there is not much you can do. Reading from the same file in multiple locations is going to make it slower overall.

Best,
Fab

score 0 · Answer 2 · 2017-05-12T11:35:40-04:00

John Murray · May 12, 2017

Can you show us the code of your while loop, in case DC members can suggest performance improvements?

0 0

score 0 · Answer 3 · 2017-05-12T11:53:11-04:00

This is the while loop in the code of a FileService class that extends EnsLib.RecordMap.Service.FileService.

While 'tInputStream.AtEnd {
Set tSC = ..GetRecordObject(tInputStream, .tRecordObject,,.tLookAhead)
If $$$ISERR(tSC) Quit

           Set tSC = ..SendRequest(tRecordObject, 1)
           If $$$ISERR(tSC) Quit
       }

The requests are asynchronously sent to a target. But there is significant time being taken for the while loop to complete

score 0 · Answer 4 · 2017-05-12T12:06:49-04:00

Sean Connelly · May 12, 2017

Are you creating 500,000 Ensemble messages from one file?

0 0

score 0 · Answer 5 · 2017-05-12T12:23:34-04:00

Previous comment recommended processing the file as a single message stream. That ended up slowing the message viewer so much for these large messages that that it is impossible to view the message at all because the stream is too large .So this line by line approach is being explored.

What is your definition of a few k? Each line is about 25000 KB.