Reading through file fast

Question

Question

Rochdi Badis · Mar 20, 2023

Hi Guys,

I've the below code that copies 15000 records into a global, problem is that taking around 11 secs to finish and I'm wondering if there is any faster way (eg. would copying the file into stream and look through the stream makes it faster!?) because I've hundreds of files I need to process every time.

Set File=##Class(%File).%New(FileAfUnZip)
Do File.Open("RS")
While 'File.AtEnd{
Set ReadData=File.ReadLine()
S ^DumpTempSeries(Equipment,Sensor,ID,MeasureDT,$I(i))=+$P(ReadData,",",5)_"^"_+$P(ReadData,",",6)_"^"_+$P(ReadData,",",7)
Quit:i=Total
}

Thanks

Product version: Caché 2014.1

Discussion (4)1

Log in or sign up to continue

Dmitry Maslennikov · Mar 20, 2023

Is it really the reading file taking so much time or using $piece on the line and setting it to global too?

There are various things here that may slow you, even $increment (best to be replaced by i+1)

You can also split the reading file and set it to global by two parts, and use $sortbegin

Try to run your code with %SYS.MONLBL started, it will help you understand where it spends more time.

0 0

score 0 · Answer 1 · 2023-03-20T08:04:15-04:00

Thanks Dmitry, and actually changing to code to copy to filestream then read from it improved a lot, down to 1 sec

Thank again

score 0 · Answer 2 · 2023-03-20T08:32:48-04:00

Hi Rochdi,

You don't need to use %File and then copy it to a file stream. You can just use %Stream.FileCharacter to directly open and read the file:

  Set stream=##class(%Stream.FileCharacter).%New()
  Set sc=stream.LinkToFile("c:\myfile.txt")
  While 'stream.AtEnd {
  Set line=stream.Read()
  ; Process the chunk here
  }

And I can confirm that I've also seen %Stream.FileCharacter significantly outperform %File for reads.

-Marc

score 0 · Answer 3 · 2023-03-20T20:01:30-04:00

Thanks Mac that's more or less what I did but didn't use LinkToFile, rather used Filename

Set stream=##class(%FileCharacterStream).%New()
Set stream.Filename=FileAfUnZip
While 'stream.AtEnd {
Set ReadFFTData=stream.ReadLine()

}

Thanks