Written by

Question Nezla · Mar 20, 2023

Reading through file fast

#Ensemble

Hi Guys,

I've the below code that copies 15000 records into a global, problem is that taking around 11 secs to finish and I'm wondering if there is any faster way (eg. would copying the file into stream and look through the stream makes it faster!?) because I've hundreds of files I need to process every time.

Set File=##Class(%File).%New(FileAfUnZip)
Do File.Open("RS")
While 'File.AtEnd{
Set ReadData=File.ReadLine()
S ^DumpTempSeries(Equipment,Sensor,ID,MeasureDT,$I(i))=+$P(ReadData,",",5)_"^"_+$P(ReadData,",",6)_"^"_+$P(ReadData,",",7)
Quit:i=Total
}

Thanks

Product version: Caché 2014.1

Discussion (4)1

Add reply

Comments

Dmitry Maslennikov · Mar 20, 2023

Is it really the reading file taking so much time or using $piece on the line and setting it to global too?

There are various things here that may slow you, even $increment (best to be replaced by i+1)

You can also split the reading file and set it to global by two parts, and use $sortbegin

Try to run your code with %SYS.MONLBL started, it will help you understand where it spends more time.

0 0

Nezla Mar 20, 2023 to Dmitry Maslennikov

Thanks Dmitry, and actually changing to code to copy to filestream then read from it improved a lot, down to 1 sec

Thank again

0 0

Marc Mundt Mar 20, 2023 to Nezla

Hi Rochdi,

You don't need to use %File and then copy it to a file stream. You can just use %Stream.FileCharacter to directly open and read the file:

Set stream=##class(%Stream.FileCharacter).%New()
  Set sc=stream.LinkToFile("c:\myfile.txt")
  While 'stream.AtEnd {
  Set line=stream.Read()
  ; Process the chunk here
  }

And I can confirm that I've also seen %Stream.FileCharacter significantly outperform %File for reads.

-Marc

0 0

Nezla Mar 20, 2023 to Marc Mundt

Thanks Mac that's more or less what I did but didn't use LinkToFile, rather used Filename

Set stream=##class(%FileCharacterStream).%New()
Set stream.Filename=FileAfUnZip
While 'stream.AtEnd {
Set ReadFFTData=stream.ReadLine()

}

Thanks

0 0