Question
· Mar 20, 2023

Reading through file fast

Hi Guys,

I've the below code that copies 15000 records into a global, problem is that taking around 11 secs to finish and I'm wondering if there is any faster way (eg. would copying the file into stream and look through the stream makes it faster!?) because I've hundreds of files I need to process every time. 

Set File=##Class(%File).%New(FileAfUnZip)
Do File.Open("RS")
While 'File.AtEnd{
Set ReadData=File.ReadLine()
^DumpTempSeries(Equipment,Sensor,ID,MeasureDT,$I(i))=+$P(ReadData,",",5)_"^"_+$P(ReadData,",",6)_"^"_+$P(ReadData,",",7)
Quit:i=Total
}

 

Thanks

Product version: Caché 2014.1
Discussion (4)1
Log in or sign up to continue

Is it really the reading file taking so much time or using $piece on the line and setting it to global too?

There are various things here that may slow you, even $increment (best to be replaced by i+1)

You can also split the reading file and set it to global by two parts, and use $sortbegin 

Try to run your code with %SYS.MONLBL started, it will help you understand where it spends more time.

Hi Rochdi,

You don't need to use %File and then copy it to a file stream. You can just use %Stream.FileCharacter to directly open and read the file:

  Set stream=##class(%Stream.FileCharacter).%New()
  Set sc=stream.LinkToFile("c:\myfile.txt")
  While 'stream.AtEnd {
  Set line=stream.Read()
  ; Process the chunk here
  }

And I can confirm that I've also seen %Stream.FileCharacter significantly outperform %File for reads.

-Marc