CRLF to LF

Question

Question

For example we have a docker app that receives a .csv file from windows environment and then do some processing on it in docker linux environment.

Windows encode new line in CRLF format, but linux can understand only LF and throw an unexpected errors like

ERROR #7207: Datatype value '100 ' is not a valid number > ERROR #5802: Datatype validation failed on property 'esh.i14y.csv.CelciusCSV.Record:Temperature', with value equal to "100 "

Is there a way to convert new line logic inside iris container in cases when .csv file (and other possible files) comes from windows environment ?

Product version: IRIS 2024.3

Discussion (4)3

Log in or sign up to continue

Julius Kavay · Feb 10

Before you start reading, set the lineterminator property to the desired value

 do myStream.Rewind()
 set myStream.LineTerminator=$c(13,10)  // or $c(10)
 
 // or more general
 set myStream.LineTerminator=$case($zversion(1), 2:$c(13,10), 3:$c(10), :"")
 
 // now start reading
 set line = myStream.ReadLine()
 ...
 ...

0 0

score 0 · Answer 1 · 2025-02-10T02:31:00-05:00

Enrico Parisi · Feb 10

What are you using to read the csv file in IRIS?

Can you provide a small sample code?

0 0

score 0 · Answer 2 · 2025-02-10T03:07:58-05:00

The code piece that reading file

If '$IsObject(pStream) {
            Set tFilename = pStream
            Set pStream = ##class(%IO.FileStream).%New()
            Do pStream.Open(tFilename,,pTimeout,"UTF-8", .tStatus)
            If $$$ISERR(tStatus) Quit
        }

score 0 · Answer 3 · 2025-02-17T09:59:53-05:00

For this specific case I recommend using the %SQL_Util.CSV stored procedure since it handles CRLF on linux correctly. Note - RFC 4180 uses CRLF as the line terminator.

In situations where you need to manually read line-by-line simply trim trailing whitespace including CR

While 'file.AtEnd {
  Set line = file.ReadLine()
  Set line = $ZStrip(line, "<>W", $C(13))
  // ...
}