New post
Question
· Feb 5, 2023

Error reading messages.log file using Embedded Python in iris-log-viewer

#Embedded Python #InterSystems IRIS

I copied a 5 MB messages.log file to AWS where I have iris-log-viewer app deployed. I ran the test to see how it takes in IRIS code to import the lines into a persistent table:

IRISAPP>set m5mb="/home/irisowner/irisdev/messages.old_20221231.log"

IRISAPP>

IRISAPP>do ##class(otw.log.irislogreader).Test1(m5mb)
Test1 begins at 02/05/2023 12:49:30
ReadLogLines
/home/irisowner/irisdev/messages.old_20221231.log
Open
Test1 ends at 02/05/2023 12:49:34
Test1 execution time: 3.500789

select count(*) from otw_log.Log

63239

 

It took 3.5 seconds and I had 63239 lines in the table. Then I tried to test using Embedded Python:

IRISAPP>do ##class(otw.log.irislogreader).Test2(m5mb)
Test2 begins at 02/05/2023 12:54:52
1675601693.693327

 Set tSC = ##class(otw.log.EmbeddedPython).ReadFileUsingPython(pFile)
 ^
<THROW>zTest2+23^otw.log.irislogreader.1 *%Exception.PythonException <UNDEFINED> 230 zTest2+23^otw.log.irislogreader.1^1^       Set tSC = ##class(otw.log.EmbeddedPython).ReadFileUsingPython(pFile) <class 'UnicodeDecodeError'>: 'ascii' codec can't decode byte 0xc2 in position 6418: ordinal not in range(128) - 
IRISAPP 2d1>

 

Embedded Python code had added 8644 lines into the table. The last line imported was:

[Utility.Event] Activating new namespace map

Here are the lines around where the import stopped:

Journaling switched to: /irissys/journal1/20221204.002
12/04/22-02:00:57:191 (1188) 0 [Utility.Event] Activating Network
12/04/22-02:00:57:276 (1198) 0 [Utility.Event] Journal File Compression: Compressed /irissys/journal1/20221122.008
12/04/22-02:00:57:345 (1188) 0 [Utility.Event] Activating new namespace map
12/04/22-02:00:57:374 (1188) 0 [Utility.Event] Namespace changes have been activated
12/04/22-02:00:57:924 (1136) 0 [Generic.Event] Starting pidtab expansion (5 chunks)
12/04/22-02:00:57:925 (1136) 0 [Generic.Event] Ending pidtab expansion (5 chunks)
12/04/22-02:00:58:347 (1198) 0 [Utility.Event] Journal File Compression: Compressed /irissys/journal1/20221204.001
 

Product version: IRIS 2022.3
$ZV: IRIS for UNIX (Ubuntu Server LTS for x86-64 Containers) 2022.3 (Build 599U) Wed Jan 18 2023 23:45:44 EST
Discussion (7)1
Log in or sign up to continue
Eduard Lebedyuk · Feb 5, 2023

Also you can simplify your code:

ClassMethod ReadFileUsingPython(pFile As %String) [ Language = python ]
{
  from datetime import datetime
  import iris
  time1 = datetime.timestamp(datetime.now())
  print(time1)
  if pFile=="":
    raise Exception("filename is required.")

  file = open(pFile,"r", encoding="utf-8", errors="ignore")
  log = iris.cls('otw.log.Log')
  for line in file:
    status = log.ImportLine(line)

  time2 = datetime.timestamp(datetime.now())
  print(time2)
  print("Execution time: ",(time2-time1))
}
0 0
Eduard Lebedyuk · Feb 6, 2023

Pythonic way is to use with. In that case close is automatic as soon as we get outsude of the context:

ClassMethod ReadFileUsingPython(pFile As %String) [ Language = python ]
{
  from datetime import datetime
  import iris
  time1 = datetime.timestamp(datetime.now())
  print(time1)
  if pFile=="":
    raise Exception("filename is required.")

  with open(pFile,"r", encoding="utf-8", errors="ignore") as file:
    log = iris.cls('otw.log.Log')
    for line in file:
      status = log.ImportLine(line)

  time2 = datetime.timestamp(datetime.now())
  print(time2)
  print("Execution time: ",(time2-time1))
}
1 0