Error reading messages.log file using Embedded Python in iris-log-viewer
I copied a 5 MB messages.log file to AWS where I have iris-log-viewer app deployed. I ran the test to see how it takes in IRIS code to import the lines into a persistent table:
IRISAPP>set m5mb="/home/irisowner/irisdev/messages.old_20221231.log"
IRISAPP>
IRISAPP>do ##class(otw.log.irislogreader).Test1(m5mb)
Test1 begins at 02/05/2023 12:49:30
ReadLogLines
/home/irisowner/irisdev/messages.old_20221231.log
Open
Test1 ends at 02/05/2023 12:49:34
Test1 execution time: 3.500789
select count(*) from otw_log.Log
63239
It took 3.5 seconds and I had 63239 lines in the table. Then I tried to test using Embedded Python:
IRISAPP>do ##class(otw.log.irislogreader).Test2(m5mb)
Test2 begins at 02/05/2023 12:54:52
1675601693.693327
Set tSC = ##class(otw.log.EmbeddedPython).ReadFileUsingPython(pFile)
^
<THROW>zTest2+23^otw.log.irislogreader.1 *%Exception.PythonException <UNDEFINED> 230 zTest2+23^otw.log.irislogreader.1^1^ Set tSC = ##class(otw.log.EmbeddedPython).ReadFileUsingPython(pFile) <class 'UnicodeDecodeError'>: 'ascii' codec can't decode byte 0xc2 in position 6418: ordinal not in range(128) -
IRISAPP 2d1>
Embedded Python code had added 8644 lines into the table. The last line imported was:
[Utility.Event] Activating new namespace map
Here are the lines around where the import stopped:
Journaling switched to: /irissys/journal1/20221204.002
12/04/22-02:00:57:191 (1188) 0 [Utility.Event] Activating Network
12/04/22-02:00:57:276 (1198) 0 [Utility.Event] Journal File Compression: Compressed /irissys/journal1/20221122.008
12/04/22-02:00:57:345 (1188) 0 [Utility.Event] Activating new namespace map
12/04/22-02:00:57:374 (1188) 0 [Utility.Event] Namespace changes have been activated
12/04/22-02:00:57:924 (1136) 0 [Generic.Event] Starting pidtab expansion (5 chunks)
12/04/22-02:00:57:925 (1136) 0 [Generic.Event] Ending pidtab expansion (5 chunks)
12/04/22-02:00:58:347 (1198) 0 [Utility.Event] Journal File Compression: Compressed /irissys/journal1/20221204.001
Try to open your file like this:
file = open(pFile, "r", encoding="utf-8", errors="ignore")
Docs.
When I click on Docs, it does not work.
Fixed the link, thanks.
Also you can simplify your code:
ClassMethod ReadFileUsingPython(pFile As %String) [ Language = python ] { from datetime import datetime import iris time1 = datetime.timestamp(datetime.now()) print(time1) if pFile=="": raise Exception("filename is required.") file = open(pFile,"r", encoding="utf-8", errors="ignore") log = iris.cls('otw.log.Log') for line in file: status = log.ImportLine(line) time2 = datetime.timestamp(datetime.now()) print(time2) print("Execution time: ",(time2-time1)) }
Thanks, Eduard. I do not need to close the file with file.close()?
Pythonic way is to use with. In that case
close
is automatic as soon as we get outsude of the context:ClassMethod ReadFileUsingPython(pFile As %String) [ Language = python ] { from datetime import datetime import iris time1 = datetime.timestamp(datetime.now()) print(time1) if pFile=="": raise Exception("filename is required.") with open(pFile,"r", encoding="utf-8", errors="ignore") as file: log = iris.cls('otw.log.Log') for line in file: status = log.ImportLine(line) time2 = datetime.timestamp(datetime.now()) print(time2) print("Execution time: ",(time2-time1)) }
Thank you very much, Eduard :-)