Journal File Indexer

Hi community!

In this article, I'm excited to introduce my latest application, Journal File Indexer. The development of this application is rooted in the portal idea  DPI-I-270.  

In short, this application lets you load and index a log file in a database.  

Why?

If you've ever used the log file search function in the management portal, you may have encountered a timeout error or even a blank page. This problem usually occurs when searching a large journal file. Journal File Indexer solves this problem by considerably increasing the speed of searches once the file has been loaded into the database.  

Another problem arises during the restoration process. When searching for global entries in a log file to restore old or new values, the management portal doesn't have this specific functionality. A routine must therefore be coded to achieve this. Journal File Indexer comes to the rescue, incorporating a restore function!

Architecture

I've used modern development standards, so this application is composed of an Angular front-end, an ObjectScript back-end of course, a REST API, JWT authentication, and WebSocket.

Note: The server-side REST API was generated from a swagger specification using OpenAPI-Suite.

About the storage

Indexed log data is considered "temporary data" and is therefore stored in IRISTEMP by default.  This behavior can be easily modified by a simple global mapping (see next section).

Run the application

There are 2 applications, one for the back-end and a second for the front-end, so the easiest way is to use docker.

Using Docker

git clone https://github.com/lscalese/journal-file-indexer.git
cd journal-file-indexer
docker-compose up -d

# If you need to rebuild images :
# docker-compose build --no-cache

And then open http://localhost:8090/login:

Non-Docker users

To install the backend application using IPM, execute the following command:

zpm "install journal-indexer"

For the front-end, ensure that you have a web server with the "URL rewrite" module and WebSocket functionality enabled. The compiled Angular application is located in the "dist" folder. Configure your web server instance to host "journal-indexer-ui." It's important to note that using the pre-built Angular application implies hosting it on the same web server instance as your web gateway.

Note: for IIS users, a default configuration file is provided with URL rewrite rules and WebSocket enabled (refer to the web.config file).
 

If you need to rebuild the front-end application with different environment parameters, you can access the repository here.  The pre-built version provided uses the following command:

ng build --configuration=webgateway

Feel free to create a new environment and build the application according to your specific requirements.

Features

I haven't included all the detailed features in this article, but a demonstration video will be created soon. The front-end application is designed to be simple and user-friendly.

Basically, Journal Indexer offers a web interface for downloading, indexing, viewing statistics, searching, and restoring data from a log file.

Start by navigating to the "New" or "Load" menu to index a log file and, from there, explore the various other functions at your disposal.

See screenshots:

Journal Indexer also provides a terminal application: 

Do ^JRNINDEXER

While the primary focus of this project is not to emphasize the terminal application, details can be found in the README.md file for those who are interested.

Warning

Journal File Indexer uses the IRISTEMP database to store data.  If you restart the instance, indexed data are lost.  

Indexing a journal file needs space, for example, a journal file size of 1GB can take up to 5 GB in IRISTEMP, so make sure to have enough space on the IRISTEMP disk. 

If you prefer to store data in another database, you can create a global mapping on "^IRIS.Temp.data.*".  In a mirror environment keep in mind, that indexing a journal file will generate a lot of journal records if you select a database in the mirror (IRISTEMP avoids this behavior).

To prevent high disk storage usage, Journal Indexer keeps only 5 indexed journals.  If you index a 6th the oldest is automatically deleted from IRISTEMP.

You can increase or decrease this limit with this class method:

; ex: to keep 7 indexed journals
Do ##class(dc.journalindexer.services.Config).SetMaxJournalRetention(7)

Furthermore, in my opinion, this type of tool should not be installed directly on a production system. Journal File Indexer makes it easy to upload external log files. I recommend using this tool on a test system and then exporting the data if necessary.

About the project development

Embarking on this development project was both a pleasure and a challenge. This was my first experience of creating a front-end application with Angular, and the process of learning and building the project proved rewarding.

Back-end development was equally rewarding, involving REST API generation, incorporating web sockets to notify indexing progress, creating custom class queries, using $SYSTEM.Event to communicate indexing progress in terminal mode, memory manipulation, implementing custom storage solutions, unit testing, and leveraging the Journal API.

Thanks to @Luis Angel Pérez Ramos for his QuinielaML (some similarities in front-end application helped me), @Dmitry Maslennikov for the reviews, and @Vadim Aniskin for challenging me to implement an idea!