Some strategies on how to manage and decrease space taken by Ensemble system

Many of us installed Ensemble a few years ago, estimated space to be sufficient and now run into surprise - hard drives are getting full! Lets have a look at the steps we can take to investigate what happened and then how to decrease the space taken.

What is taking up space in Ensemble?

Lets look at a list of things that use up space in Ensemble:

1. Ensemble Installation,
2. Backup files
3. Journal files
4. Ensemble databases,
5. Stream files,
6. Custom application files

Lets look at them one by one.

Space taken up by Ensemble Installation

This is relatively small space, depending on version about 1 GB. If you want to reduce it, chances are that you can remove some of the development packages or third party connectivity drivers ODBC/JDBC. You can do it by changing installation options, see Ensemble installation manual:

http://docs.intersystems.com/latest/csp/docbook/DocBook.UI.Page.cls?KEY=GCI_windows#GCI_windows_install

Space taken up by Backup files

So you got space used on your disk taken by backup files? If so, time to think if your system is setup well. Backup should be on different physical disks than the ones used for Ensemble databases, because if Ensemble disk will fail, you want to restore it from different disk that has not failed. Second, creating backup will incur performance hit on running Ensemble - you want to avoid it as well. So all in all, backups should be on different remote disk and therefore their space will not occupy your Ensemble system.

Space taken up by Journal files

As with backups, if your Ensemble system is well setup, journal files should be on separate disk from the database files. You want full performance gained by possibility of writing journals independently from database files and you want to avoid situation when one disk failure would cause you loosing databases and journals. If you have them on the same physical disk, make sure to read well:

http://docs.intersystems.com/latest/csp/docbook/DocBook.UI.Page.cls?KEY=GCDI_journal#GCDI_journal_config_bestpract

Space taken up by Ensemble databases

Each Ensemble database is kept in cache.dat file. If you see that those files are taking up most of your disk space, the first thing is to see the list of databases and see how much of database file is actually used up. Lets follow documentation on checking space using Management Portal:

http://docs.intersystems.com/latest/csp/docbook/DocBook.UI.Page.cls?KEY=GSA_manage#GSA_manage_databases

So how it may happened that we have free space? It is likely, that your Ensemble system was receiving messages that were being stored, and as there was no space in cache.dat, file size was extended. It was growing as long as new messages were coming in and were not deleted. At some point you may have purged messages and space was freed.

What to do if there is free space in database? First thing is to decide if you actually need and should do anything. If your system will periodically receive more than usuall number of messages that afterwards would be purged, possibly you want to keep free space for next time such spike in message number comes in. This space will be reserved and your system will be ready for next time.

If such event was unusual, for example due to stress test, you may want to reduce database file. Refer to the last documentation link on how to reclaim free space.

Ok, so my database is full and growing, what then? Lets first check what is using up space. You can use GSIZE to get exact numbers:

DO ^%GSIZE

This will take some time, depending on size of the database it may be minutes or hours. You will see listing of globals with space they are occupying. What to do depends on global type:

Diagnostic globals such as ERRORS, %ISCLOG, etc

Most likely you enabled some diagnostic logging tool (%ISCLOG). In case of ERRORS, see Application log to see what errors are being logged and filling up the global.

After solving error cause and/or disabling logging you can kill such globals.

Application specific globals

If it is your application creating global or filling up some table, check why. Correct it if needed.

Ensemble messages

If globals correspond to Ensemble messages, you need to check if their management is correct including proper purging. Here is separate post that will help you in that:

https://community.intersystems.com/post/ensemble-orphaned-messages

Space taken up by Stream files

Most likely streams were part of messages that are not being purged. Check message management and purging following previous link. If your Ensemble is older than 2014, plan to upgrade as it may have problems purging streams..

Space taken up by Custom application files

Aside from streams, your application may create any external files on disks. If this happens, review your logic and make sure they are cleaned up when not needed. If they are related to messages consider using Streams instead.