Introduction

Hello! In this article, I will be discussing database degrade, a type of data integrity issue one can face when using IRIS. First, I will be going over a review of the structure of IRIS databases. I'll then discuss how database degrade can manifest and common causes of degrade issues. I'll then conclude with general tips we give our customers about how to prevent or prepare for database degrade issues.

IRIS Database Structure

What exactly is a database? An IRIS database is a single file that consists of a fundamental unit called the "block". It contains a number of equally-sized blocks (typically 8kb) in sequential order. These blocks all "point" to each other by storing the offset in the database at which the block they point to resides. This network of blocks forms a configuration resembling a "forest" of what are called Balanced trees, or B-trees. Each global is represented by a separate B-tree, and each B-tree is accessible through a top-level Global Directory Block which holds pointers to all the global B-trees in the database.

The B-tree's defining feature is that all the "leaf"-level data blocks are on the same level. This means that anytime a top-pointer block fills up, another pointer layer is added above it, theoretically allowing the B-tree to scale indefinitely. At the beginning of each block, there are a number of bytes that indicate some metadata about that block and the blocks around it. For example, each block contains information about its block number, the number of bytes of data in the block, the type of block it is (pointer or data), the collation, etc. The block also contains information about related blocks, such as the block number of the block on its right, the first node of the block to its right, and the byte differences between the last node of the block and the block to the right.

Here is an example representation of what a global B-tree looks like. The green lines represent pointers between different levels of blocks, and the orange lines represent pointers between blocks and their neighboring block to the right. A block to the right value of 0 is used when the block is at the right edge of the tree.

Database Degrade

Database integrity issues occur when a database's internal structure either has some logical inconsistency, or the database does not match the form that IRIS expects. This can happen in a number of ways:

• Referential Integrity: The pointers between pointer blocks and data blocks, or the right-direction pointers between data blocks, are inconsistent. For example, a pointer block may indicate the first node of a data block it points to is ^x(10), while that data block's first node is actually ^x(20)
• Data Integrity: The data of the pointer or data blocks are corrupted. For example, the first few bytes of each block contains some metadata about the block itself and its related blocks, starting at offset 0. If these are located at some other offset, this would result in the block being corrupted.
• Application Integrity: This occurs when the database looks as IRIS expects, but there is some logical inconsistency related to how the application should look. For example, say we expect all doctor data to be saved in the ^Hospital.DoctorsD global, and indices on these doctors to be saved in ^Hospiral.DoctorsI. If a doctor had their data in the data global, but didn't have a corresponding index in the indices global, this would be a form of an application integrity issue.

Common causes of Degrade

Here, I'll be going over some of the common causes of database degrade we see from our customers.

Windows Write Caching

Windows write caching is a feature in Windows which can lead to database degrade issues. The idea behind this setting is that Windows will buffer disk writes in memory, and write them out to disk in larger packages. This helps improve performance in certain cases, as disk writes are slow, and doing them in larger packages allows for more efficient writing. For example, imagine your disk is broken up into 1024 byte blocks, and you have an application trying to write 256 4-byte chunks of data. Instead of doing 256 separate disk writes, which would be quite slow, write caching saves time by storing these writes in memory and writing them far less often.

However, this can create issues when power loss is involved. If the OS is storing data in memory that would normally have already been written out to disk, a loss in power can cause this data to be lost and unrecoverable, since RAM memory is volatile. For this reason, we often recommend our customers turn off Windows write caching when it is not necessary.

Hardware Write Caching

Just as OSes will use RAM to cache writes to disk, some enterprise-level disk drives will also cache disk writes for efficiency purposes. Certain SSDs for example contain a DRAM buffer which can store writes in order to write them in large packages for performance increases. Unfortunately, just as with OS-level write caching, DRAM and hardware memory in general is volatile, and will not be preserved in the case of power loss. This can be mitigated by ensuring the disk drive is powered by batteries or some other form of Uninterruptable Power Supply (UPS) so that it can flush the memory buffer in the event of a power failure.

VMware

VMware is a cloud hosting solution many of our customers use to run our products. There have been a number of known issues with VMware that can lead to data integrity issues. These are detailed in the following advisories:

• Advisory: VMWare vSAN and Data Integrity (link) : There is a known issue concerning disk extend operations with VMs running VMware vSAN, a storage solution offered by VMware
• Advisory: VMWare vSphere and Data Integrity (link) : SEsparse, a snapshot format for VMware machines, has a known data integrity issue
• Advisory: VMware vSAN Data Consistency Errors (link) : There are certain x86 instructions in the AVX-512 instruction set that can lead to data integrity issues when used with vSAN

General Advice

Here are some general recommendations we provide to our customers when helping them recover from database degrade issues.

Backups

When dealing with database degrade issues, our first recourse is surprisingly not to try and repair. Repairing databases by hand can be a cumbersome and not often fruitful process, due to the large scale and complexity of typical degrade issues. Instead, we will initially try to recover from a backup to restore the database to a healthy state. For this reason, we ask our customers to take frequent backups.

Backups are also highly useful for performing Root Cause Analysis (RCA) on why the degrade occurred in the first place. With a backup, we can see what the healthy state looked like for comparison with the unhealthy state. We can also see how the degrade changed or spread over time to track down its source.

Guide to doing backups: link

Integrity checks

It is vital for RCA that we catch database degrade as soon as possible. This is important because often degrade happens due to some aspect of the system state, whether that's on the application, OS, or hardware level.

For this reason, we ask our customers to run frequent checks of the ^Integrity routine. This routine checks databases and globals to verify that they have the format expected by IRIS. If you find there is degrade present in your databases, you can immediately contact the WRC and we can assist you in repairing and then conducting RCA to find the root cause.

Guide to running integrity checks: link

It is also important to run integrity checks outside of IRIS on the operation system and hardware that the IRIS instance is running on, as IRIS works inextricably with the OS and hardware in order to manage data correctly. This can be done through OS utilities such as fsck (UNIX) and chkdsk (Windows).

Conclusion

In the world of modern storage devices and advanced operating systems, data integrity issues are not as frequent as they once were. However they do still occur, and can lead to serious issues with application availability and data loss when they do. For this purpose, we ask our customers to frequently check the integrity of their data and systems, and contact us whenever they see an issue, so we can help restore their systems to working order as soon as possible. Thank you!