Multi-Volume Database Awesome Sauce

Article

Ron Sweeney · Jun 4 6m read

#Databases #System Administration #InterSystems IRIS

ISC Developers, I 👑 you.

Multi-Volume Database

Lifted this explanation from the doc, but just pretend I prompted an LLM for it so there are no copyright repurcussions.

In the default configuration, your database uses a single IRIS.DAT file. You can configure your database to expand automatically into additional volumes (IRIS–0001.VOL, IRIS–0002.VOL, and so on) upon reaching a specified size threshold. You can also deliberately expand your database into additional volumes. Additional volumes may reside in the same directory as IRIS.DAT and/or a set of additional directories.

Im glossing this feature over this and taking it for a spin, but what I want to do is set a small threshold and inspect the multi volume expantion operation on an alternate pvc (directory). Undoudtedly the ramifications on this to Mirroring, Performance (woot) and Management are vast and the below is a simplistic means to understanding, so here it is. Forward solutioning ponders if there is a "callback" we can inject and actually provision a new cloudy volume on the fly too prior to overflow expansion.

Environment
I have a running deployment on 2024.1 (Build 263U) with a 50Gi OpenEBS PVC for my $ISC_DATA_DIRECTORY, pulled it about a month ago.

I added an additional OpenEBS PVC to my namespace:

#kind: PersistentVolumeClaim
#apiVersion: v1
#metadata:
#  name: jiva-iris-volume-claim
#spec:
#  storageClassName: openebs-jiva-csi-default
#  accessModes:
#    - ReadWriteOnce
#  resources:
#    requests:
#      storage: 50Gi
#--
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: jiva-iris-volume-claim-mv
spec:
  storageClassName: openebs-jiva-csi-default
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 50Gi

Applied it

sween@run1:~$ kubectl apply -f deezwatts-volume.yaml -n rivian
persistentvolumeclaim/jiva-iris-volume-claim-mv created

Attached it and patched my deployment through my init container.

<snips>
volumes:
   - name: task-pv-storage
        persistentVolumeClaim:
            claimName: jiva-iris-volume-claim
        - name: task-pv-storage-mvd
          persistentVolumeClaim:
            claimName: jiva-iris-volume-claim-mv
<snips>
volumeMounts:
    - name: task-pv-storage
   mountPath: /data
    - name: task-pv-storage-mvd
   mountPath: /data-mvd

We are now rocking another volume to setup for multi-volume expansion to `/data-mvd`

Setup

So here is the speedy way I stumbled through using the System Management Clown Suit (Warning: Dark Mode not enabled 🕶 )

Lets create the database on the primary volume for the instance, and check out the new properties to the operation.

Creating DB "mvd" on the primary volume, the newness is as follows:

The New Volume Threshold is where we plan things out initially for the expansion trigger, there are other places to post configure it for the target location to create the overflow .DAT's. Of course pay attention to the placeholder hint below the configuration param, you will immediately get a best practices lesson. If you dont, you will be warmed again about the 💯💯 volume limitation.

Enter zero to disable automatic creation of new volumes. If non-zero, when the IRIS.DAT would expand past this threshold, a new volume file called IRIS-0001.VOL will be created. When expansion would grow that file past the threshold, IRIS-0002.VOL will be created, and so on. For non-zero values a minimum of 1 terabyte is recommended to avoid an excessive number of files. Each database is limited to 200 volumes.

Now, head over and mount the new database, you cant post configure it until its mounted.

You'll notice in the listing for the databases, the Volumes Column

This is where we set the alternate location for the database we want to expand multiple volumes.

Expand

With the database threshold being rather small, I sorted messed up the order of things, but it was worth showing.

To generate space exhaustion in the IRIS.DAT, I added a namespace on top the database:

Once I had the namespace out there, I did a little bit of ZPM disco, and installed a few things from the openexchange.

irisowner@iris-deezwatts-deployment-7b9bfcff8f-dssln:~$ irissession IRIS

Node: iris-deezwatts-deployment-7b9bfcff8f-dssln, Instance: IRIS

USER>zn "MVD"
MVD>zn "%SYS" d ##class(Security.SSLConfigs).Create("z") s r=##class(%Net.HttpRequest).%New(),r.Server="pm.community.intersystems.com",r.SSLConfiguration="z" d r.Get("/packages/zpm/latest/installer"),$system.OBJ.LoadStream(r.HttpResponse.Data,"c")

Load started on 06/04/2024 13:43:08
Loading file /data/IRIS/mgr/Temp/z9mu1CvnPnaGbA.xml as xml
Imported class: %ZPM.Installer
Compiling class %ZPM.Installer
Compiling routine %ZPM.Installer.1
Load finished successfully.

%SYS>zpm

=============================================================================
|| Welcome to the Package Manager Shell (ZPM). version 0.7.1               ||
|| Enter q/quit to exit the shell. Enter ?/help to view available commands ||
|| Current registry https://pm.community.intersystems.com                  ||
=============================================================================
zpm:%SYS>install "zpm-registry"

Inspecting the awesome sauce in the database ui:

And the awesome sauce in the folder structure in the pod:

irisowner@iris-deezwatts-deployment-7b9bfcff8f-dssln:/data-mvd$ ls -ltr /data-mv*
total 5140
drwxrwxrwx 2 irisowner irisowner   16384 Jun  4 11:56 lost+found
-rw-rw---- 1 irisowner irisowner      20 Jun  4 12:11 iris.dbdir
-rw-rw---- 1 irisowner irisowner 5242880 Jun  4 12:25 IRIS-0022.VOL
irisowner@iris-deezwatts-deployment-7b9bfcff8f-dssln:/data-mvd$ ls -ltr /data/IRIS/mgr/mvd
total 164
drwxrwxrwx 2 irisowner irisowner    4096 Jun  4 11:15 stream
-rw-rw---- 1 irisowner irisowner      63 Jun  4 12:01 iris.lck
-rw-rw---- 1 irisowner irisowner 5242880 Jun  4 12:11 IRIS-0001.VOL
-rw-rw---- 1 irisowner irisowner 5242880 Jun  4 12:11 IRIS-0002.VOL
-rw-rw---- 1 irisowner irisowner 5242880 Jun  4 12:11 IRIS-0003.VOL
-rw-rw---- 1 irisowner irisowner 5242880 Jun  4 12:11 IRIS-0004.VOL
-rw-rw---- 1 irisowner irisowner 5242880 Jun  4 12:11 IRIS-0005.VOL
-rw-rw---- 1 irisowner irisowner 5242880 Jun  4 12:11 IRIS-0006.VOL
-rw-rw---- 1 irisowner irisowner 5242880 Jun  4 12:11 IRIS-0007.VOL
-rw-rw---- 1 irisowner irisowner 5242880 Jun  4 12:11 IRIS-0008.VOL
-rw-rw---- 1 irisowner irisowner 5242880 Jun  4 12:11 IRIS-0009.VOL
-rw-rw---- 1 irisowner irisowner 5242880 Jun  4 12:11 IRIS-0010.VOL
-rw-rw---- 1 irisowner irisowner 5242880 Jun  4 12:11 IRIS-0012.VOL
-rw-rw---- 1 irisowner irisowner 5242880 Jun  4 12:11 IRIS-0015.VOL
-rw-rw---- 1 irisowner irisowner 5242880 Jun  4 12:11 IRIS-0018.VOL
-rw-rw---- 1 irisowner irisowner 5242880 Jun  4 12:11 IRIS-0016.VOL
-rw-rw---- 1 irisowner irisowner 5242880 Jun  4 12:11 IRIS-0019.VOL
-rw-rw---- 1 irisowner irisowner 5242880 Jun  4 12:11 IRIS-0020.VOL
-rw-rw---- 1 irisowner irisowner 5242880 Jun  4 12:11 IRIS-0017.VOL
-rw-rw---- 1 irisowner irisowner 5242880 Jun  4 12:11 IRIS-0014.VOL
-rw-rw---- 1 irisowner irisowner 5242880 Jun  4 12:11 IRIS-0013.VOL
-rw-rw---- 1 irisowner irisowner 5242880 Jun  4 12:11 IRIS-0011.VOL
-rwxrwxrwx 1 irisowner irisowner 5242880 Jun  4 13:08 IRIS.DAT
-rw-rw---- 1 irisowner irisowner 5242880 Jun  4 13:08 IRIS-0021.VOL

I think I have a new toy to play with!

Conclusion
Definitely a quick post, and gotta get back to work, I think I saw a squirrel via @jtrog and prolly should have made it a discussion instead of an article, but looking forward to future community shares and experiences with this feature as adoption occurs.

See ya'll at Summit everybody if you make the jaunt!

Robert Cemper · Jun 4

A quite interesting feature.
Did you take a look on how backup management is affected / involved by this design ?
I just fail to imagine how to keep all this in synch.
e.g. if spread over several drives:
Are they backed up in total or in volume junks ?

0 0

Ron Sweeney · Jun 4

Reduced the backup list to mvd for a quick test...

*** The time is: 2024-06-04 16:03:43 ***

              InterSystems IRIS Backup Utility
              --------------------------------
Performing a Full backup.
Backing up to device: /media/FullDBList_20240604_001.cbk
Description
Full backup of all databases that are in the backup database list.

WARNING - the following directories could not be backed up
 /data/IRIS/mgr/mvd/ Multi-volume database not supported


Nothing to Backup

I agree its a big feature with a lot of edges

Alexander Pettitt · Jun 4

In two really key ways multi-volumes datasets don't allow you to escape limits you might want to escape.
The max size of a multi-volume dataset is the same as a single file - 34 TB (33553904 MB) for 8K database. Intersystems is working to increase the max size.
An integrity check also treats it as if it was a single file. This is only going to be more of an issue as the max size increses.

2 0

Thanks Alexander, this is good feedback/info, glad I poked the bear.

Enrico Parisi · Jun 4

An integrity check also treats it as if it was a single file. This is only going to be more of an issue as the max size increses.

There is no other option for integrity check because in multi volume database a global can span across multiple volumes/files and a global structure must be fully scanned to check integrity.

If this is a concern, maybe splitting a global across multiple "small" (the concept of small may vary! 😁) database using subscript mapping can be an option.
Integrity check is performed at database/global level, not namespace/global level.

1 0

@Enrico Parisi : I fully support your point!
Global mapping - in any form - is the path for CONTROLLED managed growth.
So I wonder about the intention of the "continuation".