Early Access Program for new Table Partitioning feature
Hi,
We’re launching an Early Access Program for an upcoming Table Partitioning feature that will help IRIS customers manage very large tables, and distribute row data and associated indices across databases and storage tiers. Table Partitioning cuts deep into the core of IRIS relational data management, so we want to make sure we get things right through working with a few engaged customers who can provide feedback on the initial deliverables, and fine-tune as needed.
If you are working with very large relational datasets, looking for more operational efficiency, and willing to roll up your sleeves to test the new capability, please register at https://www.intersystems.com/early-access-program/. You will receive a welcome email that directs them to the evaluation portal, where you'll find a temporary development license, recent kits and container images that contain the new capability, and a tutorial to get you started.
We intend to reach out to registered participants periodically, when we have relevant updates to share, and of course are available to assist your explorations directly, as needed.
Thanks,
benjamin
Comments
will this implementation be employed for the HealthShare ATNA tables as these have billions of rows at customer sites
.png)
Yes, the ATNA tables are definitely on our joint HS/IRIS radar and have been a motivating use case. Right now we're making the basic functionality available on IRIS through this EAP, and are simultaneously working with the HS team to look at the best way to adopt this. Migrating existing data is a non-trivial scenario though.
Is this only for very large tables?
Because we have logging that nobody looks at after a month of three, but we have to keep them for five years.
While we moved our application to AWS, and we have some data, which we need to keep for a while. With this feature, we can move old data to a cheaper storage.
I believe the ability to move to a cheaper storage is mostly the case. Another option is that some table is too big, and someone would like to split it to be stored in multiple different databases, together with the indexes.
If this functionality allows partitioning by business criteria, such as date of sale, or store that sold it, it is possible to obtain greater performance in queries that use the same criteria.
The value should be static, at the time of creating record on table
For dates it uses range feature, so, you can split it by months, year (I suppose)
and move the whole bunch of records including indexes to another databse
yes, when your queries include criteria that correspond to the partition key, we'll ignore partitions we know cannot contain any data satisfying those criteria. e.g. if you partition by year and your query is looking at the current month, we can safely skip any partitions for 2024 or before.
Does it work in a mirror constellation?
yes, though the work to support applying updated mappings (as part of a MOVE PARTITION command) to all mirror members is still ongoing, so that's not something you can validate with the software currently published on the EAP portal.
Looking forward to exploring this! Are there plans to integrate partitioning features with existing IRIS analytics tools?
Any tools that use SQL to access partitioned tables will just work, as from the SQL query perspective there is no change. This includes Adaptive Analytics, InterSystems Reports, and any third-party BI tools. Also, IRIS BI cubes can use partitioned tables as their source class.
We currently have no plans to support partitioning of IRIS BI cubes themselves, as they have their own bucketing structure and less commonly have both hot and cold data, so some of the motivations for table partitioning don't apply.
For those excited about Table Partitioning, my colleague @Ben Schlanger just posted an updated set of kits and some extensions of the tutorial, including support for MOVE PARTITION with nonempty partitions, and how to use the ALTER TABLE t CONVERT .. command to turn a non-partitioned table into a partitioned one. All of this is on track to be included with IRIS 2026.1 in the new year!
We don't want you to get bored during the holidays ;-)