extended is not the same as unlimited :-)

We're moving from 32-bit block numbers to 40-bit block numbers, which means the max size for a database with 8k block size is 8PB. Obviously, those are huge amounts of data, and we strongly recommend against just stashing that all in a single IRIS.DAT file,, if your file system permits such files at all. Multivolume database support already helps you spread data across multiple physical files (opaquely), and the new Table Partitioning support (experimental in 2026.1) helps you spread table data across multiple databases. 

If you wonder why we stuck with 40 bit numbers rather than a power of 2: we were able to reuse a few available bits in the current block header format, meaning you can transparently and instantly upgrade to this Extended Database Format without having to migrate your almost-32TB database to a new block header format. And 8PB is quite large already...

We found a small issue affecting some Vector Search queries that may lead to a runtime error when your vector search query uses aliases to shorten or disambiguate table names. The problem can be worked around by rewriting your query to avoid using aliases. A fix for this issue will be included in the next preview kit, due in two weeks.

We also identified a separate issue with unintended changes to DSTIME behaviour, and recommend DSTIME users to skip this preview. The next preview kit will restore the original DSTIME behaviour.

I got completely distracted by this article's use of "Dynamic SQL". Dynamic SQL is a fairly specific concept on our platform, and historically there's been a small difference between Dynamic and Embedded SQL in terms of overhead, but that has all but evaporated over the past few years by consecutive optimizations. Now suddenly this article seems to suggest there's a huge gap, so I got hooked and dug into the code, only to find that this method is making a rather crazy detour through %DynamicObject for no real reason, on the way from the SQL result set to the dataframe. In other words, all you're measuring between the "Dynamic SQL" and "IRIS SQL" options is exactly that %DynamicObject overhead.

It's only a name, true, but people only doing a cursory read of the article may take away the completely wrong thing.

Benjamin De Boe · Dec 17, 2025 go to post

For those excited about Table Partitioning, my colleague @Ben Schlanger just posted an updated set of kits and some extensions of the tutorial, including support for MOVE PARTITION with nonempty partitions, and how to use the ALTER TABLE t CONVERT .. command to turn a non-partitioned table into a partitioned one. All of this is on track to be included with IRIS 2026.1 in the new year!

We don't want you to get bored during the holidays ;-)

Benjamin De Boe · Nov 27, 2025 go to post

Still a test case would be helpful.

If there's no source table rows for which userActionten=1, that value will not show up as a dimension member in Analyzer (so no row in the pivot with COUNT equal to zero). Maybe that's what you're seeing? 

Benjamin De Boe · Nov 27, 2025 go to post

Feel free to share your thoughts here, so we can make sure the AI bot's contributions are appropriately labelled. And indeed we don't want that bot to scoop up all of our precious Global Masters swag! :-)

Benjamin De Boe · Oct 29, 2025 go to post

that first line is required to initialize the mbChunk variable used in the loop, else you'd get an UNDEFINED at runtime.

I'm thinking the way how you're using this macro is slightly off, and this has the same root cause as the other thread you opened. This macro is expected to be used on its own:

  // some logic here
  $$$AndBits(^glo1,^glo2)
  // more logic here, and note there's at least a space before all these lines

hope this helps

Benjamin De Boe · Oct 29, 2025 go to post

Hi @José Pereira , you can use the maxerrors flags in a USING clause to bail out earlier, though I'm guessing you're actually already using it to achieve the opposite and make sure you process all the correct rows. 

We're intentionally writing verbose logs as that was an explicit request early on. As for retention, that was overlooked in the initial release, but is a roadmap item to make this follow the existing PurgeErrors CPF setting (and I'll add a note to the ticket you brought it up). In the absence of that, you can just use DELETE or TRUNCATE on these log tables.

Benjamin De Boe · Oct 29, 2025 go to post

I think @Alexander Koblov actually meant "Yes" :-), as row-level security was indeed designed for exactly this purpose, to inject additional filters into queries based on their credentials.

Benjamin De Boe · Oct 29, 2025 go to post

right, I was misled by the error message, which is the same for what you'd get if you omit that initial whitespace.

Given that is your macro use, the error is because the expanded statement (which you can see from the generated .INT code) is invalid:

set a =if %arr>0 QUIT 5

A macro gets expanded before code gets compiled, and should not be mistaken for a function call.

If you want to use that macro on the RHS of a set command, you'll need to rewrite it into something like $select(%arr>0, 5, 123).

Benjamin De Boe · Oct 29, 2025 go to post

Here's an example that combines two bitmaps:

#define AndBits(%dst,%src) ##continue
    set mbChunk=""                                      ##continue
    for {                                               ##continue
        set mbChunk=$o(%dst(mbChunk),1,mbBits)          ##continue
        quit:mbChunk=""                                 ##continue
        set mbBits=$bitlogic(mbBits&%src(mbChunk))      ##continue
        If $bitfind(mbBits,1) {                         ##continue
            Set %dst(mbChunk)=$bitlogic(mbBits)         ##continue
        } Else {                                        ##continue
            Kill %dst(mbChunk)                          ##continue
        }                                               ##continue
    }

Using ##continue helps you avoid cramming everything onto a single line

Benjamin De Boe · Oct 29, 2025 go to post

I think you may lack a space before your macro use. Any line of actual code other than a line labels needs to start with a space or tab.

Benjamin De Boe · Sep 23, 2025 go to post

I agree it's become more of a style preference thing, so absolutely something to have strong opinions and religious debates about :-)

The only thing to avoid is using Dynamic SQL to conveniently build your query and then feed in query parameters through string concatenation rather than as true ?-style parameters. That's a security risk you wouldn't run with Embedded SQL. That's all. Back to the debate! :-)

Benjamin De Boe · Aug 6, 2025 go to post

That REST API is indeed for querying iFind indices (hence the direct reference to an index you can provide) and the somewhat confusingly named "query" argument is actually to pass in the iFind search string. The API will then build a full SQL query for you and run it right away. 

Here's the OpenAPI spec for this endpoint (from self-documentation endpoint /api/iKnow/v1/USER/swagger):

  /table/{table}/search:
    post:
      operationId: /table/{table}/search-POST
      summary: |
        Search the given iFind index in the given table
      tags: ["iFind"]
      parameters:
        - $ref: '#/parameters/tableParam'
        - name: RequestBody
          description: JSON object with a list of query-specific arguments
          in: body
          schema:
            type: object
            properties:
              query:
                description: This is the only necessary parameter with no default value. The search terms to query against the iFind index.
                type: string
              index:
                description: the iFind index would be searched against, if you don't specify it, the first found iFind index would be used .
                type: string
              option:
                $ref: '#/definitions/OptionSpec'
              distance:
                description: only valid when option is fuzzy search (when option is 3)
                type: string
                example: "3"
              language:
                description: iKnow-supported language model to apply, for example "en"
                type: string
              includeText:
                description: whether the returned columns should include the column beging indexed by 'index'
                type: integer
                default: 0
                enum: [0, 1] 
              columns:
                description: specify the columns which also needed to be returned. For example, ["column1","column2"] 
                type: array
                example: []
                items:
                  type: string
              highlightSpec:
                $ref: '#/definitions/HighlightSpec'
                description: the parameters needed for Highlight
              rankSpec:
                $ref: '#/definitions/RankSpec'
                description: the parameters needed for Rank
              where:
                description: the valid SQL logical condition statement. For example, "column1 = ? AND column2 = ?"
                type: string
      responses:
        200:
          description: Successful response
          schema:
            type: object
            properties:
              rows:
                type: array
                default: []
                items:
                  type: object
Benjamin De Boe · Jun 23, 2025 go to post

thanks for your feedback Enrico! I also very much support the idea you filed, and it shouldn't take much to take this through into a release.

Benjamin De Boe · Apr 25, 2025 go to post

It is part of my job description to say you shouldn't touch internal globals ;-)

see you in Orlando!

Benjamin De Boe · Apr 24, 2025 go to post

FWIW, I would not recommend users touch this global. It is internal and InterSystems may (and does) change how it's used without notice.

Benjamin De Boe · Apr 23, 2025 go to post

Love the article! Very well-phrased considerations on the use of AI, almost all of which I share. Especially in the context of #1, we should not forget that the second L in LLM is for Language, and not for Fact or Solution (otherwise it would be a really bad acronym!). Therefore, if we're not qualified to spot what hallucinations crept into the response, its nicely-phrased language will probably make sure we never will.

PS: so glad you passed that Stats test and joined InterSystems :-) 

Benjamin De Boe · Apr 16, 2025 go to post

I believe what you're looking at is the new, more fine-grained set of %Native_* resources you need to use native functions. Look for DP-423341 in the upgrade guide. It seems we failed to describe this requirement in the  Native API documentation (or at least I didn't find it where I expected it), so we'll get that addressed.

I'd also recommend defaulting to the new, dynamic upgrade guide that makes it easier to filter on particular types of issues. This is now replacing the old, static pages that were more reliant on / vulnerable to manual curation. In fact you'll no longer find those static pages from the menu in the 2025.1 doc.

Benjamin De Boe · Mar 18, 2025 go to post

Any tools that use SQL to access partitioned tables will just work, as from the SQL query perspective there is no change. This includes Adaptive Analytics, InterSystems Reports, and any third-party BI tools. Also, IRIS BI cubes can use partitioned tables as their source class.

We currently have no plans to support partitioning of IRIS BI cubes themselves, as they have their own bucketing structure and less commonly have both hot and cold data, so some of the motivations for table partitioning don't apply. 

Benjamin De Boe · Mar 17, 2025 go to post

yes, though the work to support applying updated mappings (as part of a MOVE PARTITION command) to all mirror members is still ongoing, so that's not something you can validate with the software currently published on the EAP portal.

Benjamin De Boe · Mar 17, 2025 go to post

yes, when your queries include criteria that correspond to the partition key, we'll ignore partitions we know cannot contain any data satisfying those criteria. e.g. if you partition by year and your query is looking at the current month, we can safely skip any partitions for 2024 or before.

Benjamin De Boe · Mar 12, 2025 go to post

Yes, the ATNA tables are definitely on our joint HS/IRIS radar and have been a motivating use case. Right now we're making the basic functionality available on IRIS through this EAP, and are simultaneously working with the HS team to look at the best way to adopt this. Migrating existing data is a non-trivial scenario though.

Benjamin De Boe · Feb 20, 2025 go to post

Thanks for the excellent writeup @sween , and your neatly themed series is bound to inspire many to dive into the classic literature and/or tourism pond!!🤩

Benjamin De Boe · Jan 10, 2025 go to post

Nice article @Ben Schlanger !

I like how you're laying out the investigative process, though I think it's worth noting that every case is different and therefore recommendations also can differ. Especially the %NORUNTIME hint should be used with caution as it may deprive you of better plans in most scenarios. In fact, we like to say that any time you have to revert to that hint, it's worth opening a case with the WRC as it should be an improvement opportunity for our engine to make that better choice automatically (available statistics permitting) :-)

Also, I'd like to advertise a few improvements we've made since the IRIS version shown here:

  • Improved feedback in the query plan: we've been displaying a note in the query plan for a while now if there's a chance that different runtime parameter values may lead to a different plan, and as of IRIS 2023.3 are even calling out the specific predicates that drove the RTPC decision. For example, your plan may say "This query plan was selected based on the runtime parameter values that led to improved selectivity estimation of the range condition enc.EncounterTime BETWEEN '2022-01-01' AND '2023-12-31'"
  • Showing the actual runtime plan: Starting with IRIS 2023.3, we've enhanced the EXPLAIN and SMP utilities to no longer show the generic plan, after substituting out all parameter values, but the actual plan you'll get at runtime with the literal values you put in the query text. This addresses step #4 in the investigation described above.
  • SQL Process View: As of IRIS 2022.2, the Operations menu in the System Management Portal includes a "SQL Activity" link that leads you to a page listing all currently-running SQL statements, and allows you to drill through to the statement details and query plan. This also helps with step #4, and to identify any long-running queries in the first place. An aggregated form of this data is also available through the /api/metrics endpoint for consumption through a monitoring tool.
  • Query and schema recommendations: In IRIS 2024.3, released last month, we've further expanded the information contained in the query plan from the RTPC related notes described above, to also include warnings on indices marked as non-selectable (cf investigation step #1), indices that are being ignored because they have non-matching collation, whether the plan is frozen, and similar additional information that may help you improve the statement text, schema, or overall system settings

The above features are all specifically introduced to help diagnose long-running queries more quickly and identify how things can be sped up, but of course these versions also include general performance enhancements and refinements to the RTPC infrastructure too, so it'll be exciting to see how fast this customer's query runs on the latest and greatest IRIS release!

Benjamin De Boe · Dec 19, 2024 go to post

Hi @Scott Roth , the %MANAGE_FOREIGN_SERVER privilege was only just introduced with 2024.2, as part of finalizing the full production support for Foreign Servers (see also release notes). I'm not sure though why it wouldn't appear after you created it. Can you confirm whether it's still there right after the CREATE SERVER command, whether you're using the same user for both connections, and whether or not you can CREATE FOREIGN TABLEs with that server (before logging off and / or after logging back in).

I understand upgrading may not be straightforward, but the most logical explanation would be that the initial, crude privilege checking (that we replaced in 2024.2 as advertised) has a hole in it. 

thanks,
benjamin