Apache Airflow is the leading open-source platform to programmatically author, schedule, and monitor data pipelines and workflows using Python. Workflows are defined as code (DAGs), making them version-controlled, testable, and reusable. With a rich UI, 100+ built-in operators, dynamic task generation, and native support for cloud providers, Airflow powers ETL/ELT, ML pipelines, and batch jobs at companies like Airbnb, Netflix, and Spotify.

Airflow Application Layout

Modern data architectures utilize real-time data capture, transformation, movement, and loading solutions to build data lakes, analytical warehouses, and big data repositories. It enables the analysis of data from various sources without impacting the operations that use them. To achieve this, establishing a continuous, scalable, elastic, and robust data flow is essential. The most prevalent method for that is through the CDC (Change Data Capture) technique. CDC monitors for small data set production, automatically captures this data, and delivers it to one or more recipients, including analytical data repositories. The major benefit is the elimination of the D+1 delay in analysis, as data is detected at the source as soon as it is produced, and later is replicated to the destination.

This article will demonstrate the two most common data sources for CDC scenarios, both as a source and a destination. For the data source (origin), we will explore the CDC in SQL databases and CSV files. For the data destination, we will use a columnar database (a typical high-performance analytical database scenario) and a Kafka topic (a standard approach for streaming data to the cloud and/or to multiple real-time data consumers).

Overview

This article will provide a sample for the following interoperability scenario:

In the previous article, we talked about ODBC and connecting from C#. And now, let's look at JDBC and Java. The InterSystems JDBC driver is the recommended, high-performance way to integrate your Java applications.

Here is a step-by-step guide to getting your Java application connected to an IRIS instance using the JDBC driver.

Step 1: Obtain and Include the InterSystems IRIS JDBC Driver

Unlike ODBC drivers, which are often installed system-wide, JDBC drivers are typically distributed as JAR files that must be included in your Java project's classpath.

If InterSystems IRIS is installed on your local machine or another you have access to, you can find the file in install-dir/dev/java/lib/ or similar, where install-dir is the installation directory for the instance. Conversely, you can download the jar file from Driver packages page.

Or as suggested by @Dmitry Maslennikov in the comments, use the maven central repository for Maven:

<dependency>
    <groupId>com.intersystems</groupId>
    <artifactId>intersystems-jdbc</artifactId>
    <version>3.10.5</version>
</dependency>

or for Gradle:

implementation("com.intersystems:intersystems-jdbc:3.10.5")

Include the jar file in Project:

• Maven/Gradle: If you use a build tool, the simplest method is to add the InterSystems JDBC driver as a dependency in your pom.xml or build.gradle file. This automatically downloads and manages the JAR.
• Manual: For simple projects, you must place the JAR file in a project directory (e.g., /lib) and explicitly add it to your classpath when compiling and running.

Hi,

When we open a terminal in IRIS, we are entering the ObjectScript shell. Within this shell, we can execute IRIS commands, such as:

As part of improvements regarding CCR usage and usability, certain transitions are now blocked when a CCR Record has undeployed ItemSets for required Environments.

In my previous article, I structured network communications
in these 3 possible layers, and covered the last

• Client <---> Transport
• Server <---> Transport
• Client <---> Server

In fact, you have the most control over the last one.
The IRIS side as a server is yours and under your full control.
Up to now, the Transport layer was assumed to be as passive as a bare wire.

In this article, we will discuss all the debugging tools included in the Microsoft Visual Studio Code IDE.

What will be covered:

• Breakpoints
• Watch window
• Call Stack

Let's start by learning about debugging requirements!

Prerequisites

There are two plugins (extensions) for debugging ObjectScript:

The first is part of the InterSystems ObjectScript Extension Pack. The second is Serenji, a standalone plugin that provides an editor, file manager, and debugging functionality. Both plugins can be installed from the plugin store. To activate key functionality, Serenji requires a license. For this article, we'll use the InterSystems ObjectScript Extension Pack to reduce the learning curve. After you've mastered the basics, you can consider purchasing a paid license for Serenji.

In Part 1, we explored how window functions operate. We learned the logic behind PARTITION BY, ORDER BY, and such functions as ROW_NUMBER() and RANK(). Now, in Part 2, let's delve into more window functions with practical examples.

FastJsonSchema: High-Performance JSON Validation in IRIS

Validating JSON data against JSON Schema is a common requirement for modern applications. FastJsonSchema brings this capability natively to InterSystems IRIS, combining speed, simplicity, and full schema compliance.

Unlike traditional validation approaches, FastJsonSchema generates native ObjectScript code from your JSON Schemas and compiles it directly to iris object code, enabling idiomatic performance without relying on external libraries or runtimes.

For developers building external applications, especially those using familiar technologies like C#, ODBC (Open Database Connectivity) is a crucial, standardized bridge to any relational database, including InterSystems IRIS. While InterSystems offers its own native ADO.NET provider, the ODBC driver is often the most straightforward path for integration with generic database tools and frameworks.

Here is a step-by-step guide to getting your C# application connected to an IRIS instance using the ODBC driver, focusing on DSN-less connection string.

Step 1: Install the InterSystems IRIS ODBC Driver

The InterSystems ODBC driver is installed by default when you install InterSystems IRIS on a Windows machine.

• If IRIS is on the same machine: The driver is already present.
• If IRIS is on a remote server: You must download and install the standalone ODBC client driver package for your client operating system (Windows, Linux, or macOS) and bitness (32-bit or 64-bit) from WRC website if you're a client or by installing Client components and copying ODBC driver.

Once installed, you can verify its presence in the ODBC Data Source Administrator tool on Windows (look for the InterSystems IRIS ODBC35 driver).

There's a pattern I've encountered several times where I need to use a temp file/folder and have it cleaned up at some point later.

The natural thing to do here is to follow the patterns from "Robust Error Handling and Cleanup in ObjectScript" with a try/catch/pseudo-finally or a registered object to manage cleanup in the destructor. %Stream.File* also has a "RemoveOnClose" property that you can set - but use with care, as you could accidentally remove an important file, and this flag gets reset by calls to %Save() so you'll need to set it back to 1 after doing that.

There's one tricky case, though - suppose you need the temp file to survive in an enclosing stack level. e.g.:

I'm pleased to announce the publication of gj :: dataLoader, a new VS Code extension that simplifies the task of loading data from local CSV files into SQL tables on your InterSystems IRIS servers.

Here's an introductory video:

https://www.youtube.com/embed/XohVoW5rSy4
[This is an embedded link, but you cannot view embedded content directly on the site because you have declined the cookies necessary to access it. To view embedded content, you would need to accept all cookies in your Cookies Settings]

In this final part of our introduction to Window Functions, we will explore the remaining functions that have not been covered yet. You will also discover performance tips and a practical guide to help you decide when (and when not) to use window functions effectively.

1. Offset and Positional Value Functions

Overview

These functions reference values are calculated from other rows relative to the current row, or they are extracted from the first, last, or nth values within a window.

Keywords: Vibe coding, Windsurf, IRIS, TIE

Why not?

Has anyone not been trying "vibe coding" so far?

Even merely 3 years ago, if anyone asked