#Columnar Storage

Modern data architectures utilize real-time data capture, transformation, movement, and loading solutions to build data lakes, analytical warehouses, and big data repositories. It enables the analysis of data from various sources without impacting the operations that use them. To achieve this, establishing a continuous, scalable, elastic, and robust data flow is essential. The most prevalent method for that is through the CDC (Change Data Capture)  technique. CDC monitors for small data set production, automatically captures this data, and delivers it to one or more recipients, including analytical data repositories. The major benefit is the elimination of the D+1 delay in analysis, as data is detected at the source as soon as it is produced, and later is replicated to the destination.

This article will demonstrate the two most common data sources for CDC scenarios, both as a source and a destination. For the data source (origin), we will explore the CDC in SQL databases and CSV files. For the data destination, we will use a columnar database (a typical high-performance analytical database scenario) and a Kafka topic (a standard approach for streaming data to the cloud and/or to multiple real-time data consumers).

Overview

This article will provide a sample for the following interoperability scenario:

Yes, yes! Welcome! You haven't made a mistake, you are in your beloved InterSystems Developer Community in Spanish.

You may be wondering what the title of this article is about, well it's very simple, today we are gathered here to honor the Inquisitor and praise the great work he performed. 

So, who or what is the Inquisitor?

Perfect, now that I have your attention, it's time to explain what the Inquisitor is. The Inquisitor is a solution developed with InterSystems technology to subject public contracts published daily on the platform  https://contrataciondelestado.es/ to scrutiny.

Although the

Hi Community,

Enjoy the new video on InterSystems Developers YouTube:

⏯ Migrating 20TB to InterSystems IRIS with Near 0 Downtime @ Global Summit 2024

Columnar storage is one of the newer offers provided by InterSystems IRIS. Unlike traditional row-based storage, it optimizes query processing by storing data in columns rather than rows, enabling faster access and retrieval of relevant information.

A couple of articles have been written on when it should be used to give a system the biggest boost, how to create tables like that using SQL

CREATETABLEtable (column1 type1, column2 type2, column3 type3) WITH STORAGETYPE = COLUMNAR  -- ex 1CREATETABLEtable (column1 type1, column2 type2, column3 type3 WITH STORAGETYPE = COLUMNAR)  -- ex 2

and even the performance tests.

As we all know, InterSystems IRIS is a multi-model DBMS and it gives seamless access to the same data using relational and object access. So the former is covered in other articles, but what about the latter?

• The idea of this package is to compare the performance of columnar storage    inside IRIS without wrapping it to some foreign platform that is not my world   
• In addition, I do not want to measure network performance between 2 containers,     but inside a closed IRIS environment that I have fully under my control
• Even the use of SMP or some other browser-based presentation has some influence that I want to avoid.
• Measuring should be as close to the core as possible. So I flagged it NATIVE. Some people might feel it is ABORIGINAL.

How to use it

All tests are running in Namespace USER and

As most of you probably already know, since approximately the end of 2022 InterSystems IRIS included the columnar storage functionality to its database, well, in today's article we are going to put it to the test in comparison to the usual row storage.

Columnar Storage

What is the main characteristic of this type of storage? Well, if we consult the official documentation we will see this fantastic table that explains the main characteristics of both types of storage (by rows or by columns):

As you can see, columnar storage is designed primarily for analytical tasks in which queries are launched

Before we start talking about databases and different data models that exist, first we'd better talk about what a database is and how to use it.

A database is an organized collection of data stored and accessed electronically. It is used to store and retrieve structured, semi-structured, or raw data which is often related to a theme or activity.

At the heart of every database lies at least one model used to describe its data. And depending on the model it is based on, a database may have slightly different characteristics and store different types of data.

To write, retrieve, modify, sort, transform or print the information from the database, a software called Database Management System (DBMS) is used.

The size, capacity, and performance of databases and their respective DBMS have increased by several orders of magnitude. It has been made possible by technological advances in various areas, such as processors, computer memory, computer storage, and computer networks. In general, the development of database technology can be divided into four generations based on the data models or structure: navigational, relational, object and post-relational.

Hi Community,

Watch this video to get an introduction to columnar storage, and see how it can be used in InterSystems IRIS data platform to improve the speed of your analytic queries and efficiently store data:

⏯️ What is Columnar Storage?

With InterSystems IRIS 2022.2, we introduced Columnar Storage as a new option for persisting your IRIS SQL tables that can boost your analytical queries by an order of magnitude. The capability is marked as experimental in 2022.2 and 2022.3, but will "graduate" to a fully supported production capability in the upcoming 2023.1 release. 

The product documentation and this introductory video, already describe the differences between row storage, still the default on IRIS and used throughout our customer base, and columnar table storage and provide high-level guidance on choosing the appropriate storage layout for your use case. In this article, we'll elaborate on this subject and share some recommendations based on industry-practice modelling principles, internal testing, and feedback from Early Access Program participants. 

In our latest episode of Data Points, @Brenna Quirk and I had a conversation with @Benjamin De Boe about the all-new columnar storage feature in InterSystems IRIS. Benjamin tells us a bit about what columnar storage is, why it's important for InterSystems IRIS users running analytical queries, and how you can learn more. Take a listen!

As you may well remember from Global Summit 2022 or the 2022.2 launch webinar, we're releasing an exciting new capability for including in your analytics solutions on InterSystems IRIS. Columnar Storage introduces an alternative way of storing your SQL table data that offers an order-of-magnitude speedup for analytical queries. First released as an experimental feature in 2022.2, the latest 2022.3 Developer Preview includes a bunch of updates we thought were worth a quick post here.

Hello DC Members,

Please welcome the new video on InterSystems Developers YouTube:

⏯ Ten Times Faster Queries with Columnar Storage @ Global Summit 2022