Search

Hey Everyone! Not a secret that InterSystems Global Masters Advocate Hub is connected to Developer Community via API. When you make a contribution to DC, it brings you points and badges on Global Masters. We have prepared a short guide on how to best earn points on Global Masters: HOW TO EARN POINTS ON GLOBAL MASTERS Each published post on Developer Community Published post on DC ES / PT / JP / CN / FR 200400 1st Comment on DC / Each comment* Comment on DC ES / PT / JP / CN / FR 300 / 30 60 1st answer marked as Accepted / Each accepted answer 1 000 / 150 Translate an article / a question 150 / 30 Publish 1 / 5 / 10 / 25 / 50 articles on DC 1 500 / 7 500 / 15 000 / 40 000 / 75 000 Publish 1 / 5 / 10 / 25 / 50 questions on DC 500 / 2 000 / 5 000 / 15 000 / 30 000 Each application on Open ExchangeBonus points for each ZPM applicationPublish 1 / 5 / 10 / 25 applications on Open Exchange 8004001 000 / 10 000 / 25 000 / 75 000 1 / 5 / 10 / 25 / 50 Accepted Answers on DC 1 000 / 4 000 / 8 000 / 20 000 / 40 000 Your DC post gathered 750+ / 2000+ / 5000+ / 15000+ views 600 / 2 500 / 7 000 / 20 000 Read an article on DCWatch the videoShare an article / video in Social Networks 102040 Write 1 / 2 / 3 / 4 / 5 Articles with Best Practices tag 1000 / 3000 / 7000 / 10 000 / 15 000 50 / 100 / 250 / 500 / 1000 your application's downloads on Open Exchange 2 500 / 5 000 / 7 500 / 12 500 / 25 000 Make a review for InterSystems / InterSystems products 2 000 - 3 000 Invite your colleague to Developer Community 600 Create a video about your OEX application 3000 *counted only comments that were published after registration on the Global Masters Advocate Hub. Complete challenges, get badges and climb up the levels: Insider > Advocate > Specialist > Expert >Ambassador> VIP. The higher level you are, the more interesting prizes available! And... Please check the additional information about Global Masters: What is Global Masters? Start Here Global Masters Badges Descriptions Global Masters Levels Descriptions If you have not joined InterSystems Global Masters Advocacy Hub yet, let's get started right now! Feel free to ask your questions in the comments to this post. Thanks, Anastasia!Very helpful! I believe we also have a series upon the number of accepted answers, like 10,25,50,100 accepted answers. Do we? Thank you for this quick reference table (and for my *looks up amount of points for comments* 30 points!) Hi Evgeny,let me answer - we do not have so far, and I think that would be good to have such series & badges to recognize the authors. Are these automated in any way? Wondering if mine is bugged because I've certainly posted questions and comments before but those badges were never unlocked. Their descriptions below say "first" question/comment and I don't know if mine are being detected:https://community.intersystems.com/post/changes-global-masters-program-new-level-new-badges-new-possibilities Hi David! This should be automatic. We'll investigate. I wrote a post on DC in 2017? Do I have to 'register' it to get points on Global Masters?Kind regards, Stephen Hi David!We have fixed this issue. Thank you for the feedback! Thank you! You're very quick! Hi Stephen, I see you have joined recently the Global Masters, that is so great! -this post is not counted in "Write a post on Developer Community" challenge (100 points), as it has been published before you registered- it is counted in all other type of challenges listed above e.g. "Write 10 posts on DC". This was really helpful Thank you This is an excellent article and is worth bumping the thread :) Great! This is very helpful! This is helpful. Thank you!

Hi Community! We're pleased to announce that that InterSystems IRIS Community Edition is available on the Docker Store! InterSystems IRIS Community Edition is the no-cost developer edition designed to lower the barriers to entry to get started with IRIS. Now that it is listed on the Docker Store, running an IRIS Community instance is as easy as - docker run -d -p 52773:52773 store/intersystems/iris:2019.1.0.511.0-community For more on running IRIS in containers check out our Documentation or one of the many Community posts on the topic! Confirming that! Really great news, Joe! What are the limitations of Community edition? One limitation I can see straight away is that this version only includes outdated files for Node.js (ie the iris*.node files). All I see is iris610.node and iris700.node. Is there a reason why the Node.js version 8 and version 10 interface files haven't been included? What a wonderful news There's a section about Community Edition limitations in the document at https://irisdocs.intersystems.com/irislatest/csp/docbook/DocBook.UI.Page.cls?KEY=ACLOUD#ACLOUD_limitsHere's a screenshot of what it currently says:A big disappointment for me is the exclusion of namespace mapping. Those of us who create tools for the InterSystems world commonly use namespace mapping to make our tools available in the users' namespaces. See Evgeny's article here I downloaded IRIS Community edition (store/intersystems/iris:2019.1.0.510.0-community) and was able to create package and global mappings without any issues. I was able to create the mappings, but they don't seem to work for me. Not even after an IRIS restart.I'm using the %ALL pseudo-namespace. I created these mappings (no restarts):Ens* global mapping for ENSLIB -> USERDeepSee* global mapping for ENSLIB -> %ALLEns* package mapping for ENSLIB -> USERHere's the result: I notice that your mappings get code from ENSLIBMy use-case is a bit different.Can you create a new database and namespace pair called XYZ and then define a %ALL mapping so that the globals ^XYZ and the routines XYZ* and the package XYZ get fetched from your XYZ namespace from all other namespaces, e.g. USER ?I'd expect mappings to ENSLIB to work, otherwise the "Ensemble" features of IRIS would be broken in Community Edition.It seems to me that those are allowed but mappings we define to our own databases get ignored. Yeah, apparently only DeepSee and Ens packages/globals can be mapped successfully.Tried random package and it didn't show up even after mapping to %ALL. Thanks, Rob!Introduced your issue here. Please share more requests and feedback there Thanks, John!Your feedback is filed here. Also here. I beat you to it! This is a restriction in the Community Edition and it is what is meant byAll InterSystems IRIS functionality is included except the following:Namespace mapping...You can create mappings and activate configurations which contain mappings if that is useful for some purpose (eg. building a configuration for someone else) however all mappings which do not involve a system database are ignored.

Hi Developers, Get a preview of the upcoming Embedded Python feature of InterSystems IRIS: ⏯ Embedded Python in InterSystems IRIS: Sneak Peek Subscribe to InterSystems Developers YouTube and stay tuned! The "Trying embedded Python was great", Can't wait to try it out Nigel

The InterSystems IRIS has a very nice container class to allows you have your Dashboards as classes source code. It is %DeepSee.UserLibrary.Container. With this class is possible group all your dashboard and pivot table definitions. This is useful to automatically create your dashboards when you build your docker project and other automation scenarios. See: Class dc.WebsiteAnalyzer.WebsiteAnalyzerBIContainer Extends %DeepSee.UserLibrary.Container { XData Contents [ XMLNamespace = "http://www.intersystems.com/deepsee/library" ] { <items> <dashboard xmlns="http://www.intersystems.com/deepsee/library" name="WebsiteAnalyzerDashboard" folderName="dc/WebsiteAnalyzer" title="Website NLP Analysis" description="" keywords="" owner="" shared="true" public="true" locked="false" resource="" timeCreated="2020-12-24T08:29:58.107Z" createdBy="_SYSTEM" category="WebsiteAnalyzer" bookCover="" scheme="" worklistCount="2" snapTo="true" snapGrid="true" gridRows="10" gridCols="10" canResize="true" canModify="true" showTitleBar="true" titleBarOpacity="" titleBarColor="" selectedTitleBarOpacity="" selectedTitleBarColor="" titleBarTextColor="" selectedTitleBarTextColor="" titleBarFont="" companyName="" companyLogo="" companyStyle="" backgroundColor="white" backgroundImage="none" backgroundRepeat="no-repeat" backgroundSize="100% 100%" backgroundOpacity="1" widgetBorders="1px solid #F0F0F0" widgetBordersSwitch="edit" widgetBordersColor="#F0F0F0" widgetBordersStyle="solid" widgetBordersWidth="1" widgetBordersToggle="true"> <widget name="WebsiteAnalyzerTableWidget" type="pivot" subtype="pivot" subtypeClass="lineChart" title="" dataSource="TopEntities.kpi" dataLink="" drillDownDataSource="" width="200" height="200" sidebarContent="" showSidebar="false" sidebarWidth="" maximized="false" homeRowL="0" homeColL="0" colSpanL="5" rowSpanL="5" showToolbar="true" showToolbarBottomBorder="true" showToolbarOnlyWhenMaximized="false" colorToolbar="#F0F0F0" opacityToolbar="1" backgroundColor="#F0F0F0" opacity="1" theme="" dataColorList=""> </widget> <widget name="WebsiteAnalyzerChartWidget" type="pivot" subtype="barChart" subtypeClass="barChart" title="" dataSource="TopEntities.kpi" dataLink="" drillDownDataSource="" width="200" height="200" sidebarContent="" showSidebar="false" sidebarWidth="" maximized="false" homeRowL="0" homeColL="5" colSpanL="5" rowSpanL="5" showToolbar="true" showToolbarBottomBorder="true" showToolbarOnlyWhenMaximized="false" colorToolbar="#F0F0F0" opacityToolbar="1" backgroundColor="#F0F0F0" opacity="1" theme="" dataColorList=""> </widget> </dashboard> </items> } } So, you create your dashboards and pivot tables, connect to your instance using VSCode, go to other and get your dashboard and pivot definitions. Finally paste these definitions in your container class. Easy! See: This sample can be found in this the Open Exchange link. Nice dashboards, @Yuri.Gomes! Try with DSW too? In fact this class is maybe good for deployment, but not that perfect for source-control. You have ALL your DFI in the repo in one file. So when you have changes or will want to make changes you need to deal with this container class. You can export DFI as files (using isc.dev) and import it then with %SYSTEM.OBJ.ImportDir or ZPM. How to export DFI using isc.dev. First you setup the folder you export it into: do ##class(dev.code).workdir("/irisdev/app/src") Then when you need to export changes call in the terminal: d ##class(dev.code).export("*.DFI") Repo with DFI exported The line which imports it into container OR ZPM My next step! I agree, if you have many dashboards isc dev is good. I liked the container class, because I like to have all the work into classes, including the dashboards. And you have the option to have one class by a dashboard, to have more control.

We released a new version of ZPM (Package Manager) New in ZPM 0.2.10 release: 1) FileCopy Added some improvements to already existing tag FileCopy Example of module.xml <?xml version="1.0" encoding="UTF-8"?> <Export generator="Cache" version="25"> <Document name="test-binary.ZPM"> <Module> <Name>test-binary</Name> <Version>0.0.1</Version> <Packaging>module</Packaging> <SourcesRoot>src</SourcesRoot> <FileCopy Name="lib" Target="${libdir}my-lib"/> <!-- Copies content of lib folder to target --> <FileCopy Name="somefile.jar" Target="${libdir}my-lib"/> <!-- Copies just desired file to target --> </Module> </Document> </Export> Added aliases Target and Dest to attribute InstallDirectory Supports files as well as directories, no wildcards, just folder content of which should be copied, or particular file ${libdir}, ${bindir} added, installdir/lib and installdir/bin respectively. Using bindir in docker environments may cause to security issues during install. Using libdir works in this case, so, recommended way. This should work (ending slash in Target is important) <FileCopy Name="dsw/irisapp.json" Target="${cspdir}dsw/configs/"/> And this should be the same <FileCopy Name="dsw/irisapp.json" Target="${cspdir}dsw/configs/irisapp.json"/> If the Name ends with a slash, it will copy it as a folder. During uninstall it will delete Target folder <FileCopy Name="dsw/" Target="${cspdir}dsw/configs/"/> And added support for packaging and uninstalling module. When the module uninstalled it correctly deletes previously copied files. 2) Invoke improvements Don't need wrapper tag anymore Added a few more attributes: CheckStatus (Default: false), expects that invoked method returns %Status, and use it to check the status of installation. Phase, (Default: Configure), the phase during which it should be executed When, (Default: After), values: Before and After. To execute before or after the main execution for the phase added support for variables, added ${verbose} variable <Invoke Class="%EnsembleMgr" Method="EnableNamespace" Phase="Compile" When="Before" CheckStatus="true"> <Arg>${namespace}</Arg> <Arg>${verbose}</Arg> </Invoke> Docker images with IRIS 2020.4 and 2020.3 and ZPM 0.2.10 are now available. Tags: intersystemsdc/iris-community:2020.4.0.524.0-zpm intersystemsdc/iris-community:2020.3.0.221.0-zpm intersystemsdc/iris-ml-community:2020.3.0.302.0-zpm intersystemsdc/irishealth-community:2020.4.0.524.0-zpm intersystemsdc/irishealth-community:2020.3.0.221.0-zpm More info in documentation https://github.com/intersystems-community/zpm/wiki/04.-Docker-Images And to launch the image do: docker run --name my-iris -d --publish 9091:51773 --publish 9092:52773 intersystemsdc/iris-community:2020.4.0.524.0-zpm docker run --name my-iris -d --publish 9091:51773 --publish 9092:52773 intersystemsdc/iris-community:2020.3.0.221.0-zpm docker run --name my-iris -d --publish 9091:51773 --publish 9092:52773 intersystemsdc/iris-ml-community:2020.3.0.302.0-zpm docker run --name my-iris -d --publish 9091:51773 --publish 9092:52773 intersystemsdc/irishealth-community:2020.4.0.524.0-zpm docker run --name my-iris -d --publish 9091:51773 --publish 9092:52773 intersystemsdc/irishealth-community:2020.3.0.221.0-zpm

Hey Community, We're pleased to invite you to the InterSystems AI+ML Summit 2021, which will be held virtually from January 25 to February 4! Join us for a two-week event that ranges from thought leadership to technical sessions and even 1:1 “Ask the Expert” sessions. The sessions will be in both German and English. And this summit is free to attend! See details below: Discover new perspectives. Actively shape the future. According to Statista, an enormous amount of 74 zettabytes of data will be generated by the end of 2021. Without the help of artificial intelligence (AI) and machine learning (ML) technologies, this massive flood of information cannot be meaningfully channeled and transformed into value-added business processes. That’s why you shouldn’t miss the virtual InterSystems AI + ML Summit 2021, where you'll learn everything you need to know about the current state of AI and ML in general and AI + ML with InterSystems in particular. The spectrum of topics ranges from "hot" AI trends to concrete use cases in different industries and interactive workshops. In the first week, we focus on AI and ML in healthcare, while in the second week we look at the topics from an industry-agnostic standpoint. So, remember: ⏱ Time: Jan 25 - Feb 04, 2021 📍 Location: Online ✅ Registration: SAVE THE DATE Join us to find out how you can successfully use the modern AI and ML technologies with InterSystems IRIS, IRIS for Health and HealthShare to create added value for you and your customers! Hi, While registering I am getting German language. How can I get English one? Thanks Unfortunately, the registration page does not provide a choice of a language other than German. ok Thanks what's the speaker's language? Is it English or German? German is the language of the conference, but some sessions are in English: When is Artificial Intelligence Trustworthy? Convergent Analytics Making Machine Learning Actionable Enriching Master Data Management product KanoMDM with AI/ML Hi there, I am from Sales Team China. Where could we download those videos? We would like to translate the subtitle and distribute to our partners and customers. Thanks a lot and looking forward to your feedback! Kind regards, Jun Qian The recordings will be made available to all registrants on Friday this week. Uwe Hering (uwe.hering at intersystems dot com) can send you the video files for subtitling / editing - please send him an email.

Hello, everyone! InterSystems IRIS has a menu called **Interoperability**. It provides mechanisms to easily create system integrations (adapters, record maps, BPMs, data conversions, etc.) so different systems can be easily connected. A variety of operations can be included in the data relay process, as examples we can cite: to connect systems that are not normally connected, data can be received (or sent) according to the specifications of the destination system. Also, information can be acquired and added from another system before sending the data. As well as, information can be acquired and updated from a database (IRIS or other). In this series of articles, we will discuss the following topics while looking at sample code to help you understand how it works and what kind of development is required when integrating systems with Interoperability. * How it works * What a Production is * Message * Component Creation * Business Operations * Business Processes * Business Services First, let me introduce you to the case study we will be using in this series. > A company operates a shopping site and is changing the order of displaying product information to match the seasons.However, some items sell well regardless of the season, while others sell at unexpected times, which does not match the current display rule of changing the order.Therefore, we looked into the possibility of changing the order of display to match the day's temperature instead of the season. It became necessary to survey the temperature of the purchased products at that time.Since an external Web API is available for checking weather information, we plan to collect the weather information at the time of purchase and register it in the later review database. It’s very simple, but you need to use an “external Web API” to collect the information, and you need to combine the information obtained and the purchase information to register in the database. Specific instructions will be discussed in a related article (not including the creation of a web site). Please take a look! As for the "external Web API" that we are using this time, we are using OpenWeather's Current weather data. (If you want to try it out, you need to register an account and get an API ID). The following is the result of a GET request from a REST client (we will run this in the mechanism we will implement in Interoperability).  The JSON of the HTTP response is as follows: ```json { "coord": { "lon": 135.5022, "lat": 34.6937 }, "weather": [ { "id": 803, "main": "Clouds", "description": "broken clouds", "icon": "04d" } ], "base": "stations", "main": { "temp": 17.05, "feels_like": 13.33, "temp_min": 16, "temp_max": 18, "pressure": 1017, "humidity": 55 }, "visibility": 10000, "wind": { "speed": 4.63, "deg": 70 }, "clouds": { "all": 75 }, "dt": 1611635756, "sys": { "type": 1, "id": 8032, "country": "JP", "sunrise": 1611612020, "sunset": 1611649221 }, "timezone": 32400, "id": 1853909, "name": "Osaka", "cod": 200 } ``` In the next article, we will discuss how the Interoperability menu works for system integration. [OpenWeather]: https://openweathermap.org/ Great article!

Hi developers! We had a set of technology bonuses for the InterSystems Multi-Model contest. And here is how projects collected it: Project Docker zpm Key-Value SQL Object Your Model Total Bonus zapm 2 2 2 2 2 10 iris-vaccine-tracker 2 2 2 2 2 10 qewd-jsdb-kit-iris 2 2 2 6 BeI-MultiModel 2 2 2 2 2 10 iris-image-index-demo 2 2 2 2 2 2 12 IRIS Publisher 2 2 2 2 8 iris-sftp-logs 2 2 2 2 2 10 iris-multimodel-suite 2 2 2 2 2 10 GeoDatas 2 2 2 6 HealthInfoQueryLayer 2 2 2 6 This table is subject to change upon project improvements or if we missed something. Fill free to claim the bonus in the comments here or on discord. Stay tuned!

Hi Community, Find out how you can use the open-source iKnow engine in Python within InterSystems IRIS. ⏯ Using the InterSystems iKnow Engine in Python Subscribe to InterSystems Developers YouTube and stay tuned!

We released a new version of ZPM, with a set of great enhancements: $zpm function new arguments for search command and more! See the details below $zpm function With $zpm function you can execute ZPM commands programmatically and get the status. set sc=$zpm("install new-module") Search command Most often, ZPM works with the pm.community.intersystems.com registry. Now more than 100 applications are published in this registry, and their number is constantly increasing.We've added an argument to the search command that lets you specify the application name or part of the application name. zpm: USER>search zpmregistry https://pm.community.intersystems.com:zpm 0.2.7 Use an asterisk for any sequence of characters zpm: USER>search zpm*registry https://pm.community.intersystems.com:zpm 0.2.7zpm-registry 1.0.9zpmshow 1.0.1 We added the -d (-description) flag.When this flag is specified, a short description will be shown for each package in the list of found packages. Package authors should specify the description in the module.xml filе. Dark theme If you are using a dark theme in terminal (or VS Code) try this new feature in ZPM which will affect the output in the search command. Use config set ColorScheme dark to set dark theme, and config delete ColorScheme to reset to default @Nikolay.Soloviev $zpm function is excellent news! It allows us now to run ZMP install also remote to multiple instances of IRISusing IRIS Native API for ObjectScript ! This means to me: distributed installation without login to SMP or Studio or Terminal. Thanks for this improvement! I really like the opportunity to see which modules are installed in which namespaces. And quickly go there without leaving the shell of the ZPM. ``` zpm: USER>help zn namespace [] Alias: zn See list modules in namespace and go to the namespace Arguments name Name namespace, * or context name* Examples zn * Show all modules in all namespaces zn sql* Show all modules in namespaces by context ```

Hello, everyone, Now that Virtual Summit 2020 is over, we'd like to remind everyone that we would still love to hear your input on Certification topics via a very brief survey - less than 8 minutes. You can even indicate if you would like to be a Subject Matter Expert for us in the future and what kinds of incentives you'd like to receive to do so. Please let us know your thoughts: https://www.surveymonkey.com/r/2ZVHNPS Thanks, Jamie Kantor, Certification Manager

Hey Developers, Our next community session from Virtual Summit 2020 is already on InterSystems Developers YouTube: 🏆 Best Applications of InterSystems Programming Contest Series 🏆 Learn about the winning applications from a series of online contests for InterSystems developers. Developers will share their experiences of participating in the InterSystems coding marathon and show demos of their winning projects. Winning projects:⬇️ IRIS-History-Monitor⬇️ IRIS4Health-FHIR-Analytics⬇️ SQL-Builder⬇️ BlocksExplorer⬇️ IRIS-ML-Suite Presenters: 🗣 @Anastasia.Dyubaylo, Community Manager, InterSystems 🗣 @Henrique.GonçalvesDias, System Management Specialist / Database Administrator, Sao Paulo Federal Court🗣 @José.Pereira, Business Intelligence Developer, Shift Consultoria e Sistemas Ltda🗣 @Henry.HamonPereira, System Analyst, BPlus Tecnologia🗣 @Dmitry.Maslennikov, Co-founder, CTO and Developer Advocate, CaretDev Corp🗣 @Renato.Banzai, Machine Learning Engineer Coordinator, Itaú Unibanco So, big applause for all the speakers! Great developers! Great apps! We hope you enjoyed our series of online contests in 2020. Stay tuned for the next contests this year! 😉 Enjoy!

Hi developers! Often we need to deploy some data along with code pieces of the application. And for InterSystems IRIS developers the question could sound: "How can I deploy the data I have in globals?" Here I want to suggest to you one of the approaches - deploying global data using the ZPM package manager. Exporting Globals Data Suppose you have the IRIS database server where you have the global which you want to deploy. The ZPM package manager can deploy files so you need to export the global into a file and build the package with this file. ZPM can deploy globals in XML format, so we need to export a global in an XML file first. E.g. if the global you need to export has the name "DataD" the following command in the IRIS terminal will export the global DataD in XML file: d $System.OBJ.Export("DataD.GBL","/irisrun/repo/data/DataD.xml") How the Resource Looks Like To build the package with a global we should introduce certain resource element in module XML like: <Resource Name="DataD.GBL"/> See the example in documentation. Notice this resource element will look for the DataD.XML file, not DataD.GBL as you could expect. And ZPM will look for the DataD.XML file in the /gbl folder inside the folder listed in <SourceRoot> element. Example Here is a sample repository iris-dataset-countries which contains a package that deploys global with the data on different countries. Here is the module XML: <?xml version="1.0" encoding="UTF-8"?> <Export generator="Cache" version="25"> <Document name="dataset-countries.ZPM"> <Module> <Name>dataset-countries</Name> <Description>Module imports the data of Country passengers in dc.data.Tinanic class</Description> <Version>1.0.0</Version> <Packaging>module</Packaging> <SourcesRoot>src</SourcesRoot> <Resource Name="dc.data.Country.CLS"/> <Resource Name="dc.data.CountryD.GBL"/> </Module> </Document> </Export> And we could see the resource: <Resource Name="dc.data.CountryD.GBL"/> Which is located in /src/gbl/dc.data.CountryD.XML file in the repository: So when ZPM loads the module into IRIS it imports the global according to the module.xml. You can test install the global (and the class for it to make queries) with: USER>zpm "install dataset-countries" Or you are welcome to play with global packaging with the Countries or Titanic datasets.

Hey developers, As you know, the InterSystems Multi-Model contest has already ended. And now we want to get feedback from those developers who were unable to participate. Please answer some questions to help us improve our contests! 👉🏼 Quick survey: InterSystems Multi-Model Contest Survey Or please share your thoughts in the comments to this post!

As we all well know, InterSystems IRIS has an extensive range of tools for improving the scalability of application systems. In particular, much has been done to facilitate the parallel processing of data, including the use of parallelism in SQL query processing and the most attention-grabbing feature of IRIS: sharding. However, many mature developments that started back in Caché and have been carried over into IRIS actively use the multi-model features of this DBMS, which are understood as allowing the coexistence of different data models within a single database. For example, the [HIS qMS](https://openexchange.intersystems.com/package/HIS-qMS) database contains both semantic relational (electronic medical records) as well as traditional relational (interaction with PACS) and hierarchical data models (laboratory data and integration with other systems). Most of the listed models are implemented using [SP.ARM](https://openexchange.intersystems.com/company/SP-ARM)'s qWORD tool (a mini-DBMS that is based on direct access to globals). Therefore, unfortunately, it is not possible to use the new capabilities of parallel query processing for scaling, since these queries do not use IRIS SQL access. Meanwhile, as the size of the database grows, most of the problems inherent to large relational databases become right for non-relational ones. So, this is a major reason why we are interested in parallel data processing as one of the tools that can be used for scaling. In this article, I would like to discuss those aspects of parallel data processing that I have been dealing with over the years when solving tasks that are rarely mentioned in discussions of Big Data. I am going to be focusing on the technological transformation of databases, or, rather, technologies for transforming databases. It's no secret that the data model, storage architecture, and software and hardware platform are usually chosen at the early stages of system development, often when the project is still far from completion. However, some time will pass, and it is quite often the case when several years after the system was deployed, the data needs to be migrated for one reason or another. Here are just a few of the commonly encountered tasks (all examples are taken from real life): 1. A company is planning to go international, and its database with 8-bit encoding must be converted to Unicode. 2. An outdated server is being replaced with a new one, but it is either impossible to seamlessly transfer journals between servers (using Mirroring or Shadowing IRIS system features) due to licensing restrictions or lack of capabilities to meet existing needs, such as, for example, when you are trying to solve task (1). 3. You find that you need to change the distribution of globals among databases, for example, moving a large global with images to a separate database. You might be wondering what is so difficult about these scenarios. All you need to do is stop the old system, export the data, and then import it into the new system. But if you are dealing with a database that is several hundred gigabytes (or even several terabytes) in size, and your system is running in 7x24 mode, you won't be able to solve any of the mentioned tasks using the standard IRIS tools. ## Basic approaches to task parallelization ### "Vertical" parallelization Suppose that you could break a task down into several component tasks. If you are lucky, you find out that you can solve some of them in parallel. For example, - Preparing data for a report (calculations, data aggregation…) - Applying style rules - Printing out reports can all be performed at the same time for several reports: one report is still in the preparation phase, another is already being printed out at the same time, etc. This approach is nothing new. It has been used ever since the advent of batch data processing, that is, for the last 60 years. However, even though it is not a new concept, it is still quite useful. However, you will only realize a noticeable acceleration effect when all subtasks have a comparable execution time, and this is not always the case. ### "Horizontal" parallelization When the order of operations for solving a task consists of iterations that can be performed in an arbitrary order, they can be performed at the same time. For example: - Contextual search in the global: - You can split the global into sub-globals ($order by the first index). - Search separately in each of them. - Assemble the search results. - Transferring the global to another server via a socket or ECP: - Break the global into parts. - Pass each of them separately. Common features of these tasks: - Identical processing in subtasks (down to sharing the same parameters), - The correctness of the final result does not depend on the order of execution of these subtasks, - There is a weak connection between the subtasks and the "parent" task only at the level of result reporting, where any required postprocessing is not a resource-intensive operation. These simple examples suggest that horizontal parallelism is natural for data transformation tasks, and indeed, it is. In what follows, we will mainly focus on this type of parallel processing. ## "Horizontal" parallelization ### One of the approaches: MapReduce MapReduce is a distributed computing model that was introduced by [Google](https://ru.bmstu.wiki/Google_Inc.). It is also used to execute such operations to process large amounts of information at the same time. Popular open source implementations are built on a combination of [Apache Hadoop](https://ru.bmstu.wiki/Apache_Hadoop) ([https://ru.bmstu.wiki/Apache\_Hadoop](https://ru.bmstu.wiki/Apache_Hadoop)) and [Mahout](https://ru.bmstu.wiki/Apache_Mahout)[https://ru.bmstu.wiki/Apache\_Mahout](https://ru.bmstu.wiki/Apache_Mahout). Basic steps of the model: [Map](https://ru.bmstu.wiki/index.php?title=Map&action=edit&redlink=1) (distribution of tasks between handlers), the actual processing, and [Reduce](https://ru.bmstu.wiki/index.php?title=Reduce&action=edit&redlink=1) (combining the processing results). For the interested reader who would like to know more, I can recommend the series of articles by Timur Safin on his approach to creating the MapReduce tool in IRIS/Caché, which starts with [Caché MapReduce - an Introduction to BigData and the MapReduce Concept (Part I)](https://community.intersystems.com/post/cach%C3%A9-mapreduce-introduction-bigdata-and-mapreduce-concept). Note that due to the "innate ability" of IRIS to write data to the database quickly, the Reduce step, as a rule, turns out to be trivial, as in the [distributed version of WordCount](https://community.intersystems.com/post/cach%C3%A9-mapreduce-putting-it-all-together-%E2%80%93-wordcount-example-part-iii#comment-14196). When dealing with database transformation tasks, it may turn out to be completely unnecessary. For example, if you used parallel handlers to move a large global to a separate database, then we do not need anything else. ### How many servers? The creators of parallel computing models, such as MapReduce, usually extend it to several servers, the so-called data processing nodes, but in database transformation tasks, one such node is usually sufficient. The fact of the matter is that it does not make sense to connect several processing nodes (for example, via the Enterprise Cache Protocol (ECP)), since the CPU load required for data transformation is relatively small, which cannot be said about the amount of data involved in processing. In this case, the initial data is used once, which means that you should not expect any performance gain from distributed caching. Experience has shown that it is often convenient to use two servers whose roles are asymmetric. To simplify the picture somewhat: - The source database is mounted on one server (*Source DB*). - The converted database is mounted on the second server (*Destination DB*). - Horizontal parallel data processing is configured only on one of these servers; the operating processes on this server are the *master* processes to - the processes running on the second server, which are, in turn, the *slave* processes; when you use ECP, these are the DBMS system processes (ECPSrvR, ECPSrvW, and ECPWork), and when using a socket-oriented data transfer mechanism, these are the child processes of TCP connections. We can say that this approach to distributing tasks combines horizontal parallelism (which is used to distribute the load within the master server) with vertical parallelism (which is used to distribute "responsibilities" between the master and slave servers). ## Tasks and tools Let us consider the most general task of transforming a database: transferring all or part of the data from the source database to the destination database while possibly performing some kind of re-encoding of globals (this can be a change of encoding, change of collation, etc.). In this case, the old and new databases are local on different database servers. Let us list the subtasks to be solved by the architect and developer: 1. Distribution of roles between servers. 2. Choice of data transmission mechanism. 3. Choice of the strategy for transferring globals. 4. Choice of the tool for distributing tasks among several processes. Let's take a bird's eye view of them. ### Distribution of roles between servers As you are already well familiar with, even if IRIS is being installed with Unicode support, it is also able to mount 8-bit databases (both local and remote). However, the opposite is not true: the 8-bit version of IRIS will not work with a Unicode database, and there will be inevitable <WIDE CHAR> errors if you try to do so. This must be taken into account when deciding which of the servers – the source or the destination – will be the master if the character encoding is changed during data transformation. However, it is impossible here to decide on an ultimate solution without considering the next task, which is the ### Choice of data transmission mechanism You can choose from the following options here: 1. If the license and DBMS versions on both servers allow the ECP to be used, consider ECP as a transport. 2. If not, then the simplest solution is to deal with both databases (the source and destination) locally on the destination system. To do this, the source database file must be copied to the appropriate server via any available file transport, which, of course, will take additional time (for copying the database file over the network) and space (to store a copy of the database file). 3. In order to avoid wasting time copying the file (at least), you can implement your mechanism for exchanging data between the server processes via a TCP socket. This approach may be useful if: - The ECP cannot be used for some reason, e.g. due to the incompatibility of the DBMS versions serving the source and destination databases (for example, the source DBMS is of a very legacy version), - Or: It is impossible to stop users from working on the source system, and therefore the data modification in the source database that occurs in the process of being transferred must be reflected in the destination database. My priorities when choosing an approach are pretty evident: if the ECP is available and the source database remains static while it is transferred – 1, if the ECP is not available, but the database is still static – 2, if the source database is modified – 3. If we combine these considerations with master server choice, then we can produce the following possibility matrix: | **Is the source DB static during transmission?** | **Is the ECP protocol available?** | **Location of the DB source** | **Master system** | | --- | --- | --- | --- | | Yes | Yes | Remote on the target system | Target | | Yes | No | Local (copy) on the target system | Target | | No | It does not matter, as we will use our mechanism for transferring data over TCP sockets. | Local (original) on the source system | Source| ### Choice of the strategy for transferring globals At first glance, it might seem that you can simply pass globals one by one by reading the Global Directory. However, the sizes of globals in the same database can vary greatly: I recently encountered a situation when globals in a production database ranged in size between 1 MB and 600 GB. Let's imagine that we have nWorkers worker processes at our disposal, and there is at least one global ^Big for which it is true: Size(^Big) > (Summary Size of All ^Globals) / nWorkers Then, no matter how successfully the task of transferring the remaining globals is distributed between the working processes, the task that ends up being assigned the transfer of the ^Big global will remain busy for the remaining allocated part of the time and will probably only finish its task long after the other processes finish processing the rest of the globals. You can improve the situation by pre-ordering the globals by size and starting the processing with the largest ones first, but in cases where the size of ^Big deviates significantly from the average value for all globals (which is a typical case for the MIS qMS database): Size(^Big) >> (Summary Size of All ^Globals) / nWorkers this strategy will not help you much, since it inevitably leads to a delay of many hours. Hence, you cannot avoid splitting large global into parts to allow its processing using several parallel processes. I wish to emphasize that this task (number 3 in my list) turned out to be the most difficult among others being discussed here, and took most of my (rather than CPU!) time to solve it. ### Choice of the tool for distributing tasks among several processes The way that we interact with the parallel processing mechanism can be described as follows: - We create a pool of background worker processes. - A queue is created for this pool. - The initiator process (let's call it the _local manager_), having a plan that was prepared in advance in step 3, places _work units_ in the queue; as a rule, the _work unit_ comprises the name and actual arguments of a certain class method. - The worker processes retrieve work units from the queue and perform processing, which boils down to calling a class method with the actual arguments that are passed to the work units. - After receiving confirmation from all worker processes that the processing of all queued work units is complete, the local manager releases the worker processes and finalizes processing if required. Fortunately, IRIS provides an excellent parallel processing engine that fits nicely into this scheme, which is implemented in the %SYSTEM.WorkMgr class. We will use it in a running example that we will explore across a planned series of articles. In the next article, I plan to focus on clarifying the solution to task number 3 in more detail. In the third article, which will appear if you will show some interest in my writing, I will talk about the nuances of solving task number 4, including, in particular, about the limitations of %SYSTEM.WorkMgr and the ways of overcoming them. Awesome! Nice article. Very much looking forward to your views on %SYSTEM.WorkMgr, which has been getting a lot of attention to help us serve the most demanding SQL workloads from our customers. You'll also be able to monitor some of its key metrics such as active worker jobs and average queue length in SAM starting with 2021.1. Interesting article, Alexey!But we are still eagerly waiting for the 2nd part, when we should expect it? Thanks for your interest. I try to fulfill it ASAP.