Search

Hi Community, We're pleased to invite all the developers to the upcoming InterSystems AI Contest Kick-Off Webinar! The topic of this webinar is dedicated to the InterSystems AI programming contest. During the webinar, we will demo how to load data into IRIS, how to deal with it using ODBC/JDBC and REST, and how to use special AI/ML features of IRIS: IntegratedML, DataRobot, R Gateway, Embedded Python, PMML. Date & Time: Monday, June 28 — 11:00 AM EDT Speakers: 🗣 @Aleksandar.Kovacevic, InterSystems Sales Engineer🗣 @Théophile.Thierry, InterSystems Intern🗣 @Robert.Kuszewski, Product Manager - Developer Experience, InterSystems 🗣 @Evgeny.Shvarov, InterSystems Developer Ecosystem Manager Join the webinar to find out all the details about this competition and ask your questions to our speakers! ✅ REGISTER TODAY! Hey Developers, The recording of this webinar is available on InterSystems Developers YouTube! Please welcome: ⏯ InterSystems AI Contest Kick-Off Webinar Big applause to our speakers! 👏🏼

Just want to know what is the reason to user Visual Studio .Net editor over InterSystems studio. Thanks I prefer editing in Visual Studio Code for two main reasons: It's available on Windows, Mac, and Linux. So I always have it available. It's a great editor for all my code - ObjectScript, JavaScript, Python, Java. https://intersystems-community.github.io/vscode-objectscript/introduction/ The reason for that will be that it is Cross Platform my friend. Visual Studio Code is supported on all the available platforms as at present. And also it has a compatibility to to edit, run and debug any version of the code that you have chosen. Ex: F#, C#, VB, Python etc. Hope this was useful and i have made it simple and clear. Thanks. We use both depending on the situation. VS Code is more modern and certainly better for Git and Merge tools than Studio but it doesn't have all the abilities of Studio that I've found, global storage properties etc. We've found VS Code doesn't handle physical files as well as Studio at least for InterSystems so its unfriendly when using CSPs that are version controlled.

Hello Respected members, I am new with InterSystems IRIS , want to setup the environment on RHEL 7.5 with multiple lun layouts (separate LVM for DB , Journal, and WIJ), I would like to get some help in setting up the same, as far as I know, the installation will go through on a single directory ( I did not see any option to chose path/directory for DB's or WIJ /journals), hence after installation I would like to move my DB's to separate LVM and WIJ on another LVM set Please suggest or share the steps for the same document if any for the same, that will be really helpful, Thanks Regards RS 1. Install InterSystems IRIS. 2. Shut it down. 3. Open <iris>/iris.cpf, there you will find: [Databases] USER=C:\InterSystems\IRIS\Mgr\user\ [Journal] AlternateDirectory=C:\InterSystems\IRIS\Mgr\journal\ CurrentDirectory=C:\InterSystems\IRIS\Mgr\journal\ [config] wijdir=C:\InterSystems\IRIS\Mgr\WIJ 4. Edit this file as you wish and move the corresponding directories. 5. Start InterSystems IRIS. @Eduard.Lebedyuk Thank you so much for providing this information I would like to know if I can move only user database or default databases as well?? Like irislib, irisaudit, irislocaldata and other under mgr folder Please suggest if I can do this if yes then what sort of permission do I need to change for new directory path and what all files I need to copy at new location?? I want to setup my layout as given below Database LUNs = 8 x 512GB- - - > vg_iris_db Journal LUNs = 2 x 512GB - - > vg_iris_jrn WIJ LUNs = 2 x 256GB - - > vg_iris_wij Install and IRISTEMP LUNs = 2 x 128GB - - - > vg_iris_sys Please suggest if modification in iris.cpf file can do this all operation Thanks & Regards RS I would like to know if I can move only user database or default databases as well?? All databases can be moved what sort of permission do I need to change for new directory path irisusr should have RW access to the folder. The easiest way is to check what access (owner, group, permissions) is set for existing files/folders and then recreate the same in a new location. and what all files I need to copy at new location?? IRIS.DAT and stream subfolder if it exist. Please suggest if modification in iris.cpf file can do this all operation Absolutely. @Eduard.Lebedyuk Thank you so much for providing the detailed information. I am able to deploy the IRIS Instance with custom layout which I wanted to One thing I noticed that after installation I was able to see the instance running from command line but I was unable to access management portal with the URL that comes after installation, anything I missed or need to change to make it accessible? Please suggest Thanks RS Do you get Unable to connect or something else? What's the output of iris list? Hello Rahul, I'd check your messages.log to see messages regarding the private webserver. Then I'd check the OS to see if anything is holding your configured web server port. This could be worth a new post, or contacting the WRC. I am getting below error if I tried to access the URL : This site can’t be reached scsps2391082001’s server IP address could not be found. Below is the output from iris list [root@scsps2391082001 prd]# iris list Configuration 'PROD' (default) directory: /prd versionid: 2020.1.1.408.0com datadir: /prd conf file: iris.cpf (SuperServer port = 51773, WebServer = 52773) status: running, since Thu Jun 10 05:45:18 2021 state: ok product: InterSystems IRIS[root@scsps2391082001 prd]# Below are the Installation details Please review the installation options:------------------------------------------------------------------Instance name: PRODDestination directory: /prdInterSystems IRIS version to install: 2020.1.1.408.0comInstallation type: DevelopmentUnicode support: YInitial Security settings: NormalUser who owns instance: rootGroup allowed to start and stop instance: rootEffective group for InterSystems IRIS processes: irisusrEffective user for InterSystems IRIS SuperServer: irisusrSuperServer port: 51773WebServer port: 52773JDBC Gateway port: 53773Web Gateway: using built-in web server------------------------------------------------------------------ Confirm InterSystems IRIS installation <Yes>? Starting installationStarting up InterSystems IRIS for loading...../bin/irisinstall -s . -B -c c -C /prd/iris.cpf*PROD -W 1 -g2Starting Control ProcessAllocated 159MB shared memory: 2MB global buffers, 35MB routine buffersIRIS startup successful.System locale setting is 'en_US.UTF-8'This copy of InterSystems IRIS has been licensed for use exclusively by:InterSystems IRIS CommunityCopyright (c) 1986-2021 by InterSystems CorporationAny other use is a violation of your license agreement ^^/prd/mgr/> ^^/prd/mgr/>Start of IRIS initialization Loading system routines Updating system TEMP and LOCALDATA databases Installing National Language support Setting IRISTEMP default collation to IRIS standard (5) Loading system classes Updating Security database Loading system source code Building system indices Updating Audit database Updating Journal directory Updating User database Updating Interoperability databases Scheduling inventory scan IRIS initialization complete in 2 minutes See the iboot.log file for a record of the installation. Starting up InterSystems IRIS...Once this completes, users may access InterSystems IRISStarting PRODUsing 'iris.cpf' configuration file Starting Control ProcessAutomatically configuring buffersAllocated 925MB shared memory: 725MB global buffers, 35MB routine buffersCreating a WIJ file to hold 99 megabytes of dataThis copy of InterSystems IRIS has been licensed for use exclusively by:InterSystems IRIS CommunityCopyright (c) 1986-2021 by InterSystems CorporationAny other use is a violation of your license agreement You can point your browser to http://scsps2391082001:52773/csp/sys/UtilHome.cspto access the management portal. Let me know if anything else I need to provide it as a details ? Regards RS Thanks @Vic.Sun I will create a new post and will share the messages.log entry there Thank you all for providing the inputs for my queries Regards RS Hello Rahul - I didn't see your new post yet but this message is pretty clear. The problem is at the OS level in resolving the hostname, scsps2391082001. If you replace the hostname in your URL with the actual IP of your server, it should work fine. If you want the hostname to work, I'd work with your OS/networking team to set that up.

Hey Community, You've probably already heard about Discord and many of you are already chatting there. And now we invite you to become closer to the world of InterSystems technology and join the social club of our developers! Use a super-fast way to communicate with each other: 💥 InterSystems Developers Discord Channel 💥 On DC Discord Server, you'll find many InterSystems-related channels and discussions, as well as channels for everyday life conversations. Just join our club to get closer to the world of InterSystems developers! See you on the InterSystems Developers Discord ✌️ If you have any suggestions for improving our Discord Server or want to create a new channel on a specific topic – please feel free to share your thoughts in the comments. Hey everyone, To stay in touch, let's continue our tech talks on InterSystems Developers Discord Server! See you there ✌️

Try the new InterSystems IRIS® FHIR® Accelerator Learning Path from Online Learning. Learn the basics of this fully managed service, which allows for the storage and sharing of healthcare data via a secure and scalable FHIR® repository. Get an overview of the FHIR Accelerator Service—and see how to deploy it—by following this curated collection of multimedia resources. Target audience: Software developers and other technical audiences.

Hi Developers, Learn about the InterSystems IRIS FHIR Server, the fully managed FHIR data solution that empowers FHIR application developers to focus on building life-changing healthcare applications: ⏯ What is the InterSystems IRIS FHIR Accelerator Service? Subscribe to InterSystems Developers YouTube and stay tuned!

Hey everyone, The InterSystems Grand Prix Contest is over. It was an incredible competition with a record number of participating apps and developers! Thank you all for participating! And now it's time to announce the winners! A storm of applause goes to these developers and their applications: 🏆 Experts Nomination - winners were determined by a specially selected jury: 🥇 1st place and $6,000 go to the vscode-intersystems-iris project by @Dmitry.Maslennikov 🥈 2nd place and $3,000 go to the iris-rad-studio project by @Henrique.GonçalvesDias and @José.Pereira 🥉 3rd place and $2,000 go to the HealthInfoQueryLayer project by @Botai.Zhang 🏆 Community Nomination - an application that received the most votes in total: 🥇 1st place and $3,000 go to the HealthInfoQueryLayer project by @Botai.Zhang 🥈 2nd place and $1,500 go to the Dictionary comparison scheme of cache database project by @Weiwei.Yang 🥉 3rd place and $500 go to the vscode-intersystems-iris project by @Dmitry.Maslennikov And... This time, we would also like to reward the developers who took 4-10 places in the Expert nomination! 10,000 points on Global Masters go to these apps and their developers: 🏅 4th place: iris-image-index-demo by @José.Pereira 🏅 5th place: Terminal Multi-Line Command Editor​​​​ by @Robert.Cemper1003 🏅 6th place: Dictionary comparison scheme of cache database by @Weiwei.Yang 🏅 7th place: Create a unified hospital data extraction scheme based on IRIS for Health by @Deming.Xu 🏅 8th place: iris4health-fhir-analytics by @José.Pereira 🏅 9th place: iris-fhir-portal by @Henrique.GonçalvesDias 🏅 10th place: ObjectScript Kernel by @Nikita.Mullin Congratulations to all the winners and participants! Thank you all for your attention to the contest and the efforts you pay in our mega competition! Congratulations to all winners! Congratulations to all winners! Congratulations to all winners! Congratulations to all winners! Congratulations! And congratulations to all the participants - that's awesome work, great applications! Big congratulations to you and @José.Pereira!!! 🤩 Hey everyone, As always, we really appreciate the contribution of all participants to the Grand Prix competition. And this time, we would also like to reward the developers who took 4-10 places in the Expert nomination! 10,000 points on Global Masters go to these apps and their developers: 🏅 4th place: iris-image-index-demo by @José.Pereira 🏅 5th place: Terminal Multi-Line Command Editor​​​​ by @Robert.Cemper1003 🏅 6th place: Dictionary comparison scheme of cache database by @Weiwei.Yang 🏅 7th place: Create a unified hospital data extraction scheme based on IRIS for Health by @Deming.Xu 🏅 8th place: iris4health-fhir-analytics by @José.Pereira 🏅 9th place: iris-fhir-portal by @Henrique.GonçalvesDias 🏅 10th place: ObjectScript Kernel by @Nikita.Mullin Thank you all! And our big congratulations! That's good news! and a real surprise! I appreciate it highly Congratulations to all winners! Thanks @Anastasia.Dyubaylo It's our pleasure to create apps that people enjoy Hey developers! We also want to thank the rest of the participants and their cool applications that were in the InterSystems Grand Prix Contest: interoperability-integratedml-adapter , iris-integratedml-monitor-example and iris-analytics-notebook by @José Roberto Pereira IRIS-REST-API-DATABASEMANAGER by @Lucas.Bourré IRIS Interoperability Message Viewer , iris-history-monitor , npm-iris and isc-generate-db by @Henrique Dias Wsock-Embedded-Py , The adopted Bitmap , Using ZPM for Node.js and WebSocket Client JS with IRIS Native API as Docker Micro Server by @Robert Cemper iris-for-money by @Oliver.Wilms springboot-iris-crud by @Yuri.Gomes Airplane React, Material UI, and REST API by @FlávioLúcio.NavesJúnior appmsw-telestat , zapm , apptools-admin and isc-apptools-lockdown by @MikhailenkoSergey realworld-intersystems-iris , isc-tar and BlocksExplorer by @Dmitry.Maslennikov RESTFUL_API_For_Hotel_OverBooking_System by @jingqi.LIu cmPurgeBackup by @Alexey.Maslov fhir-chatbot , iris-ml-suite and iris-python-suite by @Renato.Banzai IRIS import manager by @Oleh.Dontsov ISC DEV by @Gevorg.Arutiunian Questy by Alexey Nechaev M-commands instead of Excel formulas in cells by @alex.kosinets units by @Dmitrii.Kuznetsov We are waiting for your applications in the next contests🔥🔥🔥 Congratulations to all winners! Congratulations to all Congratulations to all winners ! Well done guys! I'm very proud to had been participated! Glad to see how these contests are stimulating so many people to do their best! I see those apps as ground to create awesome projects in 2021! Indeed @Robert.Cemper1003! Thank you @Anastasia.Dyubaylo ! :) Very happy to had the chance of sharing a project with @Henrique.GonçalvesDias. More brains and hands working together it's better! Congratulations to the winners for their commitment, dedication and for the work presented. certainly, that initiatives like this contribute in a unique way to the expansion of technology and all its potential.

InterSystems has been widely used in medical and health fields. Which other industries have high market share? What's the difference between InterSystems Trakcare and InterSystems Healthshare? we also expanding into Finance, Logistics, Government etc. Trakcare is our package HIS/EMR solution: https://www.intersystems.com/cn/products/trakcare/ HealthShare is a product family to support unified care record. For more info pls connect with our local sales team. @Jun.Qian. THX! Metallurgy, construction, clothing industry, peat extraction, trade, logistics. Also as universal system for financial and accounting for any enterprise.

 This article contains the materials, examples, exercises to learn the basic concepts of IAM. You have all resources available on this git : https://github.com/grongierisc/iam-training. Solutions are in [training branch](https://github.com/grongierisc/iam-training/tree/training). This article will cover the following points : - [1. Introduction](#1-introduction) - [1.1. What is IAM ?](#11-what-is-iam-) - [1.2. What is an API Management ?](#12-what-is-an-api-management-) - [1.3. IAM Portal](#13-iam-portal) - [1.4. Flow of this training](#14-flow-of-this-training) - [2. Installation](#2-installation) - [2.1. What do you need to install?](#21-what-do-you-need-to-install) - [2.2. How IAM works with IRIS](#22-how-iam-works-with-iris) - [2.3. Setup](#23-setup) - [2.4. Install IAM](#24-install-iam) - [2.4.1. Iris Image](#241-iris-image) - [2.4.2. IAM Image](#242-iam-image) - [2.4.3. Update the docker file](#243-update-the-docker-file) - [2.4.4. Update the docker-compose](#244-update-the-docker-compose) - [2.4.5. Option : add IRIS_PASSWARD as .env](#245-option--add-iris_passward-as-env) - [2.4.6. Test it !](#246-test-it-) - [3. First Service/Route](#3-first-serviceroute) - [3.1. Create a service](#31-create-a-service) - [3.2. Create a route](#32-create-a-route) - [3.3. Test it !](#33-test-it-) - [4. Second, go further with plugin](#4-second-go-further-with-plugin) - [4.1. Add a plugin to the service](#41-add-a-plugin-to-the-service) - [4.2. Test it !](#42-test-it-) - [5. Third, add our own authentication](#5-third-add-our-own-authentication) - [5.1. Add consumers](#51-add-consumers) - [5.2. Add Basic auth plugin](#52-add-basic-auth-plugin) - [5.3. Add ACL Plugin](#53-add-acl-plugin) - [5.4. Configure USER with ACL and credentials](#54-configure-user-with-acl-and-credentials) - [5.5. Test it !](#55-test-it-) - [6. Exercice, Rate-Limiting](#6-exercice-rate-limiting) - [6.1. Solution](#61-solution) - [7. Dev Portal](#7-dev-portal) - [7.1. Overview](#71-overview) - [7.2. Enable it !](#72-enable-it-) - [7.3. Add your first spec](#73-add-your-first-spec) - [7.4. Test it !](#74-test-it-) - [7.5. Exercise](#75-exercise) - [7.5.1. Solution](#751-solution) - [8. Dev Portal, Part two, Authentication](#8-dev-portal-part-two-authentication) - [8.1. Enable Basic Auth](#81-enable-basic-auth) - [8.2. Limit access](#82-limit-access) - [8.2.1. Create a role](#821-create-a-role) - [8.2.2. Add role to Spec](#822-add-role-to-spec) - [8.2.3. Test it !](#823-test-it-) - [8.2.3.1. Register a new developer](#8231-register-a-new-developer) - [8.2.3.2. Approve this developer](#8232-approve-this-developer) - [8.2.3.3. Add role for this developer](#8233-add-role-for-this-developer) - [8.3. Add Oauth2 for developer](#83-add-oauth2-for-developer) - [8.3.1. First, remove basic auth](#831-first-remove-basic-auth) - [8.3.2. Second, add application-registration plugin](#832-second-add-application-registration-plugin) - [8.3.3. Link service and documentation](#833-link-service-and-documentation) - [8.3.3.1. Test it !](#8331-test-it-) - [9. Secure Management Portal](#9-secure-management-portal) - [9.1. Create an admin](#91-create-an-admin) - [9.2. Enable Basic Auth for Kong Manager](#92-enable-basic-auth-for-kong-manager) - [9.3. Use Kong Admin API with RBAC](#93-use-kong-admin-api-with-rbac) - [9.3.1. Create and admin user with a token](#931-create-and-admin-user-with-a-token) - [10. Plugins](#10-plugins) - [10.1. Import a community plugin](#101-import-a-community-plugin) - [10.1.1. Build a new Kong/IAM docker image with the community plugin](#1011-build-a-new-kongiam-docker-image-with-the-community-plugin) - [10.1.2. Test it !](#1012-test-it-) - [10.1.2.1. Use it !](#10121-use-it-) - [10.2. Create a new plugin](#102-create-a-new-plugin) - [10.2.1. File structure](#1021-file-structure) - [10.2.1.1. handler.lua](#10211-handlerlua) - [10.2.1.1.1. Example](#102111-example) - [10.2.1.2. schema.lua](#10212-schemalua) - [10.2.1.3. *.rockspec](#10213-rockspec) - [10.2.2. Build it](#1022-build-it) - [10.2.3. Tips](#1023-tips) - [11. CI/CD](#11-cicd) - [11.1. Create the postman collection](#111-create-the-postman-collection) - [11.1.1. Is IAM startup ?](#1111-is-iam-startup-) - [11.1.2. Delete old datas](#1112-delete-old-datas) - [11.1.3. Create Service/Route](#1113-create-serviceroute) - [11.1.3.1. Tips](#11131-tips) - [11.2. Run it with newman](#112-run-it-with-newman) # 1. Introduction  ## 1.1. What is IAM ? IAM stand for InterSystems API Manager, it's based on **Kong Enterprise Edition**. This mean you have access on top of Kong Open Source edition to : * Manager Portal * Developer Portal * Advance plugin * Oauth2 * Caching * ...  ## 1.2. What is an API Management ? API management is the process of creating and publishing web application programming interfaces (APIs), enforcing their usage policies, controlling access, nurturing the subscriber community, collecting and analyzing usage statistics, and reporting on performance. API Management components provide mechanisms and tools to support developer and subscriber community.  ## 1.3. IAM Portal Kong and IAM are design as API first, this mean, everything done in Kong/IAM can be done by rest calls or the manager portal. During this article all example / exercise will present both this way: |IAM Portal|Rest API| |-----------------|--------------| ||| ## 1.4. Flow of this article The aim of this article is to use IAM as a proxy of an IRIS rest API. Definition of this rest API can be found here : ```url http://localhost:52773/swagger-ui/index.html#/ ``` or here ```url https://github.com/grongierisc/iam-training/blob/training/misc/spec.yml ``` Start this article with the main branch. At the end of the article, you should have the same result as the training branch. # 2. Installation  ## 2.1. What do you need to install? * [Git](https://git-scm.com/downloads) * [Docker](https://www.docker.com/products/docker-desktop) (if you are using Windows, make sure you set your Docker installation to use "Linux containers"). * [Docker Compose](https://docs.docker.com/compose/install/) * [Visual Studio Code](https://code.visualstudio.com/download) + [InterSystems ObjectScript VSCode Extension](https://marketplace.visualstudio.com/items?itemName=daimor.vscode-objectscript) * InterSystems IRIS IAM enabled license file. * IAM Docker image ## 2.2. How IAM works with IRIS At Kong/IAM start, the container check for the Kong/IAM license with a curl call. The endpoint of this call is a rest API on the IRIS container. FYI : Kong license is embedded in IRIS one.  ## 2.3. Setup Git clone this repository. git clone https://github.com/grongierisc/iam-training Run the initial rest API : docker-compose up Test it : ```url http://localhost:52773/swagger-ui/index.html#/ ``` Login/Password : SuperUser/SYS ## 2.4. Install IAM ### 2.4.1. Iris Image First you need to switch for the community edition to a licensed one. To do so, you need to setup your access to InterSystems Container Registry to download IRIS limited access images. Have a look at this [Introducing InterSystems Container Registry](https://community.intersystems.com/post/introducing-intersystems-container-registry) on [Developer Community](https://community.intersystems.com). * Log-in into https://containers.intersystems.com/ using your WRC credentials and get a *token*. * Set up docker login in your computer: ```bash docker login -u="user" -p="token" containers.intersystems.com ``` * Get InterSystems IRIS image: ```bash docker pull containers.intersystems.com/intersystems/irishealth:2020.4.0.524.0 ``` ### 2.4.2. IAM Image In [WRC Software Distribution](https://wrc.intersystems.com/wrc/coDistribution.csp): * Components > Download *IAM-1.5.0.9-4.tar.gz* file, unzip & untar and then load the image: ```bash docker load -i iam_image.tar ``` ### 2.4.3. Update the docker file Change IRIS community edition to a licensed one. * containers.intersystems.com/intersystems/irishealth:2020.4.0.524.0 * add iris.key in key folder Edit the dockerfile to add on top of it this part ```dockerfile ARG IMAGE=containers.intersystems.com/intersystems/irishealth:2020.4.0.524.0 # Frist stage FROM $IMAGE as iris-iam COPY key/iris.key /usr/irissys/mgr/iris.key COPY iris-iam.script /tmp/iris-iam.script RUN iris start IRIS \ && iris session IRIS < /tmp/iris-iam.script \ && iris stop IRIS quietly # Second stage FROM iris-iam ``` This part will create a multi-stage dockerfile. * the first stage is to enable IRIS to serve IAM license. * the second stage is for the REST API build Create a new file iris-iam.script to build a new IRIS Image to enable IAM endpoint and user. ```objectscript zn "%SYS" write "Create web application ...",! set webName = "/api/iam" set webProperties("Enabled") = 1 set status = ##class(Security.Applications).Modify(webName, .webProperties) write:'status $system.Status.DisplayError(status) write "Web application "_webName_" was updated!",! set userProperties("Enabled") = 1 set userName = "IAM" Do ##class(Security.Users).Modify(userName,.userProperties) write "User "_userName_" was updated!",! halt ``` ### 2.4.4. Update the docker-compose Update the docker-compose file to : * db * postgres database for IAM * iam-migration * bootstrap the database * iam * actual IAM instance * a volume for data persistent Add this part to the end of the docker-compose file. ```yml iam-migrations: image: intersystems/iam:1.5.0.9-4 command: kong migrations bootstrap up depends_on: - db environment: KONG_DATABASE: postgres KONG_PG_DATABASE: ${KONG_PG_DATABASE:-iam} KONG_PG_HOST: db KONG_PG_PASSWORD: ${KONG_PG_PASSWORD:-iam} KONG_PG_USER: ${KONG_PG_USER:-iam} KONG_CASSANDRA_CONTACT_POINTS: db KONG_PLUGINS: bundled,jwt-crafter ISC_IRIS_URL: IAM:${IRIS_PASSWORD}@iris:52773/api/iam/license restart: on-failure links: - db:db iam: image: intersystems/iam:1.5.0.9-4 depends_on: - db environment: KONG_ADMIN_ACCESS_LOG: /dev/stdout KONG_ADMIN_ERROR_LOG: /dev/stderr KONG_ADMIN_LISTEN: '0.0.0.0:8001' KONG_ANONYMOUS_REPORTS: 'off' KONG_CASSANDRA_CONTACT_POINTS: db KONG_DATABASE: postgres KONG_PG_DATABASE: ${KONG_PG_DATABASE:-iam} KONG_PG_HOST: db KONG_PG_PASSWORD: ${KONG_PG_PASSWORD:-iam} KONG_PG_USER: ${KONG_PG_USER:-iam} KONG_PROXY_ACCESS_LOG: /dev/stdout KONG_PROXY_ERROR_LOG: /dev/stderr KONG_PORTAL: 'on' KONG_PORTAL_GUI_PROTOCOL: http KONG_PORTAL_GUI_HOST: '127.0.0.1:8003' KONG_ADMIN_GUI_URL: http://localhost:8002 KONG_PLUGINS: bundled ISC_IRIS_URL: IAM:${IRIS_PASSWORD}@iris:52773/api/iam/license volumes: - ./iam:/iam links: - db:db ports: - target: 8000 published: 8000 protocol: tcp - target: 8001 published: 8001 protocol: tcp - target: 8002 published: 8002 protocol: tcp - target: 8003 published: 8003 protocol: tcp - target: 8004 published: 8004 protocol: tcp - target: 8443 published: 8443 protocol: tcp - target: 8444 published: 8444 protocol: tcp - target: 8445 published: 8445 protocol: tcp restart: on-failure db: image: postgres:9.6 environment: POSTGRES_DB: ${KONG_PG_DATABASE:-iam} POSTGRES_PASSWORD: ${KONG_PG_PASSWORD:-iam} POSTGRES_USER: ${KONG_PG_USER:-iam} volumes: - 'pgdata:/var/lib/postgresql/data' healthcheck: test: ["CMD", "pg_isready", "-U", "${KONG_PG_USER:-iam}"] interval: 30s timeout: 30s retries: 3 restart: on-failure stdin_open: true tty: true volumes: pgdata: ``` Add the .env file in root folder : ```env IRIS_PASSWORD=SYS ``` BTW : Here are the definition of Kong ports : |Port|Protocol|Description| |----|--------|-----------| |:8000|HTTP|Takes incoming HTTP traffic from Consumers, and forwards it to upstream Services.| |:8443|HTTPS|Takes incoming HTTPS traffic from Consumers, and forwards it to upstream Services.| |:8001|HTTP|Admin API. Listens for calls from the command line over HTTP.| |:8444|HTTPS|Admin API. Listens for calls from the command line over HTTPS.| |:8002|HTTP|Kong Manager (GUI). Listens for HTTP traffic.| |:8445|HTTPS|Kong Manager (GUI). Listens for HTTPS traffic.| |:8003|HTTP|Dev Portal. Listens for HTTP traffic, assuming Dev Portal is enabled.| |:8446|HTTPS|Dev Portal. Listens for HTTPS traffic, assuming Dev Portal is enabled.| |:8004|HTTP|Dev Portal /files traffic over HTTP, assuming the Dev Portal is enabled.| |:8447|HTTPS|Dev Portal /files traffic over HTTPS, assuming the Dev Portal is enabled.| ### 2.4.5. Option : add IRIS_PASSWARD as .env For ease of use (and may be security), you can use the .env file in the IRIS dockerfile. To do so, edit the docker-compose with this in the iris service part : ```yml build: context: . dockerfile: dockerfile args: - IRIS_PASSWORD=${IRIS_PASSWORD} ``` And the dockerfile (second or first stage of the build): ```dockerfile ARG IRIS_PASSWORD RUN echo "${IRIS_PASSWORD}" > /tmp/password.txt && /usr/irissys/dev/Container/changePassword.sh /tmp/password.txt ``` ### 2.4.6. Test it ! docker-compose -f "docker-compose.yml" up -d --build # 3. First Service/Route  Remember how Kong/IAM works ?  Here, we will build : * a service * for our crud API * a route * to access this service ## 3.1. Create a service IAM Portal Rest API # Create service curl -i -X POST \ --url http://localhost:8001/services/ \ --data 'name=crud' \ --data 'url=http://iris:52773/crud/' What do we see here, to create a service we simply need it's url. ## 3.2. Create a route IAM Portal Rest API # Create route curl -i -X POST \ --url http://localhost:8001/services/crud/routes \ --data 'name=crud-route' \ --data 'paths=/persons/*' \ --data 'strip_path=false' What do we see here, to create a route we need : * it's service name * a path where RegEx is allowed ## 3.3. Test it ! **Original API** # Legacy curl –i --location --request GET 'http://localhost:52773/crud/persons/all' \ --header 'Authorization: Basic U3VwZXJVc2VyOlNZUw==' **Proxy API** # KONG curl –i --location --request GET 'http://localhost:8000/persons/all' \ --header 'Authorization: Basic U3VwZXJVc2VyOlNZUw==' What do we see here : * Nothing new on legacy side. * On kong side : * We change the port * The path corresponds to the route * We still need to authenticate # 4. Second, go further with plugin To go further, we will try to auto-authenticate Kong to the IRIS endpoint. To do so, we will use and plugin, resquest-transformer.  ## 4.1. Add a plugin to the service IAM Portal Rest API # Create plugin curl -i -X POST \ --url http://localhost:8001/services/crud/plugins \ --data 'name=request-transformer' \ --data 'config.add.headers=Authorization:Basic U3VwZXJVc2VyOlNZUw==' \ --data 'config.replace.headers=Authorization:Basic U3VwZXJVc2VyOlNZUw==' ## 4.2. Test it ! # Legacy **Original API** curl –i --location --request GET 'http://localhost:52773/crud/persons/all' **Proxy API** # KONG curl –i --location --request GET 'http://localhost:8000/persons/all' What do we see here : * Error 401 on the original API * We reach the data without authentication # 5. Third, add our own authentication What we want to achieved here is to add our own authentication without any distuption of the original API.  ## 5.1. Add consumers IAM Portal Rest API # Add consumer anonymous curl -i -X POST \ --url http://localhost:8001/consumers/ \ --data "username=anonymous" \ --data "custom_id=anonymous" # Add consumer user curl -i -X POST \ --url http://localhost:8001/consumers/ \ --data "username=user" \ --data "custom_id=user" ## 5.2. Add Basic auth plugin IAM Portal Rest API # Enable basic auth for service curl -i -X POST http://localhost:8001/routes/crud-route/plugins \ --data "name=basic-auth" \ --data "config.anonymous=5cc8dee4-066d-492e-b2f8-bd77eb0a4c86" \ --data "config.hide_credentials=false" Where : * config.anonymous = uuid of anonymous consumer ## 5.3. Add ACL Plugin IAM Portal Rest API # Enable ACL curl -i -X POST http://localhost:8001/routes/crud-route/plugins \ --data "name=acl" \ --data "config.whitelist=user" ## 5.4. Configure USER with ACL and credentials IAM Portal Rest API # Add consumer group curl -i -X POST \ --url http://localhost:8001/consumers/user/acls \ --data "group=user" # Add consumer credentials curl -i -X POST http://localhost:8001/consumers/user/basic-auth \ --data "username=user" \ --data "password=user" ## 5.5. Test it ! **Original API** # Legacy curl –i --location --request GET 'http://localhost:52773/crud/persons/all' \ --header 'Authorization:Basic dXNlcjp1c2Vy' **Proxy API ** # KONG curl –i --location --request GET 'http://localhost:8000/persons/all' \ --header 'Authorization:Basic dXNlcjp1c2Vy' # 6. Exercice, Rate-Limiting 1. Enable Unauthenticated user 2. Limit rate by 2 calls per minutes to Unauthenticated user ## 6.1. Solution 1. Enable Unauthenticated user IAM Portal Rest API # Add consumer group curl -i -X POST \ --url http://localhost:8001/consumers/anonymous/acls \ --data "group=user" 2. Limit rate by 2 calls per minutes to Unauthenticated user IAM Portal Rest API # Add rate limit consumer curl -i -X POST \ --url http://localhost:8001/consumers/anonymous/plugins \ --data "name=rate-limiting" \ --data "config.limit_by=consumer" \ --data "config.minute=2" # 7. Dev Portal  ## 7.1. Overview The Kong Developer Portal provides : * a single source of truth for all developers * intuitive content management for documentation * streamlined developer onboarding * role-based access control (RBAC)  ## 7.2. Enable it ! IAM Portal Rest API curl -X PATCH http://localhost:8001/workspaces/default --data "config.portal=true" ## 7.3. Add your first spec **IAM Portal** **Rest API** curl -X POST http://localhost:8001/default/files -F "path=specs/iam-training.yml" -F "contents=@misc/spec.yml" ## 7.4. Test it ! ```url http://localhost:8003/default/documentation/iam-training ``` What happen ? How-to solve it ? ## 7.5. Exercise 1. Add CORS plugin on route ### 7.5.1. Solution IAM Portal Rest API # Enable CORS curl -i -X POST http://localhost:8001/routes/crud-route/plugins \ --data "name=cors" # 8. Dev Portal, Part two, Authentication ## 8.1. Enable Basic Auth IAM Portal Session Config (JSON) { "cookie_secure": false, "cookie_name": "portal_session", "secret": "SYS", "storage": "kong" } Now authentication is enabled for the dev portal. ## 8.2. Limit access By default, all is accessible for unauthenticated user. We can create role to limit some access. For example, let restrict access to our CRUD API documentation. ### 8.2.1. Create a role IAM Portal Rest API # Enable role curl -i -X POST http://localhost:8001/default/developers/roles \ --data "name=dev" ### 8.2.2. Add role to Spec **IAM Portal** **Rest API** # Enable role curl 'http://localhost:8001/default/files/specs/iam-training.yml' -X PATCH -H 'Accept: application/json, text/plain, */*' --compressed -H 'Content-Type: application/json;charset=utf-8' -H 'Origin: http://localhost:8002' -H 'Referer: http://localhost:8002/default/portal/permissions/' --data-raw $'{"contents":"x-headmatter:\\n readable_by:\\n - dev\\nswagger: \'2.0\'\\ninfo:\\n title: InterSystems IRIS REST CRUD demo\\n description: Demo of a simple rest API on IRIS\\n version: \'0.1\'\\n contact:\\n email: apiteam@swagger.io\\n license:\\n name: Apache 2.0\\n url: \'http://www.apache.org/licenses/LICENSE-2.0.html\'\\nhost: \'localhost:8000\'\\nbasePath: /\\nschemes:\\n - http\\nsecurityDefinitions:\\n basicAuth:\\n type: basic\\nsecurity:\\n - basicAuth: []\\npaths:\\n /:\\n get:\\n description: \' PersonsREST general information \'\\n summary: \' Server Info \'\\n operationId: GetInfo\\n x-ISC_CORS: true\\n x-ISC_ServiceMethod: GetInfo\\n responses:\\n \'200\':\\n description: (Expected Result)\\n schema:\\n type: object\\n properties:\\n version:\\n type: string\\n default:\\n description: (Unexpected Error)\\n /persons/all:\\n get:\\n description: \' Retreive all the records of Sample.Person \'\\n summary: \' Get all records of Person class \'\\n operationId: GetAllPersons\\n x-ISC_ServiceMethod: GetAllPersons\\n responses:\\n \'200\':\\n description: (Expected Result)\\n schema:\\n type: array\\n items:\\n $ref: \'#/definitions/Person\'\\n default:\\n description: (Unexpected Error)\\n \'/persons/{id}\':\\n get:\\n description: \' Return one record fo Sample.Person \'\\n summary: \' GET method to return JSON for a given person id\'\\n operationId: GetPerson\\n x-ISC_ServiceMethod: GetPerson\\n parameters:\\n - name: id\\n in: path\\n required: true\\n type: string\\n responses:\\n \'200\':\\n description: (Expected Result)\\n schema:\\n $ref: \'#/definitions/Person\'\\n default:\\n description: (Unexpected Error)\\n put:\\n description: \' Update a record in Sample.Person with id \'\\n summary: \' Update a person with id\'\\n operationId: UpdatePerson\\n x-ISC_ServiceMethod: UpdatePerson\\n parameters:\\n - name: id\\n in: path\\n required: true\\n type: string\\n - name: payloadBody\\n in: body\\n description: Request body contents\\n required: false\\n schema:\\n type: string\\n responses:\\n \'200\':\\n description: (Expected Result)\\n default:\\n description: (Unexpected Error)\\n delete:\\n description: \' Delete a record with id in Sample.Person \'\\n summary: \' Delete a person with id\'\\n operationId: DeletePerson\\n x-ISC_ServiceMethod: DeletePerson\\n parameters:\\n - name: id\\n in: path\\n required: true\\n type: string\\n responses:\\n \'200\':\\n description: (Expected Result)\\n default:\\n description: (Unexpected Error)\\n /persons/:\\n post:\\n description: \' Creates a new Sample.Person record \'\\n summary: \' Create a person\'\\n operationId: CreatePerson\\n x-ISC_ServiceMethod: CreatePerson\\n parameters:\\n - name: payloadBody\\n in: body\\n description: Request body contents\\n required: false\\n schema:\\n type: string\\n responses:\\n \'200\':\\n description: (Expected Result)\\n default:\\n description: (Unexpected Error)\\ndefinitions:\\n Person:\\n type: object\\n properties:\\n Name:\\n type: string\\n Title:\\n type: string\\n Company:\\n type: string\\n Phone:\\n type: string\\n DOB:\\n type: string\\n format: date-time\\n"}' What's important here is this part : ```yml x-headmatter: readable_by: - dev ``` Refere to this documentation : [readable_by attribute](https://docs.konghq.com/enterprise/1.5.x/developer-portal/administration/developer-permissions/#readable_by-attribute) ### 8.2.3. Test it ! #### 8.2.3.1. Register a new developer  #### 8.2.3.2. Approve this developer  #### 8.2.3.3. Add role for this developer  curl 'http://localhost:8001/default/developers/dev@dev.com' -X PATCH --compressed -H 'Content-Type: application/json;charset=utf-8' -H 'Cache-Control: no-cache' -H 'Origin: http://localhost:8002' -H 'DNT: 1' -H 'Connection: keep-alive' -H 'Referer: http://localhost:8002/default/portal/permissions/dev/update' -H 'Pragma: no-cache' --data-raw '{"roles":["dev"]}' ## 8.3. Add Oauth2 for developer In this part we will add an Oauth2 authentication for developers to use securely our crud API. This flow will provide self-registration from developer and grant them access to the crud API. ### 8.3.1. First, remove basic auth To do so, we will replace our basic auth to a bearToken one. First disable our basic auth/acl. **IAM Portal** **Rest API** # Disable ACL Plugin curl 'http://localhost:8001/default/routes/afefe836-b9be-49a8-927a-1324a8597a9c/plugins/3f2e605e-9cb6-454a-83ec-d1b1929b1d30' -X PATCH --compressed -H 'Content-Type: application/json;charset=utf-8' -H 'Cache-Control: no-cache' -H 'Origin: http://localhost:8002' -H 'DNT: 1' -H 'Connection: keep-alive' -H 'Referer: http://localhost:8002/default/plugins/acl/3f2e605e-9cb6-454a-83ec-d1b1929b1d30/update' -H 'Pragma: no-cache' --data-raw '{"enabled":false,"name":"acl","route":{"id":"afefe836-b9be-49a8-927a-1324a8597a9c"},"config":{"hide_groups_header":false,"whitelist":["user","dev","crud"]}}' ### 8.3.2. Second, add application-registration plugin IAM Portal Rest API # Create application-registration plugin curl -i -X POST \ --url http://localhost:8001/services/crud/plugins \ --data 'name=application-registration' \ --data 'config.auth_header_name=authorization' \ --data 'config.auto_approve=true' \ --data 'config.display_name=auth' \ --data 'config.enable_client_credentials=true' ### 8.3.3. Link service and documentation **IAM Porta** **Rest API** curl 'http://localhost:8001/default/document_objects' --compressed -H 'Content-Type: application/json;charset=utf-8' -H 'Cache-Control: no-cache' -H 'Origin: http://localhost:8002' -H 'DNT: 1' -H 'Connection: keep-alive' -H 'Referer: http://localhost:8002/default/services/create-documentation' -H 'Pragma: no-cache' --data-raw '{"service":{"id":"7bcef2e6-117c-487a-aab2-c7e57a0bf61a"},"path":"specs/iam-training.yml"}' #### 8.3.3.1. Test it ! From the dev portal logged as dev@dev.com, create a new application.  This will give you client_id and client_secret. Theses can be used in the swagger dev portal. Register this application to the crud service :  Get token: curl --insecure -X POST https://localhost:8443/persons/oauth2/token \ --data "grant_type=client_credentials" \ --data "client_id=2TXNvDqjeVMHydJbjv9t96lWTXOKAtU8" \ --data "client_secret=V6Vma6AtIvl04UYssz6gAxPc92eCF4KR" Use token : curl --insecure -X GET https://localhost:8443/persons/all \ --header "authorization: Bearer u5guWaYR3BjZ1KdwuBSC6C7udCYxj5vK" # 9. Secure Management Portal ## 9.1. Create an admin As we have bootstrap Kong without a seed password. We have to create an admin before enforcing RBAC. To do so: * Go to Teams * Invite admin * Set Mail * Set Username * Set role to super admin * Invite * Go to Invited Admin * View * Generate link  ## 9.2. Enable Basic Auth for Kong Manager To enable this feature, we have to change the docker-compose file. Add this to the iam service, environment ```yml KONG_ENFORCE_RBAC: 'on' KONG_ADMIN_GUI_AUTH: 'basic-auth' KONG_ADMIN_GUI_SESSION_CONF: '{"secret":"${IRIS_PASSWORD}","storage":"kong","cookie_secure":false}' ``` Restart the container docker-compose down && docker-compose up -d Go to the invited admin link : ```url http://localhost:8002/register?email=test.test%40gmail.com&username=admin&token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2MTYzMzYzNzEsImlkIjoiY2JiZGE5Y2UtODQ3NS00MmM2LTk4ZjItNDgwZTI4MjQ4NWNkIn0.sFeOc_5UPIr3MdlQrgyGvmvIjRFvSn3nQjo2ph8GrJA ``` ## 9.3. Use Kong Admin API with RBAC As RBAC is set, we can't use kong admin api anymore : curl -s -X GET \ --url http://localhost:8001/routes Get this error : ```json {"message":"Invalid credentials. Token or User credentials required"} ``` ### 9.3.1. Create and admin user with a token * Go to Teams * RBAC Users * Add new user  curl -s -X GET \ --url http://localhost:8001/routes \ --header "Kong-Admin-Token: SYS" # 10. Plugins  Kong come with high quality plugins. But, what if, we need plugin that are not embedded. If we want community plugins ? In this chapiter, we will talk about community plugins, how to import them. Then, we will see how-to build our own plugin. ## 10.1. Import a community plugin For this part, we will be using the jwt-crafter plugin. This plugin adds the possibility to generate a JWT token within Kong itself, eliminating the need for an upstream service doing the token generation. Here is the plugin : ``` https://github.com/grongierisc/kong-plugin-jwt-crafter ``` To install this plugin, as we are using the docker version, we have to build a new image who embed the plugin. ### 10.1.1. Build a new Kong/IAM docker image with the community plugin 1. Create a folder named iam at root of this git. 2. Create a dockerfile in this new folder 3. Create a folder named plugins 1. This is where we will add all our community plugins 4. Update the docker-compose file to enable the new plug in In the plugins folder, git clone our community plugin. git clone https://github.com/grongierisc/kong-plugin-jwt-crafter The dockerfile should look like this: ```dockerfile FROM intersystems/iam:1.5.0.9-4 USER root COPY ./plugins /custom/plugins RUN cd /custom/plugins/kong-plugin-jwt-crafter && luarocks make USER kong ``` What we see in this dockerfile ? Simply to install a community plugin, we have to move to its root folder (where the rockspec is) and call luarocks make. That's it. You have installed the plugin. For the docker-compose part : 1. Edit the iam iamge tag 1. intersystems/iam:1.5.0.9-4 -> intersystems/iam-custom:1.5.0.9-4 2. Add a build context ```yml build: context: iam dockerfile: dockerfile ``` 3. Enable the plugin in the environment variables ```yml KONG_PLUGINS: 'bundled,jwt-crafter' ``` Now build our new iam image : docker-compose build iam ### 10.1.2. Test it ! docker-compose up -d If you go to plugin -> new, at the bottom of the list you should see the jwt-crafter plugin.  #### 10.1.2.1. Use it ! 1. Create a new service : IAM Portal Rest API # Create service curl -i -X POST \ --url http://localhost:8001/services/ \ --data 'name=crud-persons' \ --data 'url=http://iris:52773/crud/persons/' 2. Create a route IAM Portal Rest API # Create route curl -i -X POST \ --url http://localhost:8001/services/crud-persons/routes \ --data 'name=crud-route-jwt' \ --data 'paths=/crud/persons/*' \ --data 'strip_path=true' 3. Re-use our auto-auth IAM Portal Rest API # Create plugin curl -i -X POST \ --url http://localhost:8001/services/crud-persons/plugins \ --data 'name=request-transformer' \ --data 'config.add.headers=Authorization:Basic U3VwZXJVc2VyOlNZUw==' \ --data 'config.replace.headers=Authorization:Basic U3VwZXJVc2VyOlNZUw==' Now we are set. The real use of jwt-crafter. ```bash # Add acl to route curl -i -X POST http://localhost:8001/routes/crud-route-jwt/plugins \ --data "name=acl" \ --data "config.whitelist=test" \ --data "config.hide_groups_header=false" # Create service curl -i -X POST \ --url http://localhost:8001/services/ \ --data 'name=jwt-login' \ --data 'url=http://neverinvoked/' # Create route curl -i -X POST \ --url http://localhost:8001/services/jwt-login/routes \ --data 'name=jwt-login-route' \ --data 'paths=/jwt/log-in' # Enable basic auth for service curl -i -X POST http://localhost:8001/routes/jwt-login-route/plugins \ --data "name=basic-auth" \ --data "config.hide_credentials=false" # Enable basic auth for service curl -i -X POST http://localhost:8001/routes/jwt-login-route/plugins \ --data "name=jwt-crafter" \ --data "config.expires_in=86400" # Add consumer curl -i -X POST \ --url http://localhost:8001/consumers/ \ --data "username=test" # Add consumer group curl -i -X POST \ --url http://localhost:8001/consumers/test/acls \ --data "group=test" # Add consumer credentials curl -i -X POST http://localhost:8001/consumers/test/basic-auth \ --data "username=test" \ --data "password=test" curl -i -X POST http://localhost:8001/consumers/test/jwt \ --data "key=test" \ --data "algorithm=HS256" # JWT plugins curl -i -X POST http://localhost:8001/routes/crud-route-jwt/plugins \ --data "name=jwt" ``` Test it ! ```bash # test:test is base64 encoded curl -H 'Authorization: basic dGVzdDp0ZXN0' localhost:8000/jwt/log-in ``` ```bash curl --location --request GET 'http://localhost:8000/crud/persons/all' \ --header 'Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJuYW0iOiJ0ZXN0Iiwic3ViIjoiODJiNjcwZDgtNmY2OC00NDE5LWJiMmMtMmYxZjMxNTViN2E2Iiwicm9sIjpbInRlc3QiXSwiZXhwIjoxNjE2MjUyMTIwLCJpc3MiOiJ0ZXN0In0.g2jFqe0hDPumy8_gG7J3nYsuZ8KUz9SgZOecdBDhfns' ``` ## 10.2. Create a new plugin This is not the place to learn lua. But I'll give you some tips like how to quickly restart IAM to test our new development. ### 10.2.1. File structure ``` kong-plugin-helloworld ├── kong │ └── plugins │ └── helloworld │ ├── handler.lua │ └── schema.lua └── kong-plugin-helloworld-0.1.0-1.rockspec ``` By convention, kong plugins must be prefix by kong-plugin. In our example, the name of the plugin is helloworld. Three files are mandatory : * handler.lua: the core of your plugin. It is an interface to implement, in which each function will be run at the desired moment in the lifecycle of a request / connection. * schema.lua: your plugin probably has to retain some configuration entered by the user. This module holds the schema of that configuration and defines rules on it, so that the user can only enter valid configuration values. * *.rockspec: Rockspec: a package specification file A declarative Lua script, with rules on how to build and package rocks *.rockspec - a Lua file containing some tables. #### 10.2.1.1. handler.lua The plugins interface allows you to override any of the following methods in your handler.lua file to implement custom logic at various entry-points of the execution life-cycle of Kong: |Function name |Phase |Description| |----|----|----| |:init_worker() |init_worker |Executed upon every Nginx worker process’s startup.| |:certificate() |ssl_certificate |Executed during the SSL certificate serving phase of the SSL handshake.| |:rewrite() |rewrite |Executed for every request upon its reception from a client as a rewrite phase handler. NOTE in this phase neither the Service nor the Consumer have been identified, hence this handler will only be executed if the plugin was configured as a global plugin!| |:access() |access |Executed for every request from a client and before it is being proxied to the upstream service.| |:response() |access |Replaces both header_filter() and body_filter(). Executed after the whole response has been received from the upstream service, but before sending any part of it to the client.| |:header_filter() |header_filter |Executed when all response headers bytes have been received from the upstream service.| |:body_filter() |body_filter |Executed for each chunk of the response body received from the upstream service. Since the response is streamed back to the client, it can exceed the buffer size and be streamed chunk by chunk. hence this method can be called multiple times if the response is large. See the lua-nginx-module documentation for more details.| |:log() |log |Executed when the last response byte has been sent to the client.| ##### 10.2.1.1.1. Example ```lua local BasePlugin = require "kong.plugins.base_plugin" local HelloWorldHandler = BasePlugin:extend() function HelloWorldHandler:new() HelloWorldHandler.super.new(self, "helloworld") end function HelloWorldHandler:access(conf) HelloWorldHandler.super.access(self) if conf.say_hello then ngx.log(ngx.ERR, "============ Hello World! ============") ngx.header["Hello-World"] = "Hello World!!!" else ngx.log(ngx.ERR, "============ Bye World! ============") ngx.header["Hello-World"] = "Bye World!!!" end end return HelloWorldHandler ``` #### 10.2.1.2. schema.lua Simply the configuration file see in the portal. ```lua return { no_consumer = true, fields = { say_hello = { type = "boolean", default = true }, say_hello_body = { type = "boolean", default = true } } } ``` #### 10.2.1.3. *.rockspec ```rockspec package = "kong-plugin-helloworld" -- hint: rename, must match the info in the filename of this rockspec! -- as a convention; stick to the prefix: `kong-plugin-` version = "0.1.0-1" -- hint: renumber, must match the info in the filename of this rockspec! -- The version '0.1.0' is the source code version, the trailing '1' is the version of this rockspec. -- whenever the source version changes, the rockspec should be reset to 1. The rockspec version is only -- updated (incremented) when this file changes, but the source remains the same. -- TODO: This is the name to set in the Kong configuration `plugins` setting. -- Here we extract it from the package name. local pluginName = package:match("^kong%-plugin%-(.+)$") -- "myPlugin" supported_platforms = {"linux", "macosx"} source = { url = "https://github.com/grongierisc/iam-training", branch = "master", -- tag = "0.1.0" -- hint: "tag" could be used to match tag in the repository } description = { summary = "This a demo helloworld for Kong plugin", homepage = "https://github.com/grongierisc/iam-training", license = "Apache 2.0" } dependencies = { "lua >= 5.1" -- other dependencies should appear here } build = { type = "builtin", modules = { ["kong.plugins."..pluginName..".handler"] = "kong/plugins/"..pluginName.."/handler.lua", ["kong.plugins."..pluginName..".schema"] = "kong/plugins/"..pluginName.."/schema.lua", } } ``` ### 10.2.2. Build it We will be doing the same as here : [11.1.1. Build a new Kong/IAM docker image with the community plugin](#1111-build-a-new-kongiam-docker-image-with-the-community-plugin) But adapted to our plugin : Dockerfile : ```dockerfile FROM intersystems/iam:1.5.0.9-4 USER root COPY ./plugins /custom/plugins RUN cd /custom/plugins/kong-plugin-jwt-crafter && luarocks make RUN cd /custom/plugins/kong-plugin-helloworld && luarocks make #USER kong #Stay with root use, we will see why later ``` Enable the plugin in the environment variables ```yml KONG_PLUGINS: 'bundled,jwt-crafter,helloworld' ``` Now build our new iam image : docker-compose build iam Then docker-compose up and test it. ### 10.2.3. Tips To run the IAM container in "debug mode", to easily stop/restart it, modify the dockerfile to add/remove plugin and so on. You can stop iam service : docker-compose stop iam And start it in run mode with a shell : docker-compose run -p 8000:8000 -p 8001:8001 -p 8002:8002 iam sh In the container : ./docker-entrypoint.sh kong Happy coding :) # 11. CI/CD  We are close to the end of this article. To finish let's talk about DevOps/CI/CD. The aim of this chapter is to give you some ideas about how to implement/script ci/cd for IAM/Kong. As Kong is API first, the idea is to script all the rest calls and play then on each environment. The easiest way to script rest calls is with postman and his best friend newman (command line version of postman). ## 11.1. Create the postman collection One thing handy with postman is its ability to run script before and after a rest call. We will use this functionality in most cases. ### 11.1.1. Is IAM startup ? Our first script will check if IAM is up and running.  ```javascript var iam_url = pm.environment.get("iam_url"); var iam_config_port = pm.environment.get("iam_config_port"); var url = "http://" + iam_url + ":" + iam_config_port + "/"; SenReq(20); async function SenReq(maxRequest) { var next_request = "end request"; const result = await SendRequest(maxRequest); console.log("result:",result); if(result == -1) { console.error("IAM starting .... failed !!!!"); } } function SendRequest(maxRequest) { return new Promise(resolve => { pm.sendRequest(url, function (err) { if (err) { if (maxRequest > 1) { setTimeout(function () {}, 5000); console.warn("IAM not started...retry..next retry in 5 sec"); SendRequest(maxRequest - 1); } else { console.error("IAM starting .... failed"); resolve(-1); } } else { console.log("IAM starting .... ok"); resolve(1); } } ); }); } ``` ### 11.1.2. Delete old datas ```javascript var iam_url=pm.environment.get("iam_url"); var iam_config_port=pm.environment.get("iam_config_port"); pm.sendRequest("http://"+iam_url+":"+iam_config_port+"/plugins", function (err, res) { if (err) { console.log("ERROR : ",err); } else { var body_json=res.json(); if(body_json.data) { for( i=0; i < body_json.data.length; i++) { // Example with a full fledged SDK Request route_id = body_json.data[i].id; const delete_route = { url: "http://"+iam_url+":"+iam_config_port+"/plugins/" + route_id, method: 'DELETE', }; pm.sendRequest(delete_route, function(err, res){ console.log(err ? err : res); }); } } } }); ``` Do the same for routes, services and consumers. This order is important beause you can't remove services with routes. ### 11.1.3. Create Service/Route Routes are dependent from services. For this type of cases we can use Test function of postman to retrieve data :  Screen Script var id = pm.response.json().id; var name = pm.response.json().name; pm.globals.set("service_crud_id", id); pm.globals.set("service_crud_name", name); Here we save from the response the id and name of the new services. Then we can use it in the next route creation : Screen Script service_crud_name = pm.globals.get("service_crud_name"); Here we retrieve the global variable "service_crud_name". Then, use it in the actual call. Screen Script { "paths": [ "/persons/*" ], "protocols": [ "http" ], "name": "crud-persons", "strip_path": false, "service": { "name": "{{service_crud_name}}" } } #### 11.1.3.1. Tips * paylaod can be either json or form-data * form-data :  * json :  Easy way to get the json format, go to the manager portal, view, copy json :  ## 11.2. Run it with newman docker run --rm -v "`pwd`/ci/":"/etc/newman" \ --network="iam-training_default" \ -t postman/newman run "DevOps_IAM.postman_collection.json" \ --environment="DevOps_IAM.postman_environment.json"

Hi Developers! Here're the technology bonuses for the InterSystems Developer Tools Contest that will give you extra points in the voting. Docker container usage - 2 points The application gets a 'Docker container' bonus if it uses InterSystems IRIS running in a docker container. Here is the simplest template to start from. ZPM Package deployment - 2 points You can collect the bonus if you build and publish the ZPM(ObjectScript Package Manager) package for your Full-Stack application so it could be deployed with: zpm "install your-multi-model-solution" command on IRIS with ZPM client installed. ZPM client. Documentation. Unit Testing - 2 points Applications that have Unit Testing for the InterSystems IRIS code will collect the bonus. Learn more about ObjectScript Unit Testing in Documentation and on Developer Community. Online Demo of your project - 3 pointsCollect 3 more bonus points if you provision your project to the cloud as an online demo. You can use this template or any other deployment option. Example. Learn more on deployment in Monday's Kick-Off webinar. Code quality analysis with zero bugs - 2 points Include the code quality Github action for code static control and make it show 0 bugs for ObjectScript. Learn more in Monday's Kick-Off webinar. Article on Developer Community - 2 points Post an article on Developer Community that describes features of your project. Collect 2 points for each article. Translations to different languages work too. Video on YouTube - 3 points Make the Youtube video that demonstrates your product in action and collect 3 bonus points per each. Example. The list of bonuses is subject to change. Stay tuned!

Like hardware hosts, virtual hosts in public and private clouds can develop resource bottlenecks as workloads increase. If you are using and managing InterSystems IRIS instances deployed in public or private clouds, you may have encountered a situation in which addressing performance or other issues requires increasing the capacity of an instance's host (that is, vertically scaling). One common reason to scale is insufficient memory. As described in Memory Management and Scaling for InterSystems IRIS in the Scalability Guide, providing enough memory for all of the entities running on the host of an InterSystems IRIS instance under all normal operating circumstances is a critical factor in both performance and availability. In one common scenario, as the workload of an InterSystems IRIS instance increases, its working set becomes too large to be held by the instance's allocated database cache. This forces some queries to fall back to disk, greatly increasing the number of disk reads required and creating a major performance problem. Increasing the size of the cache solves that problem, but if doing so would leave insufficient memory remaining for other purposes, you also need to increase the total physical memory on the host to avoid pushing the bottleneck to another part of the system. Fortunately, scaling a virtual host is typically a lot easier than scaling hardware. This post discusses the two stages of the process: Scaling the cloud host's resources You can change the resource specification of a virtual host on AWS, GCP, and Azure, using the platform's command line, API, or portal. VMWare vSphere allows you to easily change a number of resource parameters for a VM through its vSphere Client interface.. Reconfiguring InterSystems IRIS to take advantage of the scaled resources There are a number of ways to reconfigure InterSystems IRIS to take advantage of scaled host resources. This document describes the use of the configuration merge feature, which merges new parameter values, specified in a merge file, into an instance's CPF. Configuration merge is an easy and effective method because it lets you address only the configuration settings you want to modify, make multiple changes to an instance's configuration in one operation, and easily make the same set of changes to multiple instances. The procedures described here are manual, but in production they would very likely be automated, for example using a script that would apply a specific merge file in an accessible location to a list of instances. Scaling Cloud Host Resources Public cloud platforms provide a range of resource templates to choose from that specify CPU, memory, network interfaces, and other resources for virtual hosts (storage is provisioned and sized separately). To resize a host, you change the template selected when the host was provisioned to one that specifies more of the resources you want to increase. On Amazon Web Services, the resource template is called an instance type; for example, the t3.large instance type specifies 2 CPUs and 8 GB of memory. On Google Cloud Platform it's a machine type, such as the e2-standard-2 (which also includes 2 CPUs and 8 GB), and on Microsoft Azure it's a size (the Standard_B2ms calls for the same 2 CPUs and 8 GB). By redefining the instance type, machine type, or size of an existing public cloud host, you can scale its resource specifications. In a VMware vSphere private cloud, you can use the vSphere Client interface to the vCenter Server management console to directly modify one or more individual resource settings of an existing virtual machine. (You can also simultaneously scale groups of hosts on each platform.) The following sections provide brief examples of resizing individual virtual hosts on the various platforms, with links to the documentation for all available methods. Please note that these methods (APIs, command line interfaces, and portal interfaces) are offered and maintained by the cloud vendors, and the examples included here are for informational purposes, to illustrate how easily you can adapt InterSystems IRIS to take advantage of increased resources AWS To modify the instance type of an AWS host (called an instance, not to be confused with an InterSystems IRIS instance), you can use the modify-instance-attribute CLI command, as shown in the following example: $ aws ec2 describe-instances --instance-ids i-01519f663af48a55e { "Instances": [ { "AmiLaunchIndex": 0, "ImageId": "ami-0abcdef1234567890, "InstanceId": "i-1234567890abcdef0, "InstanceType": "m5n.large", ... $ aws ec2 stop-instances --instance-ids i-01519f663af48a55e { "StoppingInstances": [ { "InstanceId": "i-1234567890abcdef0", ... $ aws ec2 describe-instances --instance-ids i-01519f663af48a55e { "Instances": [ { ... "State": { "Code": 80, "Name": "stopped" } ... $ aws ec2 modify-instance-attribute --instance-ids i-01519f663af48a55e \ --instance-type "{\"Value\": \"m5n.xlarge\"}" $ aws ec2 start-instances --instance-ids i-01519f663af48a55e { "StartingInstances": [ { "InstanceId": "i-1234567890abcdef0", "CurrentState": { "Code": 0, "Name": "pending" }, "PreviousState": { "Code": 80, "Name": "stopped" ... $ aws ec2 describe-instances --instance-ids i-01519f663af48a55e { "Instances": [ { "AmiLaunchIndex": 0, "ImageId": "ami-0abcdef1234567890, "InstanceId": "i-1234567890abcdef0, "InstanceType": "m5n.xlarge", ... You can also make this change using the ModifyInstanceAttribute AWS API call or the AWS EC2 console. GCP To modify the machine type of a GCP host (also known as an instance), you can use the gcloud CLI to stop, modify, and restart the instance. For example, you would use the following commands to change the machine type of an instance named scalingTest to n1-highmem-96: $ gcloud compute instances stop scalingTest $ gcloud compute instances set-machine-type scalingTest --machine-type n1-highmem-32 $ gcloud compute instances start scalingTest You can also make this change using the Google Cloud Console or GCP API. Azure When you use the Azure CLI to modify the size of a Linux VM , you can view a list of the sizes available on the hardware cluster where the VM is hosted using the list-vm-resize-options command, for example: az vm list-vm-resize-options --resource-group testingGroup --name scalingTest --output table You can then use the resize command to change the VM's size to one of the listed options, as shown. This command restarts the VM automatically. az vm resize --resource-group testingGroup --name scalingTest --size Standard_E32d_v4 If the size you want to change the VM to is not available, you can deallocate the VM, which can then be resized to any size supported by the region and restarted; the commands involved are illustrated below: az vm deallocate --resource-group testingGroup --name scalingTest az vm resize --resource-group testingGroup --name scalingTest --size Standard_M128s az vm start --resource-group testingGroup --name scalingTest You can resize a Windows VM on Azure using either the Azure portal or Powershell commands. vSphere To resize a VMware vSphere VM, do the following: Open vSphere Client or Web Client and display the VM inventory. Right-click the VM you want to modify and select Edit Settings. On the Virtual Hardware tab, Expand Memory and change the amount of RAM configured for the VM. Expand CPU and change the number of cores and optionally the number of cores per socket. Make any other desired changes to the hardware resources allocated to the VM. Reconfiguring InterSystems IRIS for Scaled Resources Once you have scaled the host, the next step is to reconfigure InterSystems IRIS to take advantage of the increased resources by changing one or more parameters in the instance's configuration parameter file (CPF). For example, to continue with the common scenario mentioned at the start of this post, now that you have increased the host's memory resources, you will want to take advantage of this by increasing the size of the InterSystems IRIS instance's database cache (which is done by changing the value of the globals parameter) so it can keep more data in memory. An easy way to make such a change, and far the easiest and most repeatable way to make multiple changes to an instance's configuration in one operation or make the same changes to multiple instances, is to use the configuration merge feature, which is available on UNIX® and Linux systems. As described in Using Configuration Merge to Deploy Customized InterSystems IRIS Instances in Running InterSystems Products in Containers and Using the Configuration Merge Feature in the Configuration Parameter File Reference, configuration merge lets you specify a merge file containing the settings you want merged into an instance's CPF immediately prior to a restart. (In release 2021.1 you'll be able to do this on a running instance without restarting it.) Not only is this more convenient than editing an instance's CPF directly, but it is highly repeatable across multiple instances, and supports reliable change management by enabling you to keep an accurate record of changes simply by versioning the configuration merge files you apply. To execute a configuration merge, you need to do the following: Create the merge file with the parameters you want to modify. Stage the merge file in a location accessible to the instance. If the instance you are modifying is in a container (which is likely on a cloud host), you can stage the file in the instance's durable %SYS directory (see Durable %SYS for Persistent Instance Data in Running InterSystems Products in Containers). Specify the merge file's location using the ISC_CPF_MERGE_FILE environment variable before restarting the instance. For example, continuing with the case of the database cache that needs updating, suppose you wanted a containerized instance's database cache size increased to 100 GB. The setting for this, in the [config] section of the CPF, would be globals=102400, which sets the database cache for 8-kilobyte blocks to 102,400 MB, or 100 GB. (As explained in the globals description in the Configuration Parameter File Reference, the parameter sets the size of the cache for multiple block sizes; if only one value is provided, however, it is applied to the 8-kilobyte block size, and 0 [zero] is assumed for the other sizes; globals=102400 is therefore the equivalent of globals=0,0,102400,0,0,0.) To make this change, you might do the following on the cloud host: 1. Create a configuration merge file, called for example mergefile2021.06.30.cpf, containing these lines: [config] globals=102400 2. Stage the merge file in the durable %SYS directory on the host's file system, which if you mounted the external volume /data as /external in the container and used the ISC_DATA_DIRECTORY variable to specify /external/iris_durable as the durable %SYS directory for the instance, would be /data/iris_durable. 3. Use the docker exec command on the host's command line to specify the variable and restart the instance with the iris command; if the instance's container is named iris and the instance is named IRIS, for example, the command would look like this: docker exec iris ISC_CPF_MERGE_FILE=/data/iris_durable/mergefile2021.06.30.cpf iris stop IRIS restart 4. When the instance is restarted, you can confirm the new globals setting with this command: docker exec iris grep globals /data/iris_durable/iris.cpf 💡 This article is considered as InterSystems Data Platform Best Practice.

Hey Developers, This week is a voting week for the InterSystems AI programming contest! So, it's time to give your vote to the best solutions built with InterSystems IRIS. 🔥 You decide: VOTING IS HERE 🔥 How to vote? Details below. Experts nomination: This time, InterSystems experienced jury will choose the best apps to nominate the prizes in the Experts Nomination. Please welcome our experts: ⭐️ @Evgeny.Shvarov, Developer Ecosystem Manager⭐️ @Raj.Singh5479, Product Manager - Developer Experience⭐️ @Robert.Kuszewski, Product Manager - Developer Experience⭐️ @Thomas.Dyar, Product Specialist - Machine Learning⭐️ @Aleksandar.Kovacevic, Sales Engineer⭐️ @Eduard.Lebedyuk, Sales Engineer⭐️ @Sergey.Lukyanchikov, Sales Engineer⭐️ @Guillaume.Rongier7183, Sales Engineer Community nomination: For each user, a higher score is selected from two categories below: Conditions Place 1st 2nd 3rd If you have an article posted on DC and an app uploaded to Open Exchange (OEX) 9 6 3 If you have at least 1 article posted on DC or 1 app uploaded to OEX 6 4 2 If you make any valid contribution to DC (posted a comment/question, etc.) 3 2 1 Level Place 1st 2nd 3rd VIP Global Masters level or ISC Product Managers 15 10 5 Ambassador GM level 12 8 4 Expert GM level or DC Moderators 9 6 3 Specialist GM level 6 4 2 Advocate GM level or ISC Employees 3 2 1 Blind vote again! The number of votes for each app will be hidden from everyone. Once a day we will publish the leaderboard in the comments to this post. The order of projects on the Contest Page will be as follows: the earlier an application was submitted to the competition, the higher it will be in the list. P.S. Don't forget to subscribe to this post (click on the bell icon) to be notified of new comments. To take part in the voting, you need: Sign in to Open Exchange – DC credentials will work. Make any valid contribution to the Developer Community – answer or ask questions, write an article, contribute applications on Open Exchange – and you'll be able to vote. Check this post on the options to make helpful contributions to the Developer Community. If you changed your mind, cancel the choice and give your vote to another application! Support the application you like! Note: contest participants are allowed to fix the bugs and make improvements to their applications during the voting week, so don't miss and subscribe to application releases! OK! After the first day of the voting we have: Top 3 nominees by voting: cryptocurrency-rate-forecasting iris_satellite_plantation fhir-integratedml-example ➡️ Vote here! Experts, we are waiting for your votes! 🔥 Participants, improve & promote your solutions! Hey Developers! The second day of voting is over! Don't forget to vote for the best application! Good luck to the participants! Hi Devs! The voting continues! Who is gonna be the winner? Developers! The top nominees are changing! Who is now in the first place?🙀 Developers! The voting period is almost over! And here are the results at the moment: fhir-integratedml-example cryptocurrency-rate-forecasting ESKLP

Hi Community, We're pleased to invite you to the online meetup with the winners of the InterSystems AI contest! Date & Time: Friday, July 30, 2021 – 11:00 AM EDT What awaits you at this Virtual Meetup? Our winners' bios. Short demos on their applications. An open discussion about technologies being used. Q&A. Plans for the next contests. Our speakers: @José.Pereira, Business Intelligence Developer at Shift Consultoria e Sistemas Ltda @Henrique.GonçalvesDias, System Management Specialist / Database Administrator, Sao Paulo Federal Court @Oleh.Dontsov, Full Stack Developer, Tax Sketches SRO @Aleksandr.Kalinin6636, Programm Engineer at LLC "Escape" @Evgeny.Shvarov, InterSystems Developer Ecosystem Manager @Thomas.Dyar, InterSystems Product Specialist - Machine Learning @Raj.Singh5479, InterSystems Product Manager - Developer Experience @Robert.Kuszewski, InterSystems Product Manager - Developer Experience You will also have the opportunity to ask any questions to our developers in a special webinar chat. We will be happy to talk to you at our Virtual Meetup! ➡️ REGISTER TODAY! TODAY! Join our meetup at https://mologaconsulting.my.webex.com/meet/Community See you! Hey Developers, Watch the recording of the meetup on InterSystems Developers YouTube: ⏯ Online Meetup with the InterSystems AI Contest Winners Big applause to all the speakers! 👏🏼

Hello everyone, I’m a French student that just arrived in Prague for an academical exchange for my fifth year of engineering school and here is my participation in the interop contest. I hadn’t much time to code since I was moving from France to Prague and I’m participating alone, so I decided to make a project that’s more like a template rather than an application. I wanted to participate since my field (Data Science and AI) is not often linked with sustainability and the contest was a way for me to express myself in this important subject that is sustainability and Environnement. As you know, Artificial Intelligence is becoming more and more popular, with many well-known firms trying to follow the movement and sell tools to easily create machine learning models, train them and use them. All of this is practical and easy, but it comes with a cost, a financial one but also an Environnmental one. Training huge models again and again can take a lot of resources and produce a lot of CO2s. Supercomputers are running for days, and days and the size of the models are just exponentially increasing taking more space than ever. All these effort for some performance improvement that are not even sure to be find in some cases. Of course, this process is needed by many firms where even 0.1% of improvement in the accuracy of a model could save thousands of lives. That's why this template is designed for more common uses. However, as I had the opportunity to work with NLP (Natural Language Process) or Image Classification, I realized that some tools and models are already almost usable as is and can help us save hundreds of hours of training and therefore a lot of CO2s production and electricity consumption. That’s why I decided to create a template using InterSystems technologies to create an interoperable solution that answer some sustainability issues by allowing you to easily, in a few clicks, download pre-trained models from the internet ONCE, use them as much as you want for your own need, and of course, fine-tune these pre-trained models with new content available on an IRIS database and add content to the existing model. That way, in the template, we are taking an NLP model, trying it, then training it on data to create five new labels in the model to grade internet review. Therefore, by doing this, we end up with (If you have time and some compute power) a great model that can be used to predict the grade of internet review, that for free and for a small amount of CO2 produced. See [the GitHub](https://github.com/LucasEnard/Contest-Sustainability) and the Open Exchange post linked to this article. Or see the ReadMe here : # 1. Contest-Sustainability This is an template using InterSystems technologies to create an interoperable solution that answer some sustainability issues by allowing you to easily, in a few clicks, download pre-trained models from the internet ONCE, use them as much as you want for your own need, and of course, fine-tune these pre-trained models with new content available on an IRIS database and add content to the existing model. In this example we are taking an NLP model, trying it, then training it on data to create five new labels in the model to grade internet review. Saving in the process a lot of resources and CO2 emission. Here are some models as example you can try : https://huggingface.co/gpt2 https://huggingface.co/Jean-Baptiste/camembert-ner https://huggingface.co/bert-base-uncased https://huggingface.co/facebook/detr-resnet-50 https://huggingface.co/facebook/detr-resnet-50-panoptic TABLE OF CONTENT : - [1. Contest-Sustainability](#1-contest-sustainability) - [2. Installation](#2-installation) - [2.1. Starting the Production](#21-starting-the-production) - [2.2. Access the Production](#22-access-the-production) - [2.3. Closing the Production](#23-closing-the-production) - [3. How it works](#3-how-it-works) - [4. HuggingFace API](#4-huggingface-api) - [5. Use any model from the web](#5-use-any-model-from-the-web) - [5.1. FIRST CASE : YOU HAVE YOUR OWN MODEL](#51-first-case--you-have-your-own-model) - [5.2. SECOND CASE : YOU WANT TO DOWNLOAD A MODEL FROM HUGGINGFACE](#52-second-case--you-want-to-download-a-model-from-huggingface) - [5.2.1. Settings](#521-settings) - [5.2.2. Testing](#522-testing) - [6. Fine-tune the models](#6-fine-tune-the-models) - [6.1. Tunning the model](#61-tunning-the-model) - [6.1.1. Download the model](#611-download-the-model) - [6.1.2. Settings](#612-settings) - [6.1.3. Train the model](#613-train-the-model) - [6.1.4. Replace the model](#614-replace-the-model) - [6.2. Use the model](#62-use-the-model) - [6.2.1. Settings](#621-settings) - [6.2.2. Test the model](#622-test-the-model) - [7. Important note](#7-important-note) - [8. TroubleShooting](#8-troubleshooting) - [9. Conclusion](#9-conclusion) # 2. Installation ## 2.1. Starting the Production While in the contest-sustainability folder, open a terminal and enter : ``` docker-compose up ``` The very first time, it may take a few minutes to build the image correctly and install all the needed modules for Python. ## 2.2. Access the Production Following this link, access the production : [Access the Production](http://localhost:52795/csp/irisapp/EnsPortal.ProductionConfig.zen?RODUCTION=INFORMATION.QuickFixProduction) ## 2.3. Closing the Production ``` docker-compose down ``` # 3. How it works For now, some models may not work with this implementation since everything is automatically done, which means, no matter what model you input, we will try to make it work through `transformers` `pipeline` library. Pipeline is a powerful tool by the HuggingFace team that will scan the folder in which we downloaded the model, then understand what library it should use between PyTorch, Keras, Tensorflow or JAX to then load that model using `AutoModel`. From here, by inputting the task, the pipeline knows what to do with the model, tokenizer or even feature-extractor in this folder, and manage your input automatically, tokenize it, process it, pass it into the model, then give back the output in a decoded form usable directly by us. # 4. HuggingFace API Some people or some systems can not download models or use them due to some restriction, that's why it is possible to use the HuggingFace API and call some models directly throught this service. It is made easier for you here : You must first start the demo, using the green `Start` button or `Stop` and `Start` it again to apply your config changes. Then, by clicking on the operation `Python.HFOperation` of your choice, and selecting in the right tab `action`, you can `test` the demo. In this `test` window, select : Type of request : `Grongier.PEX.Message` For the `classname` you must enter : ``` msg.HFRequest ``` And for the `json`, here is an example of a call to GPT2 : ``` { "api_url":"https://api-inference.huggingface.co/models/gpt2", "payload":"Can you please let us know more details about your ", "api_key":"----------------------" } ``` Now you can click on `Visual Trace` to see in details what happened and see the logs. **NOTE** that you must have an API key from HuggingFace before using this Operation ( the api-keys are free, you just need to register to HF ) **NOTE** that you can change the url to try any other models from HuggingFace, you may need to change the payload. See as example:    # 5. Use any model from the web In the section we will teach you how to use almost any pre-trained model from the internet, HuggingFace or not in order to save some resource or simply use these model inside IRIS. ## 5.1. FIRST CASE : YOU HAVE YOUR OWN MODEL In this case, you must copy paste your model, with the config, the tokenizer.json etc inside a folder inside the model folder. Path : `src/model/yourmodelname/` From here you must create a new operation, call it as you wish, go to the parameters of this operation. Then go to `settings` in the right tab, then in the `Python` part, then in the `%settings` part. Here, you can enter or modify any parameters ( don't forget to press `apply` once your are done ). Here's the default configuration for this case : %settings ``` name=yourmodelname task=text-generation ``` **NOTE** that any settings that are not `name` or `model_url` will go into the PIPELINE settings. Now you can double-click on the operation and `start` it. You must see in the `Log` part the starting of your model. From here, we create a `PIPELINE` using transformers that uses your config file find in the folder as seen before. To call that pipeline, click on the operation, and select in the right tab `action`, you can `test` the demo. In this `test` window, select : Type of request : `Grongier.PEX.Message` For the `classname` you must enter : ``` msg.MLRequest ``` And for the `json`, you must enter every arguments needed by your model. Here is an example of a call to GPT2 : ``` { "text_inputs":"Unfortunately, the outcome", "max_length":100, "num_return_sequences":3 } ``` Click `Invoke Testing Service` and wait for the model to operate. See for example:  Now you can click on `Visual Trace` to see in details what happened and see the logs. See for example :   ## 5.2. SECOND CASE : YOU WANT TO DOWNLOAD A MODEL FROM HUGGINGFACE In this case, you must find the URL of the model on HuggingFace. Find a model that does what you are looking for and use it without spending resources and using the InterSystems technologies. ### 5.2.1. Settings From here you must go to the parameters of the `Hugging`. Click on the `HuggingFace` operation of your choice then go to `settings` in the right tab, then in the `Python` part, then in the `%settings` part. Here, you can enter or modify any parameters ( don't forget to press `apply` once your are done ). Here's some example configuration for some models we found on HuggingFace : %settings for gpt2 ``` model_url=https://huggingface.co/gpt2 name=gpt2 task=text-generation ``` %settings for camembert-ner ``` name=camembert-ner model_url=https://huggingface.co/Jean-Baptiste/camembert-ner task=ner aggregation_strategy=simple ``` %settings for bert-base-uncased ``` name=bert-base-uncased model_url=https://huggingface.co/bert-base-uncased task=fill-mask ``` %settings for detr-resnet-50 ``` name=detr-resnet-50 model_url=https://huggingface.co/facebook/detr-resnet-50 task=object-detection ``` %settings for detr-resnet-50-protnic ``` name=detr-resnet-50-panoptic model_url=https://huggingface.co/facebook/detr-resnet-50-panoptic task=image-segmentation ``` **NOTE** that any settings that are not `name` or `model_url` will go into the PIPELINE settings, so in our second example, the camembert-ner pipeline requirers an `aggregation_strategy` and a `task` that are specified here while the gpt2 requirers only a `task`. See as example:  Now you can double-click on the operation and `start` it. **You must see in the `Log` part the starting of your model and the downloading.** **NOTE** You can refresh those logs every x seconds to see the advancement with the downloads.  From here, we create a `PIPELINE` using transformers that uses your config file find in the folder as seen before. ### 5.2.2. Testing To call that pipeline, click on the operation , and select in the right tab `action`, you can `test` the demo. In this `test` window, select : Type of request : `Grongier.PEX.Message` For the `classname` you must enter : ``` msg.MLRequest ``` And for the `json`, you must enter every arguments needed by your model. Here is an example of a call to GPT2 : ``` { "text_inputs":"George Washington lived", "max_length":30, "num_return_sequences":3 } ``` Here is an example of a call to Camembert-ner : ``` { "inputs":"George Washington lived in washington" } ``` Here is an example of a call to bert-base-uncased : ``` { "inputs":"George Washington lived in [MASK]." } ``` Here is an example of a call to detr-resnet-50 using an online url : ``` { "url":"http://images.cocodataset.org/val2017/000000039769.jpg" } ``` Here is an example of a call to detr-resnet-50-panoptic using the url as a path: ``` { "url":"/irisdev/app/misc/000000039769.jpg" } ``` Click `Invoke Testing Service` and wait for the model to operate. Now you can click on `Visual Trace` to see in details what happened and see the logs. **NOTE** that once the model was downloaded once, the production won't download it again but get the cached files found at `src/model/TheModelName/`. If some files are missing, the Production will download them again. See as example:    See as example:   # 6. Fine-tune the models In this part we are trying to fine-tune a model in order to repurpose an existing model and make it even better without using too much resources. ## 6.1. Tunning the model ### 6.1.1. Download the model In order to use this GitHub you need to have a model from HuggingFace compatible with pipeline to use and train, and have a dataset you want to train your model on. In order to help, we let you the possibility to use the Python script in `src/utils/download_bert.py`. It will download for you the `"https://huggingface.co/bert-base-cased"` model and put inside the `src/model/bert-base-cased` folder if it's not already here. Moreover we also give you a DataSet to train the bert model on, this dataset was already loaded inside the IRIS DataBase and nothing else needs to be done if you want to use it. ( You can access it going to the SQL part of the portal and in the iris database namespace then to the review table ) To use the script, if you are inside the container, you can execute it without worry, if you are in local, you may need to `pip3 install requests` and `pip3 install beautifulsoup4` See the output :  ### 6.1.2. Settings If you want to use the bert-base-cased model, and you did downloaded it using the script, nothing needs to be added to the settings and you can advance to the [train the model part](#43-train-the-model). If you want to use your own model, click on the `Python.TuningOperation`, and select in the right tab `Settings`, then `Python` then in the `%settings` part, enter the path to the model, the name of the folder and the number of label you want it trained on. Example : ``` path=/irisdev/app/src/model/ model_name=bert-base-cased num_labels=5 ``` ### 6.1.3. Train the model To train the model you must go the `Production` following this link : ``` http://localhost:52795/csp/irisapp/EnsPortal.ProductionConfig.zen?PRODUCTION=iris.Production ``` And connect using : ```SuperUser``` as username and ```SYS``` as password. To call the training, click on the `Python.TuningOperation`, and select in the right tab `Actions`, you can `Test` the demo. In this test window, select : Type of request : Grongier.PEX.Message For the classname you must enter : ``` msg.TrainRequest ``` And for the json, you must enter every arguments needed by the trainer to train. Here is an example that train on the first 20 rows ( This isn't a proper training but it is fast ): ``` { "columns":"ReviewLabel,ReviewText", "table":"iris.Review", "limit":20, "p_of_train":0.8, "output_dir":"/irisdev/app/src/model/checkpoints", "evaluation_strategy":"steps", "learning_rate":0.01, "num_train_epochs":1 } ``` As for example :  As you can see, you must enter - `table` to use. - `columns` to use ( first is the `label`, second is the `input` to be tokenized ) - `limit` of rows to take in ( if you don't precise a number of rows, all the data will be used ) - `p_of_train` the percentage of training data to take from the dataset and `1 - p_of_train` the percentage of testing data to take from the dataset. After that, the other parameters are up to you and can be anything according to `https://huggingface.co/docs/transformers/main_classes/trainer` parameters. **NOTE** that the batch size for training and testing is automatically calculated if not input in the request. ( It's the biggest divider of the number of rows that's less than the square root of the number of row and less than 32 ) Click Invoke Testing Service and close the testing window without waiting. Now access the `Python.TuningOperation`, and select in the right tab `log` ; Here you can see the advancement of the training and evaluations. Once it is over, you will see a log saying that the new model was saved in a temporary folder. Now access the `Python.TuningOperation`, and select in the right tab `message` and select the last one by clicking on it's header. Here you can see the advancement of the training and evaluations and at the end you can have access to the Metrics of the old and the new model for you to compare. ### 6.1.4. Replace the model **If you want to keep the old model**, nothing must be done, the old one will stay on the non-temporary folder and is still loaded for further training. **If you want to keep the new model**, you must click on the `Python.TuningOperation`, and select in the right tab `Actions` and test. In this test window, select : Type of request : Grongier.PEX.Message For the classname you must enter : ``` msg.OverrideRequest ``` And for the json, empty brackets: ``` {} ``` Click Invoke Testing Service and see the response message. The new model was moved from the temporary folder to the non-temporary folder. ## 6.2. Use the model Training a model is interesting but you can also try it out. ### 6.2.1. Settings If you want to use the bert-base-cased model, and you did downloaded it using the script, nothing needs to be added to the settings and you can advance to the [test the model part](#52-test-the-model). If you want to use your own model, click on the `Python.TuningOperation`, and select in the right tab `Settings`, then `Python` then in the `%settings` part, enter the parameter to add to the pipeline. ### 6.2.2. Test the model To test the model you must go the `Production` following this link : ``` http://localhost:52795/csp/irisapp/EnsPortal.ProductionConfig.zen?PRODUCTION=iris.Production ``` And connect using : ```SuperUser``` as username and ```SYS``` as password. To call the testing, click on the `Python.MLOperation`, and select in the right tab `Actions`, you can `Test` the demo. In this test window, select : Type of request : Grongier.PEX.Message For the classname you must enter : ``` msg.MLRequest ``` And for the json, you must enter every arguments needed by the model to work ``` { "inputs":"This was a really bad experience" } ``` Press `Call test services` and then watch the result. # 7. Important note Fine-tuning models can take a LOT of time and resources, however it will always consume less resource than training a model from scratch. You can already see that it's taking a long time and a lot of computer power to fine-tune the model so imagine the cost if you had to train it from scratch and start over multiple times to get the right results. # 8. TroubleShooting If you have issues, reading is the first advice we can give you, most errors are easily understood just by reading the logs as almost all errors will be captured by a try / catch and logged. If you need to install a new module, or Python dependence, open a terminal inside the container and enter for example : "pip install new-module" To open a terminal there are many ways, - If you use the InterSystems plugins, you can click in the below bar in VSCode, the one looking like `docker:iris:52795[IRISAPP]` and select `Open Shell in Docker`. - In any local terminal enter : `docker-compose exec -it iris bash` - From Docker-Desktop find the IRIS container and click on `Open in terminal` Some models may require some changes for the pipeline or the settings for example, it is your task to add in the settings and in the request the right information. # 9. Conclusion From here you should be able to use some models that you may need on IRIS and fine-tune them as you wish. This template is supposed to be modified to suits your need and was created as a base for any AI and IRIS project that has sustainability and interoperability in mind. Due to the lack of time I was unable to add an API that was supposed to use a Director to directly communicate with the production and allow the users to make request to the models. However, if you are still interested in an IRIS API using this Python module you can check [my GitHub](https://github.com/LucasEnard/) or you can directly go to [my example of API in Python for IRIS](https://github.com/LucasEnard/iris-python-flask-api-template). Link to my DC profile : https://community.intersystems.com/user/lucas-enard-0 Best of luck for the contest ! Thank you so much!! Hi @Lucas.Enard2487! Thanks for such a great contribution to the community! I have a question: where can I obtain an API key for GPT2 to run the test: { "api_url":"https://api-inference.huggingface.co/models/gpt2", "payload":"Can you please let us know more details about your ", "api_key":"----------------------" } Hello, Thanks for trying my template. You need to register through the HuggingFace website and get an API key for free.

Hello and welcome to the Developer Ecosystem Summer News! This summer was full of exciting events and activities in the InterSystems Developer Ecosystem. In case you missed something, we've prepared for you a selection of the hottest news and topics to read! For your convenience, here we gathered everything worth noting that happened this last season at a glance. Read on! News 🎉 Incredible Milestones on Dev Community: 10K posts, 11K members, 5M reads 💡 InterSystems Ideas announced 🆕 Brand New "About Us" Page 🆕 DC rubric: "InterSystems Memes" (+part 1) 🆕 DC rubric: Key Questions ⛱ Inquisitive Summer on DC: ask questions - get double points ☕️ Dev Random Coffee on Global Masters: join our new networking activity 🔗 DC in French > < Global Masters API integration released Contests & Events 🔥 InterSystems Global Summit 2022 Main announcement Live from the Summit: part 1, part 2 InterSystems Developers at the Summit: how it was? Watch the Keynotes 🏆 InterSystems Full Stack Contest 2022 Rules & prizes Kick-off Webinar Winners announcement Meetup with winners 🏆 InterSystems Tech Article Contest: Python Edition Rules & prizes New contest bonuses introduced Winners announcements 🧩 Code of Golf challenge ⏯ Webinar organized by DC: Scaling InterSystems FHIR Server on Amazon Web Services with ECP 👩‍💻 Women's health - FemTech panel hosted by InterSystems 👋 InterSystems Developer Meetup on Python Interoperability in Boston Latest Releases ⬇️ InterSystems IRIS, IRIS for Health, & HealthShare Health Connect 2022.2 developer previews Preview 1 Preview 2 Preview 3 Preview 4 Preview 5 Preview 6 ⬇️ InterSystems IRIS, IRIS for Health, & HealthShare Health Connect 2022.1 ⬇️ InterSystems API Manager 2.8.1 🆕 Developer Community Release, July 2022 🆕 UX/UI change on Open Exchange 🆕 New Open Exchange feature: ObjectScript Quality status Best Practices & Key Questions 🔥 Best Practices of Summer 2022 DB Migration using SQLgateway SystemPerformance Utility (pka pButtons) API (and REST API) [and sample UI] Ensemble Orphaned Messages Formation on InterSystems' interoperability framework using ONLY Python Mastering the %SYSTEM.Encryption class Method to recompile classes and routines after a major upgrade ❓ Key Questions of Summer 2022: July, August People to Know About 👋 Meet new DC Moderator: Niangang Huang 🌟 Global Masters of Summer 2022: June, July, August Job Opportunities 💼 Sr. Application Development Analyst - The Ohio State University Wexner Medical Center 💼 InterSystems IRIS Developer - French Speaking - based in Paris - France 💼 Potential HealthShare opportunity 💼 Senior Software Developer - International Healthcare 💼 Looking for a Fully Remote InterSystems IRIS Developer So... Here is our take on the most interesting and important things! What were your highlights from this summer/winter? Share them in the comments section and let's remember the joy we've had! Proud of our Dev Ecosystem Team! Thanks, @Anastasia.Dyubaylo for sharing! My favorites are Random Coffee, New About page, and Key Questions! And of course the Memes! Who says that summer is a quiet time - not in the InterSystems Developer Community! hahaha))) indeed! Proud of our milestone achievements! Great numbers, aren't they? ;) For now: 🎉🎉🎉