Search

Hey Community, We're excited to announce that Community webinars are back! Let us invite you all to @Ron.Sweeney1582's webinar on Scaling InterSystems FHIR Server on Amazon Web Services with ECP. Join this webinar to make a technical deep dive, see a demonstration, and benchmark horizontal scaling InterSystems FHIR Server on Amazon Web Services with Enterprise Cache Protocol (ECP). ⏱ Date & Time: Thursday, August 18, 8 AM ET | 2:00 PM CEST👨‍🏫 Speaker: @Ron.Sweeney1582, Full Stack Architect at Integration Required About Integration Required: We are a full-stack technical delivery team for your InterSystems® workloads, tailored to meet the requirements of your security posture and organizational deployment standards. With a decade of InterSystems® partnerships and rigorous adherence to customer satisfaction, we are trusted to best practice. So... Don't miss this opportunity to learn more about scaling FHIR, ECP and AWS and how to mix it all! >> REGISTER HERE << Hi Devs, Don't miss the upcoming webinar with @Ron.Sweeney1582! Already 27 registered people. 😎 Registration continues >> click here << Hurry, only a week left! 🔥 Developers, The upcoming webinar will be tomorrow! Don't miss this opportunity to learn more about scaling FHIR, ECP and AWS and how to mix it all! For registration >> click here << Hey Community, The webinar will start in 15 mins! Please join us in Zoom. Or enjoy watching the live stream on YouTube: https://youtu.be/DBGvt0jd4yI Hi. I fell asleep before it started :) Is there a recording available? do we hv replay for this? Thx! Hi Michael, Anastasia mentioned rerun is at: https://youtu.be/DBGvt0jd4yI Hey everyone! The recording of this webinar is available on DC YouTube: ▶️ [Webinar] Scaling InterSystems FHIR Server on Amazon Web Services with ECP Big applause to our awesome speaker @Ron.Sweeney1582 👏👏👏 And thanks everyone for joining! Really enjoyed the webinar @Ron.Sweeney1582 Great content, insights and results, and thanks for the mentions

Have you ever needed to find a record for a particular person in your inbound data stream? Searching messages will enable you to find messages using an array of search capabilities.Searching Messages Using the Message ViewerSearching Messages Using the Message Viewer introduces the Message Viewer and details how to access it,and describes the process for modifying the message list display. Learn More. Nice little course, thanks!

Learn how to push an application to both an iOS and an Android device.Configuring a Mobile Device for Application DeploymentThis course discusses the process needed to push an application to both an iOS and Android device. Learners can choose which device they would like to learn about, or they can step through both to understand the settings and steps needed to push to both device types. Learn More.

Learn the basics about HealthShare Information Exchange, the architecture and common ways it is used.Find out how to perform a patient search, identify the main parts of HealthShare Information Exchange and the main purposes of each. Learn More.*This course is available to our HealthShare customers only.

Learn why and when to use FHIR in your applications, common use cases for it, the general architecture of FHIR data, and the tools available to you in InterSystems HealthShare Health Connect. Learn More. Thank you for the online course! Very much needed.The course videos below are not available. Getting an error "No playable video sources found":WHAT IS FHIR?FHIR VS. HL7V2, HL7V3, AND CDAFHIR ARCHITECTUREAlso, it would be nice to get access to the custom code show in "FHIR IN HEALTHSHARE DEMONSTRATION", specifically, "Summit.BPL.V2ToFHIR." I'm sure many developers would like to know how to convert good old HL7v2 messages to FHIR resources using HealthConnect. Video's are not available. We've re-uploaded the videos and are working again. Please try the course again and let us know if there any further problems.

Are you going to be building an integration with Ensemble or HealthShare? Take the following course to learn about the core architecture in building an integration, the parts and pieces involved and the most common ways that data flows through that architecture.Integration Architecture: Ensemble and HealthShareLearn about the basic architecture of InterSystems' solution for integration. This course is for people who have purchased either Ensemble and HealthShare. This course covers the main parts of integration, each part's function and how data flows through that architecture. Learn More.

Hi! Want to share with you code snippet of try catch block I usually use in methods which should return %Status. { try { $$$TOE(sc,StatusMethod()) } catch e { set sc=e.AsStatus() do e.Log() } Quit sc } Here $$$TOE is a short form of $$$TROWONERROR macro. Inside macro StatusMethod is any method you call which will return %Status value. This value will be placed into sc variable. In case of sc contains error execution will be routed to try catch block. You can wrap any Status methods calls in your code if you need to catch the errors coming from them. In try catch block I place my logic and have to mandatory calls: s sc=e.AsStatus() to get the status of error. D e.Log() - to place all the stack of error to standard Application Error log which you can find in Management portal on this page: http://localhost:57772/csp/sys/op/UtilSysAppErrorNamespaces.csp?$NAMESPACE=SAMPLES&Recent=1 How do you handle errors in your COS logic? I often do this too, I think this sort of pattern is a good way to make the error handling simple and consistent. I'm not really a fan of the status codes. They seem like a legacy item for the days prior to try/catches when traps were used. Having to maintain code to handle possible exceptions AND "errors" means more control flow to check and handle problems through the various levels of code.Say that code above is within a method that's within another method that's called from the main part of the program. Now, you need three levels of try/catches. That's a lot of extra lines of code (code smell), lost errors and performance degradation.Instead, I encourage my developers to use the fail fast methodgy. We use try/catches as high as possible in the code or when absolutely necessary, like when something is prone to exceptions but shouldn't stop the overall program from running. I agree with you. As mentioned it is an option if you need to use %Status as a method result. And what about e.Log() - do you use it to log errors or use something else? Status codes still have a place along side of Try/Catch in my opinion. They really only serve to indicate the ending state of the method called. This is not necessarily an error. I agree that throwing an exception for truly fatal errors is the best and most efficient error handling method. The issues is what does "truly fatal" mean? There can be a lot of grey area to contend with. There are methods where the calling program needs to determine the correct response. For example take a method that calculates commission on a sale. Clearly this is a serious problem on a Sales order. However, it is less of an issue on a quotation. In the case of the latter the process may simply desire to return an empty commissions structure.Placement of try/catch blocks is a separate conversation. Personally I find using try/catch blocks for error handling to be clean and efficient. The programs are easier to read and any recovery can be consolidated in one place, either in or right after the catch. I have found that any performance cost is unnoticeable in a typical transactional process. It surely beats adding IF statements to handle to handle the flow. For readability and maintainability I also dislike QUITing from a method or program in multiple places. So where is the "right" place for a try/catch? If I had to pick one general rule I would say you should put the try/catch in anyplace where a meaningful recovery from the error/exception can be done and as close to the point where the error occurred as possible. I the above example of a Commission calculation method I would not put a try/catch in the method itself since the method can not perform any real recovery. However I would put one in the Sales order and Quotation code.There are many methods to manage program flow under error/exception situations; Try/Catch, Quit and Continue in loops are a couple off the top of my head. Used appropriately they can create code that is robust, readable and maintainable with little cost in performance. I agree with both Nic and Rich.The issue with e.Log() is that it clutters up the error log with repetitive entries, each subsequent one with less detail than the prior. The same thing happens in Ensemble when errors bubble through the host jobs.The trick here is knowing when to log an error verses when to just bubble it up. Using Nic's method we lose the context of the stack since the log isn't written until the entry method with the Try/Catch. Using your method we get noise in the logs, but at least the first one has the detail we'll need.I believe the root problem here is re-throwing the status. An exception should represent something fatal and usually out of the applications control (e.g. FILEFULL) while a %Status is a call success indicator. To that end your code could be refactored to just Quit on an error instead of throwing it. That way a single log is generated in the method that triggered the exception and the stack is accurate.However this doesn't work well in nested loops. In that case a Return would work unless there is a cleanup chore (e.g. releasing locks, closing transactions, etc). I haven't come up with a pattern for nested loops that doesn't clutter up the source with a bunch of extra $$$ISERR checks that are easily missed and lead to subtle bugs.Personally I use your style without logging because:Every method uses the same control structureWorks with nested loops without extra thoughtCan ignore errors by simply invoking with Do instead of $$$TOE/$$$ThrowOnErrorCleanup code is easy to find or insertUsing ByRef/Output parameters makes it trivial to refactor to return more than one valueI do lose the ability to see an accurate stack trace but most of the time the line reference in the error is enough for me to quickly debug an issue so it is an acceptable trade-off. Only in the most trivial methods is the Try/Catch skipped.All that said Nic's style is a solid approach too. By removing a bunch of boilerplate Try/Catch code it lets the programmer focus on the logic and makes the codebase much easier on the eyes. We use various methods and macros to log exceptions, it just depends on the situation. I see where you are coming from with having a status code returned to notify a problem exists. Personally, I'd calculate comissions in a seperate single responsibility class that extends an abstract class for interface segregation. That class implements three public methods: Calculate(), HasError(), GetError()//calculate then commission: Set tCommission = tUsedCarCommission.Calculate()If tUsedCarCommission.HasError() //log the error or do something with it It's very similar to what you'd do with traditional status types but without having to deal with passing in references. Also, IMO, it's very clear what's going on if the commission has an error. OK. But how do you call methods which return %Status?Do you raise Error? Or you check Error status with if? Or do you ignore status at all? I will leave the logging issue alone as I don't see it as being the main point of the example. It could also be a thread by itself.The issue of using a bunch of $$$ISERR or other error condition checks is exactly why I like using throw with try/catch. I disagree that it should only be for errors outside of the application's control. However it is true that most of the time you are dealing with a fatal error. Fatal that is to the current unit of work being performed, not necessarily to the entire process.I will often use code likeset RetStatus = MyObj.Method() throw:$$$ISERR(RetStatus) ##class(%Exception.StatusException).CreateFromStatus(RetStatus)The post conditional on the throw can take many forms, this is just one example.Where I put the Try/Catch depends on many factors such as:Where do I want recovery to happen?use of the method and my code readability and maintainability of the code...I the case of nested loops mentioned I think this is a great way to abort the process and return to a point, whether in this program or one farther up the stack, where the process can be cleanly recovered or aborted. Most of my methods look like this: Method MyMethod() As %Status { // Initialization - note locking and transactions only when needed Set sc = $$$OK Lock +^SomeGlobal:5 If '$TEST Quit $$$ERROR($$$GeneralError,"Failed to obtain lock") Try { TSTART // When error is significant $$$ThrowOnError(..SomeMethod()) // When error can be ignored Do ..SomeMethod() // When only certain errors apply Set sc = ..SomeMethod() If $$$ISERR(sc),$SYSTEM.Status.GetErrorText(sc)["Something" $$$ThrowStatus(sc) TCOMMIT } Catch e { TROLLBACK Set sc = e.AsStatus() } Lock -^SomeGlobal Quit sc } Well, I think a major question is: What do you use to return runtime information to your caller when you implement your own code? Do you return a %Status object, or something similar, or do you throw exceptions and don't return anything.Most code snippets I have seen here make use of try/catch, but do return a status code itself. Personally, I prefer to use try/catch blocks and throw errors when I encounter issues at runtime. The try/catch philosophy is optimized for the case that everything goes well and exceptions are the, well, exception. Handling a status object is not as clean from a code maintenance perspective (more lines of code within your logic), but allows you to handle multiple different scenarios at once (it was okay/not okay, and this happened...)Obviously, this is my personal preference. What do you do when you need to cleanup things like locks? Put the unlock code both in the Catch and before the Quit? Well, that depends on where you did take the lock.In your previous example you take a lock right before the try block, so you can release it directly after the try/catch block.If you take a lock in your try block, you have to put the unlock code both in the catch block and at the end of the try block. I would not place the unlock code outside of the try/catch block. This is a case where a try/catch/finally construct would definitely help, as you could place the unlock code in the finally block. Other than locks, there are a few other cases where cleanup may be needed whether or not something goes wrong:Closing SQL cursors that have been openedEnsuring that the right IO device is in use and/or returning to the previous IO redirection state.There are probably more of these too.Here's the convention we use for error handling, logging, and reporting in InSync (a large Caché-based application):We have TSTART/TCOMMIT/TROLLBACK in a try/catch block at the highest level (typically a ClassMethod in a CSP/Zen page). There isn't much business logic in here; it'll call a method in a different package.If anything goes wrong in the business logic, an exception is thrown. The classes with the business logic don't have their own try/catch blocks unless it's needed to close SQL cursors, etc. in event of an exception. After the cleanup is done, the exception is re-thrown. (Unfortunately, this means that cleanup code may be duplicated between the try and catch blocks, but there's typically not too much duplication.) The classes with business logic also don't have their own TSTART/TCOMMIT/TROLLBACK commands, unless the business logic is a batch process in which parts of the process may fail and be corrected later without impacting the whole thing; such a case may also call for a nested try/catch to do the TROLLBACK if something goes wrong in part of the batch. In this case the error is recorded rather than re-throwing the exception.We have our own type of exception (extending %Exception.AbstractException), and macros to create exceptions of this type from:Error %Status codesError SQLCODEs and messagesSQLCODE = 100 can be treated as an error, "alert", or nothing.Other types of exceptionsExceptions of our custom type can also be created to represent a general application error not related to one of those things, either a fatal error, or something the user can/should fix - e.g., invalid data or missing configuration.The macros for throwing these exceptions also allow the developer to provide a localizable user-friendly message to explain what went wrong.When an exception is caught in the top level try/catch (or perhaps in a nested try/catch in a batch process), we have a macro that logs the exception and turns it into a user-friendly error message. This might just be a general message, like "An internal error occurred (log ID _______)" - the user should never see <UNDEFINED>, SQLCODE -124: DETAILS ABOUT SOME TABLE, etc.Our persistent classes may include an XDATA block with localizable error messages corresponding foreign and unique keys in the class and types of violations of those keys. For %Status codes and SQLCODEs corresponding to foreign/unique key violations, the user-friendly error message is determined based on this metadata.Logging for these exceptions is configurable; for example, exceptions representing something the user can/should fix are not logged by default, because they're not an error in the application itself. Also, the log level is configurable - it might be all the gory detail from LOG^%ETN, or just the stack trace. Typically, verbose logging would only be enabled system-wide briefly for specific debugging tasks. For SQL errors, the SQL statement itself is logged if possible.I thought this convention was too complicated when I first started working with it, but have come to see that it is very elegant. One possible downside is that it relies on a convention that any method in a particular package (InSyncCode, in our case) might throw an exception - if that isn't respected in the calling code, there's risk of a <THROW> error.I mentioned the InSync approach previously on https://community.intersystems.com/post/message-error-csppage . Unfortunately, it's coupled with several parts of the application, so it'd be quite a bit of work to extract and publish the generally-applicable parts. I'd like to do that at some point though. >$SYSTEM.Status.GetErrorText(sc)["Something" Would not always work correctly in applications with users requesting content in several languages (for example web app). Why not use error codes? If $SYSTEM.Status.GetErrorCodes(sc)[$$$GeneralError $$$ThrowStatus(sc) Timothy! Thanks for sharing this!It' can be a standalone post as "Error and resource handling in large Caché ObjectScript project".Thank you! Agreed - GetErrorCodes() is the right thing to do from an I18N perspective. For more advanced error analysis, such as conversion of error %Status-es into user-friendly messages (as I described in another comment), $System.Status.DecomposeStatus will provide the parameters of the error message as well. These are substituted in to the localizable string. For example, here's a foreign key violation message from %DeleteId on a system running in Spanish: INSYNC>Set tSC = ##class(Icon.DB.CT.TipoDocumento).%DeleteId(50) INSYNC>k tErrorInfo d $System.Status.DecomposeStatus(tSC,.tErrorInfo) zw tErrorInfo tErrorInfo=1 tErrorInfo(1)="ERROR #5831: Error de Foreign Key Constraint (Icon.DB.CC.AllowedGuaranteeTypes) sobre DELETE de objeto en Icon.DB.CT.TipoDocumento: Al menos existe 1 objeto con referencia a la clave CTTIPODOCUMENTOPK" tErrorInfo(1,"caller")="zFKTipoDocDelete+4^Icon.DB.CC.AllowedGuaranteeTypes.1" tErrorInfo(1,"code")=5831 tErrorInfo(1,"dcode")=5831 tErrorInfo(1,"domain")="%ObjectErrors" tErrorInfo(1,"namespace")="INSYNC" tErrorInfo(1,"param")=4 tErrorInfo(1,"param",1)="Icon.DB.CC.AllowedGuaranteeTypes" tErrorInfo(1,"param",2)="Icon.DB.CT.TipoDocumento" tErrorInfo(1,"param",3)="DELETE" tErrorInfo(1,"param",4)="CTTIPODOCUMENTOPK" tErrorInfo(1,"stack")=... The "param" array allows clean programmatic access to the details of the foreign key violation, independent of language. Of course, these level of detail in these error messages may be subject to change across Caché versions, so this is a *great* thing to cover with unit tests if your application relies on it. I prefer doing this more correctly using the right API for matching status values: If $system.Status.Equals(sc,$$$ERRORCODE($$$GeneralError),$$$ERRORCODE($$$MoreSpecificStatusCode),...) { // Handle specific error(s) } The use of `'` can result in unexpected behaviour when you are checking for a 4 digit code, but are handling a 3 digit code... Note that it's also safer to wrap status codes in `$$$ERRORCODE($$$TheErrorCode)`, but that may not be necessary depending on the specific context. A rather subtle point that I haven't seen discussed here is actually about how TSTART/TCOMMIT/TROLLBACK should be handled when triggering a TROLLBACK from code that may be part of a nested transaction. Given that a lot of the code I write may be called from various contexts and those calling contexts may already have open transactions, my preferred transaction pattern is as follows: Method SaveSomething() As %Status { Set tStatus = $$$OK Set tInitTLevel = $TLevel Try { // Validate input before opening transaction so you can exit before incurring any major overhead TSTART // Do thing 1 // Do thing 2 // Check status or throw exception TCOMMIT } // Handle exception locally due to local cleanup needs Catch ex { Set tStatus = ex.AsStatus() } While ($TLevel > tInitTLevel) { // Only roll back one transaction level as you make it possible for the caller to decide whether the whole transaction should be rolled back TRollback 1 } Quit tStatus } (Edited for formatting.) What's the advantage of using $$$ERRORCODE macro? ^%qCacheObjectErrors global contains the same values.

Announcing the Custom Business Components learning path! This learning path is designed for software developers who need to build custom business components for their productions.The learning path includes the following courses: 1. Building Custom Ensemble Messages2. Building Custom Business Operations3. Building BPL Business Processes4. Building Custom Business Services 5. Coming Soon: Building Custom Business ServicesAccess the learning path. We're really excited about this learning path as its something many of you have asked for! Have you taken the Building and Managing HL7 Productions classroom course but now need to create custom components? This is a great next step! As always, please let us know if you have other suggestions for courses, learning paths, or best practices that we can incorporate in future learning content.

Hello all! As we ObjectScript developers have been experiencing, preparing an environment to run CI related tasks can be quite the chore. This is why I have been thinking about how we could improve this workflow and the result of that effort is [IRIS-CI](https://openexchange.intersystems.com/package/iris-ci). See how it works [here](https://imgur.com/N7uVDNK). ### Quickstart 1.Download the image from the Docker Hub registry: ``` docker pull rfns/iris-ci:0.5.3 ``` 2. Run the container (with the default settings): ``` docker run --rm --name ci -t -v /path/to/your/app:/opt/ci/app rfns/iris-ci:0.5.3 ``` Notice that volume mounting to `/path/to/your/app?` This is where the app should be. And that's it: the only thing required to start running the test suites is the path of the application. Also, since this is supposed to be a ephemeral and run-once container, there's no need to keep it listed after executing it, that's why there's the `--rm` flag. ### TL;DR; If you want an example on how to how use it: Check the usage with my another project [dotenv](https://github.com/rfns/dotenv/blob/master/.github/workflows/ci.yml). ### Advanced setup Some projects might need sophisticated setups in order to run the test suites, for such circunstances there's two customization levels: 1. Environment variables 2. Volume overwrite ### Environment variables Environment variables are the most simple customization format and should suffice for most situations. There's two ways to provide an environment variable: * `-e VAR_NAME="var value"` while using `docker run`. * By providing a .env file by mounting an extra volume for `docker run` like this: `-v /my/app/.env:/opt/ci/.env`. > NOTE: In case a variable is defined in both formats, using the `-e` format takes precedence over using a `.env` file. ### Types of environment variables * Variables prefixed with `CI_{NAME}` are passed down as `name` to the installer manifest. * Variables prefixed with `TESPARAM_{NAME}`are passed down as `NAME` to the unit test manager's UserFields property. * `TEST_SUITE`and `TEST_CASE` to control where to locate and which test case to target. Every variable is available to read from the `configuration.Envs` list, which is [passed](https://github.com/rfns/iris-ci/blob/master/ci/Runner.cls#L6) [down](https://github.com/rfns/iris-ci/blob/master/ci/Runner.cls#L53) through `Run` and `OnAfterRun` class methods. If `TEST_CASE` is not specified then the `recursive` flag will be set. In a project with many classes it might be interesting to at least define the `TEST_SUITE` and reduce the search scope due to performance concerns. ### Volume overwrite This image ships with a default installer that's focused on running test suites. But it's possbile to overwrite the following files in order to make it execute different tasks like: generating a XML file for old Caché versions. * `/opt/ci/App/Installer.cls` * `/opt/ci/Runner.cls` For more details on how to implement them, please check the default implementations: [Installer.cls](https://github.com/rfns/iris-ci/blob/master/ci/App/Installer.cls) [Runner.cls](https://github.com/rfns/iris-ci/blob/master/ci/Runner.cls) > TIP: Before overwriting the default Installer.cls check if you really need to, because the current implementation [also allows to create configurated web applications.](https://github.com/rfns/iris-ci#using-the-default-installer-manifest-for-unit-tests) EDIT: Link added.

Hello Developer Community! We are looking to better understand how our users use the Documentation. If you have a few minutes, please fill out this quick survey - https://www.surveymonkey.com/r/HK7F5P7! Feedback from real users like you in invaluable to us and helps us create better product. Your feedback can go further than the survey - we would love to interview you about your experience, just indicate in the survey that you’re open to talking to us! Thank you so much! If you have any questions, please contact me at Ksenia.samokhvalova@intersystems.com I look forward to hearing from you! Ksenia Ksenia SamokhvalovaUX Designer | InterSystemsKsenia.samokhvalova@intersystems.com

Hi Community, We're pleased to invite you to join the upcoming InterSystems IRIS 2020.1 Tech Talk: Integrated Development Environments on May 19 at 10:00 AM EDT! In this edition of InterSystems IRIS 2020.1 Tech Talks, we put the spotlight on Integrated Development Environments (IDEs). We'll talk about InterSystems latest initiative with the open source ObjectScript extension to Visual Studio Code, discussing what workflows are particularly suited to this IDE, how development, support, and enhancement requests will work in an open source ecosystem, and more. Speakers:🗣 @Raj.Singh5479, InterSystems Product Manager, Developer Experience🗣 @Brett.Saviano, InterSystems Developer Date: Tuesday, May 19, 2020Time: 10:00 AM EDT ➡️ JOIN THE TECH TALK! Additional Resources: ObjectScript IDEs [Documentation] Using InterSystems IDEs [Learning Course] Hi Community!Join the Tech Talk today. 😉 ➡️ You still have time to REGISTER.

Hi Community, As you may know, on Global Masters you can redeem a consultation with InterSystems expert on any InterSystems product: InterSystems IRIS, IRIS for Health, Interoperability (Ensemble), IRIS Analytics (DeepSee), Caché, HealthShare.And we have exciting news for you: now these consultations available in the following languages: English, Portuguese, Russian, German, French, Italian, Spanish, Japanese, Chinese. Also! The duration is extended to 1.5 hours for your deep dive into the topic. If you are interested, don't hesitate to redeem the reward on Global Masters! If you are not a member of Global Masters yet - you are very welcome to join here (click on the InterSystems login button and use your InterSystems WRC credentials). To learn more about Global Masters read this article: Global Masters Advocate Hub - Start Here! See you on InterSystems Global Masters today! 🙂

Introduction In a previous article, I discussed patterns for running unit tests via the InterSystems Package Manager. This article goes a step further, using GitHub actions to drive test execution and reporting. The motivating use case is running CI for one of my Open Exchange projects, AppS.REST (see the introductory article for it here). You can see the full implementation from which the snippets in this article were taken on GitHub; it could easily serve as a template for running CI for other projects using the ObjectScript package manager. Features demonstrated implementation include: Building and testing an ObjectScript package Reporting test coverage measurement (using the TestCoverage package) via codecov.io Uploading a report on test results as a build artifact The Build Environment There's comprehensive documentation on GitHub actions here; for purposes of this article, we'll just explore the aspects demonstrated in this example. A workflow in GitHub actions is triggered by a configurable set of events, and consists of a number of jobs that can run sequentially or in parallel. Each job has a set of steps - we'll go into the details of the steps for our example action in a bit. These steps consist of references to actions available on GitHub, or may just be shell commands. A snippet of the initial boilerplate in our example looks like: # Continuous integration workflow name: CI # Controls when the action will run. Triggers the workflow on push or pull request # events in all branches on: [push, pull_request] # A workflow run is made up of one or more jobs that can run sequentially or in parallel jobs: # This workflow contains a single job called "build" build: # The type of runner that the job will run on runs-on: ubuntu-latest env: # Environment variables usable throughout the "build" job, e.g. in OS-level commands package: apps.rest container_image: intersystemsdc/iris-community:2019.4.0.383.0-zpm # More of these will be discussed later... # Steps represent a sequence of tasks that will be executed as part of the job steps: # These will be shown later... For this example, there are a number of environment variables in use. To apply this example to other packages using the ObjectScript Package Manager, many of these wouldn't need to change at all, though some would. env: # ** FOR GENERAL USE, LIKELY NEED TO CHANGE: ** package: apps.rest container_image: intersystemsdc/iris-community:2019.4.0.383.0-zpm # ** FOR GENERAL USE, MAY NEED TO CHANGE: ** build_flags: -dev -verbose # Load in -dev mode to get unit test code preloaded test_package: UnitTest # ** FOR GENERAL USE, SHOULD NOT NEED TO CHANGE: ** instance: iris # Note: test_reports value is duplicated in test_flags environment variable test_reports: test-reports test_flags: >- -verbose -DUnitTest.ManagerClass=TestCoverage.Manager -DUnitTest.JUnitOutput=/test-reports/junit.xml -DUnitTest.FailuresAreFatal=1 -DUnitTest.Manager=TestCoverage.Manager -DUnitTest.UserParam.CoverageReportClass=TestCoverage.Report.Cobertura.ReportGenerator -DUnitTest.UserParam.CoverageReportFile=/source/coverage.xml If you want to adapt this to your own package, just drop in your own package name and preferred container image (must include zpm - see https://hub.docker.com/r/intersystemsdc/iris-community). You might also want to change the unit test package to match your own package's convention (if you need to load and compile unit tests before running them to deal with any load/compile dependencies; I had some weird issues specific to the unit tests for this package, so it might not even be relevant in other cases). The instance name and test_reports directory shouldn't need to be modified for other use, and the test_flags provide a good set of defaults - these support having unit test failures flag the build as failing, and also handle export of jUnit-formatted test results and a code coverage report. Build Steps Checking out GitHub Repositories In our motivating example, two repositories need to be checked out - the one being tested, and also my fork of Forgery (because the unit tests need it). # Checks out this repository under $GITHUB_WORKSPACE, so your job can access it - uses: actions/checkout@v2 # Also need to check out timleavitt/forgery until the official version installable via ZPM - uses: actions/checkout@v2 with: repository: timleavitt/forgery path: forgery $GITHUB_WORKSPACE is a very important environment variable, representing the root directory where all of this runs. From a permissions perspective, you can do pretty much whatever you want within that directory; elsewhere, you may run in to issues. Running the InterSystems IRIS Container After setting up a directory where we'll end up putting our test result reports, we'll run the InterSystems IRIS Community Edition (+ZPM) container for our build. - name: Run Container run: | # Create test_reports directory to share test results before running container mkdir $test_reports chmod 777 $test_reports # Run InterSystems IRIS instance docker pull $container_image docker run -d -h $instance --name $instance -v $GITHUB_WORKSPACE:/source -v $GITHUB_WORKSPACE/$test_reports:/$test_reports --init $container_image echo halt > wait # Wait for instance to be ready until docker exec --interactive $instance iris session $instance < wait; do sleep 1; done There are two volumes shared with the container - the GitHub workspace (so that the code can be loaded; we'll also report test coverage info back to there), and a separate directory where we'll put the jUnit test results. After "docker run" finishes, that doesn't mean the instance is fully started and ready to command yet. To wait for the instance to be ready, we'll keep trying to run a "halt" command via iris session; this will fail and continue trying once per second until it (eventually) succeeds, indicating that the instance is ready. Installing test-related libraries For our motivating use case, we'll be using two other libraries for testing - TestCoverage and Forgery. TestCoverage can be installed directly via the Community Package Manager; Forgery (currently) needs to be loaded via zpm "load"; but both approaches are valid. - name: Install TestCoverage run: | echo "zpm \"install testcoverage\":1:1" > install-testcoverage docker exec --interactive $instance iris session $instance -B < install-testcoverage # Workaround for permissions issues in TestCoverage (creating directory for source export) chmod 777 $GITHUB_WORKSPACE - name: Install Forgery run: | echo "zpm \"load /source/forgery\":1:1" > load-forgery docker exec --interactive $instance iris session $instance -B < load-forgery The general approach is to write out commands to a file, then run then in IRIS session. The extra ":1:1" in the ZPM commands indicates that the command should exit the process with an error code if an error occurs, and halt at the end if no errors occur; this means that if an error occurs, it will be reported as a failed build step, and we don't need to add a "halt" command at the end of each file. Building and Testing the Package Finally, we can actually build and run tests for our package. This is pretty simple - note use of the $build_flags/$test_flags environment variables we defined earlier. # Runs a set of commands using the runners shell - name: Build and Test run: | # Run build echo "zpm \"load /source $build_flags\":1:1" > build # Test package is compiled first as a workaround for some dependency issues. echo "do \$System.OBJ.CompilePackage(\"$test_package\",\"ckd\") " > test # Run tests echo "zpm \"$package test -only $test_flags\":1:1" >> test docker exec --interactive $instance iris session $instance -B < build && docker exec --interactive $instance iris session $instance -B < test && bash <(curl -s https://codecov.io/bash) This follows the same pattern we've seen, writing out commands to a file then using that file as input to iris session. The last part of the last line uploads code coverage results to codecov.io. Super easy! Uploading Unit Test Results Suppose a unit test fails. It'd be really annoying to have to go back through the build log to find out what went wrong, though this may still provide useful context. To make life easier, we can upload our jUnit-formatted results and even run a third-party program to turn them into a pretty HTML report. # Generate and Upload HTML xUnit report - name: XUnit Viewer id: xunit-viewer uses: AutoModality/action-xunit-viewer@v1 if: always() with: # With -DUnitTest.FailuresAreFatal=1 a failed unit test will fail the build before this point. # This action would otherwise misinterpret our xUnit style output and fail the build even if # all tests passed. fail: false - name: Attach the report uses: actions/upload-artifact@v1 if: always() with: name: ${{ steps.xunit-viewer.outputs.report-name }} path: ${{ steps.xunit-viewer.outputs.report-dir }} This is mostly taken from the readme at https://github.com/AutoModality/action-xunit-viewer. The End Result If you want to see the results of this workflow, check out: Logs for the CI job on intersystems/apps-rest (including build artifacts): https://github.com/intersystems/apps-rest/actions?query=workflow%3ACITest coverage reports: https://codecov.io/gh/intersystems/apps-rest Please let me know if you have any questions!

Hi Developers,A new exciting challenge introduced for Global Masters members of "Advocate" level and above: we invite you to record a 30-60 sec video with an answer to our question: ➥ What is the value of InterSystems IRIS to you? 🎁 Reward of your choice for doing the interview: $50 Gift Card (VISA/Amazon) or 12,000 points! Follow this direct link to the challenge for more information. Please note that the link will work for GM members of "Advocate" level and above. More about GM levels you can read here. We would love to hear from you! See you on the Global Masters Advocate Hub today!

We've been tasked with developing a file upload module as part of our wider system, storing scanned documents against a patients profile. Our Intersystems manager suggested storing those files in the DB as streams would be the best approach and it sounded like a solid idea, it can be encrypted, complex indexes, optimized for large files and so on. However the stake holder questioned why would we want to do that over storing them in windows folders and that putting it in the DB was nuts. So we were wondering what everyone else has done in this situation and what made them take that route. The nice advantage of storing them in the DB is that is makes the following easier: - refreshing earlier environments for testing- mirroring the file contents- encryption- simpler consistent backups However, if you're talking about hundreds of GBs of data, then you can run into issues which you should weigh against the above: - journaling volume- .dat size- .dat restore time One way to help mitigate the above for larger volume file management is to map the classes that are storing the the stream properties into their own .DAT so they can be managed separately from other application data, and then you can even use subscript level mapping to cap the size of the file .DATs. Hope that helps I can't disagree with Ben, there is a cut-off point where it makes more sense to store the files external to IRIS however it should be noted that if I was working with any other database technology such as Oracle or SQL Server I wouldn't even consider storing 'Blobs' in the database. However Cache/Ensemble/IRIS is extremely efficient at storing stream data especially binary steams. I agree with Ben that by storing the files in the database you will have the benefits of Journallng and Backups which support 24/7 up time. If you are using Mirroring as part of your Disaster Recovery strategy then restoring your system will be faster. If you store the files externally you will need to back up the files as a separate process from Cache/Ensemble/IRIS backups. I assume that you would have a seperate file server as you wouldn't want to keep the external files on the same server as your Cach/Ensemble/IRIS server for two reasons: 1) You would not want the files to be stored on the same disk as your database .dat files as the disk I/O might be compromised 2) If your database server crashes you may lose the external files unless they are are on separate server. 3) You would have to backup your file server to another server or suitable media 4) If the steam data is stored in IRIS then you can use iFind and iKnow on the file content which leads you into the realms of ML, NLP and AI 5) If your Cache.dat files and the External files are sored on the same disk system you potentially run into disk fragmentation issues over time and the system will get slower as the fragmentation gets worse. Far better to have your Cache.dat files on a disk system of their own where the database growth factor is set quite high but the database growth will be contiguous and fragmentation is considerably reduced and the stream data will be managed as effectively as any other global structure in Cache/Ensemble/IRIS. Yours Nigel Fragmentations issues, with SSD disks not an issue anymore. But in any way, I agree with storing files in the database. I have a system in production, where we have about 100TB of data, while more than half is just for files, stored in the database. Some of our .dat files by mapping used exclusively for streams, and we take care of them, periodically by cutting them at some point, to continue with an empty database. Mirroring, helps us do not to worry too much about backups. But If would have to store such amount of files as files on the filesystem, we would lose our mind, caring about backups and integrity. Great data point! Thanks @Dmitry.Maslennikov :) I'm throwing in another vote for streams for all the reasons in the above reply chain, plus two more: 1. More efficient hard drive usage. If you have a ton of tiny files and your hard drive is formatted with a larger allocation unit, you're going to use a lot of space very inefficiently and very quickly. 2. At my previous job, we got hit by ransomware years ago that encrypted every document on our network. (Fortunately, we had a small amount of data and good offline backup process, so we were able to recover fairly quickly!) We were also using a document management solution that ran on Cache and stored the files as Stream objects, and they were left untouched. I'm obviously not going to say streams and ransomewareproof, but that extra layer of security can't hurt! Thank you all for your input, they're all sound reasoning that I can agree with. It's not a good idea to store files in the DB that you'll simply be reading back in full. The main issue you'll suffer from if you do hold them in the database (which nobody else seems to have picked up on) is that you'll needlessly flush/replace global buffers every time you read them back (the bigger the files, the worse this will be). Global buffers are one of the keys to performance. Save the files and files and use the database to store their filepaths as data and indices. Hi Rob, what factors play a part in this though, we'd only be retrieving a single file at a time (per user session obviously) and the boxes have around 96gb-128gb memory each (2 app, 2 db) if that has any effect on your answer? I've mentioned above a system with a significant amount of streams stored in the database. And just checked how global buffers used there. And streams are just around 6%. The system is very active, including files. Tons of objects created every minute, attached files, changes in files (yeah, our users can change MS Word files online on the fly, and we keep all the versions). So, I still see no reasons to change it. And still, see tons of benefits, of keeping it as is. Hey Matthew, No technical suggestions from me, but I would say that there are pros/cons to file / global streams which have been covered quite well by the other commenters. For the performance concern in particular, it is difficult to compare different environments and use patterns. It might be helpful to test using file / global streams and see how the performance for your expected stream usage, combined with your system activity, plays into your decision to go with one or the other. I agree, for our own trust we'll most likely go with Stream. However I've suggested we plan to build both options for customers but we'll just reference the links to files and then they can implement back up etc as they see fit. Great! This was an interesting topic and I'm sure one that will help future viewers of the community. There are a lot of considerations. Questions: Can you describe what are you going to do with that streams (or files I guess)? Are they immutable? Are they text or binary? Are they already encrypted or zipped? Average stream size? Global buffers are one of the keys to performance. Yes, that's why if streams are to be stored in the db they should be stored in a separate db with distinct block size and separate global buffers. Having multiple different global buffers for different block sizes, does not make sense. IRIS will use bigger size of block for lower size blocks inefficiently. The only way to separate is, to use a separate server, right for streams. For us it will be scanned documents (to create a more complete picture of a patients record in one place) so we can estimate a few of the constants involved to test how it will perform under load. I'm not sure what you mean by this. On an IRIS instance configured with global buffers of different sizes, the different sized buffers are organized into sperate pools. Each database is assigned to a pool based on what is the smallest size available that can handle that database. If a system is configured with 8KB and 32KB buffers, the 32KB buffers could be assigned to handle 16KB database or 32KB databases but never 8KB databases. It depends. I would prefer to store the files in the linux filesystem with a directory structure based on a hash of the file and only store the meta-information (like filename, size, hash, path, author, title, etc) in the database. In my humble opinion this has the following advantages over storing the files in the database: The restore process of a single file will run shorter than the restore of a complete database with all files. Using a version control (f.e. svn or git) for the files is possible with a history. Bitrot will only destroy single files. This should be no problem if a filesystem with integrated checksums (f.e. btrfs) is used. Only a webserver and no database is needed to serve the files. You can move the files behind a proxy or a loadbalancer to increase availability without having to use a HA-Setup of Caché/IRIS. better usage of filesystem cache. better support for rsync. better support for incremental/differential backup. But the pros and cons may vary depending on the size and amount of files and your server setup. I suggest to build two PoCs, load a reasonable amount of files in each one and do some benchmarks to get some figures about the performance and to test some DR- and restore-scenarios. Jeffrey, thanks. But if I would have only 16KB blocks buffer configured and with a mix of databases 8KB (mostly system or CACHETEMP/IRISTEMP) and some of my application data stored in 16KB blocks. 8KB databases in any way will get buffered in 16KB Buffer, and they will be stored one to one, 8KB data in 16KB buffer. That's correct? So, If I would need to separate global buffers for streams, I'll just need the separate from any other data block size and a significantly small amount of global buffer for this size of the block and it will be enough for more efficient usage of global buffer? At least for non-stream data, with a higher priority? Yes, if you have only 16KB buffers configured and both 8KB and 16KB databases, then the 16KB buffers will be used to hold 8KB blocks - one 8KB block stored in one 16KB buffer using only 1/2 the space... If you allocate both 8KB and 16KB buffers then (for better or worse) you get to control the buffer allocation between the 8KB and 16KB databases. I'm just suggesting that this is an alternative to standing up a 2nd server to handle streams stored in a database with a different block size. One more consideration for whether to store the files inside the database or not is how much space gets wasted due to the block size. Files stored in the filesystem get their size rounded up to the block size of the device. For Linux this tends to be around 512 bytes (blockdev --getbsz /dev/...). Files stored in the database as streams are probably* stored using "big string blocks". Depending on how large the streams are, the total space consumed (used+unused) may be higher when stored in a database. ^REPAIR will show you the organization of a data block. *This assumes that the streams are large enough to be stored as big string blocks - if the streams are small and are stored in the data block, then there will probably be little wasted space per block as multiple streams can be packed into a single data block. Some info about blocks, in this article and others in cycle. In my opinion, it is much better & faster to store binary files out of the database. I have an application with hundreds of thousands of images. To get a faster access on a Windows O/S they are stored in a YYMM folders (to prevent having too many files in 1 folder that might slow the access) while the file path & file name are stored of course inside the database for quick access (using indices). As those images are being read a lot of times, I did not want to "waste" the "cache buffers" on those readings, hence storing them outside the database was the perfect solution. Hi, I keep everything I need in Windows folders, I'm very comfortable, I have everything organized. But maybe what you suggest won't look bad and be decent in terms of convenience! it depends on the file type , content, use frequency and so on , each way has its advantage