Hi!

The method %FileIndices(id) may help you. The problem is that you won't know which ID you must use to call it. Have new objects been added? Which have been changed?

I assume you have taken an old style global and mapped it to classes and have created an index for it. That's ok. Normally old style applications will have its own indices on the same global or another. They probably have an index on Name already that they use and you should try to map that existing subscript as your name index on your mapped class instead of creating a new index.

But if you really want to go on and create a new index, you have only two choices:

1. To change the old style code to use objects/sql and stop setting the globals directly for that specific entity
2. To change the old style code to set the new index global together with the data global (this is very common practice on old style applications).
3. Ask them (the programmers of the old style application) to call %FileIndices(id) themselves after adding/changing a record so that all the indices defined on your mapped class would be (re)computed (including the indices that they have already set manually and you have mapped on your class). You could offer them to call this method to populate the indices global for them instead of setting the indices globals themselves to eliminate the redundant work. But you would have to have your mappings very well defined to replace their code, with all their indices.

As you can see, all options imply changing the old style code. I would push for option number 1 as this can be the start of some real good improvements to your old style application. 

There is no middle ground here, unless you want to rebuild all your indices every day.

Depending on how many places this global is referred on your application, the way the application is compartmentalized, and if you are using indirection or not throughout your application, these changes can be very simple or very complex to do and test.

Another way of going about this is to try to use the journals APIs to read the journal files, detect changes on this specific global and update the index... That would prevent you from touching the old style application. I have never done this before, but I believe it is pretty feasible.

Kind regards,

AS

Hi!

I wouldn't recommend putting globals or routines on your %SYS namespace for two reasons:

• This will be shared with all namespaces. This may be what you want today, but may not be what you want in the future.
• One nice thing to organize your code, globals and classes on different databases is that you can simplify your patching procedures. Instead of shipping classes, globals and routines from DEV to TEST or LIVE you can ship the necessary databases. So, if you have a database for all your routines/classes/compiled_pages, and you have a new release of your application, you can copy the entire CODE database from DEV to TEST or LIVE and "upgrade" that environment. Of course upgrading an environment may involve other things like rebuilding some indices, running some patching code, etc. But again, you can backup your databases, move the new databases to replace the old ones, run your custom code and rebuild your indices (if necessary) and test it. If something fails, you can always revert back to the previous set of databases. Well, you can't do that if you have custom code and globals on %SYS since copying the CACHESYS database from one system to another is taking more than what you would like to take from that system to the other.
• With the new InterSystems Container Manager (ICM) that comes with InterSystems IRIS, there is this concept of "durable %SYS" where %SYS namespace is extracted from inside the container and put into an external volume. That is good because you can replace your container without loose all the system configurations you have stored on %SYS. But IRIS is not meant to be compatible with old practices such as taking care of your custom globals, routines and classes on %SYS. InterSystems may decide to change this behavior and make %SYS less custom code friendly.

So, you can perfectly create more databases to organize your libraries and globals and map them to your namespace without problems. One database for code tables, one database for some libraries you share between all your namespaces, one database for CODE another for DATA, etc. That is what namespaces are for: To rule them all.

Kind regards,

AS

Hi Robert,

You are right. Now I see there are methods on the datatype for LogicalToODBC and ODBCToLogical. 

But I insist that the methods LogicalTo*, ODBCTo* and DisplayTo* should not be called directly. Although they will work, the correct way of dealing with data type conversions are using normal functions such as $ZDateH() with the proper formatting code.

If one wants to store names in a format where the surname should be separated from the name by a comma, I would instead simply use %String for that or subclass it to create a new datatype that will make sure there is a comma in there (although I think this is too much).

Kind regards,

AS

Hi Mike,

%List is not supposed to be used that way. It doesn't have a projection to SQL, so you can't create a property on a persistent class with it and expect it to work as a %String or %Integer property will. You could have it there as a private property for some other purpose of your class but not to be used as any other property as you expect.

I think %List was created to be used on the definition of method parameters and return types of such methods to represent a $List parameter. Then one would know that such datatype must be used and this would also be used when projecting those methods to Java, .NET, etc. for the same reason (there is a Java/.NET/etc. $List like datatype for every language binding we support).

If you need to represent names in a format such as "Surname,Name", try using %String or creating another datatype that inherits from %String and validates that the string is in such format by implementing the IsValid() method of your datatype.

Also, don't try to use LogicalTo*, ODBCTo* or DisplayTo* methods directly. Those are called by the SQL and Object engine accordingly. For instance ODBCToLogical is called when storing data from ODBC/JDBC connections. You shouldn't have to call these methods. 

Kind regards,

AS

I use GitHub's issues that allows you to create issues of all sorts (bugs, enhancements, tasks, etc.) and assign them to someone or to a group of people. The issues are associated with the source code repository and there is also a functionality for projects. You create a project and then list the tasks needed to accomplish that project. Then you can make every task an issue and assign it to people. Then you can drag and drop the tasks from one stage to the next like Specification > Development > Testing > Product.

GitFlow is a good and flexible workflow. But I don't use Git to deploy on Pre-Live or LIVE. I normally would have four environments:

• Development - Your machine
• Development - where you integrate the development branch from Git with the work from all developers. Downloading the code from GitHub can be done automatically (when a change is integrated back into the develop branch) or manually.
• QA - This environment is where you download code from GitHub's master branch with a new release. Your users can test the new release here without being bothered. 
• Pre-Production/Pre-LIVE - This environment is periodically overwritten with a copy of LIVE. It is where you try and test applying your new release.
• Production

GitFlow's hotfix may be used depending on your evironment. Depending on the change and on the urgency, it will be a pain to actually test the fix on your develop machine. Your local globals may not match the storage definition of what is in production because you may have been working on a new version of your classes with different global structures. You may need larges amount of data or specific data to reproduce the problem on your developer machine, etc. You can do it, but every hot fix will be a different workflow. Depending on the urgency you may simply not have the time to prepare your develop environment with the data and conditions to reproduce the problem, fix it and produce the hot fix. But it can be done.  On the other hand, as pre-production is a copy of LIVE, you can safely fix the problem there manually (forget GitHub), apply the change to LIVE and then incorporate these changes into your next release. I think this is cleaner. Everytime you have problem in LIVE, you can investigate it on PRE-LIVE. If PRE-LIVE is outdated, you can ask Operations for an emergency unscheduled refresh of PRE-LIVE to work on it.

About patching

I recommend always creating your namespaces with two databases: One for CODE and another for DATA.

That allows you to implement patching with a simple copy of the CODE database. You stop the instance, copy the database and start the instance. Simple like that. Every release may have an associated Class that has code to be run to rebuild indices, fix some global structures that may have changed, do some other kind of maintenance, etc.

If you are using Ensemble and don't want to stop your instance, you can generate your patch as a normal XML package and a Word document explaining how to apply it. Test this on your Pre-LIVE environment. Fix the document and/or the XML package if necessary, and try again until patching works. The run it on LIVE.

Before patching applying new releases to PRE-LIVE or LIVE, execute a full snapshot of your servers' virtual machine. If the patching procedure fails for some reason, you may need to rollback to that point in time. This is specially useful on PRE-LIVE where you are still testing the patching procedure and will most likely break things until you get it right. To be able to quickly go back in time and try it again and again and again will give you the freedom you need to produce a high quality patching procedure.

If you can afford downtime, use it. Don't try to push a zero downtime policy if you don't really need it. It will only make things unnecessarily complex and risky. You can patch Ensemble integrations without down time with the right procedure though. I micro services architecture may also help you to eliminate downtime but it is complex and requires a lot of engineering.

Using External Service Registry

I recommend using External Service Registry so that when you generate your XML project with the new production definition and etc. no references to End Points, folders, etc. are there. Even if you don't send your entire production class, this will help you with the periodic refreshing of the pre-live databases from live. The External Registry Service would store the end point configurations outside your databases and they would be different on LIVE, PRE-LIVE, QA, DEV and on the developer's machine (that may be using local mock services, new versions of services elsewhere, etc.).

If you need to monitor your productions, try checking out the Ensemble Activity Monitor:

http://docs.intersystems.com/latest/csp/docbook/DocBook.UI.Page.cls?KEY=...

It will give you the information you need with minimum performance impact since the data will be stored on a cube and your queries won't affect your runtime system. You can see the counts on the last hour, week, month and year with trending graphs. You will also get the same for queuing and wait time. It's great stuff!

Agreed. I believe this information should have come inside the main exception. Many developers probably have a hard time debugging errors without the real root cause. But then, the documentation explains how to get to the root cause and even give code snippets on how you should code to always have the root cause (that is on %objlasterror).

I believe this recommendation you linked on our documentation is outdated and wrong. One must use %objlasterror on several instances. Examples:

Java Gateway: http://docs.intersystems.com/latest/csp/docbook/DocBook.UI.Page.cls?KEY=...

.NET Gateway: http://docs.intersystems.com/latest/csp/docbook/DocBook.UI.Page.cls?KEY=...

SOAP Error Handling: http://docs.intersystems.com/latest/csp/docbook/DocBook.UI.Page.cls?KEY=...

Caché ActiveX Gateway: http://docs.intersystems.com/latest/csp/docbook/DocBook.UI.Page.cls?KEY=...

%New() constructor: http://docs.intersystems.com/latest/csp/docbook/DocBook.UI.Page.cls?KEY=...

Using %Dictionary Classes: http://docs.intersystems.com/latest/csp/docbook/DocBook.UI.Page.cls?KEY=...

There are many instances where %objlasterror is the ONLY place where you can find out what REALLY happened. Not using this information on production is, IMHO, unwise.

Kind regards,

Amir Samary

Hi Dan!

Yes! I kept reading after I answered and I just noticed that. Thank you for pointing that out!

Kind regards,

AS

Hi Dan!

I don't really like macros. :) But I love exceptions. It would be awesome if %SQL.Statement simply threw the Exception when an error occurs instead of returning a SQLCODE that must be checked and transformed to either an exception or a %Status... In this way, we could keep the number of ways we deal with errors reduced to two, instead of three.

Your explanation is indeed very compelling and I will start using %SQL.Statement from now on. I was thinking about building a macro named $$$THROWONSQLERROR(result) that will receive the resultset returned by %Prepare and check it's SQLCODE and, if there is an error, throw it using result.%SQLCODE and %Message just like CreateFromSQLCODE does. This would allow me to hide SQLCODE.

Kind regards,

AS

When you receive a <ZSOAP> or <ZGTW> error, throw it away and take whatever comes into %objlasterror as your final %Status code. For <ZSOAP>, %objlasterror will have the real error description such as the timeout, XML parsing errors, authentication errors, etc. For <ZGTW> errors, %objlasterror will have the complete Java stack trace of the error.

Kind regards,

AS

I started using $System.Status.* methods about 10 years ago when I wanted to demo how we could take code from Visual Basic 6, VBA or ASP and copy most of its logic into a Caché class method and use Language = basic.

If you need to call one of our API methods from this VBScript code you would probably receive a %Status. As VBScript doesn't use the $$$ macros, the only way to parse the error was by using $System.Status methods. I believe supporting other languages as VBScript was one of the reasons we put this code in there... But I may be wrong.

So, for consistency, I started using only $System.Status methods everywhere. I could write some code in COS that would parse an error with $System.Status.IsError() and I could rewrite the same method in VBScript using the same $System.Status methods without having to explain to people why, on the same product, we would make you deal with errors in different ways. We couldn't avoid "On Error" x "Try/Catch" though.

This also helps people notice $System and %SYSTEM package of classes and see what else is in there. Very useful.

I understand using macros will result in faster code. I also believe our compile could optimize $System.Status.IsError() and $System.Status.OK() method calls to produce the same byte code as the macros. We probably don't do this, but as a Sales Engineer, that is trying to show people how simple and powerful our technology can be, I prefer consistency and clarity over speed. I would also prefer consistency and clarity over some additional speed in any professional code that must be maintained by someone else in the future... 

I have strong feelings about &SQL() too. I would avoid it at all costs whilst I know that it will be the faster way to run a query in Caché. I prefer using %SQL.Statement or %ResultSet because I hate to make my code uglier just to accommodate SQLCODE error handling. Beside this, &SQL can't be used on other supported languages such as VBScript (that is not important anymore) and will force you to compile your classes if you decide to add a new index to your class or make more dramatic changes such as changing your class storage definition. You can change your storage definition, add indices, etc. without having to recompile your classes when using %SQL.Statement or %ResultSet because those cached routines will be automatically deleted for you... That is what most people would expect. I like when things look clear, simple and natural... So I also avoid using &SQL.

Finally, people tend to not even check for errors . If you make things complex, most people will produce bad code and blame you for having a complex programming language. Consistency makes people safer.

Kind regards,

AS

Hi Dan!

I have been using %ResultSet forever and my coding style is as follows:

/// Always return a %Status
ClassMethod SomeMethod() As %Status
{
     Set tSC = $System.Status.OK()
     Try
     {
          Set oRS = ##class(%ResultSet).%New()
          Set tSC = oRS.Prepare("Select ......")
          Quit:$System.Status.IsError(tSC)        

          Set tSC = oRS.Execute()
          Quit:$System.Status.IsError(tSC)

          While oRS.Next()
          {
              //Do something...
          }     
     }
     Catch (oException)
     {
          Set tSC = oException.AsStatus()
     }
     Quit tSC
}

As you can see, it is painful enough to have to deal with both Try/Catch and %Status ways of handling errors. I have used Try/Catch in the same way I used to use $ZT back in the days. We must protect the code from unpredictable errors such as <FILEFULL>, <STORE>, etc. On the other hand, most of our API return %Status. So, there is no choice but to use a similar structure for handling both ways of reporting errors.

With the new %SQL.Statement interface I am required to check yet another way of reporting errors (SQLCODE) and translate those errors to either %Status or an Exception. That makes my code look ugly and no so much object oriented as I would like. You see, when I am doing demos and coding in front of people I tend to code the same way I code when I am building something for real and vice-versa. Caché/Ensemble is really a formidable technology and one can build things with our technology that would take anyone else months on other technologies. But the first impression is key and when I am doing demos I want to show beautiful code that is easy to read and understand. That is why I keep using %ResultSet. It's true %Prepare() will return a %Status but %Execute won't and I would have to inspect %SQL.StatementResult for it's SQLCODE and transform it into a %Status/Exception.

I opened a prodlog for this some time ago (118943), requesting an enhancement for this class to support a %Status property as well as a SQLCODE. 

Kind regards,

AS

I understand the power of %SQL.Statement but as most of my queries are simple I continue using %ResultSet since the error treatment with %Status is more consistent.

It is bad enough that we have to deal with a mix of %Status and Exception handling. I don't like to have to check for %SQLCODE being negative after %Execute() and, if it is, having to transform it to a %Status to keep error handling consistent.

%ResultSet's Execute() method will return me a %Status while %SQL.Statement interface makes me have to deal with yet another type of error (%SQLCODE) making error handling code yet uglier... 

I like consistency, so I continue using %ResultSet. But when I need more functionality or more speed, I use %SQL.Statement instead.

Respectfully,

AS

Hi Daniel!

I tend to look at REST services as a SOA Web Service an, as such, it must have a "contract". Binding this contract to the internal implementation can be problematic. Normally, you would try to work with a JSON object that is more "natural" to your web application while dealing with CRUD operations related to it on the server. That would allow you to decouple the client from the server through the contract and change your server implementation while keeping you client untouched.

So, beware that this excessive coupling can indeed increase productivity right now but may become a nightmare in the future...

Kind regards,

AS