At J2 Interactive we have created a development and deployment system that we use internally and at most of our client locations. This ranges from teams of one on a local system to greater than 100 developers on an enterprise. With most everything in between. The system has been around for a decade now and has evolved as Caché and Ensemble have.

It’s easier for me to take the questions out of order.

How do you keep track of large code bases? Thousands of folders named backup1, backups2, ..., SVN, git?

The heart of the system is a Subversion (SVN) server that Caché Studio communicates with directly using a custom hook library built off the %Studio.SourceControl framework provided natively in Caché. The developer has additional context and top level menu items that allow him or her to perform all the typical version control actions: update, check-in, revert, diff, etc. We based our system on SVN because it was important for us to have a cross platform license free solution and at the time of development SVN was the industry standard. It's still very robust, just not turning heads like it used to. ☺

It’s worth noting that many of our installations pre-date the rise of cloud repositories like GitHub and Beanstalk. But we still find today that it is usually a requirement to have an on premise server owned and controlled by the organization doing development.

Do you have a development server to which you commit and test features there, or do you rather run a local copy of Caché and implement features locally first, then push to the server?

The second and more complex part of our solution is what we call our "Deployment" system. The deployment system drives code promotion through the local developer sandbox -> development -> test -> production systems with room for customization. (Some organizations have multiple test and validation stages.) The system is Caché project based. A developer collects one or more assets that make up a change, fix or feature and adds them to a Studio project. Those assets are then bundled and moved through the environments using a web based tool running on the target Caché systems. A developer never touches the target system's code directly, instead "agents" on the target use a combination of Subversion and COS commands to fetch and install.

These assets can be COS classes, routines, CSP files, schemas, Ensemble rules, DTL, binary files and anything else that one finds in the Caché ecosystem. In addition to the standard asset types we have also introduced a few specialty classes that we call "patch classes". These are COS classes that implement a patching interface. This allows the developer to do things like create/modify globals, manipulate Ensemble production settings, and perform arbitrary actions pre and post install. Each of these tasks can also be configured to behave differently depending on the target deployment environment. For example the endpoint of a REST service may be the same for all the test environments but in prod need to go to a different location and use a SSL credential.

We have tailored the deployment system around the complexities introduced by updating live Ensemble and HealthShare systems. This includes taking in to account things like running productions, default settings and HealthShare specific configuration items. The goal is to have no external (or manual) configuration of downstream systems done. Everything is handled through the deployment system.

Do you use an issue tracking / collaboration system? If so which one. Any you would recommend or immediately dismiss based on personal experience?

This is a great question as I find that we are still circling this issue ourselves. Between all of our developers and all of our clients everyone has their own take on SDLC and even within the various disciplines of the Agile standards no one has the same idea of what is correct. We’ve used more tools than I can name or remember ranging from simple open source ticket systems to full blown Scrum on Jira with all the plugins you can throw at it. Most of the time it is best determined by what your team is familiar with and what resources they have. And different projects require different approaches. For example I’m not afraid to say out loud that I believe building a traditional HL7 interface is more suited to Waterfall type development than Agile (gasp!).

Part of our deployment system is agnostic support for ticket numbers back out to whatever system a team is using as well as a permissions system for being able to promote your code only so far. This lets you introduce gatekeepers and code reviewers. Right now we are working on integrating that with an online code review tool (exploring GitLab) as we are finding that the process of "let’s jump on a screen share for a review" doesn’t scale how we need it to. The ability to import/export COS code in UDL on newer installs is making this much easier than in the past.

We are at a very interesting point in the evolution of COS development right now with the introduction of Atelier, all the side projects that have popped up and an overall interest in bringing the advances and culture of the new generation of devops tooling we are seeing outside our community. The whole system I described above we built out of necessity and we joke about how much we look forward to throwing pieces of it away as the functionality becomes both part of the Caché tool and developer mindset! Thanks for kicking off this discussion, it's a good one.