New post
Question
· May 18, 2022

Unit Testing Conventions/Best Practices - Where Do They 'Live'?

#Testing #InterSystems IRIS

Continuing on the journey of implementing %UnitTest, @Timothy Leavitt's Test Coverage package, and automated testing with Jenkins.

My question today: why do we utilize a unit test root directory?  

I've been defining packages and classes to write unit tests as I've been developing and I run the tests on the command line or using a routine file that is setup as a debug target which has been working great.  I use DebugRunTestCase() to do this (so the classes aren't deleted).  

As I am playing with automating this on Jenkins, I'm not understanding exactly why we just don't keep and run them on the server?  I may be answering my own question here, but is the idea we don't want the classes living on the server with the application code?  Some questions for you all as I learn best practices:

- Are all your unit tests added to .gitignore so they don't get wound up in source code?

- If you do that, how can other developers access the unit tests so they don't write new ones when some already exist?

- Why does the normal RunTestCase() method automatically delete the extracted unit test class files from the folder?  Why is that the default?

- When it comes to automated testing (Jenkins specifically) what is the lifecycle?  Code is pulled from remote repo, unit test classes extracted into root, all other classes and files compiled on server, run test cases, class files are deleted from root?

Hopefully you can see my biggest gap in understanding seems to be the default functionality of deleting these classes but also needing them on the server to work and develop with.  Would another testing framework like JUnit have similar setups that I could learn more from that?

Thanks for any advice!

Discussion (4)4
Log in or sign up to continue
Pravin Barton · May 18, 2022

Hi Michael, great questions. A lot of this will depend on your own practices for source code management and deployment. In my team's case we ended up overriding a lot of the %UnitTest behavior to provide reasonable defaults for our process. Hopefully this sparks some more discussion. I'm interested in how other peoples' answers will differ.

> Are all your unit tests added to .gitignore so they don't get wound up in source code?

No - we want source code for our unit tests to be in source control, for the same reason all other code is in source control. We make sure that unit tests don't end up on production systems by maintaining different branches for test and production. Unit tests are in a separate directory that we don't merge from the test branch to the production branch. This is using Perforce. There might be a different workflow recommended for Git that would give you the same results.

> Why does the normal RunTestCase() method automatically delete the extracted unit test class files from the folder?  Why is that the default?

If I had to guess, this is a good default for a deployment process where you compile everything, run tests, and then copy over the code database to production. In that case you would always want test classes to delete themselves after running. In our case we have a different way of deploying code, so we override the RunTest methods to use the "/nodelete" flag by default.

> When it comes to automated testing (Jenkins specifically) what is the lifecycle?

We use a very similar lifecycle for Jenkins automated testing to the one you describe.

  1. Jenkins pulls all code from the remote repo
  2. Run an %Installer class on the build instance that overwrites the code database so we start from scratch
  3. Load and compile all code from the local workspace into the build instance, including tests. Report any compilation error as a build failure.
  4. Run all tests.
  5. Generate JUnit-format test reports and Cobertura test coverage reports.
2 0
Michael Davidovich · May 19, 2022

Thanks @Pravin Barton! This is helpful just to get a better send of how others use the framework.  We are still baby stepping into all of this and don't want to rush into implementing it as it's harder to untangle and "redo" once we're really rolling.  

Related to keeping your unit tests separate: so you don't export your unit test class files to a different ^UnitTestRoot?  Rather the ^UnitTestRoot is where the unit tests live in the cloned repository? 

@Timothy Leavitt thanks for the link to that post. I'm sure I've seen it before but more will make sense now that I have more experience with this.  

Related to the test coverage tool: do you have best practices or shortcuts as an individual developer?  Do you run the Test Coverage tool as your developing to see what lines of code are covered or not? We are thinking through how to arm the developer to write the most complete unit tests before sending to the remote and having Jenkins automatically rebuild everything and do the more extensive reporting.

Really appreciate the responses! 

1 0
Timothy Leavitt · May 19, 2022

@Michael Davidovich we'd generally set ^UnitTestRoot to the path in the cloned repo and then run tests right from there. Another workflow might be to have unit tests run in their own namespace that has all the same code/supporting data but is distinct from the namespace where you're doing dev work - in that case, it's safe to delete the unit tests from that second namespace. (But practically speaking we just use one namespace and don't delete.)

Re: best practices/shortcuts as an individual developer, I'd honestly just say embrace the automation and iterate in a feature branch with the tests running on commit to feature branches (not just main). It also helps to use the package manager to make a more modular codebase where each module has its own associated tests, so you're not rebuilding and testing everything every time.

In a case where you're (a) using the package manager and (b) being really strict about test coverage, combining the techniques in the article I linked earlier is a good approach - e.g., saying to run specific tests *and* measure test coverage. That helps answer "how well do the tests I'm currently writing cover the code I'm currently adding/changing?"

Of course, you can do this with the normal APIs for running unit tests with test coverage too. (TestCoverage.Manager also has DebugRunTestCase - think of it like %UnitTest.Manager but with test coverage built in.)

1 0