Test Coverage: Coverage report not generating when running unit tests with ZPM

Question

Question

Kari Vatjus-Anttila · Jan 28, 2023

#Continuous Integration #Testing #InterSystems IRIS

Hello,

I read a great article by @Timothy Leavitt here, where he wrote about unit testing and test coverage with ZPM. I am facing a slight problem and was wondering if someone has some insight into the matter.

I am running my unit tests in the following way with ZPM, as instructed. They work well and test reports are generated correctly. Test coverage is also measured correctly according to the logs. However, even though I instructed ZPM to generate Cobertura-style coverage reports, it is not generating one. When I run the GenerateReport() method manually, the report is generated correctly.

I tried to add some logging to TestCoverage Manager to see if UserParam property gets populated but it is empty no matter what I try to input to UserParam via ZPM using -D flag.

I am wondering what I am doing wrong. I used the test flags from the ObjectScript-Math repository example, but they seem not to have any effect.

Here is the ZPM command I use to run the unit tests:

zpm "common-unit-tests test -only -verbose 
-DUnitTest.ManagerClass=TestCoverage.Manager 
-DUnitTest.JUnitOutput=/opt/iris/test/TestReports/junit.xml 
-DUnitTest.FailuresAreFatal=1 
-DUnitTest.Manager=TestCoverage.Manager 
-DUnitTest.UserParam.CoverageReportClass=TestCoverage.Report.Cobertura.ReportGenerator 
-DUnitTest.UserParam.CoverageReportFile=/opt/iris/test/CoverageReports/coverage.xml":1

The test suite runs okay, but coverage reports do not generate. However, when I run these commands stated in the TestCoverage documentation, the reports are generated.

Set reportFile = "/opt/iris/test/CoverageReports/coverage.xml"
Do ##class(TestCoverage.Report.Cobertura.ReportGenerator).GenerateReport(<index>, reportFile)

Here is a short snippet from the logs where you can see that test coverage analysis is run:

Collecting coverage data for Test: .036437 seconds
  Test passed

Mapping to class/routine coverage: .041223 seconds
Aggregating coverage data: .019707 seconds
Code coverage: 41.92%

Use the following URL to view the result:
http://192.168.208.2:52773/csp/sys/%25UnitTest.Portal.Indices.cls?Index=19&$NAMESPACE=COMMON
Use the following URL to view test coverage data:
http://IRIS-LOCALDEV:52773/csp/common/TestCoverage.UI.AggregateResultViewer.cls?Index=17
All PASSED

[COMMON|common-unit-tests] Test SUCCESS

What am I doing wrong? FYI I am also using the Mocking Framework introduced here. I wonder if it causes any issues?

Thank you, and have a good day!
Kari Vatjus-Anttila

Product version: IRIS 2022.2

$ZV: IRIS for UNIX (Ubuntu Server LTS for x86-64 Containers) 2022.2 (Build 368U) Fri Oct 21 2022 17:18:04 EDT

Discussion (3)2

Log in or sign up to continue

score 0 · Answer 1 · 2023-01-31T12:22:51-05:00

I tried to read through the source code to get an idea what triggers the report generation. Didn't find any clues, so I ended up tweaking the Manager.cls source code a bit. I modified OnAfterSaveResult() method where I check if the user has defined the CoverageReportClass and CoverageReportFile parameters and trigger the report generation if defined.

#; Generate coverage reports if needed
If (..UserFields.GetAt("CoverageReportClass") '= "") && (..UserFields.GetAt("CoverageReportFile") '= "") {
  Set coverageReportClass = ..UserFields.GetAt("CoverageReportClass")
  Set coverageReportFile = ..UserFields.GetAt("CoverageReportFile")
  Set tSC = $ClassMethod(coverageReportClass, "GenerateReport", tRunIndex, coverageReportFile) 
}

That seems to work and solves my problem. If anyone has a better solution to the problem feel free to comment. For now I will stick with this.

Cheers,
Kari

score 0 · Answer 2 · 2023-02-01T13:03:41-05:00

@Kari Vatjus-Anttila I'm just seeing this now. I'm a bit mystified - currently I'm using the same flags with latest-enough IRIS and the latest package manager, and it's fine.

It's possible that the mocking framework is causing issues. It looks like it has its own unit test manager; how does that play alongside the custom TestCoverage unit test manager? Have you made further changes to TestCoverage.Manager?

score 0 · Answer 3 · 2023-02-16T03:50:48-05:00

Hello,

"Have you made further changes to TestCoverage.Manager"

No, only that if-clause which checks if Coverage related flags are set.

I am not that familiar with the Mocking Framework yet to know if the unit test manager causes any issues. I thought that setting -DUnitTest.ManagerClass=TestCoverage.Manager overrides the unit test manager. I verified that by adding some debug logging to the TestCoverage.Manager class.

At the moment, my solution works as expected. If you manage to find any clues why it does not work, I am very interested to know.

By the way, where exactly are the arguments CoverageReportClass and CoverageReportFile checked? I didn't find any trace of them in the source code.