New post
Question
· Apr 13, 2016

Followup on the unconference session about unit test and code coverage: it may be doable after all... But I need help

#Caché #Debugging #ObjectScript

Hello everyone,

I have discussed extensively with Andrew Cheshire, the author of the Atelier parser (which I'll be able to hack on and reuse for CachéQuality -- yay!), and while we were at it, we also discussed code coverage.

We ended up talking about the ZBREAK command and its /TRACE option.

It appears that combining this command and the unit test runner, we can indeed do code coverage (Andrew also told me about a way to map "back" routine lines to original lines in the source code).

So, according to the information above and this page, a sequence of possible command would be (provided we are in the correct namespace):

  • set ^UnitTestRoot = "/path/to/test/root"
  • zbreak /TRACE:ALL:someOtherFile
  • do ##class(UnitTest.Manager).RunTest("someDirUnderRoot")
  • [and the commands to disable tracing]

The problem though is that the trace format refers to int code... However Andrew told me that it was possible to "map back" to the "real", COS code.

The question is, how?

Discussion (4)0
Log in or sign up to continue
Timur Safin · Apr 13, 2016

I'm confused a bit with the set of proposed tools for "code coverage" and let me explain why...

Yes, ZBREAK is quite powerful, and ages ago, when there was no yet source-level MAC/CLS debugging in the Studio people implemented all the nasty things using smart ZBREAK expressions and some post-massaging with gathered data. Myself, for example, in my prior life, I've even implemented some "call tree" utility which was gathering information about called routines touched while executing arbitrary expression. And ZBREAK could help in many things, if there would be no kernel support...

But there is kernel support for profiling, which gather all the necessary [for code coverage] information, and which you could see used in ^%SYS.MONLBL. You don't actually interested in the actual timings for every line, but you do want to know that this particular line was called, but this was not.

Also I'm confused with you mentioned UnitTest infrastructure as key in solution, actually it's not necessary. Code coverage utilities are usually targetting not only unit-test code, but whole code size of your system. Assumption is - you do have some set of regression or unit-tests, not necessary using directly that mentioned UnitTest.Manager infrastructure, probably any other test framework, but what is the goal of execution - is collecting information of covered lines in some core of your system. This is quite independent of the test infrastructure, IMVHO.

So for me "code coverage" tool should:

  • use profiling mode with the least intrusive counters activated;
  • be able to remap from INT level gathered by profiler to the original MAC/CLS level;
  • and be able to nicely visualize the gathered information
0 0
Francis Galiegue · Apr 14, 2016

I didn't know about ^SYS.MONLBL.

As to why unit testing, it's because unit testing is used along with code coverage: you run unit tests, and you check what parts of your production code is taken/non taken. Timings are indeed irrelevant.

The plan is to run unit tests with coverage activated, scan both the test results and monitoring results and collect the results.

0 0
Mark Hanson · Apr 14, 2016

I fully agree that using the line by line monitor is the way to go, take a look at  %Monitor.System.LineByLine.cls which provides an API to turn this on/off and get information out of the line by line monitoring.

Also for mapping INT lines back to source lines you can look at the API in %Studio.Debugger.cls in method  SourceLine. This class does say it is internal but I think this particular method should be fine to use.

0 0