Is there a way to run unit tests in a given project which _does not_ require that test classes be exported first?

Hello,

It is time for me to eat my own dog's food and start implementing unit test running with coverage :) I will be inundating IRC with questions at this point, but I have a more general question first.

In this tutorial, it is supposed that your unit tests are exported as XML first... But that's not very practical. Is there a way, instead, to run all tests from a given project without having this export?

My first thought on how to do this would be to:

  • grab the project (by name, I suppose?),
  • grab the list of classes defined by this project,
  • inspect the classes,
  • consider unit tests only these classes which extend (directly or not) %UnitTest.TestCase,
  • consider test methods only those methods whose name start with Test,
  • run each method individually.

I'd have expected %UnitTest.Manager to provide such a method instead but it appears that it doesn't :/ At least, as far as I can see. And I still don't "see" much...

Do you have a better approach?

Regards

Answers

See the discussion in this previous post:

https://community.intersystems.com/post/unittest

Basically, you can use the /noload and /nodelete qualifiers, specifying the tests by class name. Be aware that if you forget the /nodelete qualifier, you'll delete your test classes.

Here's some code from the application I'm working on that might help. The "load/delete the test classes" behavior was annoying enough that we decided to always have the classes loaded on development/testing systems.

First, I think it's useful to have a Run() method in each unit test class, or in a subclass of %UnitTest.TestCase that your unit tests will extend. This code could live somewhere else too, but it's useful to be able to say:

do ##class(my.test.class).Run()

and not have to remember/type the test suite format and /nodelete. Sample implementation:

Class Tools.UnitTest.TestCase Extends %UnitTest.TestCase
{

/// Runs the test methods in this unit test class.
ClassMethod Run(ByRef pUTManager As %UnitTest.Manager = "", pBreakOnError As %Boolean = 0)
{
    If '$IsObject(pUTManager) {
        Set pUTManager = ##class(%UnitTest.Manager).%New() //Or Tools.UnitTest.Manager if you have that
        Set pUTManager.Debug = pBreakOnError
        Set pUTManager.Display = "log,error"
    }
    Set tTestSuite = $Piece($classname(),".",1,*-1)
    Set qspec = "/noload/nodelete"
    Set tSC = $$$qualifierParseAlterDefault("UnitTest","/keepsource",.qspec,.qstruct)
    Do pUTManager.RunOneTestSuite("",$Replace(tTestSuite,".","/"),tTestSuite_":"_$classname(),.qstruct)
}

}

This allows you to specify an instance of a %UnitTest.Manager to capture the test results in, which is useful if you're running a bunch of specific unit test classes (like you suggested, from a Studio project). My team organizes tests in packages rather than in projects, which makes more sense for us.

Next up, here's our %UnitTest.Manager subclass that works with the %UnitTest.TestCase subclass shown above, allowing all the classes in a particular namespace or package (or, really, with class names that contain a particular string) to be run without deleting them afterward:

Class Tools.UnitTest.Manager Extends %UnitTest.Manager
{

/// Runs all unit tests (assuming that they're already loaded)
/// May filter by package or output to a log file rather than terminal
ClassMethod RunAllTests(pPackage As %String = "", pLogFile As %String = "") As %Status
{
    Set tSuccess = 1
    Try {
        Set tLogFileOpen = 0
        Set tOldIO = $io
        If (pLogFile '= "") {
            Open pLogFile:"WNS":10
            Set tLogFileOpen = 1
            Use pLogFile
        }
        
        Write "*** Unit tests starting at ",$zdt($h,3)," ***",!
    
        Set tBegin = $zh
    
        Set tUnitTestManager = ..%New()
        Set tUnitTestManager.Display = "log,error"
        Set tStmt = ##class(%SQL.Statement).%New()
        Set tSC = tStmt.%PrepareClassQuery("%Dictionary.ClassDefinition","SubclassOf")
        $$$THROWONERROR(tSC,tStmt.%PrepareClassQuery("%Dictionary.ClassDefinition","SubclassOf"))
        Set tRes = tStmt.%Execute("Tools.UnitTest.TestCase")
        While tRes.%Next(.tSC) {
            If $$$ISERR(tSC) $$$ThrowStatus(tSC)
            Continue:(pPackage'="")&&(tRes.%Get("Name") '[ pPackage)
            Do $classmethod(tRes.%Get("Name"),"Run",.tUnitTestManager)
        }
    
        If $IsObject(tUnitTestManager) {
            Do tUnitTestManager.SaveResult($zh-tBegin)
            Do tUnitTestManager.PrintURL()
    
            &sql(select sum(case when c.Status = 0 then 1 else 0 end) as failed,
                        sum(case when c.Status = 1 then 1 else 0 end) as passed,
                        sum(case when c.Status = 2 then 1 else 0 end) as skipped
                        into :tFailed, :tPassed, :tSkipped
                   from %UnitTest_Result.TestSuite s
                   join %UnitTest_Result.TestCase c
                     on s.Id = c.TestSuite
                  where s.TestInstance = :tUnitTestManager.LogIndex)

            If (tFailed '= 0) {
                Set tSuccess = 0
            }
        } Else {
            Write "No unit tests found matching package: ",pPackage,!
        }
    } Catch anyException {
        Set tSuccess = 0
        Write anyException.DisplayString(),!
    }
    Write !,!,"Test cases: ",tPassed," passed, ",tSkipped," skipped, ",tFailed," failed",!
    If 'tSuccess {
        Write !,"ERROR(S) OCCURRED."
    }
    Use tOldIO
    Close:tLogFileOpen pLogFile
    Quit $Select(tSuccess:1,1:$$$ERROR($$$GeneralError,"One or more errors occurred in unit tests."))
}

This could probably be tweaked to use a project instead without too much work, but I think packages are a more reasonable way of organizing unit tests.

I'm only seeing this post now... Great source of information!

Now I wish I had more practice... But this is close to what I want to do.

Don't you want to contribute to the project? :p