Unit test in InterSystems with Jenkins

Question

Question

Ankita JAin · May 31, 2017

Hi ,

I am stuck with unit test failure with intersystem . In case of unit test failure, the build in jenkins is succeding while the build in jenkins should fail in case unit test failure .

In cache programming i am using %UnitTest.Manager class and DebugRunTestCase method within it. I'm able to link studio with jenkins. I wanna fail my build in jenkins, if any of the test cases fails. Could anyone help?

Discussion (14)0

Log in or sign up to continue

Dmitry Maslennikov · May 31, 2017

I don't know how you call Caché method from Jenkins, but anyway you can use $SYSTEM.Process.Terminate in Caché script to exit with an exit status. Something like this.

set tSC=##class(%UnitTest.Manager).DebugRunTestCase(....)
if 'tSC do $SYSTEM.Process.Terminate(,1)
halt

I suggest that you may use csession or cterm to call Caché code, then you should get exit code and send it to Jenkins, which will be recognized by Jenkins as an error and will fail the job.

3 0

score 3 · Answer 1 · 2017-05-31T02:21:40-04:00

Dmitry Maslennikov · May 31, 2017

Please, provide how you call Caché from Jenkins?

3 0

score 0 · Answer 2 · 2018-03-08T07:04:47-05:00

Binay Tamrakar · Mar 8, 2018

You can create a windows batch file, where you can write commands to call Cache terminal using cterm.exe

0 0

score 0 · Answer 3 · 2019-12-13T10:05:37-05:00

One gotcha in windows is making sure the encoding on your .scr file is in UTF-8. Windows thinks a .scr file is a screensaver file and can give it a weird encoding. Once I switched it to UTF-8 it worked as described in the documentation.

score 0 · Answer 4 · 2017-05-31T05:33:09-04:00

Hi,

Thanks Dmitry Maslennikov for the Response.

Could you please help us in Clarifying the below Query.
Is there any methods/ways through which will get to know whether any of the Unit Test cases is/are failing (we are getting an url of the csp page with the report which has the passed failed status) as we need to send this failure status to Jenkins for the Build to Fail (where in we have acheive this part making the build failure/success based on harcoded boolean)

score 0 · Answer 5 · 2018-03-08T12:12:15-05:00

Eduard Lebedyuk · Mar 8, 2018

Here's an object and SQL approaches to get Unit Tests status.

0 0

score 3 · Answer 6 · 2017-05-31T08:20:37-04:00

Hi,

We use something like the below to output the unit test results to an xml file in JUnit format.

/// Extend %UnitTest manager to output unit test results in JUnit format.
/// This relies on the fact that unit test results are stored in <b>^UnitTest.Result</b> global. Results displayed on CSP pages come from this global.
Class com.isc.UnitTest.Manager Extends %UnitTest.Manager
{
  
ClassMethod OutputResultsXml(pFileName As %String) As %Status
{
    set File=##class(%File).%New(pFileName)
    set i=$order(^UnitTest.Result(""),-1)
    if i="" quit $$$OK // no results
    
    kill ^||TMP // results global
    set suite="" for {
        set suite=$order(^UnitTest.Result(i,suite))
        quit:suite=""
        set ^||TMP("S",suite,"time")=$listget(^UnitTest.Result(i,suite),2)
        
        set case="" for {
            set case=$order(^UnitTest.Result(i,suite,case))
            quit:case=""
            
            if $increment(^||TMP("S",suite,"tests"))
            set ^||TMP("S",suite,"C",case,"time")=$listget(^UnitTest.Result(i,suite),2)
            set method="" for {
                set method=$order(^UnitTest.Result(i,suite,case,method))
                quit:method=""
                set ^||TMP("S",suite,"C",case,"M",method,"time")=$listget(^UnitTest.Result(i,suite,case,method),2)
                set assert="" for {
                    set assert=$order(^UnitTest.Result(i,suite,case,method,assert))
                    quit:assert=""
                    if $increment(^||TMP("S",suite,"assertions"))
                    if $increment(^||TMP("S",suite,"C",case,"assertions"))
                    if $increment(^||TMP("S",suite,"C",case,"M",method,"assertions"))
                    if $listget(^UnitTest.Result(i,suite,case,method,assert))=0 {
                        if $increment(^||TMP("S",suite,"failures"))
                        if $increment(^||TMP("S",suite,"C",case,"failures"))
                        if $increment(^||TMP("S",suite,"C",case,"M",method,"failures"))
                        set ^||TMP("S",suite,"C",case,"M",method,"failure")=$get(^||TMP("S",suite,"C",case,"M",method,"failure"))
                            _$listget(^UnitTest.Result(i,suite,case,method,assert),2)
                            _": "_$listget(^UnitTest.Result(i,suite,case,method,assert),3)
                            _$char(13,10)
                    }
                }
                if ($listget(^UnitTest.Result(i,suite,case,method))=0)
                && ('$data(^||TMP("S",suite,"C",case,"M",method,"failures"))) {
                    if $increment(^||TMP("S",suite,"failures"))
                    if $increment(^||TMP("S",suite,"C",case,"failures"))
                    if $increment(^||TMP("S",suite,"C",case,"M",method,"failures"))
                    set ^||TMP("S",suite,"C",case,"M",method,"failure")=$get(^||TMP("S",suite,"C",case,"M",method,"failure"))
                        _$listget(^UnitTest.Result(i,suite,case,method),3)
                        _": "_$listget(^UnitTest.Result(i,suite,case,method),4)
                        _$char(13,10)
                }
            }
            
            if $listget(^UnitTest.Result(i,suite,case))=0
            && ('$data(^||TMP("S",suite,"C",case,"failures"))) {
                if $increment(^||TMP("S",suite,"failures"))
                if $increment(^||TMP("S",suite,"C",case,"failures"))
                if $increment(^||TMP("S",suite,"C",case,"M",case,"failures"))
                set ^||TMP("S",suite,"C",case,"M",case,"failure")=$get(^||TMP("S",suite,"C",case,"M",case,"failure"))
                    _$listget(^UnitTest.Result(i,suite,case),3)
                    _": "_$listget(^UnitTest.Result(i,suite,case),4)
                    _$char(13,10)
            }
        }
    }
    
    do File.Open("WSN")
    do File.WriteLine("<?xml version=""1.0"" encoding=""UTF-8"" ?>")
    do File.WriteLine("<testsuites>")
    set suite="" for {
        set suite=$order(^||TMP("S",suite))
        quit:suite=""
        do File.Write("<testsuite")
        do File.Write(" name="""_$zconvert(suite,"O","XML")_"""")
        do File.Write(" assertions="""_$get(^||TMP("S",suite,"assertions"))_"""")
        do File.Write(" time="""_$get(^||TMP("S",suite,"time"))_"""")
        do File.Write(" tests="""_$get(^||TMP("S",suite,"tests"))_"""")
        do File.WriteLine(">")
        set case="" for {
            set case=$order(^||TMP("S",suite,"C",case))
            quit:case=""
            do File.Write("<testsuite")
            do File.Write(" name="""_$zconvert(case,"O","XML")_"""")
            do File.Write(" assertions="""_$get(^||TMP("S",suite,"C",case,"assertions"))_"""")
            do File.Write(" time="""_$get(^||TMP("S",suite,"C",case,"time"))_"""")
            do File.Write(" tests="""_$get(^||TMP("S",suite,"C",case,"tests"))_"""")
            do File.WriteLine(">")
            
            set method="" for {
                set method=$order(^||TMP("S",suite,"C",case,"M",method))
                quit:method=""
                do File.Write("<testcase")
                do File.Write(" name="""_$zconvert(method,"O","XML")_"""")
                do File.Write(" assertions="""_$get(^||TMP("S",suite,"C",case,"M",method,"assertions"))_"""")
                do File.Write(" time="""_$get(^||TMP("S",suite,"C",case,"M",method,"time"))_"""")
                do File.WriteLine(">")
                if $data(^||TMP("S",suite,"C",case,"M",method,"failure")) {
                    do File.Write("<failure type=""cache-error"" message=""Cache Error"">")
                    do File.Write($zconvert(^||TMP("S",suite,"C",case,"M",method,"failure"),"O","XML"))
                    do File.WriteLine("</failure>")
                }
                do File.WriteLine("</testcase>")
            }
            do File.WriteLine("</testsuite>")
        }
        do File.WriteLine("</testsuite>")
    }
    do File.WriteLine("</testsuites>")
    do File.Close()
    kill ^||TMP
    quit $$$OK
}

}

score 0 · Answer 7 · 2017-06-02T01:45:30-04:00

Please tell us very clearly how to exit the cache terminal via command line and then that batch script should fail the jenkins job ? Please tell us if it is feassible in intersystem . We have tried with all methods from intersystem doc library but still we could not progress (Jenkins job is not getting triggered based on boolean value passed from cache). Please suggest

score 0 · Answer 8 · 2017-06-02T01:50:00-04:00

Hi Ankita,

Would you be able to provide some more details about how you currently run the unit tests from your batch script?

Regards,

Caza

score 0 · Answer 9 · 2017-06-02T02:52:00-04:00

Hi Caza ,

We are facing the following issues :-

1. We are getting the boolean value for the pass and fail unit test cases in cache Intersystem, but we are not able to assingn this boolean value into a variable in a batch script( r3 in this case)

2. @ECHO %FAILUREFLAG% is not giving any output to us, can you help with that also.

Please suggest us for the following problems or some alternative approach. The Batch script code is here :-

:: Switch output mode to utf8 - for comfortable log reading
@chcp 65001
@SET WORKSPACE="https://github.com/intersystems-ru/CacheGitHubCI.git"
:: Check the presence of the variable initialized by Jenkins
@IF NOT DEFINED WORKSPACE EXIT 1
@SET SUCCESSFLG =%CD%\successflag.txt
@SET FAILUREFLAG =%CD%\failureflag.txt
@ECHO %FAILUREFLAG%
@DEL "%SUCCESSFLG%"
@DEL "%FAILUREFLAG%"

:: The assembly may end with different results
:: Let the presence of a specific file in the directory tell us about the problems with the assembly
::% CD% - [C] urrent [D] irectory is a system variable
:: it contains the name of the directory in which the script is run (bat)
:: Delete the bad completion file from the previous start
::@DEL "% ERRFLAG%"

:::::::::::::::::::::::::::::::::::::::::: :::::::::::::::::::::
:: CREATING CACHE CONTROL SCRIPT
:: line by line in the build.cos command to manage the Cache
:::::::::::::::::::::::::::::::::::::::::: :::::::::::::::::::::

:: If the Cache is installed with normal or increased security
:: then in the first two lines the name and password of the user Cache
@ECHO user>build.cos
@ECHO password>>build.cos

:: Go to the necessary NAMESPACE
@ECHO zn "MYNAMESPACE" >>build.cos
@ECHO do ##class(Util.SourceControl).Init() >>build.cos
@ECHO zn "User" >>build.cos

@ECHO set ^^UnitTestRoot ="C:\test" >>build.cos

@ECHO do ##class(%%UnitTest.Manager).RunTest("myunittest","/nodelete")     >>build.cos
@Echo set i=$order(^^UnitTest.Result(""),-1) >>build.cos
@Echo  write  i >>build.cos
@Echo set unitResults=^^UnitTest.Result(i,"myunittest") >>build.cos
@ECHO Set r3  =  $LISTGET(unitResults,1,1) >>build.cos
@ECHO if r3'= 0 set file = "C:/unittests/successflag.txt" o file:("NWS") u file do $SYSTEM.OBJ.DisplayError(r3) >>build.cos
@ECHO if r3'= 1 set file = "C:/unittests/failureflag.txt" o file:("NWS") u file do $SYSTEM.OBJ.DisplayError(r3) >>build.cos
::@ECHO if 'r3 do $SYSTEM.Process.Terminate(,1)
::@ECHO halt
@IF EXIST "%FAILUREFLAG%" EXIT 1

::@ECHO  do ##class(%%UnitTest.Manager).RunTest("User") >>build.cos
::@ECHO set ut = do ##class(%%UnitTest.Manager).RunTest("User") >>build.cos
::set var = 1 >>build.coss
:: If it did not work out, we will show the error to see it in the logs of Jenkins
::@ECHO if ut'= 1 do $SYSTEM.OBJ.DisplayError(sc) >>build.cos
::echo %var%

@Echo On
@ECHO zn "%%SYS" >>build.cos
@ECHO set pVars("Namespace")="MYDEV" >>build.cos
@ECHO set pVars("SourceDir")="C:\source\MYNAMESPACE\src\cls\User" >>build.cos
@ECHO do ##class(User.Import).setup(.pVars)>>build.cos
:: Download and compile all the sources in the assembly directory;
::% WORKSPACE% - variable Jenkins
::@ECHO set sc = $SYSTEM.OBJ.ImportDir("%WORKSPACE%", "*. Xml", "ck",. Err, 1) >>build.cos

:: If it did not work out, we will show the error to see it in the logs of Jenkins
::@ECHO if sc'= 1 do $SYSTEM.OBJ.DisplayError(sc) >>build.cos

:: and from the cos script create an error flag file to notify the script bat
::@ECHO if sc'= 1 set file = "% ERRFLAG%" o file :( "NWS") u file do $ SYSTEM.OBJ.DisplayError (sc) with file >> build.cos

:: Finish the Cache process
::@ECHO if sc'= 1 set file = "% ERRFLAG%" o file :( "NWS") u file do $ SYSTEM.OBJ.DisplayError (sc) with file >> build.cos


:::::::::::::::::::::::::::::::::::::::::: :::::::::::::::::::
:: Call the Cache control program and pass the generated script to it ::
:::::::::::::::::::::::::::::::::::::::::: ::::::::::::::::
C:\InterSystems\HealthShare\bin\cache.exe -s C:\InterSystems\HealthShare\mgr -U %%SYS <%CD%\build.cos


:: If the error file was created during the cos script execution - we notify Jenkins about this
::@IF EXIST "%ERRFLAG%" EXIT 1

score 0 · Answer 10 · 2017-06-04T19:46:00-04:00

Hi Ankita,

I found a couple of issues in the script that might affect your end results:
- the folder C:\unittests doesn't exist (at least not on my computer); unless the value of the WORKSPACE env variable is C:\untittest then you have to ensure the folder exists (you can create it either using batch mkdir or using COS ##class(%%File).CreateDirectoryChain() method )
- what is stored in the ^UnitTest.Result(i,"myunittest") global is not a status code but a numeric value; so I would suggest replacing Do $system.OBJ.DisplayError(r3) with a simple write command, like this:

@ECHO if r3'= 0 set file = "C:/unittests/successflag.txt" o file:("NWS") u file write r3 >>build.cos
@ECHO if r3'= 1 set file = "C:/unittests/failureflag.txt" o file:("NWS") u file write r3 >>build.cos

Regarding "@ECHO %FAILUREFLAG%" - make sure there are no spaces before or after the = character in the following two commands:

@SET SUCCESSFLG =%CD%\successflag.txt
@SET FAILUREFLAG =%CD%\failureflag.txt

When I did copy/paste of the example script I ended up with a space character before the = character.

Can you try these changes and let me know how you go?

Cheers,
Caza

score 1 · Answer 11 · 2017-06-05T07:26:54-04:00

Ankita JAin · Jun 5, 2017

Thanks for your response it worked

1 0

score 0 · Answer 12 · 2017-06-05T08:05:21-04:00

Caza Cazamir · Jun 5, 2017

Yay!

0 0

score 0 · Answer 13 · 2017-06-05T08:04:53-04:00

If you add to your batch script something like:

do ##class(com.isc.UnitTest.Manager).OutputResultsXml("c:/unittests/unit_test_results.xml")

then you could pass on the unit_test_results.xml file to the JUnit plugin from Jenkins. This will give you useful reports, duration and individual results breakdown, etc.

For example: https://support.smartbear.com/testcomplete/docs/_images/working-with/integration/jenkins/trend-graph.png