New post
Question
· Jul 29, 2016

How best to "break" my code

#Object Data Model #Caché

Hello,

I've been doing this for years.  Well, decades.  And, I pride myself in being able to break other people's code.

However, sometimes I don't apply the same dedication to breaking my code.  That needs to change. wink

So, I'd like to ask people, what are some of the things you like to test for when trying to stress-test your code?  I'll start with what I like to try testing to see what crashes, what works unexpectedly, and what I've handled.

I always test the following: 0; 1; -1; some 20-digit number; some 50-character string; 1/0; 2E12; " ", and an undefined variable.

What other goodies am I missing here? For this discussion, I will assume some screen input, so someone might enter a control character. Should I do something like "f I=1:1:31 s var=$c(I) do ..." with var?

Anything else?

Thanks,

Discussion (6)1
Log in or sign up to continue
Jon Willeke · Jul 29, 2016

Greetings. Did you consult at InterSystems fifteen years ago? Long time no see.

My favorite technique to keep myself and others honest is to test with random inputs. This gives rise to two challenges: generating the input, and verifying the output. The input obviously depends on the problem: string, number, list, etc. When it comes to output, I look for invariants: x*y=(y*x), sort(x)=sort(shuffle(x)), etc. I sometimes even write another version of the code under test to act as an oracle that's perhaps slower, or not as general.

Caché has at least three ways to generate random numbers: $random(), $system.Encryption.GenCryptRand(), and the Basic Rnd() function.

$random(n) returns a number from 0 to n-1, where n'>1E17.

GenCryptRand(n) returns n bytes of cryptographically random data. You can convert it to a number using one of the $ascii() functions:

  • $a($system.Encryption.GenCryptRand(1))
  • $zwa($system.Encryption.GenCryptRand(2))
  • $zla($system.Encryption.GenCryptRand(4))
  • $zqa($system.Encryption.GenCryptRand(8)) - may be negative

Rnd() is interesting for a tester, because you can seed it with Randomize. If you don't use a seeded PRNG, you'll want to log your inputs somehow. It's frustrating to find a one in a billion bug, but not be able to reproduce it.

2 0
Chad Severtson · Aug 1, 2016

When a language isn't strongly typed, I often test unexpected but possible types -- for example: Numbers or Strings instead of Dates or Booleans. 

Other favorites: 

  • ""
  • "null"
  • "undefined" 
  • Quotes, comments, and other injection attack patterns. 

As Jon Willeke suggested, I also use random data when possible (apart from the -1, 0, 1 defaults).  The %Library.PopulateUtils class can be helpful for generating other preformatted random information. 

1 0
Kyle Baxter · Aug 1, 2016

If you are going to do all the $C's, then you should definitely test $C(0) as well.  Was that "" (empty) or " " (space)?  I think you should test both.  And also probably test with a longer string - something like 600+ characters (limit for subscripts), 33K characters (normal string limit), and perhaps even try to blow out the string stack and make sure things are handled "properly" (whatever that means for you). 

1 0