::’Z’ Engine:: Efficient unit test execution

I’d like to share with you a reflexion I’m having about the unit test execution of my ‘Z’ engine. By default, I choose to have them launched each time my application is started (except for my “master” build target). Of course I can skip all tests or launch them as standalone (ie without running the full game) using some command line arguments.
This default behavior was chosen because I want to enforce me to execute them really regularly on each tested platforms (Window, Mac, iOS) and avoid the classic workflow: tests launched “sometimes”, about every two or three months after some deadlines or milestones reached (ie big features implemented, I don’t have any ship date to fulfill, that’s the benefit of an entirely personal project), observing that a lot of them are in “broken” state on some or all platforms, and waste some time to debug and fix ’em all. (Also, at home, I have no continuous-working server that could periodically launch them for me)

Currently I’m not having so many tests to launch, so the execution is not very time expensive but I’m thinking about a way to have each test executed only when required (ie, there was a code modification that could affect it), thanks to the __DATE__ & __TIME__ C++ standard preprocessor macros.

The pseudo code inside a zFooClassTest.cpp implementation (not representative) :

 // get compilation time and Date
zTime buildTime = zTime::ParseDateTime(__DATE__, __TIME__); //convert strings to my datetime format
// retrieve last launch of this test (from a local file, a database, etc…)
zTime lastLaunchedTime = zUTEST::RetrieveLastLaunch(testId); //testId could be *this, a string name, a hashcode or whatever test identification
if  (buildTime > lastLaunchedTime)
        //do the test execution
       zUTEST::SetLastLaunchedTime(testId, zTime::GetCurrentTime());

This is really simple. The major issue to solve is: How can I ensure my zFooClassTest.cpp is rebuilded (and so the “buildTime” is refreshed) each time a modification occurred on the tested zFooClass or one of its dependencies.
  1.  Adding a #include “zFooClass.cpp” directly in the zFooClassTest.cpp. You get kind of ubercpp behavior, but you also have to exclude the zFooClass.cpp file from the supported builds in your project to avoid multiple definition at link-time. This could be quite unfriendly to maintain especially if you have several platforms IDE projects manually managed instead of an automatic generation (CMake or alike). Moreover, this is ok only if your test is very low-level and don’t uses directly other objects that will then require to be added in several.cpp test files…unless you have an executable or module generated for each of your tests, loaded and launched separately by the main application execution. A viable solution only if you have a tool to manage parts of all this horrible stuff automatically for you, and if this kind of execution cost (loading modules, etc..) is still very faster than the test execution itself… I really doubt that.
  2. Having the __DATE__ & __TIME__ also used and stored inside a static member of each object classes (zFooClass, …) and retrieved by the test. The test buildTime information is still kept cause we of course want the test to be launched when modified itself. 
    During runtime it compares each used class build date & his own one with lastLaunchedTime. This class parts could be easily implemented automatically using macros, but there is more things to write and to be aware of when you implement your test. Having buggy test not launched when required doesn’t sounds pretty. Furthermore, you can’t detect inlined modifications in the .h header file.

Finally another issue (and there are probably more I’m not thinking about) is, you can’t have this kind of smart management if your test use a static or dynamic library. This is definitely a low-level tests trick.

I plan to implement one of these propositions someday in the ‘Z’ engine, but only for really low-level small tests that require minimum refactoring. Fortunately, I write test essentially for low-level critical things. Maybe I will think about a more heavy-to-implement-but-convenient-to-use solution only when I’ll have enough big and long tests to justify this work. I’m conscious this is not essential part of an engine, but hey, I’m here to play…