Policies for the test suite

Isac's test suite primarily checks interactions with the front-end (this is different from Isabelle, which primarily checks specifics of the PIDE document model). The tests started from data used for coding, and still some such tests are not yet removed from the production code. Later tracing down bugs was saved and brought to a standard format, which should make re-use with new bugs convenient. A first systematic part of the test suite is "~ ~/test/Tools/isac/Minisubpbl/*". This is run after testing basics and before testing the Interpreter.

Rules and formats for tests

 * Make tests as independent as possible: copying a test from "test/../file.sml" to "test/Tools/isac/Test_Isac.thy" should be possible without worrying about initialisation of test arguments (particularly important when comparing functions within different ISAC versions).


 * Identification of tests: each test gets a header, repeated 3 times for visibility in the Output window in case the system cannot locate an error. This header is 78 characters long, starts a text after 11 "-" and is also listed in the heading contents area; an example is below.

"--- fun identifier ";
 * Identify test of specific functions which come into tests for several reasons: in/output is demonstrated; another test creates different behaviour of this function in different ISAC versions, etc. Such tests are headed like that


 * Re-use test code: when investigating a function "xxxxx" one can search for "--- fun xxxxx" or for "~ ~ ~ fun xxxxx,", see below


 * Stepwise checking a test (usual in comparison with an elder version): Stepwise execution of functions is uneasy without debugger. Sometimes the work is worth to be kept. In order to ease re-using such work ISAC introduces the following standard (see the templates in "Test_Some.thy") for a function "xxx":

"(*xxx aarg1 .. aargm*) "~ ~ ~ ~ ~ fun xxx, args:"; val (farg1, .., fargm) = (aarg1, .., aargm); "(*xxx arg1..argn ERROR ...*) "~ ~ ~ ~ ~ and xxx, args:"; val = ; "~ ~ ~ ~ ~ to xxx return val:"; val = ;

Before descending into function "xxx", keep the call in a comment (with the actual arguments "aarg1 .. aargm". This simplifies two actions later: check the correctness of the arguments in "val (farg1, .., fargm) = (aarg1, .., aargm)" and review the return value of "xxx".

The first line is used (with the exact number of "~", and without blanks in between -- enforced by this wiki) for calling a function, which is stepped into; "args" comprises all arguments (which require "," in between, if copied from the source code). The function's identifier is inserted before the "," with exactly one blank.

The second line is used, if the respective function call is copied from a mutual recursive definition with "and".


 * In case the test protocols a search for an error, the comment holds the respective error message as well

"(*xxx aarg1 .. aargm ERROR ...*) "~ ~ ~ ~ ~ fun xxx, args:"; val (farg1, .., fargm) = (aarg1, .., aargm); "(*xxx arg1 .. argn  ERROR ...*) "~ ~ ~ ~ ~ and xxx, args:"; val = ;

Directory structure and location of test files
Location is determined by respective source files, but there are exceptions, "ADDTESTS" and "Minisubpbl".


 * (1) test files reside in some subdirectory of "test/..." parallel to the position of the file with the tested code in "src/..."


 * (2) test with extend the KEStore require a specific setup, see for instance changeset e8d9d194a96f:
 * (2.1) "setup {* ... *}" in file "/test/../filename.thy" ("filename.thy" with small leading letter: NOT Isabelle coding standard)
 * (2.2) the respective SML code is in "/test/../filename.sml" (in order to keept the "imports" phase of Test_Isac.thy short)


 * (3) tests outside the directory structure of source files are:
 * (3.1) ADDTESTS: most of these tests involve "*.thy" files themselves
 * (3.1) Minisubpbl: decomposes a minimal example (with sub-problem) into functions calls. In case you encounter errors with functions from "Interpre/" and later, then you might consider to refine these tests within the appropriate file. In case you find the culprit in a function defined in "ProgLang/" or earlier, locate a new test in the respective test file, where the function is defined.

Regression testing
Since tests have the same file structure as the sources, tests follow the sequence of bootstrapping. So it is advisable to fix errors in this sequence: this avoids to dig through too many layers of code in tests late in the sequence, because low level functions have been defined, and also tested, first.


 * (0) Test_Isac.thy requires specific handling for open/close structures as explained here.


 * (1) Run the whole test suite and try to figure out first from the pattern of red marks over the tests and from their origin in the respective test files, what the reason could be. The most important decision is, where to go into details: in "Minisubpbl/*" (see (2)) or somewhere else. Going into details (according to (4)) can be very tricky, investing time in this step might pay off.


 * (2) In case you encounter errors somewhere from the initial imports (e.g. from "ADDTESTS/All_Ctxt"), then out-comment these and fix the errors down into "Minisubpbl/*" first. See this changeset.

https://intra.ist.tugraz.at/hg/isa/rev/107330cc8b6a
 * (3) Fixing errors down to "Minisubpbl/*" requires activation of some setups in the initial imports, e.g. see

ML {* *} ML {* ... insert here and then bisect with "*} ML {*" ... *} ML {* *}
 * (4) If you encounter an error in Test_Isac.thy at a certain ML_file xxx.sml then
 * (4.1) open the file "test/../xxx.sml" by click in Test_Isac.thy and look if you see a red error icon (!) at the left margin; if there is none then
 * (4.2) look to the Output window for the error from xxx.sml; if this doesn't help
 * (4.3) copy the respective test from xxx.sml into Test_Isac.thy (i.e. into ML {* ... *}) below ML_file "xxx.sml": test setup is present here, and you can watch the red (x) there at the left margin. If this doesn't help
 * (4.4) copy the whole test file xxx.xml into Test_Isac.thy and bisect using this structure:


 * (5) Sometimes is helpful to compare with an elder Isac version, most likely the most recent version known for running "Test_Isac.thy" without errors. In this case step into a function in the old version and at specific calls copy the test back to the new version -- and not the other way round (because you might follow a misleading trace in the new version with the error).

Introduce and test a new structure
This task has become fairly convenient with Isabelle/jEdit. Here some notes on details.


 * (1) Collect code for the new structure "New" by cutting from old structures "Old". Cutting avoids confusing duplications. This also causes error messages from signature "OLD" will be helpful in steps to come


 * (2) Collect code until all definitions are present and no error in "New" is remaining.


 * (3) Collect entries for signature "NEW" by cutting from old signature "OLD" and do not autogenerate signature "NEW" from structure "New": that does not avoid duplication and might lead to different entries in the signature. Note, that PolyML does not reliably report signature entries without code in the structure (it only reports differences of the signature between signature and structure).


 * (4) Re-arrange entries in signature "NEW" until there are no errors (mostly due to the requirement to define types ahead).


 * (5) Check structure "Old" and signature "OLD" for remaining errors and remove them.


 * (6) For each entry in signature "NEW" push these changes to the source code separatedly for each entry (the reasons are given below) as follows:
 * (6.1) Query & replace "Old.xxx" with "New.xxx".
 * (6.2) . The reason for this is: Isabelle/jEdit opens updated files like "yyy.zzz" as keeps "#yyy.zzz" in the file system; subsequent updates would involve both files.
 * (6.3) Remove the "yyy.zzz~" files, which Isabelle/jEdit saves for each updated file "yyy.zzz" (use the script "~ ~/src/Tools/isac/CLEANUP"); The reason for removal is: subsequent updates would involve these files, which leads to buffer overflows soon.
 * (6.4) continue with (6.1) for the next change.


 * (7) Push these changes to the test suite in a slightly different way:
 * (7.1) Switch to the test setup by running script "~ ~/zcoding-to-test"
 * (7.2) Look for errors of the "import" files in the  Window. Remove errors by switching back "~ ~/ztest-to-coding" and forth again with "~ ~/zcoding-to-test". This part of the test suite is not run with open structures. See above Pt.(2.1) and Pt.(3.1) respectively.
 * (7.3) Use script "~ ~/test/Tools/isac/CLEANUP" instead of "~ ~/src/Tools/isac/CLEANUP".
 * (7.4) Add "open "New" to "Test_Isac.thy" and "Test_Some.thy".
 * (7.5) All tests run by "ML_File" in "Test_Isac.thy" usually run out of the box, because all structures are opened.