Policies for the test suite

Isac's test suite primarily checks interactions with the front-end (this is different from Isabelle, which primarily checks specifics of the PIDE document model). The tests started from data used for coding, and still some such tests are not yet removed from the production code. Later tracing down bugs was saved and brought to a standard format, which should make re-use with new bugs convenient. A first systematic part of the test suite is "~ ~/test/Tools/isac/Minisubpbl/*". This is run after testing basics and before testing the Interpreter.

Rules and formats for tests

 * Make tests as independent as possible: copying a test from "test/../file.sml" to "test/Tools/isac/Test_Isac.thy" should be possible without worrying about initialisation of test arguments (particularly important when comparing functions within different ISAC versions).


 * Identification of tests: each test gets a header, repeated 3 times for visibility in the Output window in case the system cannot locate an error. This header is 78 characters long, starts a text after 11 "-" and is also listed in the heading contents area; an example is below.

"--- fun identifier ";
 * Identify test of specific functions which come into tests for several reasons: in/output is demonstrated; another test creates different behaviour of this function in different ISAC versions, etc. Such tests are headed like that


 * Re-use test code: when investigating a function "xxxxx" one can search for "--- fun xxxxx" or for "~ ~ ~ fun xxxxx,", see below


 * Stepwise checking a test (usual in comparison with an elder version): Stepwise execution of functions is uneasy without debugger. Sometimes the work is worth to be kept. In order to ease re-using such work ISAC introduces the following standard (see the templates in "Test_Some.thy") for a function "xxx":

"(*xxx aarg1 .. aargm*) "~ ~ ~ ~ ~ fun xxx, args:"; val (farg1, .., fargm) = (aarg1, .., aargm); "(*xxx arg1..argn ERROR ...*) "~ ~ ~ ~ ~ and xxx, args:"; val = ; "~ ~ ~ ~ ~ to xxx return val:"; val = ;

Before descending into function "xxx", keep the call in a comment (with the actual arguments "aarg1 .. aargm". This simplifies two actions later: check the correctness of the arguments in "val (farg1, .., fargm) = (aarg1, .., aargm)" and review the return value of "xxx".

The first line is used (with the exact number of "~", and without blanks in between -- enforced by this wiki) for calling a function, which is stepped into; "args" comprises all arguments (which require "," in between, if copied from the source code). The function's identifier is inserted before the "," with exactly one blank.

The second line is used, if the respective function call is copied from a mutual recursive definition with "and".


 * In case the test protocols a search for an error, the comment holds the respective error message as well

"(*xxx aarg1 .. aargm ERROR ...*) "~ ~ ~ ~ ~ fun xxx, args:"; val (farg1, .., fargm) = (aarg1, .., aargm); "(*xxx arg1 .. argn  ERROR ...*) "~ ~ ~ ~ ~ and xxx, args:"; val = ;

Directory structure and location of test files
Location is determined by respective source files, but there are exceptions, "ADDTESTS" and "Minisubpbl".


 * (1) test files reside in some subdirectory of "test/..." parallel to the position of the file with the tested code in "src/..."


 * (2) test with extend the KEStore require a specific setup, see for instance changeset e8d9d194a96f:
 * (2.1) "setup {* ... *}" in file "/test/../filename.thy" ("filename.thy" with small leading letter: NOT Isabelle coding standard)
 * (2.2) the respective SML code is in "/test/../filename.sml" (in order to keept the "imports" phase of Test_Isac.thy short)


 * (3) tests outside the directory structure of source files are:
 * (3.1) ADDTESTS: most of these tests involve "*.thy" files themselves
 * (3.1) Minisubpbl: decomposes a minimal example (with sub-problem) into functions calls. In case you encounter errors with functions from "Interpre/" and later, then you might consider to refine these tests within the appropriate file. In case you find the culprit in a function defined in "ProgLang/" or earlier, locate a new test in the respective test file, where the function is defined.

Regression testing
Since tests have the same file structure as the sources, tests follow the sequence of bootstrapping. So it is advisable to fix errors in this sequence: this avoids to dig through too many layers of code in tests late in the sequence, because low level functions have been defined, and also tested, first.


 * (0) Test_Isac.thy requires specific handling for open/close structures as explained here.


 * (1) Run the whole test suite and try to figure out first from the pattern of red marks over the tests and from their origin in the respective test files, what the reason could be. The most important decision is, where to go into details: in "Minisubpbl/*" (see (2)) or somewhere else. Going into details (according to (4)) can be very tricky, investing time in this step might pay off.


 * (2) In case you encounter errors somewhere from the initial imports (e.g. from "ADDTESTS/All_Ctxt"), then out-comment these and fix the errors down into "Minisubpbl/*" first. See this changeset.

https://intra.ist.tugraz.at/hg/isa/rev/107330cc8b6a
 * (3) Fixing errors down to "Minisubpbl/*" requires activation of some setups in the initial imports, e.g. see

ML {* *} ML {* ... insert here and then bisect with "*} ML {*" ... *} ML {* *}
 * (4) If you encounter an error in Test_Isac.thy at a certain ML_file xxx.sml then
 * (4.1) open the file "test/../xxx.sml" by click in Test_Isac.thy and look if you see a red error icon (!) at the left margin; if there is none then
 * (4.2) look to the Output window for the error from xxx.sml; if this doesn't help
 * (4.3) copy the respective test from xxx.sml into Test_Isac.thy (i.e. into ML {* ... *}) below ML_file "xxx.sml": test setup is present here, and you can watch the red (x) there at the left margin. If this doesn't help
 * (4.4) copy the whole test file xxx.xml into Test_Isac.thy and bisect using this structure:


 * (5) Sometimes is helpful to compare with an elder Isac version, most likely the most recent version known for running "Test_Isac.thy" without errors. In this case step into a function in the old version and at specific calls copy the test back to the new version -- and not the other way round (because you might follow a misleading trace in the new version with the error).

Run tests with shell scripts
The test suite provides access to definitions hidden by structures. For that purpose certain sections of the signatures contain the signature of such definitions out-commented. These out-commented section are conveniently activated and deactivated by shell scripts:
 * $ ./xcoding_to_test.sh activates the sections mentioned, i.e. they show respective definitions in the signature and thus provide access during testing.
 * $ ./xtest_to_coding.sh deactivates the sections.
 * $ ./xcoding_to_test.sh and $ ./ztest_to_coding.sh additionally open the appropriate Isabelle/jEdit.

Theories prepared for testing
The access to ML definitions is fairly opened by a specific setup described in Policies_for_the_test_suite. And there are three theories prepared for testing:
 * 1) /test/Tools/isac/Test_Isac.thy : Runs all tests, which requires about 3 min. as with 2018. Requires session Isac. Is run by ./zcoding_to_test.sh.
 * 2) /test/Tools/isac/Test_Some.thy : Is prepared for running ad-hoc test code. Requires session Isac. So it is used by ./xcoding_to_test.sh. The advantage with this setting is, that ML definitions are fairly opened, a disadvantage is, that  does not proceed throughout src/.
 * 3) /test/Tools/isac/Text_Theory.thy : Is prepared for running ad-hoc test code. Does not require a session prepared. Advantage and disadvantage are reverse in comparison to Test_Some.thy: many ML definitions are hidden by signatures, but  proceeds throughout src/.

Resource intensive tests in Test_Isac.thy
Some tests raise exception Size raised (line 171 of "./basis/LibrarySupport.sml") if run in x86_64_32 mode of Poly/ML 5.8 (which is set as default). This exception can be avoided by ML_system_64 = "true" in ~/.isabelle/isabisac/etc/preferences. A model is in the repository at /etc/preferences.

These preferences have drawbacks, however:
 * they claim more memory such that Isabelle instances cannot run in parallel.
 * they do not reach Build_Isac.thy hanging in Build_Thydata.thy, see there.

So default for Build_Isac.thy and for general testing is Test_Isac_Short.thy is x86_64_32 mode. From time to time full testing in Test_Isac.thy is recommended. For that purpose
 * set ML_system_64 = "true" in '~/.isabelle/isabisac/etc/preferences''.