Update Isabelle version

This sequence of steps applies to the case, where ISAC's repository is updated to a new Isabelle release once: once the steps described below are done, all developers of Isabelle/Isac delete their respective workspace and follow the guidelines for setup at Development_Environment_for_ISAC_core (and do NOT update the Isabelle/Isac workspace).

ISAC's repository contains protocols of previous updates. See for instance, changesets 130688f277ba to changeset a556fcaf2ae3 at https://intra.ist.tugraz.at/hg/isa, i.e. https://intra.ist.tugraz.at/hg/isa/rev/130688f277ba : https://intra.ist.tugraz.at/hg/isa/rev/a556fcaf2ae3

Step 1: Save the current working directory
When updating Isac to a new Isabelle version (also later) it is useful to have the old Isabelle/Isac run for comparison with new versions. For this reason it is helpful to have at least the latest versions on your personal computer (presently Isabelle development has a release cycle of eight months).

Isabelle bundles all components since 2011, so going back to elder Isabelle versions within ISAC's repository is easy since 2011 (~/.isabelle/contrib/ holds the respective components).

We update from Isabelle2018 to Isabelle2019 beginning with these steps:


 * (1.1) update this webpage: 18/19 --> 19/20


 * (1.2) start from a state where Test_Isac.thy evaluates without errors, for instance changeset 76126151fa0f.


 * (1.3) ///summarize Isac development since the previous Isabelle update in Test_Isac.thy in analogy to the previous entries in section {* history of tests *}

$ hg commit -m "final isabisac18 on Isabelle2018"
 * (1.4) commit to the repository with a standardised log-entry

hg tag isabisac18
 * (1.5) tag the respective changeset

/usr/local$ sudo mv isabisac isabisac18
 * (1.6) save the current working directory:

Step 2: Adapt Isabelle's standard installation
Merging repositories of Isabelle and Isac causes dangers, so the Isac repository is trickily inserted into a standard installation of Isabelle since Isabelle2015.


 * (2.1) download the last official release from http://isabelle.in.tum.de to ~/tmp/

~/tmp$ tar -xvzpf Isabelle2019_app.tar.gz
 * (2.2) unpack with

/tmp$ mkdir /usr/local/isabisac ~/tmp$ cp -r Isabelle2019/* /usr/local/isabisac/ or /usr/local/$ sudo cp -r Isabelle2019/* isabisac/
 * (2.3) copy the sources to the installation location (isabisac has been moved to isabisac18)


 * (2.4) do NOT start Isabelle: this would cause problems later (even if the root directory already has the final name).

~/.isabelle$ rm -rf isabisac Messages like Bad component catalog file: "/home/wneuper/repos/isabisacREP/Admin/components/main" are caused by a corrupted isabisac
 * (2.5) remove the outdated heaps (if not done already by Step 1)

$ cd /usr/local/isabisac
 * (2.6) change to the new working directory indicated by $ in the sequel

Step 3: Copy ISAC's repository into Isabelle's files
Now we make an Isabelle/Isac working space ($ denotes /usr/local/isabisac$) and copy Isac into the already present Isabelle files.

$ sudo cp -r ../isabisac18/.hg*. $ sudo chown -R wneuper .hg* $ sudo chgrp -R wneuper .hg*
 * (3.1) copy repository from old installation into the new one

$ cp -r ../isabisac18/src/Tools/isac ./src/Tools/ $ cp -r ../isabisac18/test. $ cp -r ../isabisac18/doc-isac. $ cp ../isabisac18/README_WN. $ cp ../isabisac18/README_REPOSITORY. $ cp ../isabisac18/xcoding-to-test.sh. $ cp ../isabisac18/zcoding-to-test.sh. $ cp ../isabisac18/xtest-to-coding.sh. $ cp ../isabisac18/ztest-to-coding.sh. $ hg st The last command shows, that many Isabelle files have changed and that the Isac files are not listed (i.e. the Isac history is NOT affected by the following hg commit)
 * (3.2) copy Isac sources into the new installation

$ hg add
 * (3.2) add new Isabelle files to the repository (indicated by ? in the list created by hg st)

$ hg forget xxxxx         # do this file by file in thg You can use hg forget xxxxx also for non-existing directories xxxxx/.
 * (3.3) forget the old Isabelle files from the repository (indicated by ! in the list created by hg st). This is tiresome because hg forget xxxxx/* also affects files to be merged and files already added. Marking multiple files in thg and then forget by menue helps. So use 'hg forget' carefully !

hg commit -m "\- start update Isabelle2018 --> Isabelle2019" This is a large change set.
 * (3.4) do the first commit from the new working space with a specific comment

Step 4: Finish Step 1: make installation of old version usable
isabisac18$ hg commit -m "\- DO NOT PUSH FROM HERE: you are out of development"
 * (4.1) mark the saved repository (it will be still useful for coming back to tests on the old version later)

in file /urs/local/isabisac18/etc/settings: : if [ -z "$ISABELLE_IDENTIFIER" ]; then ISABELLE_HOME_USER="$USER_HOME/.isabelle/isabisac18" else ISABELLE_HOME_USER="$USER_HOME/.isabelle/isabisac18" fi : and commit with a specific comment isabisac18$ hg commit -m "define storage for installation-specific settings"
 * (4.2) specify the place for installation-specific settings (using gedit etc)
 * 1) The place for user configuration, heap files, etc.

/usr/local/isabisac18$ rm -rf ~/.isabelle/isabisac18 /usr/local/isabisac18$ ./bin/isabelle jedit & In case the saved installation does not work any more see Fix_kernel_installation.
 * (4.3) create this place from /usr/local/isabisac18 (NEVER try to change it, RATHER delete it, if existing for some reason..)

:
 * (4.4) beware the Isac frontend isac-java from unstable kernel versions and connect with the saved installation in isac-java/src/java/properties/BridgeMain.properties

Step 5: Copy libisabelle into the installation
libisabelle is a ML/Scala library "which talks to Isabelle" and is used by Isac for communication between the Isac kernel (ML) and the Isac front-end (in Java/Scala). The library is packed for other applications than Isac, so we need the following manual procedure ~/tmp$ tar -xvzpf libisabelle*
 * (5.1) Download the latest version, e.g. libisabelle 1.0.0-RC3, from the changelog and unpack

~/tmp/libisabelle-1.0.1$ sbt
 * (5.2) Using sbt download the appropriate tools (compiler, etc) and build the libisabelle project

sbt:root> cli/run --resources /tmp/libisabelle-protocol --internal --verbose This takes some time while retrieving resources via internet.
 * (5.3) Dump libisabelle's protocol into /tmp/libisabelle-protocol

sbt:root> setup/assembly In case of errors in steps (5.2-4) remove all directories created so far ~/tmp/libisabelle-1.0.1/ ~/.ivy2/ ~/.local/share/libisabelle /tmp/libisabelle-protocol/ and restart from step (5.2).
 * (5.4) Create the *.jar for the connection on the Java side (see Step 8) by

~/tmp/libisabelle-1.0.1/modules/libisabelle/src/main/isabelle$ ls -l * -rw-rw-r-- 1 wneuper wneuper 112 Okt  2  2018 Protocol_Main.thy -rw-rw-r-- 1 wneuper wneuper  90 Okt  2  2018 Protocol_Pure.thy -rw-rw-r-- 1 wneuper wneuper 251 Okt  2  2018 ROOT operations: total 24 -rw-rw-r-- 1 wneuper wneuper 284 Okt  2  2018 Basic.thy -rw-rw-r-- 1 wneuper wneuper 623 Okt  2  2018 HOL_Operations.thy -rw-rw-r-- 1 wneuper wneuper 4458 Okt 2  2018 ml_expr.ML -rw-rw-r-- 1 wneuper wneuper 1209 Okt  2  2018 ML_Expr.thy -rw-rw-r-- 1 wneuper wneuper 385 Okt  2  2018 refs.ML protocol: total 36 -rw-rw-r-- 1 wneuper wneuper 722 Okt  2  2018 Codec_Class.thy -rw-rw-r-- 1 wneuper wneuper 8179 Okt 2  2018 codec.ML -rw-rw-r-- 1 wneuper wneuper 1191 Okt  2  2018 Codec_Test.thy -rw-rw-r-- 1 wneuper wneuper 2182 Okt 2  2018 Common.thy -rw-rw-r-- 1 wneuper wneuper 3703 Okt 2  2018 protocol.ML -rw-rw-r-- 1 wneuper wneuper 2042 Okt  2  2018 Protocol.thy -rw-rw-r-- 1 wneuper wneuper 717 Okt  2  2018 ref_table.ML -rw-rw-r-- 1 wneuper wneuper  788 Okt  2  2018 typed_eval.ML
 * (5.5)  ### THIS HAS CHANGED SINCE libisabelle 2018 ### Identify the isabelle-part in the libisabelle sources, something like

$ rm -rf /usr/local/isabisac/libisabelle-protocol/* $ mkdir /usr/local/isabisac/libisabelle-1.0.1-protocol/ $ cd ~/tmp/libisabelle-1.0.1/modules/libisabelle/src/main/ ~/tmp/libisabelle-1.0.1/modules/libisabelle/src/main/$ cp -r isabelle/* /usr/local/isabisac/libisabelle-1.0.1-protocol/
 * (5.6) ### THIS HAS CHANGED SINCE libisabelle 2018 ### Copy libisabelle's protocol into Isac

$ export ISABELLE_VERSION=20YY
 * (5.7) set the environment variable required by libisabelle in the shell you build Isac to the appropriate year YY

$ ./bin/isabelle build -v -b Protocol This creates "Classy" and "Protocol" in ~/.isabelle/isabisac/heaps/..
 * (5.8) Create the session required for using libisabelle

$ hg commit -m "Isabelle2018->19: insert new libisabelle"
 * (5.9) Commit with a specific message

Step 6: Re-build Isabelle's heaps
Now everything is in place to build Isac again.

$ rm -rf ~/.isabelle/isabisac
 * (6.1) reset installation-specific settings  (NEVER try to change these, RATHER delete, if existing for some reason..)

in file /usr/local/isabisac/etc/settings: : if [ -z "$ISABELLE_IDENTIFIER" ]; then ISABELLE_HOME_USER="$USER_HOME/.isabelle/isabisac" else ISABELLE_HOME_USER="$USER_HOME/.isabelle/isabisac" fi : and in $ISABELLE_HOME_USER/etc/preferences (suggested by Makarius in Re: [isabelle] "isabelle build" for test-suite ? in order to fight exception Size raised (line 171 of "./basis/LibrarySupport.sml") in Test_Isac.thy): : ML_system_64 = "true" and commit with a specific comment hg commit -m "Isabelle2018->19: define storage for installation-specific settings"
 * (6.2) define storage for installation-specific settings (using gedit etc)
 * 1) The place for user configuration, heap files, etc.

$ ./bin/isabelle jedit & In case the installation does not work any more see Fix_kernel_installation.
 * (6.3) create the installation-specific settings from /usr/local/isabisac

file /ROOTS +src/Tools/isac +libisabelle-protocol +test/Tools/isac/ADDTESTS/session-get_theory/ and commit with a specific comment hg commit -m "Isabelle2018->19: notify Isabelle about Isac and libisabelle"
 * (6.4) notify Isabelle about Isac and libisabelle


 * (6.5) separate appearance of Isac from Isabelle

file /src/Pure/System/distribution.ML: structure Distribution = struct - val version = "Isabelle2019: June 2019"; + val version = "Isabelle2019: June 2019 + Isac"; val is_identified = true; - val is_official = true; + val is_official = false; end;

file /src/Pure/System/distribution.scala: package isabelle object Distribution { - val version = "Isabelle2019: June 2019" + val version = "Isabelle2019: June 2019 + Isac"; val is_identified = true - val is_official = true + val is_official = false } and commit with a specific comment hg commit -m "Isabelle2018->19: separate appearance of Isac from Isabelle"

hg commit -m "Isabelle2018->19: insert Isac's hooks in /src/Pure/thm.ML"
 * (6.6) insert Isac's hooks in ~ ~/src/Pure/thm.ML, which requires care in case there were major updates in Isabelle: Search /usr/local/isabisac18/Pure/thm.ML for WN and copy respective code from the old Isabelle to the new one. And commit with a specific comment

/ROOTS - src/Tools/isac + #src/Tools/isac - libisabelle-protocol + #libisabelle-protocol then compile Pure and HOL with updated thm.ML $ ./bin/isabelle jedit & Invasiveness of Isabelle's updates changes from version to version.
 * (6.7) re-build Isabelle/Isac's heaps, before outcomment entries in ROOT not yet existing

hg commit -m "Isabelle2018->19: Pure and HOL build with updated thm.ML"
 * (6.8) commit with a specific comment (after having edited 1 character)

Step 7: Build ISAC from the new Isabelle code
ISAC is deeply integrated in Isabelle and Isabelle is still under rapid development, so this step usually involves some efforts. Some general hints are helpful:
 * (7.1) Check Isac code incrementally following the dependencies shown in Build_Isac.thy


 * (7.2) Apply a change in Isabelle to the whole Isac code in src/Tools/isac when encountering an error caused by a change of Isabelle (e.g. if an Isabelle function changes to another module).


 * (7.3) In case an Isabelle function has changed
 * lookup /NEWS in the new Isabelle
 * search the function in the old Isabelle code and lookup a respective code piece in the new Isabelle.


 * (7.4) In case the Output window does not help copy all code of xxx.sml into the theory and bisect using this structure:

ML {* *} ML {* ... insert here and then bisect with "*} ML {*" ... *} ML {* *}

hg commit -m "-- finish update Isabelle2018 --> Isabelle2019 for Build_Isac.thy"
 * (7.5) If Build_Isac.thy works, finish with a specific commit message


 * (7.6) Test_Isac.thy requires specific handling introduced  here

hg commit -m "/- finish update Isabelle2018 --> Isabelle2019 for Test_Isac.thy"
 * (7.7) Finish updating /test/ to new Isabelle20YY with a specfic commit message

Step 8: Connect ISAC's Java front-end with the new version of the math-engine
As soon as Test_Isac.thy works again, Isac's overall development should continue with the new version. : ISABELLE_HOME=/usr/local/isabisac :
 * (8.1) Connect by updating isac-java/src/java/properties/BridgeMain.properties, for instance, by

libisabelle-master$ cp ./modules/setup/target/scala-2.12/setup-assembly-1.1.0-SNAPSHOT.jar ~/proto4/repos/isac-java/lib/
 * (8.2) Copy the *.jar created in step (5.4) into the prototype installation, for instance, by

isac-java >  >  > 
 * (8.3) Remove the old *.jar and add the new one to the libraries in eclipse


 * (8.4) Update the code in Isac's BridgeMain.java following the model of libisabelle's Hello_PIDE.java.

~/proto4/repos$ hg commit -m "-- connection to new math-engine on Isabelle20YY"
 * (8.5) Commit the new connection to the repository with a specific message


 * (8.6) The libisabelle repository is obsolete after creating libisabelle-protocol for the Isabelle-side(see Step 5) and setup-assembly*.jar as described in (8.2-3).