This is an introduction to the Testing Suite for big models developed at PIK RD3. For big models no standard continuous integration (CI) techniques can be deployed, while unit tests are not easy to apply and do not provide tests for the complete model workflow. We therefore developed a custom Testing Suite.
The Testing Suite tests several scenarios, including various output routines and interfaces with input data preparation. Further, it checks the model results for consistency.
It is fully automated (runs every weekend), documents its findings in a gitlab project, and sends a weekly email with each new report to the team developing the model.
A scheduled and recurring job runs on the cluster every weekend, starting model runs according to a predefined set of scenarios. Once the runs are finished the job creates an overview of the results using rs2, and (optionally) additionally runs compareScenarios comparing each run with the previous run with the same scenario name (the comparison PDF files are found in each scenario folder, they also contain comparisons with historical data). It also uploads all runs to the shinyResults::appResults app for easy and interactive viewing. Also here a comparison with historical data is found. Finally, using a gitlab integration hook an overview of the results is automatically reported to the developer team via email.
If the report contains runs that failed, or output that looks wrong (or has not been generated at all) someone has to take action. If there are cases for which you feel responsible or in the position to fix, please do so. This early warning system ensures stable workflows for all members of the team.
If you want your own scenarios to be included in future tests: For
REMIND, add your scenarios to the scenario_config.csv, add
AMT in the start column and simply commit it
to the model’s develop branch (similar for MAgPIE). They will be
automatically executed the next time the tests are run.
For wishes, changes etc, please contact RSE or open a new issue here: https://github.com/pik-piam/modelstats/issues