Tutorial for running ILAMB on CEDA JASMIN

1. 1. 1. On this PPT (30 MB) , you can view on pp. 14-16 some examples of ILAMB scoring and Model/Benchmark comparisons that were run on the CEDA JASMIN computer. On p. 14, we show the Model/Benchmark scoring output of the Rose/Cylc ILAMB suite u-ba590 which was run with the model u-aw650, which is a JULES Earth Systems (ES) model, comparing to the ILAMB benchmarks. Suite u-ba590 is the University of Reading copy of the ILAMB base suite u-ba284 from the University of Exeter and the Met Office (Carolina Duran Rojas and Eddy Robertson). On p.15 you can see one (Evapotranspiration) of the many variables from p. 14, and the ILAMB suite plots the time-averaged global distribution of evapotranspiration with the Benchmark, the Model, and the Bias (difference of Model and Benchmark). On p.16, we show similar Model/Benchmark scoring output for four different University of Reading JULES models (u-as052B, u-av764, u-av768, and u-aw198), with the different soil and spinup parameters as summarized on the slide. A different ILAMB suite was used for each of these 4 Model/Benchmark comparisons (u-ba594D, u-ba599, u-ba600, and u-ba601). The output of these 4 ILAMB suites was combined together with suite u-ba597 (currently the latest version is u-bb897), which made the scoring comparison plot shown on the slide. For these 4 University of Reading Models, we only have a few of the variables that were output for the original ILAMB suite u-ba590 for model u-aw650. On p. 17 of the PPT is a broken link that points to the data in this link , which you can explore the hyperlinks from, to study the plots for the various variables shown on the front page of the link. The hyperlinks are visible by pulling down on the triangles on the right hand side of the ‘Results Table’ tab, and show things like FLUXNET, GBAF, and GPCC Benchmarks and their model comparisons. These hyperlinks point to plots like shown on p. 15 of the PPT.
      2. To run ILAMB suites like u-ba594 to compare a model with multiple variables in monthly-averaged NETCDF format like from u-as052 on the CEDA JASMIN computer, you will first need to get your CEDA JASMIN account setup. Complete the first four steps of this tutorial for running Rose/Cylc on CEDA JASMIN. If you don’t have a UK institutional email address, then it will likely be impossible to complete step 3 to get an MOSRS repository account and you will need additional help to complete step 4.
      3. Once you have completed steps 1-4 of the tutorial in step 2 above, if you have MOSRS access, then you can do step 5 of the tutorial, but doing a rosie go with a suite-id of u-bb897 instead of u-al752. If you want to save your changes of u-bb897 into a new suite in the MOSRS repository, then you should make a copy of the suite with rosie go instead of checking out that suite. If you do not have MOSRS access, then you can copy a version of the suite to your directory as follows:
         mkdir ~/roses
cd ~/roses
cp -pr ~pmcguire/roses/u-bb897 .
      4. To run this suite, cd ~/roses/u-bb897 and modify the rose-suite.conf file to have your myusername:
         ILAMB_MODEL_DIR='/work/scratch-nopw/myusername/ILAMB_output/JULES_PROCESSED_u-as052W/'
ILAMB_RESULTS='/work/scratch-nopw/myusername/ILAMB_output/RESULTS_u-as052W/'
JULES_TMP_PROCESSED='/work/scratch-nopw/myusername/ILAMB_output/JULES_TMP_PROCESSED_u-as052W/'
RUN_ILAMB=false
RUN_ILAMB_PRE_PROC=true
The RUN_ILAMB can be set to true if this is your only model that you want to compare to the benchmarks. Then, if you do not have MOSRS access, you will need to comment out the lines of code in rose-suite.conf so that fcm doesn’t go to MOSRS archive to get the jules.py file:
         #[file:app/jules_for_ilamb/file/jules.py]
#source=fcm:jules.x_br/pkg/karinawilliams/r6715_python_packages/share/jules.py@11084
         and if you do not have MOSRS access, you will also need to copy this code at the command-line:
         cp -p ~pmcguire/Packages/JULES/r6715_python_packages/share/jules11084.py ~/roses/u-bb897/app/jules_for_ilamb/file/jules.py
         where the previous line is all one line.
         Then you should be able to type rose suite-run at the command prompt from the ~/roses/u-bb897 directory to run the suite.
         This will pop up a small pygraph window, which you should close. Then it will pop up the gcylc window, and you should see your job being submitted in sequence. The jules_for_ilamb task should take about 10-30 minutes to complete. The various run_ilamb task can take up to 1-2 hours to complete. Wait for all this to finish. You can go away for a while or overnight, if necessary. If it is still not finished, you can right-mouse-click over the various entries and look at the job.err or job.out log files, etc. If the X-windows is cut off or the session ends unexpectedly, you can go back to the ~/roses/u-bb897 directory, and type rose sgc to view the gcylc GUI again.
      5. For the current version of the u-bb897 suite, we have also added a Soil Moisture Benchmark dataset from ESA CCI that we obtained from Alberto Martinez de la Torre. Alberto had done the monthly averaging from the daily data of the ESA CCI soil moisture measurements in m3/m3 at 0.25 degrees spatial resolution. We rebinned that at 0.5 degrees resolution for ILAMB using cdo remapnn. The JULES models that we ran (i.e., u-as052) output the smc_avail_top diagnostic variable, which integrated over the top 1 meter of soil (using the default JULES zsmc value of 1 meter). To compare the JULES model (which outputs the soil moisture in kg/m2), we converted the modelled soil moisture to the same units as ESA CCI by dividing by the water density and also by the layer thickness of 1 m. This produced values that were much lower than ESA CCI. The ESA CCI measurements are only sensitive to the top 0.15 meters of soil. We also tried the same ILAMB run by dividing by 0.15 m instead of 1 m, but this produced values of the JULES model which were too high compared to ESA CCI. The value of 0.30 m is used here for the layer thickness for converting the JULES modelled soil moisture to the ESA CCI units. This is reasonably close, but as you can see from the ILAMB comparison, the JULES modeled soil moisture still doesn’t compare that well to the ESA CCI measurements. More work needs to be done. JULES can output thinner layers without using the integration over 1 m, for example. And these modeled soil moisture values for thinner layers can then be used for comparing to ILAMB.
      6. You can study the output by copying your data from the ILAMB_RESULTS directory (that you pointed to above) to your own computer, and then pointing your browser to the index.html file in your computer’s ILAMB_RESULTS directory. To transfer the data off of JASMIN, you can:
         (on JASMIN) :
         mkdir ~/ILAMB/
cp -pr /work/scratch-nopw/myusername/ILAMB_output/RESULTS_u-as052W  /home/users/myusername/ILAMB/.
         where the previous line is all one line and where you have replaced myusername with your user name.
         You normally have a 100GB quota in your JASMIN home directory. If you don’t have enough space in your home directory, you can put it in some group workspace if you have access and space. There isn’t a whole lot of space in the jules group workspace. And you can’t transfer from scratch to another computer. And then
         (on your computer):
         scp -pr jasmin-xfer1.ceda.ac.uk:/home/users/myusername/ILAMB/RESULTS_u-as052W PATH_ON_YOURCOMPUTER
         where the previous line is all one line and you have replaced myusername with your user name as before and you have given a PATH_ON_YOUR_COMPUTER value (which could be just a dot .).
      7. To change the model, you first might want to make a copy of the original roses suite directory to another name, and after changing directory to the new suite directory, you will need to change the
         JULES_OUTPUT_DIR='/gws/nopw/j04/jules/pmcguire/jules_global_monthly_mean/Euro44_bvv_nancil_CTL-BCJ-GL_jules-vn4.9p_u-as052globeE/'
         to point to a directory with your model in it, where you have replaced both the overall path and the model directory at the end. You will need to transfer your model data to this new directory with scp.
         You will also need to change:
         ILAMB_MODEL_DIR='/work/scratch-nopw/pmcguire/ILAMB_output/JULES_PROCESSED_newmodelname/'
ILAMB_RESULTS='/work/scratch-nopw/pmcguire/ILAMB_output/RESULTS_newmodelname/'
JULES_SUITE_ID='newsuitename'
JULES_TMP_PROCESSED='/work/scratch-nopw/pmcguire/ILAMB_output/JULES_TMP_PROCESSED_newmodelname/'
         where you have replaced newmodelname and newsuitename appropriately.
         Again, be careful about using the jules group workspace, since it is rather full. You can use your home directory possibly or the scratch drive. You can do an ncinfo on your monthly-averaged NETCDF files in this directory to compare the variables to those in the University of Reading models’ NETCDF files. If you don’t have variables of similar names, then the ILAMB suite when you rerun it with rose suite-run, may not work without further modification. You might want to make a copy of your directory (or check-in to MOSRS) to another directory in ~/roses prior to changing your model.
      8. To compare multiple models, we used for p. 16 of the PPT a suite with the number u-bb897. The main things that we did in there was (i) make soft links at the command-line to the ILAMB_MODEL_DIR names for each of the runs we did
         mkdir /work/scratch-nopw/pmcguire/ILAMB_output/four_models
cd /work/scratch-nopw/pmcguire/ILAMB_output/four_models
ln -s /work/scratch-nopw/pmcguire/ILAMB_output/JULES_PROCESSED_u-as052B/
ln -s /work/scratch-nopw/pmcguire/ILAMB_output/JULES_PROCESSED_u-av764/
ln -s /work/scratch-nopw/pmcguire/ILAMB_output/JULES_PROCESSED_u-av768
ln -s /work/scratch-nopw/pmcguire/ILAMB_output/JULES_PROCESSED_u-aw198
         and then (ii) pointing the ILAMB_MODEL_DIR to this four_models directory and changing ILAMB_RESULTS, RUN_ILAMB, and RUN_ILAMB_PRE_PROC:
         ILAMB_MODEL_DIR='/work/scratch-nopw/pmcguire/ILAMB_output/four_modelsL/'
ILAMB_RESULTS='/work/scratch-nopw/pmcguire/ILAMB_output/RESULTS_GL2L/all2/'
RUN_ILAMB=true
RUN_ILAMB_PRE_PROC=false
In the JULES_PROCESSED model directories above, each of those were created when we ran u-ba594, u-ba599, u-ba600, and u-ba601, respectively. For testing purposes, you can run the suite u-bb897 with JULES_PROCESSED directories in the /work/scratch/pmcguire path, but if you rerun the models for yourself, then you can replace these with your own JULES_PROCESSED directories for the different models.