Tutorial for setting up Rose/Cylc in order to run JULES on CEDA JASMIN

  1. First, complete the JASMIN setup at the following site.
    https://help.jasmin.ac.uk/category/158-getting-started
  2. Request group workspace privilege and make sure that you receive group workspace privilege for ‘jules’ at this site. You might need to wait till after you get a JASMIN account for this.
    https://accounts.jasmin.ac.uk
  3. Ask your project leader for help in getting an MOSRS account.
  4. Once you do the initial JASMIN setup and once you have an MOSRS account, do the steps: (a)  ‘Configuring your own laptop or desktop machine’, (b) ‘Logging on to JASMIN’, and (c) ‘Modify configuration files on jasmin-cylc’  (skipping the ‘Running the suite’ tutorial),  in the following guide:
    https://code.metoffice.gov.uk/trac/jules/wiki/RoseJULESonJASMIN
    When you do the ‘Modify configuration files on jasmin-cylc’ section of that guide, you should use jasmin-cylc instead of jasmin-sci1 or jasmin-sci2 (as indicated in the instructions; users should now make changes to revert to jasmin-cylc from jasmin-sci1 or jasmin-sci2, contrary to previous instructions). We’re now using jasmin-cylc instead of jasmin-sci1 or jasmin-sci2, since the graphical user interface and X-windows have recently been enabled on jasmin-cylc. The GUI and X-windows are necessary for this work.
    Once you do all of this, you should be able to ssh -AX from your laptop to jasmin-cylc.ceda.ac.uk without entering your password. This works best on campus at the University of Reading. Using a VPN off campus should also work, as long as the network setup identified in step #1 finds that you are a ‘.ac.uk’ client.
  5. Once you have your jules group workspace privilege, when logged in to jasmin-cylc.ceda.ac.uk , type rosie go at the command line. In the rosie go GUI, search for suite u-al752, where you should use the ‘trunk’ version of this suite.  Check out a copy of this suite by clicking on it once and selecting the button for checking out a suite. Once it is finished downloading the suite, you can quit rosie go. This suite, u-al752, was originally developed by Karina Williams (Met Office) and Anna Harper (U. Exeter), and it runs JULES models of (up to) 75 different FLUXNET2015 sites around the world. The TRAC site for this suite from Karina and Anna is here, where example plots generated on MONSooN can be downloaded and viewed. Patrick McGuire (U. Reading) ported this suite from MONSooN to CEDA JASMIN.
  6. In the ~/roses/u-al752 directory, you can view the various Rose/Cylc setup files like suite.rc, rose-suite.conf, rose-suite.info, site/suite.rc.CEDA_JASMIN, app/jules/rose-app.conf, and app/fcm_make/file/fcm-make.cfg . You can use vi or more to view these files. Study the files for a little while. Note that all the JULES namelists are condensed to the file: app/jules/rose-app.conf .
  7. To find out about the JULES/FLUXNET suite settings, the first file to look at is rose-suite.conf where we can set up the number of spinups, state whether we want prescribed datasets, give the path to output and plots folders, the path to fluxnet datasets, etc. The second file to check is suite.rc where the list of fluxnet sites, choice of using prescribed datasets, TRIFFID, Phenology and soil carbon pools is set. The version of the FLUXNET dataset used by the suite is also given in suite.rc. It is important to realize that the 2nd file (suite.rc) reads in the values defined in the 1st file (rose-suite.conf). The third file to check is app/jules/rose-app.conf , where all the namelists are, and setting parameters for the JULES run take place. More information on the configuration of prescribed datasets for fluxnet sites (ancillaries, driving datasets, lai, etc) are give in app/jules/opt/ . By searching the key words in the files above (in the recommended order) many questions can be answered. The parameters in rose-app.conf can be searched in JULES Userguide website (for the different JULES version numbers (here: version 5.4)). For instance looking for ‘ignition_method’ can be carried out by following this link:
    http://jules-lsm.github.io/vn5.3/namelists/jules_vegetation.nml.html#JULES_VEGETATION::ignition_method
  8. Now change your rose-suite.conf file with vi so that it has the right myusername (in two different places):
    OUTPUT_FOLDER='/work/scratch/myusername/fluxnet/run11a/jules_output'
    PLOT_FOLDER='/work/scratch/myusername/fluxnet/run11a/peg/plots'
    Also, change the location:
    LOCATION='CEDA_JASMIN'
    and the suite_data:
    SUITE_DATA='/group_workspaces/jasmin2/jules/pmcguire/fluxnet/kwilliam/suite_data'
    and the JULES archive location (it is important to use jules.x_tr instead of jules.xm_tr):
    JULES_FCM='fcm:jules.x_tr'
  9. In the ~/roses/u-al752 directory, type rose edit, and explore the suite in the GUI this time, but don’t make any changes and don’t press run. Then quit rose edit.
  10. In the ~/roses/u-al752 directory, type rose suite-run. This will pop up the gcylc window, and you should see your job being submitted in sequence. The fcm_make task should take about 10 minutes to complete. The various JULES tasks can take up to 1-2 hours to complete, and then there is the Python plotting routine, which is called when all the JULES tasks are finished. Wait for all this to finish. You can go away for a while or overnight, if necessary. If it is still not finished, you can right-mouse-click over the various entries and look at the job.err or job.out log files, etc. If the X-windows is cut off or the session ends unexpectedly, you can go back to the ~/roses/u-al752 directory, and type rose sgc to view the gcylc GUI again.
  11. You might now be able to study (with ncinfo or python2.7, etc.; if you use python2.7, you might need to use jasmin-sci1 or jasmin-sci2 instead of jasmin-cylc in order to get the proper python libraries working.) the NETCDF output files in where you set them to be in your rose-suite.conf file, e.g. in:
    OUTPUT_FOLDER='/work/scratch/myusername/fluxnet/run11a/jules_output'
    The plots are where you set them to be in your rose-suite.conf file, e.g. in:
    PLOT_FOLDER='/work/scratch/myusername/fluxnet/run11a/peg/plots'
    You can view these PDF plots by first typing module load jaspy/2.7 and then typing display filename.pdf. Using jaspy version 2.7 is important since otherwise fcm doesn’t work, and this is needed in order to run some Rose/Cylc suites. You can also use ncview to study the NETCDF files as well.
    You can compare your PDF plots to those available at the FLUXNET TRAC site listed above.
    Here are a few examples from the FLUXNET TRAC site; for the most up-to-date ones and for other plotted variables, check at the FLUXNET TRAC site.
    FLUXNET EXAMPLE PLOT: Available Soil Moisture (top layer, JULES model only)
    FLUXNET EXAMPLE PLOT: Latent Heat Flux (LE)
    FLUXNET EXAMPLE PLOT: Sensible Heat Flux (SH)
    FLUXNET EXAMPLE PLOT: Gross Primary Production
    You can also look at the log files in
    ~/cylc-run/u-al752/log/job/1/jules/01/job.err
    ~/cylc-run/u-al752/log/job/1/jules/01/job.out
    ~/cylc-run/u-al752/log/job/1/fcm_make/01/job.err
    ~/cylc-run/u-al752/log/job/1/fcm_make/01/job.out
    You can view these with vi or more. If you see anything in the log files that looks like there was something wrong, then you should investigate.
  12. You might want to transfer some PDF plots (for example) from CEDA JASMIN to your own computer. One simple way to do this is to pull them (from CEDA JASMIN to your own Linux or Macintosh machine),
    by first typing similar commands to these on CEDA JASMIN:
    mkdir ~/run11a_plots
    cp -pr /work/scratch/myusername/fluxnet/run11a/peg/plots ~/run11a_plots
    We first had to copy the files from the scratch drive to your home directory (or alternatively to a group workspace), since the transferring to external computers doesn’t work from the scratch disk.
    Then type these commands on your own machine:
    mkdir run11a_plots
    scp -pr jasmin-xfer1.ceda.ac.uk:run11a_plots run11a_plots
  13. The plotting currently (24 September 2018) doesn’t work for for GPP for the US_WCr and ZM_Mon sites on CEDA JASMIN. These 2 sites have been disabled automatically. We think that the cause of this problem with these two sites is that a new version of iris for python (2.1) was installed in July, overriding the previous version 1.13. The function in the new version of iris on JASMIN that is called ‘aggregate_by’ doesn’t seem to handle the multi-year gaps of data in GPP for US_WCr or ZM_Mon very well.
  14. You can look at the Python code (as defined in your rose-suite.conf directory). It’s copied in:
    ~/cylc-run/u-al752/bin/fluxnet_evaluation.py
    The JULES source code (FORTRAN) is copied in the subdirectories of:
    ~/cylc-run/u-al752/share/fcm_make/preprocess/
  15. So the next part of the tutorial is to make a modification to your suite and then using fcm to check in your changes to MOSRS. It can be a trivial modification like changing your plotting path in rose-suite.conf .
    But first (since you don’t have permission to commit changes to the original suite u-al752 trunk version), you need to use rosie go to check out a copy of the suite (with a new suite-number) to your account on JASMIN. There should be a copy/duplicate button in rosie go to check out a copy of the suite with a new suite number. Push that button, and then quit rosie go after it is done. Note the new suite number in your roses directory.
    Then you can make a change to the plotting path, for example.
    Next, you can do fcm diff -r HEAD in your roses/suite-number directory. It should show the changes you made.
    Finally, you can check-in (a.k.a. ‘commit’) these changes to MOSRS with fcm ci in your roses/suite-number directory. You need to add a comment to the log file before fcm ci will let you proceed.