Tutorial for setting up Rose/Cylc in order to run JULES on CEDA JASMIN

  1. First, complete the JASMIN setup at the following site.
    https://help.jasmin.ac.uk/category/158-getting-started
  2. Request group workspace privilege and make sure that you receive group workspace privilege for ‘jules’ at this site. You might need to wait till after you get a JASMIN account for this.
    https://accounts.jasmin.ac.uk
  3. Ask your project leader for help in getting an MOSRS account.
  4. Once you do the initial JASMIN setup and once you have an MOSRS account, do the numbered steps 3-5 in the following guide:
    https://code.metoffice.gov.uk/trac/jules/wiki/RoseJULESonJASMIN
    When you do the numbered step 5 of that guide, you should use jasmin-sci1 or jasmin-sci2 instead of jasmin-cylc, and the bash files might need to be re-arranged slightly from how it is suggested to have them in these directions. We’re not using jasmin-cylc right now since it doesn’t have access to the graphical user interface and X-windows, which are necessary for this work.
    Once you do all of this, you should be able to ssh -AX from your laptop to jasmin-sci1.ceda.ac.uk or jasmin-sci2.ceda.ac.uk without entering your password. This works best on campus at the University of Reading. Using a VPN off campus should also work, as long as the network setup identified in step #1 finds that you are a ‘.ac.uk’ client.
  5. Once you have your jules group workspace privilege, when logged in to jasmin-sci1.ceda.ac.uk , type ‘rosie go’ at the command line. In the rosie go GUI, search for suite u-al752, where you should use the ‘trunk’ version of this suite. Check out a copy of this suite by clicking on it once and selecting the button for checking out a suite. Once it is finished downloading the suite, you can quit ‘rosie go’. This suite, u-al752, was originally developed by Karina Williams (Met Office) and Anna Harper (U. Exeter), and it runs JULES models of 29 different FLUXNET2015 sites around the world. The TRAC site for this suite from Karina and Anna is here, where example plots generated on MONSooN can be downloaded and viewed. Patrick McGuire (U. Reading) ported this suite from MONSooN to CEDA JASMIN.
  6. In the ~/roses/u-al752 directory, you can view the various Rose/Cylc setup files like suite.rc, rose-suite.conf, rose-suite.info, site/suite.rc.CEDA_JASMIN, app/jules/rose-app.conf, and app/fcm_make/file/fcm-make.cfg . You can use vi or more to view these files. Study the files for a little while. Note that all the JULES namelists are condensed to the file: app/jules/rose-app.conf .
    Now change your rose-suite.conf file with vi so that it has the right myusername (in two different places):
    OUTPUT_FOLDER='/work/scratch/myusername/fluxnet/run11a/jules_output'
    PLOT_FOLDER='/work/scratch/myusername/fluxnet/run11a/peg/plots'
    Also, change the location:
    LOCATION='CEDA_JASMIN'
    and the suite_data:
    SUITE_DATA='/group_workspaces/jasmin2/jules/pmcguire/fluxnet/kwilliam/suite_data'
    and the jules archive location:
    JULES_FCM='fcm:jules.x_br/dev/karinawilliams/r9227_add_gpp_unstressed_diagnostic'
  7. In the ~/roses/u-al752 directory, type rose edit, and explore the suite in the GUI this time, but don’t make any changes and don’t press run. Then quit rose edit.
  8. In the ~/roses/u-al752 directory, type rose suite-run. This will pop up a small pygraph window, which you should close. Then it will pop up the gcylc window, and you should see your job being submitted in sequence. The fcm_make task should take about 10 minutes to complete. The various JULES tasks can take up to 1-2 hours to complete, and then there is the Python plotting routine, which is called when all the JULES tasks are finished. Wait for all this to finish. You can go away for a while or overnight, if necessary. If it is still not finished, you can right-mouse-click over the various entries and look at the job.err or job.out log files, etc. If the X-windows is cut off or the session ends unexpectedly, you can go back to the ~/roses/u-al752 directory, and type rose sgc to view the gcylc GUI again.
  9. You might now be able to study (with ncinfo or python2.7, etc.) the netcdf output files in where you set them to be in your rose-suite.conf file, e.g. in:
    OUTPUT_FOLDER='/work/scratch/myusername/fluxnet/run11a/jules_output'
    The plots are where you set them to be in your rose-suite.conf file, e.g. in:
    PLOT_FOLDER='/work/scratch/myusername/fluxnet/run11a/peg/plots'
    You can view these PDF plots with display filename.pdf
    You can compare your PDF plots to those available at the FLUXNET TRAC site listed above.
    Here are a few examples from the FLUXNET TRAC site; for the most up-to-date ones and for other plotted variables, check at the FLUXNET TRAC site.
    FLUXNET EXAMPLE PLOT: Available Soil Moisture (top layer, JULES model only)
    FLUXNET EXAMPLE PLOT: Latent Heat Flux (LE)
    FLUXNET EXAMPLE PLOT: Sensible Heat Flux (SH)
    FLUXNET EXAMPLE PLOT: Gross Primary Production
    You can also look at the log files in
    ~/cylc-run/u-al752/log/job/1/jules/01/job.err
    ~/cylc-run/u-al752/log/job/1/jules/01/job.out
    ~/cylc-run/u-al752/log/job/1/fcm_make/01/job.err
    ~/cylc-run/u-al752/log/job/1/fcm_make/01/job.out
    You can view these with vi or more. If you see anything in the log files that looks like there was something wrong, then you should investigate.
  10. The plotting currently (24 September 2018) doesn’t work for for GPP for the US_WCr and ZM_Mon sites on CEDA JASMIN. You can disable these easily in the
    suite.rc file, and in:
    ~/roses/u-al752/bin/list_sites_fluxnet2015.txt
           You can see how I did the disabling in:
    ~pmcguire/roses/u-al752jasmin-sci1upd20
    We think that the cause of this problem with these two sites is that a new version of iris for python (2.1) was installed in July, overriding the previous version 1.13. The function in the new version of iris on JASMIN that is called ‘aggregate_by’ doesn’t seem to handle the multi-year gaps of data in GPP for US_WCr or ZM_Mon very well.
  11. You can look at the Python code (as defined in your rose-suite.conf directory). It’s copied in:
    ~/cylc-run/u-al752/bin/fluxnet_evaluation.py
    The JULES source code (FORTRAN) is copied in the subdirectories of:
    ~/cylc-run/u-al752/share/fcm_make/preprocess/
  12. So the next part of the tutorial is to make a modification to your suite and then using fcm to check in your changes to MOSRS. It can be a trivial modification like changing your plotting path in rose-suite.conf .
    But first (since you don’t have permission to commit changes to the original suite u-al752 trunk version), you need to use rosie go to check out a copy of the suite (with a new suite-number) to your account on JASMIN. There should be a copy/duplicate button in rosie go to check out a copy of the suite with a new suite number. Push that button, and then quit rosie go after it is done. Note the new suite number in your roses directory.
    Then you can make a change to the plotting path, for example.
    Next, you can do fcm diff -r HEAD in your roses/suite-number directory. It should show the changes you made.
    Finally, you can check-in (a.k.a. ‘commit’) these changes to MOSRS with fcm ci in your roses/suite-number directory. You need to add a comment to the log file before fcm ci will let you proceed.