Tutorial for setting up Rose/Cylc in order to run JULES on CEDA JASMIN

  1. First, complete the JASMIN setup at the following site.
    https://help.jasmin.ac.uk/category/158-getting-started
  2. Request group workspace privilege and make sure that you receive group workspace privilege for ‘jules’ at this site. You might need to wait till after you get a JASMIN account for this.
    https://accounts.jasmin.ac.uk
  3. Ask your project leader for help in getting an MOSRS account.
  4. Once you do the initial JASMIN setup and once you have an MOSRS account, do the numbered steps 3-5 in the following guide:
    https://code.metoffice.gov.uk/trac/jules/wiki/RoseJULESonJASMIN
    When you do the numbered step 5 of that guide, you should use jasmin-sci1 or jasmin-sci2 instead of jasmin-cylc, and the bash files might need to be re-arranged slightly from how it is suggested to have them in these directions. We’re not using jasmin-cylc right now since it doesn’t have access to the graphical user interface and X-windows, which are necessary for this work.
    Once you do all of this, you should be able to ssh from your laptop to jasmin-sci1.ceda.ac.uk or jasmin-sci2.ceda.ac.uk without entering your password. This works best on campus at the University of Reading. Using a VPN off campus should also work, as long as the network setup identified in step #1 finds that you are a ‘.ac.uk’ client.
  5. Once you have your jules group workspace privilege, when logged in to jasmin-sci1.ceda.ac.uk , type ‘rosie go’ at the command line. In the rosie go GUI, search for the ‘multipleSites’ branch of the suite u-al752. Check out a copy of this suite by clicking on it once and selecting the button for checking out a suite. Once it is finished downloading the suite, you can quit ‘rosie go’. This suite, u-al752, was originally developed by Karina Williams (Met Office) and Anna Harper (U. Exeter), and it runs JULES models of 29 different FLUXNET2015 sites around the world. The trac site for this suite from Karina and Anna is here. Patrick McGuire (U. Reading) ported this suite from MONSooN to CEDA JASMIN.
  6. In the ~/roses/u-al752 directory, you can view the various Rose/Cylc setup files like suite.rc, rose-suite.conf, rose-suite.info, site/suite.rc.CEDA_JASMIN, app/jules/rose-app.conf, and app/fcm_make/file/fcm-make.cfg . You can use ‘vi’ or ‘more’ to view these files. Study the files for a little while. Note that all the JULES namelists are condensed to the file: app/jules/rose-app.conf .
    Now change your rose-suite.conf file with ‘vi’ so that it has the right myusername (in two different places):
    OUTPUT_FOLDER=’/work/scratch/myusername/fluxnet/run10/jules_output’
    PLOT_FOLDER=’/work/scratch/myusername/fluxnet/run10/peg/plots’
  7. In the ~/roses/u-al752 directory, type ‘rose edit’, and explore the suite in the GUI this time, but don’t make any changes and don’t press run. Then quit ‘rose edit’.
  8. In the ~/roses/u-al752 directory, type ‘rose suite-run’. This will pop up a small pygraph window, which you should close. Then it will pop up the gcylc window, and you should see your job being submitted in sequence. The fcm_make task should take about 10 minutes to complete. The various JULES tasks can take up to 1-2 hours to complete, and then there is the Python plotting routine, which is called when all the JULES tasks are finished. Wait for all this to finish. You can go away for a while or overnight, if necessary. If it is still not finished, you can right-mouse-click over the various entries and look at the job.err or job.out log files, etc. If the X-windows is cut off or the session ends unexpectedly, you can go back to the ~/roses/u-al752 directory, and type ‘rose sgc’ to view the gcylc GUI again.
  9. You might now be able to study (with ncinfo or Python, etc.) the netcdf output files in where you set them to be in your rose-suite.conf file, e.g. in:
    OUTPUT_FOLDER=’/work/scratch/myusername/fluxnet/run10/jules_output’.
    The plots are where you set them to be in your rose-suite.conf file, e.g. in:
    PLOT_FOLDER=’/work/scratch/myusername/fluxnet/run10/peg/plots’.
    You can view these PDF plots with ‘display filename.pdf’.
    You can compare your PDF plots to those in:
    /work/scratch/pmcguire/fluxnet/run10/peg/plots
    You can also look at the log files in
    ~/cylc-run/u-al752/log/job/1/jules/01/job.err
    ~/cylc-run/u-al752/log/job/1/jules/01/job.out
    ~/cylc-run/u-al752/log/job/1/fcm_make/01/job.err
    ~/cylc-run/u-al752/log/job/1/fcm_make/01/job.out
    You can view these with ‘vi’ or ‘more’. If you see anything in the log files that looks like there was something wrong, then you should investigate.
  10. You can look at the Python code (as defined in your rose-suite.conf directory). It’s copied in:
    ~/cylc-run/u-al752/bin/fluxnet_evaluation.py
    The JULES source code (FORTRAN) is copied in the subdirectories of:
    ~/cylc-run/u-al752/share/fcm_make/preprocess/
  11. So the next part of the tutorial is to make a modification to your suite and then using fcm to check in your changes to MOSRS. It can be a trivial modification like changing your plotting path in rose-suite.conf .
    But first (since you don’t have permission to commit changes to the original suite u-al752 multipleSites branch), you need to use ‘rosie go’ to check out a copy of the suite (with a new suite-number) to your account on JASMIN. There should be a copy/duplicate button in ‘rosie go’ to check out a copy of the suite with a new suite number. Push that button, and then quit ‘rosie go’ after it is done. Note the new suite number in your roses directory.
    Then you can make a change to the plotting path, for example.
    Next, you can do ‘fcm diff’ in your roses/suite-number directory. It should show the changes you made.
    Finally, you can check-in (a.k.a. ‘commit’) these changes to MOSRS with ‘fcm ci’ in your roses/suite-number directory. You need to add a comment to the log file before ‘fcm ci’ will let you proceed.