(work in progress)<\/p>\n
In the SGE resource manager on the met-cluster and the maths-cluster, a job is simply requested with a number of CPU slots allocated to it. On those clusters it is up to the user to spawn required processes, and it is user\u2019s responsibility to not oversubscribe their allocation. SLURM offers more help and flexibility in starting parallel jobs. It will also forcibly limit the resources available to the job, to those specified in the job allocation.<\/p>\n
In SLURM, a job can consist of multiple job slices. A job slice is a command or a script. In the simplest case, the execution of the commands in the job script itself is the only job slice; an example is the serial batch job script above. Other job slices within a batch script can be started using the srun command. Jobs slices can run either in parallel or sequentially within the job allocation.<\/p>\n
A task can be interpreted as an instance of a job slice. SLURM can start a number of identical tasks in parallel for each job slice, as specified with the flag \u2018\u2014ntasks\u2019 for that job slice i.e. for the job script as a whole, or for the srun call.<\/p>\n
A task can have more than one CPU cores allocated, such that the user application can spawn more processes and threads on its own. For example, if we run an openMP multi-threaded job, or an application like R or Matlab, which might be starting more processes or threads on its own, we just request one task to start the application, but with\u00a0the suitable number of CPUs for this single task.<\/p>\n
An example of a multi task job is an MPI job. We just specify the number of tasks we want\u00a0with \u2018\u2014ntasks\u2019 and SLURM will start parallel MPI processes for us. Those tasks might use more than one CPU per task if they are multi-threaded.<\/p>\n
To have more job slices we use more calls to the srun command in the job script. For example, we can have a job consisting of a data producing slice, possibly with many parallel tasks, running in parallel with a data collector slice, again with many tasks, and many CPUs per task if needed.<\/p>\n
This is an example of 16-way MPI job.<\/p>\n
#!\/bin\/bash\r\n\r\n#SBATCH --ntasks=16\r\n#SBATCH --cpus-per-task=1\r\n#SBATCH --nodes=1-1\r\n#SBATCH --job-name=test_mpi\r\n#SBATCH --output=myout.txt\r\n#SBATCH --time=120:00\r\n#SBATCH --mem-per-cpu=512\r\n\r\nmodule load MPI\/mpich\/gcc\/3.2.1\r\nsrun cd 2>\/dev\/null #a workaround, needed when node powers up\r\nsrun myMPIexecutable.exe<\/pre>\nThe above script requests16 tasks for 16 MPI processes. Typically it is better to have all the processes runing on the same node, it is requested with ‘–nodes=1-1’. In SLURM, the number of tasks represents the number of instances the command is run, typically done by a single srun command. In the above script, we have 16 identical processes, hence this is a slice with 16 tasks and not just one task using 16 CPU cores.<\/p>\n
The mpich library version loaded by the module command is built with SLURM support. The srun command will take care of creating and managing MPI processes, it replaces the mpirun or mpiexec commands.<\/p>\n
An example of an MPI program and job script can be found here:<\/p>\n
\/share\/slurm_examples\/mpi<\/span><\/p>\n
Similar like in the case of serial jobs, an MPI process (task) gets access to a whole physical core, when possible, and then it is counted as two CPUs in Slurm.<\/p>\n
Shared memory batch jobs (e.g. openMP)<\/h2>\n
#!\/bin\/bash\r\n\r\n#SBATCH --ntasks=1\r\n#SBATCH --cpus-per-task=16\r\n#SBATCH --threads-per-core=1\r\n#SBATCH --job-name=test_smp\r\n#SBATCH --output=myout.txt\r\n#SBATCH --time=120:00\r\n#SBATCH --mem-per-cpu=512\r\n\r\nexport OMP_NUM_THREADS=16\r\n.\/a.out<\/pre>\nIn the above script, one task is requested with 16 CPU cores allocated for this task. The executable can use up to 16 CPU cores for its threads or processes. In a similar fashion, parallel Matlab jobs (not tested) can be launched (with Matlab parallel toolbox, but without Matlab parallel server), or any other applications using multiple CPUs and managing them on their own.<\/p>\n
Using ‘–cpus-per-task’ is a bit tricky because of hyperthreading. For Slurm the CPU is a logical CPU (hardware thread). In RACC Slurm is configured to always allocate a whole physical core to a task. But, in case of ‘–cpus-per-task’, we are counting Slurm’s CPUs i.e. in case of processors with hyperthreading these are logical CPUs (hardware threads). On most physical nodes we have there is hyperthreading and there are two logical CPUs per physical core. On the VM nodes the CPUs are hardware threads allocated by the hypervisor. If you are happy to count CPU as hardware threads that’s easy and consistent, in both cases. However, often it is better to run just one thread per physical core, and then some customization of the job, depending on the compute node capability, is needed.<\/p>\n
<\/p>\n","protected":false},"excerpt":{"rendered":"
(work in progress) Parallel batch jobs Parallel jobs in SLURM In the SGE resource manager on the met-cluster and the maths-cluster, a job is simply requested with a number of CPU slots allocated to it. On those clusters it is up to the user to spawn required processes, and it is user\u2019s<\/p>\n","protected":false},"author":16,"featured_media":0,"template":"","meta":{"_acf_changed":false,"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"__cvm_playback_settings":[],"__cvm_video_id":"","_links_to":"","_links_to_target":""},"kbe_taxonomy":[],"kbe_tags":[],"class_list":["post-5770","kbe_knowledgebase","type-kbe_knowledgebase","status-publish","hentry"],"acf":[],"yoast_head":"\n
Reading Academic Computing Cluster - parallel batch jobs - Academic Computing Team<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/research.reading.ac.uk\/act\/knowledgebase\/reading-academic-computing-cluster-parallel-batch-jobs\/\" \/>\n<meta property=\"og:locale\" content=\"en_GB\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Reading Academic Computing Cluster - parallel batch jobs - Academic Computing Team\" \/>\n<meta property=\"og:description\" content=\"(work in progress) Parallel batch jobs Parallel jobs in SLURM In the SGE resource manager on the met-cluster and the maths-cluster, a job is simply requested with a number of CPU slots allocated to it. On those clusters it is up to the user to spawn required processes, and it is user\u2019s\" \/>\n<meta property=\"og:url\" content=\"https:\/\/research.reading.ac.uk\/act\/knowledgebase\/reading-academic-computing-cluster-parallel-batch-jobs\/\" \/>\n<meta property=\"og:site_name\" content=\"Academic Computing Team\" \/>\n<meta property=\"article:modified_time\" content=\"2020-05-25T14:55:18+00:00\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Estimated reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"4 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/research.reading.ac.uk\/act\/knowledgebase\/reading-academic-computing-cluster-parallel-batch-jobs\/\",\"url\":\"https:\/\/research.reading.ac.uk\/act\/knowledgebase\/reading-academic-computing-cluster-parallel-batch-jobs\/\",\"name\":\"Reading Academic Computing Cluster - parallel batch jobs - Academic Computing Team\",\"isPartOf\":{\"@id\":\"https:\/\/research.reading.ac.uk\/act\/#website\"},\"datePublished\":\"2020-05-25T12:04:40+00:00\",\"dateModified\":\"2020-05-25T14:55:18+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/research.reading.ac.uk\/act\/knowledgebase\/reading-academic-computing-cluster-parallel-batch-jobs\/#breadcrumb\"},\"inLanguage\":\"en-GB\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/research.reading.ac.uk\/act\/knowledgebase\/reading-academic-computing-cluster-parallel-batch-jobs\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/research.reading.ac.uk\/act\/knowledgebase\/reading-academic-computing-cluster-parallel-batch-jobs\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/research.reading.ac.uk\/act\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Knowledgebase\",\"item\":\"https:\/\/research.reading.ac.uk\/act\/knowledgebase\/\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"Reading Academic Computing Cluster – parallel batch jobs\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/research.reading.ac.uk\/act\/#website\",\"url\":\"https:\/\/research.reading.ac.uk\/act\/\",\"name\":\"Academic Computing Team\",\"description\":\"\",\"publisher\":{\"@id\":\"https:\/\/research.reading.ac.uk\/act\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/research.reading.ac.uk\/act\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-GB\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/research.reading.ac.uk\/act\/#organization\",\"name\":\"University of Reading\",\"url\":\"https:\/\/research.reading.ac.uk\/act\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-GB\",\"@id\":\"https:\/\/research.reading.ac.uk\/act\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/research.reading.ac.uk\/act\/wp-content\/uploads\/sites\/2\/2017\/08\/cropped-University_of_Reading_shield-1.png\",\"contentUrl\":\"https:\/\/research.reading.ac.uk\/act\/wp-content\/uploads\/sites\/2\/2017\/08\/cropped-University_of_Reading_shield-1.png\",\"width\":512,\"height\":512,\"caption\":\"University of Reading\"},\"image\":{\"@id\":\"https:\/\/research.reading.ac.uk\/act\/#\/schema\/logo\/image\/\"}}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Reading Academic Computing Cluster - parallel batch jobs - Academic Computing Team","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/research.reading.ac.uk\/act\/knowledgebase\/reading-academic-computing-cluster-parallel-batch-jobs\/","og_locale":"en_GB","og_type":"article","og_title":"Reading Academic Computing Cluster - parallel batch jobs - Academic Computing Team","og_description":"(work in progress) Parallel batch jobs Parallel jobs in SLURM In the SGE resource manager on the met-cluster and the maths-cluster, a job is simply requested with a number of CPU slots allocated to it. On those clusters it is up to the user to spawn required processes, and it is user\u2019s","og_url":"https:\/\/research.reading.ac.uk\/act\/knowledgebase\/reading-academic-computing-cluster-parallel-batch-jobs\/","og_site_name":"Academic Computing Team","article_modified_time":"2020-05-25T14:55:18+00:00","twitter_card":"summary_large_image","twitter_misc":{"Estimated reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/research.reading.ac.uk\/act\/knowledgebase\/reading-academic-computing-cluster-parallel-batch-jobs\/","url":"https:\/\/research.reading.ac.uk\/act\/knowledgebase\/reading-academic-computing-cluster-parallel-batch-jobs\/","name":"Reading Academic Computing Cluster - parallel batch jobs - Academic Computing Team","isPartOf":{"@id":"https:\/\/research.reading.ac.uk\/act\/#website"},"datePublished":"2020-05-25T12:04:40+00:00","dateModified":"2020-05-25T14:55:18+00:00","breadcrumb":{"@id":"https:\/\/research.reading.ac.uk\/act\/knowledgebase\/reading-academic-computing-cluster-parallel-batch-jobs\/#breadcrumb"},"inLanguage":"en-GB","potentialAction":[{"@type":"ReadAction","target":["https:\/\/research.reading.ac.uk\/act\/knowledgebase\/reading-academic-computing-cluster-parallel-batch-jobs\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/research.reading.ac.uk\/act\/knowledgebase\/reading-academic-computing-cluster-parallel-batch-jobs\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/research.reading.ac.uk\/act\/"},{"@type":"ListItem","position":2,"name":"Knowledgebase","item":"https:\/\/research.reading.ac.uk\/act\/knowledgebase\/"},{"@type":"ListItem","position":3,"name":"Reading Academic Computing Cluster – parallel batch jobs"}]},{"@type":"WebSite","@id":"https:\/\/research.reading.ac.uk\/act\/#website","url":"https:\/\/research.reading.ac.uk\/act\/","name":"Academic Computing Team","description":"","publisher":{"@id":"https:\/\/research.reading.ac.uk\/act\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/research.reading.ac.uk\/act\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-GB"},{"@type":"Organization","@id":"https:\/\/research.reading.ac.uk\/act\/#organization","name":"University of Reading","url":"https:\/\/research.reading.ac.uk\/act\/","logo":{"@type":"ImageObject","inLanguage":"en-GB","@id":"https:\/\/research.reading.ac.uk\/act\/#\/schema\/logo\/image\/","url":"https:\/\/research.reading.ac.uk\/act\/wp-content\/uploads\/sites\/2\/2017\/08\/cropped-University_of_Reading_shield-1.png","contentUrl":"https:\/\/research.reading.ac.uk\/act\/wp-content\/uploads\/sites\/2\/2017\/08\/cropped-University_of_Reading_shield-1.png","width":512,"height":512,"caption":"University of Reading"},"image":{"@id":"https:\/\/research.reading.ac.uk\/act\/#\/schema\/logo\/image\/"}}]}},"_links":{"self":[{"href":"https:\/\/research.reading.ac.uk\/act\/wp-json\/wp\/v2\/kbe_knowledgebase\/5770","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/research.reading.ac.uk\/act\/wp-json\/wp\/v2\/kbe_knowledgebase"}],"about":[{"href":"https:\/\/research.reading.ac.uk\/act\/wp-json\/wp\/v2\/types\/kbe_knowledgebase"}],"author":[{"embeddable":true,"href":"https:\/\/research.reading.ac.uk\/act\/wp-json\/wp\/v2\/users\/16"}],"version-history":[{"count":2,"href":"https:\/\/research.reading.ac.uk\/act\/wp-json\/wp\/v2\/kbe_knowledgebase\/5770\/revisions"}],"predecessor-version":[{"id":5789,"href":"https:\/\/research.reading.ac.uk\/act\/wp-json\/wp\/v2\/kbe_knowledgebase\/5770\/revisions\/5789"}],"wp:attachment":[{"href":"https:\/\/research.reading.ac.uk\/act\/wp-json\/wp\/v2\/media?parent=5770"}],"wp:term":[{"taxonomy":"kbe_taxonomy","embeddable":true,"href":"https:\/\/research.reading.ac.uk\/act\/wp-json\/wp\/v2\/kbe_taxonomy?post=5770"},{"taxonomy":"kbe_tags","embeddable":true,"href":"https:\/\/research.reading.ac.uk\/act\/wp-json\/wp\/v2\/kbe_tags?post=5770"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}