{"id":5592,"date":"2020-04-16T22:04:12","date_gmt":"2020-04-16T21:04:12","guid":{"rendered":"https:\/\/research.reading.ac.uk\/act\/?post_type=kbe_knowledgebase&p=5592"},"modified":"2020-10-14T14:57:59","modified_gmt":"2020-10-14T13:57:59","slug":"act-ssh-connection-caching-for-automated-data-transfers-from-unix","status":"publish","type":"kbe_knowledgebase","link":"https:\/\/research.reading.ac.uk\/act\/knowledgebase\/act-ssh-connection-caching-for-automated-data-transfers-from-unix\/","title":{"rendered":"act-ssh – connection caching for automated data transfers from Unix"},"content":{"rendered":"

This article will be updated to match with other act-ssh articles. The information provided here is correct, but ssh multiplexing is now also explained in the article:\u00a0 act-ssh for Linux and Mac users \u2013 step by step guide<\/a><\/p>\n

Users relying on scripted data transfers via oak<\/strong> can replace using ssh-agent with<\/strong> equally convenient connection multiplexing<\/strong>, as described at the end of this article<\/a>.\u00a0 In rare cases where connection multiplexing cannot be used (in the context of migrating from oak), we will discuss individual solutions, using act-rsync<\/a>, or Managed File Transfer<\/a> service, or even individual authentication policy exceptions on act-ssh. If there is large demand for help with setting this up we will organize a tutorial session (for users already using ssh for data transfers).<\/p>\n

This will be phased out, new users are not added to the group: Users connecting from UK academic institution with the domain name ending with ‘.ac.uk’ (other trusted institutions can be added as needed), and who rely on scripted data transfers that cannot be done with ssh connection caching, can apply to be added to the exception list (the security group act-ssh-academic-users), and use ssh key authentication without UoR password to connect from those institutions. By applying for the exception, those users commit to always use password protected private ssh keys (with ssh-agent, if needed).<\/em><\/p>\n

Allowing automatic connections from your account on some off-campus machine is potentially putting your data at risk and compromising security of the whole campus network. Please use the following procedures responsibly, only on secure systems at trusted academic institutions, or on a personal machine which is used only by you.<\/strong> A separate data transfer options, MFT<\/a> for sftp and scp and act-rsync<\/a> for rsync,\u00a0 are provided and suitable also in the case of less secure off-campus machines.<\/p>\n

Background on the ssh-agent<\/h6>\n

When using ssh key authentication on oak users where asked to protect their private key with passphrase. Ssh keys that are not protected by a passphrase were only allowed to be used on legacy poplar, and now on act-rsync<\/a>. Passphrase is independent from UoR password, it is just used to protect encrypted ssh key, such that in case the private ssh key is leaked out it cannot be used by a third party. It is the same logic as when a file is encrypted and protected with a password, e.g. you need to enter a password to open your pdf payslip. To avoid having to type the passphrase multiple times, ssh-agent requires you to enter the passphrase just once (it is needed to decrypt the key), and then runs in the background and keeps decrypted ssh key in memory, potentially for as long as the client machine is not restarted. Ssh-agent will not allow fully automated connections to act-ssh<\/strong>, because here, after the ssh key authentication, authentication with the UoR password is required.<\/p>\n

Connection reusing (ssh multiplexing)<\/h6>\n

Openssh allows to create an authenticated ssh connection, put it in the background, and reuse it for further ssh sessions, without a need to authenticate again. The procedure will be similar to using ssh-agent.<\/strong> On a trusted<\/strong> off-campus system, you will need to set up the master connection and authenticate with one the two multi-factor authentication methods described above. Once this master connection is established, it stays in the background, even after you log out, and all your subsequent ssh connections from this client machine will not require further authentication. To set it up, you need to create a personal ssh client configuration file, the default location is\u00a0 ~\/.ssh\/config, and put there the following lines<\/p>\n

Host act-ssh\r\n        HostName act-ssh.reading.ac.uk\r\n        User <UoR login name>\r\n        Port 2222\r\n        ServerAliveInterval 30\r\n        ServerAliveCountMax 3\r\n        ControlPath ~\/.ssh\/controlmasters\/%l_%r@%h:%p\r\n        ControlMaster auto \r\n        ControlPersist yes<\/pre>\n
With this set up, and the directory\u00a0~\/.ssh\/controlmasters<\/span>\u00a0 created, with the right permissions (see below), once you open a connection to act-ssh for the first time and authenticate, a persistent background process and a control socket, at the location as specified in the configuration file, will be created automatically (with ‘ControlMaster auto’), and they will be reused for subsequent connections, without a need to authenticate again. The lines ‘ControlMaster auto’\u00a0 and ‘ControlPersist yes’ are optional, and in some older versions of openssh client these options are not recognized. Without ‘ControlMaster auto’\u00a0 the\u00a0 background master connection is not created automatically, but it can be created manually. We recommend creating the master connection manually anyway, for better control whether it is created or not. We will show this procedure in the following example, on met-cluster (substitute mimas with your remote system, e.g. archer, jasmin, etc.). Met-cluster is intentionally not treated as a trusted host (requires two factor authentication) to allow testing. The option ‘User ‘ is handy if your user name on the off-campus client machine is different than your UoR user name. There is also a handy option ‘Port’ in case you want to use port 2222 and ssh key authentication. Adding the two ‘ServerAlive’ options might help with keeping the inactive master connection alive. On some systems you might need to use shorter ‘ControlPath ~\/.ssh\/controlmasters\/%L_%r@%n:%p<\/a>‘, instead of ‘ControlPath ~\/.ssh\/controlmasters\/%l_%r@%h:%p’.<\/p>\n
mimas$ mkdir .ssh\/controlmasters\r\nmimas$ chmod 700 .ssh\/controlmasters\r\nmimas$ cat .ssh\/config\r\nHost act-ssh\r\n      HostName act-ssh.reading.ac.uk\r\n      User <UoR login name> \r\n      Port 2222 \r\n      ServerAliveInterval 30 \r\n      ServerAliveCountMax 3\r\n      ControlPath ~\/.ssh\/controlmasters\/%l_%r@%h:%p   \r\n<\/pre>\n
‘ControlMaster auto’ and ‘ControlPersist’ are not available on met-cluster. We need to create the master connection by hand. The master connection will stay active after you log out. In this example we connect on default port 22, and we use Authenticator, not the ssh key, to make the two factor authentication step more obvious. Connection on port 2222 (port number can be specified in the configuration file) and uthentication with ssh key will work as well.<\/p>\n
mimas$ ssh act-ssh -fNM\r\nVerification code:\r\nPassword:\r\nmimas$ ssh -O check act-ssh \r\nMaster running (pid=1893406)\r\nmimas$ exit\r\nhomePC$<\/pre>\n
We login to met-cluster again, confirm that the master connection is still active, and we connect to act-ssh without authentication.<\/p>\n
homePC$ ssh met-cluster\r\nmimas$ ssh -O check act-ssh\r\nMaster running (pid=1893406)\r\nmimas$ ssh act-ssh\r\nLast login: Sun Nov 3 15:17:51 2019 from mimas.nerc-essc.ac.uk\r\n[qx901702@act-ssh ~]$<\/pre>\n
We can stop control master with the command ssh -O exit act-ssh<\/span>\u00a0<\/span><\/p>\n
mimas$ ssh -O check act-ssh\r\nMaster running (pid=1893406)\r\nmimas$ ssh -O exit act-ssh\r\nExit request sent.\r\nmimas$ ssh -O check act-ssh\r\nControl socket connect(\/home\/users\/qx901702\/.ssh\/controlmasters\/mimas.nerc-essc.ac.uk:qx901702@act-ssh.reading.ac.uk:22): No such file or directory<\/pre>\n
Please remember to stop unused master connections.<\/p>\n
Connection caching tips for Linux users working from home<\/h1>\n
This section is really intended for experienced Unix users, the procedures described here are not a service supported by the DTS. It is given as an informal advice and we are not able to provide individual support to users . The following example of the ~\/.ssh\/config file can be used as a source of inspiration by Linux or Mac users who can’t or don’t want to use VPN. In addition to connection caching such that you need to enter your password only once, when you log in to your home Linux machine, it sets up an local port forwarding ssh tunnel (here port 3389 for Windows Remote Desktop connection and port 4000 for nx.reading.ac.uk), and a SOCKS proxy tunnel which can be used with a Web browser, for connecting to nx.reading.ac.uk using the Enterprise NX Client for Linux and with other software which is able to connect via SOCKS proxy.<\/p>\n
Host act-ssh\r\n HostName act-ssh.reading.ac.uk\r\n User qx901702\r\n Port 2222\r\n ServerAliveInterval 30\r\n ServerAliveCountMax 3\r\n ControlPath ~\/.ssh\/controlmasters\/%l_%r@%h:%p\r\n ControlMaster auto \r\n ControlPersist yes\r\n ForwardX11 yes\r\n DynamicForward localhost:4444\r\n Localforward localhost:33389 smpc115.rdg.ac.uk:3389\r\n Localforward localhost:4000 nx.reading.ac.uk:4000\r\nHost cluster\r\n Hostname cluster.act.rdg.ac.uk\r\n ProxyCommand ssh -W %h:%p act-ssh -Y\r\n User qx901702\r\n ServerAliveInterval 30\r\n ServerAliveCountMax 3\r\n ControlPath ~\/.ssh\/controlmasters\/%l_%r@%h:%p\r\n ControlMaster auto\r\n ControlPersist yes\r\n ForwardX11 yes\r\n<\/pre>\n
With this configuration in place one can use SOCKS v4 proxy on localhost:4444 for web browsing or for connection to NX. To connect to my Windows desktop I use the command:<\/p>\n
pawel@homePC:~$ xfreerdp \/u:qx901702 \/v:localhost:33389 -f<\/pre>\n
The second part of the file sets up master connection to RACC using act-ssh as a proxy. With this setup we just type<\/p>\n
pawel@homePC:~$ ssh cluster<\/pre>\n
and we are in. Another advantage is that reusing cached connection works really fast.<\/p>\n","protected":false},"excerpt":{"rendered":"
This article will be updated to match with other act-ssh articles. The information provided here is correct, but ssh multiplexing is now also explained in the article:\u00a0 act-ssh for Linux and Mac users \u2013 step by step guide    Users relying on scripted data transfers via oak can replace using ssh-agent with equally<\/p>\n","protected":false},"author":16,"featured_media":5701,"template":"","meta":{"_acf_changed":false,"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"__cvm_playback_settings":[],"__cvm_video_id":"","_links_to":"","_links_to_target":""},"kbe_taxonomy":[80],"kbe_tags":[],"class_list":["post-5592","kbe_knowledgebase","type-kbe_knowledgebase","status-publish","has-post-thumbnail","hentry","kbe_taxonomy-user-guides"],"acf":[],"yoast_head":"\nact-ssh - connection caching for automated data transfers from Unix - Academic Computing Team<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/research.reading.ac.uk\/act\/knowledgebase\/act-ssh-connection-caching-for-automated-data-transfers-from-unix\/\" \/>\n<meta property=\"og:locale\" content=\"en_GB\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"act-ssh - connection caching for automated data transfers from Unix - Academic Computing Team\" \/>\n<meta property=\"og:description\" content=\"This article will be updated to match with other act-ssh articles. The information provided here is correct, but ssh multiplexing is now also explained in the article:\u00a0 act-ssh for Linux and Mac users \u2013 step by step guide  Users relying on scripted data transfers via oak can replace using ssh-agent with equally\" \/>\n<meta property=\"og:url\" content=\"https:\/\/research.reading.ac.uk\/act\/knowledgebase\/act-ssh-connection-caching-for-automated-data-transfers-from-unix\/\" \/>\n<meta property=\"og:site_name\" content=\"Academic Computing Team\" \/>\n<meta property=\"article:modified_time\" content=\"2020-10-14T13:57:59+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/research.reading.ac.uk\/act\/wp-content\/uploads\/sites\/2\/Unorganized\/ssh.png\" \/>\n\t<meta property=\"og:image:width\" content=\"256\" \/>\n\t<meta property=\"og:image:height\" content=\"256\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Estimated reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"7 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/research.reading.ac.uk\/act\/knowledgebase\/act-ssh-connection-caching-for-automated-data-transfers-from-unix\/\",\"url\":\"https:\/\/research.reading.ac.uk\/act\/knowledgebase\/act-ssh-connection-caching-for-automated-data-transfers-from-unix\/\",\"name\":\"act-ssh - connection caching for automated data transfers from Unix - Academic Computing Team\",\"isPartOf\":{\"@id\":\"https:\/\/research.reading.ac.uk\/act\/#website\"},\"datePublished\":\"2020-04-16T21:04:12+00:00\",\"dateModified\":\"2020-10-14T13:57:59+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/research.reading.ac.uk\/act\/knowledgebase\/act-ssh-connection-caching-for-automated-data-transfers-from-unix\/#breadcrumb\"},\"inLanguage\":\"en-GB\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/research.reading.ac.uk\/act\/knowledgebase\/act-ssh-connection-caching-for-automated-data-transfers-from-unix\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/research.reading.ac.uk\/act\/knowledgebase\/act-ssh-connection-caching-for-automated-data-transfers-from-unix\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/research.reading.ac.uk\/act\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Knowledgebase\",\"item\":\"https:\/\/research.reading.ac.uk\/act\/knowledgebase\/\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"act-ssh – connection caching for automated data transfers from Unix\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/research.reading.ac.uk\/act\/#website\",\"url\":\"https:\/\/research.reading.ac.uk\/act\/\",\"name\":\"Academic Computing Team\",\"description\":\"\",\"publisher\":{\"@id\":\"https:\/\/research.reading.ac.uk\/act\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/research.reading.ac.uk\/act\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-GB\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/research.reading.ac.uk\/act\/#organization\",\"name\":\"University of Reading\",\"url\":\"https:\/\/research.reading.ac.uk\/act\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-GB\",\"@id\":\"https:\/\/research.reading.ac.uk\/act\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/research.reading.ac.uk\/act\/wp-content\/uploads\/sites\/2\/2017\/08\/cropped-University_of_Reading_shield-1.png\",\"contentUrl\":\"https:\/\/research.reading.ac.uk\/act\/wp-content\/uploads\/sites\/2\/2017\/08\/cropped-University_of_Reading_shield-1.png\",\"width\":512,\"height\":512,\"caption\":\"University of Reading\"},\"image\":{\"@id\":\"https:\/\/research.reading.ac.uk\/act\/#\/schema\/logo\/image\/\"}}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"act-ssh - connection caching for automated data transfers from Unix - Academic Computing Team","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/research.reading.ac.uk\/act\/knowledgebase\/act-ssh-connection-caching-for-automated-data-transfers-from-unix\/","og_locale":"en_GB","og_type":"article","og_title":"act-ssh - connection caching for automated data transfers from Unix - Academic Computing Team","og_description":"This article will be updated to match with other act-ssh articles. The information provided here is correct, but ssh multiplexing is now also explained in the article:\u00a0 act-ssh for Linux and Mac users \u2013 step by step guide  Users relying on scripted data transfers via oak can replace using ssh-agent with equally","og_url":"https:\/\/research.reading.ac.uk\/act\/knowledgebase\/act-ssh-connection-caching-for-automated-data-transfers-from-unix\/","og_site_name":"Academic Computing Team","article_modified_time":"2020-10-14T13:57:59+00:00","og_image":[{"width":256,"height":256,"url":"https:\/\/research.reading.ac.uk\/act\/wp-content\/uploads\/sites\/2\/Unorganized\/ssh.png","type":"image\/png"}],"twitter_card":"summary_large_image","twitter_misc":{"Estimated reading time":"7 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/research.reading.ac.uk\/act\/knowledgebase\/act-ssh-connection-caching-for-automated-data-transfers-from-unix\/","url":"https:\/\/research.reading.ac.uk\/act\/knowledgebase\/act-ssh-connection-caching-for-automated-data-transfers-from-unix\/","name":"act-ssh - connection caching for automated data transfers from Unix - Academic Computing Team","isPartOf":{"@id":"https:\/\/research.reading.ac.uk\/act\/#website"},"datePublished":"2020-04-16T21:04:12+00:00","dateModified":"2020-10-14T13:57:59+00:00","breadcrumb":{"@id":"https:\/\/research.reading.ac.uk\/act\/knowledgebase\/act-ssh-connection-caching-for-automated-data-transfers-from-unix\/#breadcrumb"},"inLanguage":"en-GB","potentialAction":[{"@type":"ReadAction","target":["https:\/\/research.reading.ac.uk\/act\/knowledgebase\/act-ssh-connection-caching-for-automated-data-transfers-from-unix\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/research.reading.ac.uk\/act\/knowledgebase\/act-ssh-connection-caching-for-automated-data-transfers-from-unix\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/research.reading.ac.uk\/act\/"},{"@type":"ListItem","position":2,"name":"Knowledgebase","item":"https:\/\/research.reading.ac.uk\/act\/knowledgebase\/"},{"@type":"ListItem","position":3,"name":"act-ssh – connection caching for automated data transfers from Unix"}]},{"@type":"WebSite","@id":"https:\/\/research.reading.ac.uk\/act\/#website","url":"https:\/\/research.reading.ac.uk\/act\/","name":"Academic Computing Team","description":"","publisher":{"@id":"https:\/\/research.reading.ac.uk\/act\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/research.reading.ac.uk\/act\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-GB"},{"@type":"Organization","@id":"https:\/\/research.reading.ac.uk\/act\/#organization","name":"University of Reading","url":"https:\/\/research.reading.ac.uk\/act\/","logo":{"@type":"ImageObject","inLanguage":"en-GB","@id":"https:\/\/research.reading.ac.uk\/act\/#\/schema\/logo\/image\/","url":"https:\/\/research.reading.ac.uk\/act\/wp-content\/uploads\/sites\/2\/2017\/08\/cropped-University_of_Reading_shield-1.png","contentUrl":"https:\/\/research.reading.ac.uk\/act\/wp-content\/uploads\/sites\/2\/2017\/08\/cropped-University_of_Reading_shield-1.png","width":512,"height":512,"caption":"University of Reading"},"image":{"@id":"https:\/\/research.reading.ac.uk\/act\/#\/schema\/logo\/image\/"}}]}},"_links":{"self":[{"href":"https:\/\/research.reading.ac.uk\/act\/wp-json\/wp\/v2\/kbe_knowledgebase\/5592","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/research.reading.ac.uk\/act\/wp-json\/wp\/v2\/kbe_knowledgebase"}],"about":[{"href":"https:\/\/research.reading.ac.uk\/act\/wp-json\/wp\/v2\/types\/kbe_knowledgebase"}],"author":[{"embeddable":true,"href":"https:\/\/research.reading.ac.uk\/act\/wp-json\/wp\/v2\/users\/16"}],"version-history":[{"count":5,"href":"https:\/\/research.reading.ac.uk\/act\/wp-json\/wp\/v2\/kbe_knowledgebase\/5592\/revisions"}],"predecessor-version":[{"id":6021,"href":"https:\/\/research.reading.ac.uk\/act\/wp-json\/wp\/v2\/kbe_knowledgebase\/5592\/revisions\/6021"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/research.reading.ac.uk\/act\/wp-json\/wp\/v2\/media\/5701"}],"wp:attachment":[{"href":"https:\/\/research.reading.ac.uk\/act\/wp-json\/wp\/v2\/media?parent=5592"}],"wp:term":[{"taxonomy":"kbe_taxonomy","embeddable":true,"href":"https:\/\/research.reading.ac.uk\/act\/wp-json\/wp\/v2\/kbe_taxonomy?post=5592"},{"taxonomy":"kbe_tags","embeddable":true,"href":"https:\/\/research.reading.ac.uk\/act\/wp-json\/wp\/v2\/kbe_tags?post=5592"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}