New: Academic Computing Drop-in Sessions
Academic computing drop-in sessions are now run by members of our team on Wednesday afternoons! Head to our drop-in info page for details & schedule....
Knowledge Base
- /
- /
- /
Poplar replacement
This article is addressed to the users already using poplar for automated data transfers. It might be also useful to some users who will need to find an alternative for their automated data transfers currently running on oak.
Background on poplar.reading
Poplar.reading.ac.uk is a server located outside the University trusted network. It is set up to accept ssh connections authenticated with ssh key, potentially without passphrase, to allow automated data transfers with rsync. Transferring data is a two step process. Remote systems are depositing data on poplar, and then the data is moved from poplar to UoR data storage by connecting to poplar from the trusted network side. The old poplar.reading.ac.uk will be permanently switched off on 19 Dec 2019. It was set up without any funding to and it is just a repurposed old storage server, it can fail any time and it has to be replaced. We have created a replacement service act-rsync.reading.ac.uk (at some stage we used the name poplar.act.reading.ac.uk). It is a VM (virtual machine) and it will be more reliable. Formally, the cost of the VM should be charged to the projects using this custom service, but IT will provide it for free, to allow for smooth decommissioning of old poplar. Typically, the accounts actively using poplar will be migrated to the new systems, and new user accounts will not be created. We can make some exceptions tor users migrating from oak, but new users should rather use MFT or act-ssh for data transfers.
General user’s instructions and service overview
Act-rsync is a service that allows rsync connections from off-campus machines authenticated just with ssh keys. If you did not set a passphrase on your private ssh key you can transfer data with rsync from off-campus machines, without entering any password. Normally, any off-campus connection requires two factor authentication. Allowing pasword-less connection from off-campus, as it is done here, makes the system vulnerable. This is why only rsync connection is available, and any other options for of connecting to this system are disabled. Also, UoR data storage volumes are not accessed directly via act-rsync, data can be only deposited in the internal storage, and then moved to UoR storage in a second step. Act-rsync’s local storage is exported to some ACT systems and mounted at /storage/shared/act-rsync. The exported storage can be used to access your data form the trusted network side, and to manage your ssh keys. Authorized_keys should be placed in /storage/shared/act-rsync/<username>/authorized_keys . Typically, username will be the same as on the old poplar, but with UID of a proper UoR group account, such that you will be able to access the files form the UoR group account. The data copied via rsync can be found at /storage/shared/act-rsync/<username>/data/ , and the paths you specify to rsync will be relative to /storage/shared/act-rsync/<username>/data . E.g.
remotePC>$ rsync -avr my_source_dir <username>@act-rsync.reading.ac.uk:/
will place my_source_dir in /storage/shared/act-rsync/<username>/data/. If your rsync command is:
remotePC>$ rsync -avr my_source_dir <username>@act-rsync.reading.ac.uk:/home/<username>/
and if you do not want to change it, you will just need to create the directory
/storage/shared/act-rsync/<username>/data/home/<username>/
and you will find your data in
/storage/shared/act-rsync/<username>/data/home/<username>/my_source_dir
Migration from poplar.reading to act-rsync
The system is planned for the migration to be straightforward (assuming you are using only rsync). This should be just a matter of changing the host address from poplar.reading.ac.uk to act-rsync.reading.ac.uk, maybe creating some directories in /storage/shared/act-rsync/<username>/data/ , and all should work as before. To give an example, if you push data with the command
remotePC>$ rsync -avr my_source_dir <username>@poplar.reading.ac.uk:/home/<username>/
and pull data with the command
racc-cron>$ rsync -avr <username>@poplar.reading.ac.uk:/home/<username>/ /storage/silver/<project>/
You will just need to create a directory
racc-cron>$ mkdir -p /storage/shared/act-rsync/<username>/data/home/<username>
and change your commands to
remotePC>$ rsync -avr my_source_dir <username>@act-rsync.reading.ac.uk:/home/<username>/
and
racc-cron>$ rsync -avr <username>@act-rsync.reading.ac.uk:/home/<username>/ /storage/silver/<project>/
Note that using rsync from the trusted network side it is possible but it is not necessary, as the storage is mounted directly. The second command can as well be replaced with:
racc-cron>$ cp -a /storage/shared/act-rsync/<username>/data/home/<username>/* /storage/silver/<project>/
Transferring files to and from Linux storage
Research data transfer using arc-ssh.reading.ac.uk
Transfer files in and out of the university
Managed File Transfer (MFT)