sftp data transfer

SFTP (SSH File Transfer Protocol) is a commonly used protocol for transferring data. There are clients available for almost any platform that one can think of. It is available as an alternative to using Globus for retrieving data from MAX IV.

Multi factor authentication

To access MAX IV remotely you need to authenticate with your username and password, plus a One-Time Password. To prepare for using an OTP, see the guidelines for One Time Password.

Accesible file systems

  • File systems visible to sftp
    • /data/
    • /mxn/visitors
    • /mxn/staff
  • Note that /home/ in the HPC area is not visible to sftp

Log in to the sftp server from a Linux client

  1. From Linux command line, simply type

    sftp 'maxiv_user_name'@sftp.maxiv.lu.se 

    on the command line, where ‘maxiv_user_name’ is the user name you use to log in to DUO or the beamline computers.
    Enter your password and next you will be prompted for the “otp” code.
  2. Go to the directory that you want to transfer. This will typically be under /data/visitors/’beamline’/’your_proposal’/’date’/…

    cd /data/visitors/betamax/19801111/19810101/raw/
  3.  Use the get command to retrieve the data, here with the -R option which will recursively download the whole directory.

    get -R <dir>

Using the “help” argument will give you a list of available commands.

sftp help

Available commands:
bye Quit sftp
cd path Change remote directory to 'path'
chgrp grp path Change group of file 'path' to 'grp'
chmod mode path Change permissions of file 'path' to 'mode'
chown own path Change owner of file 'path' to 'own'
df [-hi] [path] Display statistics for current directory or
filesystem containing 'path'
exit Quit sftp
get [-afPpRr] remote [local] Download file
reget [-fPpRr] remote [local] Resume download file
reput [-fPpRr] [local] remote Resume upload file
help Display this help text
lcd path Change local directory to 'path'
lls [ls-options [path]] Display local directory listing
lmkdir path Create local directory
ln [-s] oldpath newpath Link remote file (-s for symlink)
lpwd Print local working directory
ls [-1afhlnrSt] [path] Display remote directory listing
lumask umask Set local umask to 'umask'
mkdir path Create remote directory
progress Toggle display of progress meter
put [-afPpRr] local [remote] Upload file
pwd Display remote working directory
quit Quit sftp
rename oldpath newpath Rename remote file
rm path Delete remote file
rmdir path Remove remote directory
symlink oldpath newpath Symlink remote file
version Show SFTP version
!command Execute 'command' in local shell
! Escape to local shell
? Synonym for help

SFTP clients for Windows and MacOS

Some popular SFTP clients for Windows and MacOS platforms are

  • WinSCP (Windows)
  • FileZilla (Windows, MacOS, Linux)
  • CyberDuck (MacOS, Windows)
  • PuTTY (Windows, sftp command line client is part of PuTTY installation)
  • MacOS also has the standard sftp command line program available.

Notes on specific clients

  • FileZilla – Do not use the “Quick Connect” bar at the top, it will not allow you to use 2-factor authentication.
  • FileZilla – In the Site Manager, set the ‘Logon Type’ to Interactive.
  • FileZilla – There is an advanced option that enables multiple download streams, do not enable it. User reports that enabling this option breaks the data transfer since the 2-factor authentication will fail for the additional streams.

Transferring files to MAX IV

It is not possible to transfer files directly from an external computer to the /mxn/ or /data/visitor areas at MAX IV. Incoming files must be placed in the /data/ingest directory:

  • Connect either to sftp.maxiv.lu.se or the Globus server.
  • Go to the directory /data/ingest and create a subdirectory (we recommend to use your user name).
  • Transfer the data from your computer to your ingest subdirectory.
  • Log in to the one of the MAX IV HPC cluster frontends, the ingest directory is only available on those. If you are offsite, open a VPN connection and log in to the offline HPC cluster.
  • Move or copy the files to an appropriate location. Any files left over in the ingest directory will be automatically deleted, typically within one month.