Login/Storage

Login and Storage

HPC and Data for Lattice QCD

Login and Storage

This text provides information on how to log into the QPACE installations at FZ Jülich and BU Wuppertal. Furthermore, it describes the storage concepts and how to access data from login/master servers and compute nodes.

Hosts

A QPACE installation consists of a front-end system and the compute nodes. The only user accessible front-end hosts are a login server and a master server. The other servers (2 metadata and 4 storage servers) implement the Lustre parallel file system.

Access to the login and master servers is only possible using ssh keys.

The compute nodes are named in the following manner:

  nc-<BP>-<NC>  (printf format: nc-%02d-%02d)

where BP is the backplane number (0..31) and NC the node position number (0..31).

Login

Login to the installations is permitted only via pass phrase-protected ssh keys. Please provide such a key to one of the administrators. Account data (like username and location of HOME directory) is synchronized between FZJ and BUW; however, home directories are not shared (which means usernames and location of HOME and WORK directories are the same on both sites whereas files are different). After opening your account, your ssh key will be installed at both sites so you can login to the corresponding login servers:

jupace.fz-juelich.de
qlogin.qpace.uni-wuppertal.de

At the sites, your home directory is shared between the login server (qloginJ or qloginW) and the master server (qmasterJ or qmasterW). To access the master servers (from which jobs should be submitted/started), generate a new ssh key pair (on the login server) and place the files in your $HOME/.ssh directory. Then use this key to login to the master server.

To create a tunnel that allows you to directly ssh the master servers, create a file $HOME/.ssh/config (on your machine) and add the following lines (access to Wuppertal is shown, username is "user"):


Host qmasterW
  User user
  Hostname qmasterW
  ProxyCommand ssh user@qlogin.qpace.uni-wuppertal.de nc %h 22

Storage

We provide the following file systems:

/home

Your $HOME directory on qmaster/qlogin can be found at /home/$USER. A quota limitation of 2 GB per user applies. It is foreseen to backup data stored under this directory.

Please note that your $HOME directory on the servers is NOT identical to your $HOME directory on the compute nodes (see below)!

/work

The main storage for data generated on the nodes is provided through a lustre file system. It is mounted under /work on the servers as well as on the nodes and accessible via the environment variable $WORK (which equals /work/$USER). On the nodes, /home is symlinked to /work. This means that content of $HOME on the nodes is identical to that of $WORK on the servers.

There is no (and will not be) backups of the data stored in /work. Please note that while we will make every possible effort to keep the data on /work as safe as possible (e.g. using RAIDs for the back-end storage), we cannot guarantee that your data will never be harmed, so please make sure you regularly transfer data out of the QPACE installations (e.g. into relevant grid storage elements).

Node access

To access the nodes, you will need to create another ssh key pair (either on qlogin or qmaster). Create a $WORK/.ssh directory and place the newly created key files into this directory, along with appropriate config, authorized_keys and known_hosts files (see detailed example below).

Now, you should be able to ssh to the nodes.

$WORK

The environment variable $WORK is setup in /etc/bashrc. It is important that you source this file, otherwise $WORK will be empty.

On the servers: If you use bash and the default .bashrc (which will be created for you together with your account), /etc/bashrc is sourced automatically. Otherwise add an entry to your startup file that includes this file. For shells that are not bach-compatible, setup your startup files so that $WORK points to the directory /work/$USER.


  For the nodes, there are no startup files by default. The recommended
  solution is to copy the following files from your $HOME (on the servers) to your
  $WORK directory (also on a server): .bashrc and
  .bash_profile.




Detailed login example

In the following, we give a detailed example of all steps necessary to login
to the sites. This example assumes we are user "myself" at host "myhost" and try
to access the Wuppertal site.

We will do the following:

  Create "key1" to login to qloginW
  On qloginW, create "key2" to login to qmasterW and the nodes
  Setup files in the appropiate locations so we can actually access qmasterW
  and nodes




# generate pass phrase protected ssh key (called "key1") to login to qpace
[myself@myhost ~]$ mkdir -vp .ssh
mkdir: created directory `.ssh'
[myself@myhost ~]$ chmod 0700 .ssh/
[myself@myhost ~]$ cd .ssh/
[myself@myhost .ssh]$ ssh-keygen -b 2048 -t rsa -f key1
Generating public/private rsa key pair.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in key1.
Your public key has been saved in key1.pub.
The key fingerprint is:
01:19:65:e8:86:fa:30:03:c9:f1:cf:f9:59:39:1f:46 myself@wmai21
# generate config file so ssh finds our key
[myself@myhost .ssh]$ cat >> config << EOF
> Host qloginW
>   User myself
>   Hostname qlogin.qpace.uni-wuppertal.de
>   IdentityFile /home/myself/.ssh/key1
> EOF
[myself@myhost .ssh]$ cat config
Host qloginW
  User myself
  Hostname qlogin.qpace.uni-wuppertal.de
  IdentityFile /home/myself/.ssh/key1
# mail public key to admin
#
# the admin will create an account and the following dirs/files:
# qloginX:/home/myself ($HOME, our home directory on the servers)
# /work/myself         ($WORK, our home directory on the nodes, accessible
#                              from servers as well as nodes)
# $HOME/.ssh
# $HOME/.ssh/authorized_keys  (this file contains our public ssh key)
#
# now, login to site (you need to enter the pass phrase)
[myself@myhost .ssh]$ ssh qloginW
The authenticity of host 'qlogin.qpace.uni-wuppertal.de (132.195.79.193)'
can't be established.
RSA key fingerprint is 2b:87:39:20:db:17:bc:40:9c:8f:72:7e:85:eb:dc:6a.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'qlogin.qpace.uni-wuppertal.de,132.195.79.193'
(RSA) to the list of known hosts.
Enter passphrase for key '/home/myself/.ssh/key1':
Last login: Wed May 12 17:18:20 2010 from myhost.math.uni-wuppertal.de
************************************************************************

                Welcome on QPACE at BU Wuppertal

************************************************************************
# verify we are on qloginW
[myself@qloginW ~]$ hostname
qloginW
[myself@qloginW ~]$ echo $HOME
/home/myself
[myself@qloginW ~]$ echo $WORK
/work/myself
# generate new ssh key pair (called "key2") to access qmasterW (without
# pass phrase protection)
[myself@qloginW ~]$ ls -Ald .ssh
drwx------ 2 myself users 4096 May 12 17:32 .ssh
[myself@qloginW ~]$ cd .ssh
[myself@qloginW .ssh]$ ls -Al
total 4
-rw-r--r-- 1 myself users 395 May 12 17:32 authorized_keys
[myself@qloginW .ssh]$ ssh-keygen -f key2
Generating public/private rsa key pair.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in key2.
Your public key has been saved in key2.pub.
The key fingerprint is:
ef:43:f8:23:80:ce:80:d9:24:47:b2:82:3a:36:04:5f myself@qloginW
# add key to authorized_keys file
[myself@qloginW .ssh]$ cat key2.pub >> authorized_keys
# copy template files (config and known_hosts) to our .ssh dir
# these files enable password less access to qmasterW and nodes
[myself@qloginW .ssh]$ cat /home/public/templates/config.template >> config
[myself@qloginW .ssh]$ cat /home/public/templates/known_hosts.template >> known_hosts
# here, we edit the config file to suite our needs: replace "USER" with
# "myself" and "KEY" with "key2"
[myself@qloginW .ssh]$ vi config
# the config file should look like this
[myself@qloginW .ssh]$ cat config
Host 10.98.1*.1* nc-*-* master qmaster qmasterW
  User myself
  IdentityFile /home/myself/.ssh/key2

# now, we can login to qmasterW
[myself@qloginW .ssh]$ ssh qmasterW
The authenticity of host 'qmasterw (10.98.0.1)' can't be established.
RSA key fingerprint is b5:9d:85:5a:90:21:54:72:dc:2e:77:5e:ee:ff:0b:55.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'qmasterw,10.98.0.1' (RSA) to the list of known hosts.
Last login: Wed May 12 17:25:02 2010 from nc-25-00.qpace
************************************************************************

                Welcome on QPACE at BU Wuppertal

************************************************************************
# verify we are indeed on qmasterW
[myself@qmasterW ~]$ hostname
qmasterW
[myself@qmasterW ~]$ echo $HOME
/home/myself
[myself@qmasterW myself]$ echo $WORK
/work/myself
# copy some files to /work/myself (remember that $WORK is our home on the
# nodes!):
# - our .ssh directory so we can access the nodes via ssh
# - our .bashrc and .bash_profile for nicer setup on the nodes
[myself@qmasterW ~]$ cp -r .bashrc .bash_profile .ssh /work/myself/
# finally, ssh to a node
[myself@qmasterW ~]$ ssh nc-25-00
Last login: Wed May 12 17:23:26 2010 from master
# verify we are indeed on the node
[myself@nc-25-00 ~]$ hostname
nc-25-00
[myself@nc-25-00 ~]$ echo $HOME
/home/myself
[myself@nc-25-00 ~]$ echo $WORK
/work/myself
# verify we can log back to qmasterW
[myself@nc-25-00 ~]$ ssh qmasterW
Last login: Wed May 12 17:37:04 2010 from qloginw.qpace
************************************************************************

                Welcome on QPACE at BU Wuppertal

************************************************************************
[myself@qmasterW ~]$ exit
Connection to qmasterW closed.
# now, we create a file in $HOME on the node and verify where it is visible
[myself@nc-25-00 ~]$ pwd
/home/myself
[myself@nc-25-00 ~]$ touch file_in_home
# it is visible in NODE:$HOME (obviously - that's where we created it...)
[myself@nc-25-00 ~]$ ls -l file_in_home
-rw-r--r-- 1 myself users 0 May 12 17:43 file_in_home
[myself@nc-25-00 ~]$ cd $WORK
# it is also visible in NODE:$WORK ($HOME and $WORK are the same on the nodes)
[myself@nc-25-00 myself]$ ls -l file_in_home
-rw-r--r-- 1 myself users 0 May 12 17:43 file_in_home
[myself@nc-25-00 myself]$ exit
logout
Connection to nc-25-00 closed.
[myself@qmasterW .ssh]$ cd ..
[myself@qmasterW myself]$ pwd
/work/myself
# file is visible in SERVER:$WORK ($WORK is the same filesystem on nodes and
# servers)
[myself@qmasterW myself]$ ls -l file_in_home
-rw-r--r-- 1 myself users 0 May 12 17:43 file_in_home
[myself@qmasterW myself]$ cd $HOME
# file is NOT visible on nodes (NFS filesystem von servers)
[myself@qmasterW ~]$ ls -l file_in_home
ls: file_in_home: No such file or directory
# leave QPACE
[myself@qmasterW ~]$ exit
Connection to qmasterW closed.
[myself@qloginW ~]$ exit
Connection to qlogin.qpace.uni-wuppertal.de closed.
# finally, setup stuff so we can login to qmasterW directly
# first, generate entry in config
[myself@myhost .ssh]$ cat >> config << EOF
>
> Host qmasterW
>   User myself
>   Hostname qmasterW
>   ProxyCommand ssh qloginW nc %h 22
>   IdentityFile /home/myself/.ssh/key2
> EOF
[myself@myhost .ssh]$ cat config
Host qloginW
  User myself
  Hostname qlogin.qpace.uni-wuppertal.de
  IdentityFile /home/myself/.ssh/key1

Host qmasterW
  User myself
  Hostname qmasterW
  ProxyCommand ssh qloginW nc %h 22
  IdentityFile /home/myself/.ssh/key2
# then, copy our "key2" key to local machine
[myself@myhost .ssh]$ scp qloginW:.ssh/key2* .
Enter passphrase for key '/home/myself/.ssh/key1':
key2                                   100% 1675     1.6KB/s   00:00
key2.pub                               100%  396     0.4KB/s   00:00
# now, login to qmasterW
[myself@myhost .ssh]$ ssh qmasterW
Enter passphrase for key '/home/myself/.ssh/key1':
The authenticity of host 'qmasterw (<no hostip for proxy command>)' can't
be established.
RSA key fingerprint is b5:9d:85:5a:90:21:54:72:dc:2e:77:5e:ee:ff:0b:55.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'qmasterw' (RSA) to the list of known hosts.
Last login: Wed May 12 18:16:30 2010 from qloginw.qpace
************************************************************************

                Welcome on QPACE at BU Wuppertal

************************************************************************
# works!
[myself@qmasterW ~]$ hostname
qmasterW
[myself@qmasterW ~]$ exit
Connection to qmasterW closed.
[myself@myhost .ssh]$
# successfully finished!