Version 9 (modified by stgo, 10 years ago) (diff)

--

Data arrival by network transfer

  1. change to arsf user on a safe machine
    ssh arsf@gridmaster2 # (or su - arsf and don't turn your machine off!)
    
  2. find the exact data path on Ops server
    ssh arsfdan@thelma.nerc-arsf.ac.uk
     # go find the path to the data to download
     # these will vary..
    cd /data/Data_2009/ ##Also try /mnt/synology1/data/Data\ 2014/ or synology2
    cd UK
    cd 281-09_GB08-02_Delamere
     # check path
    pwd
     # returns /data/Data_2009/UK/281-09_GB08-02_Delamere
     #
     # check size isn't crazy big
     # > 150GB and think about a disk transfer
    du -hsc .
     # if you get any permission denied errors, fix up permissions then redo the du.
     # fix permissions with:
     # /data/permissions_fixer/fix_perms_for_arsfdan.sh DIRECTORY_NEEDING_FIX
     #  **be very careful** with this as you could easily screw up the whole thelma server if you get the directory wrong!
    logout
    
  3. create a temporary directory in ~arsf_data/YYYY/flight_data/unpacking/ to download to, e.g.
    DOWNDIR=~/arsf_data/`date +%Y`/flight_data/unpacking/download-`date +%Y%m%d%H%M%S`
    mkdir $DOWNDIR
    cd $DOWNDIR
    
  4. do the download with rsync, using compression (-z flag). Note there is no trailing / on the PROJECTNAME part of the path below - this is important!
    • example rsync command follows, with a --dry-run option to prevent any damage occurring on the trial attempt. If it lists the files you'd expect it to transfer, remove the --dry-run option and repeat. If not, check your slashes!
      date
      rsync --dry-run -avz arsfdan@thelma.nerc-arsf.ac.uk:/PATH/YOU/FOUND/EARLIER/PROJECTNAME . ; date
       # date commands are optional, they just let you see when it started and stopped
      

If something goes wrong after an hour or two, don't delete what you've already downloaded. Instead, repeat the rsync command and it'll pick up where it left off.

Unpacking data

Return to Procedures/NewDataArrival