Workflow for data replication into the CDP

the steps should be followed by persons responsible for data ingest into the DKRZ CMIP data pool (CDP) via ESGF replication

configuration changes as described require membership in the DKRZ-ESGF group (group 933), please contact Stephan for membership requests

Setup and configure replication software

auf mistral:
  • load synda replication software module
  • documentation of synda:
    module load synda
  • set Path to synda home
    export ST_HOME=/sw/rhel6-x64/dm/sdt
  • set your credentials
    (can be omitted in this case default credentials are used)
    cd $ST_HOME/conf
    vi credentials.comf
  • default configuration settings
    • synda database and target directory:
      these are set in $ST_HOME/conf/sdt.conf:
      db_path = /sw/rhel6-x64/dm/sdt/db
      data_path = /work/kd0956/Test
      change to:
      data_path = /work/kd0956/"myproject"/data  

setup and configure your replication job

  • edit a replication description file in the specific selection folder
    cd ST_HOME
    cd selection/CMIP5 
    vi name_of_specific_replication_job.txt

check and submit the replication job

  • commit the selection/CMIP5 repo
    cd $ST_HOME/selection/CMIP5
    git add .
    git commit -m "my comment" 
  • renew your credentials and submit replication job
    synda daemon start (only if synda daemon is not running)
    synda certificate renew
    synda install $ST_HOME/selection/CMIP5/name_of_specific_replication_job.txt
  • monitor running replication jobs
    synda history