# DreqPy (Data Request Python API)

Official Link

CMIP Phase 6 Data Request Python Interface

#### IPython NoteBook created by Martin Schupfner, DKRZ¶

#### Reference: Martin Juckes, NCAS BADC (2016) -¶

####
dreqPy (Data Request Python API) & dreqPy User's Guide

## Synopsis¶

####
0 - Basic Information

####
1 - Basic Imports

####
2 - dq.coll Examples

####
3 - dq.inx Examples

####
4 - Presentation of dq.coll sections

####
5 - Advanced Example

####
6 - Data Request Volume Estimate

####
7 - Additional Functions

####
8 - Official Web Tools

####
9 - Outlook and Discussion

## 0 - Basic Information

Back to Synopsis
In [2]:

    from IPython.display import Image, display

    display(Image(filename='files/Folie1.PNG'))

In [3]:

    display(Image(filename='files/Folie2.PNG'))

In [4]:

    display(Image(filename='files/Folie3.PNG'))

In [5]:

    display(Image(filename='files/Folie4.PNG'))

## 1 - Basic Imports

Back to Synopsis
In [6]:

    #!/usr/bin/python
    # -*- coding: utf-8 -*-

    # IPython NoteBook created by Martin Schupfner, DKRZ
    # Reference: Martin Juckes 2016 -
    #            dreqPy (Data Request Python API) & dreqPy User's Guide

    from dreqPy import dreq

    print "Using DreqPy (Data Request Python API) in version %s"\
        % str(dreq.version)

    # Initialisation
    dq = dreq.loadDreq()

    Using DreqPy (Data Request Python API) in version 01.beta.42

## 2 - dq.coll Examples

Back to Synopsis
In [7]:

    # dq.coll
    # Content Object dq.coll is a dictionary containing the data request sections,
    #   with 3 elements (named tuples) for each section:
    # - header :  named tuple with info such as title, lable, etc.
    # - attDefn:  dictionary containing record attribute definitions
    # - items  :  list of records

    # Print all entries of dq.coll
    print "dq.coll Entries:\n", ", ".join(dq.coll.keys())

    dq.coll Entries:
    requestItem, grids, spatialShape, miptable, requestVar, __main__, temporalShape, __sect__, modelConfig, experiment, var, mip, __core__, tags, CMORvar, tableSection, varChoice, exptgroup, varRelLnk, remarks, varRelations, cellMethods, structure, requestVarGroup, varChoiceLinkC, requestLink, varChoiceLinkR, objectiveLink, objective, timeSlice, standardname

In [8]:

    # header content (Example MIP Variable):
    print ".header content Example 'var':\n------------------------------"

    for name in dq.coll['var'].header._fields: print "%-15s : %s" \
        %(name, getattr(dq.coll['var'].header, name))

    .header content Example 'var':
    ------------------------------
    tag             : table
    label           : var
    title           : 1.2 MIP Variable
    id              : var
    itemLabelMode   : def
    level           : 0
    maxOccurs       : 1
    labUnique       : No
    uid             : SECTION:var

In [9]:

    # attDefn content (Example MIP Variable):
    print ".attDefn content Example 'var':\n-------------------------------"

    for key in dq.coll['var'].attDefn.keys():
        print "%-15s : %s" %(key, dq.coll['var'].attDefn[key])

    .attDefn content Example 'var':
    -------------------------------
    procnote        : Item <X.1 Core Attributes>: [procnote] Processing notes
    description     : Item <X.1 Core Attributes>: [description] Record Description
    title           : Item <X.1 Core Attributes>: [title] Long name
    prov            : Item <X.1 Core Attributes>: [prov] Provenance
    unid            : Item <X.1 Core Attributes>: [unid] Link to Units section
    label           : Item <X.1 Core Attributes>: [label] Variable Name
    provmip         : Item <X.1 Core Attributes>: [provmip] MIP defining this variables
    sn              : Item <X.1 Core Attributes>: [sn] CF Standard Name
    units           : Item <X.1 Core Attributes>: [units] Units
    procComment     : Item <X.1 Core Attributes>: [procComment] Processing comments
    uid             : Item <X.1 Core Attributes>: [uid] Record identifier

In [10]:

    # items content (Example MIP Variable):
    print ".items content Example 'var':\n-----------------------------"

    for key in dq.coll['var'].attDefn.keys():
        print "%-15s : %s" %(key, getattr(dq.coll['var'].items[0], key))

    .items content Example 'var':
    -----------------------------
    procnote        : ('',)
    description     :
    title           : Ambient Aerosol Absorption Optical Thickness at 550 nm
    prov            : CMIP5_aero
    unid            : fd6fef50-3468-11e6-ba71-5404a60d96b5
    label           : abs550aer
    provmip         : CMIP5
    sn              : atmosphere_absorption_optical_thickness_due_to_ambient_aerosol
    units           : 1.0
    procComment     :
    uid             : 0baf6a333b91c4db341b515c28cd2c05

## 3 - dq.inx Examples

Back to Synopsis
In [11]:

    # Index dq.inx is a simple lookup table
    #   dq.inx.uid[UID] returns the record corresponding to 'UID'

    #   Example from above:
    item = dq.inx.uid['0baf6a333b91c4db341b515c28cd2c05']

    print item
    print "Label:", item.label
    print "Title:", item.title
    print "Units:", item.units

    Item <1.2 MIP Variable>: [abs550aer] Ambient Aerosol Absorption Optical Thickness at 550 nm
    Label: abs550aer
    Title: Ambient Aerosol Absorption Optical Thickness at 550 nm
    Units: 1.0

In [12]:

    #   dq.inx.iref_by_uid[UID]
    #    gives a list of IDs of objects linked to 'UID',
    #    as a tuple (SECT, ID)

    #   Example from above:
    id_list = dq.inx.iref_by_uid['0baf6a333b91c4db341b515c28cd2c05']

    for ID in id_list:
        print "#  Section: %-10s\n   UID: %15s\n   %s\n"\
        %(ID[0], ID[1], dq.inx.uid[ID[1]])

    #  Section: vid
       UID: 19bebf2a-81b1-11e6-92de-ac72891c3257
       Item <1.3 CMOR Variable>: [abs550aer] ambient aerosol absorption optical thickness at 550 nm

    #  Section: tid
       UID: atmosphere_absorption_optical_thickness_due_to_ambient_aerosol
       Item <3.8 Remarks about other items>: [__unset__] __unset__

In [13]:

    #   dq.inx.iref_by_sect[UID].a[SECT]
    #    gives a list of IDs of 'SECT' linked to 'UID'

    #   Example from above:
    id_list = dq.inx.iref_by_sect['0baf6a333b91c4db341b515c28cd2c05'].a['CMORvar']

    for ID in id_list:
        item = dq.inx.uid[ID]
        print "%15s | %10s | %s" \
        %(item.label, item.mipTable, item.vid)

          abs550aer | aermonthly | 0baf6a333b91c4db341b515c28cd2c05

In [14]:

    # The same can be achieved using dq.coll, though:

    #   Example from above:
    CMORvar_list = [ item for item in dq.coll['CMORvar'].items \
               if item.vid=='0baf6a333b91c4db341b515c28cd2c05']

    for item in CMORvar_list:
        print "%15s | %10s | %s" \
        %(item.label, item.mipTable, item.vid)

          abs550aer | aermonthly | 0baf6a333b91c4db341b515c28cd2c05

## 4 - Presentation of dq.coll Sections

Back to Synopsis

Overview over all dq.coll sections and their items' attributes, as well as examples.

In [15]:

    from IPython.display import Image, display

    # Display Structure of DreqPy
    display(Image(filename='files/DreqPyStructure.png'))

In [16]:

    def showSect(sect, desc=False):
        """
        Print the Content of dq.coll's sections
        Arg: sect - str     - section key
             desc - boolean - also print description
                              (Default: False)
        """

        # print header title
        print dq.coll[sect].header.title
        print "-"*len(dq.coll[sect].header.title)

        # print Section name and description
        for subsect in dq.coll[sect].attDefn.keys():
            if desc==True:
                print "# %15s  |  %s%s" \
                %(subsect, dq.coll[sect].attDefn[subsect].title, \
                "\n"+dq.coll[sect].attDefn[subsect].description if \
                (str(dq.coll[sect].attDefn[subsect].description)!="" or \
                subsect in str(dq.coll[sect].attDefn[subsect].description)) \
                else "")
            else:
                print "# %15s  |  %s" \
                %(subsect, dq.coll[sect].attDefn[subsect].title)

        # print Example of first item in list
        print "-------\nExample\n-------"
        for subsect in dq.coll[sect].attDefn.keys():
                print "# %15s  |  %s"\
                %(subsect, getattr(dq.coll[sect].items[0], subsect) if \
                 subsect not in str(getattr(dq.coll[sect].items[0], subsect)) \
                 else "")

In [17]:

    # MIPs
    print "Chicken or egg? MIP or Objective?\n"\
          "The CMIP endorsed MIP (Model Intercomparison Project) is linked to scientific objectives."
    display(Image(filename='files/DreqPyStructure_1.png'))
    showSect('mip')
    #showSect('mip', desc=True) #Activate for additional info if there is any

    Chicken or egg? MIP or Objective?
    The CMIP endorsed MIP (Model Intercomparison Project) is linked to scientific objectives.

    1.1 Model Intercomparison Project
    ---------------------------------
    #             url  |  Project Home Page
    #     description  |  Description of the Model Intercomparison Project
    #           title  |  MIP title
    #             uid  |  Record identifier
    #           label  |  MIP short name
    -------
    Example
    -------
    #             url  |  http://www.wcrp-climate.org/wgcm-cmip/wgcm-cmip6
    #     description  |  __unset__
    #           title  |  Coupled Model Intercomparison Project, Phase 6
    #             uid  |  CMIP6
    #           label  |  CMIP6

In [18]:

    # Objective
    showSect('objective')
    #showSect('objective', desc=True) #Activate for additional info if there is any

    1.6 Scientific objectives
    -------------------------
    #             mip  |  Endorsed MIP
    #     description  |  Description
    #           title  |  Record Title
    #             uid  |  Record identifier
    #           label  |  Record Label
    -------
    Example
    -------
    #             mip  |  C4MIP
    #     description  |  quantify and understand land and ocean carbon cycle feedbacks with climate
    #           title  |  climate-carbon cycle feedbacks
    #             uid  |  dc90cf0a-8308-11e5-b787-ac72891c3257
    #           label  |  CarbonFeedbacks

In [19]:

    # Experiments
    print "MIPs may define experiments or solely request data from experiments."
    print "Experiments are organised in experiment groups."
    display(Image(filename='files/DreqPyStructure_2.png'))
    showSect('experiment')
    #showSect('experiment', desc=True) #Activate for additional info if there is any

    MIPs may define experiments or solely request data from experiments.
    Experiments are organised in experiment groups.

    1.5 Experiments
    ---------------
    #          nstart  |  Number of start dates
    #             yps  |  Years per simulation
    #          starty  |  Start year
    #     description  |  Description
    #           title  |  Record Title
    #            endy  |  End year
    #            ensz  |  Ensemble size
    #           label  |  Record Label
    #            egid  |  Identifier for experiment group
    #            tier  |  Tier of experiment
    #             mip  |  MIP defining experiment
    #            ntot  |  Total number of years
    #            mcfg  |  Model category
    #         comment  |  Comment
    #             uid  |  Record identifier
    -------
    Example
    -------
    #          nstart  |  1
    #             yps  |  85
    #          starty  |  2015.0
    #     description  |  Emission-driven future scenario simulation,  biogeochemically-coupled
    #           title  |  __unset__
    #            endy  |  2100.0
    #            ensz  |  [1]
    #           label  |  esmssp5-85bgc
    #            egid  |  b111fe1c-aab7-11e6-9fd2-ac72891c3257
    #            tier  |  [2]
    #             mip  |  C4MIP
    #            ntot  |  85
    #            mcfg  |
    #         comment  |
    #             uid  |  b112010a-aab7-11e6-9fd2-ac72891c3257

In [20]:

    # Experiment Groups
    showSect('exptgroup')
    #showSect('exptgroup', desc=True) #Activate for additional info if there is any

    1.9 Experiment Group
    --------------------
    #         tierMin  |  Minimum tier of experiments in group
    #           title  |  Record Title
    #             uid  |  Record identifier
    #            ntot  |  Total number of years
    #           label  |  Record Label
    -------
    Example
    -------
    #         tierMin  |  1
    #           title  |  __unset__
    #             uid  |  b111e90e-aab7-11e6-9fd2-ac72891c3257
    #            ntot  |  140
    #           label  |  C4mip1

In [21]:

    # Request Item
    print "The request items build up the data request."
    print "They are linked to (or: requested by) either MIPs, experiments or experiment groups."
    print "Request items hold information about the time slice of the requested variables they include\n"\
          "as well as the possibility to set their priority and tier."
    display(Image(filename='files/DreqPyStructure_3.png'))
    showSect('requestItem')
    #showSect('requestItem', desc=True) #Activate for additional info if there is any

    The request items build up the data request.
    They are linked to (or: requested by) either MIPs, experiments or experiment groups.
    Request items hold information about the time slice of the requested variables they include
    as well as the possibility to set their priority and tier.

    3.2 Request Item: specifying the number of years for an experiment
    ------------------------------------------------------------------
    #          treset  |  Option to override tier set for experiment(s)
    #          nexmax  |  Maximum number of experiments requested.
    #             uid  |  Record Identifier
    #           title  |  Record Title
    #          nenmax  |  Number of ensemble members requested.
    #           nymax  |  Number of years requested.
    #            expt  |  Name of experiment or group of experiments
    #              ny  |  Default number of years.
    #          preset  |  Option to override priority set in each variable group
    #          tslice  |  Selection of years from experiment
    #            rlid  |  Identifier of corresponding requestLink
    #             tab  |  Redundant?
    #            esid  |  Identifier experiment(s): a link to an experiment, an experiment group or a MIP
    #             mip  |  The MIP making the request.
    #           label  |  Record Label
    #     esidComment  |  Comment on experiment(s) linked to.
    -------
    Example
    -------
    #          treset  |
    #          nexmax  |  -999
    #             uid  |  b7095ea2-aaa6-11e6-860b-ac72891c3257
    #           title  |  C4MIP, C4MIP-Basic, historical-ext
    #          nenmax  |  -1
    #           nymax  |  -1.0
    #            expt  |  historical-ext
    #              ny  |  160
    #          preset  |  -1
    #          tslice  |
    #            rlid  |  efc0d8f0-5629-11e6-9079-ac72891c3257__00
    #             tab  |  C4MIP-Basic
    #            esid  |  b1148498-aab7-11e6-9fd2-ac72891c3257
    #             mip  |  C4MIP
    #           label  |  C4mipC4mipBasic
    #     esidComment  |  Experiment historical-ext

In [22]:

    # Time Slice
    showSect('timeSlice')
    #showSect('timeSlice', desc=True) #Activate for additional info if there is any

    3.11 Time Slices for Output Requests
    ------------------------------------
    #             end  |  End year
    #             uid  |  Unique identifier
    #           title  |  Record Title
    #          nyears  |  Total number of years
    #           label  |  Record Label
    #           start  |  Start year
    #            step  |  Step (years)
    #       startList  |  Optional list of start times.
    #           child  |  Child experiment
    #        sliceLen  |  Length of slice
    #            type  |  Type of time slice
    #    sliceLenUnit  |  Units of slice length
    #     description  |  Description
    -------
    Example
    -------
    #             end  |  2020
    #             uid  |  _slice_DAMIP42
    #           title  |  DAMIP 42 year
    #          nyears  |  42.0
    #           label  |  DAMIP42
    #           start  |  1979
    #            step  |
    #       startList  |
    #           child  |  __unset__
    #        sliceLen  |
    #            type  |  simpleRange
    #    sliceLenUnit  |
    #     description  |

In [23]:

    # Request Variable Group
    print "The request variable groups are defined by MIPs."
    display(Image(filename='files/DreqPyStructure_4.png'))
    showSect('requestVarGroup')
    #showSect('requestVarGroup', desc=True) #Activate for additional info if there is any

    The request variable groups are defined by MIPs.

    3.1 Request variable group: a collection of request variables
    -------------------------------------------------------------
    #         refNote  |  Reference Note
    #             uid  |  Record Identifier
    #           title  |  Record Title
    #           label  |  Record Label
    #             mip  |  Endorsed MIP defining the variable group
    #             ref  |  Reference
    -------
    Example
    -------
    #         refNote  |  __unset__
    #             uid  |  6d99677c-5979-11e6-8fd9-ac72891c3257
    #           title  |  VIACSAB: CSP.4
    #           label  |  CSP-4
    #             mip  |  VIACSAB
    #             ref  |  __unset__

In [24]:

    # Request Link
    print "Each request item is linked (via request link) to a request variable group."
    print "Several request items can link to the same request variable group\n"\
          "for a different set of experiments, time slices, priorities, etc."
    print "Of course a request variable group can be requested by MIPs that did not define it."
    display(Image(filename='files/DreqPyStructure_5.png'))
    showSect('requestLink')
    #showSect('requestLink', desc=True) #Activate for additional info if there is any

    Each request item is linked (via request link) to a request variable group.
    Several request items can link to the same request variable group
    for a different set of experiments, time slices, priorities, etc.
    Of course a request variable group can be requested by MIPs that did not define it.

    3.3 Request link: linking a set of variables and a set of experiments
    ---------------------------------------------------------------------
    #         comment  |  Comment
    #         refNote  |  Note on reference
    #             uid  |  Record Identifier
    #            opar  |  parameter associated with *opt*
    #           title  |  Record Title
    #             opt  |  option for selecting a subset of variables
    #           label  |  Record Label
    #            grid  |  Grid options
    #             tab  |  Redundant
    #       objective  |  Science objectives associated with this request
    #             mip  |  Endorsed MIP requesting the data
    #             ref  |  Reference
    #           refid  |  reference to a request Variable Group
    #         gridreq  |  Grid option constraints
    -------
    Example
    -------
    #         comment  |
    #         refNote  |
    #             uid  |  26d3f07c-7c16-11e6-8a5b-ac72891c3257__02
    #            opar  |
    #           title  |  day-oth
    #             opt  |  list
    #           label  |  __unset__
    #            grid  |
    #             tab  |  day-oth
    #       objective  |
    #             mip  |  CORDEX
    #             ref  |
    #           refid  |  26d3f07c-7c16-11e6-8a5b-ac72891c3257
    #         gridreq  |

In [25]:

    # Objective Link
    print "Request items are linked to scientific objectives (via objective link and request link)."
    display(Image(filename='files/DreqPyStructure_6.png'))
    showSect('objectiveLink')
    #showSect('objectiveLink', desc=True) #Activate for additional info if there is any

    Request items are linked to scientific objectives (via objective link and request link).

    3.7 Link between scientific objectives and requests
    ---------------------------------------------------
    #             rid  |  Identifier for a request link
    #           title  |  Record Title
    #             uid  |  Record identifier
    #             oid  |  Identifier for a scientific objective
    #           label  |  Record Label
    -------
    Example
    -------
    #             rid  |  efc0e43a-5629-11e6-9079-ac72891c3257__00
    #           title  |  __unset__
    #             uid  |  8817b79e-aab8-11e6-b8d4-ac72891c3257
    #             oid  |  dc919dcc-8308-11e5-b787-ac72891c3257
    #           label  |  C4MIP

In [26]:

    # Request Variable
    print "A request variable group holds a list of request variables."
    print "Request variables are basically links to CMOR variables with an\n"\
          "additional information about the CMOR variables' priority."
    display(Image(filename='files/DreqPyStructure_7.png'))
    showSect('requestVar')
    #showSect('requestVar', desc=True) #Activate for additional info if there is any

    A request variable group holds a list of request variables.
    Request variables are basically links to CMOR variables with an
    additional information about the CMOR variables' priority.

    1.4 Request variable (carrying priority and link to group)
    ----------------------------------------------------------
    #             uid  |  Record identifier
    #             vid  |  Identifier for MIP Output Variable
    #           title  |  Record Title
    #           label  |  Record Label
    #        priority  |  Variable priority
    #            vgid  |  Identifier for Variable Group
    #             mip  |  Endorsed MIP
    -------
    Example
    -------
    #             uid  |  8709b6f4-aab8-11e6-b8d4-ac72891c3257
    #             vid  |  19bee41e-81b1-11e6-92de-ac72891c3257
    #           title  |  emioa ((isd.005))
    #           label  |  emioa
    #        priority  |  1
    #            vgid  |  ef12fa96-5629-11e6-9079-ac72891c3257
    #             mip  |  AerChemMIP

In [27]:

    # CMOR Variable
    showSect('CMORvar')
    #showSect('CMORvar', desc=True) #Activate for additional info if there is any

    1.3 CMOR Variable
    -----------------
    #         shuffle  |  Shuffle: NetCDF compression parameter
    #             uid  |  Record Identifier
    #             vid  |  MIP Variable
    #       valid_min  |  Minimum expected value for this variable.
    #       frequency  |  Frequency of time steps to be archived.
    # ok_max_mean_abs  |  Maximum expected value of the mean absolute value at each point in time
    #           title  |  Long name
    #        rowIndex  |  Row index of entry in source sheet
    #            prov  |  Provenance
    #            stid  |  Link to a record specifying the structure of the variable (dimensions and associated variable attributes).
    #        mipTable  |  The MIP table: each table identifies a collection of variables
    #           label  |  CMOR Variable Name
    #            mtid  |  Link to MIP table record
    #        subGroup  |  Sub-group of variables in a table
    #            type  |  Data value type, e.g. float or double
    #   deflate_level  |  Deflate Level: NetCDF compression parameter
    #     description  |  Description
    #      processing  |  Processing notes
    #         deflate  |  Deflate: NetCDF compression parameter
    # mipTableSection  |  Section of a table
    #        provNote  |  Provenance Note
    # ok_min_mean_abs  |  Minimum expected value of the mean absolute value at each point in time
    # defaultPriority  |  Indicative priority for this parameter, which is over-ruled by the requestVar priority setting, but provides a reference for organisation of the CMORvariables
    #  modeling_realm  |  Modeling Realm
    #        positive  |  CMOR Directive Positive
    #       valid_max  |  Maximum expected value for this variable.
    -------
    Example
    -------
    #         shuffle  |
    #             uid  |  baa123da-e5dd-11e5-8482-ac72891c3257
    #             vid  |  1333394a296e7f8af6c9bad15cb9778d
    #       valid_min  |
    #       frequency  |  mon
    # ok_max_mean_abs  |
    #           title  |  Rate of Change of Net Dissolved Inorganic Carbon
    #        rowIndex  |  125
    #            prov  |  Omon ((isd.003))
    #            stid  |  f9475c26-80ba-11e6-ab6e-5404a60d96b5
    #        mipTable  |  Omon
    #           label  |  fddtdic
    #            mtid  |  MIPtable::Omon
    #        subGroup  |  Omon_oth
    #            type  |  real
    #   deflate_level  |
    #     description  |
    #      processing  |  integral over upper 100 m only.
    #         deflate  |
    # mipTableSection  |  Omon_oth
    #        provNote  |  Omon_oth
    # ok_min_mean_abs  |
    # defaultPriority  |  3
    #  modeling_realm  |  ocnBgchem
    #        positive  |
    #       valid_max  |

In [28]:

    # MIP Variable
    print "MIP variables are defined by a MIP and linked to CMOR variables."
    print "They hold information like the variables' unit, etc."
    display(Image(filename='files/DreqPyStructure_8.png'))
    showSect('var')
    #showSect('var', desc=True) #Activate for additional info if there is any

    MIP variables are defined by a MIP and linked to CMOR variables.
    They hold information like the variables' unit, etc.

    1.2 MIP Variable
    ----------------
    #        procnote  |  Processing notes
    #     description  |  Record Description
    #           title  |  Long name
    #            prov  |  Provenance
    #            unid  |  Link to Units section
    #           label  |  Variable Name
    #         provmip  |  MIP defining this variables
    #              sn  |  CF Standard Name
    #           units  |  Units
    #     procComment  |  Processing comments
    #             uid  |  Record identifier
    -------
    Example
    -------
    #        procnote  |  ('',)
    #     description  |
    #           title  |  Ambient Aerosol Absorption Optical Thickness at 550 nm
    #            prov  |  CMIP5_aero
    #            unid  |  fd6fef50-3468-11e6-ba71-5404a60d96b5
    #           label  |  abs550aer
    #         provmip  |  CMIP5
    #              sn  |  atmosphere_absorption_optical_thickness_due_to_ambient_aerosol
    #           units  |  1.0
    #     procComment  |
    #             uid  |  0baf6a333b91c4db341b515c28cd2c05

In [29]:

    # Variable Choice Links
    print "CMOR variables can be subject to a variable choice."
    print "Those choices are either defined by a hierarchy (ranked choice),\n"\
          "or the choice is made via the model configuration (configuration choice)."
    display(Image(filename='files/DreqPyStructure_9.png'))
    showSect('varChoiceLinkC')
    #showSect('varChoiceLinkC', desc=True) #Activate for additional info if there is any
    print "\n"*2
    showSect('varChoiceLinkR')
    #showSect('varChoiceLinkR', desc=True) #Activate for additional info if there is any

    CMOR variables can be subject to a variable choice.
    Those choices are either defined by a hierarchy (ranked choice),
    or the choice is made via the model configuration (configuration choice).

    3.6 Links a variable to a choice element
    ----------------------------------------
    #           cfgid  |  Configuration Option
    #             uid  |  Record identifier
    #             vid  |  Variable
    #           title  |  Record Title
    #             cfg  |  Configuration Value
    #             cid  |  Choice -- can provide a link to related variables
    #           label  |  Record Label
    -------
    Example
    -------
    #           cfgid  |  BoussinesqOceanConstantTh
    #             uid  |  880f3984-aab8-11e6-b8d4-ac72891c3257
    #             vid  |  baa3ea2a-e5dd-11e5-8482-ac72891c3257
    #           title  |  masscello [fx]
    #             cfg  |  True
    #             cid  |  boussinesq-masscello
    #           label  |  masscello-fx

    3.9 Links a variable to a choice element
    ----------------------------------------
    #             uid  |  Record identifier
    #             vid  |  Variable
    #             cid  |  Choice
    #           title  |  Record Title
    #            rank  |  For ranked choices, the rank of this variable (higher rank makes lower ranks redundant)
    #           label  |  Record Label
    -------
    Example
    -------
    #             uid  |  880ed8fe-aab8-11e6-b8d4-ac72891c3257
    #             vid  |  8baeab28-4a5b-11e6-9cd2-ac72891c3257
    #             cid  |  choices_HighResMIP.overlap.ta
    #           title  |  ta [3hrPlev]
    #            rank  |  3
    #           label  |  ta-3hrPlev

In [30]:

    # Model Configuration
    showSect('modelConfig')
    #showSect('modelConfig', desc=True) #Activate for additional info if there is any

    #Variable Choice
    print "\n"*2
    showSect('varChoice')
    #showSect('varChoice', desc=True) #Activate for additional info if there is any

    3.5 Model configuration options
    -------------------------------
    #             uid  |  Record identifier
    #           title  |  Record Title
    #           label  |  Record Label
    #           range  |  Range of valid values, e.g. xs:boolean
    #           usage  |  How the feature is relevant to the data request
    #            MIPs  |  MIPs which make use of this feature
    #            type  |  Type of model
    -------
    Example
    -------
    #             uid  |  TiledLandUseModel
    #           title  |  Sub-grid land use reporting
    #           label  |  TiledLandUseModel
    #           range  |  xs:boolean
    #           usage  |  Used on LUMIP to indicate variables only needed from models which have the capability to report on land use tiles.
    #            MIPs  |  LUMIP
    #            type  |  capability

    3.10 Indicates variables for which a there is a range of potential CMOR Variables
    ---------------------------------------------------------------------------------
    #     description  |  Record description
    #           title  |  Record Title
    #         varList  |  A colon separated list of variable names
    #           label  |  Record Label
    #      optionList  |  A list of options, one for each variable
    #           class  |  Class of choice: heirarchy|cfg
    #             uid  |  Record identifier
    -------
    Example
    -------
    #     description  |  The 3hrPlev and 6hrPlev_extr fields overlap
    #           title  |  hus [6hrPlev_extr]
    #         varList  |  3hrPlev:hus; 6hrPlev_extr:hus;
    #           label  |  overlap-hus
    #      optionList  |  2; 2;
    #           class  |  RedundancySet
    #             uid  |  choices_HighResMIP.overlap.hus

In [31]:

    # Structure
    print "CMOR variables are linked to the structure, that holds information about dimensions ..."
    display(Image(filename='files/DreqPyStructure_10.png'))
    showSect('structure')
    #showSect('structure', desc=True) #Activate for additional info if there is any

    CMOR variables are linked to the structure, that holds information about dimensions ...

    2.3 Dimensions and related information
    --------------------------------------
    #        procNote  |  Processing Note
    #             uid  |  Record Identifier
    #           odims  |  Other Dimensions
    #   flag_meanings  |  FLag Meanings
    #            prov  |  Provenance
    #            cmid  |  Link to Cell Methods Record
    #           title  |  Record Title
    #            dids  |  Identifiers for records in grids section for dimensions
    #            tmid  |  Temporal Shape
    #           label  |  Record Label
    #    cell_methods  |  Cell Methods
    #          coords  |  Coordinates
    #   cell_measures  |  Cell Measures
    #            spid  |  Spatial Shape
    #     flag_values  |  Flag Values
    #            cids  |  Identifiers for records in grids section for coordinates
    #     description  |  Description
    -------
    Example
    -------
    #        procNote  |
    #             uid  |  947ea05c-a5a8-11e6-acf8-5404a60d96b5
    #           odims  |  spectband
    #   flag_meanings  |
    #            prov  |  RFMIP
    #            cmid  |  CellMethods::tpt
    #           title  |  Instantaneous value (i.e. synoptic or time-step value), Global mean/constant [na-na] {spectband:} [tpt]
    #            dids  |  ('',)
    #            tmid  |  7a976ce0-8042-11e6-97ee-ac72891c3257
    #           label  |  str-x189
    #    cell_methods  |  time: point
    #          coords  |
    #   cell_measures  |
    #            spid  |  a6560fba-8883-11e5-b571-ac72891c3257
    #     flag_values  |
    #            cids  |  ('dim:spectband',)
    #     description  |

In [32]:

    # Temporal and Spatial Shape
    print "... such as the spatial and temporal shape."
    display(Image(filename='files/DreqPyStructure_11.png'))
    showSect('temporalShape')
    #showSect('temporalShape', desc=True) #Activate for additional info if there is any
    print "\n"*2
    showSect('spatialShape')
    #showSect('spatialShape', desc=True) #Activate for additional info if there is any

    ... such as the spatial and temporal shape.

    2.2 Temporal dimension
    ----------------------
    #      dimensions  |  Dimensions
    #           title  |  Record Title
    #           dimid  |  Identifiers for record in grids section
    #           label  |  Record Label
    #             uid  |  Record Identifier
    #     description  |  Description
    -------
    Example
    -------
    #      dimensions  |
    #           title  |  No temporal dimensions ... fixed field
    #           dimid  |  dim:
    #           label  |  None
    #             uid  |  7a96eb30-8042-11e6-97ee-ac72891c3257
    #     description  |  __unset__

    2.1 Spatial dimensions
    ----------------------
    #       levelFlag  |  Flag set to *false* if number of levels is optional (e.g. determined by the model)
    #      dimensions  |  List of spatial dimensions
    #           title  |  Record Title
    #          dimids  |  Identifiers for records in grids section
    #           label  |  Record Label
    #           shape  |  Shape - as used in input
    #          levels  |  Number of vertical levels (ignored if levelFlag=false)
    #             uid  |  Record Identifier
    -------
    Example
    -------
    #       levelFlag  |  False
    #      dimensions  |  latitude|olevel|basin
    #           title  |  Ocean Basin Meridional Section
    #          dimids  |  ('dim:latitude', 'dim:olevel', 'dim:basin')
    #           label  |  YB-O
    #           shape  |  __unset__
    #          levels  |  0
    #             uid  |  a655ffac-8883-11e5-b571-ac72891c3257

In [33]:

    # Grids/Dimensions
    print "Spatial and temporal shape are linked to a grid entry."
    print "Structure also links to the cell methods."
    display(Image(filename='files/DreqPyStructure_12.png'))
    showSect('grids')
    #showSect('grids', desc=True) #Activate for additional info if there is any

    Spatial and temporal shape are linked to a grid entry.
    Structure also links to the cell methods.

    1.7 Specification of dimensions
    -------------------------------
    #             uid  |  Identifier
    #          isGrid  |  grid?
    #    boundsValues  |  bounds _values
    #       valid_min  |  valid_min
    #    tolRequested  |  tol_on_requests: variance from requested values that is tolerated
    #            axis  |  axis
    #          tables  |  CMOR table(s)
    #           title  |  long name
    #        positive  |  positive
    #           label  |  CMOR dimension
    #           units  |  units
    #        altLabel  |  output dimension name
    #            type  |  type
    #       requested  |  requested
    #       direction  |  stored direction
    #     description  |  description
    #         isIndex  |  index axis?
    #    standardName  |  standard name
    # boundsRequested  |  bounds_ requested
    #          bounds  |  bounds?
    #           value  |  value
    #          coords  |  coords_attrib
    #       valid_max  |  valid_max
    -------
    Example
    -------
    #             uid  |  dim:typebare
    #          isGrid  |
    #    boundsValues  |
    #       valid_min  |
    #    tolRequested  |
    #            axis  |
    #          tables  |  Lmon
    #           title  |  surface type
    #        positive  |
    #           label  |  typebare
    #           units  |
    #        altLabel  |  type
    #            type  |  character
    #       requested  |
    #       direction  |
    #     description  |
    #         isIndex  |
    #    standardName  |  area_type
    # boundsRequested  |  []
    #          bounds  |  no
    #           value  |  bare_ground
    #          coords  |  type_description
    #       valid_max  |

In [34]:

    # Cell Methods
    showSect('cellMethods')
    #showSect('cellMethods, desc=True) #Activate for additional info if there is any if there is any

    7.1 Cell Methods
    ----------------
    #           title  |  Record Title
    #             uid  |  Record Identifier
    #    cell_methods  |  Cell Methods String
    #           label  |  Record Label
    -------
    Example
    -------
    #           title  |  Temporal Sum
    #             uid  |  CellMethods::tsum
    #    cell_methods  |  time: sum
    #           label  |  tsum

In [35]:

    # Standard Name
    print "Grids and MIP variables have a defined standard name."
    print "This concludes the list with the most important objects \n"\
        "of the DreqPy CMIP6 Data Request structure."
    display(Image(filename='files/DreqPyStructure.png'))
    showSect('standardname')
    #showSect('standardname', desc=True) #Activate for additional info if there is any

    Grids and MIP variables have a defined standard name.
    This concludes the list with the most important objects
    of the DreqPy CMIP6 Data Request structure.

    1.8 CF Standard Names
    ---------------------
    #           units  |  Canonical Units
    #     description  |  Record Description
    #           title  |  Record Title
    #             uid  |  CF Standard Name
    #           label  |  Record Label
    -------
    Example
    -------
    #           units  |  s
    #     description  |  The quantity with standard name acoustic_signal_roundtrip_travel_time_in_sea_water is the time taken for an acoustic signal to propagate from the emitting instrument to a reflecting surface and back again to the instrument. In the case of an instrument based on the sea floor and measuring the roundtrip time to the sea surface, the data are commonly used as a measure of ocean heat content.
    #           title  |  Acoustic Signal Roundtrip Travel Time in Sea Water
    #             uid  |  acoustic_signal_roundtrip_travel_time_in_sea_water
    #           label  |  AcousticSignalRoundtripTravelTimeInSeaWater

## 5 - Advanced Example

Back to Synopsis

Get a list of all variables requested by the CMIP6 Endorsed MIP 'C4MIP':

To attain the requested variables for an experiment, experiment group, or MIP, the first step is to find all requestItem-items (or requestLink-items) related to this experiment/experiment group/MIP.

Then proceed to find all requestVarGroup-items by following the Request Links. The requestVarGroup-items hold the requestVar-items, which finally are linked to the desired CMORvar-items.

In [36]:

    # Collect all 'requestLink'-items, defined by 'C4MIP':
    C4MIP_rL = [ rLink for rLink in dq.coll['requestLink'].items\
               if rLink.mip=='C4MIP']
    print "%i requestLink-items found for 'C4MIP'.\n" % len(C4MIP_rL)

    12 requestLink-items found for 'C4MIP'.

In [37]:

    # Find the 'requestVarGroup'-items, associated with these requestLinks,
    #   (not all requestVarGroups have to be defined by C4MIP!):
    C4MIP_rVG = set([ dq.inx.uid[rLink.refid] for rLink in C4MIP_rL ])

    print "%i requestVarGroup-items found for 'C4MIP'.\n" % len(C4MIP_rVG)

    12 requestVarGroup-items found for 'C4MIP'.

In [38]:

    # Get the UIDs of the requestVarGroup-items:
    C4MIP_rVG_UIDs = [ rVG.uid for rVG in C4MIP_rVG ]

    # Find the 'requestVar'-items, associated with these requestVarGroups:
    C4MIP_rV = set([ rV for rV in dq.coll['requestVar'].items\
                   if rV.vgid in C4MIP_rVG_UIDs])

    print "%i requestVar-items found for 'C4MIP'.\n" % len(C4MIP_rV)

    702 requestVar-items found for 'C4MIP'.

In [39]:

    # Get the CMORvar-UIDs from the requestVar-items:
    C4MIP_rV_VIDs = [ rV.vid for rV in C4MIP_rV ]

    # Find the 'CMORvar'-items, associated with these requestVars:
    C4MIP_CMORvars = set([ cmvar for cmvar in dq.coll['CMORvar'].items\
                     if cmvar.uid in C4MIP_rV_VIDs ])

    print "%i CMORvar-items found for 'C4MIP'\n" % len(C4MIP_CMORvars)
    print "Here are the first 10 list entries:"

    C4MIP_CMORvars=list(C4MIP_CMORvars) # set -> list conversion
    for i in range(0, 10):
        if i<len(C4MIP_CMORvars):
            cmvar = C4MIP_CMORvars[i]
            print " # %-15s | mipTable: %-10s | Frequency: %-2s\n   LongName: %s\n"\
                %(cmvar.label, cmvar.mipTable, cmvar.frequency, cmvar.title)

    629 CMORvar-items found for 'C4MIP'

    Here are the first 10 list entries:
     # mrlso           | mipTable: emMon      | Frequency: mon
       LongName: Soil Liquid Water Content

     # cSoilTree       | mipTable: emMon      | Frequency: mon
       LongName: __from new

     # netAtmosLandC14Flux | mipTable: emMon      | Frequency: mon
       LongName: Net Mass Flux of 14C between atmosphere and land (positive into land) as a result of all processes.

     # fFireNat        | mipTable: emMon      | Frequency: mon
       LongName: Carbon Mass Flux into Atmosphere due to CO2 Emission from natural Fire

     # sispeed         | mipTable: SIday      | Frequency: day
       LongName: Sea-ice speed

     # mrso            | mipTable: Lmon       | Frequency: mon
       LongName: Total Soil Moisture Content

     # siage           | mipTable: SImon      | Frequency: mon
       LongName: Age of sea ice

     # intppdiaz       | mipTable: emMon      | Frequency: mon
       LongName: Net Primary Mole Productivity of Carbon by Diazotrophs

     # intpppico       | mipTable: emMon      | Frequency: mon
       LongName: Net Primary Mole Productivity of Carbon by Picophytoplankton

     # raShrub         | mipTable: emMon      | Frequency: mon
       LongName: __from new

## 6 - Data Request Volume Estimate

Back to Synopsis

Estimate the Volume from a Data Request

In [40]:

    # Import
    from dreqPy import scope

    # Initialisation
    sc = scope.dreqQuery()

    print "Using DreqPy (Data Request Python API) in version %s"\
        % str(sc.dq.version)

    # Type   Storage Allocation
    #--------------------------
    # char   1 Byte
    # bool   1 Byte
    # short  2 Byte
    # int    4 Byte
    # long   4 Byte
    # float  4 Byte
    # double 8 Byte

    # Assume 50% compression:
    bytesPerFloat = 2.

    # Set maximum tier [1,4]
    tierMax       = 4

    # Set maximum priority [1,3]
    priorityMax   = 3

    # Print Volume Estimate Configuration settings
    print "\nDefault model configuration settings:"
    for key in sc.mcfg.keys():
        print "%-5s  %s"  %(str(key), str(sc.mcfg[key]))

    # Configure Volume Estimate:
    # Example MPI-ESM1.2-HR
    #----------------------
    # MPIOM Ocean (Res. TP04L40, ~0.4°)
    nho  = 6811850/40  # Nr. of hor. mesh points in ocean
    nlo  = 40          # Nr. of vertical levels in ocean
    # ECHAM 6.3 Atmosphere (Res T127L95, ~1°)
    nha  = 7004160/95  # Nr. of hor. mesh points in atm.
    nla  = 95          # Nr. of vertical levels in atm.
    nlas = 43 #guess   # Nr. of vertical levels in StrSph.
    nh1  = 3*128       # Nr. of latitude points
    # JSBACH Land Model
    nls  = 5           # Nr. of vertical levels in soil model

    # Apply settings
    MCFG=[nho, nlo, nha, nla, nlas, nls, nh1]
    sc.setMcfg(MCFG)

    # Print Volume Estimate Configuration settings
    print "\nUpdated model configuration settings (Check!):"
    for key in sc.mcfg.keys():
        print "%-5s  %s"  %(str(key), str(sc.mcfg[key]))

    WARNING: spid has no size info: a6562dba-8883-11e5-b571-ac72891c3257 [8670b670-aab8-11e6-b8d4-ac72891c3257]
    Using DreqPy (Data Request Python API) in version 01.beta.42

    Default model configuration settings:
    nho    259200
    nlo    60
    nha    64800
    nla    40
    nlas   20
    nls    5
    nh1    100
    WARNING: spid has no size info: a6562dba-8883-11e5-b571-ac72891c3257 [8670b670-aab8-11e6-b8d4-ac72891c3257]

    Updated model configuration settings (Check!):
    nho    170296
    nlo    40
    nha    73728
    nla    95
    nlas   43
    nls    5
    nh1    384

In [41]:

    def VolumeEstimate(MIPorExpt, isMIP=True, priorityMax=3, tierMax=4, bytesPerFloat=2.):
        """
        Function to set up a configuration for
        "dreqPy.scope.dreqQuery"
        and call its function volByMip
        Args:
        MIPorExpt:    set(str)  - Set of MIP or Experiment UIDs
        isMIP:        bool - MIPorExpt is MIP or Experiment
                             Default: True (MIPorExpt is MIP!)
        priorityMax : int  - Maximum priority of the variables
                             taken into account [1,3]
                             Default: 3
        tierMax     : int  - Maximum tier of the experiments
                             taken into account [1,2,3,4]
                             Default: 4
        bytesPerFloat: float - Compression factor
                               Default: 2. # 50% compression
        """

        #Apply tier setting
        sc.setTierMax(tierMax)

        # Use dreqPy.scope.dreqQuery-function volByMip
        if isMIP==True:
            ss = 0.
            for mip in MIPorExpt:
                    # Calculate volByMip and sum up
                    x = sc.volByMip(mip, pmax=priorityMax)\
                                    *1.e-12*bytesPerFloat
                    print ('%-15s:  %5.1fTb'  % (mip,x))
                    ss += x
            z = sc.volByMip( set(MIPorExpt), pmax=priorityMax)\
                                    *1.e-12*bytesPerFloat

        # Use dreqPy.scope.dreqQuery-function volByMip with
        #    additional experiment arg
        else:
            VolByE    = list()
            MIPorExpt = list(MIPorExpt)
            for i in range(0, len(MIPorExpt)):
                #VolByE.append(list)
                Esize = 0.
                # Calculate volByMip and sum up for every experiment
                # if statement to avoid bug in version 01.beta.41 and older
                Esize+=sc.volByMip(set([mip.uid for mip in dq.coll['mip'].items \
                                if not mip.uid in ['CMIP6', 'DECK', \
                                'PDRMIP', 'SolarMIP', 'SPECS', 'CCMI', 'CMIP5']]), \
                                pmax=priorityMax, exptid=MIPorExpt[i])\
                                *1.e-12*bytesPerFloat
                print ('%-15s:  %5.1fTb'  % (dq.inx.uid[MIPorExpt[i]].label, Esize))
                VolByE.append(Esize)
            ss=0.
            for val in VolByE: ss+=val
            z = sc.volByMip(set([mip.uid for mip in dq.coll['mip'].items \
                            if not mip.uid in ['CMIP6', 'DECK', \
                            'PDRMIP', 'SolarMIP', 'SPECS', 'CCMI', 'CMIP5']]), \
                            pmax=priorityMax, exptid=set(MIPorExpt))\
                            *1.e-12*bytesPerFloat

        # Print Info and calculate Overlap (Overlap is 0 for Experiments only!)
        print ( 'Combined:  %5.1fTb'  % z )
        print ( 'Overlap:   %5.1fTb'  % (ss-z) )

In [42]:

    # Volume Estimate by MIP

    print ( '######### All variables ###########' )
    VolumeEstimate(set(["AerChemMIP", "C4MIP", "LUMIP"]), isMIP=True, \
                   priorityMax=3, tierMax=4, bytesPerFloat=2.)

    print ( '\n######### Top priority variables ###########' )
    VolumeEstimate(set(["AerChemMIP", "C4MIP", "LUMIP"]), isMIP=True, \
                   priorityMax=1, tierMax=4, bytesPerFloat=2.)

    ######### All variables ###########
    C4MIP          :  151.7Tb
    AerChemMIP     :   25.6Tb
    LUMIP          :   36.8Tb
    Combined:  187.1Tb
    Overlap:    27.0Tb

    ######### Top priority variables ###########
    C4MIP          :   46.8Tb
    AerChemMIP     :   13.6Tb
    LUMIP          :   24.0Tb
    Combined:   70.2Tb
    Overlap:    14.2Tb

In [43]:

    # Volume Estimate by Experiments

    # Get Experiment ids:
    ExpIDs=list()
    print "%-15s  %-8s  %-36s  %s\n%s" %("Experiment", "Ens.Size", "UID", "MIP", "-"*75)
    for exp in dq.coll['experiment'].items:
        if exp.label in ['historical', 'AMIP', 'abrupt4xCO2', '1pctCO2', 'control', \
                         'SSP245', 'SSP585', 'SSP370', 'SSP126']:
            ExpIDs.append(exp.uid)
            print "%-15s  %8s  %36s  %s" % (exp.label, ",".join([str(i) for i in exp.ensz]), exp.uid, exp.mip)

    # Get Volume Estimates
    print ( '\n######### All variables ###########' )
    VolumeEstimate(set(ExpIDs), isMIP=False, \
                   priorityMax=3, tierMax=4, bytesPerFloat=2.)

    print ( '\n######### Top priority variables ###########' )
    VolumeEstimate(set(ExpIDs), isMIP=False, \
                   priorityMax=1, tierMax=4, bytesPerFloat=2.)

    Experiment       Ens.Size  UID                                   MIP
    ---------------------------------------------------------------------------
    SSP245                  1  b1141b2a-aab7-11e6-9fd2-ac72891c3257  ScenarioMIP
    historical              1  b1147afc-aab7-11e6-9fd2-ac72891c3257  CMIP
    SSP126                  1  b1141ddc-aab7-11e6-9fd2-ac72891c3257  ScenarioMIP
    SSP585                  1  b1141468-aab7-11e6-9fd2-ac72891c3257  ScenarioMIP
    SSP370               1,10  b1141864-aab7-11e6-9fd2-ac72891c3257  ScenarioMIP
    1pctCO2                 1  b11475d4-aab7-11e6-9fd2-ac72891c3257  CMIP

    ######### All variables ###########
    ########### Selecting first in list .............
    ########### Selecting first in list .............
    ########### Selecting first in list .............
    ########### Selecting first in list .............
    1pctCO2        :   11.5Tb
    SSP126         :   12.9Tb
    SSP370         :   26.8Tb
    SSP245         :   11.9Tb
    historical     :   43.5Tb
    SSP585         :   12.9Tb
    ########### Selecting first in list .............
    ########### Selecting first in list .............
    ########### Selecting first in list .............
    ########### Selecting first in list .............
    Combined:  119.5Tb
    Overlap:     0.0Tb

    ######### Top priority variables ###########
    ########### Selecting first in list .............
    ########### Selecting first in list .............
    1pctCO2        :    7.8Tb
    SSP126         :   10.9Tb
    SSP370         :   16.6Tb
    SSP245         :    9.9Tb
    historical     :   28.0Tb
    SSP585         :   10.9Tb
    ########### Selecting first in list .............
    ########### Selecting first in list .............
    Combined:   84.1Tb
    Overlap:     0.0Tb

## 7 - Additional Functions

Back to Synopsis

"Talk" to the Data Request using this functions

In [44]:

    # Auxiliary Functions to find request variables for request Links or request Items
    def requestVarTOrequestLink(rv, toScreen=False):
            """
            Args:
              + rv - 'dreqPy.dreq.dreqItem_requestVar'-object
              + toScreen - Boolean (print result to std.out or not)
            Returns:
              + List of corresponding:
                  - 'dreqPy.dreq.dreqItem_requestLink'-objects
            """

            #requestVar -> requestVarGroup.uid
            rvg= set([ y.uid for y in dq.coll['requestVarGroup'].items if y.uid==rv.vgid ])
            #requestVarGroup.uid -> requestLink
            rl = set([ x for x in dq.coll['requestLink'].items if x.refid in rvg])

            #print to std.out
            if toScreen==True:
                    print "\nrequestLink(s) of requestVar '%s' (uid: %s):" \
                        % (rv.label, rv.uid)
                    for i in rl: print " %s (uid: %s) requested by mip '%s'" \
                            % (i.title, i.uid, i.mip)
                    print

            #return list of requestLink-objects
            return list(rl)

    def requestVarTOrequestItem(rv, toScreen=False):
            """
            Args:
              + rv - 'dreqPy.dreq.dreqItem_requestVar'-object
              + toScreen - Boolean (print result to std.out or not)
            Returns:
              + List of corresponding:
                  - 'dreqPy.dreq.dreqItem_requestItem'-objects
            """

            #requestVar->requestLink
            rl=set([ x.uid for x in requestVarTOrequestLink(rv) ])
            #requestLink->requestItem
            ri=set([ x for x in dq.coll['requestItem'].items if x.rlid in rl ])

            #print to std.out
            if toScreen==True:
                    print "\nrequestItem(s) of requestVar '%s' (uid: %s):" \
                        % (rv.label, rv.uid)
                    for i in ri: print "%s (uid: %s) requested by mip '%s'" \
                            % (i.title, i.uid, i.mip)
                    print

            #return list of requestItem-objects
            return list(ri)

    # Examples
    rl = requestVarTOrequestLink(dq.coll['requestVar'].items[9], True)
    ri = requestVarTOrequestItem(dq.coll['requestVar'].items[9], True)

    requestLink(s) of requestVar 'rlus' (uid: 8709fd80-aab8-11e6-b8d4-ac72891c3257):

    requestItem(s) of requestVar 'rlus' (uid: 8709fd80-aab8-11e6-b8d4-ac72891c3257):

In [45]:

    # Find all MIPs requesting a CMOR Variable
    def CMORvarTOmip(cmvar, toScreen=False):
            """
            Args:
              + cmvar - 'dreqPy.dreq.dreqItem_CMORvar'-object
              + toScreen - Boolean (print result to std.out or not)
            Returns:
              + List of corresponding 'dreqPy.dreq.dreqItem_mip'-objects
            """
            #CMORvar -> requestVar -> mip.label
            mips=set([ x.mip for x in dq.coll['requestVar'].items if x.vid==cmvar.uid ])
            #mip.label -> mip
            mips=set([ x for x in dq.coll['mip'].items if x.label in mips ])

            #print to stdout
            if toScreen==True:
                    print
                    print str(cmvar.label), ":", [mip.label for mip in mips]
                    print

            #return matches
            return list(mips)

    # Example for arbitrary CMOR Variable UID
    mips = CMORvarTOmip(dq.inx.uid['d22da9f2-4a9f-11e6-b84e-ac72891c3257'], True)

    nwdFracLut : ['LUMIP']

In [46]:

    #Find all experiments requesting a CMOR Variable
    def CMORvarTOexperiment(cmvar, toScreen=False):
            """
            Args:
              + cmvar - 'dreqPy.dreq.dreqItem_CMORvar'-object
              + toScreen - Boolean (print result to std.out or not)
            Returns:
              + List of corresponding 'dreqPy.dreq.dreqItem_experiment'-objects
            """

            #CMORvar -> requestVar
            rv=set([ x for x in dq.coll['requestVar'].items if x.vid==cmvar.uid ])
            #requestVar -> requestItem
            ri=list()
            for i in rv:
                    for j in requestVarTOrequestItem(i): ri.append(j)
            ri = set(ri)
            #requestItem -> mip,experiment,exptgroup -> experiment
            exp=list()
            for i in ri:
                    if type(dq.inx.uid[i.esid])==type(dq.coll['experiment'].items[0]):
                            exp.append(dq.inx.uid[i.esid])
                    elif type(dq.inx.uid[i.esid])==type(dq.coll['exptgroup'].items[0]):
                            for e in dq.coll['experiment'].items:
                                    if e.egid==i.esid: exp.append(e)
                    elif type(dq.inx.uid[i.esid])==type(dq.coll['mip'].items[0]):
                            #print "mip", dq.inx.uid[i.esid].uid, dq.inx.uid[i.esid].label
                            currmip=dq.inx.uid[i.esid].uid
                            for item in dq.coll['experiment'].items:
                                    if item.mip==currmip:
                                            exp.append(item)
                    else:
                            raise TypeError("Type must be dreqPy.dreq.dreqItem_experiment, \
                            dreqPy.dreq.dreqItem_exptgroup or dreqPy.dreq.dreqItem_mip!")
                            sys.exit(1)

            #print to stdout
            exp=set(exp)
            if toScreen==True:
                    print "\n%i Experiments linked to CMORvar '%s':" % (len(exp), cmvar.label)
                    for i in exp: print "  - %-15s :  %s" % (i.label, i.description)
                    print

            #return matches
            return list(exp)

    # Example for arbitrary CMOR Variable UID
    exps = CMORvarTOexperiment(dq.inx.uid['d22da9f2-4a9f-11e6-b84e-ac72891c3257'], True)

    35 Experiments linked to CMORvar 'nwdFracLut':
      - Land-Future     :  Land only simulations
      - SSP585ext       :  Extension of SSP5_85 to 2300, in style of CMIP5 extension. Assumes emissions eventually decline from 2100 levels to produce stabilized forcing by 2300.
      - esm-hist        :
      - SSP585          :  Future scenario with high radiative forcing by the end of century. Following approximately RCP8.5 global forcing pathway but with new forcing based on SSP5. Concentration-driven.
      - hist-noLULCC-01 :  Same as CMIP6 historical but with land cover held at 1850, no human activity; concentration driven
      - SSP370          :  Future scenario with high radiative forcing by the end of century. Reaches about 7.0 W/m2 by 2100; fills gap in RCP forcing pathways between 6.0 and 8.5 W/m2. Concentration-driven.
      - SSPXY           :  Future scenario with low radiative forcing by the end of century. Following a forcing pathway below RCP2.6. Specific SSP and 2100 forcing level to be finalized with IAM groups within next few months. Concentration-driven.
      - Land-Hist-cruNcep :  Land only simulations with different forcing data
      - esm-hist-ext    :
      - SSP3-7wSSP1-26landuse-01 :  Additional land use policy sensitivity simulation for high radiative forcing scenario, keep all forcings the same as ScenarioMIP SSP3-7 (deforestation scenario), but replace land use from SSP1-2.6 (afforestation) scenario; concentration-driven
      - SSP160          :  Future scenario with medium radiative forcing by the end of century. Following approximately RCP6.0 global forcing pathway but with new forcing based on SSP. Concentration-driven.
      - esmhistbgc      :  Emission-driven historical simulation,  biogeochemically-coupled
      - esmssp5-85bgc   :  Emission-driven future scenario simulation,  biogeochemically-coupled
      - historical-ext  :
      - SSP126over      :  21st century overshoot scenario relative to SSP1_26. Specific design to be finalized with IAM groups within next several months.
      - esmSSP5-85wSSP1-26landuse :  Additional land use policy sensitivity simulation for high radiative forcing scenario, keep all forcings the same as in C4MIP esmssp5-8.5 scenario except use SSP1-2.6 land use; emission driven
      - historical      :  CMIP6 historical
      - SSP245          :  Future scenario with medium radiative forcing by the end of century. Following approximately RCP4.5 global forcing pathway but with new forcing based on SSP. Concentration-driven.
      - idealized-global-deforest :  Idealized transient global deforestation with all other forcings held constant
      - hist-noLULCC-LND :  Historical land only simulation with land use held at 1850; no human activity; derivative of LMIP-hist (LS3MIP)
      - Land-Hist-wfdei :  Land only simulations with different forcing data
      - Land-Hist       :  Land only simulations
      - SSP126          :  Future scenario with low radiative forcing by the end of century. Following approximately RCP2.6 global forcing pathway but with new forcing based on SSP1. Concentration-driven.
      - Land-Hist-princeton :  Land only simulations with different forcing data
      - SSP3-7wSSP1-26landuse-02 :  Additional land use policy sensitivity simulation for high radiative forcing scenario, keep all forcings the same as SSP3-7 (deforestation scenario), but replace land use from SSP1-2.6 (afforestation) scenario; concentration-driven
      - SSP585extover   :  Extension of SSP5_85 to 2300. Assumes emissions decline linearly from 2100 to SSP1_26 levels by 2200, which is expected to produce substantial overshoot in radiative forcing.
      - SSP437          :  Future scenario with low radiative forcing by the end of century. Reaches about 3.7 W/m2 by 2100; fills gap in RCP forcing pathways between 4.5 and 2.6 W/m2. Concentration-driven.
      - SSP126ext       :  Extension of SSP1_26 to 2300, in style of CMIP5 extension. Extends negative emissions level reached in 2100 to produce slowly declining forcing.
      - hist-noLULCC-02 :  Same as CMIP6 historical but with land cover held at 1850, no human activity; concentration driven
      - landcover-manage-LND :  Factorial set of land only experiments with increasingly realistic treatment of land management; derivatives of LMIP-hist (LS3MIP)
      - SSP1-26wSSP3-7landuse :  Additional land use policy sensitivity simulation for low radiative forcing scenario, keep all forcings the same as ScenarioMIP SSP1-2.6 (afforestation scenario), but replace land use from SSP3-7 (afforestation) scenario; concentration-driven
      - idealized-reg-deforest :  Paired idealized timeslice control and deforestation experiments for specific regions (tropical, boreal, temperate?, TBD)
      - esmssp5-85extbgc :  Emission-driven future scenario simulation extension to 2300 biogeochemically-coupled
      - esmhistbgc-noLULCC :  Same as esmhistbgc (C4MIP) but with land cover held at 1850, no human activity; emission-driven
      - esmssp5-85      :  Emissions-driven future scenario simulation

In [47]:

    # Find objectives linked to a CMOR variable
    def CMORvarTOobjectives(cmvar, toScreen=False):
            """
            Args:
              + cmvar - 'dreqPy.dreq.dreqItem_CMORvar'-object
              + toScreen - Boolean (print result to std.out or not)
            Returns:
              + List of corresponding:
                  - 'dreqPy.dreq.dreqItem_objective'-objects
            """

            #CMORvar -> requestVar -> requestVarGroup.uid
            rvg = set([ x.vgid for x in dq.coll['requestVar'].items if x.vid==cmvar.uid ])
            #requestVarGroup.uid -> requestLink.uid
            rl  = set([ x.uid for x in dq.coll['requestLink'].items if x.refid in rvg ])
            #requestLink.uid -> objectiveLink -> objective
            ob  = set([ dq.inx.uid[x.oid] for x in dq.coll['objectiveLink'].items if x.rid in rl ])

            #print to std.out
            if toScreen==True:
                    print
                    for i in ob: print str(i.label), i.description
                    print

            #return objective(s)
            return list(ob)

    # Example for arbitrary CMOR Variable UID
    objectives = CMORvarTOobjectives(dq.inx.uid['d22da9f2-4a9f-11e6-b84e-ac72891c3257'], True)

    Lulcc Assess and quantify the biogeophysical and carbon cycle consequences for climate of historic and future land cover and land use change

In [48]:

    # List structure/dimensions connected to a CMOR Variable
    def CMORvarTOstructure(cmvar, toScreen=False):
            """
            Args:
              + cmvar - 'dreqPy.dreq.dreqItem_CMORvar'-object
              + toScreen - Boolean (print result to std.out or not)
            Returns:
              + List of corresponding:
                 - 'dreqPy.dreq.dreqItem_CMORvar'-object
                 - 'dreqPy.dreq.dreqItem_cellMethods'-object
                 - 'dreqPy.dreq.dreqItem_temporalShape'-object
                 - 'dreqPy.dreq.dreqItem_spathialShape'-object
                 - List of corresponding 'dreqPy.dreq.dreqItem_grids'-objects
            """

            #Follow links to related objects:
            struc=dq.inx.uid[cmvar.stid]
            cm=dq.inx.uid[struc.cmid]
            ts=dq.inx.uid[struc.tmid]
            ss=dq.inx.uid[struc.spid]
            grid=list()
            grid.append(dq.inx.uid[ts.dimid])
            for i in ss.dimids: grid.append(dq.inx.uid[i])

            #Print info to std.out
            if toScreen==True:
                    print
                    print "Dimension Info for CMORvar "+str(cmvar.label)
                    print "cellMethods  :" , cm.cell_methods
                    print "temporalShape:", ts.label, "|", \
                        ts.dimensions, "|", ts.description
                    print "spatialShape :", ss.label, "|", \
                        ss.dimensions, "|", ss.shape
                    print "grids        :\n------"
                    for i in grid: print "", i.label, "|", i.units, "|", \
                            i.standardName, "|", i.description
                    print

            #Return List of objects
            return [cmvar, cm, ts, ss, grid]

    # Example for arbitrary CMOR Variable UID
    structure_info = CMORvarTOstructure(dq.inx.uid['d22da9f2-4a9f-11e6-b84e-ac72891c3257'], True)

    Dimension Info for CMORvar nwdFracLut
    cellMethods  : time: mean
    temporalShape: time-mean | time | __unset__
    spatialShape : XY-na | longitude|latitude | __unset__
    grids        :
    ------
     time | days since ? | time | for time-mean fields
     longitude | degrees_east | longitude |
     latitude | degrees_north | latitude |

In [49]:

    # Find priorities linked to a CMOR Variable
    def CMORvarTOpriority(cmvar, toScreen=False):
            """
            Args:
              + cmvar - 'dreqPy.dreq.dreqItem_CMORvar'-object
              + toScreen - Boolean (print result to std.out or not)
            Returns:
              + integer List of corresponding priorities
            """

            #priority stored in requestVar and as preset in requestItem
            #CMORvar->requestVar
            rv=[ x for x in dq.coll['requestVar'].items if x.vid==cmvar.uid ]
            #requestVar->requestItem
            ri=list()
            for i in rv:
                    for j in requestVarTOrequestItem(i):
                            ri.append(j)

            #add all priorities from requestVar and requestItem
            plist=[ x.preset for x in ri ]
            for i in [ x.priority for x in rv ]: plist.append(i)
            #remove double entries and "-1", the default preset value
            plist=set(plist)
            plist.remove(-1)

            #print to std.out
            if toScreen==True:
                    pliststr=[str(i) for i in plist]
                    prioritystring=", ".join(pliststr)
                    print "\nThe CMORvar %s is linked to \n  \
                    - %i requestVar-items\n  - %i requestItem-items.\n\
                    Its priorities are: \033[1m%s\033[0m\n" \
                    % (cmvar.label, len(rv), len(ri), prioritystring)

            #return list of integers/priorities
            return list(sorted(plist))

    # Example for arbitrary CMOR Variable UID
    prts = CMORvarTOpriority(dq.inx.uid['6f4eb4e4-9acb-11e6-b7ee-ac72891c3257'], True)

    The CMORvar hfgeoubed is linked to
                      - 1 requestVar-items
      - 7 requestItem-items.
                    Its priorities are: 3

In [50]:

    # Find time slices linked to CMOR Variables for any request items
    def CMORvarTOtimeSlice(cmvar, toScreen=False):
            """
            Args:
              + cmvar - 'dreqPy.dreq.dreqItem_CMORvar'-object
              + toScreen - Boolean (print result to std.out or not)
            Returns:
              + List of corresponding 'dreqPy.dreq.dreqItem_timeSlice'-objects
                beware: str("notSet") will be one item in the list,
                        if any requestItems requests the CMORvar for
                        the entire simulation length!
            """

            #CMORvar -> requestVar
            rv=set([ x for x in dq.coll['requestVar'].items if x.vid==cmvar.uid ])
            #requestVar -> requestItem
            ri=list()
            for i in rv:
                    for j in requestVarTOrequestItem(i): ri.append(j)
            ri = list(set(ri))
            #requestItem -> timeSlice
            ts=list()
            for i in ri:
                    try:
                            ts.append(dq.inx.uid[i.tslice])
                    except KeyError:
                            ts.append("notSet")
            tsset =list(set(ts))

            #print to stdout
            if toScreen==True:
                    #Group timeSlices and requestItems:
                    riset=list()
                    n=0
                    for i in tsset:
                            riset.append(list())
                            for j in ri:
                                    try:
                                            if i.uid==j.tslice:
                                                    if j.label not in riset[n]: riset[n].append(j.label)
                                    except AttributeError:
                                            if j.label not in riset[n]: riset[n].append(j.label)
                            n+=1
                    #print
                    ristr=riset
                    for i in range(0, len(ristr)): ristr[i]=", ".join(ristr[i])
                    print "\n%i different timeSclices linked to CMORvar '%s' (%s):" % (len(tsset), cmvar.label, cmvar.uid)
                    j=0
                    for i in tsset:
                            if type(i)==str:
                                    print "  - %s (= entire simulation length)\n   \
                                    for requestItems %s" %(i, ristr[j])
                            else:
                                    print "  - %-15s :  %s\n   %s %s %s"\
                                            "%s\n    for requestItems %s" \
                                            % (i.label, i.description \
                                               if not "description" in str(i.description) \
                                               else "", \
                                               "Start: "+str(i.start) \
                                               if not "start" in str(i.start) \
                                               else "", \
                                               "End: "+str(i.end) \
                                               if not "end" in str(i.end) \
                                               else "", \
                                               "Step [a]: "+str(i.step) \
                                               if not "step" in str(i.step) \
                                               else "", \
                                               "SliceLen: "+str(i.sliceLen) \
                                               if not "slice" in str(i.sliceLen) \
                                               else "", ristr[j])
                            j+=1
                    print

            #return matches
            return tsset

    # Example for arbitrary CMOR Variable UID
    ts = CMORvarTOtimeSlice(dq.inx.uid['bab5d898-e5dd-11e5-8482-ac72891c3257'], True)

    8 different timeSclices linked to CMORvar 'rsdcs' (bab5d898-e5dd-11e5-8482-ac72891c3257):
      - piControl030    :
       Start: __unset__  Step [a]: 135.0
        for requestItems PmipPmipCfmon3dstd
      - piControl200    :
       Start: __unset__  Step [a]: 0.0
        for requestItems HighresmipHighresmipCfmon3dstd
      - piControl140    :
       Start: __unset__  Step [a]: 25.0
        for requestItems CfmipCfmon3dstd
      - piControl165    :
       Start: __unset__  Step [a]: 0.0
        for requestItems AerchemmipCfmon3dstd
      - piControl100    :
       Start: __unset__  Step [a]: 65.0
        for requestItems DamipCfmon3dstd
      - DAMIP20         :
       Start: 2081 End: 2100
        for requestItems DamipCfmon3dstd
      - notSet (= entire simulation length)
                                       for requestItems AerchemmipCfmon3dstd, abrupt2xco2, PmipPmipCfmon3dstd, GeomipCfmon3dstd, cfmip3, DamipCfmon3dstd, CfmipCfmon3dstd, scenariomip2, HighresmipHighresmipCfmon3dstd, amip4xco2, ssp437, scenariomip3, 1pctco2, ssp370, highresmip1, ssp126, scenariomip1, gmmip2, historical, amip
      - DAMIP42         :
       Start: 1979 End: 2020
        for requestItems DamipCfmon3dstd

In [51]:

    # Find Variable Choices the CMOR Variable is linked to
    def CMORvarTOvarChoice(cmvar, toScreen=False):
            """
            Args:
              + cmvar - 'dreqPy.dreq.dreqItem_CMORvar'-object
              + toScreen - Boolean (print result to std.out or not)
            Returns:
              + List of corresponding 'dreqPy.dreq.dreqItem_varChoice'-objects
            """

            #CMORvar -> var.uid
            #var=set([ x.uid for x in dq.coll['var'].items if x.uid==cmvar.vid ])
            var=set([x.uid for x in dq.coll['CMORvar'].items])
            #var -> varChoiceLinkC, varChoiceLinkR -> varChoice
            vclc=list(set([ x for x in dq.coll['varChoiceLinkC'].items if x.vid in var ]))
            vclr=list(set([ x for x in dq.coll['varChoiceLinkR'].items if x.vid in var ]))
            vcc=[ dq.inx.uid[x.cid] for x in vclc ]
            vcr=[ dq.inx.uid[x.cid] for x in vclr ]

            #print to std.out
            if toScreen==True:
                    print "%i configuration varChoices found for CMORvar '%s' (%s)"\
                            %(len(vcc), cmvar.label, cmvar.uid)
                    for i in range(0, len(vcc)):
                            print "  - %s:\n    %s\n    %s\n    for cfg value %s "\
                                    "in modelConfig %s (%s)"\
                                    %(str(vcc[i].label), str(vcc[i].varList), \
                                      str(vcc[i].optionList), str(vclc[i].cfg), \
                                      str(dq.inx.uid[vclc[i].cfgid].label), \
                                      str(vclc[i].cfgid))
                    print
                    print "%i rank varChoices found for CMORvar '%s' (%s)" \
                        %(len(vcr), cmvar.label, cmvar.uid)
                    for i in range(0, len(vcr)):
                            print "  - %s:\n    %s\n    %s\n    for rank %s"\
                                    %(str(vcr[i].label), str(vcr[i].varList), \
                                      str(vcr[i].optionList), str(vclr[i].rank))
                    print

            #return varChoice-objects
            return vcc+vcr

    # Example for arbitrary CMOR Variable UID
    vc = CMORvarTOvarChoice(dq.inx.uid['bab5d898-e5dd-11e5-8482-ac72891c3257'], True)

    11 configuration varChoices found for CMORvar 'rsdcs' (bab5d898-e5dd-11e5-8482-ac72891c3257)
      - msftmzsmpa:
        msftmzsmpa

        for cfg value False in modelConfig __unset__ (CartesianOceanGrid)
      - masscello:
        masscello

        for cfg value True in modelConfig BoussinesqOceanConstantTh (BoussinesqOceanConstantTh)
      - masscello:
        masscello

        for cfg value False in modelConfig BoussinesqOceanConstantTh (BoussinesqOceanConstantTh)
      - ficeberg:
        ficeberg

        for cfg value False in modelConfig __unset__ (DepthResolvedIcebergMelt)
      - msftmzsmpa:
        msftmzsmpa

        for cfg value False in modelConfig __unset__ (SometimesFixed-msftmzsmpa)
      - ficeberg:
        ficeberg

        for cfg value True in modelConfig __unset__ (DepthResolvedIcebergMelt)
      - msftmzsmpa:
        msftmzsmpa

        for cfg value False in modelConfig __unset__ (CartesianOceanGrid)
      - msftmzsmpa:
        msftmzsmpa

        for cfg value False in modelConfig __unset__ (SometimesFixed-msftmzsmpa)
      - msftmzsmpa:
        msftmzsmpa

        for cfg value False in modelConfig __unset__ (CartesianOceanGrid)
      - msftmzsmpa:
        msftmzsmpa

        for cfg value False in modelConfig __unset__ (CartesianOceanGrid)
      - msftmzsmpa:
        msftmzsmpa

        for cfg value False in modelConfig __unset__ (CartesianOceanGrid)

    8 rank varChoices found for CMORvar 'rsdcs' (bab5d898-e5dd-11e5-8482-ac72891c3257)
      - overlap-va:
        3hrPlev:va; 6hrPlev_extr:va; 6hrPlev:va;
        2; 2; 1;
        for rank 2
      - overlap-hus:
        3hrPlev:hus; 6hrPlev_extr:hus;
        2; 2;
        for rank 3
      - overlap-hus:
        3hrPlev:hus; 6hrPlev_extr:hus;
        2; 2;
        for rank 2
      - overlap-ta:
        3hrPlev:ta; 6hrPlev_extr:ta; 6hrPlev:ta;
        2; 2; 1;
        for rank 3
      - overlap-ta:
        3hrPlev:ta; 6hrPlev_extr:ta; 6hrPlev:ta;
        2; 2; 1;
        for rank 2
      - overlap-ua:
        3hrPlev:ua; 6hrPlev_extr:ua; 6hrPlev:ua;
        2; 2; 1;
        for rank 3
      - overlap-ua:
        3hrPlev:ua; 6hrPlev_extr:ua; 6hrPlev:ua;
        2; 2; 1;
        for rank 2
      - overlap-va:
        3hrPlev:va; 6hrPlev_extr:va; 6hrPlev:va;
        2; 2; 1;
        for rank 3

In [52]:

    # Find all CMORvars requested by a MIP
    def mipTOCMORvar(mip, toScreen=False, info=False):
            """
            Args:
              + mip - 'dreqPy.dreq.dreqItem_mip'-object
              + toScreen - Boolean (print result to std.out or not)
              + info - Boolean (print number of matches to std.out or not)
            Returns:
              + List of corresponding:
                  - 'dreqPy.dreq.dreqItem_CMORvar'-objects
            """

            #mip.label -> requestLink -> requestVarGroup.uid
            requestVarGroup_uid=set([ x.refid for x in dq.coll['requestLink'].items if x.mip==mip.uid ])
            #requestVarGroup.uid -> requestVar -> CMORvar.uid
            CMORvar_uid=set([ x.vid for x in dq.coll['requestVar'].items if x.vgid in requestVarGroup_uid ])

            #print result to std.out
            if toScreen==True:
                    print "\nCMORvars in mip %s:" % mip.label
                    for uid in CMORvar_uid: print dq.inx.uid[uid].label,
                    print "\n"
            if info==True:
                    print "[Info] mipTOCMORvar : %i CMORvars have been found for mip '%s'." \
                        % (len(CMORvar_uid), mip.label)

            #return List of CMORvar objects
            return list(set([dq.inx.uid[uid] for uid in CMORvar_uid]))

    # Example for arbitrary MIP
    cmvars = mipTOCMORvar(dq.inx.uid["VolMIP"], True, True)

    CMORvars in mip VolMIP:
    rtmt tauv ta23 zmswaero agesno epfy lwtoafluxaerocs psl uas n2o n2oClim vas ta va cropFrac hfevapds dpco2 tntsw clw co2mass masso tossq cl vas vtendogw23 friver rhs rsuscs fLuc mrsos sfdsi zg10 va23 snw expc100 snmLi fHarvest ta hur soga hfsnthermds hfbasin cLitterAbove clt c4PftFrac mrro tos sltovovrt ua23 wtem rlus rsus hus23 nppRoot thkcello ts ua hur o2min mrsos cfc12global c3PftFrac clivi baresoilFrac rldscs sltovgyre cLitterBelow vtendnogw23 cLitter co2massClim va fco2antt rsdscs mfo rlds msftmrho tro3Clim phalf clwvi cRoot cVeg hfsifrazil mlotst ch4 vtem hfss froc hfsifrazil2d rlut zg23 pflw cct rsntds ua tro3 mc pr zg co2Clim treeFracSecEver lai tauu hfrunoffds evs omldamax ps tas treeFracPrimDec fFire snd rsds rMaint nppLeaf rhsmax rsds hfsnthermds2d sci nbp uo uas cMisc tasmin vo pastureFrac snw ua10 tsnLi sic rsutcs masscello treeFracSecDec so cfc11global expsi100 gpp grassFrac hfibthermds2d treeFracPrimEver sfcWindmax hfls evspsbl pfull wfo rGrowth ta10 rlds cSoilMedium snc sblLi fVegSoil msftyrhompa fVegLitter hus19 psl ta expcalc100 psl hfgeou hfsithermds2d hus10 hfds zos tasmax tossq fgco2 nppWood tsl fsitherm n2oglobalClim prc fco2fos hfibthermds ps zg19 hfls hfsithermds hfls mrfso ch4globalClim rsut fGrazing mrlsl ci sbl hus clt htovgyre rsdt residualFrac utendnogw19 fco2nat sfcWind evspsblveg rlds burntArea hfrainds cli thetao wap10 rh fLitterSoil wap thetaoga tasmax sos cProduct prsn hfcorr rlut wap prc tntlw23 vtendogw19 huss zmtnt msftmyz ua19 ch4Clim msftbarot cfc113global cLeaf ch4global nep tpf zg prw prsn ua va evspsblsoi hfss va ficeberg2d swsffluxaero hcfc22global cWood mrros ficeberg pr zmlwaero tntogw va10 shrubFrac hfss utendepfd ra lwsffluxaero tos htovovrt ccb sncLi cSoilFast swtoafluxaerocs prveg hus tran prsn tas huss wap19 treeFrac mrro landCoverFrac epfz utendogw19 ta ps tasmin hfdsn aod550volso4 rlutcs pr fgo2 rhsmin rsus tntnogw lwsnl rlus ua cCwd sfcWind cSoilSlow vtendnogw19 utendogw23 usi tslsi hfrunoffds2d hurs n2oglobal npp vsi co2 cSoil hus wap23 ta19 utendnogw23 sootsn mrso sithick va19

    [Info] mipTOCMORvar : 301 CMORvars have been found for mip 'VolMIP'.

In [55]:

    # Find all CMORvars requested by all MIPs for a certain experiment
    def experimentTOCMORvar(exp, toScreen=False, info=False):
            """
            Args:
              + exp - 'dreqPy.dreq.dreqItem_experiment'-object
              + toScreen - Boolean (print result to std.out or not)
              + info - Boolean (print number of matches to std.out or not)
            Returns:
              + List of corresponding:
                  - 'dreqPy.dreq.dreqItem_CMORvar'-objects
            """

            #experiment -> exptgroup.uid
            egid=list(set([ x.uid for x in dq.coll['exptgroup'].items \
                               if exp.egid==x.uid ]))
            #experiment -> requestItem.uid
            #allitems=set([ x.uid for x in dq.coll['requestItem'].items if x.esid==exp.uid ])
            #experiment,exptgroup.uid -> requestLink.uid
            alllinks=set([ x.rlid for x in dq.coll['requestItem'].items \
                              if exp.uid==x.esid or x.esid in egid ])
            #requestLink.uid -> requestVarGroup.uid
            allvargroups=set([ x.refid for x in dq.coll['requestLink'].items \
                                  if x.uid in alllinks ])
            #Find requestVar from requestVarGroup:requestVarGroup.uid -> CMORvar.uid
            allcmorvars=set([ x.vid for x in dq.coll['requestVar'].items \
                                 if x.vgid in allvargroups ])
            #CMORvar -> var.uid
            #allvars=set([ x.vid for x in dq.coll['CMORvar'].items if x.uid in allcmorvars])

            #print results to std.out
            if len(egid)==1:
                    egid=egid[0]
            else:
                    raise ValueError("Multiple experiment groups found for experiment! \
                    An experiment should only belong to one group of experiments!")

            if toScreen==True:
                    print "\nThe following CMORvars are requested for experiment  %s (%s):" \
                        % (exp.label, exp.uid)
                    for i in allcmorvars: print dq.inx.uid[i].label,
            print
            if info==True:
                    print "\n[Info] experimentTOCMORvar : "\
                        "Your specified experiment '%s' is in exptgroup '%s'." \
                        % (exp.label, dq.inx.uid[egid].label)
                    #print len(allitems), "RequestItems found for your specifications." \
                    #    % (len(allitems), exp.label)
                    print "[Info] experimentTOCMORvar : "\
                        "%i RequestLinks found for experiment '%s'." \
                        % (len(alllinks), exp.label)
                    print "[Info] experimentTOCMORvar : "\
                        "%i RequestVarGroups found for experiment '%s'." \
                        % (len(allvargroups), exp.label)
                    #print "[Info] experimentTOCMORvar : \
                    #   %i requested Variables found for experiment '%s'." \
                    #   % (len(allvars), exp.label)
                    print "[Info] experimentTOCMORvar : "\
                        "%i CMORvariables found for experiment '%s'." \
                        % (len(allcmorvars), exp.label)

                    #return List of CMORvars
            return list(allcmorvars)

    # Example for arbitrary Experiment
    cmvars = experimentTOCMORvar(dq.inx.uid["b1141b2a-aab7-11e6-9fd2-ac72891c3257"], False, True)

    [Info] experimentTOCMORvar : Your specified experiment 'SSP245' is in exptgroup 'ScenarioMIP1'.
    [Info] experimentTOCMORvar : 38 RequestLinks found for experiment 'SSP245'.
    [Info] experimentTOCMORvar : 38 RequestVarGroups found for experiment 'SSP245'.
    [Info] experimentTOCMORvar : 317 CMORvariables found for experiment 'SSP245'.

In [56]:

    # Find all CMOR Variables linked to a certain cellMethod
    def cellMethodsTOCMORvar(cm, toScreen=False, info=False):
            """
            Args:
              + cm - 'dreqPy.dreq.dreqItem_cellMethods'-object
              + toScreen - Boolean (print result to std.out or not)
              + info - Boolean (print number of matches to std.out or not)
            Returns:
              + List of corresponding:
                  - 'dreqPy.dreq.dreqItem_CMORvar'-objects
            """

            #cellMethod -> structure.uid
            struc=set([ x.uid for x in dq.coll['structure'].items if x.cmid==cm.uid ])
            #structure.uid -> CMORvar
            cmvar=set([ x for x in dq.coll['CMORvar'].items if x.stid in struc ])

            #print results to std.out
            if toScreen==True:
                    print "\nThe following CMORvars have the cellMethod %s (%s):" \
                        % (cm.label, cm.cell_methods)
                    for i in cmvar: print i.label,
                    print
            if info==True:
                    print "\n[Info] cellMethodsTOCMORvar : "\
                    "%i CMORvars have been found for cellMethod '%s' (%s)." \
                    % (len(cmvar), cm.label, cm.cell_methods)

            #return list of CMORvars
            return list(cmvar)

    # Example for an arbitrary cell method
    cmvars = cellMethodsTOCMORvar(dq.coll['cellMethods'].items[4], True, True)

    The following CMORvars have the cellMethod amnsi-tpt (area: mean where sea_ice time: point):
    sishevel sistremax sistresave sidivvel

    [Info] cellMethodsTOCMORvar : 4 CMORvars have been found for cellMethod 'amnsi-tpt' (area: mean where sea_ice time: point).

In [57]:

    # Find Objectives linked to a MIP
    def mipTOobjective(mip, toScreen=False):
            """
            Args:
              + mip - 'dreqPy.dreq.dreqItem_mip'-object
              + toScreen - Boolean (print result to std.out or not)
            Returns:
              + List of corresponding:
                  - 'dreqPy.dreq.dreqItem_objective'-objects
            """

            #mip -> objective
            ob=set([ x for x in dq.coll['objective'].items if x.mip==mip.label ])

            #print to std.out
            if toScreen==True:
                    print "\nThe MIP '%s' has the following objective(s):" % mip.label
                    for i in ob: print " - %-15s: %s" % (i.label, i.description)
                    print
            #return List of objective-objects
            return list(ob)

    # Example for arbitrary MIP
    objectives = mipTOobjective(dq.inx.uid["VolMIP"], True)

    The MIP 'VolMIP' has the following objective(s):
     - Forcing        : Improve understanding of volcanic radiative forcing and constrain related uncertainties. Volcanic eruptions are among the largest sources of uncertainty in historical radiative forcing .
     - Predictability : A systematical assessment of regional climate variability - and associated predictability and prediction - during periods of strong volcanic forcing at both intraseasonal-to-seasonal and interannual-to-decadal time scales
     - Variability    : Improve understanding of the mechanism(s) underlying the dynamical climate response to large volcanic eruptions, in particular concerning influence on decadal variability
     - Paleo          : Evaluate models against paleo reconstructions This objective is motivated by the mismatch between simulated and reconstructed post-eruption surface cooling for volcanic eruptions during the last millennium
     - Sensitivity    : Improve the characterization of volcanic forcing and associated climate sensitivity through improved understanding of how the hydrological cycle and the large-scale circulation respond to volcanic forcing, through improved understanding of forced responses of the coupled ocean-atmosphere system, and through improved assessment of internal variability.
     - Dynamics       : Improve understanding of the mechanism(s) underlying the dynamical atmospheric response to large volcanic eruptions, in particular separating them from the direct radiative response
     - Evaluation     : Evaluate models against observations. This objective is motivated by the mismatch between observed and modeled seasonal to interannual dynamical responses to volcanic eruptions during the instrumental period, for instance the 1991 Pinatubo eruptions

In [59]:

    # Find objectives linked to an experiment
    def experimentTOobjective(exp, toScreen=False, info=False):
            """
            Args:
              + exp - 'dreqPy.dreq.dreqItem_experiment'-object
              + toScreen - Boolean (print result to std.out or not)
            Returns:
              + List of corresponding:
                  - 'dreqPy.dreq.dreqItem_objective'-objects
            """

            #experiment -> exptgroup.uid
            egid=list(set([ x.uid for x in dq.coll['exptgroup'].items \
                           if exp.egid==x.uid ]))
            #experiment -> requestItem.uid
            #allitems=set([ x.uid for x in dq.coll['requestItem'].items \
            #    if x.esid==exp.uid ])
            #experiment,exptgroup.uid -> requestLink.uid
            alllinks=set([ x.rlid for x in dq.coll['requestItem'].items \
                          if x.esid==exp.uid or x.esid in egid])
            #requestLink.uid -> requestVarGroup.uid
            ob=set([ dq.inx.uid[x.oid] for x in dq.coll['objectiveLink'].items \
                    if x.rid in alllinks ])

            #print results to std.out
            if len(egid)==1:
                    egid=egid[0]
            else:
                    raise ValueError("Multiple experiment groups found for experiment!" \
                        " An experiment should only belong to one group of experiments!")

            if toScreen==True:
                    print "\nThe following %i objective(s) are found for experiment %s (%s):" \
                        % (len(ob), exp.label, exp.uid)
                    for i in ob: print " - %-15s: %s" % (i.label, i.description)
            print
            if info==True:
                    print "[Info] experimentTOobjective : Your specified experiment "\
                            "'%s' is in exptgroup '%s'." \
                            % (exp.label, dq.inx.uid[egid].label)
                    print "[Info] experimentTOobjective : "\
                            "%i RequestLinks found for experiment '%s'." \
                            % (len(alllinks), exp.label)
                    print "[Info] experimentTOobjective : "\
                            "%i objectives found for experiment '%s'." \
                            % (len(ob), exp.label)

            #return List of objectives
            return list(ob)

    # Example for arbitrary experiment
    objectives = experimentTOobjective(dq.inx.uid["b1141b2a-aab7-11e6-9fd2-ac72891c3257"], True, True)

    The following 9 objective(s) are found for experiment SSP245 (b1141b2a-aab7-11e6-9fd2-ac72891c3257):
     - ISI-MIP        : ISI-MIP requirements input to VIACSAB
     - AgMIP          : AgMIP requirements input to VIACSAB
     - DBEM           : DBEM requirements input to VIACSAB
     - FISH-MIP       : FISH-MIP requirements input to VIACSAB
     - SMHI-FoUo      : SMHI-FoUo requirements input to VIACSAB
     - Atlantis       : Atlantis requirements input to VIACSAB
     - APECOSM        : APECOSM requirements input to VIACSAB
     - climateServices: climateServices requirements input to VIACSAB
     - Madingley      : Madingley requirements input to VIACSAB

    [Info] experimentTOobjective : Your specified experiment 'SSP245' is in exptgroup 'ScenarioMIP1'.
    [Info] experimentTOobjective : 38 RequestLinks found for experiment 'SSP245'.
    [Info] experimentTOobjective : 9 objectives found for experiment 'SSP245'.

In [62]:

    # Find all experiments
    def mipTOexperimentBYtier(mip, tier=[1,2,3,4], toScreen=False):
            """
            Args:
              + mip - 'dreqPy.dreq.dreqItem_mip'-object
              + tier - List containing desired tiers of the experiment, eg. [2] or [1,2]
              + toScreen - Boolean (print result to std.out or not)
            Returns:
              + List of corresponding 'dreqPy.dreq.dreqItem_experiment'-objects
            """

            #mip -> requestItem -> [mip/experiment/exptgroup.uid, [tier,reset]]
            exps=[ [x.esid, [] if "Option to override" in str(x.treset) \
                    else x.treset] for x in dq.coll['requestItem'].items if x.mip==mip.label]
            #[mip/experiment/exptgroup.uid, [tier,reset]] -> [experiment, [tier, reset]]
            expstier=list()
            exp=list()
            for i in exps:
                    if type(dq.inx.uid[i[0]])==type(dq.coll['experiment'].items[0]):
                            exp.append([dq.inx.uid[i[0]], i[1]])
                    elif type(dq.inx.uid[i[0]])==type(dq.coll['exptgroup'].items[0]):
                            for e in dq.coll['experiment'].items:
                                    if e.egid==i[0]: exp.append([e, i[1]])
                    elif type(dq.inx.uid[i[0]])==type(dq.coll['mip'].items[0]):
                            currmip=dq.inx.uid[i[0]].uid
                            for item in dq.coll['experiment'].items:
                                    if item.mip==currmip:
                                            exp.append([item, i[1]])
                    elif type(dq.inx.uid[i[0]])==type(dq.coll['remarks'].items[0]):
                        continue
                    else:
                        raise TypeError("Type must be dreqPy.dreq.dreqItem_experiment, "\
                                            "dreqPy.dreq.dreqItem_exptgroup or "\
                                            "dreqPy.dreq.dreqItem_mip!\n Type is %s" % (str(type(dq.inx.uid[i[0]]))))

            #Filter experiments by requested tier
            for i in exp:
                    #if treset performs a experiment.tier override
                    if i[1]!=[]:
                            for t in tier:
                                    if t in i[1]:
                                            expstier.append(i[0])
                    #if the experiment.tier specifies the tier
                    else:
                            for t in tier:
                                    if t in i[0].tier:
                                            expstier.append(i[0])

            #print to std.out
            expstier=set(expstier)
            if toScreen==True:
                    tierstring=[str(t) for t in tier]
                    tierstring=", ".join(tierstring)
                    print "\n%i Experiments linked to MIP '%s' for tier %s:" \
                        % (len(expstier), mip.uid, tierstring)
                    for i in expstier: print "  - %-15s :  %s" % (i.label, i.description)
                    print

            #Return experiment-objects of requested tier
            return list(expstier)

    # Example for an arbitrary MIP
    exps = mipTOexperimentBYtier(dq.inx.uid["C4MIP"], tier=[1], toScreen=True)

    32 Experiments linked to MIP 'C4MIP' for tier 1:
      - lgm             :  last glacial maximum
      - idealized-global-deforest :  Idealized transient global deforestation with all other forcings held constant
      - esmSSP5-85wSSP1-26landuse :  Additional land use policy sensitivity simulation for high radiative forcing scenario, keep all forcings the same as in C4MIP esmssp5-8.5 scenario except use SSP1-2.6 land use; emission driven
      - G7cirrus        :  increase cirrus ice crystal fall speed to reduce net forcing in SSP585 by 1 W m-2
      - esm1pcbgc       :  Biogeochemically-coupled specified concentration simulation in which CO2 increases at a rate of 1% per year until quadrupling
      - esmssp5-85      :  Emissions-driven future scenario simulation
      - hist-GHG        :  historical well-mixed GHG-only run
      - hist-nat        :  historical natural-only run
      - PlioExp         :  main forcings : trace gases, orography, ice-sheet
      - hist-aer        :  historical anthropogenic aerosols-only run
      - past1000        :  last millenium
      - SSP3-7wSSP1-26landuse-01 :  Additional land use policy sensitivity simulation for high radiative forcing scenario, keep all forcings the same as ScenarioMIP SSP3-7 (deforestation scenario), but replace land use from SSP1-2.6 (afforestation) scenario; concentration-driven
      - G1ext           :  Beginning from a preindustrial control run, simultaneously quadruple the CO2 concentration and reduce the solar constant such that the TOA radiative flux remains within +/m0.1 W/m2.
      - G6sulfate       :  Using equatorial SO2 injection, return the radiative forcing from a background of the ScenarioMIP high forcing to the ScenarioMIP middle forcing.
      - LIG             :  main forcings : orbital parameters, ice-sheet, trace gases
      - SSP245          :  Future scenario with medium radiative forcing by the end of century. Following approximately RCP4.5 global forcing pathway but with new forcing based on SSP. Concentration-driven.
      - esm-piControl   :
      - hist-noLULCC-LND :  Historical land only simulation with land use held at 1850; no human activity; derivative of LMIP-hist (LS3MIP)
      - midHolocene     :  mid-Holocene
      - Land-Hist       :  Land only simulations
      - SSP126          :  Future scenario with low radiative forcing by the end of century. Following approximately RCP2.6 global forcing pathway but with new forcing based on SSP1. Concentration-driven.
      - LFMIP-rmLC      :  Prescribed land conditions 30yr running mean
      - LFMIP-pdLC      :  Prescribed land conditions 1980-2014 climate
      - hist-noLULCC-01 :  Same as CMIP6 historical but with land cover held at 1850, no human activity; concentration driven
      - SSP585          :  Future scenario with high radiative forcing by the end of century. Following approximately RCP8.5 global forcing pathway but with new forcing based on SSP5. Concentration-driven.
      - piControl       :  DECK: control
      - esm-hist        :
      - historical      :  CMIP6 historical
      - SSP1-26wSSP3-7landuse :  Additional land use policy sensitivity simulation for low radiative forcing scenario, keep all forcings the same as ScenarioMIP SSP1-2.6 (afforestation scenario), but replace land use from SSP3-7 (afforestation) scenario; concentration-driven
      - 1pctCO2         :  DECK: 1pctCO2
      - G6solar         :  total solar irradiance reduction to reduce net forcing from SSP585 to SSP245
      - SSP370          :  Future scenario with high radiative forcing by the end of century. Reaches about 7.0 W/m2 by 2100; fills gap in RCP forcing pathways between 6.0 and 8.5 W/m2. Concentration-driven.

In [63]:

    # Find all CMOR Variables linked to a certain standardname
    def standardnameTOCMORvar(sn, toScreen=False):
            """
            Args:
              + sn - 'dreqPy.dreq.dreqItem_var'.sn or 'dreqPy.dreq.dreqItem_standardname'.uid,
                      which is the name of a standardname
              + toScreen - Boolean (print result to std.out or not)
            Returns:
              + List of corresponding 'dreqPy.dreq.dreqItem_CMORvar'-objects
            """

            #standardname->var.uid
            vl=set([ x.uid for x in dq.coll['var'].items if x.sn==sn ])
            #var->CMORvar
            cmvar=set([ x for x in dq.coll['CMORvar'].items if x.vid in vl ])

            #print to std.out
            if toScreen==True:
                    print "\nThe following %i CMORvars have been found for standardname '%s':" \
                        % (len(cmvar), sn)
                    for i in cmvar: print "  - %-10s (%-15s)" \
                            % (str(i.label), str(i.mipTable))
                    print

            #return corresponding 'dreqPy.dreq.dreqItem_CMORvar'-objects
            return list(cmvar)

    # Example for an arbitrary standard name
    cmvars = standardnameTOCMORvar(dq.inx.uid[dq.inx.uid['6f4eb4e4-9acb-11e6-b7ee-ac72891c3257'].vid].sn, True)

    The following 4 CMORvars have been found for standardname 'upward_geothermal_heat_flux_at_ground_level':
      - hfgeoubed  (LIyrgre        )
      - hfgeoubed  (LIfxant        )
      - hfgeoubed  (LIfxgre        )
      - hfgeoubed  (LIyrant        )

In [64]:

    # Find all CMOR Variables whose standard name includes a certain part
    def partOFstandardnameTOCMORvar(sn, toScreen=False):
            """
            Args:
              + sn - 'dreqPy.dreq.dreqItem_var'.sn or 'dreqPy.dreq.dreqItem_standardname'.uid,
                     which is the name of a standardname
              + toScreen - Boolean (print result to std.out or not)
            Returns:
              + List of corresponding 'dreqPy.dreq.dreqItem_CMORvar'-objects
            """

            #standardname->var.uid
            vl=set([ x.uid for x in dq.coll['var'].items if sn.lower() in x.sn.lower() ])
            #var->CMORvar
            cmvar=set([ (x, dq.inx.uid[x.vid].sn) \
                           for x in dq.coll['CMORvar'].items if x.vid in vl ])

            #print to std.out
            if toScreen==True:
                    print "\nThe following %i CMORvars have a standardname including '%s':" \
                        % (len(cmvar), sn.lower())
                    for i in cmvar: print "  - %-10s (%-15s): %s" \
                            % (str(i[0].label), str(i[0].mipTable), str(i[1]))
                    print

            #return corresponding 'dreqPy.dreq.dreqItem_CMORvar'-objects
            cmvar=[x[0] for x in cmvar]
            return cmvar

    # Example for an arbitrary part of a standardname
    cmvars = partOFstandardnameTOCMORvar("Wet_Deposition", True)

    The following 9 CMORvars have a standardname including 'wet_deposition':
      - wetnh3     (aermonthly     ): tendency_of_atmosphere_mass_content_of_ammonia_due_to_wet_deposition
      - wetnh4     (aermonthly     ): tendency_of_atmosphere_mass_content_of_ammonium_dry_aerosol_due_to_wet_deposition
      - wetss      (aermonthly     ): tendency_of_atmosphere_mass_content_of_seasalt_dry_aerosol_due_to_wet_deposition
      - wetso2     (aermonthly     ): tendency_of_atmosphere_mass_content_of_sulfur_dioxide_due_to_wet_deposition
      - wetbc      (aermonthly     ): tendency_of_atmosphere_mass_content_of_black_carbon_dry_aerosol_due_to_wet_deposition
      - wetnoy     (aermonthly     ): tendency_of_atmosphere_mass_content_of_all_nitrogen_oxides_due_to_wet_deposition
      - wetso4     (aermonthly     ): tendency_of_atmosphere_mass_content_of_sulfate_dry_aerosol_due_to_wet_deposition
      - wetdust    (aermonthly     ): tendency_of_atmosphere_mass_content_of_dust_dry_aerosol_due_to_wet_deposition
      - wetoa      (aermonthly     ): tendency_of_atmosphere_mass_content_of_particulate_organic_matter_dry_aerosol_due_to_wet_deposition

## 8 - Official Web-tools

Back to Synopsis

Official Link

MIP-Variables Search

Data Request Overview

## 9 - Outlook and Discussion

Back to Synopsis

### Outlook¶

Above functions can help to gain further insight into the data request. However to attain a complete and un-ambiguous list of variables you need to retain the links between the CMOR variable and the request item as well as the request variable. This information can then be written to a comma separated variable list to serve for example for automatic processing.

#### To be done by the dreqPy-developers:¶

- Several links are missing.
- Some experiment labels, etc. require fixing or update.
- Variable choices not yet fully implemented.
- Some bugs of the command line tool require fixing.
- According to the developers, the tool will be finalized by end of 2016.

### Discussion¶

The first question would be what information is needed for every entry in the csv list that the data request will be retrieved?

- var-label
- CMORvar-label
- units
- positive
- mipTable
- frequency
- valid_min
- valid_max
- Priority (incl. preset)
- tier of experiment (incl. treset)
- long-name
- standard-name
- processing
- description
- variable choice options
- grid (native, interpolated, ...)
- ... ?

For the future proceeding there are several options:

- Solely write the "master function", to translate the data request into a custom csv list and leave it to the user to configure the source code according to his wishes to generate the .csv file
- Embed above (and maybe more) functions into a Web-GUI: . offer customisable searches for all objects in DreqPy . offer customisable "master function" .csv output file as download
- Write a python based GUI (eg. using tkinter) with above features
- Write a command line based program with above features

In [ ]: