DMF Resources

Data items stored in the DMF, whether they be property data, property models, flowsheets, or other user-defined objects, are called “resources”. They are represented in the Python API by a class, Resource, with attributes for metadata and a generic “data” section where resource-type-specific information can be held. You store Resources in the DMF with the add method on an instance of the DMF class. See the DMF API Examples documentation for all the available operations.

Resource classes documented on this page: Resource, PropertyDataResource, FlowsheetResource.

A “special” type of resource is an “experiment”. The idea of experiments is that they anchor a set of resources that were the result of one coherent set of activities. Experiments are initialized with their associated DMF instance and have some convenience methods to make them a handy way to track a set of resources. See the experiment documentation page <experiment> for more details.

Here is a diagram of the resource structure:

../_images/resource-structure.png

Resources are populated by simply setting their attributes. The attributes are actually traits, using the Traitlets library, to make them “smarter” and easier to use. Some of the attributes, such as creator or sources, are lists of containers with more attributes. All the attributes are documented in the API Reference section.

Example

Below is an example of creating and then setting all (well, most) of the attributes for a resource that is a plot of the standard ‘iris’ dataset.

##############################################################################
# Institute for the Design of Advanced Energy Systems Process Systems
# Engineering Framework (IDAES PSE Framework) Copyright (c) 2018, by the
# software owners: The Regents of the University of California, through
# Lawrence Berkeley National Laboratory,  National Technology & Engineering
# Solutions of Sandia, LLC, Carnegie Mellon University, West Virginia
# University Research Corporation, et al. All rights reserved.
# 
# Please see the files COPYRIGHT.txt and LICENSE.txt for full copyright and
# license information, respectively. Both files are also available online
# at the URL "https://github.com/IDAES/idaes".
##############################################################################
from idaes.dmf import resource

# Create an empty resource
rsrc = resource.Resource(type='plot')

# Populate the resource. All the attributes are optional.
# This is set automatically to current time
rsrc.created = '2017-10-31'
# This is set automatically to current time
rsrc.modified = '2017-11-23'
# Description of the resource
rsrc.desc = 'Plots of the Iris dataset'
# Version of the resource
rsrc.version = resource.Version(revision='1.0.3-a7',
                                name='november-release')
# Name and contact email of creator of the resource
rsrc.creator = resource.Contact(name='Dan Gunter',
                                email='dang@science-lab.gov')
# Name and contact email of other people involved
rsrc.collaborators = [resource.Contact(name='John Eslick',
                      email='john@science-lab.gov'),
                     resource.Contact(name='David Miller',
                                      email='david@science-lab.gov')]
# Provenance: sources, such as publications -- but
# really anything is allowed -- for the resource.
rsrc.sources = [resource.Source(
             source='R. A. Fisher. "The use of multiple measurements '
                    'in taxonomic problems". '
                    'Annals of Eugenics. 7 (2): 179-188.',
             doi='10.1111/j.1469-1809.1936.tb02137.x', date='1936'),
           resource.Source(source='Edgar Anderson. '
                                  '"The species problem in Iris". '
                                  'Annals of the Missouri Botanical Garden. '
                                  '23 (3): 457-509. '
                                  'JSTOR 2394164',
                           date='1936')]
# Associated code files, including Jupyter notebooks.
# If these codes are worth making into their own Resources,
# you should not put them here, but instead link to them
# through the `relations` attribute.
rsrc.codes = [resource.Code(type='notebook',
                        desc='Python plotting code',
                       name='plot_iris_dataset.ipynb',
                       language='Python',
                       location='http://scikit-learn.org/stable/_downloads/'
                                'plot_iris_dataset.ipynb')]
# Associated data files. Same comment as for `codes` re: linking.
rsrc.datafiles = [resource.FilePath(path='iris-data.csv',
                                    desc='Iris data')]
# Short names that make it easier to find this resource.
rsrc.aliases = ['iris', 'iris-plots']
# Tags that make it easier to find this resource later.
rsrc.tags = ['iris', 'anderson', 'fisher']
# Arbitrary additional data to add into the resource.
# Note that larger data units should be made into files and
# added into the `datafiles` attributes.
rsrc.data = {
    'features': ['sepal length (cm)', 'sepal width (cm)',
                  'petal length (cm)', 'petal width (cm)'],
    # the data is already defined in the datafile,
    # but we could also put it here, inline
    'array': [[5.1, 3.5, 1.4, 0.2], [4.9, 3.0, 1.4, 0.2],
              [4.7, 3.2, 1.3, 0.2], [4.6, 3.1, 1.5, 0.2],
              [5.0, 3.6, 1.4, 0.2], [5.4, 3.9, 1.7, 0.4],
              # ... you get the idea..
             ]
}

API Reference

This section shows the documentation from the source code for the Resource and ResourceType classes, as well as Python classes that inherit from Resource in order to add custom functionality: PropertyDataResource and FlowsheetResource.

class idaes.dmf.resource.Resource(*args, **kwargs)[source]

A dynamically typed resource.

Resources have metadata and (same for all resoures) a type-specific “data” section (unique to that type of resource).

id_

Integer identifier for this Resource. You should not set this yourself. The value will be automatically overwritten with the database’s value when the resource is added to the DMF (with the .add() method).

uuid

Universal identifier for this resource

type

Type of this Resource. See ResourceTypes for standard values for this attribute.

name

Human-readable name for the resource (optional)

desc

Description of the resource

created

Date and time when the resource was created. This defaults to the time when the object was created. Value is a DateTime.

modified

Date and time the resource was last modified. This defaults to the time when the object was created. Value is a DateTime.

version

Version of the resource. Value is a SemanticVersion.

creator

Creator of the resource. Value is a Contact.

collaborators

List of other people involved. Each value is a Contact.

sources

Sources from which resource is derived, i.e. its provenance. Each value is a Source.

codes

List of code objects (including repositories and packages) associated with the resource. Each value is a Code.

datafiles

List of data files associated with the resource. Each value is a FilePath.

datafiles_dir

Datafiles subdirectory (single directory name)

aliases

List of aliases for the resource

tags

List of tags for the resource

help(name)[source]

Return descriptive ‘help’ for the given attribute.

Parameters:name (str) – Name of attribute
Returns:Help string, or error starting with “Error: “
Return type:str
static create_relation(subj, pred, obj)[source]

Create a relationship between two Resource instances.

Parameters:
Returns:

None

Raises:

TypeError – if subject & object are not Resource instances.

copy(**kwargs)[source]

Get a copy of this Resource.

As a convenience, optionally set some attributes in the copy.
Parameters:kwargs

Attributes to set in new instance after copying.

Returns:
Resource: A deep copy.

The copy will have an empty (zero) identifier and a new unique value for uuid. The relations are not copied.

table
For tabular data resources, this property builds and returns
a Table object.
Returns:
A representation of metadata and data
in this resource.
Return type:tabular.Table
Raises:TypeError – if this resource is not of the correct type.
property_table

For property data resources, this property builds and returns a PropertyTable object.

Returns:
A representation of metadata and data
in this resource.
Return type:propdata.PropertyTable
Raises:TypeError – if this resource is not of the correct type.
class idaes.dmf.resource.DateTime(default_value=traitlets.Undefined, allow_none=False, read_only=None, help=None, config=None, **kwargs)[source]

A trait type for a datetime.

Input can be a string, float, or tuple. Specifically:
  • string, ISO8601: YYYY[-MM-DD[Thh:mm:ss[.uuuuuu]]]
  • float: seconds since Unix epoch (1/1/1970)
  • tuple: format accepted by datetime.datetime()

No matter the input, validation will transform it into a floating point number, since this is the easiest form to store and search.

class idaes.dmf.resource.SemanticVersion(default_value=traitlets.Undefined, allow_none=False, read_only=None, help=None, config=None, **kwargs)[source]

Semantic version.

Three numeric identifiers, separated by a dot. Trailing non-numeric characters allowed.

Inputs, string or tuple, may have less than three numeric identifiers, but internally the value will be padded with zeros to always be of length four.

A leading dash or underscore in the trailing non-numeric characters is removed.

Some examples:

  • 1 => valid => (1, 0, 0, ‘’)
  • rc3 => invalid: no number
  • 1.1 => valid => (1, 1, 0, ‘’)
  • 1a => valid => (1, 0, 0, ‘a’)
  • 1.a.1 => invalid: non-numeric can only go at end
  • 1.12.1 => valid => (1, 12, 1, ‘’)
  • 1.12.13-1 => valid => (1, 12, 13, ‘1’)
  • 1.12.13.x => invalid: too many parts
class idaes.dmf.resource.Source(*args, **kwargs)[source]

A work from which the resource is derived.

doi

Digital object identifier

isbn

ISBN

source

The work, either print or electronic, from which the resource was derived

language

The primary language of the intellectual content of the resource

date

Date associated with resource

class idaes.dmf.resource.Contact(*args, **kwargs)[source]

Person who can be contacted.

name

Name of the contact

email

Email of the contact

class idaes.dmf.resource.Code(*args, **kwargs)[source]

Some source code, such as a Python module or C file.

This can also refer to packages or entire Git repositories.

type

Type of code resource, must be one of – ‘method’, ‘function’, ‘module’, ‘class’, ‘file’, ‘package’, ‘repository’, or ‘notebook’.

desc

Description of the code

name

Name of the code object, e.g. Python module name

language

Programming language, e.g. “Python” (the default).

release

Version of the release, default is ‘0.0.0’

idhash

Git or other unique hash

location

Flie path or URL location for the code

class idaes.dmf.resource.FilePath(tempfile=False, copy=True, **kwargs)[source]

Path to a file, plus optional description and metadata.

So that the DMF does not break when data files are moved or copied, the default is to copy the datafile into the DMF workspace. This behavior can be controlled by the copy and tempfile keywords to the constructor.

For example, if you have a big file you do NOT want to copy when you create the resource:

FilePath(path='/my/big.file', desc='100GB file', copy=False)

On the other hand, if you have a file that you want the DMF to manage entirely:

FilePath(path='/some/file.txt', desc='a file', tempfile=True)
path

Path to file

subdir

Unique subdir

desc

Description of the file’s contents

mimetype

MIME type

metadata

Metadata to associate with the file

__init__(tempfile=False, copy=True, **kwargs)[source]

Constructor.

Parameters:tempfile (bool) – if True, when copying the file, remove original.
class idaes.dmf.resource.RelationType(default_value=traitlets.Undefined, allow_none=False, read_only=None, help=None, config=None, **kwargs)[source]

Traitlets type for RDF-style triples relating resources to each other.

class idaes.dmf.resource.ResourceTypes[source]

Standard resource type names.

Use these as opaque constants to indicate standard resource types. For example, when creating a Resource:

rsrc = Resource(type=ResourceTypes.property_data, ...)
data = 'data'

Data (e.g. result data)

experiment = 'experiment'

Experiment

fs = 'flowsheet'

Flowsheet resource.

nb = 'notebook'

Jupyter Notebook

property_data = 'propertydb'

Property data resource, e.g. the contents are created via classes in the idaes.dmf.propdata module.

python = 'python'

Python code

surrmod = 'surrogate_model'

Surrogate model

tabular_data = 'tabular_data'

Tabular data

class idaes.dmf.resource.PropertyDataResource(property_table=None, **kwargs)[source]

Property data resource & factory.

class idaes.dmf.resource.FlowsheetResource(*args, **kwargs)[source]

Flowsheet resource & factory.

Resource schema

Below are HTML and raw JSON versions of the resource “schema”. This describes the structure of a resource. Having a schema is useful for other programs to be able to independently manipulate the resource representation.

Resource Schema (HTML)

schema
  • aliases
    List of:
    string
  • codes
    List of:
    • desc string
    • name string
    • language string
    • idhash string
    • location string
    • version
      List of:
      0 integer
      1 integer
      2 integer
      3 string
  • collaborators
    List of:
    • email string
    • name string
  • created number
  • creator
    • email string
    • name string
  • data
  • datafiles
    List of:
    • desc string
    • metadata
    • mimetype string
    • path string
    • subdir string
  • datafiles_dir string
  • desc string
  • id_ integer
  • modified number
  • relations
    List of:
    • predicate string
      Possible values:
      • WasGeneratedBy
      • Used
      • WasDerivedFrom
      • WasTriggeredBy
      • WasControlledBy
      • WasRevisionOf
    • identifier string
    • role string
      Possible values:
      • subject
      • object
  • sources
    List of:
    • date number
    • doi string
    • isbn string
    • language string
    • source string
  • tags
    List of:
    string
  • version
    • created number
    • name string
    • revision
      List of:
      0 integer
      1 integer
      2 integer
      3 string

Resouce Schema (JSON)

{
  "$schema": "http://json-schema.org/draft-04/schema#",
  "id": "http://idaes.org",
  "definitions": {
    "SemanticVersion": {
      "type": "array",
      "items": [
        {
          "type": "integer"
        },
        {
          "type": "integer"
        },
        {
          "type": "integer"
        },
        {
          "type": "string"
        }
      ],
      "minItems": 4
    }
  },
  "type": "object",
  "properties": {
    "aliases": {
      "type": "array",
      "items": {
        "type": "string"
      }
    },
    "codes": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "type": {
            "type": "string",
            "enum": [
              "method",
              "function",
              "module",
              "class",
              "file",
              "package",
              "repository",
              "notebook"
            ]
          },
          "desc": {
            "type": "string"
          },
          "name": {
            "type": "string"
          },
          "language": {
            "type": "string"
          },
          "idhash": {
            "type": "string"
          },
          "location": {
            "type": "string"
          },
          "version": {
            "$ref": "#/definitions/SemanticVersion"
          }
        }
      }
    },
    "collaborators": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "email": {
            "type": "string",
            "format": "email"
          },
          "name": {
            "type": "string"
          }
        },
        "required": [
          "name"
        ]
      }
    },
    "created": {
      "type": "number"
    },
    "creator": {
      "type": "object",
      "properties": {
        "email": {
          "type": "string",
          "format": "email"
        },
        "name": {
          "type": "string"
        }
      },
      "required": [
        "name"
      ]
    },
    "data": {
      "type": "object"
    },
    "datafiles": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "desc": {
            "type": "string"
          },
          "metadata": {
            "type": "object"
          },
          "mimetype": {
            "type": "string"
          },
          "path": {
            "type": "string"
          },
          "subdir": {
            "type": "string"
          }
        },
        "required": [
          "desc",
          "metadata",
          "mimetype",
          "path",
          "subdir"
        ]
      }
    },
    "datafiles_dir": {
      "type": "string"
    },
    "desc": {
      "type": "string"
    },
    "id_": {
      "type": "integer"
    },
    "modified": {
      "type": "number"
    },
    "relations": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "predicate": {
            "type": "string",
            "enum": [
              "WasGeneratedBy",
              "Used",
              "WasDerivedFrom",
              "WasTriggeredBy",
              "WasControlledBy",
              "WasRevisionOf"
            ]
          },
          "identifier": {
            "type": "string"
          },
          "role": {
            "type": "string",
            "enum": [
              "subject",
              "object"
            ]
          }
        },
        "required": [
          "predicate",
          "identifier",
          "role"
        ]
      }
    },
    "sources": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "date": {
            "type": "number"
          },
          "doi": {
            "type": "string"
          },
          "isbn": {
            "type": "string"
          },
          "language": {
            "type": "string"
          },
          "source": {
            "type": "string"
          }
        }
      }
    },
    "tags": {
      "type": "array",
      "items": {
        "type": "string"
      }
    },
    "type": {
      "type": "string"
    },
    "version": {
      "type": "object",
      "properties": {
        "created": {
          "type": "number"
        },
        "name": {
          "type": "string"
        },
        "revision": {
          "$ref": "#/definitions/SemanticVersion"
        }
      }
    }
  },
  "required": [
    "id_"
  ],
  "additionalProperties": false
}