DMF Resources¶
Data items stored in the DMF, whether they be property data, property models,
flowsheets, or other user-defined objects, are called “resources”. They
are represented in the Python API by a class, Resource
, with
attributes for metadata and a generic “data” section where
resource-type-specific information can be held. You store Resources in the
DMF with the add method on an instance of the DMF class. See the
DMF API Examples documentation for all the available operations.
Resource classes documented on this page: Resource
,
PropertyDataResource
,
FlowsheetResource
.
A “special” type of resource is an “experiment”. The idea of experiments is that they anchor a set of resources that were the result of one coherent set of activities. Experiments are initialized with their associated DMF instance and have some convenience methods to make them a handy way to track a set of resources. See the experiment documentation page <experiment> for more details.
Here is a diagram of the resource structure:

Resources are populated by simply setting their attributes. The attributes are actually traits, using the Traitlets library, to make them “smarter” and easier to use. Some of the attributes, such as creator or sources, are lists of containers with more attributes. All the attributes are documented in the API Reference section.
Example¶
Below is an example of creating and then setting all (well, most) of the attributes for a resource that is a plot of the standard ‘iris’ dataset.
##############################################################################
# Institute for the Design of Advanced Energy Systems Process Systems
# Engineering Framework (IDAES PSE Framework) Copyright (c) 2018, by the
# software owners: The Regents of the University of California, through
# Lawrence Berkeley National Laboratory, National Technology & Engineering
# Solutions of Sandia, LLC, Carnegie Mellon University, West Virginia
# University Research Corporation, et al. All rights reserved.
#
# Please see the files COPYRIGHT.txt and LICENSE.txt for full copyright and
# license information, respectively. Both files are also available online
# at the URL "https://github.com/IDAES/idaes".
##############################################################################
from idaes.dmf import resource
# Create an empty resource
rsrc = resource.Resource(type='plot')
# Populate the resource. All the attributes are optional.
# This is set automatically to current time
rsrc.created = '2017-10-31'
# This is set automatically to current time
rsrc.modified = '2017-11-23'
# Description of the resource
rsrc.desc = 'Plots of the Iris dataset'
# Version of the resource
rsrc.version = resource.Version(revision='1.0.3-a7',
name='november-release')
# Name and contact email of creator of the resource
rsrc.creator = resource.Contact(name='Dan Gunter',
email='dang@science-lab.gov')
# Name and contact email of other people involved
rsrc.collaborators = [resource.Contact(name='John Eslick',
email='john@science-lab.gov'),
resource.Contact(name='David Miller',
email='david@science-lab.gov')]
# Provenance: sources, such as publications -- but
# really anything is allowed -- for the resource.
rsrc.sources = [resource.Source(
source='R. A. Fisher. "The use of multiple measurements '
'in taxonomic problems". '
'Annals of Eugenics. 7 (2): 179-188.',
doi='10.1111/j.1469-1809.1936.tb02137.x', date='1936'),
resource.Source(source='Edgar Anderson. '
'"The species problem in Iris". '
'Annals of the Missouri Botanical Garden. '
'23 (3): 457-509. '
'JSTOR 2394164',
date='1936')]
# Associated code files, including Jupyter notebooks.
# If these codes are worth making into their own Resources,
# you should not put them here, but instead link to them
# through the `relations` attribute.
rsrc.codes = [resource.Code(type='notebook',
desc='Python plotting code',
name='plot_iris_dataset.ipynb',
language='Python',
location='http://scikit-learn.org/stable/_downloads/'
'plot_iris_dataset.ipynb')]
# Associated data files. Same comment as for `codes` re: linking.
rsrc.datafiles = [resource.FilePath(path='iris-data.csv',
desc='Iris data')]
# Short names that make it easier to find this resource.
rsrc.aliases = ['iris', 'iris-plots']
# Tags that make it easier to find this resource later.
rsrc.tags = ['iris', 'anderson', 'fisher']
# Arbitrary additional data to add into the resource.
# Note that larger data units should be made into files and
# added into the `datafiles` attributes.
rsrc.data = {
'features': ['sepal length (cm)', 'sepal width (cm)',
'petal length (cm)', 'petal width (cm)'],
# the data is already defined in the datafile,
# but we could also put it here, inline
'array': [[5.1, 3.5, 1.4, 0.2], [4.9, 3.0, 1.4, 0.2],
[4.7, 3.2, 1.3, 0.2], [4.6, 3.1, 1.5, 0.2],
[5.0, 3.6, 1.4, 0.2], [5.4, 3.9, 1.7, 0.4],
# ... you get the idea..
]
}
API Reference¶
This section shows the documentation from the source code for the
Resource and ResourceType classes, as well as Python classes that inherit
from Resource in order to add custom functionality:
PropertyDataResource
and FlowsheetResource
.
-
class
idaes.dmf.resource.
Resource
(*args, **kwargs)[source] A dynamically typed resource.
Resources have metadata and (same for all resoures) a type-specific “data” section (unique to that type of resource).
-
id_
Integer identifier for this Resource. You should not set this yourself. The value will be automatically overwritten with the database’s value when the resource is added to the DMF (with the .add() method).
-
uuid
Universal identifier for this resource
-
type
Type of this Resource. See
ResourceTypes
for standard values for this attribute.
-
name
Human-readable name for the resource (optional)
-
desc
Description of the resource
-
created
Date and time when the resource was created. This defaults to the time when the object was created. Value is a
DateTime
.
-
modified
Date and time the resource was last modified. This defaults to the time when the object was created. Value is a
DateTime
.
-
version
Version of the resource. Value is a
SemanticVersion
.
-
creator
Creator of the resource. Value is a
Contact
.
-
collaborators
List of other people involved. Each value is a
Contact
.
-
sources
Sources from which resource is derived, i.e. its provenance. Each value is a
Source
.
-
codes
List of code objects (including repositories and packages) associated with the resource. Each value is a
Code
.
-
datafiles
List of data files associated with the resource. Each value is a
FilePath
.
-
datafiles_dir
Datafiles subdirectory (single directory name)
-
aliases
List of aliases for the resource
-
tags
List of tags for the resource
-
help
(name)[source] Return descriptive ‘help’ for the given attribute.
Parameters: name (str) – Name of attribute Returns: Help string, or error starting with “Error: “ Return type: str
-
static
create_relation
(subj, pred, obj)[source] Create a relationship between two Resource instances.
Parameters: Returns: None
Raises: TypeError
– if subject & object are not Resource instances.
-
copy
(**kwargs)[source] Get a copy of this Resource.
As a convenience, optionally set some attributes in the copy.Parameters: kwargs – Attributes to set in new instance after copying.
- Returns:
- Resource: A deep copy.
The copy will have an empty (zero) identifier and a new unique value for uuid. The relations are not copied.
-
table
- For tabular data resources, this property builds and returns
- a Table object.
Returns: - A representation of metadata and data
- in this resource.
Return type: tabular.Table Raises: TypeError
– if this resource is not of the correct type.
-
property_table
For property data resources, this property builds and returns a PropertyTable object.
Returns: - A representation of metadata and data
- in this resource.
Return type: propdata.PropertyTable Raises: TypeError
– if this resource is not of the correct type.
-
-
class
idaes.dmf.resource.
DateTime
(default_value=traitlets.Undefined, allow_none=False, read_only=None, help=None, config=None, **kwargs)[source] A trait type for a datetime.
- Input can be a string, float, or tuple. Specifically:
- string, ISO8601: YYYY[-MM-DD[Thh:mm:ss[.uuuuuu]]]
- float: seconds since Unix epoch (1/1/1970)
- tuple: format accepted by datetime.datetime()
No matter the input, validation will transform it into a floating point number, since this is the easiest form to store and search.
-
class
idaes.dmf.resource.
SemanticVersion
(default_value=traitlets.Undefined, allow_none=False, read_only=None, help=None, config=None, **kwargs)[source] Semantic version.
Three numeric identifiers, separated by a dot. Trailing non-numeric characters allowed.
Inputs, string or tuple, may have less than three numeric identifiers, but internally the value will be padded with zeros to always be of length four.
A leading dash or underscore in the trailing non-numeric characters is removed.
Some examples:
- 1 => valid => (1, 0, 0, ‘’)
- rc3 => invalid: no number
- 1.1 => valid => (1, 1, 0, ‘’)
- 1a => valid => (1, 0, 0, ‘a’)
- 1.a.1 => invalid: non-numeric can only go at end
- 1.12.1 => valid => (1, 12, 1, ‘’)
- 1.12.13-1 => valid => (1, 12, 13, ‘1’)
- 1.12.13.x => invalid: too many parts
-
class
idaes.dmf.resource.
Source
(*args, **kwargs)[source] A work from which the resource is derived.
-
doi
Digital object identifier
-
isbn
ISBN
-
source
The work, either print or electronic, from which the resource was derived
-
language
The primary language of the intellectual content of the resource
-
date
Date associated with resource
-
-
class
idaes.dmf.resource.
Contact
(*args, **kwargs)[source] Person who can be contacted.
-
name
Name of the contact
-
email
Email of the contact
-
-
class
idaes.dmf.resource.
Code
(*args, **kwargs)[source] Some source code, such as a Python module or C file.
This can also refer to packages or entire Git repositories.
-
type
Type of code resource, must be one of – ‘method’, ‘function’, ‘module’, ‘class’, ‘file’, ‘package’, ‘repository’, or ‘notebook’.
-
desc
Description of the code
-
name
Name of the code object, e.g. Python module name
-
language
Programming language, e.g. “Python” (the default).
-
release
Version of the release, default is ‘0.0.0’
-
idhash
Git or other unique hash
-
location
Flie path or URL location for the code
-
-
class
idaes.dmf.resource.
FilePath
(tempfile=False, copy=True, **kwargs)[source] Path to a file, plus optional description and metadata.
So that the DMF does not break when data files are moved or copied, the default is to copy the datafile into the DMF workspace. This behavior can be controlled by the copy and tempfile keywords to the constructor.
For example, if you have a big file you do NOT want to copy when you create the resource:
FilePath(path='/my/big.file', desc='100GB file', copy=False)
On the other hand, if you have a file that you want the DMF to manage entirely:
FilePath(path='/some/file.txt', desc='a file', tempfile=True)
-
path
Path to file
-
subdir
Unique subdir
-
desc
Description of the file’s contents
-
mimetype
MIME type
-
metadata
Metadata to associate with the file
-
-
class
idaes.dmf.resource.
RelationType
(default_value=traitlets.Undefined, allow_none=False, read_only=None, help=None, config=None, **kwargs)[source] Traitlets type for RDF-style triples relating resources to each other.
-
class
idaes.dmf.resource.
ResourceTypes
[source] Standard resource type names.
Use these as opaque constants to indicate standard resource types. For example, when creating a Resource:
rsrc = Resource(type=ResourceTypes.property_data, ...)
-
data
= 'data' Data (e.g. result data)
-
experiment
= 'experiment' Experiment
-
fs
= 'flowsheet' Flowsheet resource.
-
nb
= 'notebook' Jupyter Notebook
-
property_data
= 'propertydb' Property data resource, e.g. the contents are created via classes in the
idaes.dmf.propdata
module.
-
python
= 'python' Python code
-
surrmod
= 'surrogate_model' Surrogate model
-
tabular_data
= 'tabular_data' Tabular data
-
-
class
idaes.dmf.resource.
PropertyDataResource
(property_table=None, **kwargs)[source] Property data resource & factory.
-
class
idaes.dmf.resource.
FlowsheetResource
(*args, **kwargs)[source] Flowsheet resource & factory.
Resource schema¶
Below are HTML and raw JSON versions of the resource “schema”. This describes the structure of a resource. Having a schema is useful for other programs to be able to independently manipulate the resource representation.
Resource Schema (HTML)¶
-
aliasesList of:string
-
codesList of:
-
desc string
-
name string
-
language string
-
idhash string
-
location string
-
versionList of:0 integer1 integer2 integer3 string
-
-
collaboratorsList of:
-
email string
-
name string
-
-
created number
-
creator
-
email string
-
name string
-
-
data
-
datafilesList of:
-
desc string
-
metadata
-
mimetype string
-
path string
-
subdir string
-
-
datafiles_dir string
-
desc string
-
id_ integer
-
modified number
-
relationsList of:
-
predicate stringPossible values:
- WasGeneratedBy
- Used
- WasDerivedFrom
- WasTriggeredBy
- WasControlledBy
- WasRevisionOf
-
identifier string
-
role stringPossible values:
- subject
- object
-
-
sourcesList of:
-
date number
-
doi string
-
isbn string
-
language string
-
source string
-
-
tagsList of:string
-
version
-
created number
-
name string
-
revisionList of:0 integer1 integer2 integer3 string
-
Resouce Schema (JSON)¶
{
"$schema": "http://json-schema.org/draft-04/schema#",
"id": "http://idaes.org",
"definitions": {
"SemanticVersion": {
"type": "array",
"items": [
{
"type": "integer"
},
{
"type": "integer"
},
{
"type": "integer"
},
{
"type": "string"
}
],
"minItems": 4
}
},
"type": "object",
"properties": {
"aliases": {
"type": "array",
"items": {
"type": "string"
}
},
"codes": {
"type": "array",
"items": {
"type": "object",
"properties": {
"type": {
"type": "string",
"enum": [
"method",
"function",
"module",
"class",
"file",
"package",
"repository",
"notebook"
]
},
"desc": {
"type": "string"
},
"name": {
"type": "string"
},
"language": {
"type": "string"
},
"idhash": {
"type": "string"
},
"location": {
"type": "string"
},
"version": {
"$ref": "#/definitions/SemanticVersion"
}
}
}
},
"collaborators": {
"type": "array",
"items": {
"type": "object",
"properties": {
"email": {
"type": "string",
"format": "email"
},
"name": {
"type": "string"
}
},
"required": [
"name"
]
}
},
"created": {
"type": "number"
},
"creator": {
"type": "object",
"properties": {
"email": {
"type": "string",
"format": "email"
},
"name": {
"type": "string"
}
},
"required": [
"name"
]
},
"data": {
"type": "object"
},
"datafiles": {
"type": "array",
"items": {
"type": "object",
"properties": {
"desc": {
"type": "string"
},
"metadata": {
"type": "object"
},
"mimetype": {
"type": "string"
},
"path": {
"type": "string"
},
"subdir": {
"type": "string"
}
},
"required": [
"desc",
"metadata",
"mimetype",
"path",
"subdir"
]
}
},
"datafiles_dir": {
"type": "string"
},
"desc": {
"type": "string"
},
"id_": {
"type": "integer"
},
"modified": {
"type": "number"
},
"relations": {
"type": "array",
"items": {
"type": "object",
"properties": {
"predicate": {
"type": "string",
"enum": [
"WasGeneratedBy",
"Used",
"WasDerivedFrom",
"WasTriggeredBy",
"WasControlledBy",
"WasRevisionOf"
]
},
"identifier": {
"type": "string"
},
"role": {
"type": "string",
"enum": [
"subject",
"object"
]
}
},
"required": [
"predicate",
"identifier",
"role"
]
}
},
"sources": {
"type": "array",
"items": {
"type": "object",
"properties": {
"date": {
"type": "number"
},
"doi": {
"type": "string"
},
"isbn": {
"type": "string"
},
"language": {
"type": "string"
},
"source": {
"type": "string"
}
}
}
},
"tags": {
"type": "array",
"items": {
"type": "string"
}
},
"type": {
"type": "string"
},
"version": {
"type": "object",
"properties": {
"created": {
"type": "number"
},
"name": {
"type": "string"
},
"revision": {
"$ref": "#/definitions/SemanticVersion"
}
}
}
},
"required": [
"id_"
],
"additionalProperties": false
}