DMF API Examples¶
The DMF API allows you to add, find, retrieve, and delete
Resources <resources> from
the DMF. See the dmf.DMF
class docs for the full API reference.
The rest of this section has a brief description, with examples, of the major operations of the API.
Initialize¶
To perform these functions, you should first create an instance of the DMF object, which is linked of course to its configuration and the underlying file system:
from idaes.dmf import dmf
my_dmf = dmf.DMF(path='/path/to/my-workspace')
The path given to the constructor is the DMF’s “workspace” directory, that holds the configuration files, resource metadata and data. This directory is laid out like this:
my-workspace: Root DMF "workspace" directory
|
+- config.yaml: Configuration file
+- resourcedb.json: Resource metadata "database" (if using TinyDB)
+- files: Data files for all resources
|
+- 195d6e5e-73da-4a0e-85f4-129bfcfda64b: Unique hash of directory
| with the datafiles (this is in the `datafiles_dir` attribute
| of the Resource object).
| |
| +- <fileA>: Datafile for that resource
| +- <fileB>: Another datafile, etc.
|
+- 36206516-88ce-400f-ac72-e42918e34aaf: Another directory, etc.
The “resourcedb.json” is a JSON file that is used by TinyDB. If you configured the DMF to use another database backend (not currently an option, but definitely in the roadmap), this file will not exist.
Find resources¶
You can find
resources by searching on any of the fields that
are part of the resource metadata. The basic syntax is a filter
consisting of key/value pairs. The key is the name of the
attribute in the Resource
class;
sub-attributes should be referred to with dotted notation.
For example, the query for looking for version 1.0.0 would
be:
spec = {'version.revision': '1.0.0'}
resources = my_dmf.find(spec)
For dates, you should pass a datetime.datetime or an instance the Pendulum class, from the pendulum package. The pendulum package is pretty easy to use, as shown here:
import pendulum
last_week = pendulum.now().subtract(weeks=1)
spec = {'version.created': last_week}
Combining expressions. You can combine multiple filter expressions by simply putting them together in the same dictionary. A given record must match all fields to match. There is currently no built-in way to select records matching less than all the fields:
halloween = pendulum.Pendulum(2017, 10, 31)
scary _query = {'version.revision': '13.13.13',
'version.created': halloween}
Inequalities. There are modifiers for inequalities in the style of MongoDB,
i.e. are nested dictionaries
with the key being the inequality prefixed with a “$” and the value
being the end of the range. An example should clarify the general idea,
see the documentation for find
for details:
from pendulum import Pendulum
# see if 'version.created' is in October 2017
# $ge : greater than or equal
# $le : less than or equal
spec = {'version.created': {'$ge': Pendulum('2017-10-1'),
'$le': Pendulum('2017-10-31')}
}
Lists. There is some special notation to help with lists of values. If the resource’s attribute value is a list, e.g. the tags attribute, then you need to pass a list of values to match against. By default, the method will return resources that match any of the provided values. If you want to only get resources that match all of the provided values, then add a ‘!’ after the attribute name. For example:
# match any resource with tags "MEA" or "model"
spec = {'tags': ['MEA', 'model']}
# match resources that have both tags "MEA" and "model"
spec = {'tags!': ['MEA', 'model']}
Retrieve resources¶
Resources can be retrieved directly, and reasonably quickly,
by their identifiers. The methods that do that start with the word “fetch”.
The fetch_one
method gets a single resource
and fetch_many
gets a number of them at once:
from idaes.dmf import DMF
from idaes.dmf.resource import Resource
my_dmf = DMF(path='/my/workspace')
def add_test_resource():
r = Resource(desc='test resource')
dmf.add(r) # Resource id is assigned at this point
return r.id_
# Demonstrate, in a somewhat contrived way, how "fetch_one" works
rid = add_test_resource()
r = my_dmf.fetch_one(rid) # returns the new resource
# Demonstrate, in a somewhat contrived way, how "fetch_many" works
rids = [add_test_resource() for i in range(10)]
rlist = my_dmf.fetch_many(rids) # returns the new resources
Add resources¶
The add
method puts a resource into the DMF.
This is mostly simply adding the metadata in the resource.
Datafiles. At this time, files in the datafiles attribute are copied and, if they were marked as temporary, the original is deleted. The default is to copy, and not delete the original. If a file is not copied, it is the user’s responsibility to update the record if the original file moves. Below are examples of the three possibilities:
from idaes_dmf import dmf
from idaes_dmf.resource import Resource, FilePath
my_dmf = dmf.DMF(path='/where/my/config/lives/')
r = Resource(desc='test resource')
# (1) Copy, and don't remove
r.datafiles.append(FilePath(path='/tmp/my-data.csv'))
# (2) Copy, and remove original
r.datafiles.append(FilePath(path='/tmp/temp-data.csv',
tempfile=True))
# (3) Neither copy nor remove
r.datafiles.append(FilePath(path='/home/bigfile', copy=False))
# Att this point, and not before, the copies occur
dmf.add(r)
# This is an error! It makes no sense to ask the file
# to be removed, but not copied (just a file delete?!)
r.datafiles.append(FilePath(path='foo', copy=False, tempfile=True))
# ^^ raises ValueError
Update resources¶
Resources are updated by providing a new Resource object to the
update
method. However, you cannot change the “type” of a Resource
in the process of updating it:
from idaes_dmf import dmf
from idaes_dmf.resource import Resource
my_dmf = dmf.DMF(path='/where/my/config/lives/')
r = Resource(desc='test resource')
my_dmf.add(r)
# Valid update
r.aliases.append('tester')
my_dmf.update(r)
# Invalid update
r.type = 'free to be me'
my_dmf.update(r) # !! Raises: errors.DMFError
Delete resources¶
Resources are deleted by their id, or a filter, using the remove method:
from idaes_dmf import dmf
from idaes_dmf.resource import Resource, FilePath
my_dmf = dmf.DMF(path='/where/my/config/lives/')
r = Resource(desc='test resource')
my_dmf.add(r)
# (a) delete by ID:
my_dmf.remove(identifier=r.id_)
# (b) alternatively, delete by filter:
my_dmf.remove(filter_dict={'desc':'test_resource'})