About IDAES and the DMF¶
The Data Management Framework (DMF) is part of the Institute for the Design of Advanced Energy Systems (IDAES, pronounced “eye-DAY-iss”) project. IDAES’ goal is to be the “world’s premier resource for the development and analysis of innovative advanced energy systems through the use of process systems engineering tools and approaches.” You can read more about IDAES on the project website.
The core functionality of IDAES is building and executing mathematical models of energy systems. The DMF complements this functionality by providing the ability to track the inputs (parameters, property models, etc.), processing (model code, Jupyter notebooks, scripts, etc.) and outputs (images, tables, etc.) of IDAES experiments. The DMF does not attempt to record all aspects of the environment, such as operating system and library versions. Instead, it focuses on tracking the environment-agnostic elements of the analysis that are specific to process engineering within the IDAES framework.
Below is a conceptual diagram of the DMF, showing the Workspaces residing on the local disk, and the contents of the workspaces being the tracked Resources, linked by Relations and containing metadata and (optionally) data files.
+--+ Local disk +-----------------------------------------+
| +-----------+ +-----------+ +-----------+ +-------+ |
| | Workspace | | Workspace | | Workspace | |Global | |
| | | | | .. | | |config.| |
| +-----------+ +-----------+ +-----------+ +-------+ |
+--------------+.+---------+.+----------------------------+
/ \
/ \
/ \
+-------------------------------------------+
| Workspace (detail) |
| |
| +-----------+ |
| | Resource <------+ |
| +-----------+ | Relation |
| | e.g. "derivedFrom" |
| +-----------+ | or "versionOf" |
| | Resource <---+ +
| +-----------+
| +-----------------------------+
| .+----+` | Resource (detail) |
| +-----------+ | * Type |
| | Resource | | * Name, ID, Date |
| +-----------+ | * Metadata |
| `+----+. | * Data files |
| | |
+---------------------+ | +------+ +-----+ +------+ |
| | File | |File |...| File | |
| +------+ +-----+ +------+ |
+-----------------------------+
The DMF provides a command line (terminal) interface and a Python library to create, retrieve, update, and delete resources and workspaces. The Python library is designed to be relatively easy to use interactively from within, e.g., Jupyter notebooks.