DMF Properties API example

July 30, 2017 - Dan Gunter dkgunter@lbl.gov

This example shows usage of the DMF “Properties” API in the DMF.

# Standard library imports
import json
import os
import tempfile
# Third-party imports
import pandas as pd
import numpy as np

Initialize the DMF

The DMF is initialized with a root directory, called the “workspace”. For this example, we’ll use a temporary directory that we create.

from idaes_dmf import dmf
from idaes_dmf import propdata
tmpdir = tempfile.mkdtemp()
mydmf = dmf.DMF(tmpdir, create=True)

Create property data from Python dictionary

Create some inline property data and write it to a temporary file. This file could have been created in other ways, e.g. by using Python or CLI functions to convert the data from comma-separated values.

amundsen_properties = {
                "meta": {
                    "datatype": "MEA",
                    "info": "J. Chem. Eng. Data, 2009, Vol 54, pg. 3096-30100",
                    "notes": "r is MEA weight fraction in aqueous soln.",
                    "authors": "Amundsen, T.G., Lars, E.O., Eimer, D.A.",
                    "title": "Density and Viscosity of Monoethanolamine + .etc.",
                    "date": "2009"
                },
                "data": {
                    "properties" : [
                        {
                            "name": "Viscosity Value",
                            "units": "mPa-s",
                            "values": [2.6, 6.2],
                            "error_type": "absolute",
                            "errors": [0.06, 0.004]
                        }
                    ],
                    "states": [
                        {
                            "name": "r",
                            "units": "",
                            "values": [0.2, 1000]
                        }
                    ]
                }
            }
# Make temporary directory in which to write the file
scratchdir = os.path.join(mydmf.root, 'scratch')
if not os.path.exists(scratchdir):
    os.mkdir(scratchdir)
# Dump the Python dictionary as JSON to the file
tmpf = open(os.path.join(scratchdir, 'resource.json'), 'w')
tmpf.write(json.dumps(amundsen_properties))
tmpf.close()

Load property data into DMF

The newly created property data is just a file, that could be anywhere on the file system. So the DMF can locate it and manipulate it, we need to explicitly “add” it.

# Add the resource
factory = dmf.ResourceFactory()
resource = factory.create_propertydb(tmpf.name)
rid = mydmf.add(resource, copy_files=(tmpf.name,))

Get the property data as an object

Now that the property data is in the DMF, we can retrieve it as an object. Since we just loaded it, we can take a shortcut and “get” the data using the resource ID returned by the “add_resource” method.

r2 = mydmf.fetch_one(rid)
table = dmf.get_propertydb_table(r2)
table.data.values_dataframe()
r Viscosity Value
0 0.2 2.6
1 1000.0 6.2

Extract values and errors as Pandas Dataframes

After extracting each dataframe, merge them together.

vf = table.data.values_dataframe()
ef = table.data.errors_dataframe()
vf.join(ef, rsuffix='.err').head()
r Viscosity Value Viscosity Value.err
0 0.2 2.6 0.060
1 1000.0 6.2 0.004

The ‘states=False’ parameter means we only see measured properties, not the associated states – which in this case is temperature.

vf = table.data.values_dataframe(states=False)
ef = table.data.errors_dataframe(states=False)
vf.join(ef, rsuffix='.err').head()
Viscosity Value Viscosity Value.err
0 2.6 0.060
1 6.2 0.004

Create property data from CSV

The DMF properties classes also understand how to convert data from specifically formatted CSV files into the JSON format shown above. Once that is done, the process for importing that data into the DMF is the same. In this example, we’ll start with some example CSV data, which we write to a file and then convert before importing to the DMF.

Han properties as CSV

The CSV format has two files, one for “metadata” and one for “data”.

han_meta = '''Source,Han, J., Jin, J., Eimer, D.A., Melaaen, M.C.,"Density of Water(1) + Monoethanolamine(2) + CO2(3) from (298.15 to 413.15) K and Surface Tension of Water(1) + Monethanolamine(2) from (303.15 to 333.15)K", J. Chem. Eng. Data, 2012, Vol. 57, pg. 1095-1103
Retrieval,J. Morgan, date unknown
Notes,r is MEA weight fraction in aqueous soln. (CO2-free basis)
'''
han_data = '''Data No.,T (K),Absolute Error,CO2 Loading (mol CO2/MEA),Absolute Error,r,Absolute Error,Density Data (g/cm3),Absolute Error
1,298.15,0,0.1,0,0.3,0,1.0333,5E-05
2,298.15,0,0.21,0,0.3,0,1.0534,5E-05
3,298.15,0,0.32,0,0.3,0,1.0756,5E-05
4,298.15,0,0.44,0,0.3,0,1.0964,5E-05
5,298.15,0,0.56,0,0.3,0,1.1142,5E-05
6,313.15,0,0.1,0,0.3,0,1.0253,5E-05
7,313.15,0,0.21,0,0.3,0,1.0464,5E-05
8,313.15,0,0.32,0,0.3,0,1.0669,5E-05
9,313.15,0,0.44,0,0.3,0,1.0891,5E-05
10,313.15,0,0.56,0,0.3,0,1.1068,5E-05
11,323.15,0,0.1,0,0.3,0,1.0196,5E-05
12,323.15,0,0.21,0,0.3,0,1.0412,5E-05
13,323.15,0,0.32,0,0.3,0,1.0613,5E-05
14,323.15,0,0.44,0,0.3,0,1.0838,5E-05
15,323.15,0,0.56,0,0.3,0,1.1014,5E-05
16,333.15,0,0.1,0,0.3,0,1.0138,5E-05
17,333.15,0,0.21,0,0.3,0,1.0356,5E-05
18,333.15,0,0.32,0,0.3,0,1.0556,5E-05
19,333.15,0,0.44,0,0.3,0,1.0782,5E-05
20,333.15,0,0.56,0,0.3,0,1.0957,5E-05
21,343.15,0,0.1,0,0.3,0,1.0076,5E-05
22,343.15,0,0.21,0,0.3,0,1.0297,5E-05
23,343.15,0,0.32,0,0.3,0,1.0496,5E-05
24,343.15,0,0.44,0,0.3,0,1.0723,5E-05
25,343.15,0,0.56,0,0.3,0,1.0887,5E-05
26,353.15,0,0.1,0,0.3,0,1.0002,5E-05
27,353.15,0,0.21,0,0.3,0,1.0234,5E-05
28,353.15,0,0.32,0,0.3,0,1.0434,5E-05
29,353.15,0,0.44,0,0.3,0,1.066,5E-05
30,353.15,0,0.56,0,0.3,0,1.0812,5E-05
31,298.15,0,0.1,0,0.4,0,1.0376,5E-05
32,298.15,0,0.21,0,0.4,0,1.0627,5E-05
33,298.15,0,0.33,0,0.4,0,1.0945,5E-05
34,298.15,0,0.45,0,0.4,0,1.1296,5E-05
35,313.15,0,0.1,0,0.4,0,1.0295,5E-05
36,313.15,0,0.21,0,0.4,0,1.0547,5E-05
37,313.15,0,0.33,0,0.4,0,1.0867,5E-05
38,313.15,0,0.45,0,0.4,0,1.1199,5E-05
39,323.15,0,0.1,0,0.4,0,1.0237,5E-05
40,323.15,0,0.21,0,0.4,0,1.049,5E-05
41,323.15,0,0.33,0,0.4,0,1.0811,5E-05
42,323.15,0,0.45,0,0.4,0,1.1138,5E-05
43,333.15,0,0.1,0,0.4,0,1.0178,5E-05
44,333.15,0,0.21,0,0.4,0,1.043,5E-05
45,333.15,0,0.33,0,0.4,0,1.0752,5E-05
46,333.15,0,0.45,0,0.4,0,1.1087,5E-05
47,343.15,0,0.1,0,0.4,0,1.011,5E-05
48,343.15,0,0.21,0,0.4,0,1.0367,5E-05
49,343.15,0,0.33,0,0.4,0,1.0686,5E-05
50,343.15,0,0.45,0,0.4,0,1.1032,5E-05
51,353.15,0,0.1,0,0.4,0,1.0048,5E-05
52,353.15,0,0.21,0,0.4,0,1.0292,5E-05
53,353.15,0,0.33,0,0.4,0,1.0626,5E-05
54,353.15,0,0.45,0,0.4,0,1.0963,5E-05
'''
# use 'scratchdir' from above to write the files,
# saving the names of these files in some local variables
prop_meta = os.path.join(scratchdir, 'prop-meta.csv')
open(prop_meta, 'w').write(han_meta)
prop_data = os.path.join(scratchdir, 'prop-data.csv')
open(prop_data, 'w').write(han_data)
2151

Convert from CSV to JSON

A single convenience function can convert the properties data from CSV to JSON. In addition to the files, it needs two parameters:

  • The “type” of the data, which is a free-form string such as “MEA” to identify what kinds of property data this is. Right now, the vocabulary, etc. of this string is outside the scope of the DMF code.
  • The number of initial columns of data (value,error = one “column”) that are state variables (as opposed to measured variables). This information is necessary for good display of the data, as well as merging data from multiple sources. In the case of the dataset here, the first 3 values are “states” and only the last one (Density) is measured.
from idaes_dmf import propdata
output_file = os.path.join(scratchdir, 'prop.json')
propdata.convert_csv(prop_meta, 'MEA', prop_data, 3, output_file)
jdata = json.load(open(output_file))
println(json.dumps(jdata, indent=2))
{
  "data": {
    "properties": [
      {
        "name": "Density Data",
        "units": "g/cm3",
        "values": [
          1.0333,
          1.0534,
          1.0756,
          1.0964,
          1.1142,
          1.0253,
          1.0464,
          1.0669,
          1.0891,
          1.1068,
          1.0196,
          1.0412,
          1.0613,
          1.0838,
          1.1014,
          1.0138,
          1.0356,
          1.0556,
          1.0782,
          1.0957,
          1.0076,
          1.0297,
          1.0496,
          1.0723,
          1.0887,
          1.0002,
          1.0234,
          1.0434,
          1.066,
          1.0812,
          1.0376,
          1.0627,
          1.0945,
          1.1296,
          1.0295,
          1.0547,
          1.0867,
          1.1199,
          1.0237,
          1.049,
          1.0811,
          1.1138,
          1.0178,
          1.043,
          1.0752,
          1.1087,
          1.011,
          1.0367,
          1.0686,
          1.1032,
          1.0048,
          1.0292,
          1.0626,
          1.0963
        ],
        "errors": [
          5e-05,
          5e-05,
          5e-05,
          5e-05,
          5e-05,
          5e-05,
          5e-05,
          5e-05,
          5e-05,
          5e-05,
          5e-05,
          5e-05,
          5e-05,
          5e-05,
          5e-05,
          5e-05,
          5e-05,
          5e-05,
          5e-05,
          5e-05,
          5e-05,
          5e-05,
          5e-05,
          5e-05,
          5e-05,
          5e-05,
          5e-05,
          5e-05,
          5e-05,
          5e-05,
          5e-05,
          5e-05,
          5e-05,
          5e-05,
          5e-05,
          5e-05,
          5e-05,
          5e-05,
          5e-05,
          5e-05,
          5e-05,
          5e-05,
          5e-05,
          5e-05,
          5e-05,
          5e-05,
          5e-05,
          5e-05,
          5e-05,
          5e-05,
          5e-05,
          5e-05,
          5e-05,
          5e-05
        ],
        "error_type": "absolute"
      }
    ],
    "states": [
      {
        "name": "T",
        "units": "K",
        "values": [
          298.15,
          298.15,
          298.15,
          298.15,
          298.15,
          313.15,
          313.15,
          313.15,
          313.15,
          313.15,
          323.15,
          323.15,
          323.15,
          323.15,
          323.15,
          333.15,
          333.15,
          333.15,
          333.15,
          333.15,
          343.15,
          343.15,
          343.15,
          343.15,
          343.15,
          353.15,
          353.15,
          353.15,
          353.15,
          353.15,
          298.15,
          298.15,
          298.15,
          298.15,
          313.15,
          313.15,
          313.15,
          313.15,
          323.15,
          323.15,
          323.15,
          323.15,
          333.15,
          333.15,
          333.15,
          333.15,
          343.15,
          343.15,
          343.15,
          343.15,
          353.15,
          353.15,
          353.15,
          353.15
        ]
      },
      {
        "name": "CO2 Loading",
        "units": "mol CO2/MEA",
        "values": [
          0.1,
          0.21,
          0.32,
          0.44,
          0.56,
          0.1,
          0.21,
          0.32,
          0.44,
          0.56,
          0.1,
          0.21,
          0.32,
          0.44,
          0.56,
          0.1,
          0.21,
          0.32,
          0.44,
          0.56,
          0.1,
          0.21,
          0.32,
          0.44,
          0.56,
          0.1,
          0.21,
          0.32,
          0.44,
          0.56,
          0.1,
          0.21,
          0.33,
          0.45,
          0.1,
          0.21,
          0.33,
          0.45,
          0.1,
          0.21,
          0.33,
          0.45,
          0.1,
          0.21,
          0.33,
          0.45,
          0.1,
          0.21,
          0.33,
          0.45,
          0.1,
          0.21,
          0.33,
          0.45
        ]
      },
      {
        "name": "r",
        "units": "",
        "values": [
          0.3,
          0.3,
          0.3,
          0.3,
          0.3,
          0.3,
          0.3,
          0.3,
          0.3,
          0.3,
          0.3,
          0.3,
          0.3,
          0.3,
          0.3,
          0.3,
          0.3,
          0.3,
          0.3,
          0.3,
          0.3,
          0.3,
          0.3,
          0.3,
          0.3,
          0.3,
          0.3,
          0.3,
          0.3,
          0.3,
          0.4,
          0.4,
          0.4,
          0.4,
          0.4,
          0.4,
          0.4,
          0.4,
          0.4,
          0.4,
          0.4,
          0.4,
          0.4,
          0.4,
          0.4,
          0.4,
          0.4,
          0.4,
          0.4,
          0.4,
          0.4,
          0.4,
          0.4,
          0.4
        ]
      }
    ]
  },
  "meta": {
    "datatype": "MEA",
    "authors": "Han, J., Jin, J., Eimer, D.A., Melaaen, M.C.",
    "date": 2012,
    "title": "Density of Water(1) + Monoethanolamine(2) + CO2(3) from (298.15 to 413.15) K and Surface Tension of Water(1) + Monethanolamine(2) from (303.15 to 333.15)K",
    "info": "J. Chem. Eng. Data, 2012, Vol. 57, pg. 1095-1103",
    "retrieval": "J. Morgan, date unknown",
    "notes": "r is MEA weight fraction in aqueous soln. (CO2-free basis)"
  }
}

Load PropertyTable

Once there is a standard JSON version of the data, you can simply load it back into a PropertyTable

table2 = propdata.PropertyTable.load(output_file)

Show the data

As before, we extract the data to a Pandas dataframe so it is easy to view in the notebook.

vf = table2.data.values_dataframe()
ef = table2.data.errors_dataframe()
vf.join(ef, rsuffix='.err').head(10)
T CO2 Loading r Density Data Density Data.err
0 298.15 0.10 0.3 1.0333 0.00005
1 298.15 0.21 0.3 1.0534 0.00005
2 298.15 0.32 0.3 1.0756 0.00005
3 298.15 0.44 0.3 1.0964 0.00005
4 298.15 0.56 0.3 1.1142 0.00005
5 313.15 0.10 0.3 1.0253 0.00005
6 313.15 0.21 0.3 1.0464 0.00005
7 313.15 0.32 0.3 1.0669 0.00005
8 313.15 0.44 0.3 1.0891 0.00005
9 313.15 0.56 0.3 1.1068 0.00005