This is the main package, containing the API to create and run templates; and perform and query monitoring and provenance. For background information on what Tigres is meant to do, see the Concepts in Tigres. The top-level tigres module has tools for initializing and finalizing tigres programs. This can be done explicitly using tigres.start() and tigres.end() or implicitly when the first Tigres data object is initialized.
platform: | Unix, Mac |
---|---|
synopsis: | The high level Tigres API. Initialization of Tigres templates, tasks and their inputs. |
module author: | Dan Gunter <dkgunter@lbl.gov>, Gilberto Pastorello <gzpastorello@lbl.gov>, Val Hendrix <vhendrix@lbl.gov> |
Example: |
>>> import tempfile as temp_file
>>> from tigres import *
>>> f = temp_file.NamedTemporaryFile()
>>> program=start(log_dest=f.name)
>>> set_log_level(Level.INFO)
>>> log_info("Tasks", message="Starting to prepare tasks and inputs")
>>> def adder(a, b): return a + b
>>> task1 = Task("Task 1", FUNCTION, adder)
>>> task2 = Task("Task 2", EXECUTABLE, "/bin/echo")
>>> tasks = TaskArray(None, [task1, task2])
>>> inputs = InputArray(None, [InputValues(None, [2, 3]), InputValues(None, ['world'])])
>>> log_info("Template", message="Before template run")
>>> sequence("My Sequence", tasks, inputs)
'world'
>>> # find() returns a list of records
>>> for o in find(activity="Tasks"):
... assert o.message
>>> end()
- InputArray - Array of one or more InputValues, which will be inputs to a TaskArray in a Template.
- InputTypes - List of types for inputs of a Task.
- InputValues - List of values for inputs matched to a task Task.
- Task - Function or program. A task is the atomic unit of execution in Tigres.
- TaskArray - List of one or more Tasks, which will be executed in a Template
- EXECUTABLE - identifies that a task implementation is an executable
- FUNCTION - identifies that a task implementation is a function
Users should use start() to initialize the Tigres library, and this will initialize the logging as well. When start() is called, the log_dest keyword in that function is passed as the first argument logger and any other keywords beginning in log_ will have the prefix stripped and will be passed as keywords.
One of the design goals of the state management is to cleanly combine user-defined logs, e.g. about what is happening in the application, and the automatic logs from the execution system. The user-defined logging uses the familiar set of functions. Information is logged as a Python dict of key/value pairs. It is certainly possible to log English sentences, but breaking up the information into a more structured form will simplify querying for it later.
PREVIOUS - a syntax where the user can specify that the output of a previously executed task as input for future task.
- handles dependencies between templates. PREVIOUS can be both implicit and explicit.
- The semantics of PREVIOUS are affected by the template type.
List of types for inputs of a Task.
Example: |
---|
>>> from tigres import *
>>> task1_types = InputTypes('Task1Types', [int, str])
Array of one or more InputValues, which will be inputs to a TaskArray in a Template.
Example: |
---|
>>> from tigres import InputArray,InputValues
>>> input_array = InputArray("input_array1", [InputValues("values",[1,'hello world'])])
Values for inputs of a Task.
Example: |
---|
>>> from tigres import InputValues
>>> values = InputValues("my_values", [1, 'hello world'])
PREVIOUS is a syntax where the user can specify that the output of a previously executed task as input for a task. PREVIOUS creates dependencies between templates and can be both implicit and explict.
- PREVIOUS: Use the entire output of the previous task as the input.
- PREVIOUS.i: sed to split outputs across parallel tasks from the previous task.It matches the i-th output of | the previous task/template to the i-th InputValues of the task to be run. This only works for parallel tasks.
- PREVIOUS.i[n]: Use the n-th output of the previous task or template as input.
- PREVIOUS.taskOrTemplateName: Use the entire output of the previous template/task with specified name.
- PREVIOUS.taskOrTemplateName.i: Used to split outputs from the specified task or template across parallel tasks. | Match the i-th output of the previous task/template to the i-th InputValues of the task to be run. This only works for parallel tasks.
- PREVIOUS.taskOrTemplateName.i[n] Use the n-th output of the previous task or template as input.
List of one or more Tasks, which will be executed in a Template
Instances act just like a list, except that they have a name.
Function or program. A task is the atomic unit of execution in Tigres.
Example: |
---|
>>> def abide(dude): return dude - 1
>>> from tigres import Task, FUNCTION, EXECUTABLE
>>> fn_task = Task("abiding", FUNCTION, abide)
>>> exe_task = Task("more_abiding", EXECUTABLE, "abide.sh")
List of tasks placed in series; If no inputs are given for any task, the output of the previous task or template is the input of the next.
Valid configurations of task_array and input_array are:
Parameters: |
|
---|---|
Returns: | results from the last task executed in the sequence. If the results of the last sequence is from a python function the output type will be defined by the type returned otherwise executable output will be a string. |
Return type: | object or str |
Example: |
>>> from tigres import *
>>> def adder(a, b): return a + b
>>> task1 = Task("Task 1", FUNCTION, adder)
>>> task2 = Task("Task 2", EXECUTABLE, "echo")
>>> tasks = TaskArray(None, [task1, task2])
>>> inputs = InputArray(None, [InputValues(None, [2, 3]), InputValues(None, ['world'])])
>>> sequence("My Sequence", tasks, inputs)
'world'
Collect output of parallel tasks
Single ‘merge’ task is being fed inputs from a set of parallel tasks. The parallel task inputs (input_array) must be explicitly defined with PREVIOUS or explicit values.
Note
If the following conditions are met: a) no inputs are given to the parallel task (input_array) b) the results of the previous template or task is iterable c) there is only one parallel task in the task_array then each item of the iterable results becomes a parallel task.
Parameters: |
|
---|---|
Returns: | results from the merge task. If the results of the merge task is from a python function the output type will be defined by the type returned otherwise executable output will be a string. |
Return type: | object or string |
Single ‘split’ task feeding inputs to a set of parallel tasks. The parallel task inputs (input_array) must be explicitly defined with PREVIOUS or explicit values.
Note
If the following conditions are met: a) no inputs are given to the parallel task (input_array) b) the results of the split_task is iterable c) there is only one parallel task in the task_array then each item in the iteration becomes a parallel task.
Parameters: |
|
---|---|
Returns: | Output of the parallel operation |
Return type: | list(object) |
List of tasks processing their inputs in parallel.
Valid configurations of task_array and input_array are:
Note
If the following conditions are met: a) no inputs are given to the parallel task (input_array) b) the results of the previous template or task is iterable c) there is only one parallel task in the task_array then each item of the iterable results becomes a parallel task.
Parameters: |
|
---|---|
Returns: | list of task outputs |
Return type: | list |
Example: |
>>> from tigres import *
>>> def adder(a, b): return a + b
>>> task1 = Task(None, FUNCTION, adder)
>>> task2 = Task(None, EXECUTABLE, "/bin/echo")
>>> inputs = InputArray(None,[InputValues("One",[1,2]), InputValues("Two",['world'])])
>>> tasks = TaskArray(None,[task1, task2])
>>> parallel("My Parallel", tasks, inputs)
[3, 'world']
A list of tasks is given in the task_array. Those tasks will be run sequentially multiple times in parallel lanes depending on the size of input_array.
Valid configurations of task_array and input_array are:
Parameters: |
|
---|---|
Returns: | results from the last task executed in the sequence. If the results of the last sequence is from a python function the output type will be defined by the type returned otherwise executable output will be a string. |
Return type: | object or str |
Write a user log entry at level DEBUG.
This simply calls write() with the level argument set to DEBUG. See documentation of write() for details.
Write a user log entry at level ERROR.
This simply calls write() with the level argument set to ERROR. See documentation of write() for details.
Write a user log entry at level TRACE.
This simply calls write() with the level argument set to TRACE. See documentation of write() for details.
Write a user log entry at level DEBUG.
This simply calls write() with the level argument set to DEBUG. See documentation of write() for details.
Write a user log entry at level INFO.
This simply calls write() with the level argument set to INFO. See documentation of write() for details.
Write a user log entry at level ERROR.
This simply calls write() with the level argument set to ERROR. See documentation of write() for details.
Set the level of logging for the user-generated logs.
After this call, all messages at a numerically higher level (e.g. DEBUG if level is INFO) will be skipped.
Parameters: | level (int) – One of the constants defined in this module: NONE, FATAL, ERROR, WARN[ING], INFO, DEBUG, TRACE |
---|---|
Returns: | None |
Raise: | ValueError if level is out of range or not an integer |
Configures and initializes tigres monitoring
WARNING: this will end any previously started programs
Parameters: |
|
---|---|
Returns: | Workflow identifier |
Return type: | str |
Raises: | TigresException |
return: | None |
---|
Write a user log entry.
If the API is not initialized, this does nothing.
Parameters: |
|
---|---|
Returns: | None |
Example:
# Write a simple message
write(Level.WARN, "looking_up", "I see a Vogon Constructor Fleet")
# Or, using the more convenient form
warn("looking_up", "I see a Vogon Constructor Fleet")
# Use key/value pairs
warn("looking_up", vehicle="spaceship", make="Vogon", model="Constructor", count=42)
Get status of a task or template (etc.).
Parameters: |
|
---|---|
Returns: | list(LogStatus). See parameter multiple. |
Find and return log records based on simple criteria of the name of the activity and matching attributes (key/value pairs).
Parameters: |
|
---|---|
Returns: | List of Record instances representing the original user log data |
Find and return log records based on a list of filter expressions.
Supported operators for spec are:
Example expressions: | |
---|---|
foo > 1.5 – find records where field foo is greater than 1.5, ignoring records where foo is not a number. foo ~ 1.d – find records where field foo is a ‘1’ followed by a decimal point followed by some other digit. |
|
Parameters: |
|
Returns: | Generator function. Each item represents one record. |
Return type: | list of Record (metadata = False), list of tuple (Record,metadata) |
Raise: | BuildQueryError, ValueError |
Configures and initializes tigres monitoring
WARNING: this will end any previously started programs
Parameters: |
|
---|---|
Returns: | Workflow identifier |
Return type: | str |
Raises: | TigresException |
return: | None |
---|
Defined levels, and some related functions.
Note that the numeric equivalents of these levels is not the same as the Python logging package. Basically, they are in the reverse order – higher levels are less fatal. The Level class has static methods for manipulating levels and converting to/from the Python logging levels.
Return list of all names of levels, in no particular order.
Returns: | List of level names |
---|---|
Return type: | list of str |
Get Python logging module level.
Parameters: | level – Tigres logging level |
---|---|
Returns: | Equivalent Python logging module level. |
Return type: | int |
Convert a numeric level to its string equivalent.
Parameters: | levelno (int) – Numeric level |
---|---|
Returns: | String level, or empty string |
Return type: | str |
Convert a string level to its numeric equivalent.
Parameters: | level_str (str or basestring) – String level (case-insensitive) |
---|---|
Returns: | Numeric level, or NONE |
Return type: | int |
Get the results from the latest task or template of the running Tigres program
Returns: |
---|
Provide convenient access to status information related to a single log entry.
Instances of this class are returned by the check() function.
Return JSON representation of status.
Record used for formatting
Add values from dict
Convert timestamp to a number of seconds since the epoch (1-1-1970) Convert the level name to a number.
Check whether all keys/values in given record, which are present in this one, are the same.
Parameters: |
|
---|---|
Return type: | bool |
Project the record onto just these fields.
Parameters: |
|
---|---|
Returns: | New record, or nothing if copy=False |
Return type: | Record or None |
Creates an execution graph of the currently running Tigres program and writes it to a file
Parameters: | path – the file path to write the DOT file to |
---|---|
Returns: | |
Sideeffect: | writes a DOT file of the currently running Tigres tigres.core.monitoring.state.Program |
platform: | Unix, Mac |
---|---|
synopsis: | Tigres shared classes/methods/constants |
module author: | Gilberto Pastorello <gzpastorello@lbl.gov>, Val Hendrix <vchendrix@lbl.gov> |
- get_new_output_file() - Get a new unique output file name.
- Execution - Execution plugins available for a Tigres program
- State - States of a Tigres program
- TigresException - A Tigres Program Exception
- TaskFailure - Object to be returned in case of failures in task execution
Execution plugins available
get the execution plugin for the name
Example: |
---|
>>> from tigres.utils import Execution
>>> Execution.get('EXECUTION_LOCAL_THREAD')
'tigres.core.execution.plugin.local.ExecutionPluginLocalThread'
Parameters: | name – execution plugin string |
---|---|
Returns: | plugin module |
Return type: | str |
Tigres State constants
Combine a name and a state into a single string.
Object to be returned in case of failures in task execution
Parameters: |
|
---|
A Tigres Program Exception
Get a new unique output file name.
Parameters: |
|
---|---|
Returns: | Name of file (caller should open) |