.. _tigresarch: .. currentmodule:: tigres Architecture ************ This section details the Tigres architecture. In this section, Tigres data types and their relationships with one another, a high-level overview of the major Tigres components and how they interact and the special Tigres syntax that allows you to explicitly specify data dependencies between workflow tasks is discussed. .. contents:: :depth: 1 :local: :backlinks: top .. _tigresdatamodel: .. include:: includes/data_model.rst System Components ================== .. image:: _static/images/tigres_arch_overview.png :width: 50% :align: left Tigres has five major components in a layered architecture: `User API`, `Core API`, `State Management`, `Execution Management` and `Monitoring`. The top most layer, the `User API`, is directly supported by `Monitoring` and `Core API` components. The execution type semantics is managed deeper down the stack by `State` and `Execution Management` components. The user interacts with a thin User API to define the elements of their workflows, log messages and monitor their program execution. The core API is a collections of interfaces that support the User API with graphing, creating tasks, managing task dependencies and running templates. Monitoring is used by most other components to log system or user-defined events and to monitor the program with a set of entities for querying those events. State management encompasses different execution management aspects of the workflow. For instance, it validates the user input to confirm they result in a valid workflow both prior to start as well as during execution. It transforms the Tigres tasks into `Work` that are then handed off to execution management. The state management layer also provides monitoring and provenance capabilities. It maintains state as each template is invoked and integrity of the running Tigres program. The execution layer can work with one or more resource management techniques including HPC and cloud environments. In addition, the separation of the API and the execution layer allows us to leverage different existing workflow execution tools while providing a native programming interface to the user. .. include:: includes/monitoring.rst Execution Management ===================== A Tigres program may be written once and run multiple times on different resource systems. The execution engine is specified at run time. A Tigres program can be executed on your personal desktop with local threads or processes. It can also be distributed across a cluster of nodes. Additionally, if you have access to a batch job scheduler, Tigres can use that. The interchangeability of different execution mechanisms allows you to develop your analysis on a desktop or laptop and scale it up to a department clusters or HPC centers when your code is ready for production without actually changing the workflow code. The different Tigres execution plugins require minimal setup. The plugins can be classified as desktop (single node), cluster and job manager execution plugins. Each execution plugin has its own mechanism for executing tasks as well as parallelization. =========== ===================== ===================== ================================== ---------------------------------------------------------------------------------------------- Type Name Execution Mechanism Parallelization =========== ===================== ===================== ================================== Desktop | Local Thread | Threads | Local Worker Queue | Local Process | Processes | Local Worker Queue Cluster | Distribute Process | Processes | Task Server and Task Clients Job Manager | SGE | SGE Job | Sun Grid Engine (:code:`qsub`) | SLURM | SLURM Job | SLURM (:code:`squeue`) =========== ===================== ===================== ================================== .. _desktopexecution: Desktop Execution +++++++++++++++++ .. image:: _static/images/execution_local_process_or_thread.png :width: 50% :align: right There are two execution plugins for use on a single node: :code:`LOCAL_THREAD` and :code:`LOCAL_PROCESS`. Both plugins behave similarly except that :code:`LOCAL_THREAD` spawns threads for each task while :code:`LOCAL_PROCESS` spawns a process. The default Tigres execution is :code:`LOCAL_THREAD`. You change the execution mechanism by passing in a keyword argument to :func:`tigres.start`:: from tigres import start from tigres.utils import Execution start(execution=Execution.LOCAL_PROCESS) ... The figure to the right shows a simple Tigres program that launches *n* number of parallel tasks (A) and follows it up with a sequence of *m* tasks (B). This workflow is run on a single node (localhost). Parallel execution is a very simple mechanism. All parallel tasks are placed in a worker queue, the number of processing cores is determined and then a worker is spawned for each processing core. In the case of *local thread* execution, the worker is a :class:`threading.Thread` for *local process* execution a :class:`multiprocessing.Process`. .. _clusterexecution: Cluster Execution +++++++++++++++++ Running the Tigres program from above on a set of nodes in a local cluster takes a little more set up. You change the execution plugin (``Execution.LOCAL_PROCESS``) and then set a one or more environment variables. ``DISTRIBUTE_PROCESS`` distributes tasks across user specified host machines. A `TaskServer` is run in the main Tigres program. `TigresClient`s`, which run workers that consume the tasks from the `TaskServer`, are launched on the hosts specified in ``TIGRES_HOSTS`` environment variable . If no host machines are specified, the `TaskServer` and a single `TaskClient` will be run on the local machine. .. image:: _static/images/execution_distribute_process.png :width: 60% :align: left The Tigres program may have additional nodes configured with special environment variables: ``TIGRES_HOSTS`` and ``OTIGRES_*``. ``TIGRES_HOSTS`` is a comma separated list of hosts names for distributing tasks. (i. e. ``export TIGRES_HOSTS=n1,n2,n3``). Additionally, any environment variable prefixed with ``OTIGRES_`` will be passed to the `TaskClient` program on each node. For example, ``OTIGRES_PATH=tigres_home/bin`` will be translated to the command ``PATH=tigres_home/bin; tigres-client`` which is launched on the `TaskClient` nodes specified in ``TIGRES_HOSTS``. The figure above shows the same Tigres program as the previous figure. The program still has *n* parallel tasks and *m* sequential tasks but this time it is distributed across *p* nodes.:: localhost$ export TIGRES_HOSTS=node1,node2...nodep localhost$ OTIGRES_PATH=tigres_home/bin my_analysis.py ... The parallel template tasks are distributed across **Node 1** through **Node p**. The main Tigres program starts a `TaskServer` on **Node 1** and then launches *p* `TaskClients` on the nodes specified in ``TIGRES_HOSTS``. The client programs run a process for each processing core until the `TaskServer` runs out of tasks. The sequence template tasks are run on **Node 1** where the main Tigres program is executing. Job Manager Execution +++++++++++++++++++++ .. image:: _static/images/execution_job_manager.png :width: 75% :align: left If you have access to resources managed by a job scheduler, Tigres supports Sun Grid Engine (SGE) and Simple Linux Utility for Resource Management (SLURM). Each task is executed as a job on the batch queue. Parallelism is managed by the job manager. Tigres submits all the jobs at the same time and then waits for completion. The requirements for using a job manager execution mechanism are that you must have an account and the ability to load the Tigres python module. Of course the correct execution plugin must be chosen:: from tigres import start from tigres.utils import Execution start(execution=Execution.SGE) ... The naive assumption is that you would be able to run your Tigres program from the submission node and have unfettered access to the queue. At `National Energy Research Scientific Computing Center (NERSC)`_, this is achieved with a resource reservation that provides a personal installation of SGE called MySGE. .. _National Energy Research Scientific Computing Center (NERSC): http://www.nersc.gov In the figure above, ``my_analysis.py`` is run at scale with managed resources. The program is run on the node where jobs are submitted to the batch queue. As with the other two examples, :ref:`desktopexecution` and :ref:`clusterexecution`, the workflow first runs a parallel template with *n* tasks and follows it up with a sequence template of *m* tasks. The parallel template submits *n* jobs simultaneously to the job manager. Upon their completion, the sequence template is executed. This template submits one task job at a time and waits until it completes before the next one is submitted.