Workflow Automation Documentation

Overview

What Workflow Automation provides

Workflow Automation combines dependency-aware experiment scheduling with scientific data management, data processing, visualization, optimization and quality control.

Before you begin

Install and activate SIGCLEAR, then run sglicense-check to verify your license.

Quick start

Start a simple experiment in SIGCLEAR

Create an empty project directory and add a file named SConstruct:

from experiment import *

Process('raw', None,
    '''
    sgcreate nx=10 data.ns=500
    | sgfieldmath offset:f="50*ix"
      data="sinc(30*(data.time-(600^2+offset^2)^0.5/1500))"
    ''')

Process('processed', 'raw',
    'sgfieldmath data="2*data"')

Figure('./processed.png', 'processed',
    'sgplotps')

Run the experiment from that directory:
```
scons
```

The workflow creates the dataset data/raw.sg, transforms it into data/processed.sg, and generates processed.png as a quality-control figure.

Core concepts

Process, Program and Package

As shown in the Quick Start, a SIGCLEAR experimental workflow is defined in a Python-based SConstruct file using the SIGCLEAR Experiment API. Three concepts organize the workflow: Process, Program and Package.

Process

A workflow is composed of Processes. Each Process defines one or more output datasets, its input datasets, and the processing flow that transforms the inputs into outputs.

A complex experiment can contain many Processes connected in a dependency graph. When an input, parameter or Program changes, SIGCLEAR rebuilds the affected datasets and regenerates their dependent figures. Unaffected Processes and figures are not rerun.

The Quick Start contains three Processes. The first creates raw; the second reads raw, doubles its data vector, and writes processed; the third generates processed.png from processed:

raw → processed → processed.png

Figure is a specialized Process

Within the workflow model, Figure is a specialized Process for visual quality-control output. It participates in the same dependency graph as other Processes.

A workflow imports the SIGCLEAR Experiment API with from experiment import * before defining Processes and Figures.

Program

A Program is an executable that performs a specific operation within a Process. A processing flow can run one Program or connect several Programs through a pipeline to transform input datasets into output datasets.

To create raw, the first Process runs sgcreate to create a dataset containing a 500-sample data vector across 10 columns. It then runs sgfieldmath to add the offset field and calculate the sinc signals stored in data.

The second Process runs sgfieldmath to double the data vector. The Figure runs sgplotps to generate the plot.

SIGCLEAR Programs can also run directly from the Linux command line. For example, this command doubles the data vector in the raw dataset:

sgfieldmath data="data*2" <data/raw.sg >data/processed-cli.sg

Program manuals

Use man with a SIGCLEAR Program name to open its command reference:

man sgcreate

Package

A Package is an installable collection of related Programs, libraries, modules and supporting resources. Each Package provides focused capabilities for experimental workflows.

sigclear-experimentExperiment API and process scheduling sigclear-soigLarge-scale multidimensional dataset management sigclear-ioConversion between SIGCLEAR datasets and other data formats sigclear-dspSignal synthesis, filtering and analysis sigclear-plotVisualization and experiment QC sigclear-optimOptimization and parameter inversion

In the Quick Start, sigclear-experiment provides the Experiment API and execution scheduling. The Program sgcreate is provided by sigclear-soig; sgfieldmath and sgplotps are supplied by sigclear-dsp and sigclear-plot, respectively.

Users can install additional Packages or develop custom Packages to add specialized functions to their experimental workflows.

Examples

Follow complete research workflows

NASA air-pressure fittingImport atmospheric measurements and compare robust model fits Lunar LPR data analysisImport instrument records, filter radar channels and generate quality control

Next steps

Move from example to your research workflow

Request a trial Explore implementation services Discuss your workflow

Build a reproducible scientific workflow.