sigclear-experiment

Overview

Describe and schedule an experiment

sigclear-experiment provides the Process and Figure definitions used in a Python-based SConstruct workflow. SIGCLEAR analyzes dependencies among inputs, outputs, Programs and parameters before scheduling the required work.

Package contents

Workflow definitions rather than processing Programs

The Package installs the Python experiment module that provides Process and Figure. The Programs invoked by a flow come from functional Packages such as sigclear-soig, sigclear-dsp and sigclear-plot, or from third-party and custom Packages.

sigclear-experiment depends on SCons for dependency scheduling and ImageMagick for supported raster-figure conversion.

Process definition

Transform source datasets into targets

Process(target, source, flow,
    nodes=None, ngroup=0, split="sgwindow --hist=n ",
    cat=None, src_split=None, **options)

A Process defines one or more targets, their sources, and the Program flow that performs the transformation. Targets depend on the sources, Programs and parameters in the definition, so only affected Processes are rerun after a change.

By default, dataset names are resolved under data/ with the .sg suffix.

Quality control

Generate dependency-aware Figures

Figure(target, sources, flow, **options)

When the source dataset has the same base name as the target, use the shorter form:

Figure(target, flow, **options)

Figure is a specialized Process for visual quality-control output. Its defaults accept SIGCLEAR datasets from data/, produce PostScript output, and suppress processing verbosity. PDF, PNG, GIF and JPEG targets are converted automatically from the generated PostScript stream.

Quick example

Connect a Process to its QC Figure

from experiment import *

Process('result', 'input',
    'sgfieldmath data="data*2"')

Figure('./result.png', 'result',
    'sgplotps')

Run scons to build data/result.sg and then generate result.png. Use man sgfieldmath and man sgplotps for the Program options.

Configuration

Common Process and Figure options

Scheduling

Run independent Processes concurrently

Use the SCons job option to set the maximum number of concurrent jobs:

scons -j 8

The scheduler respects dependencies while running independent Processes concurrently. Processes configured for grouped execution can also distribute groups across the nodes supplied through the nodes option.