Use this reference
Start with the Experiment package guide. These API details describe version 1.2.0. Execution profiles are documented in Execution and HPC setup.
Process definition
Transform source datasets into targets
Process(target, source, flow,
nodes=None, ngroup=None, split="sgwindow --hist=ignore ",
cat=None, src_split=None, *, chunks=1, resources=None,
**options)A Process defines one or more targets, their sources, and the Program flow that performs the transformation. Targets depend on the sources, Programs and parameters in the definition, so only affected Processes are rerun after a change.
By default, dataset names are resolved under data/ with the .sg suffix.
Use keyword arguments for optional settings. nodes is retained only for compatibility checking and must stay None; host dictionaries are no longer accepted.
Quality control
Generate dependency-aware Figures
Figure(target, sources, flow, **options)When the source dataset has the same base name as the target, use the shorter form:
Figure(target, flow, **options)Figure is a specialized Process for visual quality-control output. Its defaults accept SIGCLEAR datasets from data/, produce PostScript output, and suppress processing verbosity. PDF, PNG, GIF and JPEG targets are converted automatically from the generated PostScript stream.
Configuration
Common Process and Figure options
| Option | Process default | Figure default | Purpose |
|---|---|---|---|
sprefix | data/ | data/ | Source path prefix |
tprefix | data/ | data/ | Target path prefix |
ssuffix | .sg | .sg | Source suffix |
tsuffix | .sg | .ps | Target suffix |
verb | True | False | Program verbosity |
stdin | True | True | Pass the first source through standard input |
stdout | True | True | Write standard output to the first target |
chunks | 1 | Not applicable | Number of independent group partitions; one leaves the flow unpartitioned |
ngroup | None | Not applicable | Detect input group count when partitioning; optionally supply a known positive count |
resources | None | None | Inherit the selected profile's per-action resources; override only when this action needs different resources |
Scheduling
Choose parallelism that fits the experiment
| Level | Work unit | Control |
|---|---|---|
| Program | Threads within a parallel-capable Program on one node | --nthread, consistent with the action's CPU request |
| Within a Process | Independent ranges of SOIG groups | Explicit chunks; automatic ngroup=None |
| Workflow | Dependency-ready Processes and chunk actions | Profile defaults, resource limits and optional scons -j N |
Profiles do not partition datasets automatically. First validate the serial workflow, then opt into chunks only when each group can be processed independently:
Process('scaled', 'raw',
'sgfieldmath data="2*data"', chunks=4)This example requires at least four input groups and scales each group's data independently. The group count is detected after the input is built; missing count metadata requires an extra read pass. A known positive ngroup can avoid that scan. Chunk results are kept for ordered merging and incremental reuse.
Chunks share the workflow's job pool and resource budget; four chunks do not request four nodes or guarantee four concurrent actions. Each chunk's pipeline runs on one node. Global fits, cross-group filters and reductions need a scientifically valid split/merge design before partitioning. An explicit -j overrides the default job limit while resource and policy limits still apply.
When upgrading from earlier packages, replace host dictionaries with profiles and explicit chunks where appropriate. Legacy rsh, SGNODES and Python config.py node settings are no longer used.