Execution and HPC setup

Choose an execution environment

Ordinary local workflows need only scons. This reference is for users and administrators preparing reusable resource profiles. A Slurm profile requires an existing allocation or configured automatic allocation; installing Slurm tools alone does not select HPC execution.

Execution profiles

Keep machine settings outside the workflow

Start with scons. Without configuration, the built-in local profile schedules ready actions using available CPU slots. A profile supplies the executor, job limit and per-action CPU, memory and time requests, so ordinary Process and Figure definitions need no machine settings.

Optional config.json files combine in this order:

Site: /etc/sigclear/config.json
User: ~/.sigclear/config.json
Experiment: config.json beside the main SConstruct script

More specific settings inherit unspecified values. Site policies and Slurm allocation limits remain authoritative. The selected profile comes from profile=NAME, then SIGCLEAR_EXECUTION_PROFILE, then the configured default. The default auto selects local execution outside a Slurm allocation and the configured HPC profile inside one.

scons profile=local
scons profile=hpc
scons profile=hpc mode=batch

The HPC commands require a configured Slurm profile. Foreground mode is the default: it waits for resources and shows live output. Batch mode submits the workflow and returns a job ID and log location; submission does not mean the experiment has completed. An existing allocation is reused without submitting a nested job.

Example HPC profile for a site administrator

Adapt these example budgets to the cluster and workload. Save the file at one of the scope locations above. Users can then select profile=hpc without adding Slurm commands to their workflows.

{
  "schema_version": 1,
  "profiles": {
    "hpc": {
      "executor": "slurm",
      "allocation": {
        "nodes": 2,
        "cpus_per_node": 8,
        "memory_per_node": "24G",
        "walltime": "02:00:00"
      },
      "resources": {"cpus": 2, "memory": "4G"},
      "jobs": "auto"
    }
  }
}

The allocation reserves the run's overall capacity; resources describe each action. Set "default_profile": "hpc" at the top level to make plain scons select this profile. Without allocation settings, an HPC profile requires an existing allocation.

All nodes need matching packages, Python helpers, applicable licenses and shared workflow/data paths. Keep workflow files stable while a batch job is queued or running. Slurm and the site's job-management tools control queue state and cancellation.

Resources and run records

Configuration defaults combine from built-in settings through site, user and experiment files. Policies accumulate: a more specific file can tighten earlier limits but cannot relax them. An explicit scons -j N overrides the default job count while resource and policy limits still apply.

A profile's resources request applies to each action, including chunk planning, splits and merges. Local admission shares a configured budget; it is not an operating-system memory sandbox. Pipeline stages share their action's resources. Set --nthread consistently with the requested CPUs.

The startup summary reports resolved execution settings. Local execution reports its host once; Slurm actions report their compute host and job ID. Real runs save execution records under .sigclear/runs/; batch submission reports its job ID and private log paths under .sigclear/batch/. Submission is not completion. Keep the session open for foreground runs; batch runs continue independently.

Use the site's Slurm tools to inspect queue state and cancel submitted jobs. Leave workflow code and shared data paths available while jobs are queued or running. Dry runs, help and cleaning do not request allocations.