SOIG dataset reference

Dataset structure

One schema, many measurement records

The header stores processing history, dataset-level attributes and a field schema. Every column contains values described by that same schema. Groups are consecutive runs of columns, rather than separate field definitions for each group.

A column is not necessarily one numerical sample or one CSV row. A scalar-only dataset may represent one observation per column; a trace dataset can contain hundreds of samples within each column. Different vector fields may have different lengths, but each field's declared length applies to every column. Text fields have a fixed byte width.

seed.sg
├── Header
│   ├── Attributes: data.ns=100, data.s0=0, data.ds=0.001
│   │               group key → group field; index key → index field
│   └── Field schema: group (integer scalar), index (integer scalar),
│                    data (100 floating-point samples)
└── Columns: one group with group-key value 0
    ├── Column 0: group=0, index=0, data=[0, ..., 0]
    └── Column 1: group=0, index=1, data=[0, ..., 0]

Grouping rules

Choose the processing boundary explicitly

The group attribute identifies the scalar field used as the group key. Values in that field determine membership; the field need not itself be named group. The subgroup and index attributes likewise identify fields used by Programs that need subdivisions or indexing.

Group reads follow adjacent columns, not a global SQL-style group-by:

Column key values:  1, 1, 2, 1
Consecutive groups: [1, 1] [2] [1]  → three groups, not two

Decide which measurements belong together, arrange their order, and inspect the key before group-wise processing such as sgstack. Sorting or concatenating can change group boundaries.

Sampling and units

Keep coordinates consistent with the vector

For a regularly sampled vector named data, these header attributes describe its sampling:

The coordinate at zero-based sample index i is data.s0 + i * data.ds, for i=0,...,data.ns-1. The example spans 0 through 0.099, not 0.100. If the coordinate is time in seconds, the interval is 1 ms.

Use the vector's own name for other sampling attributes, such as pressure.ns, pressure.s0 and pressure.ds. These are common metadata, not extra samples or per-column measurements.

Record physical units, field meanings and missing-value conventions explicitly. Field names do not establish units, and a zero-filled vector is not automatically missing data. Check required types, lengths and spacing in each Program's manual; recheck sampling metadata after windowing, resampling or concatenation.

Inspect and exchange

Validate the schema before processing

• Use sgattribute to check field names, types, element counts, byte offsets and grouping/sampling attributes.
• Use sgdump to inspect representative values and key sequences. Its diagnostic text is not a replacement binary dataset or a general-purpose interchange export.
• Keep binary outputs redirected to a file or another Program; do not display them directly in the terminal.
• Retain raw inputs, converter parameters, physical units and acceptance checks in the experiment definition.

SOIG uses a native binary representation. Do not assume byte-order or architecture/ABI independence when moving files between systems. For external tools, select the appropriate I/O conversion Program and verify the resulting types, shapes and coordinates on a small sample.

Use sgloadcsv/sgoutcsv for supported table layouts, sgloadmat/sgoutmat for MATLAB exchange, and sgloadbin for explicitly described fixed-size binary records. Choose conversion parameters from the matching manual; conversion does not establish units or correct grouping automatically.

Processing considerations

Structure supports processing; the Program defines behavior

Scope

Numerical research data, not business records

SOIG is designed for numerical arrays and measurements produced by sensors, simulations and scientific algorithms. A relational database remains more suitable for customers, orders and other transactional records.