Offline Processing

A realtime pipeline is started, left running, and stopped by whoever started it. An offline pipeline is driven: you call run(), and it returns when the whole recording has passed through the last node.

That difference is not just convenience. It changes what the framework can offer you, and this page is about the things that only hold offline.

Driving a recording

A pipeline is offline because its source says so. Every reader takes mode="batch":

import gpype as gp

with gp.Pipeline() as p:
    reader = gp.CsvReader(file_name="recording.csv",
                          sampling_rate=250, mode="batch")
    band = gp.Bandpass(f_lo=8, f_hi=12)
    out = gp.Collector()

    p.connect(reader, band)
    p.connect(band, out)

    result = p.run()

print(result.data.shape)

run() executes on the calling thread and creates none of its own, which is why an exception in a node arrives as an ordinary Python traceback through your own line rather than as a log entry from a monitoring thread.

Readers: CsvReader, EDFReader, MatReader, HDF5Reader.

run() refuses a realtime pipeline, naming the mode it found and both ways out:

ValueError: run() drives a batch pipeline, and this one is
'realtime'. Either build it with a source in batch mode --
CsvReader(..., mode='batch') -- or use start() and stop() to run it
against the clock.

The reverse is not symmetric: start() accepts a batch pipeline and runs it against the clock, because a recording paced to the clock is a perfectly ordinary realtime pipeline. If you meant to drive it as fast as the machine allows, call run().

Season 2, Episode 16 does both halves: it records five seconds against the clock, then filters the whole file in one run() with the same Bandpass.

What a Result carries

run() hands back a Result, not a bare array: .data is the samples, and beside it are .labels, .roles, .rate and .units, and, where the source records them, .start_time, .events, .gaps and .trust. A field the source does not record is None or empty rather than guessed. See the Result reference.

Consult .gaps before cutting epochs: samples either side of a dropout are not adjacent, however they look.

Zero-phase filtering

The reason to validate offline. Every filter takes a phase:

gp.Bandpass(f_lo=8, f_hi=12, phase="zero")

phase="causal" is the default and is the same state machine as a realtime run over the same samples, so a causal offline run and a live run produce identical output. That is the property that makes validating offline worth anything at all.

phase="zero" filters forwards and backwards, leaving no group delay at any frequency. Measured on a 10 Hz sine at 250 Hz: 0 samples of delay, against 1 for the causal pass. A realtime pipeline cannot do this — it would have to know the future — which is exactly why the option exists here and nowhere else.

Season 2, Episode 17 measures it on a symmetric burst, by symmetry and cross-correlation rather than by peak position, and ends by showing the same filter refused in a realtime pipeline.

A function as a node

Offline, a transform that is three lines of numpy does not need a class:

import numpy as np

@gp.node
def to_microvolts(block):
    return block * 1e6

p.connect(reader, to_microvolts())

Or inline, with Apply:

p.connect(reader, gp.Apply(np.abs))

The function is handed the whole recording as a (samples, channels) array and must return one of the same shape and dtype. A bare callable has no way to declare a contract to the node downstream, so changing either is refused with both values named — write a node class when you need to change the shape, and see Node Design Pattern.

@gp.node leaves your function reachable, so it stays testable with no pipeline involved:

assert to_microvolts.function(np.ones((2, 2))).max() == 1e6

This is offline only, and a realtime pipeline refuses it. Two reasons, and neither is stylistic. An arbitrary callable has no bound on its execution time, and the budget it would have to fit inside is one frame — about 16 ms at the derived frame size and 4 ms at frame_size 1. And purity is enforceable offline and not online: a closure quietly accumulating state across frames looks identical to a pure transform at the call site, and is wrong in a way that surfaces only under load or after a restart. Offline neither applies, because there is no deadline and the recording arrives in one call.

One consequence worth knowing: a pipeline holding a function cannot be saved. A function is not a configuration value, so a serialized pipeline could not rebuild the node, and serialize() refuses the pipeline with a ValueError that names each function node. Run such a pipeline from the script that builds it, or write a node class whose parameters are configuration.

s7e6_a_function_as_a_node.py runs the same function both ways, shows that the decorated function is still callable on its own, and ends with the refusal a realtime pipeline gives.

When a node refuses

A node says what it cannot do in setup(), and offline you get that exception unwrapped, through your own line:

p.connect(reader, gp.Bandpass(f_lo=200, f_hi=300))   # above Nyquist
p.run()
  File ".../filters/base/butterworth.py", line 188, in setup
    raise ValueError(
ValueError: f_lo must be below the Nyquist frequency (125 Hz, half
the 250 Hz sampling rate); got 200 Hz.

Nothing is caught and re-reported. run() is the caller, on the caller’s thread, so the framework has no reason to stand between you and the traceback – the failing frame is the last one, and if you wrote the node it is yours.

A realtime pipeline cannot do that, because the exception happens on a source thread nobody is waiting on. There start() raises a RuntimeError naming the node, with the original attached as __cause__ – the same information, one layer further away, which is the best that can be done once a thread is involved.

Samples are 32-bit floats

Every sample in a pipeline is a 32-bit float, offline as well as live; a .mat or .h5 file stores it as 64-bit, which changes no value, and an .edf file as a 16-bit integer, which does. The shapes, units and timestamps of what a run hands back are in Data Structures, and each format’s layout in File Formats.

Carrying each channel’s calibration beside its samples, so that the conversion can be checked, is decided and not built yet. Whether an offline run should compute in 64-bit floats is still open. Until both are settled, treat an offline result as exact to 32-bit precision and no further.

The MNE bridge

MNE-Python is not a g.Pype dependency, so the bridge is a function in an example rather than in g.Pype. A Result already carries what MNE needs, and s4e9_into_mne.py builds an mne.io.RawArray from one:

  • the samples, turned to MNE’s (channel, time), each signal channel multiplied by its si_scale – MNE works in volts, a source in whatever unit it declares;

  • mne.create_info from .labels, .rate and .roles: signal becomes eeg and trigger becomes stim, whose codes pass through unscaled. Any other role is refused, so select the signal and trigger channels first;

  • mne.Annotations from .events, their samples divided by the rate.

Where a result records no units, si_scale is None and the bridge refuses rather than assume microvolts: a recording in millivolts would reach MNE a thousand times too small, and nothing would say so. Units reach a recording from an amplifier source, or from ChannelLabeler(units=[...]) in front of the writer, and a .csv file stores none. s4e9_into_mne.py records one run to HDF5 and to CSV, converts the first and prints the refusal for the second.

The scikit-learn bridge

Sklearn bridges one scikit-learn estimator into g.Pype’s “fit offline, deploy online” contract: LinearDiscriminantAnalysis and LogisticRegression as classifiers, StandardScaler and PCA as transformers – a fixed allow-list, refused otherwise, naming the four. scikit-learn is not a dependency: constructing this node without it installed raises an ImportError naming the package.

A classifier fits on the (time, channel, trial) block Epochs cuts, with labels from the trials it publishes, and predicts one class index per trial or per realtime epoch. A transformer fits and applies plain (samples, channels) rows, in batch and in realtime alike.

s4e10_into_scikit_learn.py writes its own two-class recording, a plain HDF5 file, so it needs no hardware and no download. It cuts each condition in its own Epochs run – a batch pipeline merges no inputs, so the two conditions are joined in numpy between the runs – and reports a plain sklearn.model_selection.cross_val_score well above chance. Then it fits the bridge with fit_offline(), the escape hatch for training data that does not come from one batch source, and deploys it through step() on trials it never trained on.

What a fit marks

A fitted artifact records whether it is marked: the non-commercial mark an unlicensed run writes into its recordings travels with what was trained on it. An artifact is marked when the fitting run was limited – no licence held – when it read an unsealed or recovered recording, or when it read a file a marked run wrote.

fit_offline() runs outside any pipeline, so nothing checks a licence or a device for it, and it always marks its artifact, on a licensed machine too. A fit that skipped the check must not produce an artifact that claims to have passed it.

fit() on a licensed machine, from a file that carries no mark, gives an unmarked artifact: a licence lifts the mark a run would otherwise get for having no amplifier.

What you see: state["marked"] is True, and a pipeline that loads or deploys the artifact runs marked – its writers mark their files and its log says why – even where the run would otherwise be unmarked, on a licensed machine. Nothing else changes: the run is not degraded or refused.

Two things that catch people out

Writers timestamp their filenames. gp.CsvWriter(file_name="rec.csv") writes something like rec_20260907_110441_205.csv, so a recording is never silently overwritten. A script that asserts on the configured path will report a working pipeline as broken.

Stop, then close. stop() releases threads, file handles and hardware; close() releases logging and the master timeline. A pipeline can be started again after stop(), which is why the two are separate — or use with, as above, and neither is your problem.