Offline Processing
A realtime pipeline is started, left running, and stopped by whoever
started it. An offline pipeline is driven: you call
run(), and it returns when the whole recording has
passed through the last node.
That difference is not just convenience. It changes what the framework can offer you, and this page is about the things that only hold offline.
Driving a recording
A pipeline is offline because its source says so. Every reader takes
mode="batch":
import gpype as gp
with gp.Pipeline() as p:
reader = gp.CsvReader(file_name="recording.csv",
sampling_rate=250, mode="batch")
band = gp.Bandpass(f_lo=8, f_hi=12)
out = gp.Collector()
p.connect(reader, band)
p.connect(band, out)
result = p.run()
print(result.data.shape)
run() executes on the calling thread and creates none of its own,
which is why an exception in a node arrives as an ordinary Python
traceback through your own line rather than as a log entry from a
monitoring thread.
Readers: CsvReader, EDFReader,
MatReader, HDF5Reader.
run() refuses a realtime pipeline, naming the mode it found and both
ways out:
ValueError: run() drives a batch pipeline, and this one is
'realtime'. Either build it with a source in batch mode --
CsvReader(..., mode='batch') -- or use start() and stop() to run it
against the clock.
The reverse is not symmetric: start() accepts a batch pipeline and
runs it against the clock, because a recording paced to the clock is a
perfectly ordinary realtime pipeline. If you meant to drive it as fast
as the machine allows, call run().
Season 2, Episode 16
does both halves: it records five seconds against the clock, then
filters the whole file in one run() with the same
Bandpass.
What a Result carries
run() hands back a Result, not a bare
array: .data is the samples, and beside it are .labels,
.roles, .rate and .units, and, where the source records them,
.start_time, .events, .gaps and .trust. A field the
source does not record is None or empty rather than guessed. See the
Result reference.
Consult .gaps before cutting epochs: samples either side of a
dropout are not adjacent, however they look.
Zero-phase filtering
The reason to validate offline. Every filter takes a phase:
gp.Bandpass(f_lo=8, f_hi=12, phase="zero")
phase="causal" is the default and is the same state machine as a
realtime run over the same samples, so a causal offline run and a live
run produce identical output. That is the property that makes validating
offline worth anything at all.
phase="zero" filters forwards and backwards, leaving no group delay
at any frequency. Measured on a 10 Hz sine at 250 Hz: 0 samples of
delay, against 1 for the causal pass. A realtime pipeline cannot do this
— it would have to know the future — which is exactly why the option
exists here and nowhere else.
Season 2, Episode 17 measures it on a symmetric burst, by symmetry and cross-correlation rather than by peak position, and ends by showing the same filter refused in a realtime pipeline.
A function as a node
Offline, a transform that is three lines of numpy does not need a class:
import numpy as np
@gp.node
def to_microvolts(block):
return block * 1e6
p.connect(reader, to_microvolts())
Or inline, with Apply:
p.connect(reader, gp.Apply(np.abs))
The function is handed the whole recording as a (samples, channels)
array and must return one of the same shape and dtype. A bare
callable has no way to declare a contract to the node downstream, so
changing either is refused with both values named — write a node class
when you need to change the shape, and see
Node Design Pattern.
@gp.node leaves your function reachable, so it stays testable with
no pipeline involved:
assert to_microvolts.function(np.ones((2, 2))).max() == 1e6
This is offline only, and a realtime pipeline refuses it. Two
reasons, and neither is stylistic. An arbitrary callable has no bound on
its execution time, and the budget it would have to fit inside is one
frame — about 16 ms at the derived frame size and 4 ms at
frame_size 1. And purity is enforceable offline and not online: a
closure quietly accumulating state across frames looks identical to a
pure transform at the call site, and is wrong in a way that surfaces
only under load or after a restart. Offline neither applies, because
there is no deadline and the recording arrives in one call.
One consequence worth knowing: a pipeline holding a function cannot
be saved. A function is not a configuration value, so a serialized
pipeline could not rebuild the node, and serialize() refuses the pipeline
with a ValueError that names each function node. Run such a
pipeline from the script that builds it, or write a node class whose
parameters are configuration.
s7e6_a_function_as_a_node.py runs the same function both ways, shows that the decorated function is still callable on its own, and ends with the refusal a realtime pipeline gives.
When a node refuses
A node says what it cannot do in setup(), and offline you get that
exception unwrapped, through your own line:
p.connect(reader, gp.Bandpass(f_lo=200, f_hi=300)) # above Nyquist
p.run()
File ".../filters/base/butterworth.py", line 188, in setup
raise ValueError(
ValueError: f_lo must be below the Nyquist frequency (125 Hz, half
the 250 Hz sampling rate); got 200 Hz.
Nothing is caught and re-reported. run() is the caller, on the
caller’s thread, so the framework has no reason to stand between you and
the traceback – the failing frame is the last one, and if you wrote the
node it is yours.
A realtime pipeline cannot do that, because the exception happens on a
source thread nobody is waiting on. There start() raises a
RuntimeError naming the node, with the original attached as
__cause__ – the same information, one layer further away, which is
the best that can be done once a thread is involved.
Samples are 32-bit floats
Every sample in a pipeline is a 32-bit float, offline as well as live;
a .mat or .h5 file stores it as 64-bit, which changes no value,
and an .edf file as a 16-bit integer, which does. The shapes, units
and timestamps of what a run hands back are in Data Structures, and each format’s layout in File Formats.
Carrying each channel’s calibration beside its samples, so that the conversion can be checked, is decided and not built yet. Whether an offline run should compute in 64-bit floats is still open. Until both are settled, treat an offline result as exact to 32-bit precision and no further.
The MNE bridge
MNE-Python is not a g.Pype dependency, so the bridge is a function in an
example rather than in g.Pype. A Result
already carries what MNE needs, and s4e9_into_mne.py builds an
mne.io.RawArray from one:
the samples, turned to MNE’s
(channel, time), each signal channel multiplied by itssi_scale– MNE works in volts, a source in whatever unit it declares;mne.create_infofrom.labels,.rateand.roles:signalbecomeseegandtriggerbecomesstim, whose codes pass through unscaled. Any other role is refused, so select the signal and trigger channels first;mne.Annotationsfrom.events, their samples divided by the rate.
Where a result records no units, si_scale is None and the bridge
refuses rather than assume microvolts: a recording in millivolts would
reach MNE a thousand times too small, and nothing would say so. Units
reach a recording from an amplifier source, or from
ChannelLabeler(units=[...]) in front of the writer, and a .csv
file stores none. s4e9_into_mne.py
records one run to HDF5 and to CSV, converts the first and prints the
refusal for the second.
The scikit-learn bridge
Sklearn bridges one scikit-learn estimator into g.Pype’s
“fit offline, deploy online” contract: LinearDiscriminantAnalysis
and LogisticRegression as classifiers, StandardScaler and
PCA as transformers – a fixed allow-list, refused otherwise, naming
the four. scikit-learn is not a dependency: constructing this node
without it installed raises an ImportError naming the package.
A classifier fits on the (time, channel, trial) block
Epochs cuts, with labels from the trials it publishes,
and predicts one class index per trial or per realtime epoch. A
transformer fits and applies plain (samples, channels) rows, in
batch and in realtime alike.
s4e10_into_scikit_learn.py writes its
own two-class recording, a
plain HDF5 file, so it needs no hardware and no download. It cuts each
condition in its own Epochs run – a batch pipeline
merges no inputs, so the two conditions are joined in numpy between the
runs – and reports a plain sklearn.model_selection.cross_val_score
well above chance. Then it fits the bridge with
fit_offline(), the escape
hatch for training data that does not come from one batch source, and
deploys it through step() on trials it never trained on.
What a fit marks
A fitted artifact records whether it is marked: the non-commercial
mark an unlicensed run writes into its recordings travels with what was
trained on it. An artifact is marked when the fitting run was limited
– no licence held – when it read an unsealed or recovered
recording, or when it read a file a marked run wrote.
fit_offline() runs outside any pipeline, so nothing checks a licence
or a device for it, and it always marks its artifact, on a licensed
machine too. A fit that skipped the check must not produce an artifact
that claims to have passed it.
fit() on a licensed machine, from a file that
carries no mark, gives an unmarked artifact: a licence lifts the mark a
run would otherwise get for having no amplifier.
What you see: state["marked"] is True, and a pipeline that loads
or deploys the artifact runs marked – its writers mark their files and
its log says why – even where the run would otherwise be unmarked, on
a licensed machine. Nothing else changes: the run is not degraded or
refused.
Two things that catch people out
Writers timestamp their filenames.
gp.CsvWriter(file_name="rec.csv") writes something like
rec_20260907_110441_205.csv, so a recording is never silently
overwritten. A script that asserts on the configured path will report a
working pipeline as broken.
Stop, then close. stop() releases threads, file handles and
hardware; close() releases logging and the master timeline. A
pipeline can be started again after stop(), which is why the two are
separate — or use with, as above, and neither is your problem.