Episode 5: Choosing a File Format
Write one recording to CSV, EDF, HDF5 and MATLAB at once, read each back, and know what every format keeps and what it costs.
Episode 1 recorded to CSV, because CSV is the format you can open with anything. It is not always the one you want. g.Pype writes four, and they differ in what metadata survives, how large the file is, and which program opens it without an import step.
A Generator drives four channels, and a ChannelLabeler names them, so there is metadata for the formats to differ about. Without it, every format writes Ch01 to Ch04.
29 p = gp.Pipeline()
30 source = gp.Generator(sampling_rate=SAMPLING_RATE,
31 channel_count=len(ELECTRODES),
32 signal_frequency=10, signal_amplitude=50.0,
33 noise_amplitude=5.0)
34
35 # Name the channels first, so there is metadata for the formats to
36 # differ about; without this every format writes Ch01..Ch04
37 labeler = gp.ChannelLabeler(labels=ELECTRODES)
38 p.connect(source, labeler)
A CsvWriter, an EDFWriter, an HDF5Writer and a MatWriter, all fed from the labeler. Each adds the date and time to the name it is given, so the script finds its files by pattern afterwards.
40 # The same stream into four writers
41 writers = {
42 ".csv": gp.CsvWriter(file_name=os.path.join(OUT, "s4e5.csv")),
43 ".edf": gp.EDFWriter(file_name=os.path.join(OUT, "s4e5.edf")),
44 ".h5": gp.HDF5Writer(file_name=os.path.join(OUT, "s4e5.h5")),
45 ".mat": gp.MatWriter(file_name=os.path.join(OUT, "s4e5.mat")),
46 }
47 for writer in writers.values():
48 p.connect(labeler, writer)
A Collector beside the writers keeps in memory what they were given. The pipeline runs for four seconds and stops.
50 # What the writers were given, kept in memory to compare against
51 recorded = gp.Collector(name="recorded")
52 p.connect(labeler, recorded)
53
54 p.start()
55 time.sleep(SECONDS)
56 p.stop()
57 p.close()
Every format has a reader, and mode="batch" hands a whole file back from one run(), as Season 2, Episode 16 showed. CsvReader and EDFReader take the same arguments. HDF5Reader and MatReader take two more: variable_name names the array inside the file, and has_time_row says whether its first row is a time axis rather than a channel. For a file g.Pype wrote, a wrong value is a read failure; a file from elsewhere carries nothing to check against, so check the channel count and the rate after reading one.
67 # Every format has a reader, and mode="batch" hands the whole file
68 # back from one run(). HDF5Reader and MatReader take two arguments
69 # the other two do not: variable_name names the array inside the
70 # file, and has_time_row says its first row is a time axis
71 readers = {
72 ".csv": lambda path: gp.CsvReader(file_name=path, mode="batch"),
73 ".edf": lambda path: gp.EDFReader(file_name=path, mode="batch"),
74 ".h5": lambda path: gp.HDF5Reader(file_name=path, mode="batch",
75 variable_name="data",
76 has_time_row=True),
77 ".mat": lambda path: gp.MatReader(file_name=path, mode="batch",
78 variable_name="data",
79 has_time_row=True),
80 }
81 reference = recorded.result.data
82 print()
83 print("Read back, against what the writers were given:")
84 for extension, reader_for in readers.items():
85 with gp.Pipeline() as back:
86 back.connect(reader_for(written(extension)), gp.Collector())
87 result = back.run()
88 n = min(result.sample_count, len(reference))
89 worst = np.max(np.abs(result.data[:n] - reference[:n]))
90 print(f" {extension:<5} largest difference {worst:<8.3g} uV "
91 f"labels {', '.join(result.labels)}")
Run the script. It prints each file’s size, then the largest difference between what was read back and what the writers were given, with the labels each file kept:
4 s of 4 channels at 250 Hz, written four ways:
.csv ... bytes s4e5_<date>_<time>.csv
.edf ... bytes s4e5_<date>_<time>.edf
.h5 ... bytes s4e5_<date>_<time>.h5
.mat ... bytes s4e5_<date>_<time>.mat
Read back, against what the writers were given:
.csv largest difference ... labels Fz, C3, Cz, C4
...
The sizes follow from how each format stores a value: CSV as text at 17 significant digits, about 20 characters a value; EDF as a 16-bit integer, 2 bytes; HDF5 and MATLAB as a 64-bit float, 8 bytes. So CSV is the largest and EDF the smallest, by a wide margin. HDF5, MATLAB and CSV read back exactly; EDF reads back less than one quantisation step off. Every file kept the labels. If the EDF writer printed a UserWarning from pyedflib about an invalid character, that is expected: g.Pype stamps the non-commercial mark into a header field the EDF specification allows no spaces in. The mark is written and survives; the warning is cosmetic.
CSV is text, so it is the only one you can open in an editor or a spreadsheet with nothing installed. Use it for a short recording someone else has to look at without your toolchain. It stores no channel roles, so a CsvReader reports every channel as a signal; here all four are, but a trigger channel would come back as a signal too, and a filter downstream would then smooth it.
EDF is the clinical interchange format, the one every EEG viewer already reads. It stores samples as 16-bit integers scaled between a physical minimum and maximum, which is what the writer’s physical_min and physical_max are for, and why a signal outside that range is clipped. At the default range of plus or minus 10000 µV one step is 0.305 µV. That quantisation is the price of the compatibility.
HDF5 keeps the array and its metadata, labels and roles included. Reach for it when the recording is large, or when you want to read part of it without loading all of it. Python, MATLAB and R all read it.
MATLAB (.mat) is HDF5 underneath, written so that load() hands the array back under variable_name. Use it when the analysis is already in MATLAB.
In this episode, you learned how to:
write one stream to four formats at once, and find the files afterwards
read each format back in a batch run, and what HDF5Reader and MatReader need to be told
choose a format by what it keeps and what it costs
Proceed to Episode 6, where a recording becomes a source: replayed in real time, or run offline in one go.
Nodes in this episode: Generator, ChannelLabeler, CsvWriter, EDFWriter, HDF5Writer, MatWriter, Collector, CsvReader, EDFReader, HDF5Reader, MatReader
File s4e5_choosing_a_file_format.py – View file on GitHub
1"""Season 4, Episode 5: one recording written four ways, and read back.
2
3The files are left in the temporary folder, as s4e5_<date>_<time>.*
4"""
5
6import glob
7import os
8import tempfile
9import time
10
11import numpy as np
12
13import gpype as gp
14
15SAMPLING_RATE = 250
16SECONDS = 4.0
17ELECTRODES = ["Fz", "C3", "Cz", "C4"]
18OUT = tempfile.gettempdir()
19
20
21def written(extension: str) -> str:
22 """The file a writer produced last; writers timestamp their names."""
23 found = sorted(glob.glob(os.path.join(OUT, "s4e5_*" + extension)))
24 return found[-1] if found else ""
25
26
27if __name__ == "__main__":
28
29 p = gp.Pipeline()
30 source = gp.Generator(sampling_rate=SAMPLING_RATE,
31 channel_count=len(ELECTRODES),
32 signal_frequency=10, signal_amplitude=50.0,
33 noise_amplitude=5.0)
34
35 # Name the channels first, so there is metadata for the formats to
36 # differ about; without this every format writes Ch01..Ch04
37 labeler = gp.ChannelLabeler(labels=ELECTRODES)
38 p.connect(source, labeler)
39
40 # The same stream into four writers
41 writers = {
42 ".csv": gp.CsvWriter(file_name=os.path.join(OUT, "s4e5.csv")),
43 ".edf": gp.EDFWriter(file_name=os.path.join(OUT, "s4e5.edf")),
44 ".h5": gp.HDF5Writer(file_name=os.path.join(OUT, "s4e5.h5")),
45 ".mat": gp.MatWriter(file_name=os.path.join(OUT, "s4e5.mat")),
46 }
47 for writer in writers.values():
48 p.connect(labeler, writer)
49
50 # What the writers were given, kept in memory to compare against
51 recorded = gp.Collector(name="recorded")
52 p.connect(labeler, recorded)
53
54 p.start()
55 time.sleep(SECONDS)
56 p.stop()
57 p.close()
58
59 print(f"{SECONDS:.0f} s of {len(ELECTRODES)} channels at "
60 f"{SAMPLING_RATE} Hz, written four ways:")
61 for extension in writers:
62 path = written(extension)
63 size = os.path.getsize(path) if path else 0
64 print(f" {extension:<5} {size:>9,d} bytes "
65 f"{os.path.basename(path) or '(not written)'}")
66
67 # Every format has a reader, and mode="batch" hands the whole file
68 # back from one run(). HDF5Reader and MatReader take two arguments
69 # the other two do not: variable_name names the array inside the
70 # file, and has_time_row says its first row is a time axis
71 readers = {
72 ".csv": lambda path: gp.CsvReader(file_name=path, mode="batch"),
73 ".edf": lambda path: gp.EDFReader(file_name=path, mode="batch"),
74 ".h5": lambda path: gp.HDF5Reader(file_name=path, mode="batch",
75 variable_name="data",
76 has_time_row=True),
77 ".mat": lambda path: gp.MatReader(file_name=path, mode="batch",
78 variable_name="data",
79 has_time_row=True),
80 }
81 reference = recorded.result.data
82 print()
83 print("Read back, against what the writers were given:")
84 for extension, reader_for in readers.items():
85 with gp.Pipeline() as back:
86 back.connect(reader_for(written(extension)), gp.Collector())
87 result = back.run()
88 n = min(result.sample_count, len(reference))
89 worst = np.max(np.abs(result.data[:n] - reference[:n]))
90 print(f" {extension:<5} largest difference {worst:<8.3g} uV "
91 f"labels {', '.join(result.labels)}")
92 print()
93 print("Files left in:", OUT)