Preprocessing Functions

See CHANGELOG.md for user-facing release notes.

1. preprocess_sirenia
2. preprocess_sleep_data

Which One to Use? preprocess_sirenia.m is used for extracting signals from the Pinnacle Sirenia equipment, optionally synchronizing them with Tucker-Davis Technologies (TDT) Synapse photometry data, and then preparing them to be used in the Sleep Scoring App.

preprocess_sleep_data.m is used for extracting and synchronizing signals from the Tucker-Davis Technologies (TDT) Synapse equipment and/or the Viewpoint SleepScore equipment then preparing them to be used in the Sleep Scoring App.

preprocess_sirenia

Usage example

First, make sure you "import" the preprocess_sleep_data folder and its helper functions, the EEGtoolbox folder. If the preprocess_sleep_data folder is not in your current directory. You could use absolute path in addpath. For example,

addpath('C:\Users\yzhao\matlab_projects\preprocess_sleep_data')
addpath('C:\Users\yzhao\matlab_projects\preprocess_sleep_data\EEGtoolbox')

Assuming you placed the .edf file from Sirenia in the same folder as the TDT data, you can call the function (in a new script) by providing the path to the TDT folder like this

preprocess_sirenia('C:\Users\yzhao\matlab_projects\pinnacle\Isabelle_M1_Fi2PostICH24h_M2_Fi1PostICH1-3h');

If you placed the .edf file elsewhere, you need to provide its path by adding the optional argument edf_file, like this:

preprocess_sirenia( ...
    'C:\Users\yzhao\matlab_projects\pinnacle\Isabelle_M1_Fi2PostICH24h_M2_Fi1PostICH1-3h', ...
    'edf_file', 'C:\Users\yzhao\matlab_projects\pinnacle\Isabelle_M1_Fi2PostICH24h_M2_Fi1PostICH1-3h\M1_Fi2PostICH24hM2_Fi1PostICH1-3h.edf' ...
);

If you only have the Sirenia .edf file and no TDT photometry, pass the .edf file path directly:

preprocess_sirenia('C:\Users\yzhao\matlab_projects\sleep_data\Default_2026-05-17_10_18_13_export.edf');

When no TDT block is provided or detected, the output still includes ne and ne_frequency, saved as ne = [] and ne_frequency = NaN.

The output also includes recording_start_time, an ISO-8601 character string for the first EEG/EMG sample saved in the MAT file. Sirenia timestamps come from the EDF header and preserve its source clock without inferring a timezone. If interval trims the beginning of the recording, the saved timestamp is advanced to the first retained sample.

Below is a list of other optional arguments that you may need to specify.

1. eeg_col, defaults to 'EEGEEG1A_B'. Column name for the EEG signal in Sirenia.
2. emg_col, defaults to 'EMGEMG'. Column name for the EMG signal in Sirenia.
3. chan_465, defaults to 'x465A'. Channel name for signal 465 in TDT.
4. chan_405, defaults to 'x405A'. Channel name for signal 405 in TDT.
5. chan_ttl_pulse, defaults to 'PC0_'. Channel name for the TTL pulse in TDT.
6. interval, defaults to []. By default, the entire recording after the TTL synchronization will be extracted. If needed you may supply a start and end time (in seconds) array like [10, 500]. To supply only a start time, write Inf as the end time, e.g., [10, Inf]. If you only want an end time, set the the start time to 0, for example, [0, 500], meaning the first 500 seconds.
7. save_path, defaults to a .mat file with the same name as the .edf file but saved to its parent folder. Where and what to name the .mat file that will be saved.
8. show_figure, defaults to false. Whether or not to show the plots of the preprocessed data.

preprocess_sleep_data

Usage example

First, make sure you "import" the preprocess_sleep_data folder and its helper functions, the EEGtoolbox folder. If the preprocess_sleep_data folder is not in your current directory. You could use absolute path in addpath. For example,

addpath('C:\Users\yzhao\matlab_projects\preprocess_sleep_data')
addpath('C:\Users\yzhao\matlab_projects\preprocess_sleep_data\EEGtoolbox')

After you have taken care of the "import", you can call the function like this

preprocess_sleep_data( ...
    'C:\Users\yzhao\matlab_projects\sleep_data_extraction\opto_NOR\408_yfp\408_YFP_NOR\408_YFP_NOR_2021-03-08_10-29-08-223.exp', ...
    'ne_dir', 'C:\Users\yzhao\matlab_projects\sleep_data_extraction\opto_NOR\408_yfp', ...
    'chan_465', 'BuBo', ...
    'chan_405', 'V_Bo', ...
    'chan_ttl_pulse', 'Pu2_', ...
    'fit_interval', (100:7000), ...
    'time_correction', 2, ...
    'sleep_score_file', 'C:\Users\yzhao\matlab_projects\sleep_data_extraction\opto_NOR\408_yfp\408_YFP_NOR\408_NOR_sleepscore.xlsx', ...
    'save_path', '408_yfp.mat', ...
    'show_figure', true ...
);

The first parameter is the path to your EEG/EMG data. If your EEG/EMG data was recorded with Viewpoint, then this should be the path to the .exp file. If your EEG/EMG data was recorded using the TDT system, then this should be the directory of the TDT files here.

The rest of the parameters are optional name-value pairs. 'fit_interval' and 'time_correction' are always optional. The last parameter, 'show_figure' is for visualizing the preprocessed data. If you want to see it, set it to 'true'. If you don't need it, just skip it. You may or may not need to provide the other optional parameters depending on your data source. The order in which you write them does not matter. See the flowchart below for the parameters you need. Also check Section More Examples for example usage.

For single-bin, single-video Viewpoint recordings with no TDT photometry input, video_start_time is computed from the Viewpoint metadata start times. It is a signed offset in seconds such that video_time = eeg_time + video_start_time. Negative values mean the video starts after the EEG/EMG trace, so early EEG/EMG samples have no matching video. Positive values mean the video starts before the EEG/EMG trace, so EEG/EMG time zero maps later into the source video. The legacy 'video_alignment' argument is still accepted for compatibility, but it no longer controls single-bin Viewpoint alignment.

Output

The following fields are saved in the .mat file

1. eeg
2. emg
3. ne
4. sleep_scores
5. start_time (0 if only one .bin file or for 12-hour recordings or less. See Note below)
6. video_start_time (signed seconds to add to an EEG/EMG timestamp to find the matching video timestamp: video_time = eeg_time + video_start_time)
7. recording_start_time (ISO-8601 character string for the first EEG/EMG sample saved in this MAT file; TDT values are UTC and end in Z, while Viewpoint source clocks have no inferred timezone)
8. num_class (fixed to be 3 for now)
9. eeg_frequency
10. ne_frequency
11. video_name
12. video_path (path to .avi file if found in .exp file)

Note

1. ne is downsampled by a factor of 100 before being saved.
2. eeg, emg, and ne are 1D arrays.
3. Make sure that your ne_dir only contains one set of TDT files.
4. Long recordings. When you have EEG/EMG data from Viewpoint and you have multiple bin files, the data in each bin file will be written to a .mat file named after the bin file. When you have EEG/EMG from TDT and the recording is longer than 12 hours, your recording will be segmented into 12-hour bins and the resulting .mat files will be named with a suffix accordingly. The start_time will mark the start time of the segmented .mat file.

More Examples

1. EEG/EMG from .exp file, no NE data

preprocess_sleep_data( ...
    'C:\Users\yzhao\matlab_projects\sleep_data_extraction\opto_NOR\408_yfp\408_YFP_NOR\408_YFP_NOR_2021-03-08_10-29-08-223.exp', ...
);

2. EEG/EMG from .exp file + NE from TDT files.

preprocess_sleep_data( ...
    'C:\Users\yzhao\matlab_projects\sleep_data_extraction\opto_NOR\408_yfp\408_YFP_NOR\408_YFP_NOR_2021-03-08_10-29-08-223.exp', ...
    'ne_dir', 'C:\Users\yzhao\matlab_projects\sleep_data_extraction\opto_NOR\408_yfp', ...
    'chan_465', 'BuBo', ...
    'chan_405', 'V_Bo', ...
    'chan_ttl_pulse', 'Pu2_', ...
    'save_path', '408_yfp.mat', ...
    'show_figure', true ...
);

3. EEG/EMG + NE from TDT files.

preprocess_sleep_data( ...
    'C:\Users\yzhao\matlab_projects\sleep_data_extraction\830', ...
    'ne_dir', 'C:\Users\yzhao\matlab_projects\sleep_data_extraction\830', ...
    'EEG_stream', 'EEGw', ...
    'EEG_chan', 1, ...
    'EMG_stream', 'EMG1', ...
    'chan_465', 'x465C', ...
    'chan_405', 'x405C', ...
    'save_path', 'C:\Users\yzhao\matlab_projects\sleep_data_extraction\830.mat', ...
    'show_figure', true ...
);

4. EEG/EMG from TDT files, no NE.

preprocess_sleep_data( ...
    'C:\Users\yzhao\matlab_projects\sleep_data_extraction\20241113_1_263_2_259_24h_test', ...
    'EEG_stream', 'EEG1', ...
    'EEG_chan', 1, ...
    'EMG_stream', 'EMG1', ...
    'save_path', 'C:\Users\yzhao\matlab_projects\sleep_data_extraction\20241113_1_263_2_259_24h_test.mat', ...
    'show_figure', true ...
);

5. if you already have the sleep scores (this happens if you want to contribute training data for improving the sleep scoring models).

preprocess_sleep_data( ...
    'C:\Users\yzhao\matlab_projects\sleep_data_extraction\20221221_adra_1_238_2_242', ...
    'ne_dir', 'C:\Users\yzhao\matlab_projects\sleep_data_extraction\20221221_adra_1_238_2_242', ...
    'EEG_stream', 'EEGw', ...
    'EEG_chan', 1, ...
    'EMG_stream', 'EMG1', ...
    'chan_465', 'x465A', ...
    'chan_405', 'x405A', ...
    'fit_interval', (1:8000), ...
    'time_correction', 0, ...
    'sleep_score_file', 'C:\Users\yzhao\matlab_projects\sleep_data_extraction\20221221_adra_1_238_2_242\211222_238_adjusted_score.xlsx', ...
    'save_path', 'C:\Users\yzhao\matlab_projects\sleep_data_extraction\20221221_adra_1_238_2_242.mat', ...
    'show_figure', true ...
);