[Buildathon] Add Card et al. (2025) dataset (Brain-to-Text 2025)

[nandahkrishna]

This PR adds the Card et al. (2025) dataset used for the Brain-to-Text 2025 challenge. This PR is a work in progress and should only be merged (post rebasing) after #49 is merged.

[Copilot]

Pull Request Overview

This PR adds two new brain-to-text datasets to the brainsets pipeline: the Willett & Henderson (2023) speech dataset and the Card & Brandman (2025) accurate decoding dataset (for the Brain-to-Text 2025 challenge). Both pipelines follow the established pattern of downloading data, processing it into a standardized format, and saving it as HDF5 files with train/validation/test splits.

Key Changes

• Added processing pipeline for Willett & Henderson (2023) speech dataset with 128 electrode arrays and phoneme annotations
• Added processing pipeline for Card & Brandman (2025) dataset with 256 electrode arrays
• Both pipelines include Snakefiles for automated data download and processing workflows

Reviewed Changes

Copilot reviewed 4 out of 6 changed files in this pull request and generated 19 comments.

Show a summary per file

File	Description
brainsets_pipelines/willett_henderson_speech_2023/prepare_data.py	Processes speech data from .mat files, extracting spike trains, transcripts, and phoneme sequences with train/valid splits
brainsets_pipelines/willett_henderson_speech_2023/Snakefile	Automates downloading and processing of Willett dataset from DataDryad
brainsets_pipelines/willett_henderson_speech_2023/requirements.txt	Empty requirements file for the Willett pipeline
brainsets_pipelines/card_brandman_accurate_2025/prepare_data.py	Processes HDF5 neural activity files into standardized format with train/val/test splits
brainsets_pipelines/card_brandman_accurate_2025/Snakefile	Workflow for downloading and processing Card dataset (requires URL configuration)
brainsets_pipelines/card_brandman_accurate_2025/requirements.txt	Empty requirements file for the Card pipeline

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Empty source and description fields should be populated with the actual dataset source URL and description. These fields are important for dataset documentation and provenance tracking.

Suggested change

	source="",
	description="",
	source="https://github.com/ChangLabUcsf/accurate2025", # Example: replace with actual dataset URL if different
	description="Utah array threshold crossing recordings from human participant T15 performing continuous sentence speaking tasks, as described in Brandman et al. (2025).",

[Copilot]

The result of .encode("ascii", errors="ignore") is a bytes object, but the transcript array likely expects strings. This will cause type inconsistency. Consider using .decode("ascii") after encoding, or using a different approach to handle non-ASCII characters (e.g., .encode("ascii", errors="ignore").decode("ascii")).

Suggested change

	transcript = ["".join(chr(c) for c in s if c != 0).replace(u"\u2019", "'").encode("ascii", errors="ignore") for s in transcripts]
	transcript = ["".join(chr(c) for c in s if c != 0).replace(u"\u2019", "'").encode("ascii", errors="ignore").decode("ascii") for s in transcripts]

[Copilot]

Using np.arange(1/FREQ, 1/FREQ * x.shape[0] + 1e-6, 1/FREQ) may not produce exactly x.shape[0] timestamps due to floating-point precision issues. Consider using np.arange(x.shape[0]) / FREQ + 1/FREQ or (np.arange(x.shape[0]) + 0.5) / FREQ for more reliable timestamp generation.

Suggested change

	[np.arange(1/FREQ, 1/FREQ * x.shape[0] + 1e-6, 1/FREQ) + trial_bounds[i] for i, x in enumerate(activity)]
	[(np.arange(x.shape[0]) + 0.5) / FREQ + trial_bounds[i] for i, x in enumerate(activity)]

[Copilot]

[nitpick] The dataset ID is "card_brandman_accurate_2025" but the PR title references "Card et al. (2025)". The naming pattern should be consistent. Consider verifying whether the dataset ID should include "brandman" or if it should match the citation pattern more closely (e.g., "card_et_al_2025").

Suggested change

	id="card_brandman_accurate_2025",
	id="card_et_al_2025",

[Copilot]

Empty REMOTE_URL will cause wget to fail. This needs to be populated with the actual URL for the dataset download.

Suggested change

	REMOTE_URL = ""
	# TODO: Set REMOTE_URL to the actual dataset download URL
	REMOTE_URL = "https://example.com/path/to/competitionData.tar.gz"

[Copilot]

Mode 1 performs no normalization (subtracts 0 and divides by 1). This appears to be a no-op and may be unintentional. Consider removing this mode or implementing proper normalization logic.

Suggested change

	if mode == 1:
	spikes = (spikes - 0.) / 1.
	elif mode == 2:
	for idx in np.where(unique_block_idx == i)[0]:
	spikes[idx] = (spikes[idx] - mean) / std
	elif mode == 3:
	for idx in np.where(blockIdx == i)[0]:
	spikes[idx] = (spikes[idx] - mean) / std
	if mode == 2:
	for idx in np.where(unique_block_idx == i)[0]:
	spikes[idx] = (spikes[idx] - mean) / std
	elif mode == 3:
	for idx in np.where(blockIdx == i)[0]:
	spikes[idx] = (spikes[idx] - mean) / std
	else:
	raise ValueError(f"Unsupported normalization mode: {mode}")

[Copilot]

The return type annotation indicates Tuple[np.ndarray, np.ndarray] but the function actually returns 4 values: spike_timestamps, spike_ids, spike_counts, spike_power. The type annotation should be Tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray].

Suggested change

	) -> Tuple[np.ndarray, np.ndarray]:
	) -> Tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray]:

[Copilot]

Import of 're' is not used.

Suggested change

import re

[Copilot]

Import of 'Tuple' is not used.

Suggested change

from typing import Tuple

[Copilot]

Import of 'sio' is not used.

Suggested change

import scipy.io as sio