🧩 Developer Notes

Author: Ifra Ilyas Ansari
Created: 2025-10-15
Updated: 2025-11-11

NeuroEng Voltage Transient Processor

This repository contains a set of Python scripts for processing, analyzing, and plotting voltage transient data from Tektronix oscilloscopes. The pipeline is broken into modular stages to handle pulse detection, data alignment, voltage extraction, and metadata merging.

🚀 Before Use

⚙️ Dependencies

This pipeline requires the following Python libraries. You can install them using pip:

• pandas
• numpy
• matplotlib
• rich (for optional folder tree previews)
• openpyxl (for reading Metadata.xlsx)

🛠️ Installation

Tested on Python 3.10–3.11 (macOS/Windows).

# (recommended) create a virtual environment
python -m venv .venv && source .venv/bin/activate   # on Windows: .venv\Scripts\activate

# install dependencies
pip install -r requirements.txt

---

📜 Code Versions

• Stages:
  • VT_Processor_Py_Stage1_PulseDetector: v3.6.4
  • VT_Processor_Py_Stage2_VoltageSelector: v1.7.0
  • VT_Processor_Py_BatchPlotter: v1.2.3
• Modules:
  • VT_Module_Py_GlobalSettings: v1.1.0
  • VT_Module_Py_System: v1.0.2
  • VT_Module_Py_DataIO: v1.2.0
  • VT_Module_Py_Processing: v1.0.0
  • VT_Module_Py_PlotHelpers: v1.0.1
  • VT_Module_Py_UserInput: v1.0.1

---

🚀 Quick Start / How to Use

1. Prepare Input Files:
   • Place all your raw Tektronix .CSV files (e.g., TEK00001.CSV, TEK00002.CSV) in a single data folder.
   • In that same folder, create and populate a Metadata.xlsx file. (See "Input File Formats" below for details).
2. Run Stage 1:
   • Execute VT_Processor_Py_Stage1_PulseDetector.py.
   • When prompted, select all the .CSV files from your data folder.
   • This stage will auto-detect pulses, correct baseline drift, and save a ...wAutoOffset.pkl file inside a new VT_Py_Outputs subfolder.
3. Run Stage 2:
   • Execute VT_Processor_Py_Stage2_VoltageSelector.py.
   • When prompted, select the PyPD_Dictionary_AllDataFiles_wAutoOffset.pkl file generated by Stage 1.
   • This stage will read your Metadata.xlsx snapshot to auto-calculate time points (t1-t5).
4. Interactive Validation:
   • For each file, Stage 2 will show you a plot with the auto-detected points.
   • In your terminal, press [Enter] to accept the points or type in 5 new comma-separated values to adjust them.
   • Close the plot window to proceed to the next file.
5. Get Results:
   • Once complete, find your final, merged data in .../VT_Py_Outputs/VT_Py_Outputs_VoltageSelector.../PyVS_VoltageSummary_Final.CSV.

---

Workflow Overview

This pipeline processes voltage transient data in two main stages, with an optional plotting utility.

• Stage 1: Pulse Detector (VT_Processor_Py_Stage1_PulseDetector.py)

  • Input: Raw TEK...csv files.
  • Action:
    1. Loads and standardizes each CSV.
    2. Auto-detects the primary biphasic pulse in the current trace.
    3. Calculates and applies a baseline correction (Y-offset) based on the pre-pulse region.
    4. Aligns all pulses to a common start time (X-offset).
    5. Finds and saves a snapshot of Metadata.xlsx from the same folder.
  • Output: A .pkl dictionary (PyPD_Dictionary_AllDataFiles_wAutoOffset.pkl) containing a clean, aligned DataFrame for each file.

• Stage 2: Voltage Selector (VT_Processor_Py_Stage2_VoltageSelector.py)

  • Input: The .pkl file from Stage 1 and the PyPD_Metadata_Snapshot.CSV it created.
  • Action:
    1. Reads PhaseWidth(µs) and InterphaseDelay(µs) from the metadata.
    2. Uses these values to auto-calculate 5 time points (t1-t5) for voltage extraction.
    3. Enters an interactive loop for the user to validate or manually correct these 5 points for each trace.
    4. Extracts the final V/I values at the validated points (with a special local-minimum search for V3).
  • Output: The final summary table (PyVS_VoltageSummary_Final.CSV), which merges your original metadata with all the new calculated voltage values (V1-V5, ΔV, etc.).

• Utility: Batch Plotter (VT_Processor_Py_BatchPlotter_v1.2.3.py)

  • This is a separate tool for creating comparison plots. It loads raw CSVs, allows you to apply manual time offsets (via the terminal), and overlays all traces on a single set of V/I plots.

---

📦 Input File Formats

1. Raw Data (TEK000xx.csv)

These are the raw data files exported directly from the Tektronix oscilloscope.

• Header: The script assumes a standard Tektronix header and skips the first 15 rows by default (see skip_rows=15 in Stage1).
• Columns: The script does not rely on fixed column positions. Instead, it auto-detects the necessary columns by searching for common names (aliases).
  • Time: Searches for Time(s), Time (µs), T (ms), etc., and converts to microseconds (µs).
  • Voltage: Searches for Voltage(V), V, CH1, etc.
  • Current: Searches for Current(A), I, CH2, etc.
• Channel Map: The default mapping is VOLTAGE: "CH1" and CURRENT: "CH2". This can be changed in VT_Module_Py_GlobalSettings.py.

2. Metadata (Metadata.xlsx)

This single Excel file must be in the same directory as the raw .CSV files. It provides experimental context and parameters for processing.

• FileID (Required): This column is the join key. It must contain the base filename of the corresponding data file (e.g., TEK00001, TEK00002).
• Stage 2 Calculation Columns (Required):
  • PhaseWidth(µs): User-defined. This value is read by Stage 2 to auto-calculate the initial t3 (Vc peak) and t4 (Emc) time points.
  • InterphaseDelay(µs): User-defined. This value is read by Stage 2 to auto-calculate the initial t5 (E IPend) time point.
• User-Defined Info Columns (Recommended):
  • These columns are read, ignored during calculation, and merged directly into the final PyVS_VoltageSummary_Final.CSV for complete documentation.
  • Examples: Date, WaferID, DeviceID, ElectrodeID, Electrolyte, Current(µA), etc.
• TotalPulses (Informational):
  • This is a user-defined column. You should enter the total number of pulses for your reference (e.g., from a pulse counter).
  • The scripts do not use this value for any calculations. It is simply loaded and carried through to the final summary file for your records.

⚙️ Configuration

All scripts import their settings from the VT_Module_... files. The most important one is VT_Module_Py_GlobalSettings.py. If your data is not loading or your plots look wrong, you probably need to edit this file:

• To swap Voltage/Current channels: Change DICT_TEK_CHANNEL_MAP.

  # e.g., if Voltage is on CH2 and Current is on CH1
  DICT_TEK_CHANNEL_MAP = {
      "VOLTAGE": "CH2",
      "CURRENT": "CH1",
  }

• To change plot appearance: Edit the rcParams.update({...}) dictionary to change fonts, line widths, and figure sizes.

---

📊 Key Output Files

The entire pipeline is designed to generate one primary file:

• PyVS_VoltageSummary_Final.CSV
  • Location: .../VT_Py_Outputs/VT_Py_Outputs_VoltageSelector_[timestamp]/
  • Contents: This is the main result. It is a single CSV file containing all columns from your original Metadata.xlsx plus all the new calculated voltage columns (e.g., Eipp (V1), Vc peak (V3), ΔV (V3-V1), etc.) for every processed file.

Other important intermediate files include:

• PyPD_Dictionary_AllDataFiles_wAutoOffset.pkl (from Stage 1)
  • This is the primary input for Stage 2. It's a Python pickle file containing a dictionary where each key is a FileID and each value is a fully cleaned, baseline-corrected, and time-aligned DataFrame.
• PyPD_Metadata_Snapshot.CSV (from Stage 1)
  • This is a snapshot of your Metadata.xlsx file, saved into the Stage 1 output folder. Stage 2 reads this file (not your original Excel) to ensure reproducibility.

---

Summary: Repository Tree

NeuroEng-Voltage-Transient-Processor/
├── README.md
├── LICENSE.txt
│
├── VT_Processor.html
├── __init__.py
│
├── VT_Processor_Py_Stage1_PulseDetector.py
├── VT_Processor_Py_Stage2_VoltageSelector.py
├── VT_Processor_Py_BatchPlotter.py
│
├── VT_Module_Py_GlobalSettings.py
├── VT_Module_Py_Paths.py
├── VT_Module_Py_Utilities.py
├── VT_Module_Py_PulseAlgorithm.py
├── VT_Module_Py_PlotHelpers.py
│
├── test_global_settings.py
│
└── 📁 Example_Data_Files/
    ├── 📄 Metadata_Template.csv
    ├── 📄 Metadata.csv
    ├── 📄 TEK00001.CSV
    ├── 📄 TEK00002.CSV
    ├── 📄 TEK00003.CSV
    ├── 📄 TEK00004.CSV
    ├── 📄 TEK00005.CSV  ... TEK000xx.CSV
    └── 📁 VT_Py_Outputs/
        ├── 📁 VT_Py_Outputs_PulseDetector_YYYYMMDD_HHMM/
        ├── 📁 VT_Py_Outputs_VoltageSelector_YYYYMMDD_HHMM/
        └── 📁 VT_Py_Outputs_BatchPlotter_YYYYMMDD_HHMM/ 
    └── 📁 VT_Py_Outputs/
        ├── 📁 VT_Py_Outputs_PulseDetector_YYYYMMDD_HHMM
        │   ├── 📄 PyPD_Summary.txt
        │   ├── 📄 PyPD_Summary.json
        │   ├── 📄 PyPD_Dictionary_AllDataFiles_wAutoOffset.pkl
        │   │
        │   └── 📁 PyPD_ProcessedDataFiles/
        │       ├── 📁 PyPD_CSV_Files/
        │       │   ├── 📄 PyPD_TEK0001_AutoOffset.csv
        │       │   ├── 📄 PyPD_TEK0002_AutoOffset.csv
        │       │   └── ...
        │       │
        │       ├── 📁 PyPD_PNG_Files/
        │       │   ├── 🖼️ PyPD_TEK0001_AutoOffset.png
        │       │   ├── 🖼️ PyPD_TEK0002_AutoOffset.png
        │       │   └── ...
        │       │
        │       └── 📁 PyPD_ExtraInfo_Files/
        │           ├── 📄 PyPD_Dictionary_AllDataFiles_wAutoDetected.pkl
        │           ├── 📄 PyPD_Dictionary_AllDataFiles_wRawTimeµs.pkl
        │           ├── 📄 PyPD_TEK0001_RawTimeµs.csv
        │           ├── 📄 PyPD_TEK0001_Detected.csv
        │           └── ...
        │
        ├── 📁 VT_Py_Outputs_VoltageSelector_YYYYMMDD_HHMM
        │   ├── 📄 PyVS_VoltageSummary_Final.CSV
        │   ├── 📄 PyVS_Dictionary_AllDataFiles_SelectedVals.pkl
        │   ├── 📄 PyVS_Summary.txt
        │   ├── 📄 PyVS_Summary.json
        │   │
        │   └── 📁 PyVS_ProcessedDataFiles/
        │       ├── 📁 PyVS_CSV_Files/
        │       │   ├── 📄 PyVS_TEK0001_SelectedVals.csv
        │       │   ├── 📄 PyVS_Metadata_Input.CSV
        │       │   ├── 📄 PyVS_VoltageSummary_CalculatedVals.CSV
        │       │   └── ...
        │       │
        │       └── 📁 PyVS_PNG_Files/
        │           ├── 🖼️ PyVS_TEK0001_SelectedVals_wOverlay.png
        │           ├── 🖼️ PyVS_TEK0002_SelectedVals_wOverlay.png
        │           └── ...
        │
        └── 📁 VT_Py_Outputs_BatchPlotter_YYYYMMDD_HHMM
            ├── 📄 PyBP_Summary.txt
            ├── 📄 PyBP_Summary.json
            ├── 📄 PyBP_Dictionary_AllDataFiles_Processed.pkl
            │
            └── 📁 PyBP_ProcessedDataFiles/
                ├── 📁 PyBP_CSV_Files/
                │   ├── 📄 PyBP_TEK00001_UserOffset.csv
                │   ├── 📄 PyBP_TEK00002_Raw.csv
                │   └── ...
                │
                ├── 📁 PyBP_PNG_Files/
                │   ├── 🖼️ PyBP_Current_wBatchPlot_{custom_suffix}.png
                │   ├── 🖼️ PyBP_Voltage_wBatchPlot_{custom_suffix}.png
                │   └── ...
                │
                └── 📁 PyBP_SVG_Files/
                    ├── 📄 PyBP_Current_wBatchPlot_{custom_suffix}.svg
                    ├── 📄 PyBP_Voltage_wBatchPlot_{custom_suffix}.svg
                    └── ...

---