Market Data Collector & ML Training Pipeline

This project consists of two main components:

1. Market Data Collector (Go) — Streams and stores market data frames
2. Machine Learning Pipeline (Python) — Trains models on collected data

---

Prerequisites

• Go (latest stable recommended)
• Python 3.13+

---

1. Running the Data Collector

The collector gathers market data and stores it as sequential frames.

Steps

go run main.go

Output

Data is written to:

data/frames.jsonl

Notes

• Let the collector run for a sufficient amount of time to accumulate long sequences per ticker
• Longer sequences generally improve model training quality

---

2. Training the Machine Learning Model

Once you have collected enough data, you can train the model.

Command

python ML/train.py --frames ./data/frames.jsonl --epochs 30 --arch {fast, full, auto}

Parameters

• --frames
  Path to the collected data file (.jsonl)

• --epochs
  Number of training epochs (e.g., 30)

• --arch
  Model architecture:

  • fast — Lightweight, quicker training
  • full — More complex, higher performance
  • auto — Automatically selects architecture

---

Workflow

1. Start the collector:

   go run main.go

2. Let it run until sufficient data is collected

3. Train the model:

   python ML/train.py --frames ./data/frames.jsonl --epochs 30 --arch {auto, fast, full}

---

Important

• Ensure frames.jsonl is non-trivial in size before training
• Start with --arch fast for quick iteration
• Use --arch full once data volume is sufficient