Skip to content

Commit 15c3d90

Browse files
feat: prepare to publish to pypi (#7)
1 parent 02e31c2 commit 15c3d90

69 files changed

Lines changed: 497 additions & 193 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.env.example

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
# Copy to .env and fill in values (optional for many runs):
22
# cp .env.example .env
33
#
4-
# On startup, `orchestrator.main` loads `.env` automatically (python-dotenv).
4+
# On startup, `resilient-stt` loads `.env` automatically (python-dotenv).
55
# Variables already set in your shell are not overwritten.
66

77
# --- Shortest full pipeline (OpenAI ASR + repair) ---
@@ -15,7 +15,7 @@
1515
# ASR_BASE_URL=https://openrouter.ai/api/v1
1616
# ASR_API_KEY=sk-or-...
1717
#
18-
# uv run python -m orchestrator.main --audio path/to/audio.mp3 --output data/output/run
18+
# resilient-stt --audio path/to/audio.mp3 --output data/output/run
1919
#
2020
# Local ASR is preferred when available. Hosted ASR auto-detection requires:
2121
# no --model, no ASR_BASE_URL/ASR_ENDPOINT, and no reachable local worker.

.github/workflows/publish.yml

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
name: Publish to PyPI
2+
3+
on:
4+
push:
5+
tags:
6+
- "v*"
7+
8+
permissions:
9+
id-token: write
10+
contents: read
11+
12+
jobs:
13+
publish:
14+
runs-on: ubuntu-latest
15+
steps:
16+
- uses: actions/checkout@v4
17+
18+
- name: Verify tag matches package version
19+
run: |
20+
TAG="${GITHUB_REF_NAME#v}"
21+
VERSION=$(python -c "import tomllib; print(tomllib.load(open('pyproject.toml','rb'))['project']['version'])")
22+
if [ "$TAG" != "$VERSION" ]; then
23+
echo "Tag v$TAG does not match pyproject.toml version $VERSION"
24+
exit 1
25+
fi
26+
27+
- uses: actions/setup-python@v5
28+
with:
29+
python-version: "3.12"
30+
31+
- name: Build sdist and wheel
32+
run: |
33+
python -m pip install --upgrade build
34+
python -m build
35+
36+
- name: Publish to PyPI
37+
uses: pypa/gh-action-pypi-publish@release/v1
Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
# Opens/updates a Release PR on main; merging it creates vX.Y.Z and triggers publish.yml.
2+
name: release-please
3+
4+
on:
5+
push:
6+
branches:
7+
- main
8+
workflow_dispatch:
9+
10+
permissions:
11+
contents: write
12+
pull-requests: write
13+
14+
jobs:
15+
release-please:
16+
runs-on: ubuntu-latest
17+
steps:
18+
- uses: googleapis/release-please-action@v4

.release-please-manifest.json

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
{
2+
".": "0.3.2"
3+
}

CHANGELOG.md

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
# Changelog
2+
3+
All notable changes to this project are documented in this file.
4+
Release versions and notes are managed by [release-please](https://github.com/googleapis/release-please)
5+
from [Conventional Commits](https://www.conventionalcommits.org/) on `main`.
6+
7+
## [0.3.2](https://github.com/gitcommitshow/resilient-stt/releases/tag/v0.3.2) (2026-05-25)
8+
9+
### Features
10+
11+
* PyPI packaging (`src/resilient_stt` layout), bundled qwen worker, and tag-based publish workflow

MANIFEST.in

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
graft src
2+
prune tests
3+
prune scripts
4+
global-exclude *.egg-info

README.md

Lines changed: 133 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,51 @@
1-
# Resilient STT (Speech-To-Text)
1+
# Resilient STT
22

3-
A local speech transcription pipeline for English, Hindi, and Hinglish audio.
4-
The orchestrator owns chunking, diarization, repair, and exports; ASR inference is **externalized** behind an OpenAI-compatible `/v1/audio/transcriptions` endpoint that you choose (vLLM, hosted APIs including OpenAI and Open Router API, faster-whisper wrapper, Parakeet wrapper, etc.).
3+
**The only Speech-To-Text pipeline you need.**
4+
5+
A provider-agnostic speech-to-text pipeline that plugs into **any OpenAI-compatible ASR endpoint** — OpenAI, OpenRouter, vLLM, faster-whisper, or the bundled Qwen worker — and returns rich, timestamped transcripts with speaker labels and validated metadata. Resilient STT handles everything around inference: audio preprocessing, voice-activity detection, intelligent chunking, pyannote diarization, and LLM-powered transcript repair. Tested on English, Hindi, and Hinglish but it is designed to support all the languages the ASR model of your choice supports.
6+
7+
## Features
8+
9+
- **Any ASR provider** — Point at any service that implements `POST /v1/audio/transcriptions` (or OpenRouter STT); swap models with flags, not code changes.
10+
- **Resilient ASR discovery** — Auto-detects local workers and hosted APIs, retries failed requests, and starts a bundled Qwen fallback when nothing else is reachable.
11+
- **Audio preprocessing** — ffmpeg normalization to mono 16 kHz WAV with optional enhancement before transcription.
12+
- **Voice activity detection** — Silero, webrtcvad, or RMS fallback to skip silence and focus compute on speech.
13+
- **Smart chunking & stitching** — Fixed or pause-aligned chunks for long files, with global word and segment timestamps stitched back together.
14+
- **Speaker diarization** — pyannote on the full audio, with word-level speaker assignment (IoU + segment fallback).
15+
- **LLM transcript repair** — Optional two-pass repair via any OpenAI-compatible chat endpoint, with validation to preserve timing and mixed-script Hinglish.
16+
- **Rich exports** — JSON, SRT, and VTT with segments, words, speakers, and pipeline metadata.
17+
- **Lightweight orchestrator** — No ASR model weights in the core package; inference stays in the microservice you choose.
18+
- **CLI and Python API**`resilient-stt` from the terminal, or `pipeline.run(JobConfig)` for programmatic use.
519

620
> **Experimental:** This project is under active development and is **not production-ready**. Expect breaking changes, incomplete features, and behavior that may shift between releases. Use for evaluation and prototyping only.
721
822
**Architecture & design decisions:** [docs/design.md](docs/design.md)
923

24+
**CLI reference:** [docs/cli.md](docs/cli.md)
25+
26+
## Repository layout
27+
28+
```text
29+
resilient-stt/
30+
src/resilient_stt/ # installable package (PyPI name: resilient-stt)
31+
orchestrator/ # CLI, pipeline, ASR discovery
32+
core/ # audio, VAD, chunking, schemas, exports
33+
asr/ # OpenAI-compatible ASR client, fallback worker
34+
diarization/ # pyannote + speaker assignment
35+
alignment/ # optional aligner (stubs in v1)
36+
repair/ # LLM transcript repair
37+
workers/ # bundled qwen-asr HTTP service (shipped in wheel)
38+
workers/ # docs for optional ASR workers (vLLM, Whisper, …)
39+
scripts/ # bootstrap helpers (not installed from PyPI)
40+
tests/
41+
docs/
42+
data/ # input / work / output (local runs)
43+
```
44+
45+
- **PyPI / pip:** `pip install resilient-stt` → CLI command **`resilient-stt`** (hyphen).
46+
- **Python imports:** `resilient_stt` (underscore), e.g. `from resilient_stt.orchestrator.pipeline import run`.
47+
- **From source:** `uv sync` then `uv run resilient-stt …` (or activate `.venv` and run `resilient-stt`).
48+
1049
## Architecture
1150

1251
```
@@ -87,7 +126,7 @@ The orchestrator sends 16 kHz mono WAV chunks.
87126

88127
| Worker | Endpoint (default) | Example `--model` | Status |
89128
| ------ | ------------------ | ----------------- | ------ |
90-
| [qwen_transformers_service](workers/qwen_transformers_service/README.md) | `http://127.0.0.1:8002/v1` | `Qwen/Qwen3-ASR-0.6B` | **Bundled** — auto-started when no other ASR is reachable |
129+
| [qwen_transformers_service](src/resilient_stt/workers/qwen_transformers_service/README.md) | `http://127.0.0.1:8002/v1` | `Qwen/Qwen3-ASR-0.6B` | **Bundled** — auto-started when no other ASR is reachable |
91130
| [qwen_vllm_service](workers/qwen_vllm_service/README.md) | `http://127.0.0.1:8001/v1` | `Qwen/Qwen3-ASR-1.7B` | **Bundled bootstrap** — Linux + NVIDIA GPU |
92131
| Custom (faster-whisper, vLLM, hosted proxy, etc.) | Your URL | Your model id | **Supported** — implement or deploy separately |
93132
| [whisper_openai_service](workers/whisper_openai_service/README.md) ||| **Planned Roadmap** |
@@ -121,13 +160,74 @@ Any OpenAI-compatible **`/chat/completions`** endpoint. Set `REPAIR_BASE_URL`,
121160

122161
| Component | Status |
123162
| --------- | ------ |
124-
| Qwen forced aligner (`alignment/qwen_aligner.py`) | **Planned Roadmap** |
163+
| Qwen forced aligner (`src/resilient_stt/alignment/qwen_aligner.py`) | **Planned Roadmap** |
125164
| WhisperX | **Planned Roadmap** |
126165

127166
Today, alignment runs only when `--align` is set or ASR returned weak
128167
timestamps; the default aligner is a no-op pass-through.
129168

130-
## Install (uv)
169+
## Install (PyPI)
170+
171+
Prerequisites: **Python 3.11 or 3.12**, **ffmpeg** on PATH.
172+
173+
```bash
174+
# Minimal orchestrator (webrtcvad VAD; ASR via API or bundled qwen worker)
175+
pip install resilient-stt
176+
177+
# Recommended on Apple Silicon / Linux (Silero VAD + pyannote diarization + torch)
178+
pip install "resilient-stt[full]"
179+
180+
# Contributors
181+
pip install "resilient-stt[full,dev]"
182+
```
183+
184+
Verify the CLI:
185+
186+
```bash
187+
resilient-stt --help
188+
```
189+
190+
### Usage after `pip install`
191+
192+
Run from any directory (creates `data/work/` under the current working dir unless you pass `--work-root`):
193+
194+
```bash
195+
resilient-stt \
196+
--audio /path/to/audio.wav \
197+
--output /path/to/output-dir \
198+
--language hi
199+
```
200+
201+
With diarization and Silero VAD you need the `[full]` extra and usually `HF_TOKEN` in `.env` (see [Environment variables](#environment-variables)). Quick smoke test without pyannote:
202+
203+
```bash
204+
resilient-stt \
205+
--audio /path/to/audio.wav \
206+
--output /path/to/output-dir \
207+
--language hi \
208+
--skip-diarization \
209+
--repair false
210+
```
211+
212+
Hosted ASR (no local qwen worker) — set `OPENAI_API_KEY` or `OPENROUTER_API_KEY` in `.env`:
213+
214+
```bash
215+
resilient-stt \
216+
--audio /path/to/audio.wav \
217+
--output /path/to/output-dir \
218+
--no-asr-fallback \
219+
--skip-diarization
220+
```
221+
222+
**Bundled qwen-asr worker:** On first auto-start, the CLI creates an isolated venv at
223+
`~/.cache/resilient-stt/qwen-transformers-worker/.venv` (requires network to download
224+
`qwen-asr` and model weights). If inference fails on Apple Silicon, start the worker
225+
manually with `python scripts/bootstrap_qwen_asr_fallback.py --no-aligner` (from a git
226+
checkout) or see the [bundled worker README](src/resilient_stt/workers/qwen_transformers_service/README.md).
227+
228+
Platform notes for `[full]` match [Platform notes (torch / diarization)](#platform-notes-torch--diarization) below (Intel Mac: base install + `--skip-diarization`).
229+
230+
## Install from source (uv)
131231

132232
Prerequisites: [uv](https://docs.astral.sh/uv/), **ffmpeg** on PATH. ASR is optional to
133233
configure manually — see [ASR auto-detection](#asr-auto-detection) below.
@@ -192,19 +292,21 @@ uv run python -c "import torch; print('mps', torch.backends.mps.is_available())"
192292

193293
## Usage
194294

295+
From a **git checkout**, prefix commands with `uv run` (or activate `.venv` and use `resilient-stt` directly). After **`pip install`**, use `resilient-stt` only.
296+
195297
Minimal run (no ASR setup — auto-starts local **qwen-asr** on CPU/MPS when needed):
196298

197299
```bash
198-
uv run python -m orchestrator.main \
300+
uv run resilient-stt \
199301
--audio data/input/meeting.mp3 \
200302
--output data/output/meeting \
201303
--language hi
202304
```
203305

204-
With an external ASR service (vLLM, hosted API, etc.):
306+
With an external ASR service (vLLM, hosted API, etc.) — the endpoint must already be running and respond to `GET {base}/v1/models`:
205307

206308
```bash
207-
uv run python -m orchestrator.main \
309+
uv run resilient-stt \
208310
--audio data/input/meeting.mp3 \
209311
--asr-endpoint http://localhost:8001/v1 \
210312
--model Qwen/Qwen3-ASR-1.7B \
@@ -218,14 +320,14 @@ auto-detect (no local ASR on `:8001`/`:8002`) or point at the API explicitly:
218320

219321
```bash
220322
# Auto-detect when no local ASR is running (--no-asr-fallback avoids starting qwen-asr)
221-
uv run python -m orchestrator.main \
323+
uv run resilient-stt \
222324
--audio data/input/speech.wav \
223325
--output data/output/openai \
224326
--no-asr-fallback \
225327
--skip-diarization
226328

227329
# Explicit endpoint
228-
uv run python -m orchestrator.main \
330+
uv run resilient-stt \
229331
--audio data/input/speech.wav \
230332
--output data/output/openai \
231333
--asr-endpoint https://api.openai.com/v1 \
@@ -237,7 +339,7 @@ uv run python -m orchestrator.main \
237339
the key via `ASR_API_KEY`:
238340

239341
```bash
240-
uv run python -m orchestrator.main \
342+
uv run resilient-stt \
241343
--audio data/input/speech.wav \
242344
--output data/output/openrouter/chirp \
243345
--asr-endpoint https://openrouter.ai/api/v1 \
@@ -257,10 +359,12 @@ When `--asr-endpoint` is omitted (and `ASR_BASE_URL` / `ASR_ENDPOINT` are unset)
257359
2. Probe an existing **qwen-asr** worker at `http://127.0.0.1:8002/v1`
258360
3. **OpenRouter** when `OPENROUTER_API_KEY` is set (no `--model`)
259361
4. **OpenAI** when `OPENAI_API_KEY` is set (no `--model`)
260-
5. Otherwise **start** the local qwen-asr fallback ([workers/qwen_transformers_service/README.md](workers/qwen_transformers_service/README.md)) — slow on CPU/MPS but needs no NVIDIA GPU
362+
5. Otherwise **start** the local qwen-asr fallback ([bundled worker](src/resilient_stt/workers/qwen_transformers_service/README.md)) — slow on CPU/MPS but needs no NVIDIA GPU
261363

262364
Use `--no-asr-fallback` to require an explicit or already-running ASR service.
263365

366+
If you pass `--asr-endpoint` explicitly, that URL must respond to `GET …/v1/models` before the pipeline runs (otherwise the CLI exits with “endpoint configured but unreachable”).
367+
264368
Useful flags:
265369

266370

@@ -340,13 +444,29 @@ Final exports land under `--output`: `transcript.json`, `transcript.srt`,
340444

341445
## Tests
342446

447+
From source:
448+
343449
```bash
450+
uv sync --extra dev
344451
uv run pytest
345452
```
346453

454+
After `pip install "resilient-stt[dev]"` from a checkout (with `src/` on `PYTHONPATH`) or when developing in the repo, the same `pytest` command applies if `src` is configured in `pyproject.toml` (`pythonpath = ["src"]`).
455+
347456
The included tests use synthetic fixtures and mocked HTTP responses; they do
348457
not invoke ffmpeg, pyannote, or any LLM.
349458

459+
## Publishing (maintainers)
460+
461+
1. **Commits on `main`** — use [Conventional Commits](https://www.conventionalcommits.org/) in PR titles or squash messages (`feat:`, `fix:`, `chore:`, etc.).
462+
2. **release-please** (`.github/workflows/release-please.yml`) — opens/updates a **Release PR** that bumps `pyproject.toml`, `CHANGELOG.md`, and `.release-please-manifest.json`.
463+
3. **Ship** — merge the Release PR; release-please creates GitHub Release + tag `vX.Y.Z`.
464+
4. **PyPI** (`.github/workflows/publish.yml`) — runs on that tag when it matches `version` in `pyproject.toml`.
465+
466+
One-time: configure a [PyPI trusted publisher](https://docs.pypi.org/trusted-publishers/) for workflow `publish.yml` on repo `gitcommitshow/resilient-stt`. In the org/repo settings, allow GitHub Actions to create and approve pull requests if Release PRs do not appear.
467+
468+
See [CHANGELOG.md](CHANGELOG.md) for release history.
469+
350470
## License
351471

352472
This project is licensed under the [GNU General Public License v3.0 or

conftest.py

Lines changed: 2 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,5 @@
1-
"""Make repo root importable so `import core` etc. works in tests without install."""
1+
"""Pytest hooks: disable third-party telemetry before any tests run."""
22

3-
import sys
4-
from pathlib import Path
5-
6-
_ROOT = Path(__file__).resolve().parent
7-
if str(_ROOT) not in sys.path:
8-
sys.path.insert(0, str(_ROOT))
9-
10-
from core.privacy import disable_dependency_telemetry # noqa: E402
3+
from resilient_stt.core.privacy import disable_dependency_telemetry
114

125
disable_dependency_telemetry()

0 commit comments

Comments
 (0)