feat(stt): add speech-to-text support in neurolink using google cloud… - #793
Closed
shambhavik-25 wants to merge 1 commit into
Closed
Conversation
|
Important Review skippedAuto incremental reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the Use the checkbox below for a quick retry:
WalkthroughAdds Speech-to-Text (STT): new types and runtime guards, a provider-agnostic STTProcessor and STTError model, a Google Cloud STT handler, SDK integration (generateSTT, getSTTLanguages, getSTTModels), CLI commands/options and output handling, provider registration, and extensive documentation. Changes
Sequence Diagram(s)sequenceDiagram
actor User
participant CLI as "CLI Parser / CommandFactory"
participant NL as "NeuroLink SDK"
participant Proc as "STTProcessor"
participant GHandler as "GoogleSTTHandler"
participant GAPI as "Google Cloud Speech API"
User->>CLI: run transcribe/generate --stt with audio file
CLI->>NL: generate(options { audioFiles, stt })
NL->>Proc: transcribe(audioBuffer, provider, options)
Proc->>GHandler: transcribe(audioBuffer, options)
GHandler->>GAPI: SpeechClient.recognize(request)
GAPI-->>GHandler: recognition response
GHandler->>Proc: STTResult (text, words, confidence, metadata)
Proc-->>NL: STTResult
NL-->>CLI: GenerateResult { transcription }
CLI->>User: render/save transcription and metadata
Estimated code review effort🎯 4 (Complex) | ⏱️ ~50 minutes Possibly related PRs
Suggested labels
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
shambhavik-25
force-pushed
the
BZ-48219-add-speech-to-text-support-in-neuro-link
branch
from
January 29, 2026 09:52
a2c1fd6 to
c545eaf
Compare
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 13
🤖 Fix all issues with AI agents
In `@docs/features/stt.md`:
- Around line 520-526: The shell snippet has inline comments after a trailing
backslash which breaks continuations; update the block around the neurolink
generate <audio-file> example so every line ending with a backslash has no
trailing comment, move explanatory comments for flags (e.g., --stt-language,
--stt-model, --stt-enable-timestamps, --stt-enable-diarization) to their own
separate lines above or below the flag lines (or remove backslashes on commented
lines), and ensure each continuation backslash is the last character on the line
so the CLI example copies and pastes cleanly.
In `@src/cli/factories/commandFactory.ts`:
- Around line 1138-1140: Remove the dangling JSDoc "Create STT (Speech-to-Text)
transcribe command" comment or implement the actual command: either delete that
comment block entirely to avoid misleading docs/IDE tooltips, or add a matching
command factory function (e.g., createSttTranscribeCommand) and wire it into the
command registration (where other commands are created/registered in this
module, e.g., the existing createCommands/registerCommand logic) so the comment
accurately describes real code.
- Around line 315-320: The CLI option definition for sttModel in
commandFactory.ts currently restricts choices to
["default","command_and_search","phone_call","video"]; update the sttModel
choices array in the sttModel option to include the additional SDK-supported
values "medical_dictation", "latest_long", and "latest_short" so the CLI
--stt-model accepts those models; locate the sttModel object in the
commandFactory.ts file and add those three strings to the choices list (keeping
the type and default intact).
- Around line 2410-2413: The STT output handler call inside the conditional is
not awaited, so async work in handleSTTOutput can be interrupted by process
exit; update the call in the block that currently checks isSTTMode and
result.transcription to await this.handleSTTOutput(result, options) and ensure
the enclosing function (e.g., the method in commandFactory where this if lives)
is marked async or properly returns a Promise so the await is valid and the
process waits for writes to finish.
- Around line 1254-1289: The command handler calls sdk.getSTTLanguages and only
logs a count via spinner.succeed but never prints the actual languages; after
obtaining languages (the languages variable returned by
NeuroLink.getSTTLanguages) and after spinner.succeed, output the full list to
the console (e.g., join with newline or iterate and console.log each language,
or use any existing CLI formatter utility) so users see the language entries
they requested; ensure you reference the async (argv) handler, NeuroLink,
sdk.getSTTLanguages and spinner when adding the print.
- Around line 2220-2318: handleSTTOutput currently nests all stdout under if
(!options.quiet) so when quiet is true nothing is printed and --format json is
only written to file; change the logic in handleSTTOutput to separate "human"
logging from machine/json output: keep all logger.always(...) calls inside the
if (!options.quiet) block (so human-readable output is suppressed when quiet)
but ensure that when options.format === "json" and no options.output is provided
you write JSON to stdout (e.g.,
process.stdout.write(JSON.stringify(transcription))) irrespective of
options.quiet, and when options.output is present keep current file write
behavior for both json and plain text; locate this change in the handleSTTOutput
method referenced above and adjust the conditions around the options.quiet and
options.format checks accordingly.
In `@src/lib/adapters/stt/googleSTTHandler.ts`:
- Around line 233-236: The recognize call in GoogleSTTHandler should be wrapped
with the withTimeout utility instead of relying on the client option alone;
update the call in the method that currently does
"this.client.recognize(request, { timeout:
GoogleSTTHandler.DEFAULT_API_TIMEOUT_MS })" to invoke withTimeout around the
async operation (using the client.recognize function bound to this.client or an
arrow wrapper) and pass GoogleSTTHandler.DEFAULT_API_TIMEOUT_MS as the timeout
value so the operation uses the library-standard timeout wrapper.
- Around line 248-317: The handler currently only uses response.results[0]
(primaryResult/primaryAlternative) and drops later segments; modify the logic to
iterate over response.results to aggregate full transcript text, merge
word-level timings (concatenate maps using convertDurationToSeconds), and
combine alternatives and confidences (e.g., append alternatives per segment or
compute an overall confidence) so the returned object fields text, confidence,
languageCode, words, alternatives, and duration represent the complete
multi-segment response; update where primaryResult/primaryAlternative are
referenced and where words, alternatives, duration are computed to use the
aggregated values.
In `@src/lib/constants/enums.ts`:
- Line 770: The ErrorCategory enum contains an inconsistent casing: the member
named STT is set to uppercase "STT"; change its string value to lowercase "stt"
so it matches the lowercase pattern used by other members and downstream
consumers; locate the ErrorCategory enum (the STT member) and update the
right-hand string literal from "STT" to "stt" while leaving the enum member name
unchanged.
In `@src/lib/neurolink.ts`:
- Around line 1839-1847: The getSTTLanguages method currently declares provider
as optional but throws a generic Error when missing; update the API to either
make provider required (change signature to provider: string) or provide a
sensible default (e.g., default to the primary provider constant) and replace
the generic Error with a typed error created via ErrorFactory (e.g.,
ErrorFactory.createInvalidArgument or the project's equivalent) for the
validation failure; apply the same change pattern to the other methods noted
(those around getTTSLanguages / similar at 1863-1871) so all provider validation
uses ErrorFactory and the parameter signatures/behavior are consistent.
- Around line 1983-1986: When options.stt is true the current branch only calls
generateSTT when options.input.audioFiles exists, causing STT requests with
missing audio to be treated as normal text generation; change the conditional to
unconditionally route STT requests to generateSTT by replacing the check with a
simple if (options.stt) return this.generateSTT(options); and ensure
generateSTT(options) is responsible for validating options.input.audioFiles and
throwing/handling the error; refer to the generateSTT method and the options.stt
/ options.input.audioFiles properties to locate and update the logic.
- Around line 1768-1823: The generateSTT method currently throws generic Errors
and performs unbounded I/O; update it to use ErrorFactory for typed errors and
wrap async ops with withTimeout: replace direct throws for missing options and
invalid audio file types with ErrorFactory (e.g., ErrorFactory.create(...)) and
wrap the fs.readFile call and the STTProcessor.transcribe call with withTimeout,
passing sensible timeouts and propagating timeout errors via ErrorFactory;
ensure you reference generateSTT, options.input.audioFiles, fs.readFile,
STTProcessor.transcribe, withTimeout, and ErrorFactory when making these
changes.
In `@src/lib/utils/sttProcessor.ts`:
- Around line 246-253: Replace the plain Error throws in
STTProcessor.registerHandler with typed errors produced by ErrorFactory: when
providerName is falsy, call ErrorFactory.create or the appropriate ErrorFactory
method to create a validation/argument error that includes context (e.g.,
"Provider name is required") and throw that instead; likewise for the handler
null check, use ErrorFactory to construct and throw the typed error. Update
references in the registerHandler(providerName: string, handler: STTHandler)
method so both validation failures use ErrorFactory-produced errors rather than
new Error(...).
shambhavik-25
force-pushed
the
BZ-48219-add-speech-to-text-support-in-neuro-link
branch
4 times, most recently
from
January 29, 2026 12:03
0c94261 to
b754e05
Compare
murdore
force-pushed
the
BZ-48219-add-speech-to-text-support-in-neuro-link
branch
3 times, most recently
from
February 1, 2026 11:33
d1a911c to
89bb415
Compare
shambhavik-25
force-pushed
the
BZ-48219-add-speech-to-text-support-in-neuro-link
branch
2 times, most recently
from
February 13, 2026 15:44
a9149d0 to
e4fdb1a
Compare
|
@shambhavik-25 is attempting to deploy a commit to the Sachin Sharma's projects Team on Vercel. A member of the Team first needs to authorize it. |
shambhavik-25
force-pushed
the
BZ-48219-add-speech-to-text-support-in-neuro-link
branch
9 times, most recently
from
February 16, 2026 10:01
20b4e1a to
c01ffb6
Compare
murdore
force-pushed
the
BZ-48219-add-speech-to-text-support-in-neuro-link
branch
from
February 16, 2026 17:28
c01ffb6 to
83511fe
Compare
Contributor
|
@coderabbitai full review and check if all project standards are being followed. also check if all the previous comments have been addressed |
|
I'll review all changes with a focus on ensuring they align with the NeuroLink project's established patterns and best practices. 🧠 Learnings used✅ Actions performedFull review triggered. |
There was a problem hiding this comment.
Actionable comments posted: 6
🤖 Fix all issues with AI agents
In `@docs/features/stt.md`:
- Around line 633-640: The fenced code block mixes shell and TypeScript which
confuses highlighting; split the single ```bash block into two separate fenced
blocks: one ```bash block containing the echo $GOOGLE_AI_API_KEY line and a
second ```typescript block containing the TypeScript snippet that calls
neurolink.getSTTLanguages() and logs the result, preserving the order and
surrounding backticks so each block has the correct language tag for syntax
highlighting.
In `@package.json`:
- Line 198: Remove the unused "commander" dependency from package.json by
deleting the "commander": "^14.0.2" entry and then update the lockfile and
installed packages (run your package manager's install or prune command) so the
removal is applied; verify no references to commander remain and that CLI
behavior still relies on yargs/CommandFactory code in src/cli/**/*.ts.
In `@src/cli/factories/commandFactory.ts`:
- Around line 3692-3694: The stub function flushTraces() unconditionally throws
and is called from the --list-conversations flow, causing a crash; replace calls
to flushTraces() with a call to the existing
CLICommandFactory.flushLangfuseTraces() (or have flushTraces() delegate to
CLICommandFactory.flushLangfuseTraces()) and remove the dead stub to avoid the
unhandled exception, ensuring all references use
CLICommandFactory.flushLangfuseTraces() so the list-conversations path no longer
throws.
- Around line 2196-2208: When isSTTMode is true and you build audioFiles from
argv.input, validate that argv.input exists and is readable before adding it to
generateInput: check the filesystem (e.g., fs.existsSync or fs.promises.access)
for argv.input and if the file is missing or not readable, throw or return a
clear CLI error (with a message like "STT input file not found or not readable:
<path>") rather than passing the path into audioFiles; update the logic around
isSTTMode / audioFiles / generateInput so the existence check runs before
constructing generateInput and before any SDK calls.
- Around line 2290-2392: The transcription text is only printed inside the if
(!options.quiet) block in handleSTTOutput, so with quiet=true the plain-text
transcript is never emitted; move the plain transcription output (the line that
emits transcription.text) out of the quiet-only block so the transcript is
always written (use process.stdout.write or logger.always to emit
transcription.text when options.format !== 'json' and no output file specified),
while keeping decorative chrome, metadata lines and alternatives gated behind if
(!options.quiet); ensure references to transcription.text, handleSTTOutput, and
the JSON stdout path are preserved and not duplicated.
In `@src/lib/neurolink.ts`:
- Around line 1914-1920: The returned GenerateResult incorrectly sets model to
the text-generation option (options.model); replace that with the STT model from
sttResult.metadata.model (or remove the top-level model field) so the top-level
model reflects the actual STT model used; update the return object in the
function that returns the GenerateResult (reference symbols: sttResult,
options.model, GenerateResult, transcription.metadata.model,
sttResult.metadata.model) to use sttResult.metadata.model for the model property
or omit the property entirely.
🧹 Nitpick comments (5)
src/lib/types/index.ts (1)
233-249: Consolidate duplicate re-exports; keep only the STT export here.This block repeats exports already listed above, which adds noise and risks confusion about the public surface. Keep a single STT export and drop the duplicates.
♻️ Proposed cleanup
-// Middleware Types - Middleware system types -export * from "./middlewareTypes.js"; - -// File detection and processing types -export * from "./fileTypes.js"; - -// Content types for multimodal support (includes multimodal re-exports for backward compatibility) -export * from "./content.js"; - -// TTS (Text-to-Speech) types -export * from "./ttsTypes.js"; - -// STT (Speech-to-Text) types -export * from "./sttTypes.js"; - -// HITL (Human-in-the-Loop) types -export * from "./hitlTypes.js"; +// STT (Speech-to-Text) types +export * from "./sttTypes.js";src/cli/factories/commandFactory.ts (1)
50-50: Commented-out import is dead code.
normalizeEvaluationDataimport is fully commented out. Remove it to keep the module clean.Proposed fix
-//import { normalizeEvaluationData } from "../../lib/utils/evaluationUtils.js";src/lib/types/sttTypes.ts (1)
57-64: Consider deriving themodelunion fromVALID_STT_MODELSto keep them in sync automatically.Currently the model union in
STTOptions.model(lines 57–64) and theVALID_STT_MODELSarray (lines 168–176) list the same values independently. If one is updated without the other, the type guard would silently diverge from the type.Example: derive the union from the const array
export const VALID_STT_MODELS = [ "default", "command_and_search", "phone_call", "video", "medical_dictation", "latest_long", "latest_short", ] as const; export type STTModel = (typeof VALID_STT_MODELS)[number]; // Then in STTOptions: // model?: STTModel;Also applies to: 168-176
src/lib/adapters/stt/googleSTTHandler.ts (2)
92-163: Hardcoded language list only covers ~55 of 125+ claimed languages.
getLanguages()returns a static array and doesn't query the Google API (which lacks a dedicated language discovery endpoint, so this is understandable). However, the module doc and PR description claim "125+ languages". Consider either expanding the list to be more complete or adjusting the messaging to say "commonly used languages" to set accurate expectations.
444-472: Hardcoded numeric enum values are fragile but functional.The encoding map uses raw integers (1–9) matching
google.cloud.speech.v1.RecognitionConfig.AudioEncodingproto values. If the proto definition ever changes, these would silently break. Consider importing the enum directly from the proto if available, e.g.,protos.google.cloud.speech.v1.RecognitionConfig.AudioEncoding.LINEAR16.#!/bin/bash # Check if the enum is accessible from the proto import rg -n "AudioEncoding" --type ts -g '!node_modules' -g '!*.d.ts' | head -20 echo "---" # Check the proto definition structure fd -t f "protos.d.ts" -p "google-cloud/speech" --exec head -200 {} \; 2>/dev/null || echo "Proto file not found in repo"
shambhavik-25
force-pushed
the
BZ-48219-add-speech-to-text-support-in-neuro-link
branch
3 times, most recently
from
February 17, 2026 06:59
e838f7a to
1270db5
Compare
Author
|
@coderabbitai please do a full review and check if all project standards are being followed. also check if all the previous comments have been addressed |
|
I'll examine all the changes systematically and provide detailed feedback on any issues found. 🧠 Learnings used✅ Actions performedFull review triggered. |
There was a problem hiding this comment.
Actionable comments posted: 4
🧹 Nitpick comments (7)
src/lib/types/sttTypes.ts (1)
157-160: Normalize audio format before checking support.
Trimming a leading dot and lowercasing avoids false negatives from inputs like.WAV.♻️ Suggested tweak
export function isSupportedAudioFormat( format: string, ): format is SupportedAudioFormat { - return SUPPORTED_AUDIO_FORMATS.includes(format as SupportedAudioFormat); + const normalized = format.replace(/^\./, "").toLowerCase(); + return SUPPORTED_AUDIO_FORMATS.includes(normalized as SupportedAudioFormat); }🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@src/lib/types/sttTypes.ts` around lines 157 - 160, The isSupportedAudioFormat function should normalize the incoming format string before checking membership: strip any leading '.' and lowercase the value (e.g., const normalized = format.replace(/^\./, '').toLowerCase()) and then use SUPPORTED_AUDIO_FORMATS.includes(normalized as SupportedAudioFormat). Update the check in isSupportedAudioFormat to use the normalized value so inputs like ".WAV" or "Wav" are correctly recognized.src/lib/utils/sttProcessor.ts (2)
347-398: Add withTimeout around provider discovery calls.
WrappinggetLanguages()andgetModels()prevents indefinite hangs; importwithTimeoutfrom the existing utils module.As per coding guidelines: Wrap async operations with withTimeout utility.♻️ Suggested change
- return handler.getLanguages(); + return withTimeout( + handler.getLanguages(), + 60_000, + `STT getLanguages timed out for provider "${providerName}"` + );- return handler.getModels(); + return withTimeout( + handler.getModels(), + 60_000, + `STT getModels timed out for provider "${providerName}"` + );🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@src/lib/utils/sttProcessor.ts` around lines 347 - 398, Import the existing withTimeout helper from the utils module and wrap the provider discovery promises to avoid hangs: in getLanguages() replace the direct return of handler.getLanguages() with an awaited withTimeout(handler.getLanguages(), <timeoutMs>) (and do the same in getModels()); ensure you import withTimeout at the top and await the wrapper, and if desired catch a timeout error and rethrow an STTError that includes STT_ERROR_CODES.PROVIDER_NOT_SUPPORTED or a new timeout-specific code and the provider context to preserve error semantics.
129-180: Prefer atypealias for STTHandler to match repo conventions.Based on learnings: In the juspay/neurolink repository, new type definitions should use the `type` keyword instead of `interface`, unless there is a valid and justified exception.♻️ Suggested change
-export interface STTHandler { +export type STTHandler = { /** * Transcribe audio to text using provider-specific STT API * * **IMPORTANT: Timeout Responsibility** @@ -} +};🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@src/lib/utils/sttProcessor.ts` around lines 129 - 180, Replace the exported STTHandler interface with an exported type alias named STTHandler while preserving the exact shape (methods transcribe, getLanguages, getModels, isConfigured and readonly properties maxAudioSizeMB and maxDurationSeconds) and JSDoc comments; ensure the transcribe signature Promise<STTResult>, optional methods are marked optional, and exported name remains STTHandler so existing imports (and references to transcribe, getLanguages, getModels, isConfigured, maxAudioSizeMB, maxDurationSeconds) continue to work.src/lib/neurolink.ts (1)
167-168: Remove the commented-out STTResult import.It adds noise without value. Clean it up if no longer needed.
🧹 Proposed cleanup
-import { STTProcessor } from "./utils/sttProcessor.js"; -//import type { STTResult } from "./types/sttTypes.js"; +import { STTProcessor } from "./utils/sttProcessor.js";🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@src/lib/neurolink.ts` around lines 167 - 168, Remove the unnecessary commented import line for STTResult — delete the line "//import type { STTResult } from "./types/sttTypes.js";" so only the active import (STTProcessor) remains, keeping the top of neurolink.ts clean and free of unused commented imports.src/lib/adapters/stt/googleSTTHandler.ts (2)
206-233: Consider omittingsampleRateHertzfor self-describing formats like MP3.Google STT ignores
sampleRateHertzfor MP3 and OGG_OPUS (the rate is in the file header), but for LINEAR16/FLAC it's required. The hardcoded default of16000works fine today but could silently produce degraded results if a user supplies a 44.1 kHz WAV without specifying the rate. A small improvement would be to omit the field entirely when the encoding is self-describing, so Google uses auto-detection.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@src/lib/adapters/stt/googleSTTHandler.ts` around lines 206 - 233, The request currently always sets sampleRateHertz to 16000 which can be wrong for self-describing formats; update the request-building in googleSTTHandler.ts (inside the function that builds the request using this.mapAudioEncoding and options) to only include sampleRateHertz when the mapped encoding is a non-self-describing type (e.g., LINEAR16 or FLAC) and omit it for self-describing encodings like MP3 and OGG_OPUS so Google will auto-detect the rate; use the result of this.mapAudioEncoding(options.encoding || "MP3") to drive this conditional and keep the existing defaulting behavior for required encodings.
92-164: Hardcoded language list means the cache serves no real purpose.Since
getLanguages()returns a static array (not fetched from an API), the 5-minute TTL cache adds complexity without benefit. If this is a placeholder for a future API call, consider adding a comment; otherwise, the cache can be removed.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@src/lib/adapters/stt/googleSTTHandler.ts` around lines 92 - 164, The getLanguages() function currently builds a static languages array and then stores it in this.languagesCache with a timestamp, which is unnecessary; remove the caching assignment (this.languagesCache = { languages, timestamp: Date.now() }) and any related TTL logic so getLanguages() simply returns the static languages array, and also remove or clean up the languagesCache property/uses elsewhere; if this was intended as a future placeholder instead, replace the assignment with a clear TODO comment referencing getLanguages() and this.languagesCache.src/cli/factories/commandFactory.ts (1)
2346-2356: Minor: Dynamicfs/promisesimport shadows the top-levelfsimport.
fsis already imported at the top of the file (import fs from "node:fs"). Re-importingfs/promisesasfsinside this method shadows the module-level binding, which is confusing and unnecessary.♻️ Use the already-imported module
- const fs = await import("fs/promises"); - if (options.format === "json") { - await fs.writeFile( - options.output as string, - JSON.stringify(transcription, null, 2), - ); - } else { - await fs.writeFile(options.output as string, transcription.text); - } + if (options.format === "json") { + fs.writeFileSync( + options.output as string, + JSON.stringify(transcription, null, 2), + ); + } else { + fs.writeFileSync(options.output as string, transcription.text); + }🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@src/cli/factories/commandFactory.ts` around lines 2346 - 2356, The dynamic import of "fs/promises" as fs shadows the top-level fs import and is unnecessary; remove the await import("fs/promises") line and use the existing top-level fs to write files (e.g., call fs.promises.writeFile for both JSON and text branches when handling options.output and transcription). Update the branches that call writeFile to use fs.promises.writeFile and keep the same arguments (options.output and either JSON.stringify(transcription, null, 2) or transcription.text).
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@CHANGELOG.md`:
- Around line 1-5: Update the 9.8.0 release notes in CHANGELOG.md to include a
new Features bullet announcing STT support and the new runtime dependency
introduced by this PR: under the "## [9.8.0] ... (2026-02-17)" section add a
line like "- **STT:** add speech-to-text support and include new runtime
dependency (name the actual package added in this PR)" so the entry explicitly
mentions STT and the runtime dependency alongside the existing TS migration
note.
In `@docs/features/stt.md`:
- Around line 33-41: The docs currently list GOOGLE_AI_API_KEY as an STT auth
option but the Google STT handler (googleSTTHandler.ts / the GoogleSTT handler)
only supports service account credentials via GOOGLE_APPLICATION_CREDENTIALS or
a credentialsPath parameter; remove the "Option 2" API key snippet from
docs/features/stt.md and update the text to state that STT requires service
account credentials (GOOGLE_APPLICATION_CREDENTIALS or credentialsPath) only,
ensuring no references to GOOGLE_AI_API_KEY remain; if you prefer to keep
API-key support instead, implement API-key handling inside
src/lib/adapters/stt/googleSTTHandler.ts (add logic to accept GOOGLE_AI_API_KEY
and construct the client accordingly) and then keep the docs, but do one or the
other—not both.
In `@src/cli/factories/commandFactory.ts`:
- Around line 2238-2314: The buildGenerateOptions flow constructs sdkOptions but
omits the STT configuration, so generate() gets audio but no transcription
settings; update buildGenerateOptions to add an stt (or speechToText) object
into sdkOptions using the CLI-extracted STT fields (language, model, enhanced,
maxAlternatives, enablePunctuation, enableTimestamps, enableConfidence,
enableDiarization, speakerCount) so the SDK receives them; place it alongside
other top-level sdkOptions properties (near provider/model/temperature) and map
each property from the existing extracted variables (argv or enhancedOptions
whichever holds them) to the corresponding STT field.
In `@src/lib/neurolink.ts`:
- Around line 1868-1895: The code currently only reads the first entry of
options.input.audioFiles into audioBuffer and silently drops any additional
files; update the validation in the audio input block (where audioBuffer is set
and options.input.audioFiles is checked) to detect if
options.input.audioFiles.length > 1 and fail fast by throwing
ErrorFactory.invalidParameters (e.g., key "audioFiles" with an error stating
multiple files are not supported) so callers are informed, or alternatively
implement batching support if desired; keep the existing Buffer/string handling
and withTimeout/fs.readFile logic for the single-file path (references: variable
audioBuffer, options.input.audioFiles, withTimeout,
ErrorFactory.invalidParameters, fileReadTimeoutMs).
---
Nitpick comments:
In `@src/cli/factories/commandFactory.ts`:
- Around line 2346-2356: The dynamic import of "fs/promises" as fs shadows the
top-level fs import and is unnecessary; remove the await import("fs/promises")
line and use the existing top-level fs to write files (e.g., call
fs.promises.writeFile for both JSON and text branches when handling
options.output and transcription). Update the branches that call writeFile to
use fs.promises.writeFile and keep the same arguments (options.output and either
JSON.stringify(transcription, null, 2) or transcription.text).
In `@src/lib/adapters/stt/googleSTTHandler.ts`:
- Around line 206-233: The request currently always sets sampleRateHertz to
16000 which can be wrong for self-describing formats; update the
request-building in googleSTTHandler.ts (inside the function that builds the
request using this.mapAudioEncoding and options) to only include sampleRateHertz
when the mapped encoding is a non-self-describing type (e.g., LINEAR16 or FLAC)
and omit it for self-describing encodings like MP3 and OGG_OPUS so Google will
auto-detect the rate; use the result of this.mapAudioEncoding(options.encoding
|| "MP3") to drive this conditional and keep the existing defaulting behavior
for required encodings.
- Around line 92-164: The getLanguages() function currently builds a static
languages array and then stores it in this.languagesCache with a timestamp,
which is unnecessary; remove the caching assignment (this.languagesCache = {
languages, timestamp: Date.now() }) and any related TTL logic so getLanguages()
simply returns the static languages array, and also remove or clean up the
languagesCache property/uses elsewhere; if this was intended as a future
placeholder instead, replace the assignment with a clear TODO comment
referencing getLanguages() and this.languagesCache.
In `@src/lib/neurolink.ts`:
- Around line 167-168: Remove the unnecessary commented import line for
STTResult — delete the line "//import type { STTResult } from
"./types/sttTypes.js";" so only the active import (STTProcessor) remains,
keeping the top of neurolink.ts clean and free of unused commented imports.
In `@src/lib/types/sttTypes.ts`:
- Around line 157-160: The isSupportedAudioFormat function should normalize the
incoming format string before checking membership: strip any leading '.' and
lowercase the value (e.g., const normalized = format.replace(/^\./,
'').toLowerCase()) and then use SUPPORTED_AUDIO_FORMATS.includes(normalized as
SupportedAudioFormat). Update the check in isSupportedAudioFormat to use the
normalized value so inputs like ".WAV" or "Wav" are correctly recognized.
In `@src/lib/utils/sttProcessor.ts`:
- Around line 347-398: Import the existing withTimeout helper from the utils
module and wrap the provider discovery promises to avoid hangs: in
getLanguages() replace the direct return of handler.getLanguages() with an
awaited withTimeout(handler.getLanguages(), <timeoutMs>) (and do the same in
getModels()); ensure you import withTimeout at the top and await the wrapper,
and if desired catch a timeout error and rethrow an STTError that includes
STT_ERROR_CODES.PROVIDER_NOT_SUPPORTED or a new timeout-specific code and the
provider context to preserve error semantics.
- Around line 129-180: Replace the exported STTHandler interface with an
exported type alias named STTHandler while preserving the exact shape (methods
transcribe, getLanguages, getModels, isConfigured and readonly properties
maxAudioSizeMB and maxDurationSeconds) and JSDoc comments; ensure the transcribe
signature Promise<STTResult>, optional methods are marked optional, and exported
name remains STTHandler so existing imports (and references to transcribe,
getLanguages, getModels, isConfigured, maxAudioSizeMB, maxDurationSeconds)
continue to work.
shambhavik-25
force-pushed
the
BZ-48219-add-speech-to-text-support-in-neuro-link
branch
2 times, most recently
from
February 17, 2026 08:10
17f6cb5 to
f6dccbd
Compare
Author
|
@coderabbitai can you check if all the previous comments have been addressed? |
|
Tip For best results, initiate chat on the files or code changes. 🧩 Analysis chain
🏁 Script executed: Length of output: 135989
✅ Summary: All Previous Comments Are ResolvedTotal review threads analyzed: 22
📊 Breakdown by Review DateJanuary 29, 2026 (First Review)
February 16, 2026 (Second Review)
February 17, 2026 (Third Review)
🔍 Key Areas That Were AddressedThe resolved comments covered:
All review threads have been marked as resolved by the development team, indicating that the feedback has been either implemented through code changes or discussed and closed. The PR appears to have addressed all previous review feedback comprehensively. 🧠 Learnings used |
murdore
requested changes
Feb 17, 2026
murdore
requested changes
Feb 17, 2026
Contributor
|
@shambhavik-25 code approach is not correct. Can you check other multi-modality pull requests that have been merged and verify all the changes according to that? This will include documentation, usage and other changes. |
shambhavik-25
force-pushed
the
BZ-48219-add-speech-to-text-support-in-neuro-link
branch
4 times, most recently
from
February 19, 2026 11:50
38ec7b2 to
d8f3e15
Compare
Author
|
@coderabbitai can you please review my pr? |
|
Tip For best results, initiate chat on the files or code changes. 🧩 Analysis chain
🏁 Script executed: Length of output: 17177 🏁 Script executed: Length of output: 33710 🏁 Script executed: Length of output: 4399 🏁 Script executed: Length of output: 5836 🏁 Script executed: Length of output: 7926
🔴 Critical Issue: STT Does NOT Follow the Established Multi-modality PatternCurrent Implementation (Incorrect)Your STT implementation uses a separate routing path that bypasses the core In // Check if STT output requested (audio transcription)
if (options.stt) {
return this.generateSTT(options); // ❌ Bypasses baseProvider
}Problem: This creates a How TTS Implements It (Correct Pattern) ✅TTS PR In // ===== TTS MODE 1: Direct Input Synthesis (useAiResponse=false) =====
if (options.tts?.enabled && !options.tts?.useAiResponse) {
// Synthesize input text directly WITHOUT AI generation
const textToSynthesize = options.prompt ?? options.input?.text ?? "";
// ... synthesis logic ...
return baseResult; // Return early with audio
}
// ... Normal AI generation flow ...
// ===== TTS MODE 2: AI Response Synthesis (useAiResponse=true) =====
if (options.tts?.enabled && options.tts?.useAiResponse) {
// Synthesize AI-generated response AFTER generation
// ... synthesis logic ...
enhancedResult.audio = audioResult;
}Key Points:
How Video Generation Implements It (Another Example) ✅In private async handleVideoGeneration(
options: TextGenerationOptions,
startTime: number,
): Promise<EnhancedGenerateResult> {
const { generateVideoWithVertex, VideoError, VIDEO_ERROR_CODES } =
await import("../adapters/video/vertexVideoHandler.js");
// ... video generation logic directly in baseProvider ...
return await this.enhanceResult(baseResult, options, startTime);
}Called from within 📋 Required Changes1. Remove STT Routing from neurolink.tsRemove these lines from // Check if STT output requested (audio transcription)
if (options.stt) {
return this.generateSTT(options); // ❌ DELETE THIS
}Remove the entire 2. Integrate STT into baseProvider.generate()Add to // ===== STT: Audio Transcription =====
// Check for audio transcription request before AI generation
if (options.stt && options.input?.audioFiles) {
return await this.handleSTTTranscription(options, startTime);
}
// ... existing AI generation flow continues ...Add the handler method in baseProvider.ts: /**
* Handle Speech-to-Text transcription
*
* Converts audio input to text transcription using STT processor.
* Can optionally pass transcription to AI for further processing.
*/
private async handleSTTTranscription(
options: TextGenerationOptions,
startTime: number,
): Promise<EnhancedGenerateResult> {
const { STTProcessor } = await import("../utils/sttProcessor.js");
// Get audio buffer from input
const audioFile = options.input.audioFiles![0];
const audioBuffer = Buffer.isBuffer(audioFile)
? audioFile
: await this.readAudioFile(audioFile);
const provider = options.provider || this.providerName;
// Transcribe audio
const transcription = await STTProcessor.transcribe(
audioBuffer,
provider,
options.stt!
);
// Build result with transcription
const baseResult: EnhancedGenerateResult = {
content: transcription.text,
provider,
model: options.model || transcription.metadata?.model || "unknown",
usage: { input: 0, output: 0, total: 0 },
transcription,
};
return await this.enhanceResult(baseResult, options, startTime);
}3. Move STT Methods to baseProviderMove these public methods from neurolink.ts to baseProvider.ts: // In baseProvider.ts
public async getSTTLanguages(provider?: string): Promise<string[]> {
const { STTProcessor } = await import("../utils/sttProcessor.js");
return STTProcessor.getLanguages(provider || this.providerName);
}
public async getSTTModels(provider?: string): Promise<string[]> {
const { STTProcessor } = await import("../utils/sttProcessor.js");
return STTProcessor.getModels(provider || this.providerName);
}Then expose them in neurolink.ts by delegating: // In neurolink.ts
public async getSTTLanguages(provider?: string): Promise<string[]> {
const baseProvider = await this.getProvider();
return baseProvider.getSTTLanguages(provider);
}
public async getSTTModels(provider?: string): Promise<string[]> {
const baseProvider = await this.getProvider();
return baseProvider.getSTTModels(provider);
}📚 Reference PRs to Study
🎯 SummaryThe core issue is architectural inconsistency: STT currently bypasses the standard
This will ensure STT follows the same architectural patterns as TTS, video, and other modalities, maintaining consistency across the codebase. Would you like me to provide more detailed code examples for any specific part of the refactoring? 🧠 Learnings used |
Author
|
@coderabbitai can you help me with refactoring it? |
|
Tip For best results, initiate chat on the files or code changes. 🧩 Analysis chain
🏁 Script executed: Length of output: 13633 Perfect! I now have a complete picture of the current implementation. Let me create a detailed refactoring plan that follows the TTS pattern. Refactoring Plan: Move STT to baseProvider PatternThis refactoring will align STT with the established multi-modality architecture by integrating it into Overview:
Testing ChecklistAfter implementing these changes, verify:
SummaryThis refactoring:
The implementation mirrors TTS Mode 1 (direct processing without AI generation) and follows the same handler delegation pattern used by video and image generation. 🧠 Learnings used💡 Use |
shambhavik-25
force-pushed
the
BZ-48219-add-speech-to-text-support-in-neuro-link
branch
3 times, most recently
from
February 19, 2026 18:33
e6e1e6d to
2b55532
Compare
Author
|
@coderabbitai can you please review if the pr is in accordance to the established multi-modality architecture? |
|
Tip For best results, initiate chat on the files or code changes. 🧩 Analysis chain🏁 Script executed: Length of output: 4692 🏁 Script executed: Length of output: 5578 🏁 Script executed: Length of output: 4432 🏁 Script executed: Length of output: 6949 🏁 Script executed: Length of output: 10890 🏁 Script executed: Length of output: 4512
✅ What's Architecturally Correct### 1. Core STT Transcription Flow ✅The main transcription flow is correctly integrated into // In baseProvider.generate() (lines 895-902)
if (sttOptionsValue) {
logger.info("[BaseProvider.generate] Routing to STT transcription", {
provider: this.providerName,
});
return this.handleSTTTranscription(options, startTime);
}Flow path:
✅ This follows the established pattern used by:
❌ What's Architecturally Incorrect2. STT Discovery Methods Missing/Incomplete ❌Issue A: Only Issue B: Current implementation (lines 1854-1866): async getSTTModels(providerName?: string): Promise<string[]> {
// ... initialization ...
const { STTProcessor } = await import("./utils/sttProcessor.js");
return STTProcessor.getModels(provider); // ❌ Direct call
}Should delegate to baseProvider: async getSTTModels(providerName?: string): Promise<string[]> {
const baseProvider = await this.getProvider();
return baseProvider.getSTTModels(providerName); // ✅ Delegation
}Issue C: 🔧 Required FixesFix 1: Add
|
| Component | Status | Notes |
|---|---|---|
| STT transcription flow | ✅ Correct | Integrated into baseProvider.generate() |
handleSTTTranscription() |
✅ Correct | Follows video generation pattern |
| Old routing removed | ✅ Correct | No generateSTT() in neurolink |
baseProvider.getSTTModels() |
✅ Correct | Exists and implemented |
baseProvider.getSTTLanguages() |
❌ Missing | Needs to be added |
neurolink.getSTTModels() |
❌ Incorrect | Calls STTProcessor directly |
neurolink.getSTTLanguages() |
❌ Missing | Doesn't exist |
✅ Final Verdict
The STT implementation is ~85% architecturally correct:
- ✅ Core transcription flow follows the multi-modality pattern
- ✅ No separate routing in neurolink.ts
- ✅ Integrated into baseProvider.generate()
- ❌ Discovery methods don't follow the delegation pattern
The refactoring successfully moved STT transcription to baseProvider, but the discovery methods (getSTTLanguages, getSTTModels) were not fully migrated to follow the established delegation pattern seen in TTS.
Recommendation: Apply the 3 fixes above to achieve 100% architectural compliance with the established multi-modality patterns. The changes are straightforward and follow the delegation pattern where a first-level object receives a request and delegates said action to a second-level object.
Would you like me to generate a `` to implement these fixes?
🧠 Learnings used
Learnt from: vigneshJuspay
Repo: juspay/neurolink PR: 237
File: memory-bank/tts-provider-implementation-plan.md:92-106
Timestamp: 2025-11-17T13:53:20.209Z
Learning: In PR 237's TTS modality implementation approach, TTS functionality uses GOOGLE_AI_API_KEY (not GOOGLE_TTS_API_KEY) when using the google-ai provider. TTS is implemented as an output modality that leverages the existing google-ai provider authentication.
Learnt from: vigneshJuspay
Repo: juspay/neurolink PR: 693
File: src/lib/core/baseProvider.ts:490-517
Timestamp: 2025-12-18T15:13:28.435Z
Learning: In juspay/neurolink TTS integration (PR `#693`), when options.provider is "auto" and passed to TTSProcessor.synthesize, it will fail automatically during handler lookup since only concrete providers ("google-ai", "vertex") are registered as TTS handlers. No explicit validation against "auto" is needed—the implicit failure at handler registration lookup is by design.
Learnt from: vigneshJuspay
Repo: juspay/neurolink PR: 0
File: :0-0
Timestamp: 2025-12-15T18:35:37.783Z
Learning: In juspay/neurolink TTS implementation (PR `#691`), the new StreamChunk discriminated union type introduced in TTS-019 will be integrated with StreamResult.stream during the actual TTS streaming implementation PR (TTS-020/TTS-021), not in the type-definition PR. This phased approach keeps type updates and implementation changes atomic.
Learnt from: vigneshJuspay
Repo: juspay/neurolink PR: 693
File: src/lib/core/baseProvider.ts:490-517
Timestamp: 2025-12-18T15:13:28.435Z
Learning: In juspay/neurolink TTS architecture (PR `#693`), timeout handling for TTS synthesis is intentionally managed at the provider/handler level (within TTS handler implementations like GoogleTTSHandler), not at the BaseProvider orchestration level. This allows each provider to enforce its own timeout constraints appropriate to its synthesis capabilities.
Learnt from: RajuSudhar
Repo: juspay/neurolink PR: 173
File: src/lib/types/index.ts:58-62
Timestamp: 2025-09-17T18:14:34.960Z
Learning: RajuSudhar explained that in the Neurolink codebase, there are multiple ProviderConfig types causing inconsistency. One existing ProviderConfig type better suited the "ProviderConfig" name, so they renamed the less-suitable one to AIModelProviderConfig to free up the name. Adding backward compatibility aliases would worsen naming inconsistency rather than help. The remaining duplicates will be systematically deduplicated in the 07-Types-Module.md TODO as part of their phased refactor approach.
Learnt from: vigneshJuspay
Repo: juspay/neurolink PR: 693
File: src/lib/utils/ttsProcessor.ts:318-319
Timestamp: 2025-12-18T15:21:37.311Z
Learning: In juspay/neurolink TTS implementation (PR `#693`), timeout handling for TTSProcessor.synthesize() is intentionally delegated to individual TTS handler implementations (e.g., GoogleTTSHandler) rather than enforced at the TTSProcessor orchestration level. This design allows each provider to implement its own timeout constraints appropriate to its API characteristics and requirements.
💡 Use @coderabbitai plan to create a plan for code changes
Author
|
We are not using getSTTLanguages() and getSTTModels() right now as per the requirement. |
shambhavik-25
force-pushed
the
BZ-48219-add-speech-to-text-support-in-neuro-link
branch
from
February 20, 2026 11:22
2b55532 to
11000e2
Compare
murdore
requested changes
Feb 21, 2026
|
|
||
| - **(generateText):** add prepareStep and toolChoice passthrough support for multi-step agentic generation ([4cd340a](https://github.com/juspay/neurolink/commit/4cd340af7d39f72006d09fe86569232d751dcd8d)) | ||
|
|
||
| ## [9.9.0](https://github.com/juspay/neurolink/compare/v9.8.0...v9.9.0) (2026-02-17) |
| { | ||
| "name": "@juspay/neurolink", | ||
| "version": "9.10.0", | ||
| "version": "9.9.0", |
| */ | ||
| static createSTTCommands(): CommandModule { | ||
| return { | ||
| command: "stt <subcommand>", |
| private static registered = false; | ||
| private static options: ProviderRegistryOptions = { | ||
| enableManualMCP: false, // Default to disabled for safety | ||
| }; |
Contributor
There was a problem hiding this comment.
There are a lot of changes in your file. I feel I don't think you need to do so much change. Something seems wrong here.
Contributor
|
@coderabbitai the review approach needs to be changed. You need to identify recently merged pull requests where we have done multi modality additions for different different use cases and The recent documentation updates that we have done identify all the gaps in the current pull request related to documentation the process for documentation and the recent updates that we have done Compare the pull request against the multi modality integrations and the amount of changes we need we have done there |
|
Tip For best results, initiate chat on the files or code changes. 🧩 Analysis chain
🏁 Script executed: Length of output: 13919 🏁 Script executed: Length of output: 6778 🏁 Script executed: Length of output: 12346 🏁 Script executed: Length of output: 10209 🏁 Script executed: Length of output: 771
🏁 Script executed: Length of output: 31429 🏁 Script executed: Length of output: 7634 🏁 Script executed: Length of output: 6742 🏁 Script executed: Length of output: 37304 🏁 Script executed: Length of output: 17227
🔍 Gap Analysis: STT PR
|
| PR # | Feature | Merged Date | Pattern Quality |
|---|---|---|---|
| #824 | Video Analysis | Feb 17, 2026 | ⭐⭐⭐⭐⭐ Best Reference |
#809 |
File Processors | Feb 6, 2026 | ⭐⭐⭐⭐⭐ |
#810 |
RAG | Feb 8, 2026 | ⭐⭐⭐⭐⭐ |
#774 |
Video Generation (docs) | Jan 22, 2026 | ⭐⭐⭐⭐ |
| #693 | TTS | Dec 18, 2025 | ⭐⭐⭐⭐ Audio Modality |
❌ Critical Gaps Identified
1. CHANGELOG.md Entry Missing 🚨
Current State: Version 9.8.0 CHANGELOG has NO STT entry
Expected Pattern (from video-analysis #824):
## [9.8.0](https://github.com/juspay/neurolink/compare/v9.7.0...v9.8.0) (2026-02-17)
### Features
- **(stt):** add speech-to-text support in neurolink using Google Cloud Speech-to-Text ([commit-hash])Action Required:
# Add to CHANGELOG.md under version 9.8.0:
### Features
- **(stt):** add speech-to-text support in neurolink using Google Cloud Speech-to-Text with 7 specialized models, word-level timestamps, speaker diarization, and 8 audio format support2. memory-bank Implementation Plan Missing 🚨
Observed Pattern: ALL recent multi-modality PRs include implementation plans
Examples Found:
- ✅
memory-bank/tts-provider-implementation-plan.md(TTS) - ✅
memory-bank/video-analysis-implementation-plan.md(Video Analysis) - ✅
memory-bank/PPT-generation-implementation-plan.md(PPT) - ❌ Missing:
memory-bank/stt-implementation-plan.md
Action Required:
Create memory-bank/stt-implementation-plan.md with:
- Architecture overview
- Implementation strategy
- Provider integration approach
- CLI/SDK usage patterns
- Technical implementation details
- Testing strategy
Reference Template: Use memory-bank/tts-provider-implementation-plan.md structure
3. docs/features/index.md Not Updated 🚨
Current State: STT is NOT listed in the features index
Action Required:
Add STT to docs/features/index.md under "Latest Features (Q1 2026)" section:
| :material-microphone-variant: **[Speech-to-Text (STT)](stt.md)** | Convert audio to text with Google Cloud Speech-to-Text. 7 specialized models, word-level timestamps, speaker diarization, 8 audio formats. |4. Code Examples Missing 🚨
Observed Pattern:
- ✅ Video-analysis:
examples/video-analysis.ts - ✅ Image generation:
examples/image-generation.ts - ❌ STT: No example files
Action Required:
Create examples/stt-transcription.ts:
import { NeuroLink } from "@juspay/neurolink";
import { readFileSync } from "fs";
/**
* Speech-to-Text Examples
*
* Demonstrates:
* - Basic audio transcription
* - Word-level timestamps
* - Speaker diarization
* - Multiple audio formats
*/
const neurolink = new NeuroLink();
// Example 1: Basic Transcription
async function basicTranscription() {
const audioBuffer = readFileSync("./data/meeting.wav");
const result = await neurolink.generate({
input: {
text: "Transcribe this audio",
files: [audioBuffer],
},
provider: "google-ai",
stt: {
languageCode: "en-US",
enableAutomaticPunctuation: true,
},
});
console.log("Transcription:", result.transcription?.text);
console.log("Confidence:", result.transcription?.confidence);
}
// Example 2: Word-Level Timestamps
async function timestampedTranscription() {
const result = await neurolink.generate({
input: {
text: "Transcribe with timestamps",
files: ["./data/presentation.mp3"],
},
provider: "google-ai",
stt: {
languageCode: "en-US",
enableWordTimeOffsets: true,
enableWordConfidence: true,
},
});
result.transcription?.words?.forEach((word) => {
console.log(
`${word.word}: ${word.startTime}s - ${word.endTime}s (${word.confidence})`
);
});
}
// Example 3: Speaker Diarization
async function speakerDiarization() {
const result = await neurolink.generate({
input: {
text: "Transcribe meeting with speaker identification",
files: ["./data/conference-call.wav"],
},
provider: "google-ai",
stt: {
languageCode: "en-US",
model: "phone_call",
enableSpeakerDiarization: true,
diarizationSpeakerCount: 3,
},
});
console.log("Full Transcript:", result.transcription?.text);
console.log("\nSpeaker-tagged words:");
result.transcription?.words?.forEach((word) => {
console.log(`Speaker ${word.speakerTag}: ${word.word}`);
});
}
// Run examples
async function main() {
console.log("=== Basic Transcription ===");
await basicTranscription();
console.log("\n=== Timestamped Transcription ===");
await timestampedTranscription();
console.log("\n=== Speaker Diarization ===");
await speakerDiarization();
}
main().catch(console.error);Also create example audio file in examples/data/:
- Add a small sample audio file (public domain)
- Or add README explaining where to get test audio
5. README.md Feature Table Missing ⚠️
Pattern Inconsistency Observed:
- Video Generation: ✅ In README feature table
- RAG: ✅ In README feature table
- File Processors: ✅ In README feature table
- TTS: ❌ NOT in README (but has docs)
- Video Analysis: ❌ NOT in README (but has docs)
Recommendation: Add STT to README.md feature table to match video generation pattern:
| Feature | Version | Description | Guide |
| ---------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------- |
| **Speech-to-Text (STT)** | v9.8.0 | Audio transcription with Google Cloud Speech-to-Text. 7 models, word timestamps, speaker diarization, 8 formats (WAV, MP3, FLAC...) | [STT Guide](docs/features/stt.md) |Insert after "Video Generation with Veo" line (currently line 46).
📋 Documentation Quality Comparison
File Size & Structure:
| Feature | Doc Size | Sections | Example Code | Quality Score |
|---|---|---|---|---|
| TTS | 21,928 bytes | 47 sections | ✅ Extensive | ⭐⭐⭐⭐⭐ |
| STT | 19,241 bytes | 36 sections | ✅ Good | ⭐⭐⭐⭐ |
| Video Analysis | 5,015 bytes | Short, focused | ✅ Minimal | ⭐⭐⭐⭐ |
| Video Generation | 38,176 bytes | Comprehensive | ✅ Extensive | ⭐⭐⭐⭐⭐ |
Analysis: STT documentation is comprehensive and well-structured, comparable to TTS. Good coverage of features, configuration, troubleshooting.
Minor Improvement Suggestions:
- Add "Use Cases & Examples" section like TTS has
- Add pricing information section (like TTS)
- Add "Coming Soon" features section if applicable
✅ What's Correct in STT PR
- ✅ Documentation file exists -
docs/features/stt.md(comprehensive) - ✅ Architecture follows TTS pattern - Uses processor/handler pattern
- ✅ CLI integration - Has
createSTTCommands()for discovery - ✅ Type definitions - Complete STT types in
src/lib/types/sttTypes.ts - ✅ Error handling -
STT_ERROR_CODESandSTTErrorclass - ✅ Provider integration - Registered in
providerRegistry.ts - ✅ Exports - Properly exported in
src/lib/index.ts
🎯 Action Items Summary
🚨 Critical (Must Fix):
- Add CHANGELOG.md entry for v9.8.0 with STT feature description
- Create
memory-bank/stt-implementation-plan.md(use TTS as template) - Update
docs/features/index.mdto include STT in features table - Create
examples/stt-transcription.tswith usage examples
⚠️ Important (Should Fix):
- Update
README.mdfeature table to include STT - Update
docs-site/sidebars.tsif it exists in the repository
💡 Nice-to-Have:
- Add sample audio files to
examples/data/ - Enhance STT docs with pricing section (like TTS)
- Add "Use Cases & Examples" section to docs
📝 Comparison: TTS vs STT Patterns
| Aspect | TTS (#693) |
STT (#793) |
Match? |
|---|---|---|---|
| Documentation | ✅ Yes | ✅ Yes | ✅ |
| Implementation plan | ✅ Yes | ❌ No | ❌ |
| CHANGELOG entry | ✅ Yes | ❌ No | ❌ |
| README update | ❌ No | ❌ No | ✅ |
| Features index | ❌ No | ❌ No | ✅ |
| Code examples | ❌ No | ❌ No | ✅ |
| Processor pattern | ✅ Yes | ✅ Yes | ✅ |
| CLI discovery | ✅ Yes | ✅ Yes | ✅ |
Key Difference: Video-analysis (#824) is the better pattern - includes implementation plan and examples.
🔗 Reference Documents to Study
-
Best Overall Pattern: PR
#824(video-analysis)- Implementation plan:
memory-bank/video-analysis-implementation-plan.md - Example:
examples/video-analysis.ts - CHANGELOG: Version 9.9.0 entry
- Implementation plan:
-
Audio Modality Reference: PR
#693(TTS)- Implementation plan:
memory-bank/tts-provider-implementation-plan.md - Documentation:
docs/features/tts.md
- Implementation plan:
-
Documentation Standard: PR
#774(video-generation docs)- Comprehensive guide:
docs/features/video-generation.md - README integration
- Comprehensive guide:
Would you like me to generate the missing files (implementation plan, examples, CHANGELOG entry) using the `` tag?
🧠 Learnings used
Learnt from: vigneshJuspay
Repo: juspay/neurolink PR: 693
File: src/lib/core/baseProvider.ts:490-517
Timestamp: 2025-12-18T15:13:28.435Z
Learning: In juspay/neurolink TTS integration (PR `#693`), when options.provider is "auto" and passed to TTSProcessor.synthesize, it will fail automatically during handler lookup since only concrete providers ("google-ai", "vertex") are registered as TTS handlers. No explicit validation against "auto" is needed—the implicit failure at handler registration lookup is by design.
Learnt from: vigneshJuspay
Repo: juspay/neurolink PR: 237
File: memory-bank/tts-provider-implementation-plan.md:92-106
Timestamp: 2025-11-17T13:53:20.209Z
Learning: In PR 237's TTS modality implementation approach, TTS functionality uses GOOGLE_AI_API_KEY (not GOOGLE_TTS_API_KEY) when using the google-ai provider. TTS is implemented as an output modality that leverages the existing google-ai provider authentication.
Learnt from: RajuSudhar
Repo: juspay/neurolink PR: 173
File: src/lib/types/index.ts:58-62
Timestamp: 2025-09-17T18:14:34.960Z
Learning: RajuSudhar explained that in the Neurolink codebase, there are multiple ProviderConfig types causing inconsistency. One existing ProviderConfig type better suited the "ProviderConfig" name, so they renamed the less-suitable one to AIModelProviderConfig to free up the name. Adding backward compatibility aliases would worsen naming inconsistency rather than help. The remaining duplicates will be systematically deduplicated in the 07-Types-Module.md TODO as part of their phased refactor approach.
Learnt from: vigneshJuspay
Repo: juspay/neurolink PR: 0
File: :0-0
Timestamp: 2025-12-15T18:35:37.783Z
Learning: In juspay/neurolink TTS implementation (PR `#691`), the new StreamChunk discriminated union type introduced in TTS-019 will be integrated with StreamResult.stream during the actual TTS streaming implementation PR (TTS-020/TTS-021), not in the type-definition PR. This phased approach keeps type updates and implementation changes atomic.
Learnt from: sudharsan-juspay
Repo: juspay/neurolink PR: 160
File: src/lib/providers/index.ts:43-44
Timestamp: 2025-09-10T08:22:11.910Z
Learning: In the Neurolink project, type deduplication across modules (like ProviderName definitions) should be handled as separate tasks rather than mixed with other refactoring efforts, as there are multiple such occurrences throughout the codebase that need systematic cleanup.
Learnt from: vigneshJuspay
Repo: juspay/neurolink PR: 693
File: src/lib/core/baseProvider.ts:490-517
Timestamp: 2025-12-18T15:13:28.435Z
Learning: In juspay/neurolink TTS architecture (PR `#693`), timeout handling for TTS synthesis is intentionally managed at the provider/handler level (within TTS handler implementations like GoogleTTSHandler), not at the BaseProvider orchestration level. This allows each provider to enforce its own timeout constraints appropriate to its synthesis capabilities.
Learnt from: CR
Repo: juspay/neurolink PR: 0
File: CLAUDE.md:0-0
Timestamp: 2026-02-14T02:42:22.821Z
Learning: Applies to test/**/*.test.ts : Mock external API calls for unit tests. Use real API calls sparingly in integration tests. Test provider consistency across all providers and validate multimodal content handling.
Learnt from: RajuSudhar
Repo: juspay/neurolink PR: 0
File: :0-0
Timestamp: 2025-11-04T22:14:18.719Z
Learning: In the juspay/neurolink repository, do not flag existing type or interface definitions located outside src/lib/types/ - these are part of a phased migration plan and will be addressed in upcoming PRs. Only enforce type centralization rules on new code going forward.
💡 Use @coderabbitai plan to create a plan for code changes
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
feat(stt): add speech-to-text support in neurolink using google cloud speech-to-text
Pull Request
Description
What does this PR do?
Adds integrated Speech-to-Text (STT) support to NeuroLink using Google Cloud Speech-to-Text. Enables high-accuracy audio transcription in 125+ languages, with advanced features like word-level timestamps and model selection.
Related Issues
Does this PR close any issues?
Closes #48219
Type of Change
Please select the type of change:
Motivation and Context
Why is this change needed? What problem does it solve?
Changes Made
What specific changes were made?
Added STT types, runtime type guards, and error codes.
STTProcessororchestrator andGoogleSTTHandler.generate()flow.--stt-language,--stt-model, and related flags.Breaking Changes
Does this PR introduce breaking changes?
Testing
How has this been tested?
Please describe the tests you ran and their results:
google-aiTest Coverage
Code Quality
Have you followed code quality standards?
Documentation
Have you updated documentation?
Commit Message Format
Does your commit follow semantic commit conventions?
type(scope): descriptionExample:
feat(providers): add support for LiteLLM proxyDependencies
Does this PR add, update, or remove dependencies?
If yes, list dependencies and justification:
Performance Impact
Does this change affect performance?
If applicable, provide benchmark results:
Security Considerations
Are there any security implications?
If applicable, describe:
Deployment Notes
Special deployment instructions?
Reviewer Checklist
For reviewers:
Pre-submission Checklist
Before submitting, ensure you have:
pnpm testpnpm buildpnpm run validate:alland all checks passThank you for contributing to NeuroLink!
Summary by CodeRabbit
New Features
Documentation