Skip to content

feat(tts): support API-based voice cloning #2191

Description

@qbc2016

Background

AgentScope supports API-based speech synthesis, but creating a custom voice from reference audio currently requires users to interact with the provider outside AgentScope.

This issue adds an API-based voice enrollment workflow and makes the resulting voice ID directly usable by the existing TTS models.

This work is tracked by #.

Task Breakdown

  • Add a provider-facing voice cloning abstraction that is separate from speech synthesis.
  • Define common request and response models containing:
    • Provider and target model.
    • Voice name or prefix.
    • Reference audio input.
    • Provider-side voice ID.
    • Voice status and provider metadata.
  • Implement DashScope voice cloning as the initial provider:
    • Support Qwen TTS voice enrollment.
    • Support CosyVoice voice enrollment.
    • Support create, list, query, update, and delete operations where available.
  • Support provider-compatible reference audio inputs:
    • Local audio or base64/data URI where accepted by the provider.
    • Publicly accessible audio URL where required.
  • Validate that the selected target model supports voice cloning.
  • Allow the returned voice ID to be passed through the existing TTS voice parameter.
  • Mark supported models with voice_cloning: true in their model cards.
  • Return actionable errors for unsupported models, invalid audio, quota errors, and provider failures.
  • Ensure API keys and raw reference audio are never written to logs.
  • Add mocked unit tests without requiring live provider calls.
  • Add an end-to-end example covering:
    1. Creating a voice from reference audio.
    2. Using the returned voice ID for synthesis.
    3. Querying or deleting the voice.
  • Document that users must have permission to clone and use the reference voice.

Acceptance Criteria

  • A reference audio sample can be submitted through AgentScope and produces a provider voice ID.
  • The voice ID can be used by an existing AgentScope TTS model to synthesize audio.
  • Supported voice-management operations are covered by unit tests.
  • Unsupported providers and models fail with clear error messages.
  • Existing preset voices remain fully compatible.
  • Unit tests do not perform external network requests.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Status
    In Progress

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions