Skip to content

CONTRIBUTING.md

Latest commit

 

History

History
216 lines (160 loc) · 10 KB

CONTRIBUTING.md

File metadata and controls

216 lines (160 loc) · 10 KB
Raw

Contributing Content

This guide explains how to add or update datasheet content for your language.

What Are Datasheets?

Datasheets describe each language dataset in Common Voice. They include information about the language, its writing system, text sources, and community contributors.

How to Contribute

Via Pull Request (preferred)

  1. Fork this repository
  2. Look at content/_template/ for the directory structure and file list
  3. Look at content/_example/ for a filled-in reference
  4. Create or edit files under content/locales/{your-locale-code}/
  5. Submit a Pull Request

Via Google Form

Datasheets can also be submitted via Google Form (English and Spanish):

Scripted/Read Speech:

Spontaneous Speech:

Via Email

If you cannot access GitHub or the form, email the Common Voice team at commonvoice@mozilla.com.

Directory Structure

content/locales/{locale}/
  shared/              # Language-level (shared across SCS and SPS)
    description.md       About the language
    variants.md          Dialect / accent variants
    accents.md           Accent options
    writing_system.md    Writing system description
    alphabet.md          Symbol table
    community_links.md   Community resource links

  scripted/            # Scripted Speech (SCS) only
    corpus.md            Text corpus description
    sources.md           Corpus sources
    text_domain.md       Text domain descriptions
    processing.md        Text processing applied
    postprocessing.md    Recommended post-processing
    discussion_links.md  Discussion links
    contribute_links.md  Additional contribute links
    authors.md           Datasheet authors
    citation.md          Citation guidelines
    funding.md           Non-OMSF or additional funding

  spontaneous/         # Spontaneous Speech (SPS) only
    transcriptions.md    Transcription process description
    postprocessing.md    Recommended post-processing
    discussion_links.md  Discussion links
    contribute_links.md  Additional contribute links
    authors.md           Datasheet authors
    citation.md          Citation guidelines
    funding.md           Non-OMSF or additional funding

Guidelines

  • Only create files you have content for. Empty files are ignored.
  • shared/ fields apply to both SCS and SPS. Write them once here instead of duplicating across scripted/ and spontaneous/.
  • Write plain Markdown -- no Jinja2 or HTML needed.
  • List items (bullets, numbered lists) are deduplicated at compile time as a safety net.

Field Edit Modes

Datasheet content is built by the community over time. Fields have two edit modes that indicate how they should be maintained.

Additive fields -- extend, do not replace:

Field Why it grows
sources.md New text sources are added across releases
community_links.md New community spaces are created
authors.md New contributors and community leads join over time
citation.md New publications reference the dataset
funding.md Additional funders may support the project

When editing additive fields, add your entries below the existing content. Do not remove previous contributions -- they represent the work of earlier community members. Include version references where helpful (e.g. v24.0, v22.0-present).

Descriptive fields -- rewrite or improve freely:

Field What it describes
description.md The language itself
variants.md Dialect / accent variants
accents.md Accent options
corpus.md Text corpus description
text_domain.md Text domain descriptions
transcriptions.md Transcription process
writing_system.md How the language is written
alphabet.md Symbol table or character list
processing.md Current text processing pipeline
postprocessing.md Current recommended post-processing
discussion_links.md Current discussion channels
contribute_links.md Current contribution paths

These fields describe the current state. They can be rewritten entirely when the understanding improves -- they are not a historical record.

Mergeable fields -- community content + auto-generated data:

Some fields are also mergeable: the final datasheet includes auto-generated data alongside your content. Your community content is optional -- if you provide it, your text appears before the auto-generated data. If you don't provide it, only the auto-generated data appears. Each mergeable field is still either additive or descriptive -- follow the edit rules above.

Field Edit mode Auto-generated data
sources.md additive Per-source counts (bundler)
corpus.md descriptive Corpus statistics (bundler)
text_domain.md descriptive Domain breakdown (bundler)
variants.md descriptive Per-variant recording stats (bundler)
accents.md descriptive Per-accent recording stats (bundler)
transcriptions.md descriptive Transcription quality metrics (bundler)
community_links.md additive Pontoon + Communities links (template)
discussion_links.md descriptive Matrix/Discourse/Discord/Telegram links (template)
contribute_links.md descriptive Speak/Write/Listen/Review links (template)

See content/_example/ for examples, particularly scripted/sources.md and scripted/authors.md for the additive style.

Fields With Auto-Defaults

Some fields always include default links in the final datasheet. Your community content appears before the defaults. Do not duplicate the default links in your files.

  • community_links.md -- Pontoon translators link + Communities page are always included
  • discussion_links.md -- Matrix, Discourse, Discord, and Telegram links are always included
  • contribute_links.md -- Standard contribution links are always included (Speak/Write/Listen/Review for SCS, question/transcription links for SPS)
  • funding.md -- OMSF funding text is added for eligible locales (only when no community file exists)

How to Add a Translated Datasheet

By default, datasheets are generated in English. You can add support for a new language so that speakers of that language see the datasheet in their own language.

A translated datasheet requires two things:

  1. i18n strings -- section titles and boilerplate text in an i18n JSON file
  2. Template language mapping -- telling the compile script which locales use the new language

The compile script auto-discovers available template languages by scanning templates/i18n/*.json. No code changes are needed -- just add the files.

Step 1: Add i18n Strings

Create a new file in templates/i18n/{lang-code}.json by copying templates/i18n/en.json and translating all values. Keep the JSON keys unchanged.

The compile script detects which modalities the translation supports by checking for header_intro_scs and/or header_intro_sps keys. To support both SCS and SPS, include both keys. To support only one modality, include only the relevant key.

Use the English file as the reference for the complete list of keys. All shared keys (titles, labels) must be present; modality-specific keys can be omitted for unsupported modalities.

Existing translations:

File Language Keys Coverage
en.json English 55 Complete (SCS + SPS)
es.json Spanish 55 Complete (SCS + SPS)
zh-TW.json Traditional Chinese 55 Complete (SCS + SPS)

Step 2: Map Locales to the Template Language

Edit metadata/template-languages.json to assign locales to your new template language. Only non-English locales need entries -- English is the default.

{
  "_comment": "Template language overrides. Locales not listed here default to 'en'.",
  "fr": "fr",
  "fr-CA": "fr"
}

Step 3: Add Translated Defaults (optional)

If you want default link text in the new language, add the translated community_links_defaults, discussion_links_defaults, scs_contribute_links_defaults, and sps_contribute_links_defaults keys to your i18n JSON file. See en.json for the pattern.

For OMSF funding text, add a translated string to OMSF_FUNDING_TEXT in compile_datasheets.py.

Step 4: Compile and Verify

python3 compile_datasheets.py {version} \
    --api-snapshot metadata/api-snapshots/languagedata-{date}.json \
    --pretty

Check that the output JSON contains your new template in templates.scs.{lang} (or templates.sps.{lang}) and that the mapped locales reference the correct template_language.