Skip to content

Use the display title - critical for low-resource languages #4

Description

@bittlingmayer

Some Wikipedias use DISPLAYTITLE to override the titles of almost all articles. Typically this is in the case of a low language in a high-low disglossia, a good example would be Alemannic (~"Swiss German").

For example, see https://als.wikipedia.org/wiki/Zürich:

Screenshot 2020-01-17 at 15 33 24

(The URL using the Standard German (de) Zürich instead of the actual Alemannic (als) Züri is a workaround for the fact that Alemannic has no single standardised orthography, so it's more practical to allow searches and lookups in the standard language.)

Currently, the actual output extracted is Zürich, but the expected output is Züri.

So in order to build a viable parallel titles corpus for such a language, we need to prefer DISPLAYTITLE and only take the underlying title if DISPLAYTITLE is unset.

(Not sure what the default should be , but it's probably good to make it an option not a hard rule, because for example for building a corpus for translation from als to en it's often useful to additionally include the de to en data, because of how often de segments occurs in real als data.)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions