Vietnamese month uncertainty (discussion)

[Allactaga]

We need to come up with generic solution for this.

Vietnamese language does not have names for months and simply use "Month One", "Month Two" etc.
Some sites use numeric form like "Month 1", "Month 2" etc. So when we translate tokens from Vietnamese for dates like "1 Year 1 Month 1 Day" it is not quite clear whether it is "1 year 1 month 1 day" or "1 year 1 January day".

[Liechti]

Chinese, Japanese, and Korean (and probably others) need a solution for this too.

One simplistic way to do this is to have a lookup table like I put in this comment in #14. Keep a list of the Year, Month, and Day tokens for each language(ex: '年', '月', '日' in Chinese). The data could be pulled in from a formatted file like in #26.

[Allactaga]

But how we would understand when to treat "Token Number" as an ordinal form of token, and when as two separate words?

[Liechti]

In Chinese at least (can't speak for Japanese/Korean as well), ordinal numbers are prefixed with 第。
So，
第一 first
第二 second

第 will never appear in a yyyy年mm月dd日 context. You will commonly see

第一個月 the first month (not January, just an arbitrary month)
第一年 the first year
第一天 the first day

but they don't refer to absolute dates/times, only relative. Does that make sense?

Edit: if you wanted to say something like "one year and three months later", it would look like this:
一年三個月以後 (以後＝after/later). In this case the measure word 個 tells you it is not a date, but rather expressing a length of time.

[asadurski]

My research shows that in Vietnamese this distinction would be achieved by word order, so "tháng sáu" is "June", "sáu tháng" is "six months". But I still need a native speaker to confirm this...
If that causes problems only in Vietnamese, then perhaps using entries like (?<!(tháng|thứ)\s)năm in languages.yaml is enough for now to exclude those mixed possibilities?

[Allactaga]

@asadurski not sure if i understand what are you suggesting, could you please provide a piece of code?
Also would it work with digits?

[asadurski]

After rethinking and initial testing, I think my approach will not work.
What I wanted to do is using regex in language data to restrict "year" and "month" to mean "year" only if it has digits before it, like this:

year:
        - (?<=\d\s)năm
        - (?<=một\s)năm
        - (?<=hai\s)năm
month:
        - (?<=\d\s)tháng

and so forth, but currently only the "simplifications" part of language data is taken as regex, right?
Second weakness of this approach is that it only helps a little and will not solve the entire problem - it would need to be coupled with formats.

[waqasshabbir]

I think our greatest aid is the keyword ago. No native speaker would write relative date without adding 'ago' considering the duplicity of meaning that would ensue otherwise. Hence, we probably need an approach where we can switch between relative and absolute dates context on the basis of the presence of keyword ago in the datestring for such cases. If ago is present, the date should be considered as relative and only be translated for calendar literals (day/week/month/year) instead of month names.

[Allactaga]

I agree, 'ago' would be proper solution. But we would need to collect dates parsed particularly with freshness parser and see if they all contain 'ago'.

[zxq9]

@Liechti As for Japanese, the rule is extremely similar. In Korean it is somewhat less common to see dates written with Chinese characters (sometimes hangul, though).

Japan has one extra issue I haven't checked whether is handled here: imperial dates. These are very easy to detect when they have the era name (appearing, for example, as 平成, 平, "Heisei", or "H." followed by date characters) but require a conversion table to translate to an ISO year. Sometimes they are not labeled, so one has to assume that a year value less than, say, 100 is actually a current-era imperial year (sometimes which era was "current" at the time of a document's creation is only discernable with some extra context, though).

Numbers can be tricky because years may be in traditional characters, half-width western characters, or full-width western characters (this is a perennial issue in interpreting Japanese numbers, though, only ever solved very well by game libraries, tbh -- that is to say, Japanese users will not be surprised in the slightest for full functionality to be lacking, but will be pleasantly surprised if it exists).

[noviluni]

I reopen this issue as it seems that it was closed by error