Skip to content

A Chinese date with spaces around its parts is read, so "2024 年 3 月 15 日" becomes 2024-03-15 instead of no value - #1436

Open
rudycelekli wants to merge 1 commit into
deeplethe:devfrom
rudycelekli:extract-spaced-chinese-date
Open

rudycelekli wants to merge 1 commit into
deeplethe:devfrom
rudycelekli:extract-spaced-chinese-date

Conversation

@rudycelekli

Copy link
Copy Markdown
Contributor

Why

written_date reads Chinese dates through year_first_date, which splits on 年 / 月 / 日 and requires every part to be ASCII digits. Text pulled out of a PDF very often keeps a space on each side of the characters: "2024 年 3 月 15 日" is how many government notices, filings and annual reports come out of the text layer. The parts are then "2024 ", " 3 ", " 15 ", the digit check fails, and the date is not read. The same day written "2024年3月15日" works.

So a date attribute quoted from such a PDF is dropped, an interval endpoint is left open, and literal_object lets "2024 年 3 月 15 日" through as a thing instead of a value.

What changes

Each part is trimmed before the digit check. Nothing else changes: the year must still come first, the month-only and year-only forms still need their 月 / 年, and an impossible day is still no value.

How it was checked

  • A new test, a_chinese_date_with_spaces_around_its_parts_is_read, covers "2024 年 3 月 15 日", "2024年 3月 15日", "2024 年 3 月", "1982 年", read_time on the spaced form, and that "3 月 15 日" and "2024 年 2 月 30 日" stay unread.
  • With the fix reverted the test fails: left: None right: Some(String("2024-03-15")).
  • cargo test -p utopia-extract --lib: 113 passed.
  • cargo fmt -p utopia-extract --check and cargo clippy -p utopia-extract --all-targets -- -D warnings pass.

Before review

  • Every commit is signed off (git commit -s)
  • cargo fmt --all --check, cargo clippy and cargo test pass for the touched crate

…5 日" becomes 2024-03-15 instead of no value

Signed-off-by: Rudy Celekli <rudy@gradiahq.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant