Repository navigation
A Chinese date with spaces around its parts is read, so "2024 年 3 月 15 日" becomes 2024-03-15 instead of no value - #1436
Open
rudycelekli wants to merge 1 commit into
Conversation
…5 日" becomes 2024-03-15 instead of no value Signed-off-by: Rudy Celekli <rudy@gradiahq.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
written_datereads Chinese dates throughyear_first_date, which splits on 年 / 月 / 日 and requires every part to be ASCII digits. Text pulled out of a PDF very often keeps a space on each side of the characters: "2024 年 3 月 15 日" is how many government notices, filings and annual reports come out of the text layer. The parts are then"2024 "," 3 "," 15 ", the digit check fails, and the date is not read. The same day written "2024年3月15日" works.So a date attribute quoted from such a PDF is dropped, an interval endpoint is left open, and
literal_objectlets "2024 年 3 月 15 日" through as a thing instead of a value.What changes
Each part is trimmed before the digit check. Nothing else changes: the year must still come first, the month-only and year-only forms still need their 月 / 年, and an impossible day is still no value.
How it was checked
a_chinese_date_with_spaces_around_its_parts_is_read, covers "2024 年 3 月 15 日", "2024年 3月 15日", "2024 年 3 月", "1982 年",read_timeon the spaced form, and that "3 月 15 日" and "2024 年 2 月 30 日" stay unread.left: None right: Some(String("2024-03-15")).cargo test -p utopia-extract --lib: 113 passed.cargo fmt -p utopia-extract --checkandcargo clippy -p utopia-extract --all-targets -- -D warningspass.Before review
git commit -s)cargo fmt --all --check,cargo clippyandcargo testpass for the touched crate