Skip to content

A full-width yen sign is read as the currency, so "¥5,000万" is 50000000 instead of no value - #1442

Open
rudycelekli wants to merge 1 commit into
deeplethe:devfrom
rudycelekli:extract-full-width-yen
Open

rudycelekli wants to merge 1 commit into
deeplethe:devfrom
rudycelekli:extract-full-width-yen

Conversation

@rudycelekli

Copy link
Copy Markdown
Contributor

Why

currency_unit maps the symbols it knows to one canonical symbol. Chinese financial text (contracts, invoices, prospectuses, price lists) usually writes amounts with the full-width yen sign ¥ (U+FFE5), not ¥ (U+00A5). ¥ is not in the table, so scan_quantity finds no currency, the number scan starts at ¥ and reads nothing:

  • "¥5,000万" under a number attribute is no value, while "¥5,000万" is 50000000.
  • "¥5,000万" under 万元 is no value instead of 5000.
  • literal_object("¥5000万") is false, so an amount written as an object becomes an entity named "¥5000万".

What changes

¥ is read as ¥, the same symbol in its full-width form. Nothing else changes: "¥" stays ambiguous between yen and yuan as before, and a mismatch with a declared currency ("¥5,000万" under 万美元) is still no value.

How it was checked

  • A new test, a_full_width_yen_sign_is_read_as_the_currency, covers "¥5,000万" with no unit and under 万元, "¥1,280" under 元, the mismatch under 万美元, and literal_object("¥5000万").
  • With the fix reverted the test fails: left: None right: Some(Number(50000000.0)).
  • cargo test -p utopia-extract --lib: 113 passed.
  • cargo fmt -p utopia-extract --check and cargo clippy -p utopia-extract --all-targets -- -D warnings pass.

Before review

  • Every commit is signed off (git commit -s)
  • cargo fmt --all --check, cargo clippy and cargo test pass for the touched crate

…00 instead of no value

Signed-off-by: Rudy Celekli <rudy@gradiahq.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant