Summary
Email subject and snippet cleanup can remove meaningful trailing Unicode characters. A decomposed accented character such as e followed by U+0301 COMBINING ACUTE ACCENT loses its accent, and an emoji variation selector can also be stripped.
Current implementation
remove_trailing_special_chars in services/email_service/src/util/process_pre_insert/clean_message/clean_subject_snippet.rs classifies the full Unicode combining-mark ranges and variation-selector range as invisible. It then truncates everything after the last character outside those ranges.
Combining marks and variation selectors modify the preceding visible character, so removing them changes user-authored content.
Expected result
- Trailing combining marks are preserved.
- Variation selectors are preserved.
- Existing cleanup of trailing whitespace, zero-width spaces, joiners, BOM, word joiner, and similar formatting noise remains intact.
- Characters in the middle of a subject or snippet remain unchanged.
Suggested direction
Restrict the trailing-noise predicate to actual whitespace and formatting controls already targeted by the function, and add focused regression tests.
Non-goals
- Adding Unicode normalization.
- Rewriting or transliterating subjects and snippets.
- Changing leading or internal whitespace handling.
Summary
Email subject and snippet cleanup can remove meaningful trailing Unicode characters. A decomposed accented character such as
efollowed by U+0301 COMBINING ACUTE ACCENT loses its accent, and an emoji variation selector can also be stripped.Current implementation
remove_trailing_special_charsinservices/email_service/src/util/process_pre_insert/clean_message/clean_subject_snippet.rsclassifies the full Unicode combining-mark ranges and variation-selector range as invisible. It then truncates everything after the last character outside those ranges.Combining marks and variation selectors modify the preceding visible character, so removing them changes user-authored content.
Expected result
Suggested direction
Restrict the trailing-noise predicate to actual whitespace and formatting controls already targeted by the function, and add focused regression tests.
Non-goals