Double ended iterator

[psg-19]

Addresses #6996

Summary :-

Adding another field right_boundary_property to iterate from the end, and trying to implement DoubleEndedIterator for Utf16Indices and Latin1Indices

• Added right_boundary_property to grapheme.rs, word.rs and sentence.rs .
• Updated the iterator macro to expose next_back(), allowing wrapper segmenters to delegate reverse iteration in iterator_helper.rs .
• Implemented next_back() function for Utf16Indices to allow construction of Unicode code points from the end of the string backwards while correctly merging surrogate pairs. (in indices.rs)
• Implemented next_back() function for Latin1Indices which yields byte-index and <u8> character pairs starting from the end. (in indices.rs)
• Added DoubleEndedIterator for RuleBreakIterator implementation, it enables finding segmentation boundaries in reverse, iterating from the end of the text using rule-based logic.

[sffc]

Thanks for the work; we've been needing this for a long time. We need to be super careful that this produces correct results.

@makotokato @aethanyc @eggrobin

[sffc]

I think you need to consider more than 1 code point back. See .next()

[sffc]

Please add more test coverage; .next() and .next_back() should produce identical breakpoints. Just take a bunch of real test data (which we should already have) and run it both ways and check that the vectors are the same (just reversed).

[psg-19]

thanks for the review, started working on it

[psg-19]

Hi @sffc,

What I added :-

• impl DoubleEndedIterator for LineBreakIterator This implementation enables reverse iteration for
  LineBreakIterator, fulfilling the requirement to support DoubleEndedIterator across all segmenters. It currently provides a stub implementation that delegates to the cache.

• get_break_property_with_state It simulates the state machine in reverse by scanning backwards to a safe point which is Start of Text and re-running the forward state logic to determine the correct break property and backtrack length for the current position, handling complex looking-ahead rules that simple table lookups miss.

• Added code in spec_test.rs to test on real data as you suggested.

• Although my implementation fails for 4 testcases :-

icu4x> cargo test --test spec_test
running 12 tests
test run_grapheme_break_extra_test ... ok
test run_sentence_break_extra_test ... FAILED
test run_sentence_break_test ... FAILED
test run_word_break_extra_test ... ok
test run_grapheme_break_test ... ok
test run_line_break_extra_test ... ok
test run_word_break_test ... ok
test run_sentence_break_random_test ... FAILED
test run_line_break_test ... ok
test run_line_break_random_test ... ok
test run_word_break_random_test ... FAILED
test run_grapheme_break_random_test ... ok

failures:
  run_sentence_break_extra_test
    run_sentence_break_random_test
    run_sentence_break_test
    run_word_break_random_test

test result: FAILED. 8 passed; 4 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.81s

Its very complex to mimic the logic used in the forward iteration, can you help me out reviewing my current code and give me some insight. The DoubleEndedIterator for LineBreakIterator does not implement full reverse logic with LB9 support.