preprocess: strip bidi control chars from values; map a revised welfare header

Found via CI's first real run against freshly-downloaded data:

- fix_value() flagged a batch of legitimate decimals as "BAD FLOATS"
  because Excel wraps numbers in invisible RTL-embedding/pop-directional-
  formatting Unicode control characters in Hebrew sheets - float() can't
  parse a string containing them even though the digits are fine. Also
  folded in a stray '. .' null-sentinel found in the same set.
- CBS revised a source file between my local download and CI's fresh
  one, surfacing a new raw header variant for the income-security-
  recipients metric with different sheet-prefix/end-of-year phrasing.
  Reviewed against the canonical's other 6 orig variants (which already
  cover the same prefix/phrasing pattern for the "during the year"
  counterpart) and confirmed it's the same metric, not a new one.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Author: akariv

Lamas/data/header_mapping.yaml

@@ -3584,6 +3584,7 @@ headers:
     - מקבלי הבטחת הכנסה (נפשות, סוף שנה)
     - רווחה/מקבלי גמלת הבטחת הכנסה (נפשות, בסוף השנה)
     - שכר ורווחה/מקבלי גמלת הבטחת הכנסה (נפשות, במשך השנה)
+    - שכר ורווחה/מקבלי גמלת הבטחת הכנסה (נפשות, סוף שנה)
   שכר ורווחה - מקבלי הכשרה מקצועית (סה"כ):
     orig:
     - מקבלי הכשרה מקצועית - סה"כ

Lamas/lamas/preprocess.py

@@ -20,6 +20,11 @@
 ]
 SPACES = re.compile(r'\s+')
 LETTERS = re.compile('[א-ת]')
+# Excel embeds invisible Unicode bidi control characters (RIGHT-TO-LEFT EMBEDDING U+202B, POP
+# DIRECTIONAL FORMATTING U+202C, and similar) around numbers in RTL Hebrew sheets to force correct
+# visual rendering direction - found for real, flagged as "BAD FLOATS" because float() can't
+# parse a string with invisible control characters in it even though the visible digits are fine.
+BIDI_CONTROL_CHARS = re.compile('[‎‏‪-‮⁦-⁩]')
 MAGICS = [re.compile(x) for x in MAGICS_RAW]
 MIN_SIZE = 30
 HEADER_SIZE = 5
@@ -67,7 +72,9 @@ def get_safe(data, r, c):
 
 
 def fix_value(v, bad_floats):
-    if v in ('-', '..', '', '.', None):
+    if v is not None:
+        v = BIDI_CONTROL_CHARS.sub('', v).strip()
+    if v in ('-', '..', '', '.', '. .', None):
         return None
     if 'http' in v:
         return None

Lamas/tests/test_preprocess.py

@@ -0,0 +1,37 @@
+from lamas.preprocess import fix_value
+
+
+def test_strips_bidi_control_characters_around_numbers():
+    # Excel wraps numbers in invisible RTL-embedding/pop-directional-formatting control
+    # characters in Hebrew (RTL) sheets - found for real, flagged as "BAD FLOATS" because
+    # float() can't parse a string containing them even though the visible digits are fine.
+    bad_floats = set()
+    assert fix_value('‫22.9‬', bad_floats) == '22.9'
+    assert fix_value('‫5.5 ‬', bad_floats) == '5.5'
+    assert bad_floats == set()
+
+
+def test_dot_space_dot_treated_as_null():
+    # A stray malformed null-sentinel found in the raw data (likely meant to be '..').
+    bad_floats = set()
+    assert fix_value('. .', bad_floats) is None
+    assert bad_floats == set()
+
+
+def test_ordinary_sentinels_still_null():
+    bad_floats = set()
+    for v in ('-', '..', '', '.', None):
+        assert fix_value(v, bad_floats) is None
+    assert bad_floats == set()
+
+
+def test_genuinely_unparseable_value_still_flagged():
+    bad_floats = set()
+    fix_value('=Q94/R94100', bad_floats)
+    assert '=Q94/R94100' in bad_floats
+
+
+def test_hebrew_text_value_passed_through_unchanged():
+    bad_floats = set()
+    assert fix_value('טקסט כלשהו', bad_floats) == 'טקסט כלשהו'
+    assert bad_floats == set()