Symptom
pipelines/knesset/ethics_committee_decisions.py produces an empty index.csv (0 rows). Published datapackage at next.obudget.org/datapackages/knesset/ethics_committee_decisions/datapackage.json reports count_of_rows: 0 and has been frozen since 2025-09-01.
Root cause
Same as #243 — main.knesset.gov.il/Activity/committees/Ethics/Pages/CommitteeDecisionsPast.aspx and the per-Knesset CommitteeDecisions{NN}.aspx pages all return the Radware Bot Manager JavaScript challenge (HTTP 247, rbzns body) instead of the real HTML when accessed by the pipeline's Selenium-via-SSH driver. The CSS selectors in the pipeline (tr, .link-item, .ComEthicsTdDate, a[href$=".pdf"]) find nothing on the challenge body, so the loop never executes.
$ curl -sw '%{http_code}\n' \
'https://main.knesset.gov.il/Activity/committees/Ethics/Pages/CommitteeDecisionsPast.aspx' \
-o /tmp/r.html
247
$ head -c 200 /tmp/r.html
<!DOCTYPE html><html><head><meta charset="utf-8">
<script type="text/javascript" src="/kramericaindustries.ac_v2.lib.js"></script>
<script type="text/javascript">;;window.rbzns={"seed":"…"};winsocks();
</script></head><body></body></html>Why I'm not opening this as part of #243
#243 is specifically about the legal_advisor + letters pipelines and proposes Radware-bypass approaches for them. This issue is identical in root cause but attached to a different output, so I'm filing it as a tracker so it doesn't get lost when #243 is fixed. Once the Radware bypass lands (whichever path #243 settles on), this pipeline should resume working with no code changes — the existing CSS selectors should match the post-challenge real HTML.
Why an OData migration won't work for this feed
Worth noting up front: I checked whether ethics-committee decisions are reachable through the KNS_* OData endpoints (the path used by knesset_committee_decisions and proposed for that pipeline in #242). They are not:
- All 20 ethics committees across Knessets 13–25 (
Name contains 'אתיקה', CommitteeTypeID=71) are queryable via KNS_Committee.
KNS_CommitteeSession returns 0–3 sessions per ethics committee.- For every one of those sessions,
KNS_DocumentCommitteeSession?$filter=CommitteeSessionID eq … returns zero documents.
- No alternative
KNS_DocumentEthics-style table exists.
So the Ethics decisions live only on main.knesset.gov.il's public HTML pages and the only way to scrape them is to pass through the Radware barrier — there isn't a clean OData escape hatch like there is for committee_decisions.
Action
No action requested separately on this issue — fixing #243 is sufficient. Filing this only as a tracker so the dependency is explicit.
Symptom
pipelines/knesset/ethics_committee_decisions.pyproduces an emptyindex.csv(0 rows). Published datapackage atnext.obudget.org/datapackages/knesset/ethics_committee_decisions/datapackage.jsonreportscount_of_rows: 0and has been frozen since 2025-09-01.Root cause
Same as #243 —
main.knesset.gov.il/Activity/committees/Ethics/Pages/CommitteeDecisionsPast.aspxand the per-KnessetCommitteeDecisions{NN}.aspxpages all return the Radware Bot Manager JavaScript challenge (HTTP 247,rbznsbody) instead of the real HTML when accessed by the pipeline's Selenium-via-SSH driver. The CSS selectors in the pipeline (tr,.link-item,.ComEthicsTdDate,a[href$=".pdf"]) find nothing on the challenge body, so the loop never executes.Why I'm not opening this as part of #243
#243 is specifically about the
legal_advisor+letterspipelines and proposes Radware-bypass approaches for them. This issue is identical in root cause but attached to a different output, so I'm filing it as a tracker so it doesn't get lost when #243 is fixed. Once the Radware bypass lands (whichever path #243 settles on), this pipeline should resume working with no code changes — the existing CSS selectors should match the post-challenge real HTML.Why an OData migration won't work for this feed
Worth noting up front: I checked whether ethics-committee decisions are reachable through the
KNS_*OData endpoints (the path used byknesset_committee_decisionsand proposed for that pipeline in #242). They are not:Name contains 'אתיקה',CommitteeTypeID=71) are queryable viaKNS_Committee.KNS_CommitteeSessionreturns 0–3 sessions per ethics committee.KNS_DocumentCommitteeSession?$filter=CommitteeSessionID eq …returns zero documents.KNS_DocumentEthics-style table exists.So the Ethics decisions live only on
main.knesset.gov.il's public HTML pages and the only way to scrape them is to pass through the Radware barrier — there isn't a clean OData escape hatch like there is forcommittee_decisions.Action
No action requested separately on this issue — fixing #243 is sufficient. Filing this only as a tracker so the dependency is explicit.