Code fix for WV Event Time Wrong#5727
Conversation
|
Code looks OK at a glance, but I think there is a likely data problem. I ran the scraper and it resulted in 25 events in July that I don't think are accurate. Here is a selection of data from those events, with the event name, start date, and source URL: (BTW I'm using duckdb to scan through data as documented here https://docs.openstates.org/data/query-scraper-output-data/#querying-a-full-set-of-scraper-output-files) Looking at several of those events by going to the source URL, they appear to be in January (matches what the date part of the source URL says). Can you please investigate? |
jessemortenson
left a comment
There was a problem hiding this comment.
Looks like you fixed the July events issue, thanks. I did find one data issue though:
http://www.wvlegislature.gov/committees/senate/Senate_com_agendas.cfm?input=March 22,2017&chart=int
Senate Interstate Cooperation Committee
2026-03-22T14:00:00+00:00
On that source URL, I cannot find any meeting that is in 2026. Is it possible it is coding the meeting described at the top ("March 22,2017, 10:00 AM") as 2026-03-22 ?
Please look into that, let me know what you found (and of course push a fix if applicable
Summary
Fixes the incorrect meeting times for West Virginia committee events.
Changes
Updated the scraper to read the meeting date/time from the current page structure (h2) and the committee name from h1.
Fixed clean_date() to preserve valid time segments while still removing trailing non-date text.
Ensures meeting times (e.g. 2:00 PM, 9:00 AM) are parsed correctly instead of defaulting to 12:00 AM.
Validation
Verified that committee events are now scraped with the correct meeting times.
Confirmed that events are stored with the expected UTC timestamps after timezone conversion.
Existing agenda extraction, bill parsing, and event creation continue to work as expected.