Skip to content

Code fix for WV Event Time Wrong#5727

Open
harshitasingh360 wants to merge 3 commits into
openstates:mainfrom
harshitasingh360:WV-Event-Time-Wrong
Open

Code fix for WV Event Time Wrong#5727
harshitasingh360 wants to merge 3 commits into
openstates:mainfrom
harshitasingh360:WV-Event-Time-Wrong

Conversation

@harshitasingh360

Copy link
Copy Markdown
Contributor

Summary
Fixes the incorrect meeting times for West Virginia committee events.

Changes
Updated the scraper to read the meeting date/time from the current page structure (h2) and the committee name from h1.
Fixed clean_date() to preserve valid time segments while still removing trailing non-date text.
Ensures meeting times (e.g. 2:00 PM, 9:00 AM) are parsed correctly instead of defaulting to 12:00 AM.
Validation
Verified that committee events are now scraped with the correct meeting times.
Confirmed that events are stored with the expected UTC timestamps after timezone conversion.
Existing agenda extraction, bill parsing, and event creation continue to work as expected.

@jessemortenson

jessemortenson commented Jul 8, 2026

Copy link
Copy Markdown
Member

Code looks OK at a glance, but I think there is a likely data problem. I ran the scraper and it resulted in 25 events in July that I don't think are accurate. Here is a selection of data from those events, with the event name, start date, and source URL:

Senate Military Committee	2026-07-03T04:00:00+00:00	http://www.wvlegislature.gov/committees/senate/Senate_com_agendas.cfm?input=3-4-14&chart=mil
Senate Natural Resources Committee	2026-07-03T04:00:00+00:00	http://www.wvlegislature.gov/committees/senate/Senate_com_agendas.cfm?input=3-9-15&chart=nrs
Senate Military Committee	2026-07-02T04:00:00+00:00	http://www.wvlegislature.gov/committees/senate/Senate_com_agendas.cfm?input=2-4-20&chart=mil
Senate Natural Resources Committee	2026-07-02T04:00:00+00:00	http://www.wvlegislature.gov/committees/senate/Senate_com_agendas.cfm?input=2-13-17&chart=nrs
Senate Workforce Committee	2026-07-02T04:00:00+00:00	http://www.wvlegislature.gov/committees/senate/Senate_com_agendas.cfm?input=02-02-2022&chart=wrk
Senate Military Committee	2026-07-02T04:00:00+00:00	http://www.wvlegislature.gov/committees/senate/Senate_com_agendas.cfm?input=2-11-20&chart=mil
Senate Economic Development Committee	2026-07-02T04:00:00+00:00	http://www.wvlegislature.gov/committees/senate/Senate_com_agendas.cfm?input=2-5-20&chart=ecd
Senate Workforce Committee	2026-07-02T04:00:00+00:00	http://www.wvlegislature.gov/committees/senate/Senate_com_agendas.cfm?input=2-12-20&chart=wrk
Senate Natural Resources Committee	2026-07-02T04:00:00+00:00	http://www.wvlegislature.gov/committees/senate/Senate_com_agendas.cfm?input=2-25-15&chart=nrs
Senate Natural Resources Committee	2026-07-02T04:00:00+00:00	http://www.wvlegislature.gov/committees/senate/Senate_com_agendas.cfm?input=2-18-15&chart=nrs
Senate Military Committee	2026-07-02T04:00:00+00:00	http://www.wvlegislature.gov/committees/senate/Senate_com_agendas.cfm?input=2-25-14&chart=mil
Senate Economic Development Committee	2026-07-02T04:00:00+00:00	http://www.wvlegislature.gov/committees/senate/Senate_com_agendas.cfm?input=2-12-20&chart=ecd
Senate Military Committee	2026-07-02T04:00:00+00:00	http://www.wvlegislature.gov/committees/senate/Senate_com_agendas.cfm?input=2-11-14&chart=mil
Senate Economic Development Committee	2026-07-02T04:00:00+00:00	http://www.wvlegislature.gov/committees/senate/Senate_com_agendas.cfm?input=2-29-20&chart=ecd
Senate Military Committee	2026-07-02T04:00:00+00:00	http://www.wvlegislature.gov/committees/senate/Senate_com_agendas.cfm?input=2-29-20&chart=mil
Senate Military Committee	2026-07-02T04:00:00+00:00	http://www.wvlegislature.gov/committees/senate/Senate_com_agendas.cfm?input=2-18-14&chart=mil
Senate Economic Development Committee	2026-07-02T04:00:00+00:00	http://www.wvlegislature.gov/committees/senate/Senate_com_agendas.cfm?input=2-26-20&chart=ecd
Senate Workforce Committee	2026-07-01T04:00:00+00:00	http://www.wvlegislature.gov/committees/senate/Senate_com_agendas.cfm?input=1-22-20&chart=wrk
Senate Military Committee	2026-07-01T04:00:00+00:00	http://www.wvlegislature.gov/committees/senate/Senate_com_agendas.cfm?input=1-28-14&chart=mil
Senate Economic Development Committee	2026-07-01T04:00:00+00:00	http://www.wvlegislature.gov/committees/senate/Senate_com_agendas.cfm?input=1-29-20&chart=ecd
Senate Interstate Cooperation Committee	2026-07-01T04:00:00+00:00	http://www.wvlegislature.gov/committees/senate/Senate_com_agendas.cfm?input=1-20-14&chart=int
Senate Interstate Cooperation Committee	2026-07-01T04:00:00+00:00	http://www.wvlegislature.gov/committees/senate/Senate_com_agendas.cfm?input=1-25-12&chart=int
Senate Military Committee	2026-07-01T04:00:00+00:00	http://www.wvlegislature.gov/committees/senate/Senate_com_agendas.cfm?input=1-14-14&chart=mil
Senate Military Committee	2026-07-01T04:00:00+00:00	http://www.wvlegislature.gov/committees/senate/Senate_com_agendas.cfm?input=01-21-14&chart=mil
Senate Economic Development Committee	2026-07-01T04:00:00+00:00	http://www.wvlegislature.gov/committees/senate/Senate_com_agendas.cfm?input=1-22-20&chart=ecd

260708wv_events.zip

(BTW I'm using duckdb to scan through data as documented here https://docs.openstates.org/data/query-scraper-output-data/#querying-a-full-set-of-scraper-output-files)

Looking at several of those events by going to the source URL, they appear to be in January (matches what the date part of the source URL says). Can you please investigate?

@jessemortenson jessemortenson left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like you fixed the July events issue, thanks. I did find one data issue though:

http://www.wvlegislature.gov/committees/senate/Senate_com_agendas.cfm?input=March 22,2017&chart=int
Senate Interstate Cooperation Committee
2026-03-22T14:00:00+00:00

On that source URL, I cannot find any meeting that is in 2026. Is it possible it is coding the meeting described at the top ("March 22,2017, 10:00 AM") as 2026-03-22 ?

Please look into that, let me know what you found (and of course push a fix if applicable

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants