Skip to content

Commit ed8b505

Browse files
njncalubnjncalub
committed
feat: add more missing
1 parent 3c4010a commit ed8b505

36 files changed

Lines changed: 2284 additions & 57 deletions

‎README.md‎

Lines changed: 93 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -73,6 +73,94 @@ deno task preview
7373

7474
The server will be available at `http://localhost:8000`
7575

76+
## Senate Bill Scraper
77+
78+
The Senate bill scraper (`senate/main.py`) collects legislative data from the Philippine Senate website.
79+
80+
### Prerequisites
81+
82+
```bash
83+
cd senate/
84+
pip3 install selenium beautifulsoup4 aiohttp pyyaml toml
85+
```
86+
87+
Also ensure you have Chrome/Chromium installed for Selenium.
88+
89+
### Usage
90+
91+
```bash
92+
# Discover and fetch all bills for congress 19
93+
python senate/main.py --congress 19
94+
95+
# Only discover bill numbers (saves to metadata cache)
96+
python senate/main.py --congress 19 --discover
97+
98+
# Only fetch bill details from cached metadata
99+
python senate/main.py --congress 19 --fetch
100+
101+
# Fetch only missing files (skip existing ones)
102+
python senate/main.py --congress 16 17 --fetch --skip-existing
103+
104+
# Extract metadata for all congresses (13-20)
105+
python senate/main.py --metadata
106+
107+
# Fetch bills for multiple congresses
108+
python senate/main.py --congress 16 17 18 19 20 --fetch
109+
110+
# Specify custom workers for concurrent fetching
111+
python senate/main.py --congress 19 --fetch --workers 30
112+
113+
# Show browser window during discovery (debugging)
114+
python senate/main.py --congress 19 --discover --show-browser
115+
```
116+
117+
### Command-line Options
118+
119+
- `--congress`: Congress number(s) to scrape (e.g., 16 17 18 19 20)
120+
- `--type`: Type of bills to scrape (SBN, HBN, ALL)
121+
- `--discover`: Discover bill numbers and save to metadata cache
122+
- `--fetch`: Fetch bill details from cached metadata
123+
- `--skip-existing`: Only download missing files, skip existing ones
124+
- `--metadata`: Extract metadata (senators, committees, statuses)
125+
- `--workers`: Number of concurrent workers for fetching (default: 20)
126+
- `--dir`: Base directory for output (default: current directory)
127+
- `--metadata-dir`: Directory for metadata and cache files (default: metadata)
128+
- `--show-browser`: Show browser window during discovery
129+
- `--force`: Force rediscovery even if cache exists
130+
131+
### Output Structure
132+
133+
```
134+
senate/
135+
├── congress/ # Bill data organized by congress
136+
│ ├── 16/
137+
│ │ ├── SBN/ # Senate bills
138+
│ │ │ ├── SBN-00001.toml
139+
│ │ │ ├── SBN-00002.toml
140+
│ │ │ └── index.yml
141+
│ │ └── HBN/ # House bills
142+
│ └── 17/
143+
└── metadata/ # Metadata and cache files
144+
├── congress_16.json
145+
├── bills_congress_16_SBN.json
146+
└── bills_congress_16_HBN.json
147+
```
148+
149+
### Recovery from Errors
150+
151+
If the scraper encounters timeouts or connection errors:
152+
153+
```bash
154+
# Use --skip-existing to only download missing files
155+
python senate/main.py --congress 16 17 --fetch --skip-existing
156+
```
157+
158+
This will:
159+
- Check metadata JSON files for expected bills
160+
- Scan congress directories for existing TOML files
161+
- Only download missing files
162+
- Update index.yml with all existing bills
163+
76164
## Available Deno Tasks
77165

78166
| Task | Command | Description |
@@ -329,6 +417,10 @@ The following indexes are created for optimal query performance:
329417
├── psa/ # Philippine Statistics Authority data
330418
│ ├── parse_psgc.py # PSGC data parser
331419
│ └── requirements.txt # Python dependencies
420+
├── senate/ # Senate bills data and scraper
421+
│ ├── main.py # Senate bill scraper
422+
│ ├── congress/ # Bill data organized by congress
423+
│ └── metadata/ # Bill metadata and cache files
332424
├── tasks/ # Deno task scripts
333425
│ └── import_psgc_data_to_neo4j.ts
334426
├── src/ # Shared libraries
@@ -347,6 +439,7 @@ The following indexes are created for optimal query performance:
347439
(PSGC)
348440
- **PHLPost:** ZIP codes
349441
- **FOI (Freedom of Information):** Government data
442+
- **Senate of the Philippines:** Legislative bills and resolutions
350443

351444
## Deployment
352445

‎senate/README.md‎

Lines changed: 65 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ A comprehensive scraper that:
88
- **Metadata Extraction**: Extracts senators, committees, and legislative statuses for Neo4j mapping
99
- **Discovery Phase**: Uses Selenium to navigate pages and discover all bill numbers
1010
- **Fetching Phase**: Uses fast async HTTP requests to download bill details with "All Information" view
11+
- **Skip Existing**: Can resume from failures by skipping already downloaded files
1112

1213
## Installation
1314

@@ -22,6 +23,18 @@ pip install -r requirements.txt
2223

2324
## Usage
2425

26+
### Basic workflow (discover and fetch)
27+
```bash
28+
# Default: Congress 19, all bill types
29+
python main.py
30+
31+
# Specific congress
32+
python main.py --congress 19
33+
34+
# Multiple congresses
35+
python main.py --congress 16 17 18 19 20
36+
```
37+
2538
### Extract metadata for Neo4j mapping
2639
```bash
2740
# Extract metadata for all congresses (13-20)
@@ -41,6 +54,18 @@ python main.py --discover --congress 19 --type ALL
4154
python main.py --fetch --congress 19 --type ALL --workers 30
4255
```
4356

57+
### Resume from failures (NEW)
58+
```bash
59+
# Only download missing files for ALL cached congresses
60+
python main.py --fetch --skip-existing
61+
62+
# For specific congresses
63+
python main.py --congress 16 17 --fetch --skip-existing
64+
65+
# The scraper will automatically detect all congresses with cached metadata
66+
# and process them all when no --congress is specified
67+
```
68+
4469
### Complete workflow (all steps)
4570
```bash
4671
python main.py --metadata --discover --fetch --congress 19 --workers 30
@@ -60,7 +85,8 @@ python main.py --discover --congress 19 --show-browser
6085
- `--metadata`: Extract metadata (senators, committees, statuses)
6186
- `--discover`: Run discovery phase to get bill numbers
6287
- `--fetch`: Run fetching phase to download bill details
63-
- `--congress`: Congress number(s) (e.g., 19, 20)
88+
- `--skip-existing`: Only download missing files, skip existing ones (NEW)
89+
- `--congress`: Congress number(s) (e.g., 19, 20) - if omitted, processes all cached congresses when fetching
6490
- `--type`: Bill type (SBN, HBN, or ALL)
6591
- `--workers`: Number of concurrent workers for fetching (default: 20)
6692
- `--dir`: Output directory for bills
@@ -106,8 +132,38 @@ congress/
106132
└── ...
107133
```
108134

135+
## Recovery from Errors
136+
137+
If the scraper encounters timeouts or connection errors:
138+
139+
```bash
140+
# Use --skip-existing to only download missing files
141+
python main.py --congress 16 17 --fetch --skip-existing
142+
```
143+
144+
This will:
145+
1. Check metadata JSON files for expected bills
146+
2. Scan congress directories for existing TOML files
147+
3. Only attempt to download missing files
148+
4. Update index.yml with all existing bills
149+
150+
### Common Error Messages
151+
152+
**Timeout errors:**
153+
```
154+
⏱️ Timeout for SBN-1233
155+
```
156+
Solution: Use `--skip-existing` to retry only failed downloads
157+
158+
**Server disconnection:**
159+
```
160+
❌ Error fetching SBN-1791: Server disconnected
161+
```
162+
Solution: Use `--skip-existing` and optionally reduce `--workers` to decrease server load
163+
109164
## Known Issues
110165

166+
- **Senate Website Data Errors**: Some bills appear in search results but their detail pages return server errors (e.g., SBN-1273 in Congress 19). These are bugs on the Senate website itself.
111167
- **HBN Bill Gaps**: HBN bills have non-consecutive numbering with massive gaps. For Congress 19, bills include: 1, 4, 5, 14, 24, 102, 198, 425, 1028... up to 11545
112168
- **ASP.NET Forms**: The website uses ASP.NET postback forms for dropdown changes, which is why Selenium is used for discovery
113169
- **Rate Limiting**: Be respectful and don't overwhelm the server. The scraper includes built-in delays
@@ -117,4 +173,11 @@ congress/
117173
1. **Extract metadata** to get congress-specific reference data for senators, committees, and statuses (defaults to congresses 13-20)
118174
2. **Discover bills** to get all bill numbers (handles non-consecutive HBN bills)
119175
3. **Fetch details** with multiple workers for fast parallel processing
120-
4. **Import to Neo4j** using the structured TOML files with proper codes for relationships
176+
4. **Import to Neo4j** using the structured TOML files with proper codes for relationships
177+
178+
## Performance Tips
179+
180+
- Use `--workers` to adjust concurrent requests (default: 20)
181+
- Higher worker counts speed up fetching but may cause more timeouts
182+
- Use `--skip-existing` for incremental updates or recovery from failures
183+
- Discovery phase requires browser automation and is slower than fetching

‎senate/congress/13/HBN/HBN-04768.toml‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
billNumber = "HBN-04768"
2+
billType = "HBN"
3+
congress = 13
4+
url = "https://web.senate.gov.ph/lis/bill_res.aspx?congress=13&q=HBN-4768"
5+
lastScraped = "2025-09-21T00:29:21.843745"
6+
title = "BRGY. STO. NIÑO AGRICULTURAL LAND (OCCIDENTAL MINDORO)"
7+
longTitle = "AN ACT DECLARING A PARCEL OF LAND OF THE PUBLIC DOMAIN LOCATED IN BARANGAY STO. NIÑO, MUNICIPALITY OF SABLAYAN, PROVINCE OF OCCIDENTAL MINDORO AN AGRICULTURAL LAND OPEN TO DISPOSITION FOR RESIDENTIAL, COMMERCIAL, INDUSTRIAL AND OTHER PRODUCTIVE PURPOSES"
8+
scope = "Local"
9+
subject = [ "Buffer Zones (Brgy. Sto. Niño Agricultural Land", "Occidental Mindoro)",]
10+
pdfUrl = "https://web.senate.gov.ph/lisdata/50034982!.pdf"
11+
pdfFileName = "HBN-4768 (as filed)"
12+
[[legislativeHistory]]
13+
date = ""
14+
action = "Entitled:"
15+
16+
[[legislativeHistory]]
17+
date = "4/25/2006"
18+
action = "Filed in the Senate requesting for concurrence by Representatives MA. AMELITA C. VILLAROSA and LEOVIGILDO B. BANAAG, as authors;"
19+
20+
[[legislativeHistory]]
21+
date = "5/17/2006"
22+
action = "Read on First Reading and Referred to the Committee on ENVIRONMENT AND NATURAL RESOURCES;"
23+
24+
[status]
25+
status = "Pending in the Committee"
26+
date = "5/17/2006"
27+
28+
[committee]
29+
name = "Environment and Natural Resources"
30+
type = "primary"

‎senate/congress/13/HBN/HBN-04813.toml‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
billNumber = "HBN-04813"
2+
billType = "HBN"
3+
congress = 13
4+
url = "https://web.senate.gov.ph/lis/bill_res.aspx?congress=13&q=HBN-4813"
5+
lastScraped = "2025-09-21T00:29:22.063602"
6+
title = "SIEMBRE NATIONAL HIGH SCHOOL (CAMARINES SUR)"
7+
longTitle = "AN ACT SEPARATING THE SULPICIO A. ROCO MEMORIAL HIGH SCHOOL - SIEMBRE ANNEX IN BARANGAY SIEMBRE, MUNICIPALITY OF BOMBON, PROVINCE OF CAMARINES SUR FROM THE SULPICIO A. ROCO INDEPENDENT NATIONAL HIGH SCHOOL TO BE KNOWN AS THE SIEMBRE NATIONAL HIGH SCHOOL AND APPROPRIATING FUNDS THEREFOR"
8+
scope = "Local"
9+
subject = [ "Schools, Conversion (Camarines Sur)", "Siembre National High School (Camarines Sur)",]
10+
pdfUrl = "https://web.senate.gov.ph/lisdata/44543697!.pdf"
11+
pdfFileName = "HBN-4813 (as filed)"
12+
[[legislativeHistory]]
13+
date = ""
14+
action = "Entitled:"
15+
16+
[[legislativeHistory]]
17+
date = "12/22/2005"
18+
action = "Filed in the Senate requesting for concurrence by Representatives LUIS R. VILLAFUERTE, EDMUNDO O. REYES JR. and ROLANDO G. ANDAYA JR., as authors;"
19+
20+
[[legislativeHistory]]
21+
date = "1/18/2006"
22+
action = "Read on First Reading and Referred to the Committee(s) on EDUCATION, ARTS AND CULTURE and FINANCE;"
23+
24+
[status]
25+
status = "Pending in the Committee"
26+
date = "1/18/2006"
27+
28+
[committee]
29+
name = "Education, Arts and Culture"
30+
type = "primary"

‎senate/congress/13/HBN/HBN-04814.toml‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
billNumber = "HBN-04814"
2+
billType = "HBN"
3+
congress = 13
4+
url = "https://web.senate.gov.ph/lis/bill_res.aspx?congress=13&q=HBN-4814"
5+
lastScraped = "2025-09-21T00:29:21.906105"
6+
title = "BALILI NATIONAL HIGH SCHOOL"
7+
longTitle = "AN ACT SEPARATING THE LEPANTO NATIONAL HIGH SCHOOL - BALILI ANNEX IN BARANGAY \\BALILI, MUNICIAPALITY OF MANKAYAN, PROVINCE OF BENGUET FROM THE LEPANTO NATIONAL HIGH SCHOOL, CONVERTING IT INTO AN INDEPENDENT NATIONAL HIGH SCHOOL TO BE KNOWN AS THE BALILI NATIONAL HIGH SCHOOL AND APPROPRIATING FUNDS THEREFOR"
8+
scope = "Local"
9+
subject = [ "Balili National High School (Benguet)",]
10+
pdfUrl = "https://web.senate.gov.ph/lisdata/46423815!.pdf"
11+
pdfFileName = "HBN-4814 (as filed)"
12+
[[legislativeHistory]]
13+
date = ""
14+
action = "Entitled:"
15+
16+
[[legislativeHistory]]
17+
date = "12/22/2005"
18+
action = "Filed in the Senate requesting for concurrence by Representatives SAMUEL M. DANGWA, EDMUNDO O. REYES JR., ROLANDO G. ANDAYA JR. and EULOGIO 'AMANG' R. MAGSAYSAY, as authors;"
19+
20+
[[legislativeHistory]]
21+
date = "1/18/2006"
22+
action = "Read on First Reading and Referred to the Committee(s) on EDUCATION, ARTS AND CULTURE and FINANCE;"
23+
24+
[status]
25+
status = "Pending in the Committee"
26+
date = "1/18/2006"
27+
28+
[committee]
29+
name = "Education, Arts and Culture"
30+
type = "primary"

‎senate/congress/13/HBN/HBN-04832.toml‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
billNumber = "HBN-04832"
2+
billType = "HBN"
3+
congress = 13
4+
url = "https://web.senate.gov.ph/lis/bill_res.aspx?congress=13&q=HBN-4832"
5+
lastScraped = "2025-09-21T00:29:22.094845"
6+
title = "DR. SANTIAGO DAKUDAO SR. NATIONAL HIGH SCHOOL (DAVAO)"
7+
longTitle = "AN ACT SEPARATING THE F. BANGOY NATIONAL HIGH SCHOOL – PANACAN ANNEX IN BARANGAY PANACAN, CITY OF DAVAO FROM THE F. BANGOY NATIONAL HIGH SCHOOL, CONVERTING IT INTO AN INDEPENDENT NATIONAL HIGH SCHOOL TO BE KNOWN AS THE DR. SANTIAGO DAKUDAO SR. NATIONAL HIGH SCHOOL AND APPROPRIATING FUNDS THEREFOR"
8+
scope = "Local"
9+
subject = [ "Schools, Conversion (Davao)", "Dr. Santiago Dakudao Sr. National High School (Davao)",]
10+
pdfUrl = "https://web.senate.gov.ph/lisdata/50764154!.pdf"
11+
pdfFileName = "HBN-4832 (as filed)"
12+
[[legislativeHistory]]
13+
date = ""
14+
action = "Entitled:"
15+
16+
[[legislativeHistory]]
17+
date = "4/25/2006"
18+
action = "Filed in the Senate requesting for concurrence by Representatives VINCENT J. GARCIA, EDMUNDO O. REYES JR., ROLANDO G. ANDAYA JR. and EULOGIO 'AMANG' R. MAGSAYSAY, as authors;"
19+
20+
[[legislativeHistory]]
21+
date = "5/18/2006"
22+
action = "Read on First Reading and Referred to the Committee(s) on EDUCATION, ARTS AND CULTURE and FINANCE;"
23+
24+
[status]
25+
status = "Pending in the Committee"
26+
date = "5/18/2006"
27+
28+
[committee]
29+
name = "Education, Arts and Culture"
30+
type = "primary"

‎senate/congress/13/HBN/index.yml‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -402,6 +402,7 @@ bills:
402402
- HBN-04762
403403
- HBN-04763
404404
- HBN-04767
405+
- HBN-04768
405406
- HBN-04775
406407
- HBN-04776
407408
- HBN-04777
@@ -415,6 +416,8 @@ bills:
415416
- HBN-04810
416417
- HBN-04811
417418
- HBN-04812
419+
- HBN-04813
420+
- HBN-04814
418421
- HBN-04815
419422
- HBN-04816
420423
- HBN-04817
@@ -432,6 +435,7 @@ bills:
432435
- HBN-04829
433436
- HBN-04830
434437
- HBN-04831
438+
- HBN-04832
435439
- HBN-04833
436440
- HBN-04834
437441
- HBN-04835
@@ -967,6 +971,6 @@ bills:
967971
- HBN-06063
968972
- HBN-06073
969973
congress: 13
970-
count: 967
971-
generated: '2025-09-20T23:49:58.918563'
974+
count: 971
975+
generated: '2025-09-21T00:31:57.689209'
972976
type: HBN

0 commit comments

Comments
 (0)