Skip to content

Commit 43bbf07

Browse files
SynthLuvrSynthLuvr
committed
Merge branch 'main' into healf-crawler
2 parents 689afa9 + ecf4ac6 commit 43bbf07

266 files changed

Lines changed: 25415 additions & 0 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/nhs-crawler.yml‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
name: nhs-crawler
2+
3+
on:
4+
pull_request:
5+
paths:
6+
- "nhs-crawler/**"
7+
- ".github/workflows/nhs-crawler.yml"
8+
9+
jobs:
10+
nhs-crawler:
11+
name: nhs-crawler
12+
runs-on: ubuntu-latest
13+
defaults:
14+
run:
15+
working-directory: nhs-crawler
16+
steps:
17+
- uses: actions/checkout@v7.0.0
18+
19+
- name: Install uv
20+
uses: astral-sh/setup-uv@v8.2.0
21+
22+
- name: Set up Python
23+
run: uv python install 3.11
24+
25+
- name: Install dependencies
26+
run: uv sync --extra dev
27+
28+
- name: Lint (ruff)
29+
run: uv run ruff check .
30+
31+
- name: Format check (ruff)
32+
run: uv run ruff format --check .
33+
34+
- name: Type check (pyright)
35+
run: uv run pyright

‎nhs-crawler/README.md‎

Lines changed: 164 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,164 @@
1+
# NHS Conditions Scraper
2+
3+
A Python scraper for all [NHS Health A to Z condition
4+
pages](https://www.nhs.uk/health-a-to-z/conditions/). It discovers every
5+
condition listed on the index (198 conditions), crawls all subpages for
6+
each condition (254 pages total), and saves the content as clean
7+
Markdown to `data/`.
8+
9+
## Prerequisites
10+
11+
- **[uv](https://docs.astral.sh/uv/)** — install with:
12+
13+
``` bash
14+
curl -LsSf https://astral.sh/uv/install.sh | sh
15+
```
16+
17+
- **Python ≥ 3.11** — uv manages this automatically.
18+
19+
## Quick Start
20+
21+
``` bash
22+
cd nhs-crawler
23+
uv sync
24+
25+
# Scrape ALL conditions (198 conditions, 254 pages)
26+
uv run nhs-crawler
27+
28+
# Scrape a single condition (discovers subpages automatically)
29+
uv run nhs-crawler --url https://www.nhs.uk/conditions/type-2-diabetes/
30+
31+
# Scrape a single specific page
32+
uv run nhs-crawler --page https://www.nhs.uk/conditions/type-2-diabetes/treatment/
33+
34+
# Show all options
35+
uv run nhs-crawler --help
36+
```
37+
38+
## Output
39+
40+
Markdown files are saved to `data/<condition-slug>/<subpage>.md`:
41+
42+
data/
43+
├── asthma/
44+
│ └── index.md # Inline hub page — no subpages
45+
├── type-2-diabetes/
46+
│ ├── what-is-type-2-diabetes.md
47+
│ ├── symptoms.md
48+
│ ├── treatment.md
49+
│ ├── complications.md
50+
│ └── support.md
51+
├── covid-19/
52+
│ ├── covid-19-symptoms-and-what-to-do.md
53+
│ ├── how-to-avoid-catching-and-spreading-covid-19.md
54+
│ └── treatments-for-covid-19.md
55+
└── ... # 198 conditions, 254 Markdown files
56+
57+
Each file starts with a title and source URL header:
58+
59+
``` markdown
60+
# Asthma
61+
62+
> Source: https://www.nhs.uk/conditions/asthma/
63+
64+
Asthma is a common condition that affects your breathing...
65+
```
66+
67+
### Reports
68+
69+
After a full crawl, two files are written to `reports/`:
70+
71+
- **`condition_urls.json`** — ordered list of all discovered condition
72+
hub URLs.
73+
- **`summary.json`** — aggregate statistics for the run:
74+
75+
``` json
76+
{
77+
"run_at": "2026-06-20T18:38:08.937007+00:00",
78+
"index_url": "https://www.nhs.uk/health-a-to-z/conditions/",
79+
"conditions_discovered": 198,
80+
"pages_discovered": 254,
81+
"pages_scraped": 254,
82+
"error_count": 0,
83+
"total_markdown_chars": 1136966,
84+
"empty_pages": [],
85+
"errors": []
86+
}
87+
```
88+
89+
### Logs
90+
91+
A detailed crawl log is appended to `logs/crawl.log` on every run.
92+
93+
> **Note:** The `data/`, `logs/`, and `reports/` directories are created
94+
> automatically at runtime by `src/constants.py`. They do not need to
95+
> exist beforehand. Consider adding `logs/` and `reports/` to
96+
> `.gitignore` if you don’t want to track generated output.
97+
98+
## How It Works
99+
100+
1. **Discover conditions** — fetches the A-to-Z index page and extracts
101+
all `/conditions/<slug>/` links using a regex filter.
102+
2. **Discover subpages** — for each condition hub page, looks for a
103+
`<ul class="nhsuk-hub-key-links">` navigation element to find
104+
subpages. Some conditions (like asthma) are single-page with inline
105+
content — the hub page itself is saved as `index.md`. Others (like
106+
type-2-diabetes) have dedicated subpages for Symptoms, Treatment,
107+
etc. — in this case only the subpages are saved (the hub is excluded
108+
since the subpages contain all the content).
109+
3. **Fetch & convert** — each page is fetched and its main content
110+
element (`<div class="nhsuk-grid-column-two-thirds">` inside
111+
`<article>`) is converted to clean Markdown using a custom NHS-aware
112+
converter that handles:
113+
- **Care cards** → blockquotes with emoji prefixes (🚨 ⚠️ 📋)
114+
- **Do/don’t lists** → bullet lists with ✅/❌ markers
115+
- **Inset text** → blockquotes with 💡 prefix
116+
- **Clutter removal** — nav, breadcrumbs, feedback banners, SVGs,
117+
etc.
118+
4. **Save** — Markdown is written to `data/<condition>/<subpage>.md`
119+
with a title and source URL header.
120+
5. **Report** — `reports/summary.json` with aggregate statistics and
121+
`reports/condition_urls.json` with the full condition URL list.
122+
123+
### HTTP Strategy
124+
125+
All requests go through a shared `requests.Session` with:
126+
127+
- **Retry**: up to 4 retries with exponential backoff (factor 1.2) on
128+
HTTP 429, 500, 502, 503, 504.
129+
- **Rate limiting**: 0.5s delay before each request (politeness).
130+
- **Connection pooling**: 10 connections, 20 max pool size.
131+
- **User-Agent**: identifies the scraper with a reference to nhs.uk.
132+
- **Concurrency**: pages are fetched in parallel using a thread pool (4
133+
workers by default).
134+
135+
## CLI Reference
136+
137+
usage: nhs-crawler [-h] [--url URL] [--page PAGE]
138+
139+
NHS Conditions scraper
140+
141+
options:
142+
-h, --help show this help message and exit
143+
--url URL Scrape a single condition URL (including subpages).
144+
If omitted, scrapes all conditions from the A-to-Z index.
145+
--page PAGE Scrape a single specific page URL (no subpage discovery).
146+
147+
## Configuration
148+
149+
Key settings in `src/constants.py`:
150+
151+
| Constant | Default | Description |
152+
|---------------|---------|---------------------------------------|
153+
| `DELAY` | `0.5` | Seconds between requests (politeness) |
154+
| `TIMEOUT` | `30` | HTTP request timeout (seconds) |
155+
| `MAX_WORKERS` | `4` | Parallel threads for page fetching |
156+
157+
## Development
158+
159+
``` bash
160+
uv sync --all-extras
161+
uv run ruff format src/
162+
uv run ruff check src/
163+
uv run pyright src/
164+
```

‎nhs-crawler/data/acute-pancreatitis/index.md‎

Lines changed: 103 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,103 @@
1+
# Acute pancreatitis
2+
3+
> Source: https://www.nhs.uk/conditions/acute-pancreatitis/
4+
5+
![Diagram of the stomach area with labels showing the liver, stomach, gallbladder and pancreas. The pancreas is highlighted just below the stomach.](https://assets.nhs.uk/nhsuk-cms/images/Pancreatic_Cancer_NEW_copy.width-320.jpg)
6+
7+
The pancreas is an organ in the middle of your tummy. It helps you digest food and makes hormones such as insulin.
8+
9+
> **💡 Information:** Acute pancreatitis is different to chronic pancreatitis , a long-term condition where the pancreas has become permanently damaged.
10+
11+
## Symptoms of acute pancreatitis
12+
13+
The main symptom of acute pancreatitis is pain in your tummy (abdomen). You may also have a high temperature and feel sick or be sick (nausea and vomiting).
14+
15+
Tummy pain may:
16+
17+
- start suddenly and not go away
18+
- be severe, sharp or knife-like
19+
- affect the upper part of your tummy (between your ribs), one side of your tummy or your whole tummy
20+
- spread to your sides and back
21+
- feel worse after you eat, move around or lie down
22+
- feel better when you lean forward or bring your knees to your chest (fetal position)
23+
24+
> **💡 Information:** Acute pancreatitis symptoms can be similar to other conditions such as appendicitis or stomach ulcer . Find out about what else can cause stomach ache.
25+
26+
> **⚠️ Urgent advice:**
27+
>
28+
> You get sudden, severe pain in your tummy and:
29+
>
30+
> - it does not go away or keeps coming back
31+
> - you have a high temperature, or you feel hot, cold or shivery
32+
>
33+
> You can call 111 or [get help from 111 online](https://111.nhs.uk/triage/check-your-symptoms).
34+
>
35+
> If a GP thinks you have acute pancreatitis they will refer you to hospital for tests straight away.
36+
37+
> **🚨 Immediate action required:**
38+
>
39+
> You get sudden, severe pain in your tummy and:
40+
>
41+
> - the pain is spreading to your back
42+
> - you have bloating that does not go away or keeps coming back
43+
> - you have a fast heartbeat or difficulty breathing
44+
> - the skin around your belly button, waist or upper outer thigh appears blue or bruised – this may be more difficult to see on black or brown skin
45+
>
46+
> [Find your nearest A&E](https://www.nhs.uk/service-search/find-an-accident-and-emergency-service/)
47+
48+
> **💡 Information:** Do not drive to A&E. Ask someone to drive you or call 999 and ask for an ambulance. Bring any medicines you take with you.
49+
50+
## Treatment for acute pancreatitis
51+
52+
Acute pancreatitis is usually diagnosed using blood tests and sometimes a CT scan. It's a serious condition that needs treatment in hospital straight away.
53+
54+
You'll be monitored to see how serious your condition is and if it's causing any other problems, such as an infection.
55+
56+
Hospital treatment may include:
57+
58+
- fluids and nutrients – given through a tube into a vein
59+
- painkillers
60+
- antibiotics – if you have an infection
61+
62+
You may also need treatment for what is causing your acute pancreatitis, such as surgery for gallstones or support to stop drinking alcohol.
63+
64+
Most people with acute pancreatitis start to get better within a week and can leave hospital in 5 to 10 days.
65+
66+
If you have severe pancreatitis or it's causing other problems, you may need to stay in hospital for longer.
67+
68+
## Problems caused by acute pancreatitis
69+
70+
Most people with acute pancreatitis recover fully. But some people develop serious complications that will need treatment.
71+
72+
Complications of acute pancreatitis include:
73+
74+
- small growths in your pancreas (cysts) – these often go away on their own but may need to be removed if they become infected
75+
- pancreatic necrosis – where some of the tissue in the pancreas dies and you need surgery and antibiotics to prevent a serious condition called [sepsis](/conditions/sepsis/)
76+
- [chronic pancreatitis](/conditions/chronic-pancreatitis/) – if you keep getting acute pancreatitis it can develop into a serious long-term condition
77+
78+
Acute pancreatitis can be life-threatening. You'll be monitored while you're in hospital, to check for any problems caused by acute pancreatitis.
79+
80+
## Causes of acute pancreatitis
81+
82+
The most common causes of acute pancreatitis are:
83+
84+
- [gallstones](/conditions/gallstones/) – which can block the opening of the pancreas
85+
- drinking a lot of alcohol
86+
87+
Less common causes include:
88+
89+
- injury to the pancreas, such as during surgery
90+
- medicines, including certain steroids, heart and epilepsy medicines
91+
- other conditions including [lupus](/conditions/lupus/), [mumps](/conditions/mumps/), [pancreatic cancer](/conditions/pancreatic-cancer/) or having high levels of calcium in your blood (hypercalcaemia)
92+
93+
## How to prevent acute pancreatitis
94+
95+
If you have had acute pancreatitis once, it's possible to get it again.
96+
97+
There are some things you can do to help stop it coming back.
98+
99+
It's a good idea to:
100+
101+
- drink less alcohol, or not drink any alcohol at all
102+
- stop smoking
103+
- eat a healthy, low-fat diet

0 commit comments

Comments
 (0)