Skip to content

Commit 19a7c69

Browse files
rosuHrosuH
committed
feat: engineering best practices, tests, and legal docs
Code Quality: - Format all Python code with black (line-length 127) - Fix flake8 errors: remove unused imports, unused variables - Add mypy type stubs and resolve all type errors - Optimize remove_duplicated_files: avoid redundant MD5 recomputation CI/CD: - GitHub Actions: fetch-depth 1, timeout 15min, concurrency control - Pin flake8 version in CI for reproducible builds - Add pytest step to Action pipeline - Add dependabot config for pip + github-actions Testing: - Add 13 unit tests for directory naming, MD5, and dedup logic - Configure pytest with pythonpath in pyproject.toml Developer Experience: - Add Makefile, pyproject.toml, .flake8, .pre-commit-config.yaml - Add requirements-dev.txt (black, flake8, mypy, pytest, pre-commit) Documentation: - Rewrite README with compliance statement, badges, proper English - Add robots.txt compliance and public domain attribution guidance - Fix LICENSE copyright holder - Update CHANGELOG with complete history
1 parent 1ce5837 commit 19a7c69

12 files changed

Lines changed: 366 additions & 109 deletions

‎.flake8‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
[flake8]
2+
max-line-length = 127
3+
extend-ignore = E203,W503

‎.github/dependabot.yml‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
version: 2
2+
updates:
3+
- package-ecosystem: "pip"
4+
directory: "/"
5+
schedule:
6+
interval: "monthly"
7+
open-pull-requests-limit: 5
8+
9+
- package-ecosystem: "github-actions"
10+
directory: "/"
11+
schedule:
12+
interval: "monthly"

‎.github/workflows/spider_action.yml‎

Lines changed: 51 additions & 51 deletions
Original file line numberDiff line numberDiff line change
@@ -1,71 +1,71 @@
11
name: Spider Data Collection
22

33
on:
4-
workflow_dispatch: # 仅允许手动触发
4+
workflow_dispatch: # 仅允许手动触发
5+
6+
concurrency:
7+
group: ${{ github.workflow }}-${{ github.ref }}
8+
cancel-in-progress: true
59

610
jobs:
711
build:
812
runs-on: ubuntu-latest
13+
timeout-minutes: 15
914
permissions:
1015
contents: write
1116

1217
steps:
13-
- name: Checkout code
14-
uses: actions/checkout@v4
15-
with:
16-
fetch-depth: 0
18+
- name: Checkout code
19+
uses: actions/checkout@v4
20+
with:
21+
fetch-depth: 1
1722

18-
- name: Set up Python
19-
uses: actions/setup-python@v5
20-
with:
21-
python-version: '3.12'
23+
- name: Set up Python
24+
uses: actions/setup-python@v5
25+
with:
26+
python-version: "3.12"
2227

23-
- name: Cache pip dependencies
24-
uses: actions/cache@v4
25-
with:
26-
path: ~/.cache/pip
27-
key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt') }}
28-
restore-keys: |
29-
${{ runner.os }}-pip-
28+
- name: Cache pip dependencies
29+
uses: actions/cache@v4
30+
with:
31+
path: ~/.cache/pip
32+
key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt') }}
33+
restore-keys: |
34+
${{ runner.os }}-pip-
3035
31-
- name: Install dependencies
32-
run: |
33-
python -m pip install --upgrade pip
34-
pip install -r requirements.txt
36+
- name: Install dependencies
37+
run: |
38+
python -m pip install --upgrade pip
39+
pip install -r requirements.txt
3540
36-
- name: Lint with flake8
37-
run: |
38-
pip install flake8
39-
flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
40-
flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
41+
- name: Lint with flake8
42+
run: |
43+
pip install "flake8==7.3.0"
44+
flake8 . --count --show-source --statistics
4145
42-
- name: Run Spider 🤖
43-
run: |
44-
python spider.py
45-
env:
46-
PYTHONUNBUFFERED: 1
46+
- name: Run tests
47+
run: |
48+
pip install pytest
49+
pytest -v
4750
48-
- name: Commit and push changes
49-
env:
50-
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
51-
run: |
52-
if [[ -z $(git status -s) ]]; then
53-
echo "No changes to commit"
54-
exit 0
55-
fi
51+
- name: Run Spider 🤖
52+
run: |
53+
python spider.py
54+
env:
55+
PYTHONUNBUFFERED: "1"
5656

57-
git config --local user.email "github-actions[bot]@users.noreply.github.com"
58-
git config --local user.name "github-actions[bot]"
57+
- name: Commit and push changes
58+
env:
59+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
60+
run: |
61+
if [[ -z $(git status -s) ]]; then
62+
echo "No changes to commit"
63+
exit 0
64+
fi
5965
60-
git add .
61-
git commit -m "Update assets [skip ci]" -m "Automated update via GitHub Actions"
62-
git push
66+
git config --local user.email "github-actions[bot]@users.noreply.github.com"
67+
git config --local user.name "github-actions[bot]"
6368
64-
- name: Upload logs as artifacts
65-
uses: actions/upload-artifact@v4
66-
if: always()
67-
with:
68-
name: spider-logs
69-
path: |
70-
CHANGELOG.md
71-
retention-days: 14
69+
git add .
70+
git commit -m "Update assets [skip ci]" -m "Automated update via GitHub Actions"
71+
git push

‎.pre-commit-config.yaml‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
repos:
2+
- repo: https://github.com/psf/black
3+
rev: 24.10.0
4+
hooks:
5+
- id: black
6+
language_version: python3
7+
8+
- repo: https://github.com/PyCQA/flake8
9+
rev: 7.1.1
10+
hooks:
11+
- id: flake8
12+
13+
- repo: https://github.com/pre-commit/mirrors-mypy
14+
rev: v1.13.0
15+
hooks:
16+
- id: mypy
17+
additional_dependencies: [types-requests, types-tqdm]

‎CHANGELOG.md‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,8 +22,21 @@
2222
- **Disabled** the weekly scheduled cron job — the NPS sound library is essentially static, so automatic runs only produced empty commits.
2323
- **Retained** `workflow_dispatch` so the spider can still be triggered manually when needed.
2424
- Upgraded Action runtime to Python 3.12.
25+
- Added `concurrency` control, `timeout-minutes: 15`, and `fetch-depth: 1` for faster, safer runs.
2526

2627
### Spider Rewrite
2728
- **Unified HTTP client**: replaced mixed `urllib.request.urlopen` + `requests` usage with a single `requests.Session()`.
2829
- **URL-level deduplication**: the crawler now tracks processed URLs, preventing duplicate folders when the NPS site lists the same page under multiple titles.
30+
- **Directory name normalization**: new downloads prefer short names and reuse existing folders with suffix variants (e.g. `Old Faithful Geyser`).
2931
- Improved error handling, type hints, and logging consistency.
32+
33+
### Developer Experience
34+
- Added **13 unit tests** covering directory resolution, MD5 hashing, and duplicate removal logic.
35+
- Added linting & formatting toolchain: `black`, `flake8`, `mypy`, `pytest`, `pre-commit`.
36+
- Added `Makefile`, `pyproject.toml`, `.pre-commit-config.yaml`, and `.github/dependabot.yml`.
37+
- Added `requirements-dev.txt` for reproducible development dependencies.
38+
39+
### Documentation
40+
- **Rewrote README** with clearer copyright notices, compliance statement, and CI badges.
41+
- Added explicit **robots.txt compliance** and **public domain attribution** guidance.
42+
- Fixed LICENSE copyright holder name.

‎LICENSE‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
MIT License
22

3-
Copyright (c) 2019
3+
Copyright (c) 2019 rosuH
44

55
Permission is hereby granted, free of charge, to any person obtaining a copy
66
of this software and associated documentation files (the "Software"), to deal

‎Makefile‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
.PHONY: install lint test run clean
2+
3+
install:
4+
pip install -r requirements.txt
5+
pip install -r requirements-dev.txt
6+
7+
lint:
8+
flake8 spider.py tests/
9+
black --check spider.py tests/
10+
mypy spider.py
11+
12+
format:
13+
black spider.py tests/
14+
15+
test:
16+
pytest -v
17+
18+
run:
19+
python spider.py
20+
21+
clean:
22+
find . -type d -name __pycache__ -exec rm -rf {} +
23+
find . -type f -name "*.pyc" -delete

‎README.md‎

Lines changed: 44 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -1,57 +1,77 @@
11
# YSL
2-
### 🏞Yellowstone Sound Library 🎵
32

4-
This warehouse collects the public sound library of Yellowstone Park. These sounds contain the sounds of the natural environment and animals.
5-
You can use this audio as a BGM for your work👩‍💻, study📖 or meditation🙇‍♂️.
3+
[![Spider Data Collection](https://github.com/rosuH/YSL/actions/workflows/spider_action.yml/badge.svg)](https://github.com/rosuH/YSL/actions/workflows/spider_action.yml)
4+
[![Python 3.12](https://img.shields.io/badge/python-3.12-blue.svg)](https://www.python.org/downloads/)
5+
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
66

7-
# Download
7+
### 🏞 Yellowstone Sound Library 🎵
88

9-
Since Github has a size limit on the uploaded file, I uploaded it to the [Web Archive](https://archive.org/details/YSL.7z).
10-
Or you can just clone the repo.
9+
This repository collects the public sound library of Yellowstone National Park. These sounds contain the natural environment and animals.
10+
You can use this audio as BGM for your work 👩‍💻, study 📖 or meditation 🙇‍♂️.
1111

12-
# Inspiration
12+
## Download
1313

14-
[tonyq0802’s tweets](https://twitter.com/tonyq0802/status/1084364955506290688)
14+
Since GitHub has a size limit on uploaded files, the full archive is also available on the [Web Archive](https://archive.org/details/YSL.7z).
15+
Or you can just clone this repository.
1516

16-
# Credit
17+
## Inspiration
1718

18-
This library collects public sound libraries from [Yellowstone National Park](https://www.nps.gov/yell/learn/photosmultimedia/soundlibrary.htm).
19+
[tonyq0802's tweet](https://twitter.com/tonyq0802/status/1084364955506290688)
1920

20-
> Welcome to the Yellowstone sound library, where you can immerse yourself in the aural landscape of America's first national park. The files available here were recorded in the park and are in the public domain. They may be downloaded and used without limitation; however, please credit the "National Park Service " where appropriate.
21+
## Credit & Legal Notice
2122

22-
[National Park Service](https://www.nps.gov/index.htm)
23+
This repository collects public sound libraries from [Yellowstone National Park](https://www.nps.gov/yell/learn/photosmultimedia/soundlibrary.htm).
24+
25+
> Welcome to the Yellowstone sound library, where you can immerse yourself in the aural landscape of America's first national park. The files available here were recorded in the park and are in the public domain. They may be downloaded and used without limitation; however, please credit the "National Park Service" where appropriate.
26+
> — [National Park Service](https://www.nps.gov/index.htm)
27+
28+
### ⚖️ Compliance Statement
29+
30+
**Copyright Status**: According to the [NPS Disclaimer](https://www.nps.gov/aboutus/disclaimer.htm), works created by U.S. federal government employees as part of their official duties are generally in the public domain under 17 U.S.C. §§ 101, 105. The Yellowstone Sound Library explicitly states that its audio files are "in the public domain."
31+
32+
**Web Crawling**: This project uses an automated crawler to download files. The crawler:
33+
- Respects the site's [`robots.txt`](https://www.nps.gov/robots.txt) (the sound library path is **not** disallowed).
34+
- Identifies itself with a descriptive `User-Agent` including this repository's URL.
35+
- Uses a polite request delay (2 seconds by default) to avoid overloading NPS servers.
36+
37+
**Usage Risk**: While NPS explicitly labels these files as public domain, not all materials on NPS websites are guaranteed to be free of third-party rights. Users are responsible for determining whether their specific use case requires additional permissions. If any file in this repository is found to infringe on third-party rights, please [open an issue](https://github.com/rosuH/YSL/issues) immediately and it will be removed.
2338

2439
> *If there is any infringement in this repo, please create an issue immediately and I will deal with it.*
25-
> *I am a beginner of Python. If you have any suggestions for the code, please start a PR immediately, thank you!*
2640
27-
----
41+
---
2842

2943
# YSL
3044

3145
### 🏞 黄石公园声音库 🎵
3246

33-
这个仓库收集了黄石公园的公开声音库。这些声音包含了自然环境、动物产生的声音。
47+
这个仓库收集了黄石国家公园的公开声音库。这些声音包含了自然环境、动物产生的声音。
3448
您可以使用它们作为您工作 👩‍💻、学习 📖 或冥想的背景音乐 🙇‍♂️。
3549

36-
# 下载
50+
## 下载
3751

38-
因为 Github 对上传的文件有大小限制，我将之上传到了 [Web Archive](https://archive.org/details/YSL.7z)。
52+
因为 GitHub 对上传的文件有大小限制，完整的归档也上传到了 [Web Archive](https://archive.org/details/YSL.7z)。
3953
当然，您也可以直接 clone 本项目。
4054

41-
# 启发
55+
## 启发
4256

4357
[tonyq0802 的此条推特](https://twitter.com/tonyq0802/status/1084364955506290688)
4458

45-
# 版权声明
59+
## 版权声明与法律说明
4660

61+
本库收集自 [Yellowstone National Park](https://www.nps.gov/yell/learn/photosmultimedia/soundlibrary.htm) 的公开声音库。
4762

48-
This library collects public sound libraries from [Yellowstone National Park](https://www.nps.gov/yell/learn/photosmultimedia/soundlibrary.htm).
63+
> 欢迎来到黄石声音库，在这里您可以沉浸于美国第一个国家公园的听觉景观中。这里提供的文件是在公园内录制的，属于公共领域。它们可以无限制地下载和使用；但请在适当的地方注明 "National Park Service"。
64+
> — [National Park Service](https://www.nps.gov/index.htm)
4965
50-
> Welcome to the Yellowstone sound library, where you can immerse yourself in the aural landscape of America's first national park. The files available here were recorded in the park and are in the public domain. They may be downloaded and used without limitation; however, please credit the "National Park Service " where appropriate.
66+
### ⚖️ 合规声明
5167

52-
[National Park Service](https://www.nps.gov/index.htm)
68+
**版权状态**：根据 [NPS 免责声明](https://www.nps.gov/aboutus/disclaimer.htm)，美国联邦政府雇员在履行公务过程中创作的作品通常属于公共领域（依据 17 U.S.C. §§ 101, 105）。黄石声音库也明确声明其音频文件 "属于公共领域"。
5369

54-
> *如果此仓库有任何侵权行为，请您立刻发起一条 issue，我将会马上处理。*
55-
> *我是 Python 的初学者，如果您对代码有任何建议，请您立刻发起一个 PR，感谢您！*
70+
**网络爬虫**：本项目使用自动化爬虫下载文件。该爬虫：
71+
- 遵守网站的 [`robots.txt`](https://www.nps.gov/robots.txt)（声音库路径**未被**禁止）。
72+
- 使用包含本仓库 URL 的描述性 `User-Agent`。
73+
- 默认使用 2 秒的请求延迟，以避免对 NPS 服务器造成压力。
5674

75+
**使用风险**：虽然 NPS 明确将这些文件标记为公共领域，但并非 NPS 网站上的所有材料都保证没有第三方权利。用户有责任确定其特定使用场景是否需要额外的许可。如果本仓库中的任何文件被发现侵犯了第三方权利，请立即[提交 issue](https://github.com/rosuH/YSL/issues)，我们将立即处理。
5776

77+
> *如果此仓库有任何侵权行为，请您立刻发起一条 issue，我将会马上处理。*

‎pyproject.toml‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
[tool.mypy]
2+
python_version = "3.10"
3+
warn_return_any = true
4+
warn_unused_configs = true
5+
disallow_untyped_defs = true
6+
ignore_missing_imports = true
7+
follow_untyped_imports = false
8+
9+
[tool.pytest.ini_options]
10+
pythonpath = ["."]
11+
addopts = "-v"
12+
13+
[tool.black]
14+
line-length = 127
15+
target-version = ['py310']

‎requirements-dev.txt‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
black==24.10.0
2+
flake8==7.1.1
3+
mypy==1.13.0
4+
pytest==8.3.5
5+
pre-commit==4.0.1
6+
types-requests==2.32.0
7+
types-tqdm==4.66.0

0 commit comments

Comments
 (0)