Skip to content

Commit 152c657

Browse files
committed
chore: major cleanup and modernization
- Merge 12 pairs of duplicate folders (Bird - * prefixes) - Remove 3 empty directories and invalid/corrupt downloads - Delete bloated change_log.txt, replace with CHANGELOG.md - Update all dependencies: requests 2.33.1, urllib3 2.6.3, bs4 4.14.3, etc. - Fix 13 known CVEs in requests/urllib3/idna/tqdm - Disable weekly cron Action; keep manual workflow_dispatch trigger - Rewrite spider: unify requests.Session, URL deduplication, short-name dirs - Extend dedup logic to cover both MP3 and JPG files
1 parent d228087 commit 152c657

44 files changed

Lines changed: 361 additions & 2591 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
Lines changed: 12 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -1,32 +1,27 @@
11
name: Spider Data Collection
22

33
on:
4-
schedule:
5-
- cron: '0 2 * * 5' # 每周五 UTC 2:00 运行
6-
push:
7-
branches:
8-
- master
9-
workflow_dispatch: # 允许手动触发
4+
workflow_dispatch: # 仅允许手动触发
105

116
jobs:
127
build:
138
runs-on: ubuntu-latest
149
permissions:
15-
contents: write # 明确声明需要的权限
10+
contents: write
1611

1712
steps:
1813
- name: Checkout code
19-
uses: actions/checkout@v4 # 升级到最新版本
14+
uses: actions/checkout@v4
2015
with:
21-
fetch-depth: 0 # 获取完整历史以确保正确的提交
16+
fetch-depth: 0
2217

2318
- name: Set up Python
24-
uses: actions/setup-python@v5 # 升级到最新版本
19+
uses: actions/setup-python@v5
2520
with:
26-
python-version: '3.10' # 使用更新的 Python 版本
21+
python-version: '3.12'
2722

2823
- name: Cache pip dependencies
29-
uses: actions/cache@v4 # 添加缓存以加速构建
24+
uses: actions/cache@v4
3025
with:
3126
path: ~/.cache/pip
3227
key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt') }}
@@ -41,46 +36,35 @@ jobs:
4136
- name: Lint with flake8
4237
run: |
4338
pip install flake8
44-
# 检查 Python 语法错误或未定义的变量名
4539
flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
46-
# 将所有错误作为警告处理
4740
flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
4841
4942
- name: Run Spider 🤖
5043
run: |
5144
python spider.py
5245
env:
53-
PYTHONUNBUFFERED: 1 # 确保 Python 输出不被缓冲
54-
55-
- name: Add timestamp to log
56-
run: |
57-
echo -e "\n==================$( date +%Y-%m-%d_%H:%M:%S)====================\n" >> change_log.txt
46+
PYTHONUNBUFFERED: 1
5847

5948
- name: Commit and push changes
6049
env:
6150
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
6251
run: |
63-
# 检查是否有更改需要提交
6452
if [[ -z $(git status -s) ]]; then
6553
echo "No changes to commit"
6654
exit 0
6755
fi
68-
69-
# 设置 Git 用户信息
56+
7057
git config --local user.email "github-actions[bot]@users.noreply.github.com"
7158
git config --local user.name "github-actions[bot]"
72-
73-
# 添加文件、提交并推送
59+
7460
git add .
7561
git commit -m "Update assets [skip ci]" -m "Automated update via GitHub Actions"
76-
echo -e "\n==================$(git status)====================\n" >> change_log.txt
77-
7862
git push
7963
8064
- name: Upload logs as artifacts
8165
uses: actions/upload-artifact@v4
82-
if: always() # 即使任务失败也上传日志
66+
if: always()
8367
with:
8468
name: logs
8569
path: change_log.txt
86-
retention-days: 14 # 保留两周
70+
retention-days: 14

Bird - American Dipper/Sound Library - American Dipper.mp3 renamed to American Dipper/Sound Library - American Dipper.mp3

File renamed without changes.
5.15 KB
File renamed without changes.
File renamed without changes.
-8 MB
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.

0 commit comments

Comments
 (0)