Add wordlist templates and backend selection

Author: maurosoria

CHANGELOG.md

@@ -8,6 +8,8 @@
 - Added `dirsearch[mysql]`, `dirsearch[postgresql]`, and `dirsearch[db]` install extras.
 - Updated Docker, CI, PyInstaller, and Nuitka release builds for the 5.0.0 baseline.
 - Added an importable Python API with `FuzzerConfig`, `DirsearchFuzzer`, structured results, wordlists, and template wordlists.
+- Added template wordlist placeholders, curated templates, `--wordlist-status`, and generation limits.
+- Added the initial wordlist backend interface with `auto`, `python`, and reserved `native` selection.
 - Ability to use multiple output formats
 - MySQL and PostgreSQL report formats
 - Support variables in file path and SQL table name for saving results

README.md

@@ -126,6 +126,9 @@ Wordlists (IMPORTANT)
   - To apply your extensions to wordlist entries that have extensions already, use **-O** | **--overwrite-extensions** (Note: some extensions are excluded from being overwritted such as *.log*, *.json*, *.xml*, ... or media extensions like *.jpg*, *.png*)
   - To use multiple wordlists, you can separate your wordlists with commas. Example: `wordlist1.txt,wordlist2.txt`.
   - Bundled wordlist categories live in `db/categories/` and can be selected with **--wordlist-categories**. Available: `extensions`, `conf`, `vcs`, `backups`, `db`, `logs`, `keys`, `web`, `common` (use `all` to include everything).
+  - Wordlist generation uses **--wordlist-backend=auto** by default. **python** selects the built-in backend and **native** requires a native backend build.
+  - Template wordlists live in `db/templates/` and support placeholders such as `%SUBJECT%`, `%CRUD_OP%`, `%AUTH_OP%`, `%ADMIN_OP%`, `%ENV%`, `%DATE%`, `%API_VERSION%`, `%CATEGORY:name%`, and `%EXT%`.
+  - Use **--wordlist-status** to preview resolved wordlist files and generated entry count before scanning. Use **--wordlist-max-size** to cap generation.
 
 <details>
 <summary><strong>Wordlist Examples (click to expand)</strong></summary>
@@ -214,6 +217,14 @@ Options:
                         Comma-separated wordlist category names (e.g.
                         common,conf,web). Use 'all' to include all bundled
                         categories
+    --wordlist-backend=BACKEND
+                        Wordlist generation backend: auto, python, native
+                        (default: auto)
+    --wordlist-status   Show resolved wordlist files and generated entry
+                        count, then exit
+    --wordlist-max-size=COUNT
+                        Maximum generated wordlist entries before aborting
+                        (default: 500000)
     -e EXTENSIONS, --extensions=EXTENSIONS
                         Extension list separated by commas (e.g. php,asp)
     -f, --force-extensions

TODO.md

@@ -2,9 +2,9 @@
 
 ## Current Status
 
-Phase 1 is implemented for optional database dependencies. Phase 2 is implemented for the Python 3.14 / `5.0.0` release baseline. Phase 3 is implemented for the first importable Python API.
+Phase 1 is implemented for optional database dependencies. Phase 2 is implemented for the Python 3.14 / `5.0.0` release baseline. Phase 3 is implemented for the first importable Python API. Phase 4 is implemented for template wordlists and generation controls.
 
-This is a useful foundation for a future MCP server or REST API because import-time side effects are lower, base installs no longer require database drivers, and the first supported Python API surface is now available. The next work expands wordlist templates and generation controls.
+This is a useful foundation for a future MCP server or REST API because import-time side effects are lower, base installs no longer require database drivers, and the first supported Python API surface is now available. The next work adds the native wordlist implementation and benchmark gates.
 
 ## Completed
 
@@ -47,6 +47,13 @@ This is a useful foundation for a future MCP server or REST API because import-t
 - Added isolation coverage proving two configs do not leak state in one process.
 - Added packaged install smoke coverage for the public API imports.
 - Added optional DB and importable API coverage to `testing.py`.
+- Added shared template wordlist expansion for CLI and Python API callers.
+- Added named placeholders for subjects, CRUD/auth/admin ops, envs, separators, dates, DB/archive tokens, API versions, categories, and `%EXT%`.
+- Added curated template wordlists under `db/templates/`.
+- Added `--wordlist-status`.
+- Added `--wordlist-max-size` generation limits.
+- Added template wordlist regression coverage to `testing.py`.
+- Added the wordlist backend interface and `auto|python|native` selection.
 
 ## Verification Run
 
@@ -74,22 +81,16 @@ This is a useful foundation for a future MCP server or REST API because import-t
 - `docker build -t dirsearch:v5-importable-api .`
 - `docker run --rm --entrypoint python3 dirsearch:v5-importable-api testing.py`
 - `docker run --rm --entrypoint sh dirsearch:v5-importable-api -c "python3 -m pip install . && python3 tests/check_packaged_install.py"`
+- `/home/mauro/dirsearch/.venv/bin/python -m unittest tests.core.test_dictionary_templates tests.core.test_importable_api`
+- `/home/mauro/dirsearch/.venv/bin/python dirsearch.py --wordlist-status -w db/templates/crud.txt -e php`
+- `/home/mauro/dirsearch/.venv/bin/python dirsearch.py --wordlist-status -w db/templates/crud.txt -e php --wordlist-max-size 1`
 
 ## Next Phase
 
-### Phase 4: Template Wordlists
-
-- Add named placeholder expansion for `%SUBJECT%`, `%CRUD_OP%`, `%AUTH_OP%`, `%ADMIN_OP%`, `%ENV%`, `%SEP%`, date tokens, DB/archive tokens, `%API_VERSION%`, and `%CATEGORY:name%`.
-- Add curated template files under `db/templates/`.
-- Add `--wordlist-status`.
-- Add generation limits to prevent accidental huge expansions.
-
-### Later Phases
-
 ### Phase 5: Native Wordlist Backend
 
-- Add backend interface with Python backend first.
-- Add optional native backend with `auto|python|native` selection.
+- Add backend interface with Python backend first. (done)
+- Add optional native backend with `auto|python|native` selection. (selection done, native implementation pending)
 - Keep native output byte-for-byte compatible with Python output for ordering and deduplication.
 - Add opt-in large dictionary benchmark for generation time, iteration throughput, memory, and startup cost.
 

config.ini

@@ -41,6 +41,8 @@ capital = False
 #suffixes = ~,.bak
 #wordlists = /path/to/wordlist1.txt,/path/to/wordlist2.txt
 #wordlist-categories = common,conf,web
+wordlist-backend = auto
+wordlist-max-size = 500000
 
 [request]
 http-method = get

db/templates/admin.txt

@@ -0,0 +1,5 @@
+%ADMIN_OP%.%EXT%
+admin/%ADMIN_OP%
+%ADMIN_OP%/%SUBJECT%
+%ADMIN_OP%_%SUBJECT%.%EXT%
+%ADMIN_OP%-%SUBJECT%.%EXT%

db/templates/api.txt

@@ -0,0 +1,5 @@
+api/%API_VERSION%/%SUBJECT%
+api/%API_VERSION%/%CRUD_OP%/%SUBJECT%
+api/%SUBJECT%/%CRUD_OP%
+%API_VERSION%/%SUBJECT%
+%SUBJECT%.json

db/templates/auth.txt

@@ -0,0 +1,5 @@
+%AUTH_OP%.%EXT%
+auth/%AUTH_OP%
+account/%AUTH_OP%
+user/%AUTH_OP%
+oauth/%AUTH_OP%

db/templates/backups.txt

@@ -0,0 +1,5 @@
+backup.%ARCHIVE%
+backup_%DATE%.%ARCHIVE%
+backup_%DATE_COMPACT%.%ARCHIVE%
+%ENV%_backup.%ARCHIVE%
+%SUBJECT%_backup.%ARCHIVE%

db/templates/crud.txt

@@ -0,0 +1,5 @@
+%CRUD_OP%_%SUBJECT%.%EXT%
+%SUBJECT%_%CRUD_OP%.%EXT%
+%CRUD_OP%/%SUBJECT%.%EXT%
+%SUBJECT%/%CRUD_OP%.%EXT%
+api/%CRUD_OP%_%SUBJECT%

db/templates/db.txt

@@ -0,0 +1,5 @@
+db.%ARCHIVE%
+database.%ARCHIVE%
+backup_%DB%.%ARCHIVE%
+%DB%_backup.%ARCHIVE%
+dump_%DATE_COMPACT%.%ARCHIVE%