You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
(uppercase). Add AST drift-guard tests for new settings.
45
54
46
55
## Architectural patterns
47
-
-**Pipeline stage pattern** (v2.0): each stage subclasses `BaseStage` and mutates `ProcessingContext`. Add new behaviour as a new stage, not by extending an existing one. Keep stages single-responsibility.
48
-
- Prefer editing the modular `jobs/` package — `jobs_legacy.py` is kept only for reference.
49
-
- CKAN plugin interfaces are wired in `plugin.py`; new actions go through `logic/action.py` + `logic/schema.py` + `logic/auth.py`.
56
+
-**Pipeline stage pattern**: each stage subclasses `BaseStage` and mutates
57
+
`ProcessingContext`. Add new behaviour as a new stage, not by extending
58
+
an existing one. Keep stages single-responsibility.
59
+
-**Prefect @task wrappers** in `jobs/prefect_flow.py` are thin —
60
+
they delegate to `BaseStage.process()` bodies. Don't embed business
61
+
logic in the task functions.
62
+
- The v2-era `jobs_legacy.py` is gone — there is nothing to "edit the
63
+
modular jobs/ package instead of". All work happens under `jobs/`.
64
+
- CKAN plugin interfaces are wired in `plugin.py`; new actions go through
DataPusher+ is a **CKAN extension (v2.0.0)** for ultra-fast, robust data ingestion into CKAN's datastore. It replaces the legacy Datapusher webservice and combines the speed/robustness of `ckanext-xloader` with the data-type guessing of Datapusher — supercharged with metadata inference/suggestion via Jinja2 formulas defined in scheming YAML.
4
+
DataPusher+ is a **CKAN extension (v3.0.0a0)** for ultra-fast, robust data
5
+
ingestion into CKAN's datastore. It replaces the legacy Datapusher webservice
6
+
and combines the speed/robustness of `ckanext-xloader` with the data-type
7
+
guessing of Datapusher — supercharged with metadata inference/suggestion via
8
+
Jinja2 formulas defined in scheming YAML.
5
9
6
10
Key differentiators:
7
-
-**Guaranteed type inference** — scans the entire file (not first N rows) via [qsv](https://github.com/dathere/qsv), a Rust CSV toolkit.
11
+
-**Guaranteed type inference** — scans the entire file (not first N rows) via
12
+
[qsv](https://github.com/dathere/qsv), a Rust CSV toolkit.
8
13
-**PostgreSQL COPY** for direct datastore loading (no API overhead).
9
-
-**Jinja2 formula system** for metadata inference/suggestion (`formula` and `suggest_formula` in scheming YAML).
14
+
-**Jinja2 formula system** for metadata inference/suggestion (`formula` and
15
+
`suggestion_formula` in scheming YAML — NOTE the production key is
- Three formula namespaces available: `dpps` (per-field stats), `dppf` (per-field freq tables), `dpp` (inferred metadata: RECORD_COUNT, DATE_FIELDS, LAT_FIELD, LON_FIELD, etc.).
12
-
- No longer a separate webservice — it is now a full CKAN extension (`ckan.plugins` entry point: `datapusher_plus = ckanext.datapusher_plus.plugin:DatapusherPlusPlugin`).
-**Prefect plumbing**: `prefect_client.py` is the single place CKAN admin
22
+
paths touch `prefect.*`; `jobs/__init__.py` uses PEP 562 lazy
23
+
`__getattr__` to defer the Prefect import so CKAN CLI commands don't
24
+
spin up a Prefect server.
13
25
14
26
## Workflow recipes
15
-
- Understanding a CKAN action handler: `find_symbol` with name_path `datapusher_submit` (or other action) in `logic/action.py`, `include_body=True`.
27
+
- Understanding a CKAN action handler: `find_symbol` with name_path
28
+
`datapusher_submit` (or other action) in `logic/action.py`,
29
+
`include_body=True`.
16
30
- Adding a pipeline stage:
17
31
1.`get_symbols_overview` on `jobs/stages/base.py`.
18
32
2.`find_symbol` on `BaseStage`, `include_body=True`.
19
-
3. Create new module under `jobs/stages/` using `replace_symbol_body` on a placeholder, or write a new file with the editor.
20
-
4. Hook the new stage into the pipeline by editing `jobs/pipeline.py` with `replace_symbol_body` on the orchestration function.
21
-
- Renaming or moving a function: prefer `rename` / `move` Serena tools so all references update. Use `find_referencing_symbols` first to scope blast radius.
22
-
- Inspecting how a CKAN interface is wired: `find_symbol` on `DatapusherPlusPlugin` with `depth=1`, then drill into the method that implements the interface hook (`update_config`, `get_actions`, etc.).
33
+
3. Create new module under `jobs/stages/` and add the matching
34
+
`@task` wrapper in `jobs/prefect_flow.py`.
35
+
4. Hook the new task into the flow by editing `datapusher_plus_flow`
36
+
in `jobs/prefect_flow.py` (use `replace_symbol_body`).
37
+
- Renaming or moving a function: prefer `rename` / `move` Serena tools so
38
+
all references update. Use `find_referencing_symbols` first to scope
39
+
blast radius.
40
+
- Inspecting how a CKAN interface is wired: `find_symbol` on
41
+
`DatapusherPlusPlugin` with `depth=1`, then drill into the method that
42
+
implements the interface hook (`update_config`, `get_actions`, etc.).
23
43
24
44
## Known oddities
25
-
-`plugin.py` defines `DatastoreException` twice (lines around the top) — likely vestigial; leave alone unless asked to clean up.
26
-
-`jobs.py` and `jobs_legacy.py` both exist at the package root in addition to the modular `jobs/` package. The legacy module is kept for reference; do not edit it.
27
-
-`# flake8: noqa: E501` is conventional at the top of many modules; preserve when editing.
45
+
-`plugin.py` defines `DatastoreException` twice (lines around the top) —
46
+
likely vestigial; leave alone unless asked to clean up.
47
+
- The v2-era `jobs.py`, `jobs_legacy.py`, and `jobs/pipeline.py` files
48
+
have all been removed; if you see references to them in old docs or
49
+
memories, they are stale.
50
+
-`jobs/__init__.py` re-exports `datapusher_plus_to_datastore` as an
51
+
alias for `datapusher_plus_flow` — preserves the v2 import path.
52
+
-`# flake8: noqa: E501` is conventional at the top of many modules;
Pytest defaults (from `setup.cfg`): SQLAlchemy and `DeprecationWarning` warnings are filtered; `--pdbcls=IPython.terminal.debugger:TerminalPdb` is preset.
34
+
35
+
> **Pytest config lives in `pyproject.toml`** under
36
+
> `[tool.pytest.ini_options]` (warning filters + the IPython `--pdbcls`
37
+
> addopt). It moved from `setup.cfg` during the v3.0 refactor — comments
38
+
> referencing `setup.cfg` for pytest config are stale.
ckan -c /etc/ckan/default/ckan.ini db upgrade -p datapusher_plus
39
64
```
40
65
41
66
## Linting / formatting
42
-
No formatter is enforced; flake8 is the implicit linter with `E501` disabled. There is **no** project-level `make lint`, `ruff`, or `black` config — match existing style.
67
+
No formatter is enforced; flake8 is the implicit linter with `E501`
68
+
disabled. There is **no** project-level `make lint`, `ruff`, or `black`
0 commit comments