v3.0 — package the library as simcenter-dlml with an API, CLI, and the DLML Explorer web app#40
Merged
Merged
Conversation
This reverts commit 57a8bce.
tests/test_data_scripts.py parses every Python script shipped under src/dlml/data/ (the per-dataset pelicun_config.py auto-population scripts and their helpers) and asserts each imports only from an allowed set: the standard library, pelicun, pandas/numpy/jsonschema, sibling modules, and any names a script declares in a module-level REQUIRES list. The audit is static (parsed with ast, never executed) so the test suite stays pelicun-free, and it catches a script quietly gaining an undeclared dependency that would fail a user mid-run. Skipped below Python 3.10 (needs sys.stdlib_module_names).
src/dlml/vocabulary.py establishes dlml as the authoritative owner of the controlled vocabularies in the model parameters: - EDP_to_demand_type (and DEMAND_TYPES, its key set): valid demand-type names, mirrored from pelicun base.py; - DISTRIBUTION_FAMILIES: distribution families a parameter may declare, mirrored from pelicun uq.py rv_class_map. Re-exported from the package root so pelicun can import them from here rather than redefine them after a future pelicun update. Tests verify the structures and prove the vocabularies cover every demand type and family in the packaged data.
…adata The SimCenter Wind Component Library generator emitted the per-model field as "RoudUpToIntegerQuantity" (missing the n). Fix it in generate_library_files.py and regenerate fragility.json; the only change to the library file is the 123 renamed keys (CSV unchanged).
Rewrite tests/test_data_format.py to validate cell values, not just column presence: closed-world columns (only known LS<n>/DS<n> and fixed groups); binary flags; positive dispersions; Theta_0 as a positive/non-negative scalar or a numeric-string curve; normalized DamageStateWeights; and an LS/DS that names a Family must supply its parameters unless the model is Incomplete. Demand-type/Family membership remains enforced in test_vocabulary.
RoundUpToIntegerQuantity was stored inconsistently: "True"/"False", "YES"/"NO" (FEMA P-58 consequence), and 1 as a number (SimCenter Hurricane). Normalize to the canonical "True"/"False": - fix the SimCenter generator (1 -> "True") and regenerate; - fix the FEMA P-58 consequence generator (YES/NO -> True/False) and sync the committed consequence_repair.json to match; This allows us to tighten the metadata schema to enum ["True","False"] in the following commit.
Validate every shipped metadata file against a per-collection JSON Schema: _GeneralInformation (required fields, closed), an optional References bibliography, and every model a closed-world entry with the nested LimitStates/DamageStates structure. Enforce that top-level keys are exactly the model IDs plus the reserved keys, and that every per-model Reference citation resolves to the References bibliography. Add test_metadata_states_match_parameters: each model's metadata LimitStates/DamageStates (minus the LS0/DS0 undamaged baseline) must equal exactly the LS<n>/DS<n> groups it defines in the CSV.
Build the wheel with uv and assert it ships exactly the library and nothing more: every packaged data file is present (and only those), the package modules are present, no authoring material leaks in (data_sources, spreadsheets, notebooks, scratch files), every entry is a data file / package module / wheel metadata, and the wheel stays well under PyPI's size limit. Skipped where uv is unavailable.
tests/test_pelicun_equivalence.py proves dlml's tabular deserializer (a port of pelicun's load_data) stays faithful: for every packaged parameters table, dlml.get_parameters equals load_data(path, None, orientation=1, reindex=False), and the re-exported convert_to_MultiIndex matches pelicun's on both axes. Requires pelicun (the test extra); skipped only when it is genuinely absent. The import is made offline by pointing DLML_DATA_DIR at a stub dir with the manifest pelicun probes for plus a fresh version-check cache, so importing pelicun downloads nothing.
Prepare the merged Streamlit app for packaging into src/dlml/web/: - delete dead top-level modules figures.py and visuals_core.py (unimported; the live figures live in st_visuals/); - remove the dormant auth; - delete three temp files; - broaden the secrets gitignore to **/secrets.toml so credentials can never be committed.
Fold the merged app into the package so it ships and installs with dlml: - relocate app.py, pages/, st_core/, st_search/, st_ui/, st_visuals/, assets/ into src/dlml/web/ (adding __init__.py for the new subpackages); - rewrite intra-app imports to dlml.web.* and swap the app's only pelicun use (convert_to_MultiIndex) for dlml's re-export — the app no longer imports pelicun; - exclude src/dlml/web from ruff (force-exclude), mypy, codespell, and coverage — it is not yet held to the package's standards (a tracked follow-up); - allow the web tree (code + assets) in the wheel test while still forbidding secrets/.env/.ds_store, and assert the app ships. Data discovery still resolves against the repo root; repointing it at the packaged data is the next chunk.
The Streamlit app located its data through CWD-relative paths (tree_corpus_files' base_path defaulted to "."), which only worked when launched from the old repo root. With the data now under src/dlml/data, default that base to dlml._catalog.data_root() so discovery works for a pip-installed package from any working directory; an explicit path stays overridable for the CLI and tests. tree_corpus_files is the sole discovery choke point; every downstream read resolves siblings off the corpus path. Also drop the now-redundant "." at the three call sites and refresh stale module-path references left by the src/dlml/web relocation.
Add src/dlml/cli.py registering a `dlml` console script with three subcommands: - `web` (launch the packaged Streamlit app headless, with the base theme passed as --theme.* flags so pip users need no config file; prints an install hint when the web extra is absent), and - `list` / `info` (dataset inspection using only the core dependencies, so they work from a bare install). Reconcile the `web` extra to the app's actual imports. Wrap the search-index build so missing optional dependencies or an offline first-run degrade to a friendly notice with the browse tree intact, instead of a traceback. Replace requirements.txt with a one-line `.[web]` pointer so there is a single source of dependency truth in pyproject.
The "Running locally" section still pointed at the old Streamlit fork: Tshajlij's clone URL, a pelicun/numpy<2 prerequisite, `pip install -r requirements.txt`, and `streamlit run app.py`. Update it for the packaged app.
Restructure the optional-dependency extras so CI can install exactly what each job needs without the explorer stack : add a `lint` extra (ruff, codespell, mypy) and a `doc` extra (sphinx toolchain plus plotly, colorlover, scipy, which the doc generators need). Point `dev` at all of them. Pin `pelicun>=3.9,<3.10` in `test`: it matches the offline-import guard in test_pelicun_equivalence and keeps the resolver from selecting a pelicun that would fetch data from the frozen releases/latest during CI. Delete run_checks.sh - superseded by scripts/check.sh. Regenerate uv.lock.
Replace the three ad-hoc lint/format/spell workflows with one ci.yml that mirrors the local gate, so CI can't drift from scripts/check.sh: - checks: uv sync --locked --extra test --extra lint, then run scripts/check.sh (ruff, format, codespell, mypy, pytest + coverage>=90), with UV_NO_SYNC=1 so its inner `uv run` calls reuse the synced env. - test: matrix over ubuntu 3.9/3.12/3.13, windows 3.12, macOS 3.12, on the test extra only. - bare-install: build and install from the sdist with no extras, then smoke the core API and CLI (proves the sdist ships the data tree). Least-privilege permissions, cancel-in-progress concurrency, pinned uv.
Add publish.yml in the PyPA build-then-publish shape. It triggers on a git tag `v[3-9].*` (or a manual dry run), never on GitHub Release creation, so `releases/latest` stays frozen at v2.1.0 for legacy pelicun. The build job builds sdist+wheel, asserts the tag matches the built version (blocking a stray tag or a forgotten .dev bump), and runs `twine check`. The publish job runs only on a tag, in a reviewer-gated `pypi` environment, and uploads via pypa/gh-action-pypi-publish using OIDC trusted publishing. A manual dispatch builds and validates but does not upload.
PyPI rejected `dlml` as too similar to an existing project. Rename the distribution to `simcenter-dlml`; the import package (`import dlml`, src/dlml/) and the `dlml` console command are unchanged.
The Codespaces devcontainer still installed requirements.txt + streamlit and ran `streamlit run app.py` from the old root path. Point it at the package: install editable with the explorer extra, launch via `dlml explorer`, and open src/dlml/web/app.py. The Codespaces proxy needs Streamlit's CORS/XSRF protection off; since `dlml explorer` doesn't expose those flags, set them via containerEnv (Streamlit's run CLI reads config from env vars).
The Sphinx doc generators read model data from the old repo-root location (./seismic etc.) and imported pelicun. Repoint them at the packaged data and dlml's own convert_to_MultiIndex: - generate_dl_doc.py: chdir to dlml.data_root() (in main(), not at import), and resolve the doc output folders from __file__. - compute_global_hash.py: hash the packaged CSVs explicitly. - visuals.py: import convert_to_MultiIndex from dlml; delete the dead argparse main() that carried the last pelicun reference. - sphinx_generate_dl_doc.py: run the generator with sys.executable. Fix build_docs.yaml: install the .[doc] extra instead of the removed requirements.txt, bump the actions, and build (without deploying) on PRs to main and manual runs so the generators are exercised before they deploy.
The Sphinx doc generators read model data from the old repo-root location (./seismic etc.) and imported pelicun. Repoint them at the packaged data and dlml's own convert_to_MultiIndex: - generate_dl_doc.py: chdir to dlml.data_root() (in main(), not at import), and resolve the doc output folders from __file__. - compute_global_hash.py: hash the packaged CSVs explicitly. - visuals.py: import convert_to_MultiIndex from dlml; delete the dead argparse main() that carried the last pelicun reference. - sphinx_generate_dl_doc.py: run the generator with sys.executable. Fix build_docs.yaml: install the .[doc] extra instead of the removed requirements.txt, bump the actions, and build (without deploying) on PRs to main and manual runs so the generators are exercised before they deploy.
Rewrite the README to lead with the data library and its principles (version-controlled, standardized, documented provenance), then present the ways to access it: the simcenter-dlml package and the dlml API, the DLML Explorer (hosted app first, local install as an alternative), Pelicun and the SimCenter tools, and the raw data. Acknowledge Degenkolb Engineers for the DLML Explorer. Add the 3.0 changelog entry and a matching v3.0 release-notes page, reframe the Sphinx About and landing pages around the package and API, demote the old docs website to a legacy note, and fix the intersphinx numpy/scipy URLs so the docs build without warnings. Align the publish workflow's freeze comment to the v3.0 tag. The version stays 3.0.0.dev0; the bump to 3.0 is the final step before tagging.
… panels Navigation & branding - The About page is now the landing page (app.py); Browse & Search moves to pages/1_Browse_and_Search.py, with a prominent call-to-action from About. - A slim sticky brand bar (SimCenter + Degenkolb logos, "DLML Explorer" wordmark, page links, dark-mode toggle) replaces the sidebar nav and stays visible on scroll, wrapping to a second row on narrow widths. - The hero becomes a text-only editorial header (title + red rule + subtitle + managed-by note); the Browse page drops it to save space. Theme - Adopt the SimCenter palette: red (#C8382E) as the signature accent, NHERI blue for utility actions. - Tighten the vertical rhythm; dark-mode fixes for code blocks, the search spinner, and the sidebar collapse control. - Responsive column stacking now uses container queries, so it reacts to the content width (e.g. a widened sidebar), not just the viewport. Detail panels & sidebar - Restructure the model detail panels: shared header (metadata, technical notes, references, add/remove action), demand/consequence characteristics, and themed HTML parameter tables that size headers correctly and follow dark mode (st.dataframe did neither). - The sidebar is reserved for the user's "Selected models"; Remove moves into the panel header where Add sits in Browse. - Downloads become a checklist + single button, zipping multiple files. Copy & icons - "components" -> "models" throughout; consistent hazard icons (seismic 〰️, hurricane 🌀); About copy rewritten with GitHub, NIST NED, Pelicun and ATC links.
- Rename the SimCenter-calibrated hurricane portfolio dataset from the duplicate "Hazus Hurricane Methodology - Buildings" to "SimCenter Coupled Hurricane Models (Hazus-calibrated)", with a description explaining the normal/lognormal fit and per-damage-state loss-ratio calibration, so it is not mistaken for the unmodified Hazus data. - FEMA P-58 dataset description now renders the changelog URL and links the "2nd edition of FEMA P-58" to femap58.atcouncil.org. Applied to the bundled JSONs and the data_sources source-of-truth metadata.
…nc theme - __init__.py: look up the distribution as `simcenter-dlml` (was `dlml`), so dlml.__version__ reports the real version instead of 0.0.0+local. - Bump version 3.0.0.dev0 -> 3.0 (+ uv.lock); date the changelog / release notes. - Point the native Streamlit theme (config.toml + cli._THEME) at the SimCenter red #C8382E to match the app palette.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Graduates the Damage and Loss Model Library into an installable Python package,
simcenter-dlml(import namedlml), and brings the DLML Explorer webapp into the repo. The model data is unchanged and backward compatible —
this release adds tooling around it.
What's in it
pip install simcenter-dlml; a small, pelicun-free API(
list_datasets,get_parameters/get_fragility/ …,get_metadata,get_schema,validate_asset) with filesystem-based dataset discovery.dlml(includingdlml explorer).under
dlml.weband installable via theexplorerextra.schema/format/wheel gates, a bare-install job, and an OS/Python matrix.
v3.*tag push(manual-approval gated); the docs site now builds from the packaged data.
Per-commit messages carry the details.
Release / maintainer notes⚠️
releases/latestmust stay at v2.1.0.Existing Pelicun (≤3.9) downloads the model data from the latest GitHub
release, so that pin keeps it working. v3.0+ ships to PyPI only.
v3.0) that triggerspublish.yml— do notcreate a GitHub Release, which would move
releases/latest.