Sync 2.x into master: pipeline refactor (PR-A through PR-E2) and changelog#40
Open
ramonski wants to merge 15 commits into
Open
Sync 2.x into master: pipeline refactor (PR-A through PR-E2) and changelog#40ramonski wants to merge 15 commits into
ramonski wants to merge 15 commits into
Conversation
The package declares Python 3.8+ in setup.py, so the legacy compat layer is dead code. Cleaning it up reduces noise for future readers and removes a layer of indirection over plain str / int / bytes. - Delete src/senaite/astm/compat.py (basestring, unicode, long, make_string, b, u, buffer) - Replace compat imports with direct str / int / bytes usage - Inline make_string into Field._set_value (decode bytes as utf-8, str() everything else) - Replace try/except izip_longest with a direct 'from itertools import zip_longest' in utils.py and mapping.py - Replace try/except 'from collections import Iterable' with 'from collections.abc import Iterable' in codec.py - Replace deprecated logger.warn with logger.warning in utils.py and lims.py - Drop the u() helper in tests/test_fields.py — Python 3 string literals are already unicode
Drop Python 2 compatibility shims
Bumps the package to 2.0.0 — the LIMS push API is intentionally incompatible with the 1.x line. The Session class now creates a single requests.Session in __init__ and reuses it across all calls, so the TLS handshake is amortised across the connection rather than repeated for every request. auth(), get() and post() raise typed exceptions (SenaiteAuthError, SenaiteHTTPError, SenaiteUnreachableError) instead of swallowing every Exception into an empty dict, so the caller can react to the specific failure mode. post_to_senaite() now authenticates once per call. Retries on push failure only re-call session.post(); auth is not re-run on every attempt as before. The function returns a PushResult dataclass (success, attempts, last_error) so the server can act on the result instead of fire-and-forget. The top-level senaite.astm.lims module is removed in the same PR; server.py, sender.py and the tests import from senaite.astm.core.lims directly. We own all callers, so we don't keep a compat shim.
Lift LIMS push into core/ with typed errors and PushResult
Adds senaite.astm.core.envelope with a pydantic-based Envelope and Metadata model. The envelope is now a pinned contract: - ENVELOPE_VERSION = '1.0' is exposed in metadata.envelope_version on every output, so consumers can detect schema changes. - Metadata declares the required keys (envelope_version, astm, lis2a) and accepts vendor extras (e.g. Roche c111's parsed sender component) via extra='allow'. - The per-record buckets (H, P, O, R, C, M, L, Q) default to empty lists so the top-level shape is stable regardless of which record types a given instrument emits. - Per-record dict shapes are intentionally left loose — that lives in the per-instrument record classes. Wrapper now exposes to_envelope() returning the typed model and keeps to_dict()/to_json() as JSON-serialisable convenience wrappers around it. The 11 golden snapshots are regenerated to include the envelope_version field and the empty-list defaults. pydantic>=2 is added to install_requires.
Define a typed Envelope schema for Wrapper.to_dict()
Three behaviours bit us in production and made the test output hard to read. This PR fixes the symptoms without rewriting the descriptor framework. NotUsedField no longer warns on assignment. The cobas_c311 fixture alone produced ~78 UserWarning entries per parse, drowning out real warnings without giving the operator anything actionable. The field now silently drops the assigned value. SetField accepts unknown values by default and logs them at debug level. A device firmware update that introduces a new status code should not crash parsing of every message that contains it. Pass strict=True to restore the legacy raise-on-unknown behaviour. DateField, TimeField and DateTimeField now accept a tuple of parse_formats in addition to the canonical format. Subclasses can extend it to handle vendor-specific date strings without rewriting parse logic. The canonical format is always tried first and is still used for serialisation, so existing snapshots are unchanged.
Make field descriptors quiet and tolerant
First step of unifying senaite.astm.instruments and
senaite.astm.adapters into a single mechanism.
- New core.instrument module:
- Instrument base class (name, header_regex, record_map,
can_handle, preparse, get_metadata)
- register_instrument decorator with shape validation
- find_instrument resolver that raises
AmbiguousInstrumentError instead of silently picking one
match when two regexes overlap
- Wrapper.get_mapping() now consults the registry first and
falls back to today's pkgutil-based discovery. Instrument-
specific metadata is merged via either path.
No instrument has been migrated yet, so behaviour is unchanged
for all existing analyzers. PR-E2 will migrate them and remove
the legacy discovery path together with senaite.astm.adapters.
269 tests pass (+10 new); flake8 clean.
Introduce the instrument registry (PR-E1)
Second and final step of the instrument unification: - Every senaite.astm.instruments.* module now declares an Instrument subclass at the bottom and registers itself via @register_instrument. Module-level HEADER_RX is a compiled bytes regex; the old get_mapping()/get_metadata() helpers are gone. Each module also exposes INSTRUMENT for direct test access. - The two zope-adapter data handlers (mini_vidas, spotchem se1520) have moved onto their corresponding Instrument via a new raw_data_regex attribute and a handle_raw_data(protocol, data) hook. ASTMProtocol.handle_data now dispatches via find_raw_data_handler instead of a Components registry. - Wrapper.get_mapping resolves entirely through the registry and falls back to DEFAULT_MAPPING for unknown headers. The pkgutil iter_modules path is gone, along with self.module. - senaite.astm.adapters, senaite.astm.interfaces, the adapter_registry global, and the zope.interface dependency are removed. - instruments/__init__.py imports every submodule so the decorators run at import time. - The Instrument base class also gained get_metadata; the wrapper now actually calls it (the old wrapper had latent code that never fired for any module). Envelope snapshots regenerated accordingly to include version + header_rx in metadata. - New test_replay_corpus.py walks $ASTM_REPLAY_DIR (~50k CERMEL captures) through Wrapper and asserts the parse failure ratio stays under 5%. Pre-/post-migration ratios are identical at 1549/50382 (~3.07%), confirming no real-traffic regression. - Existing per-instrument tests updated to access <module>.INSTRUMENT.record_map. 270 tests pass (269 + replay). Existing pre-migration parser quirks (mostly truncated c111 captures) are explicitly tolerated by the replay threshold.
Migrate every instrument to the registry (PR-E2)
ramonski
added
Enhancement ✨
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Brings
masterup to date with the pipeline refactor series merged on2.x:core/with typed errors andPushResultEnvelopeschema forWrapper.to_dict()The head branch is pinned at the last refactor commit so the HL7 stack and the disk-capture / transport-split work that landed on 2.x afterwards (#33-#39) is intentionally excluded from this merge.
Test plan
bin/test --package senaite.astmpassesWrapper.to_dict()stillwork with the typed envelope