feat(types): enhance crypto column types and add TOTP/OTP storage#758
Merged
Conversation
github-actions
Bot
added
area/alembic
area/dependencies
area/docs
area/private-api
size: medium
type/feat
pr/internal
size: large
and removed
size: medium
labels
cofin
changed the title
Fix and harden the at-rest encrypted column types (Ch.1 of crypto-types hardening). - PGCryptoBackend now actually works: encryption/decryption run server-side via SQLAlchemy bind_expression/column_expression. Writes wrap as armor(pgp_sym_encrypt(:v, passphrase)) and reads as pgp_sym_decrypt(dearmor(col), passphrase), so the ASCII-armored ciphertext stays in a String column and the column type is unchanged. Previously the base mount_vault never called init_engine (AttributeError), and even past that, returning a SQL Function from process_bind_param could not bind. - Add EncryptionBackend.bind_expression/column_expression hooks (default None for client-side backends like Fernet). - Wire cryptography through the optionals facade: CRYPTOGRAPHY_INSTALLED in _typing/typing; FernetBackend raises MissingDependencyError when missing; cryptography is imported lazily inside the Fernet methods. - Emit a DeprecationWarning when EncryptedString/EncryptedText is constructed without an explicit key (the random default changes per process restart, making rows undecryptable). Internal fixtures pass an explicit key. - Cache the derived cipher for static keys; callable keys still re-resolve. - __repr__ renders type(self).__name__ so EncryptedText reconstructs as EncryptedText (not EncryptedString) in Alembic autogenerated migrations. - Drop the false "built-in rotation support" docstring claim. Adds unit coverage (facade flag, missing-dep guard, default-key warning, static/callable cache, ciphertext backward-compat golden token, backend SQL wrapping, cache-key isolation, subclass repr) and a real PostgreSQL + Fernet round-trip integration test.
Ch.2 of crypto-types hardening. - Add needs_rehash(hashed) to the HashingBackend interface, implemented by the argon2, passlib, and pwdlib backends (argon2/pwdlib via check_needs_rehash, passlib via needs_update). All return False (never raise) for unparsable or foreign hashes. - Add HashedPassword.verify_and_update(plain) -> (bool, Optional[str]): (False, None) on mismatch, (True, None) when current, (True, new_hash) when the stored hash should be upgraded after a successful verify. - Wire argon2/passlib/pwdlib through the optionals facade: ARGON2_INSTALLED, PASSLIB_INSTALLED, PWDLIB_INSTALLED in _typing/typing. argon2 guards at import (obstore pattern, MissingDependencyError); passlib/pwdlib guard in __init__ via the flag. - Raise the default PasswordHash length from 128 to 255 for stacked passlib schemes (explicit-length callers unaffected). - Narrow Argon2Hasher.verify: drop the bare except Exception. Guard non-str/bytes input with an explicit isinstance check (returns False) and catch only argon2 verification errors (VerifyMismatchError, InvalidHash, VerificationError). The worksheet's (TypeError, ValueError) fallback was insufficient — argon2 raises AttributeError for non-encodable inputs. - Document compare_expression/compare_value as an optional server-side extension point (all shipped backends raise NotImplementedError); kept as public pre-2.0 API, not removed. Adds unit coverage for flags, guards, needs_rehash, verify_and_update, default length, and foreign-hash handling.
Ch.3 of crypto-types hardening: a reversible TOTP shared-secret column type built on the hardened EncryptedString. - TOTPSecret(EncryptedString) stores a base32 authenticator seed encrypted at rest (same on-disk behavior as EncryptedString) and returns a TOTPProvider on read. Unlike EncryptedString, key is REQUIRED — the new type does not carry over the deprecated random-default-key footgun. - TOTPProvider wraps the decrypted secret with pyotp-backed now(), verify(code, valid_window=1) (one tick of drift tolerated by default), and provisioning_uri() (falls back to the type's configured issuer). - generate_totp_secret(length=32) helper. - pyotp wired through the optionals facade (PYOTP_INSTALLED in _typing/typing); TOTPSecret and generate_totp_secret raise MissingDependencyError when pyotp is absent. - Exported from advanced_alchemy.types; registered in both Alembic mako templates (import + sa.TOTPSecret binding) so autogenerated migrations resolve the type. - pyotp added to the test dependency group. Unit coverage (facade flag, guards, provider round-trip, required key, distinct cache keys) plus an encrypted round-trip integration test that verifies the stored DB column is ciphertext, not the plaintext seed.
Ch.4 of crypto-types hardening: a hashed, single-use one-time-code column type for transient email/SMS OTPs, reusing the password-hash backends. - OneTimeCode(PasswordHash) hashes the code on bind (identical storage to PasswordHash) and returns HashedOneTimeCode on read. Requires an explicit backend (no default — each backend pulls a different optional dep). - HashedOneTimeCode(HashedPassword) inherits verify/verify_and_update; exists for intent and a focused docstring. - Expiry, single-use invalidation, and attempt throttling are documented as model/service-layer concerns (e.g. an expires_at column plus deleting the code after a successful verify); the type only hashes and verifies. - Exported from advanced_alchemy.types; registered in both Alembic mako templates (import + sa.OneTimeCode binding). Unit coverage plus a hashed round-trip integration test confirming the stored DB value is an argon2 hash, not the plaintext code.
Ch.5 (final) of crypto-types hardening. - Declare the cryptography and pyotp optional-dependency extras; add both to the test dependency group so the FernetBackend facade and TOTP tests run deterministically. - Usage docs (docs/usage/modeling/types.rst): explicit-key guidance + the random-default DeprecationWarning and callable-key pattern for EncryptedString; PGCrypto CREATE EXTENSION + cryptography install requirements; new Password Hashing (incl. verify_and_update login flow), TOTP Secrets, and One-Time Codes sections (the OTP section spells out that expiry/single-use/throttling are model-level). - Changelog: six entries in the 1.11.0 block covering the PGCrypto fix, default-key deprecation + migration guidance, rehash-on-verify, TOTPSecret, OneTimeCode, and the new extras. Placed in 1.11.0 (the upcoming unreleased block) to match the deprecation warning's version. Docs build clean; make lint green.
Follow-up to the Ch.1 subclass-repr fix: two more types in the family did not reconstruct faithfully under Alembic autogenerate. - PasswordHash.__repr__ now uses type(self).__name__, so OneTimeCode renders as OneTimeCode(...) instead of the hardcoded PasswordHash(...). Both are String(255) so the generated DDL was unaffected, but the migration source named the wrong type. - TOTPSecret overrides __repr__ to include its digits/interval/digest/issuer, so a reconstructed type carries the configured parameters instead of the defaults (the base EncryptedString repr only knew key/backend/length). Each type now evals back to its own class with its own arguments. Adds repr unit tests for both; the existing PasswordHash/EncryptedString repr tests are unchanged.
- Move the crypto type unit tests into tests/unit/test_types/ (alongside test_boolean/test_json/test_vector) where custom-type unit tests live, instead of the top-level tests/unit/ directory. - Add a parametrized integration test running FernetBackend and PGCryptoBackend through identical assertions on PostgreSQL (plain value, unicode, empty string, None), confirming the two backends are drop-in interchangeable: same API, same column type, same round-trip behavior, and both store ciphertext at rest. The only difference is inherent and documented — pgcrypto is PostgreSQL-only (server-side), Fernet works on any backend (client-side).
…pmost imports - Replace per-test create_all(tables=[...]) + drop_all in the encrypted-type integration tests with a shared fixture doing metadata.create_all(engine) (mirrors the password-hash tests). The old pattern did not create the BigIntBase PK sequence on a fresh database, causing "relation <table>_id_seq does not exist" on PostgreSQL/Oracle/MSSQL CI. - Make the new docs code examples self-contained and runnable under Sybil: build HashedPassword/TOTPProvider directly instead of referencing undefined model instances, and import HashedPassword/StoredObject from the topmost advanced_alchemy.types path. - Add TOTP secret, one-time-code, and rehash-on-verify to the README feature list. The crypto type imports remain safe when their optional libraries are absent: importing advanced_alchemy.types pulls in none of cryptography/pyotp/argon2/ passlib/pwdlib (guards fire at construction time, not import time).
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #758 +/- ##
==========================================
+ Coverage 81.97% 82.45% +0.47%
==========================================
Files 103 105 +2
Lines 8844 9039 +195
Branches 1198 1219 +21
==========================================
+ Hits 7250 7453 +203
+ Misses 1273 1260 -13
- Partials 321 326 +5 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
The previous OneTimeCode was a thin alias over PasswordHash — it did nothing a
hashed string column did not. Redesign it to store the code's full lifecycle in
a single JSON column so the type itself enforces one-time-ness.
- OneTimeCode now stores {hash, expires_at, used_at, attempts} as JSON (impl
JsonB, dialect-aware). OneTimeCode(backend, ttl_seconds=None, max_attempts=3).
- HashedOneTimeCode gains is_expired/is_used/is_locked/is_redeemable; verify
succeeds only while the code is still redeemable; redeem(code) verifies and
returns the updated value to persist (marks used on success, records a failed
attempt otherwise); consume()/register_failure() are the explicit primitives.
- Single-use is always enforced (a successful redeem marks the code used);
max_attempts (default 3) locks the code after that many wrong guesses;
ttl_seconds expires it. State persists by assigning the redeemed value back
and committing — the type owns the state model, the caller commits it.
- Add generate_one_time_code(length=6, digits_only=True) using secrets.
- Oracle returns JSON as bytes/str; decode via decode_json like StoredObject.
Updates the README feature entry, consolidates the crypto changelog into a
single PR-referenced entry, and documents the redeem lifecycle.
Describe the PGCrypto change as a correction rather than emphasizing that it never worked. Also bumps the ruff pre-commit hook (v0.15.14 -> v0.15.15) and refreshes uv.lock.
…mplate Argon2Hasher imported argon2-cffi at module top, so importing the module failed when the package was absent — which forced the Alembic migration template to wrap the password-hash backend imports in try/except fallbacks. - Argon2Hasher now imports argon2-cffi lazily (in __init__ and verify) and guards construction on the ARGON2_INSTALLED facade flag, matching the cryptography/pyotp backends. Importing the module no longer requires the package; constructing it without argon2-cffi raises MissingDependencyError. - The sync and asyncio migration templates drop the try/except blocks, import the three backends directly, group them with the other advanced_alchemy imports, and use a tuple for __all__.
…changelog PR - TOTPProvider.__init__ now raises MissingDependencyError when pyotp is absent, matching the construction-time flag guard used by the other backends so a directly-constructed provider fails consistently rather than with a bare ImportError from the lazy pyotp import. - Add the missing :pr: 757 link to the 1.11.0 "deprecate DictProtocol" changelog entry.
The Spanner client's built-in metrics exporter runs a background thread that flushes to Cloud Monitoring. Without GCP credentials it fails to authenticate and logs the failure during interpreter teardown — after the safe-logging handlers are closed — surfacing as "Failed to export metrics to Cloud Monitoring" and "I/O operation on closed file", which intermittently failed the test job on a non-zero exit (seen on 3.9/3.10 while 3.11-3.14 passed with the same code). Set SPANNER_DISABLE_BUILTIN_METRICS=true at conftest import (before any Spanner client is created) so the exporter thread is never started. The emulator does not need these metrics.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Enhance the at-rest cryptographic column types and add two new types — an encrypted TOTP shared-secret type and a hashed, self-expiring one-time-code type.
What changed
EncryptedString/ PGCryptoPGCryptoBackendto run encryption/decryption server-side via SQLAlchemybind_expression/column_expression: writes wrap asarmor(pgp_sym_encrypt(:v, key)), reads aspgp_sym_decrypt(dearmor(col), key). ASCII-armored ciphertext stays in aStringcolumn, so the column type is unchanged. Verified interchangeable withFernetBackendon PostgreSQL — same API, same round-trip for value/None/empty/unicode, ciphertext at rest.cryptographythrough the optionals facade (CRYPTOGRAPHY_INSTALLED);FernetBackendraisesMissingDependencyErrorwhen it's absent. Importingadvanced_alchemy.typesnever eagerly imports the optional crypto libraries — guards fire at construction time.keynow emits aDeprecationWarning(the random default changes per process restart → undecryptable rows). Non-breaking.__repr__renderstype(self).__name__soEncryptedTextreconstructs asEncryptedText(notEncryptedString) in autogenerated migrations.PasswordHashneeds_rehashto all three backends andHashedPassword.verify_and_updatefor transparent rehash-on-login.length128 → 255; narrowedArgon2Hasher.verify(no bareexcept).New types
TOTPSecret— base32 TOTP secret encrypted at rest, returns a pyotp-backedTOTPProvider(now/verify/provisioning_uri);generate_totp_secrethelper.keyis required (no deprecated default).OneTimeCode— a transient code hashed in a JSON column that also tracks expiry, single-use redemption, and wrong-guess lockout, so the whole lifecycle lives in one column:OneTimeCode(backend, ttl_seconds=None, max_attempts=3)stores{hash, expires_at, used_at, attempts}.HashedOneTimeCodeexposesis_expired/is_used/is_locked;verifysucceeds only while the code is redeemable;redeem(code)verifies and returns the updated value to persist (marks used on success, records a failed attempt otherwise).max_attempts(default 3) locks after that many wrong guesses;ttl_secondsexpires it. State persists by assigning the redeemed value back and committing.generate_one_time_code(length=6, digits_only=True)helper.Docs & packaging
cryptographyandpyotpoptional-dependency extras.1.11.0block. All new docs examples execute under Sybil.Backward compatibility
compare_value/compare_expressionretained (documented as an advanced extension point), not removed.📚 Documentation preview: https://litestar-org.github.io/advanced-alchemy-docs-preview/758