iCourse Subscriber V2#25
Merged
Merged
Conversation
Replaces the split LLM_*/GEMINI_* config constants with a single ordered
provider list in config.py. Each entry records api_key_env, base_url_env
(both optional), default_base_url, and a list of models. resolve_model_providers()
filters to providers whose API key env is set, preserving declared order.
Summarizer now iterates providers → models in order and returns the first
success with model_id "{provider}/{model}". Setting only DASHSCOPE_API_KEY
still works via the modelscope default entry.
Users can edit MODEL_PROVIDERS directly to add openai-compatible endpoints.
New password = sha256("ICSv2:" + stuid + ":" + uispsw).hexdigest().
PBKDF2 iterations 10000 → 100000. Keeps OpenSSL Salted__ envelope so
sql.js / Web Crypto frontends work unchanged.
Backward compat: workflows + frontend try v2 first, fall back to legacy
(stuid + uispsw + dashscope + smtp, 10k iter). On next deploy, data is
re-encrypted with v2. Forks with existing legacy DB upgrade silently.
- src/crypto_box.py: encrypt / decrypt / decrypt_with_fallback
- frontend/js/crypto.js: buildPasswordV2, decryptWithFallback
- 4 workflows (check/single_run/export/reset_data): v2-or-legacy bash
decrypt_blob() helper; deploys re-encrypt with v2 + 100k iter
Pull per-page PPT screenshots through RapidOCR and align them with the
audio transcript on a 10-minute grid. The LLM now sees a bucketed prompt
containing both audio text (from sherpa-onnx FireRed ASR2 CTC, replacing
SenseVoice) and PPT board text per time window.
OCR results live in a new ppt_pages child table keyed by (sub_id, page_num)
with an ocr_status column ('pending' / 'done' / 'failed') so the work is
resumable across interrupted runs and safe under concurrent workers via
WAL row-level locks. Pages identified as the classroom desktop wallpaper
or near-duplicate of the previous slide (Jaccard ≥ 0.85) are filtered
in-memory at prompt-build time.
Transcript-level VAD segments are produced by the transcriber and held in
memory only — they feed the bucketer and are then discarded, so the DB
doesn't double-store the transcript text.
A summary_format_version=1 marker tags PPT-aware summaries; older v0
summaries are re-OCR'd and re-summarized in flat mode at the end of each
run, with emailed_at reset and a 「（含 PPT 识别·更新）」subject suffix
plus inline 更新 badge in the email digest.
Replace the single icourse.db.gz.enc file on the data branch with a
sharded layout:
data/icourse-index.enc encrypted JSON manifest
data/shards/shard-NNNN.db.gz.enc per-shard encrypted gzipped sqlite
Each shard is a self-contained sqlite holding only the courses + lectures
+ ppt_pages rows for the courses it owns; courses are packed greedily into
~10MB shards so a single course growing doesn't churn unrelated data.
The frontend now fetches the data branch tree once, content-addresses
shards by their git blob sha, and only downloads shards whose sha changed
since last load (cached as decrypted bytes in IndexedDB). Typical
incremental updates fall from re-downloading the whole DB to <1MB.
CI workflows (check, single_run, export, reset_data) reassemble the
shards into a working sqlite via the new scripts/db_shard.py, run as
before, then re-shard for deploy. A legacy fall-back keeps reading
icourse.db.gz.enc / icourse.db.enc until the first sharded deploy
completes.
Manual summary editing in the frontend is retired (the sharded layout
makes single-row push-back unworkable); buttons stay as no-op stubs
until subproject D removes them along with the new three-state viewer.
The lecture detail page now cycles between summary, transcript, and PPT
identification on a single button. The button label always advertises the
*next* state ("切换到转录" / "切换到 PPT 识别" / "切换到摘要"), so it
reads as an action regardless of the current view.
The PPT view renders one block per kept page with a `[页 N @ mm:ss]` tag
above the OCR text, mirroring the format the bucketer feeds the LLM.
Pages whose ocr_status isn't 'done' (or whose text is empty) are filtered
out at query time, so the UI never shows 'pending' placeholders.
Manual summary editing was retired in subproject C when the data branch
moved to sharded storage (no clean way to push back a single row from
the browser). Removing the button + textarea + the now-unused
edit-textarea CSS keeps the UI honest about what the frontend actually
supports.
The previous design encrypted each shard with a fresh os.urandom salt and gzip-compressed with default mtime, so re-sharding identical course content produced different bytes each deploy. Combined with `git push --force` on the data branch, every deploy invalidated every shard's git blob sha and forced the frontend to re-download every shard — defeating the entire point of sharding for incremental fetch. Add a deterministic flag to crypto_box.encrypt that derives the salt from sha256(plaintext)[:8], and pin gzip mtime=0 in sharder. Same plaintext now produces identical ciphertext, so unchanged courses keep their git blob sha across deploys and the IndexedDB shard cache hits as designed. Updates the sharder stability test from "only name+course_ids stable" to asserting actual sha256 equality, plus a raw-bytes equality double-check.
…lidators AES-CBC + PKCS7 has a ~1/256 chance of accepting a wrong key (last byte of the garbage plaintext happens to be 0x01). Without a sanity check the bytes get propagated downstream and either crash unhelpfully or fail silently. Add 3 magic-byte validators (is_sqlite, is_gzip, is_json_obj) and wire them through: - crypto_box.decrypt_with_fallback gains a validate= param: if v2 plaintext fails validation, fall through to legacy; if legacy also fails, raise. - sharder.load_index / reassemble_database validate plaintext after each plain decrypt so the CI reassembly loop fails loudly on wrong key. - frontend mirror in js/crypto.js plus pass-through in app.js _loadShard / _loadFromShardManifest / _loadFromLegacyBlob / testAndSave.
Previously after exhausting all max_attempts on IncompleteAudioError, we fell back to transcriber._last_*, i.e. the *most recent* partial result. That punished the case where the first attempt got 90% of the audio and a retry only got 30% — we'd ship the 30%. Track best_duration / best_transcript / best_segments across attempts so the final fallback uses the longest partial we saw. Updates the misleading 'using best result' log to 'using longest (Ns)'.
The previous shape returned a fresh ICourseClient on session expiry, which forced every caller to ' client = _check_session(client) ' or silently discard the refreshed instance. Worse, anything that captured the old client (closures, threads, retry helpers) kept hitting the dead session. Mutate client.vpn / client._userinfo in place so all references see the re-login automatically. Five caller sites simplified.
Schema CREATE statements + migration column lists previously lived in four places (database.py, sharder.py, merge_db.py, frontend/js/db.js), drifting silently every time we added a column. This commit makes src/schema.py the single Python source and frontend/js/schema.js a manual mirror — both files carry a header reminding the next editor to sync the other. Adds the ppt_pages.dhash column (TEXT, nullable) used by commit 6's perceptual-hash dedup pipeline; existing DBs pick it up via ALTER TABLE in Database._init_tables and merge_db._ensure_schema. frontend/index.html now loads js/schema.js before js/db.js so the schema constant is available when the SQL.Database is created.
… filter The old LCS-based 'classroom desktop' filter only caught one screen and the Jaccard adjacent-frame dedup ran post-OCR — meaning we paid OCR cost for every dropped page. Replace both: * Pre-OCR dHash dedup. compute_dhash() runs on raw image bytes; dedup_dhash walks a sliding window (w=5, threshold=4) so adjacent near-duplicates collapse to one survivor. Already-dropped pages are skipped as anchors, preventing 'next-near-last-kept' chains from cascading. * Post-OCR INVALID_PAGE_PATTERNS. ~20 long, topic-specific feature strings drawn from both the classroom desktop wallpaper and the e-learning resource portal screen, all matched after stripping punctuation / whitespace so OCR noise still hits. Bare campus names that could legitimately appear in real PPTs (邯郸校区 etc.) are deliberately omitted. Pipeline status flow per page: pending → (downloaded + dhash written, still pending) → dedup_dropped | (OCR'd: done | invalid | failed). get_done_ppt_pages only returns 'done', so dropped/invalid/failed naturally vanish from prompts. Also: _resummarize_old_lectures now takes course_ids and uses the new get_lectures_to_resummarize_for_courses, so we don't pay re-OCR cost for courses outside the current run's COURSE_IDS list.
Single-lecture wall-clock now collapses from download_video + transcribe + sum(page_download + page_ocr) to roughly max(download_video + transcribe, all_page_downloads + all_page_ocr) plus across-lecture overlap of next lecture's image fetching with the current lecture's LLM round-trip. Pieces: * 20-worker image-download pool (IO bound — GitHub runner has bandwidth). * 3-worker OCR pool (CPU bound; RapidOCR's ONNX runtime releases the GIL on native calls so threads get real parallelism). 2 ASR cores + 3 OCR workers on a 4-vCPU runner is mild oversubscription, hidden behind IO waits — matches the 'a touch hotter than perfect' tuning request. * PrefetchCache: per-lecture image pre-fetcher. main schedules N+1's downloads before calling summarizer.summarize on N, so the LLM round-trip masks the next lecture's network IO. * run() flattens lectures across all courses into one queue, so prefetch works across course boundaries instead of restarting per course. * Database: check_same_thread=False + threading.Lock around update_ppt_page / update_ppt_page_dhash. Reads are main-thread only between worker dispatches so they skip the lock. * OCR worker function classifies + persists (done | invalid | failed) and returns status; main collects via as_completed for accounting. * _check_session is main-thread only; workers see refreshed client.vpn cookies automatically because commit 4 made it mutate in place rather than returning a new client.
Reorganize 18 modules into 5 themed subpackages so the layering is visible at the directory level rather than buried in import statements. Mapping: api/ webvpn, icourse, emailer data/ database, schema, sharder, crypto_box ai/ transcriber, ocr, summarizer, bucketer, ppt_dedup runtime/ scheduler, reporter, config pipeline/ lecture_runner, ppt_pipeline All imports rewritten to absolute form (from src.SUBPKG.X import Y) so the dependency graph reads consistently across src/ internals, tests/, scripts/, and main.py. 72 tests pass post-move. Also incidentally bundles work already in flight on this branch: - merge_db all_courses section guarded for old local DBs lacking the table - explicit DETACH on Windows so tests' os.unlink doesn't hit a file lock - PPT pipeline (dHash dedup + 30-pattern invalid-page filter) - 10-min bucketed prompts + summary_format_version v2 - v2 password (sha256+PBKDF2) with legacy fallback - Sharded encrypted DB with deterministic shard hashes - Frontend: subscriptions editor, prev/next, star pin, summary/PPT cycle - MODEL_PROVIDERS list with DASHSCOPE backward compat - Reporter image/CPU/concurrency snapshots
…e log Three coordinated changes to fix the CPU-under-utilization observed in run #126 (ASR sat at ~45% CPU on a 4-vCPU runner because num_threads was hard- coded to 2): 1. transcriber: num_threads 2 → 4 so ASR can use all cores when alone. 2. scheduler.ResourceMonitor: track an asr_active flag, exposed via Scheduler.set_asr_active(). When True, OCR concurrency caps at OCR_MAX_TARGET_WHEN_ASR_ACTIVE (default 2 = half cores). When False (resummarize phase, between lectures), OCR can ramp up to the full OCR_MAX_TARGET (8). Flag flip immediately clamps the in-flight target down if needed so a 5-worker OCR run doesn't keep stealing cores from ASR's first second. 3. lecture_runner: wrap the transcribe_tail call with set_asr_active(True/False) in a try/finally so even an ASR exception restores the OCR cap. Also adds the OCR-throughput logging that was missing: every 20 finished pages (and at end) print `[OCR <sub_id>] [bar] N/total (R page/s)`. This is throughput, distinct from the existing cpu_snapshot's "OCR busy/target/ max" which reports pool occupancy. Both are needed to read run health — high occupancy with low throughput would now be visible immediately. Resummarize log: per-lecture progress index (idx/total) at start + total elapsed seconds at end, so a long resummarize phase is observable. Summarizer log: include prompt_tokens / completion_tokens in the per-call "Done" line when the provider returns response.usage. Token counts drive both billing and rate-limit decisions, character counts don't.
…MMARIZE_OLD
Production-mode end-to-end test (scripts/test_prod_lecture.py):
CLI that picks one lecture and runs the full pipeline (PPT crawl, dHash
dedup, OCR, audio download + ASR, bucketed prompt, LLM summary) without
touching the SQLite database. Every intermediate artifact lands in
test_prod_out/<sub_id>/: raw images, dedup decisions, per-page OCR
text, transcript, segments JSON, bucketed prompt, summary, plus
metrics.json with per-stage timing and throughput numbers.
Used for A/B comparing ASR backends, measuring per-stage CPU/rate on
the runner, and validating OCR / dedup / bucketer quality without
needing to re-decrypt the production DB. Default target is course
33974 (机器学习系统) — picked because it has real slide PPTs, not
just 板书 camera, so OCR + summary are actually exercised.
Resummarize gate:
RESUMMARIZE_OLD=1 (default off) — re-OCRing + re-summarizing every old
lecture turns a 5-min nightly run into a 2-hour run; we don't want
that as the default cadence. When unset, the resummarize phase logs
a reminder line and skips. Flip on for one-shot manual workflow runs.
.gitignore hygiene:
- Stop tracking _run_logs/, 任务描述2.md, .pytest_cache
- Add sherpa-onnx-fire-red-* and sherpa-onnx-zipformer-* (downloaded
at runtime; same treatment as the existing sense-voice glob)
- Add test_prod_out/ for the new test script's artifacts
Make Transcriber dispatch on a backend string at construction so the
production runtime can switch between ASR models via env without touching
code, and the upcoming A/B profiling can build several Transcribers with
different backend= kwargs side-by-side over the same audio.
Backends:
firered sherpa-onnx-fire-red-asr2-ctc-* (CTC, single model.onnx)
OfflineRecognizer.from_fire_red_asr_ctc(...)
sensevoice sherpa-onnx-sense-voice-* (multi-lang CTC, single model)
OfflineRecognizer.from_sense_voice(..., use_itn=True)
zipformer sherpa-onnx-zipformer-* (transducer)
OfflineRecognizer.from_transducer(encoder, decoder, joiner)
The zipformer branch globs the model_dir for encoder*/decoder*/joiner*
since the exact filename suffix (epoch-N-avg-M) varies across upstream
releases.
New env knobs in src/runtime/config.py:
ASR_BACKEND default firered
ASR_NUM_THREADS default 4 (matches the GitHub runner's 4 vCPU)
The 4-thread default is the same as the previous commit's hard-coded
value, now centralised + tunable so a runner with more cores doesn't
need a code edit.
Refactor extracted _resolve_first() helper for the FireRed/SenseVoice
case (same single-file lookup pattern); zipformer keeps its own glob
since it picks three files.
The end-to-end test script (one-lecture no-DB run with full artifact dump + per-stage metrics) is a profiling/benchmarking tool, not a production code path. Keep dev clean by holding the script on a dedicated 'profiling' branch instead. The Transcriber backend pluggability stays on dev — that's a config knob production users will want regardless of profiling.
- config.ASR_BACKEND defaults to "sensevoice"; ASR_MODEL_DIR to sherpa-onnx-sense-voice-zh-en-ja-ko-yue-2024-07-17 bundle. - workflows (check.yml, single_run.yml) now cache + download the SenseVoice bundle instead of FireRed. - transcriber: generic per-segment cleanup applied to every backend's output — strip JP kana, Korean hangul, bracket tokens (<sil>, <|zh|>), and a fixed whitelist of English filler words (tech terms survive). Collapse orphan-punctuation runs left behind by the deletions. - VAD: bump min_silence_duration 0.25 → 0.8 and cap max_speech_duration at 30s so SenseVoice gets longer, more coherent chunks (its training receptive field) and fewer hallucinated leading kana. - scripts/dump_lectures.py: local-only helper to dump decrypted DB contents per-lecture as markdown.
When resummarize_old_lectures upgrades a v0 row to v2, the original
summary now lands in lectures.old_summary instead of being overwritten.
The frontend can render an old-vs-new diff, and the email pipeline can
optionally include the historical version.
- schema: add old_summary TEXT to the CREATE TABLE + migration list, so
existing DBs get the column via ALTER TABLE on startup.
- frontend/js/schema.js: mirror the new column.
- Database.update_summary_v2 uses COALESCE(old_summary, summary) so:
• first overwrite (typically v0 → v2): captures old text.
• later overwrites: never clobber the once-captured value.
• first write on a fresh lecture: stays NULL.
- merge_db.py: include old_summary in the cross-DB INSERT/UPDATE so
merging a recently-resummarized local into remote doesn't drop it.
deploy-frontend.yml now accepts a 'branch' input on workflow_dispatch so you can test frontend changes from dev (or any feature branch) without merging to main first. The auto push-to-main trigger is unchanged — actions/checkout's 'ref' falls back to github.ref when inputs.branch is empty.
Data-driven stopword list (derived from 7 lectures x 5 courses) removes standalone lines that exactly match known ribbon labels, status-bar items, and keyboard shortcut hints. Regex patterns handle variable-status-bar text (zoom %, word count, page N/M). Cleanup is per-line and only drops exact matches — the same word embedded in lecture content (e.g. "编辑" in a sentence) survives. Reduces OCR text volume 7.8-18.1% across real lectures, all of which is wasted LLM prompt budget from PowerPoint window chrome.
Old shards produced by prior code versions lack migration columns (e.g. old_summary). INSERT ... SELECT * would fail with 'N columns but M values'. _migrate_shard_schema now aligns attached shard tables to main before the copy, matching on LECTURES_MIGRATION_COLUMNS and PPT_PAGES_MIGRATION_COLUMNS from src/data/schema.py.
RapidOCR is single-threaded CPU-bound — one page saturates one core. Raising concurrency past 2 never increases throughput but produces noisy target-churn logs every second as ResourceMonitor oscillates.
…rrency OCR scheduling analysis showed that the CPU-based dynamic adjustment between 1 and 2 concurrent workers provides no real throughput benefit since RapidOCR is single-threaded CPU-bound. The complexity of DynamicSemaphore, ResourceMonitor, psutil polling, and six config knobs was not justified for a ±1 adjustment on a 4-core runner. Changes: - Remove DynamicSemaphore class entirely - Remove ResourceMonitor class entirely - Replace both with a fixed threading.BoundedSemaphore(OCR_MAX_TARGET=2) - Remove set_asr_active() from Scheduler and LectureRunner - Remove psutil dependency from requirements.txt - Remove config knobs: OCR_INITIAL_TARGET, OCR_MIN_TARGET, OCR_MAX_TARGET_WHEN_ASR_ACTIVE, RESOURCE_MONITOR_CPU_HIGH/LOW - Remove cpu_snapshot() and cpu_target_changed() from Reporter - Remove scheduler.start() from main.py (no monitor thread to start) Also fix: GITHUB_TOKEN push auth in check.yml and single_run.yml — use 'x-access-token' username format required by the GitHub API instead of the actor username.
Existing resummarize_old_lectures() serializes PPT OCR and LLM calls for each lecture. This script uses a ThreadPoolExecutor to overlap OCR (CPU-bound) with LLM API waits (IO-bound), reducing wall-clock time from the sum of both phases to the max of either. Usage: python scripts/resummarize_parallel.py # all courses python scripts/resummarize_parallel.py --workers 6 # 6 concurrent python scripts/resummarize_parallel.py --limit 50 # first 50 only
Phase 1 — Catalog: scan all targets, register PPT pages in DB. Phase 2a — OCR saturation: N threads each run PPT pipeline for a batch of lectures, push completed results to a shared queue. Phase 2b — LLM concurrent: M workers pull from the queue, assemble prompts, call the summarizer API in parallel. The two pipelines overlap: CPU-bound OCR and IO-bound LLM API calls run simultaneously. Wall-clock time = max(ΣOCR, ΣLLM) not ΣOCR+ΣLLM.
Groups courses into ~5MB shards by update recency:
- "Active" courses (processed within last 14 days or with
pending/errored lectures) are grouped together so frequently
changing data stays in the same shard.
- "Stable" courses (fully processed, no recent activity) are
packed by first-fit into separate shards. Since they never
change, these shard hashes are stable across runs.
- A course is never split across shards.
Output format uses the same index + shard layout as sharder.py
so the frontend and db_shard.py can consume it unchanged.
Active courses now use a 3MB per-shard target (instead of 5MB) so frequently-changing data produces smaller update payloads. Stable courses remain at 5MB. Factorised the packing logic into a shared _pack() helper used by both groups.
When a page is dropped from the window, the next non-dropped page beyond the original window boundary fills the vacated slot. This ensures the anchor always compares against exactly ``window`` non-dropped items, making dedup more aggressive on long runs of near-identical screenshots (e.g. a static slide held for minutes).
Sliding-window dedup for PPT animation reveals. Uses 3-gram directional containment (not symmetric Jaccard): if a page's 3-gram set is >85% contained in a nearby page that is >10% longer, the shorter page is dropped as a progressive-disclosure subset. No line heuristics (OCR line breaks are unreliable). <55 lines total.
Use explicit refspec for git fetch so origin/data tracking ref is always created. Replace rev-parse with actual file-existence check so force-pushed branches with different history are detected.
Shallow clones reject non-fast-forward ref updates without the + prefix. Use git ls-tree (reads tree directly) instead of git cat-file -e (requires blob fetch) to detect which DB format is on the data branch.
OCR is now deferred (defer_ocr=True) in LectureRunner. The PPT pipeline downloads images and runs dHash dedup before ASR, but OCR submission is postponed to handle.drain() which runs after ASR completes. This gives ASR exclusive CPU access during transcription — previously OCR competed with ASR and degraded transcription throughput. The handle stores pre-dedup'd image bytes and submits OCR lazily on drain(), so no architecture changes are needed beyond the defer_ocr flag.
Appends (cpu=N% mem=N%) to Transcriber progress, OCR progress, and image download progress lines using psutil (added back to deps). The meter is lazy-imported on first call, so startup is unaffected on systems without psutil.
Between Phase E (current OCR drain) and Phase F (LLM call), kick off the next lecture's PPT pipeline (download → dedup → OCR) so its OCR runs in the background pool during the 60-120s LLM API wait. prefetch_and_ocr() is a light submit() that doesn't drain or discard the prefetch cache. submit() detects if pages were already OCR'd by checking for pre-existing dhash values and skips re-processing.
prefetch_and_ocr stores OCR futures in PPTPipeline._prefetched_ocr. submit() pops and drains them before starting ASR if the LLM returned before OCR completed. This prevents OCR from competing with ASR for CPU in the edge case where the LLM API responds faster than expected.
Progress lines now include: [CPU] or [IO] tag indicating the limiting resource down=XXXKB/s (network receive throughput, delta-sampled) Example output: [Transcriber] Progress: 1154s ... 65 segments so far [CPU] (cpu=99% mem=68% down=120KB/s) [Images 616503] [####] 30/94 (10.4 pic/s) [IO] (cpu=12% mem=68% down=8500KB/s) [OCR 616503] [####] 20/88 (0.53 page/s) [CPU] (cpu=100% mem=68% down=5KB/s)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces a major refactor of the GitHub Actions workflows to support a new sharded and versioned database format, as well as improvements to deployment flexibility and model caching. The workflows now use a more robust and scalable approach for storing, fetching, decoding, and deploying the database, with backward compatibility for legacy formats. There are also enhancements to secret management and deployment configuration.
Database Sharding and Workflow Refactor
check.yml,export.yml,reset_data.yml) to use a sharded database format, with scripts to shard/reassemble the database and support for both new and legacy formats. This includes new logic for fetching, decoding, and deploying sharded data, and improved handling of concurrent updates. [1] [2] [3] [4] [5]Workflow and Model Improvements
Deployment Flexibility
deploy-frontend.ymlto allow manual selection of the deployment branch via workflow dispatch input, improving deployment flexibility for different environments. [1] [2]Secret and Configuration Management
CRAWL_TERM,RESUMMARIZE_OLD, andDEEPSEEK_API_KEY. [1] [2]These changes collectively make the workflows more robust, scalable, and maintainable, while setting the stage for future database growth and improved deployment practices.