fix(db): additional cross-DB and node traffic edge cases (migration scan + node reset time)#5045
Open
rqzbeh wants to merge 3 commits into
Open
fix(db): additional cross-DB and node traffic edge cases (migration scan + node reset time)#5045rqzbeh wants to merge 3 commits into
rqzbeh wants to merge 3 commits into
Conversation
- ExternalProxy migration: change StreamSettings scan from []byte to string (text column on both SQLite and Postgres). Use []byte() for json.Unmarshal. Avoids potential scan/encoding differences in migration of old multi-domain data on PG. - Node mirror inbound creation and updates in SetRemoteTraffic: now copy LastTrafficResetTime from the node's reported snapshot. Previously, resets done on the node (or via API that affects node-owned inbounds) would not update the grace period tracking on the central mirror. This improves traffic reset + node traffic combining accuracy when using the public API to manage node inbounds or when nodes perform resets. These are independent additional issues around node traffic combining, creating mirrored node inbounds from snapshots, and migration code that can affect Postgres (or mixed) setups after API changes or node operations. They do not depend on the previous enable-merge or tag/sub fixes. Base: upstream/main (separate PR).
… tag cleanup)
- In NodeService.GetAll (used for node list/stats): the load of client_traffics
for node inbound IDs used a direct "IN ?" with all IDs. On SQLite this
can hit the bind var limit ("too many SQL variables") when there are
many nodes/inbounds. Chunked using the existing chunkInts + sqliteMaxVars
(same pattern as other large IN queries in the package). This is a
specific scale issue for "node" setups on SQLite (PG is fine with large IN).
- Tag cleanup raw in MigrationRequirements (always runs at startup):
was using SQLite-only INSTR. Fixed to use position() on PG (same as
the previous tag fix on the main branch). Prevents startup crash on PG
after node/inbound API changes that leave old tags.
These are additional specific cases around node traffic/stats combining,
node inbound counts, and startup migrations that can affect Postgres
users or large SQLite node deployments. They are independent of the
enable/traffic core fixes and the prior additional ones.
Added to the clean additional-issues branch for the separate PR.
…r + chunk gone deletes)
- Full dialect-safe client enable merge in setRemoteTrafficLocked:
- Added ClientTrafficEnableMergeExpr() helper (PG CASE with ::boolean
casts to avoid type errors; SQLite numeric for affinity).
- Updated GreatestExpr with ::bigint casts on PG.
- Switched the merge UPDATE from "enable AND ?" to the helper.
This completes the node traffic sync safety for the "only node can
disable" logic across DBs (core of the original symptom after API
inbound updates on nodes).
- Chunked the NodeClientTraffic delete for "goneEmails" (when a node's
snapshot no longer includes clients previously attached to a mirrored
inbound). The "email IN ?" could exceed SQLite bind limit for nodes
with many clients (after API deletes, bulk ops, or structural changes).
Uses chunkStrings + sqliteMaxVars (consistent with the node stats chunk
we added earlier).
These are direct extensions of node traffic combining, mirrored inbound
lifecycle, and API-driven changes that affect client_traffics / NodeClientTraffic
for nodes. Stayed on the clean 5045 branch as requested.
Pushed to update MHSanaei#5045
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Additional database compatibility and node-specific edge cases found during audit of legacy SQLite assumptions vs Postgres + node features.
Fixes in this PR (clean base on mhsanaei/main, separate from the main node-traffic-sync PG+SQLite chain)
ExternalProxy migration (MigrationRequirements):
Node mirror inbound creation and config sync (setRemoteTrafficLocked in node traffic merge):
These are specific to:
Does not include the previous commits (tag cleanup, sub fallback JSON, enable merge, etc.) — those are on the parallel branch/fix/pg-node-traffic-sync-api-update.
Hunted during review of node traffic combining, inbound creation on/push to nodes, and API node/inbound paths for other raw SQL, JSON, type, or reset-related assumptions.