Telegram support & broadcasts, and the scaling work that came out of it - #34
Merged
Conversation
A user writes to a separate support bot; the bot forwards into a per-user topic of the operator's forum supergroup, and an admin's reply in that topic is copied back. No message history is stored — Telegram is the store, and relaying by message id is what makes screenshots and voice notes work without the bot parsing a single attachment. The bot is separate from the user bot on purpose: inside the user bot every message would need a "support request, or just tapping around the menu?" decision. Here there is no menu, no plans and no registration, so nothing has to be guessed, and the user bot changes in only two places — a link button in the menu and in the welcome screen, both hidden unless support resolves. - one topic per chat, re-opened if the admins delete it - "✅ sent" only after a successful forward: a confirmation the operator will never see is worse than an honest failure, because the user then waits - a "//" prefix keeps a message between admins - per-chat flood limit, since everything received lands in the admin group - a check button that validates group, topics and bot rights in one go — a bot added as a plain member still receives users' messages but, under Telegram's group privacy mode, never the replies, which is invisible from outside Sends go through the existing rate limiter and APIError retry path; permanent failures are classified by status code rather than description text.
Groundwork for mass broadcasts: who they go to, and how someone leaves. The audience is a new tg_subscribers table rather than users.tg_chat_id. That column only knows people holding a VPN account right now, and misses the ones a broadcast most needs to reach — waiting on moderation, mistyped invite code, account deleted but still sitting in the bot. It is also the only place "this chat blocked the bot" can live, and a blocked chat must stop being addressed or it burns a send slot every run, forever. The migration backfills from linked users, without which the first broadcast after an upgrade would reach only those who happened to write since, and read as broken. Opt-out is /mailing_off, /mailing_on and /mailing (current state), published through setMyCommands — an opt-out nobody can find isn't one. Both directions are named explicitly rather than hidden behind one toggle. Unsubscribing says what was NOT switched off, since the alternative to a findable unsubscribe isn't a captive audience, it's people blocking the bot, which is irreversible and kills payment confirmations and support replies too. Contact re-activates a blocked chat but never reverts an opt-out. upload now returns the sent message so callers can keep the file_id: one upload re-sent by id, instead of one transfer per recipient.
…ivery Compose in the panel (text, one attachment, URL buttons, audience filter), send through the user bot, watch an exact progress bar. The recipient list is a table, not a slice in memory. A run to thousands of chats takes minutes; holding it in memory means a restart either loses it or, retried, sends everything twice. On disk, the worker just picks up the remaining pending rows, so a restart is a pause — and the primary key on (broadcast_id, chat_id) is what makes a resume unable to repeat anyone whatever the worker does. Counters are derived from those rows rather than stored, so they cannot drift from what actually happened. Pacing is left to the existing rate limiter. A second regulator alongside it would only let the two together exceed the ceiling each was written to respect. - audience snapshotted at launch, never recomputed, or the total would move under the progress bar it is measured against - attachment uploaded once, then re-sent by file_id - a broadcast is created paused and started after its attachment is on disk: the worker addresses that file by id - permanent refusals mark the subscriber blocked, so later runs stop spending a send slot on a chat that can never receive - retry re-queues failures but not blocked chats, which would refuse identically - test-send to the operator's own chats first: broken markup seen by the whole audience can only be corrected by another broadcast
…casts Adversarial review of the three preceding commits. Every fix below is a real failure path, not a style change. Broadcasts: - A cancelled run could be fully resurrected. Cancel leaves its queued recipients untouched, and retry re-opened any run with a failure — so "повторить неудачные (12)" delivered the whole remainder of a message the operator had just stopped. Retry is now refused unless the run finished, and the store's re-open is conditional on that too. - Retry ran as three unsynchronised writes; a crash between them left a 'done' broadcast holding pending rows, which no API could move. Now one transaction. - The worker wrote status unconditionally, so a pause or cancel landing during a slow upload was silently reverted — the same invariant, reached another way. All worker writes are now conditional on the run still being running. - The first recipient of a broadcast with media got no buttons: the upload path never sent reply_markup. So did the operator's own test send, meaning the one guard against bad markup disagreed with what the audience receives. - Attachments were deleted only on the 'done' path, leaking a file for every cancelled or abandoned run into the directory that gets backed up. Swept now. - A run stalls invisibly if the user bot is switched off; it now says so. Support relay: - A topic closed by an admin — the natural "handled" gesture — permanently ended that user's support: TOPIC_CLOSED isn't "thread not found", so nothing reopened it and every later message failed. Reopened and delivered now. - The "//" internal-note escape only read Text, so a note written as a photo caption was relayed to the user it was about. - Topic titles were truncated by bytes against a character limit, so a name with emoji produced invalid UTF-8 and Telegram refused it — a user could break their own support permanently just by choosing a name. - Forum service messages (rename, close) were relayed as replies, failing and posting alarming delivery warnings for routine housekeeping. - The flood limit sat behind the /start branch, leaving the one command everyone sends first unlimited, and it answered every rejected message — turning a flood into more outbound traffic than inbound. Counted first, answered once. - An offset survived a token change in both bots. Update ids are per-bot, so the old offset ACKed away the new bot's backlog and swallowed messages silently. - Switching support to another group wrote the new id before clearing the stale topic mappings; if the clear failed, nothing ever retried it and replies would land on whoever held that topic id in the old group. - Token-collision checks only compared against enabled bots, so a shared token passed validation and broke on the day the other bot was switched on. - A transient lookup failure could overwrite a known user_id with NULL, dropping that person out of every audience that filters on having an account. Not fixed, documented instead: anyone in the support group can read every thread and post as support. Enforcing admin-only would break operators whose support staff are ordinary members, so the settings page now says to keep the group closed.
…its id Finding a supergroup id by hand meant reading it out of a Telegram Web URL and remembering to prepend "-100", or adding a stranger's id-printing bot to the group where customer support conversations will live. The bot already receives its own membership events; it just wasn't listening. The chicken-and-egg was in the poll loop: it started only once support was enabled AND a group was set, so the bot went silent exactly when it could have supplied the one thing the operator was missing. It now polls on a token alone and relays only when fully configured — and answers "поддержка ещё не настроена" meanwhile, instead of swallowing messages sent mid-setup. Groups reach the picker via my_chat_member (added to allowed_updates for this bot only) and via any message seen in a group, which also picks up groups joined before this existed. Each option shows why it can't be used yet — no topics, bot not admin — at the moment of choosing rather than after clicking Проверить. They are candidates, never an automatic choice: the bot is reachable by @username, so anyone can add it to a group and land in the list. Applying one without the operator saying so would let a stranger redirect every support conversation to a chat they control. Manual entry stays as a fallback, and a pasted bare id is now repaired to its -100 form — the single mistake everyone makes. Also drops self-service unlink from the user bot. It only ever cost the person their access: the account survives, but they land back on the welcome screen and write to support to get it back. An operator who genuinely needs to detach a chat has the button in the user's card. Old menus still carrying the button now do nothing, which is the intended outcome.
… commands
Three things the first cut got wrong, all found by actually walking the setup.
The group field showed a bare ID input whenever no groups were known — which is
exactly the first moment an operator looks at it, directly under a line promising
they wouldn't have to type an ID. It now shows what the panel is waiting for
("добавьте бота в группу — она появится здесь сама"), with manual entry one click
away for anyone who wants it.
Enabling support without a group is still refused — a support button leading
nowhere is worse than no button — but nothing said so, leaving a dead Save
button and no way forward. The reason is now stated where the problem is, along
with the non-obvious way out: the bot polls on a token alone, so saving with the
switch OFF is what makes the group list appear.
Worse, that same rule was disabling the WHOLE save bar, which covers every
Telegram section. A half-filled support block froze the admin bot, the user bot
and the backup schedule behind it. Only a missing token disables saving now.
Separately: /mailing_on and /mailing_off are gone, leaving /mailing alone. The
card it opens already shows the current state and the one button that flips it,
so a command per direction made the menu longer without saying anything the card
doesn't.
It implied a load in progress that would resolve on its own. Nothing is loading — the panel is waiting on the operator to add the bot to a group, and an animation that never stops promises progress that isn't happening. The text now also says discovery works while support is still switched off, which is the order the setup actually has to happen in.
The empty state said 'add the bot to a group and it will appear here', which strands the most common case: the bot is normally already in the group by the time anyone opens these settings. Telegram gives a bot no way to enumerate the groups it belongs to, and it never replays the 'you were added' event — so a group joined before this feature existed stays invisible until something happens in it. The panel now says to post any message in the group, which is the one thing that surfaces it.
…an admin A group discovered through a message was recorded with is_admin=false, because a message says nothing about the bot's own rights. That guess was then shown as fact — "бот не админ" next to a bot that was an administrator — sending the operator off to grant a permission it already had. The rights are now looked up with getChatMember (bot id resolved once per token), and the group is recorded first, so a failed lookup costs an accurate label rather than the candidate. The Client's API root is now injectable, which is what makes these paths testable at all: the previous test could only pass a nil client and assert that nothing was sent, which stopped working the moment discovery needed to ask Telegram anything. UI: the group picker shows the chat id next to the name — names repeat and get renamed, and the id is what actually gets saved. Also lands the empty-state text that a previous commit claimed to add but did not: a silent string replace missed, the build went out unchanged, and the "already added it?" instructions were never actually there.
…elected Changing the support group has to drop the topic mappings — thread ids belong to the group that issued them, and a reply in the new group's topic 7 would otherwise reach whoever owned topic 7 in the old one. But the test fired on 0 → X as well, so simply saving the settings form after the group field had been empty wiped mappings that were still perfectly valid for that same group. The consequence is not recoverable. Telegram gives a bot no way to list its topics, so an orphaned thread can never be found again: the next message from that user opens a SECOND topic with the same title, and the operator ends up with two threads for one person and no way to tell which is live. Both sides of the comparison must now be a real, different group. Topic creation is also logged now — it was the one event with no trace anywhere, which is why a duplicate could only be guessed at after the fact.
The 8-goroutine fan-out shares one *model.Broadcast and one SQLite connection, and sending the same person twice is the failure that cannot be taken back — so it is worth exercising rather than reasoning about. Uses the client's injectable API root: 24 recipients, each must receive exactly one message, and permanent vs transient refusals must be told apart (a 500 deactivating a subscriber would quietly shrink every future audience). Passes under -race.
Comment alignment drifted when the offset field was added between the commandsFor and pending lines.
Alignment drifted when the broadcast actions and category were added.
…feature Three independent reviews of the support relay, the broadcast path, and the config/HTTP/UI surface. Two of them found the same message-misrouting hole independently, which is why it is fixed at the root rather than patched. Secrets - The bot token leaked wherever an error was printed. Go puts the request URL — and the token lives in its path — into every transport error, so a DNS blip or a timeout wrote full control of the bot into the panel's world-readable log file, into admin toasts, and, for the support relay, into the support group itself, whose members the UI already warns may not all be trusted. The client now redacts its own token from every error it returns. Message misrouting (found twice, independently) - Topic mappings were keyed by thread id alone, but a thread id is a message id and those are only unique within a chat. Mappings from a previous group stayed addressable after a group switch: an admin writing in the new group's topic 7 reached whoever owned topic 7 in the old one, and that user's next message landed in a stranger's thread. Guarding it with "reset on change" had to be exactly right on every path (A→B, A→0→B, re-picking A after clearing the field) and each way of being wrong either leaked across customers or orphaned live conversations Telegram gives no way to find again. Rows now carry their group, so a foreign mapping simply never matches and no transition needs handling. The unique index is per group too — the global one wedged a new user out of support whenever a fresh group reused a thread id. - The "//" internal note was matched only at the start of the whole message, while the pinned card promises it works per line. An admin who answered the customer and added a line about them below delivered both. Broadcasts - A file Telegram rejects (an oversized photo — our cap is 20 MB, sendPhoto's is 10) was treated as one bad recipient, so the run walked the entire audience one person per pass, re-uploading the whole file each time and failing all of them. Only an unreachable recipient consumes a target now; anything else pauses. - Selecting pending rows is not claiming them, so a failing MarkTarget (disk full, DB locked) re-sent the same 50 people every pass, forever. Losing the ability to record an outcome now stops the run. - An attachment was deleted once a run finished even if nothing had ever uploaded, leaving "повторить неудачные" with nothing to send and no way back. - Unsubscribing mid-run was ignored: the audience snapshot fixes who is in scope, but it must not override a decision the bot has since confirmed to the person. Skipped recipients get their own state so progress still reaches its total. Also: absent JSON fields no longer read as empty (a stale browser tab could wipe a bot token or the whole support relay and get a 200); getMe is skipped when the token hasn't changed, so saving no longer fails during a Telegram outage; all three bots reject each other's tokens, not just the support one; the rights lookup is debounced per group and no longer overwrites a verified "admin" with a guess when it fails; the candidate list is capped and pruned; an unknown broadcast id is a 404 rather than a raw SQL 500; the test send is validated like the real one; and the UI stops polling a paused run forever.
The support relay and broadcasts grew across five files as the feature was built and reviewed — including one that only existed to fix another's index. Nothing has released them, so they collapse into a single 0031 that states the final schema and keeps the reasoning behind it. Numbered 0031 because this branch is cut from main, whose head is 0030. Databases that already ran the five (the test server) need their schema_migrations rewritten to name the merged file instead: migrations are keyed by filename, so the runner would otherwise replay DDL against tables that already exist and refuse to start. The resulting schema is identical apart from the column order inside tg_support_topics, which is addressed by name everywhere.
…created Reviewed the FIXES this time, which is where the worst finding was. The per-line "//" check I added last round made things worse, not better. It detected a note anywhere in the message and then dropped the whole message — copyMessage copies, it cannot edit — so an admin who answered the customer and appended a note below sent them nothing at all, silently, while their text sat in the topic looking delivered. Withheld messages now say so in the thread. The same silence covered topics that belong to nobody (opened by hand, or left over from a previous group): an answer typed there reached no one and nothing said so. Support relay - The rights debounce compared against last-SEEN, which every message bumps. A busy group was therefore checked once and never again — so one transient getChatMember failure pinned it at "бот не админ" forever, sending the operator to grant a permission the bot already held. A quiet group, meanwhile, was checked on every single message. Separate rights_at column. - The candidate prune sat inside the "no token configured" branch, so on a working install it never ran and the table it bounds grew forever. Worse, the picker ordered by recency under a cap: anyone can add a public bot to 30 groups, and those fresh rows evicted the operator's own group from the list. Usable groups (forum + admin) now sort first. - createForumTopic returning no thread id stored 0, which addresses General, where replies are dropped by the thread guard and no recovery ever fires. Broadcasts - primeMedia paused the whole run on ANY upload error, including a timeout or a routine 502 — over-correcting last round's fix. Only a 400 about the content pauses now; a transient fault leaves the recipient pending and retries. - A failing status write in finish() left the row running with nothing pending, and step() returned "worked" — a tight loop on the panel's single DB connection. Same for a persistently failing upload. - The pause that stops a resend loop writes to the database, which is precisely what has failed in the case it exists for. An in-memory quarantine now holds even when nothing can be persisted at all. - The UI never learned about skipped recipients, so a run where anyone unsubscribed mid-flight froze short of 100% with polling already stopped. Settings - All three bots are validated before any is written. Saved in sequence, a failure on the third — a support token unverifiable while Telegram was unreachable — left the first two committed, the request reported as failed, and nothing in the audit trail. - The cached support @username could never be refreshed, so renaming the bot in BotFather left a dead t.me link forever. The check button now persists it.
…poser UI
Broadcasts now sit as a sub-tab of Пользователи rather than at the top level —
the audience is the bot's users, and composing one is something you do while
looking at them. The tab is hidden unless the user bot is on, since that is what
delivers; a tab whose every action errors is worse than no tab. /api/me carries
the flag, alongside billing_enabled.
A user's card gains "Отправить сообщение" in its Telegram section, shown only
with a linked chat and a running bot. It is a broadcast of one, but answered
synchronously: for a single recipient the operator wants to know now whether it
arrived, not to watch a progress bar. Telegram's refusal is translated ("the user
blocked the bot") instead of surfacing a raw API error, and it gets its own audit
action — nothing else records that a customer was written to.
UI fixes from using it:
- The history block used a bare Card, which carries no padding, so its contents
sat flush against the edges.
- The attachment field was a raw file input, rendering the browser's own
untranslated "Файл не выбран" next to styled controls. Now a button that says
what it does, plus a line about what happens to an image versus a document.
- The text field asked for hand-typed HTML tags. It now has a formatting bar that
wraps the selection. Deliberately not a Markdown editor: Telegram accepts a
small fixed set of HTML and nothing else, so a converter would add a layer
whose mistakes only surface once the audience has the message.
…ssages, UI tidy The test send used the ADMIN bot's linked chat list but the USER bot's token. That only worked by accident — chat ids are global Telegram user ids, so another bot's token reaches the same chat, but only if the admin had separately started the user bot; otherwise Telegram refuses outright, since a bot cannot write first. It now goes through the admin bot, whose linked chats exist for exactly this. No fidelity is lost: a preview has to prove the markup, the buttons and the attachment, and every bot renders those identically — only the sender's name differs, which is not what anyone is checking. The single-user message now takes an attachment, through the same multipart parser the broadcast composer uses, so whether a file goes out as a photo or a document does not depend on which screen sent it. The formatting bar moved into a shared HtmlEditor used by both composers, and its controls are no longer a mix of letters and emoji: each letter renders in the style it applies (Ж bold, К italic, Ч underlined, S struck), and code/link/spoiler get SVGs in the same stroke style as the rest of the panel's icons. Payments moved under Пользователи too — they are about what users pay for, not a separate destination.
…ences
Until now the panel told the OPERATOR that somebody's subscription ran out. The
person it happened to found out by failing to connect — the worst moment to learn
it, and the point at which they write to support instead of renewing.
The user bot now has its own notification switches, mirroring the admin ones:
subscription running out (horizon configurable, 1–30 days), expired, traffic
running low (80%), traffic exhausted, too many devices, access switched off, a
payment confirmed, and a decision on a moderated signup. The last two already
sent messages unconditionally; they are gated now, so an operator who turns the
notices off does not still have the bot writing to people.
Two of these are not edge-triggered, so they carry their own markers:
- The expiry warning stores the expiry it warned about. A renewal moves
expire_at, the stored value stops matching, and the warning re-arms itself with
no extra bookkeeping — and no way to leave it stuck armed or stuck spent.
- The quota warning has no changing value to key off (usage grows inside one
limit), so it stores a flag cleared once usage drops back under the threshold,
which is exactly what a reset or a bigger plan does.
Broadcast audiences gained targeting: expiring within N days, seen within N days,
not seen for N days, never connected. The horizon travels inside the audience
value ("seen:7") rather than in a second column, since an audience is written
once at launch and read back only to be displayed — a separate field would exist
purely to be kept in sync with this one. Out-of-range and malformed values are
refused rather than resolving to an empty list, which would read as "nobody
qualifies" when the truth is "the panel didn't understand you".
…e's one file The branch is going up as a squashed PR, so the feature ships as a single migration rather than as the trail of files it grew through. Nothing has released either of them. A database that already ran both (the test server) needs no surgery this time: 0031 is already marked applied, so the merged file is skipped, and the row naming the now-deleted 0032 is inert — the runner only ever asks whether a file it has was already applied.
…s over The crash alert had no counterpart, so an operator saw "⚠️ Xray аварийно завершился, перезапускается автоматически" and then nothing — leaving them unable to tell a two-second blip from an ongoing outage, and reaching for SSH either way. The all-clear is fired by the SUPERVISED restart path only. Apply-driven restarts — a reconcile, a renewed certificate — happen routinely, and reporting those as recovery would drain the meaning from the one message that says an outage ended. It is also only sent when this panel actually raised the alarm. The crash alert is throttled to one per five minutes, so a crash loop would otherwise report a stream of good news for outages nobody was told about.
…anch added The worst finding was self-inflicted: adding the StatusDisabled case cut the device-limit branch in half, so its audit row and webhook moved under it. Since then, exceeding the device limit wrote no journal entry and fired no webhook, while manually disabling a user emitted a false user.device_limited event to every integration subscribed to it. Both statements are back where they belong. Audiences - A deleted account left its id on the subscriber row, so the filters read a missing user's zero values as facts: "ни разу не подключался" collected ex-customers who had connected the day before, while "без аккаунта" — the audience documented to hold exactly those people — excluded them. Deletion now detaches the row, and the resolver keys off roster presence rather than a non-zero id, which also repairs rows that already went stale. - The preview skipped validation, so an unrecognised audience previewed as "0 получателей" while the send would refuse it — and an omitted one previewed 0 and then went to everybody. Preview and send now normalise identically. - "Не заходили N дней" matched accounts registered minutes ago, since never connecting counts as not being seen. It is now floored by the account's own age. Notifications - One registration notice still bypassed the new gate, so an operator who turned the notices off could still have the bot write to somebody. - The quota marker was never re-armed for a user moved to an unlimited plan. Moving them back to a limited one carries usage over, so a user already past 80% would never be warned again on the new plan. - The expiry horizon could only be saved together with the category map; a body carrying just the horizon returned 200 and changed nothing. Elsewhere - The per-user message audit row now names the recipient. The body is deliberately never stored, so without it the row could not answer the one question it exists for. - That endpoint parsed inline buttons and dropped them silently. - userBotEnabled was captured at login and never refreshed, so the broadcast tab and the send button stayed hidden after enabling the user bot — and stayed visible after disabling it, with every action behind them answering 400.
…d money atomic
The panel's SQLite pool is a single connection with synchronous=FULL, so every
statement is its own commit and its own fsync. Measured on the 1-core test box:
223 writes/sec, exactly the disk's fsync rate. The write paths that mattered ran
row by row, so the ceiling scaled with the number of users — ~450 online was a
hard wall, and six seconds of every sixty-second poll cycle was spent writing
while holding the one connection the whole panel shares.
Batching, not synchronous=NORMAL: durability is kept.
stats poll (500 users) 6.075s -> 70ms (87x)
access-log tap (500 hits) 2.217s -> 53ms (42x)
dashboard summary, per tick 8.1ms -> 3.4ms (2.4x)
RecordAccess now only buffers in memory — it runs per access-log line, so it does
no I/O at all; a 5s loop writes the batch. That also moved the per-sighting
WorkingUsers query to once per flush. CountUsers replaces loading every user row
(and decrypting every password) to compute four numbers, and statusFeed computes
the dashboard payload once for all viewers instead of once per open tab, idling
when nobody is watching. Write load is now roughly constant instead of linear in
users.
Making those paths transactional exposed a class of bug they had been hiding.
Payment confirmation wrote five separate autocommits with the terminal status
FIRST and the plan LAST — and every retry path selects status = 'pending', which
the claim had already cleared. An ordinary restart in between took the money and
left nothing able to notice. Node traffic ingest had the same shape: the
watermark committed before the traffic it covered, so a failure meant the node's
resend was rejected as a duplicate and that batch was lost for good. Both now
commit the claim together with what it pays for.
Two consequences of batching that row-by-row code shrugged off:
- a foreign-key violation now rolls back the whole batch, so one departed user
could void everyone else's traffic (and wedge a node forever). Both inserts
guard on EXISTS.
- the abandoned-order sweep asked the provider only after cancelling by age,
so an outage longer than a day cancelled orders that had in fact been paid.
It now asks first.
Also from review: traffic_daily had no retention sweep and grew forever (capped
at a year, with covering indexes for the queries that read it); designating a
paid plan as the free/trial one stranded its existing subscribers permanently
expired; one plan could be picked for both roles, which stranded every
registrant; and Xray's exit during shutdown was reported as a crash, so an
ordinary `systemctl stop` paged the operator with an alert no all-clear ever
followed.
Verified on the test box: migrations applied, covering indexes in use, service
healthy, data intact.
…ard missing a secret Two gaps a security pass turned up. Both predate this branch; both are cheap to close and unpleasant to leave. The public user bot had no per-chat limit, though the support bot was built with one. Its updates loop is a single goroutine answering synchronously, and every reply waits on the outbound one-second-per-chat slot — so ~60 junk messages from one chat stalled registration, menus and payments for everyone else for a minute, and each one wrote a subscriber row before any gate. The support bot's limiter is now a shared chatLimiter used by both, applied ahead of trackSubscriber. Invite codes get their own, much tighter budget. The comparison was already constant-time, but that only closes a timing oracle: nothing bounded how many codes a chat could try, operators pick short memorable ones, and a hit mints a real account on the trial plan. Five attempts per ten minutes, charged on every attempt so a correct guess mixed into a run of wrong ones doesn't refill it. Separately, dbHasEncryptedSecrets did not know about tg_support_bot_token. That guard is what tells "fresh install, no key yet" apart from "the key is gone", so an install configured with only the support bot looked fresh: boot would mint a new key and orphan the ciphertext, silently. Every encrypted column is listed now — including the nodes table — and the column was missing from the re-encrypt list too. Covered by tests that walk the columns one at a time, since the failure mode is precisely a column nobody remembered.
"Общий объём трафика" summed users.used_up/used_down, which the quota reset zeroes per user — so it added up a different period for everybody (all time for one, the current 30-day cycle for another, today for a third) while reading as a lifetime figure, and a deleted user's share vanished from it entirely. The per-day history on "Статистика" is the honest answer to that question, and it is already what the "Трафик сегодня" tile beside it was using. "Сеть сервера" showed NIC throughput for the whole host — panel, SSH, updates and all — sitting directly above the VPN figure it never matched. The fields stay in Summary/SystemStatus: /api/v1 serves them, and CountUsers computes the sums in the same query either way.
The dashboard compared a node's reported Xray version to PinnedVersion with ==. PinnedVersion carries a leading "v" and `xray version` output does not, so the two never matched: every node was stale forever, and updating one changed nothing. VersionMatchesPinned exists for exactly this and the Nodes tab already used it — which is why that tab called the same node healthy while the dashboard asked the operator to update it. Found from a live install: a node reporting 26.6.27 against a pinned v26.6.27.
Both the stats page and a user's card plotted one total line, so "where does this person actually connect?" had no answer in the panel — the data was there (traffic_daily carries node_id) but nothing read it. StatsSeriesNode had been sitting in the store with no caller at all. A compact per-server split under the chart, shared by both pages: they differ only by user_id. Bytes plus a share bar, busiest first. Kept as numbers rather than more lines on the chart: it already draws two, and a line per server would be 2xN — six lines on three nodes, which answers the question worse than a list does. Names resolve server-side so the client needs no node list, and an operator without rights to the Nodes tab still sees the split. NodeNames covers tombstoned nodes too: traffic rows outlive the node they name, so a period reaching back before a deletion can still label its bytes instead of printing a bare id. Renders nothing on a single-server install — with only the panel's own node the split would just repeat the total above it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Telegram support & broadcasts, and the scaling work that came out of it
Closes #29.
Two things landed on this branch. The first is the feature the issue asked for; the
second is what fell out of asking "how many users can this actually hold?" while
testing it.
Support relay, broadcasts, user notifications
A third bot relays user support into a forum group, one topic per person, so
operators answer from Telegram instead of the panel. Broadcasts have a
per-recipient table, so a delivery that dies halfway resumes rather than restarts —
and nobody gets messaged twice. Users now hear about their own account (expiry,
quota, payment), not just admins.
Scaling and durability
The panel's SQLite pool is a single connection with
synchronous=FULL, so everystatement is its own commit and its own fsync. Measured on the 1-core test box:
223 writes/sec, exactly the disk's fsync rate. The hot paths wrote row by row,
so the ceiling scaled with users — ~450 online was a hard wall, and six seconds of
every sixty-second poll cycle went to writing while holding the one connection the
whole panel shares.
Fixed by batching, not by relaxing
synchronous— durability is kept.RecordAccessruns per access-log line, so it now only buffers in memory; a 5sloop writes the batch (which also moved a full
WorkingUsersquery fromper-sighting to per-flush).
CountUsersreplaces loading every user row — anddecrypting every stored password — to compute four numbers.
statusFeedcomputesthe dashboard payload once for all viewers instead of once per open tab, and idles
when nobody is watching. Write load is now roughly constant rather than linear in
users.
The bugs batching exposed
Wrapping these paths in transactions surfaced a class of bug they had been hiding:
a claim committing separately from the thing it pays for.
and the plan last. Every retry path selects
status = 'pending', which theclaim had already cleared — so an ordinary restart in between took the money and
left nothing in the codebase able to notice. Claim and grant now commit together.
traffic it covered, so a failure meant the node's resend was rejected as a
duplicate and that batch was gone for good.
And two consequences of batching that row-by-row code used to shrug off:
could void everyone else's traffic — and wedge a node into resending the same
poison batch forever. Both inserts guard on
EXISTS.an outage longer than a day cancelled orders that had in fact been paid. It asks
first now.
Also fixed (from review)
traffic_dailyhad no retention sweep and grew forever — capped at a year, withcovering indexes for every query that reads it (all five now plan as
COVERING INDEX).on paid terms: they expired, and then nothing could rescue or renew them. Their
rows are now rewritten to the plan's new terms.
every self-registered user when their trial ended. Refused, in the UI and on the
server.
systemctl stoppaged the operator with an alert no all-clear ever followed(
KillMode=mixed+ the supervisor treats any exit while closing as intentional).an "Активен" toggle — it is never offered for sale, and the toggle was quietly
gating whether trials happened at all.
Panel fixes found by using it
check compared against
PinnedVersionwith==, and that constant carries aleading "v" while
xray versionoutput does not.VersionMatchesPinnedalreadyexisted and the Nodes tab already used it — so the two screens disagreed about
the same node while the operator kept "updating" it.
the stats page and a user's card. The data was always there (
traffic_dailycarries
node_id) but nothing read it —StatsSeriesNodesat in the store withno caller. Numbers rather than more lines: the chart already draws two, and a
line per server would be 2×N.
users.used_up/used_down, which the quota reset zeroes per user — so it added upa different period for everybody while reading as a lifetime figure. "Сеть
сервера" showed whole-host NIC throughput directly above the VPN number it never
matched.
Migrations
Everything is folded into
0031_telegram_support_broadcast.sql, except0032_drop_billing_trial_days.sql.The index DDL deliberately lives in 0032, not 0031. The migration runner keys
off the filename with no checksum, so anything appended to an already-applied file
silently never runs — a fresh install would get it and an upgraded one would not,
with no error either way. This was confirmed on the test box, whose
schema_migrationsstill holds both the pre-squash and post-squash series.Verification
go build, the fullgo test ./... -race, andgolangci-lint v2.12.2are clean;tsc --noEmitandvite buildpass.Deployed to the test box and checked live: migrations applied, both covering
indexes in use on the real database, service healthy (0 restarts, no panic), data
intact.
The riskiest changes are pinned by tests that were verified to fail against the
old code: the payment-atomicity test reproduces the historical "order paid, plan
never granted" symptom, the FK test reproduces the wedged node, and the shutdown
test reproduces the false crash alert.
Hardening (from a security pass)
loop is one goroutine answering synchronously, and every reply waits on the
outbound one-second-per-chat slot — so one chat could stall registration, menus
and payments for everyone, writing a subscriber row per message before any gate.
Both bots now share one
chatLimiter, applied ahead of that write.The comparison was already constant-time, but that only closes a timing oracle —
nothing bounded how many codes a chat could try, and a hit mints a real account.
dbHasEncryptedSecretsdid not know abouttg_support_bot_token. That guard iswhat tells "fresh install" apart from "the key is gone", so an install running
only the support bot looked fresh: boot would mint a new key and orphan the
ciphertext, silently. Every encrypted column is listed now, the nodes table
included, and the column was missing from the re-encrypt list too.