Skip to content

Track additional statistics#6390

Draft
malsadev wants to merge 16 commits intoLemmyNet:mainfrom
malsadev:additional-statistics-scheduled-task
Draft

Track additional statistics#6390
malsadev wants to merge 16 commits intoLemmyNet:mainfrom
malsadev:additional-statistics-scheduled-task

Conversation

@malsadev
Copy link
Copy Markdown
Contributor

@malsadev malsadev commented Mar 8, 2026

Issue: #6288

Summary of changes:

In local_site table:

  1. Renamed columns posts, users, (and others) to local_posts, local_users, etc.
  2. Added columns total_posts, total_users, etc
  3. Added language_usage_percent jsonb column with a non null constraint and default value {}

There were tests that were exercising some trigger logic in the db so I fixed the db triggers/functions here.

  1. Added update_stats method which runs some basics queries and updates local_site table and added it to the daily scheduled tasks
  2. Added process_language_breakdown (runs as part of update_stats) which calculates the percentage breakdown of posts per language tag and updates the newly added language_usage_percent field in local_site table
  3. Added test_update_stats and test_process_language_breakdown tests
  4. Added --features full flag to ./scripts/dump_schema.rs

Outstanding items:

  • Comment percentage calculation (is this needed?)
  • Ban rate tracking
  • Accepted/failed signups tracking
  • Are extra filters needed for the counts? For example, should banned/deleted users be included?
  • Manual testing
  • pipeline fail

Comment thread migrations/2026-03-08-173221-0000_additional-statistics/up.sql Outdated
Comment thread migrations/2026-03-08-173221-0000_additional-statistics/up.sql Outdated
ADD COLUMN total_communities integer;

ALTER TABLE local_site
ADD COLUMN user_retention_percent integer;
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All the percentages/rates should probably be floats.

Comment on lines +20 to +21
ALTER TABLE local_site
ADD COLUMN local_post_english_percent integer;
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure how useful this metric would be. I spose it could be useful for servers who are multi-lingual, and wanting to try to get rid of english usage.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah it shouldnt be limited to a single language, but have usage percentage for every language. Maybe with a new column language.usage_percentage.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added language_usage_percent jsonb column. It will contain something like:

{
"en" : "30.10",
"und": "10.00",
"de": "59.90"
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not a fan of slamming json into sql, we really should try to avoid that. Either add a new table, or better yet just remove this.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Best put it in the existing language table.

malsadev and others added 10 commits March 10, 2026 15:32
- Rename `users/posts/comments/communities` to `local_users/local_posts/local_comments/local_communities` to distinguish from the new total_* columns
- Add `NOT NULL DEFAULT 0` to all new statistics columns (linked_instances, total_*, rates) instead of nullable
- Update triggers.sql, nodeinfo.rs, convert.rs, and tests to use new column names
@malsadev malsadev changed the title Additional statistics scheduled task Track additional statistics Mar 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants