Skip to content

Conversation

@Erethon
Copy link
Collaborator

@Erethon Erethon commented Sep 17, 2025

After running some tests, these settings seem to perform about 100-200x times faster than the previous values, at the expense of more memory.

Original values we're getting ~1 attribute every 3 seconds with some exceptions:

Sep 17 20:58:16 nixos web-security-tracker-worker-start[20234]: 2025-09-17 20:58:16,367 INFO 1 attributes were ingested in 0.064309 seconds                                                    
Sep 17 20:58:20 nixos web-security-tracker-worker-start[20234]: 2025-09-17 20:58:20,107 INFO 1 attributes were ingested in 0.086749 seconds                                                    
Sep 17 20:58:21 nixos web-security-tracker-worker-start[20234]: 2025-09-17 20:58:21,303 INFO 1 attributes were ingested in 0.064423 seconds                                                    
Sep 17 20:58:21 nixos web-security-tracker-worker-start[20234]: 2025-09-17 20:58:21,704 INFO 3 attributes were ingested in 0.062850 seconds                                                    
Sep 17 20:58:24 nixos web-security-tracker-worker-start[20234]: 2025-09-17 20:58:24,760 INFO 1 attributes were ingested in 0.053967 seconds                                                    
Sep 17 20:58:28 nixos web-security-tracker-worker-start[20234]: 2025-09-17 20:58:28,668 INFO 1 attributes were ingested in 0.086924 seconds                                                    
Sep 17 20:58:29 nixos web-security-tracker-worker-start[20234]: 2025-09-17 20:58:29,457 INFO 1 attributes were ingested in 0.073036 seconds                                                    
Sep 17 20:58:29 nixos web-security-tracker-worker-start[20234]: 2025-09-17 20:58:29,877 INFO 1 attributes were ingested in 0.085220 seconds

With these new values:

Sep 17 21:09:45 nixos web-security-tracker-worker-start[73821]: 2025-09-17 21:09:45,977 INFO 6052 attributes were ingested in 51.348887 seconds
Sep 17 21:10:58 nixos web-security-tracker-worker-start[73821]: 2025-09-17 21:10:58,225 INFO 3290 attributes were ingested in 25.561116 seconds
Sep 17 21:11:18 nixos web-security-tracker-worker-start[73821]: 2025-09-17 21:11:18,507 INFO 211 attributes were ingested in 1.572250 seconds
Sep 17 21:11:22 nixos web-security-tracker-worker-start[73821]: 2025-09-17 21:11:22,130 INFO 1 attributes were ingested in 0.047971 seconds
Sep 17 21:11:29 nixos web-security-tracker-worker-start[73821]: 2025-09-17 21:11:29,400 INFO 152 attributes were ingested in 1.143473 seconds
Sep 17 21:13:15 nixos web-security-tracker-worker-start[73821]: 2025-09-17 21:13:15,129 INFO 2571 attributes were ingested in 16.635387 seconds
Sep 17 21:15:35 nixos web-security-tracker-worker-start[73821]: 2025-09-17 21:15:35,448 INFO 5500 attributes were ingested in 25.364185 seconds
Sep 17 21:16:20 nixos web-security-tracker-worker-start[73821]: 2025-09-17 21:16:20,744 INFO 1527 attributes were ingested in 9.952927 seconds
Sep 17 21:16:25 nixos web-security-tracker-worker-start[73821]: 2025-09-17 21:16:25,913 INFO 3 attributes were ingested in 0.063238 seconds
Sep 17 21:16:36 nixos web-security-tracker-worker-start[73821]: 2025-09-17 21:16:36,906 INFO 45 attributes were ingested in 0.423121 seconds

@Erethon Erethon marked this pull request as draft September 17, 2025 23:15
@Erethon Erethon marked this pull request as ready for review October 29, 2025 12:24
After running some tests, these settings seem to perform about 100-200x
times faster than the previous values, at the expense of more memory.
Copy link
Member

@balsoft balsoft left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would be nice to make it configurable (e.g. from the NixOS module), so that we don't have to muck with the codebase when Nixpkgs gets bigger still

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants