-
Notifications
You must be signed in to change notification settings - Fork 4
Overview of cleanup - suggested indexes and export tool #1975
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
RagnarFatland
wants to merge
25
commits into
main
Choose a base branch
from
chore/cleanupMigration2026
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Changes from all commits
Commits
Show all changes
25 commits
Select commit
Hold shift + click to select a range
7d8dd5c
Generated documentation, scipt and console app to export data neccesa…
35b2ce1
Adjusted A2Parties table to handle SelfIdentified users having partyu…
794beec
Various corrections after initial coderabbit review.
424b440
Cleanup of various issue; including fixing proper cursor for export.
1905fd3
minor tweaks, added test script for export query so I can manually te…
b24755c
fixes some sql syntas issues, added manual test scripts
e6f58ba
handle edge cases for index creation
ad7cb68
Misc optimizations after creating indexes and testing queries manually.
8e608aa
various attempts to tweak and test
ed447d1
fix comments from coderabbit
2c43cfe
more fix
b8b2c92
more testing
a4997e6
Updates after massive testing of export for issue 1716, should be pro…
6ccdfa7
updates after comments from Coderabbits
b0ea142
more fixes
9e3e14f
Fix broken syntax of export script.
c81321b
changed cursor usage to per-status
52b434d
Additional troubleshooting and optimization.
169d26a
Improved status logging.
31ee9fc
Tweaks after feedback from Coderabbit.
7034317
Updated for optimized Issue 1951.
7cfb8ca
cleanup documentation
4c9f7e4
tweaked export with abillity to have a delay between each export to a…
3f0ea46
Major documentation cleanup.
0ebf622
minor fixes to documentation and cleanup-scripts.
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,91 @@ | ||
| -- ===================================================================================== | ||
| -- Check Disk Space and Table Statistics | ||
| -- ===================================================================================== | ||
| -- For use BEFORE creating large indexes on 1.94 billion row table | ||
| -- ===================================================================================== | ||
|
|
||
| -- 1. Check available disk space | ||
| SELECT | ||
| pg_tablespace_name(oid) AS tablespace, | ||
| pg_size_pretty(pg_tablespace_size(oid)) AS size | ||
| FROM pg_tablespace; | ||
|
|
||
| -- 2. Current CorrespondenceStatuses table size | ||
| SELECT | ||
| schemaname, | ||
| tablename, | ||
| pg_size_pretty(pg_total_relation_size(relid)) AS total_size, | ||
| pg_size_pretty(pg_relation_size(relid)) AS table_size, | ||
| pg_size_pretty(pg_indexes_size(relid)) AS indexes_size, | ||
| n_live_tup AS live_rows, | ||
| n_dead_tup AS dead_rows, | ||
| ROUND(100.0 * n_dead_tup / NULLIF(n_live_tup + n_dead_tup, 0), 2) AS dead_row_pct | ||
| FROM pg_stat_user_tables | ||
| WHERE schemaname = 'correspondence' | ||
| AND tablename = 'CorrespondenceStatuses'; | ||
|
|
||
| -- 3. Existing indexes on CorrespondenceStatuses | ||
| SELECT | ||
| schemaname, | ||
| tablename, | ||
| indexname, | ||
| pg_size_pretty(pg_relation_size(indexrelid)) AS index_size, | ||
| idx_scan AS times_used, | ||
| idx_tup_read AS tuples_read, | ||
| idx_tup_fetch AS tuples_fetched | ||
| FROM pg_stat_user_indexes | ||
| WHERE schemaname = 'correspondence' | ||
| AND tablename = 'CorrespondenceStatuses' | ||
| ORDER BY pg_relation_size(indexrelid) DESC; | ||
|
|
||
| -- 4. Estimate new index sizes (rough calculation) | ||
| -- Based on 1.94B total rows: | ||
| -- - Issue #1716 (SyncedFromAltinn2 IS NOT NULL): ~1% = ~19M rows → ~3 GB | ||
| -- - Issue #1951 (SyncedFromAltinn2 IS NULL): ~15% = ~300M rows → ~24 GB | ||
| SELECT | ||
| 'Estimated disk space needed for new indexes:' AS info, | ||
| '3 GB (Issue #1716) + 24 GB (Issue #1951) = ~27 GB' AS estimate, | ||
| 'Plus 50% temp space during creation = ~40 GB total' AS with_temp; | ||
|
|
||
| -- 5. Check if table needs VACUUM | ||
| SELECT | ||
| schemaname, | ||
| tablename, | ||
| last_vacuum, | ||
| last_autovacuum, | ||
| n_dead_tup, | ||
| n_mod_since_analyze, | ||
| CASE | ||
| WHEN n_dead_tup > 1000000 THEN 'VACUUM RECOMMENDED' | ||
| ELSE 'OK' | ||
| END AS vacuum_recommendation | ||
| FROM pg_stat_user_tables | ||
| WHERE schemaname = 'correspondence' | ||
| AND tablename = 'CorrespondenceStatuses'; | ||
|
|
||
| -- 6. Current database size | ||
| SELECT | ||
| pg_database.datname, | ||
| pg_size_pretty(pg_database_size(pg_database.datname)) AS size | ||
| FROM pg_database | ||
| WHERE datname = current_database(); | ||
|
|
||
| -- ===================================================================================== | ||
| -- Recommendations based on 1.94 billion rows: | ||
| -- ===================================================================================== | ||
| -- | ||
| -- DISK SPACE: | ||
| -- - Ensure at least 50 GB free space (27 GB indexes + 20 GB temp + buffer) | ||
| -- - Monitor disk space during creation with: df -h (Linux) or Get-PSDrive (PowerShell) | ||
| -- | ||
| -- TIMING: | ||
| -- - Index #1 (Issue #1716): 1-2 hours (currently running) | ||
| -- - Index #2 (Issue #1951): 6-10 hours (run overnight or maintenance window) | ||
| -- | ||
| -- PERFORMANCE: | ||
| -- - Set maintenance_work_mem = '4GB' before creating Index #2 | ||
| -- - Run VACUUM ANALYZE if dead_row_pct > 5% | ||
| -- - Schedule during lowest traffic period | ||
| -- - Consider pausing ETL/batch jobs during creation | ||
| -- | ||
| -- ===================================================================================== | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.