infoschema v2: optimize partition table memory usage

[tiancaiamao]

Problem Description

Partition tables in InfoSchema V2 are currently classified as "special attributes" and their full PartitionInfo is stored in tableInfoResident, bypassing the Sieve cache. This violates the V2 design principle of lazy loading and causes significant memory bloat for large-scale partitioned table clusters.

Current State

For a cluster with:

• 1,000 partitioned tables
• Average 100 partitions per table

Estimated memory waste: ~100MB of permanent memory for PartitionInfo (including partition definitions, expressions, etc.)

Root Cause

The original design assumed:

1. Background tasks need to iterate over tables with special attributes
2. Different background tasks iterate over different attribute types
3. Without resident storage, repeated iterations would be expensive

However, this assumption is incorrect for partition tables:

• Partition tables have the lowest access frequency among all special attributes
• No periodic background tasks iterate over partition tables
• Access patterns are primarily: startup, user-initiated queries, and DDL events

Comparison with Other Special Attributes

Special Attribute	Access Frequency	Background Tasks	Iteration Frequency
TTL	⚡⚡⚡⚡ Very High	ttlworker (every 5s or schema change)	Every 5s
TiFlash	⚡⚡⚡ High	Async replica progress monitoring	Continuous
Placement	⚡⚡ Medium	Policy validation, placement tracking	On demand
Partition	⚡⚡ Low	No periodic tasks	Startup / Query / DDL only
TableLock	⚡ Low	Lock timeout cleanup	On demand

Expected Benefits

• Memory: Tens to hundreds of MB saved for large-scale partitioned table clusters
• Startup: ~95% reduction in LIST partition scan scope
• Query Performance: No regression (partition pruning remains O(1))

Proposed Solution

1. Remove PartitionAttribute from HasSpecialAttributes() filter
2. Add version-aware GetPartitionedTableIDsV2() API (extract from pid2tid btree)
3. Update all callers with V2 API + V1 fallback
4. Optimize LIST partition startup rebuild (scan only LIST partitions, not all partitioned tables)

See design document for detailed analysis: explorer/partition-table-optimization-design.md