Commit 7a46fc1
Rewrite AnalysisScheduler with trace-based scheduling algorithm for HHBBC
Summary:
The previous AnalysisScheduler required N analysis rounds to propagate
information through a dependency chain of length N. This rewrite introduces a
trace-based scheduling algorithm that processes entire dependency chains
together, allowing information to propagate through the entire chain in a
single round, dramatically reducing total analysis time.
TRACE-BASED SCHEDULING (index.cpp:23683-25388, index.h:2389-2707):
The new scheduler builds "traces" - sets of transitive dependencies that should
be analyzed together. If A depends on B depends on C depends on D, then A's
trace includes [A,B,C,D]. All entities in the trace are processed in the same
job, collapsing what would be 4 sequential rounds into 1.
The scheduler operates by:
- Determining entity eligibility (which entities might change this round)
- Building traces by following dependency chains up to configurable size/depth
- Packing traces into size-bounded buckets using greedy bin-packing with
overlap optimization (multiple traces can share bundles)
- Classifying bundles as reportBundles (analyze and report), noReportBundles
(analyze speculatively but don't report), or pureDepBundles (information only)
Entities are tracked at fine granularity but scheduled at bundle granularity
since entities are stored in bundles for distributed execution.
DATA STRUCTURE CHANGES (index.h:2236-2387):
Restructured AnalysisInput and AnalysisOutput to work with bundle-based
scheduling. Added comprehensive metadata for dependency tracking, eligibility
determination, and bundle classification. Inputs now carry reportBundles and
noReportBundles with metadata describing which specific entities to process
and their dependencies.
INTEGRATION CHANGES (whole-program.cpp:432-935):
Adapted whole-program analysis to work with the new scheduler:
- Unified AnalyzeConstantsJob and regular analysis into templated
AnalyzeJob<Mode> that works with bundles
- Converted from per-entity job inputs to bundle-based inputs
- Added final_pass() function (whole-program.cpp:837-897) to run the final
analysis pass with no traces (immediate dependencies only)
- Added emit_units() function (whole-program.cpp:900-935) to emit optimized
unit emitters from final pass results
This completes the main work of distributed hhbbc!
The full functionality is gated by a compile-time only option. The
default is the old behavior of doing analyze constants distributed,
then doing the rest locally. The reason it is not on by default is
that AnalysisIndex still needs to implement a few optimizations to be
at parity with the old way. This avoids shipping regressions and will
be short-lived.
Reviewed By: mdko
Differential Revision: D89634732
fbshipit-source-id: 6eceb97fcda760605422dda3c4a5ea95084258721 parent c673925 commit 7a46fc1
13 files changed
Lines changed: 4991 additions & 3958 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
317 | 317 | | |
318 | 318 | | |
319 | 319 | | |
320 | | - | |
321 | | - | |
322 | | - | |
323 | | - | |
324 | | - | |
325 | | - | |
326 | | - | |
327 | | - | |
328 | | - | |
329 | | - | |
330 | | - | |
331 | | - | |
332 | 320 | | |
333 | 321 | | |
334 | 322 | | |
| |||
349 | 337 | | |
350 | 338 | | |
351 | 339 | | |
| 340 | + | |
| 341 | + | |
| 342 | + | |
| 343 | + | |
| 344 | + | |
| 345 | + | |
| 346 | + | |
| 347 | + | |
| 348 | + | |
| 349 | + | |
| 350 | + | |
| 351 | + | |
| 352 | + | |
| 353 | + | |
352 | 354 | | |
353 | 355 | | |
354 | 356 | | |
| |||
0 commit comments