feat(codegen): add solar-codegen crate with MIR and EVM codegen#693
feat(codegen): add solar-codegen crate with MIR and EVM codegen#693gakonst wants to merge 128 commits into
Conversation
This adds the solar-codegen crate which provides: **MIR (Mid-level IR) Structure:** - Core MIR index types (ValueId, InstId, BlockId, FunctionId) - MIR types (UInt(u16), Address, MemPtr, StoragePtr) - Value (SSA values: Inst, Arg, Immediate, Phi, Undef) and Immediate constants - Comprehensive InstKind enum (Arithmetic, Bitwise, Comparison, Memory, Storage, Environment, Calls, Control Flow, SSA Phi/Select) - BasicBlock and Terminator (Jump, Branch, Switch, Return, Revert, Stop, SelfDestruct) - Function structure with SSA value/instruction storage, entry block, attributes - Module as top-level container (functions, data segments, storage layout) - FunctionBuilder API for constructing MIR **Lowering (HIR → MIR):** - Main Lowerer context (contract iteration, storage slot allocation, function setup) - Expression lowering for literals, identifiers, binary/unary ops, calls, ternary - Statement lowering for variable declarations, blocks, loops, if-statements **Code Generation (MIR → EVM):** - Opcode enum and EvmCodegen struct - generate_dispatcher: Check calldata, dispatch to function by 4-byte selector - generate_inst: Push operands, emit EVM opcode based on InstKind - Stack management: push_value for PUSH opcodes, generate_terminator for JUMP/REVERT/RETURN
Implements --standard-json input mode to enable Foundry compatibility: - Parse standard JSON input from stdin - Handle import remappings by configuring FileResolver - Correct contract-to-source file mapping in output - Write sources to temp directory with proper base_path setup Amp-Thread-ID: https://ampcode.com/threads/T-019bbe07-a540-7123-b1de-e9cd40659193 Co-authored-by: Amp <amp@ampcode.com>
- Add LoopContext stack to track break/continue targets in nested loops - Implement proper function argument loading from calldata with CALLDATALOAD - Store mutable local variables in memory (offset 0x80+) using MLOAD/MSTORE - Function parameters still handled as SSA values for efficiency Amp-Thread-ID: https://ampcode.com/threads/T-019bbe07-a540-7123-b1de-e9cd40659193 Co-authored-by: Amp <amp@ampcode.com>
- Add StackScheduler with LoadArg operation for calldata argument loading - Implement memory load/store operations for mutable local variables - Add assembler module for bytecode emission - Improve stack model tracking and manipulation Amp-Thread-ID: https://ampcode.com/threads/T-019bbe07-a540-7123-b1de-e9cd40659193 Co-authored-by: Amp <amp@ampcode.com>
- Add MemoryLoad, MemoryStore instructions for local variable storage - Add CalldataLoad instruction for function argument access - Extend MIR builder with memory allocation helpers Amp-Thread-ID: https://ampcode.com/threads/T-019bbe07-a540-7123-b1de-e9cd40659193 Co-authored-by: Amp <amp@ampcode.com>
- Implement dataflow-based liveness analysis for register allocation - Add phi node elimination with parallel copy support - Fix compiler warnings with #[allow(dead_code)] annotations Amp-Thread-ID: https://ampcode.com/threads/T-019bbe07-a540-7123-b1de-e9cd40659193 Co-authored-by: Amp <amp@ampcode.com>
- Add tempfile, alloy-json-abi dependencies for standard-json mode - Add --standard-json CLI flag to opts - Update version reporting for Foundry compatibility Amp-Thread-ID: https://ampcode.com/threads/T-019bbe07-a540-7123-b1de-e9cd40659193 Co-authored-by: Amp <amp@ampcode.com>
- Add example programs for codegen testing - Add integration tests validating EVM bytecode output
- Add function_to_dot() and module_to_dot() for graphviz output - Format instructions and terminators with operand values - Color-coded edges for branch conditions (green=true, red=false) - Add --dot flag to compile example for easy CFG generation
- Convert while-with-break to if statement in stack scheduler - Use format string variables directly - Replace extend with append for vector ranges - Replace manual div_ceil with method call - Collapse nested if statements - Update help CLI test expectations for --standard-json flag
Record shape syntax was causing parsing issues with graphviz. Simple box nodes work correctly with left-aligned labels.
- Fix uninlined_format_args in display.rs - Fix manual_div_ceil in liveness.rs - Fix collapsible_if patterns across multiple files - Fix drain_collect and extend_with_drain issues - Fix field_reassign_with_default in standard_json.rs - Add type aliases to reduce type complexity - Fix never_loop warning in spill_excess_values - Add allow attributes for test harness modules - Fix unused variable in build.rs - Add unused_crate_dependencies allows for test targets Amp-Thread-ID: https://ampcode.com/threads/T-019bbe1b-6309-775e-9653-f407558bb00b Co-authored-by: Amp <amp@ampcode.com>
- Use stderr emitter so compilation errors are visible - Fix panic on emitted_errors check - Use box shape instead of record to fix graphviz parsing
…nsfer - Add keccak256(key, slot) computation for mapping storage access - Handle compound assignments (+=, -=, etc.) by reading current value first - Implement address.transfer() and address.send() with 2300 gas stipend - Fix CALL/STATICCALL/DELEGATECALL to track result value in scheduler - Create fresh ValueIds for CALL arguments to avoid stack reuse issues Fixes MemoryLimitOOG when running Advanced.sol tests with mappings and external calls (transfer).
…ution - Add can_emit_value() to check if a value is available for emission - Add instruction_executed_untracked() for values that become stale in loops - Fix builtin member resolution to preserve Builtin reference
- Handle type conversion calls (ICallee(addr), uint256(x)) that were returning 0 - Fix compute_member_selector to resolve type cast calls for external contracts - Add nested mapping slot computation for multi-level mappings (m[a][b]) - Support dynamic array storage access and .length member - Implement pre/post increment/decrement operators - Add builtin module member access (msg.sender, block.timestamp)
- Add generate_synthetic_constructor() to create constructor for contracts without one - Constructor emits SSTORE for each state variable with an initializer - Modify generate_deployment_bytecode() to run constructor code before CODECOPY/RETURN - Add generate_constructor_code() helper that strips trailing STOP from constructor bytecode State variables like `uint256 public value = 42` are now properly initialized.
- Implement lower_emit() in stmt.rs that computes event signature hash for topic0 - Handle indexed parameters as additional topics, non-indexed as ABI-encoded data - Add compute_event_signature() and type_to_abi_string() helpers - Add log0-log4 builder methods in builder.rs emit statements were previously no-ops, now properly emit LOG0-LOG4.
- Fix typo: UEUse -> upward-exposed uses and defs - Escape doc comments with brackets to prevent broken intra-doc links - Mark README code block as text to avoid arrow parsing errors - Add #[ignore] to foundry tests that require anvil/solc
…suite - Add testdata/foundry-tests with 6 test contracts and 21 test cases - Test contracts: Counter, Events, ExternalCall, StorageInit, Showcase, StackDeep - Rewrite test harness to use FOUNDRY_SOLC=solar and forge test - Use per-test output directories (--out, --cache-path) for parallel execution - Parses forge output and validates all tests pass
- Add payable check in function dispatcher: non-payable/view/pure
functions revert if called with ETH (CALLVALUE != 0)
- Support {value: X} call options for external calls
- Add extract_call_value helper to parse call options
- Add Payable.sol test contract and Payable.t.sol tests
The payable check is emitted in emit_payable_check() after the function
JUMPDEST, before generating the function body. For payable functions,
no check is emitted.
External calls now correctly pass the value from {value: X} options
to the CALL opcode instead of hardcoding 0.
Optimization passes for the Solar codegen: 1. Constant Folding (HIR-level): - Evaluates constant expressions at compile time - Handles binary ops, unary ops, ternary expressions - 15 unit tests covering arithmetic, bitwise, comparison ops 2. Dead Code Elimination (MIR-level): - Removes instructions whose results are never used - Preserves side-effect instructions (SSTORE, CALL, LOG, etc.) - Uses value use analysis to identify dead code Also adds InstKind::has_side_effects() helper for DCE correctness. Amp-Thread-ID: https://ampcode.com/threads/T-019bbfc4-4637-71ce-a483-63911a6290f5 Co-authored-by: Amp <amp@ampcode.com>
1. Peephole Optimizer (not yet integrated): - 15+ optimization patterns (PUSH0 ADD→nop, SWAP1 SWAP1→nop, etc.) - 17 unit tests passing - Not integrated into pipeline yet (breaks jump targets) - TODO: integrate at assembler level before label resolution 2. Dynamic Arrays - fix pop() bug: - Fixed StackUnderflow in pop() caused by reusing slot_val - Reorder operations to avoid consuming values twice - Added DynamicArray.sol and DynamicArray.t.sol (6 tests pass) Amp-Thread-ID: https://ampcode.com/threads/T-019bbfec-3633-7296-8139-120e925d8fb3 Co-authored-by: Amp <amp@ampcode.com>
- Split monolithic test into 6 parallel projects by category:
arithmetic, control-flow, storage, events, calls, stack-deep
- Each project runs both Solar and solc for comparison
- Reports compilation time, bytecode sizes, and per-test gas usage
- Separate out-{compiler}/ directories for artifacts
Test results (95 tests):
- Bytecode: Solar 61-72% smaller than solc
- Gas: Solar 2-48% cheaper on most operations
- Stack-deep: Solar compiles what solc cannot
Added edge case tests exposing 5 bugs (tests skipped, tasks created):
- Signed arithmetic (SDIV, SLT, SGT)
- Ternary operator
- Continue statement
- Storage pre/post increment
- Bitwise NOT
- Test contracts (.t.sol) always compiled with solc for reliable test logic - Contract-under-test bytecode injected via SOLAR_<NAME>_BYTECODE env vars - Tests deploy Solar bytecode when env var present, fallback to solc - Updated .gitignore to exclude out/ and cache/ directories - stack-deep tests marked #[ignore] (solc can't compile, Solar has bugs)
The scheduler's StackModel was drifting from the actual EVM stack during complex codegen sequences, causing incorrect DUP operations. Changes: - Add StackEffect struct to describe pops/pushes for each opcode - Add StackPush enum to specify tracked/untracked stack entries - Add emit_stack_op() helper for DUP/SWAP/POP with automatic model updates - Add emit_op_with_effect() helper for opcodes with known stack effects - Fix InstKind::Select to use per-operation stack tracking - Update CALL/STATICCALL/DELEGATECALL to use emit_op_with_effect The root cause was Select emitting 6 stack-mutating opcodes but only updating the scheduler once at the end, causing cumulative drift.
…onment opcodes - Fix Select instruction (ternary) stack manipulation: use DUP3,DUP3 instead of DUP2,DUP4 and SWAP1 instead of SWAP2 to correctly compute f + cond*(t-f) - Add handling for environment opcodes in emit_value_fresh: CallValue, Caller, Origin, CalldataSize, Timestamp, BlockNumber - these can be safely re-emitted - Move spill slot base from 0x100 to 0x1000 to avoid conflicts with dynamic memory allocations (structs, arrays)
Tests that calling between overloaded library functions works correctly (e.g., find(key) calling find(key, true)).
Tests for functions that take multiple struct parameters, which currently have issues with memory layout.
Implement solc-style backward layout analysis for better stack scheduling: - Add StackShuffler: converts source to target layout with minimal ops - Phase 1: Ensure multiplicities (DUP values needing copies) - Phase 2: Arrange positions (SWAP to correct slots) - Phase 3: Pop excess values - Add LayoutAnalysis: backward analysis through instructions - analyze_backward() computes ideal entry layout from exit - compute_entry_for_instruction() for single instruction - Add helper functions for ideal operand layouts - ideal_binary_op_entry/ideal_unary_op_entry - is_freely_generable() for literals/labels - Integration with StackScheduler - shuffle_to_layout(), prepare_binary_op(), prepare_unary_op() All 8 shuffler unit tests pass.
Add CFG simplification passes: 1. Block Merging (CfgSimplifier): - Merge A→B when A has single successor B and B has single predecessor A - Saves 8 gas per eliminated JUMP 2. Empty Block Elimination: - Remove blocks with only unconditional jump - Redirect predecessors to final target 3. Dead Function Elimination (DeadFunctionEliminator): - Build reachability from entry points (public/external/constructor) - Remove unreachable functions 4. Call Graph Analysis (CallGraphAnalyzer): - Build call graph for module - Detect recursive functions via DFS Includes CfgSimplifyStats for tracking optimization metrics. All 8 unit tests pass.
Add infrastructure for stack layout merging at control flow merge points: - BlockStackLayout: represents stack layouts with SmallVec slots - combine_stack_layouts(): computes common layout from predecessors - estimate_shuffle_cost(): estimates DUP/SWAP/POP operations needed StackScheduler block layout methods: - set/get_block_entry_layout: manage target layouts per block - record/get_block_exit_layout: track actual layouts - compute_merge_layout: compute entry from predecessor exits - shuffle_to_block_entry: shuffle current stack to target - init_from_block_entry_layout: initialize stack from expected 12 new unit tests for layout merging functionality. All 117 tests pass.
Partial implementation of nested struct handling: - calculate_memory_words_for_type: compute flattened memory layout - get_struct_field_memory_offset: get byte offset for nested fields - compute_nested_memory_struct_info: handle chained member access - copy_struct_storage_to_memory/memory_to_storage: recursive copy helpers - Fix variable declaration to allocate memory for uninitialized structs testNestedMemoryValue passes - nested memory struct access works. Cross-location copying needs further work.
Implement storage-to-memory and memory-to-storage copying for nested structs: - copy_storage_to_memory(): recursive copy from storage to memory - copy_memory_to_storage(): recursive copy from memory to storage - compute_nested_storage_slot_with_type(): recursive storage slot calculation - compute_nested_memory_struct_info_with_type(): recursive memory offset calc Supports arbitrarily deep nesting (3+ levels tested). Test files: - DeepNested.sol/t.sol: 3-level nested struct storage-memory round trip - DeepNestedSimple.sol/t.sol: 1/2/3-level storage access All 117 unit tests and 33 struct foundry tests pass.
These instruction kinds can be re-emitted when their results are needed as CALL operands but aren't on the stack. This fixes ICE when compiling contracts that use keccak256 results in external calls (e.g., unifap-v2). 6/8 unifap-v2 tests now pass. Remaining 2 failures are runtime errors in setUp() - needs further investigation.
- Fix ExprKind::Ternary lowering to use proper branching instead of select() which only works for single values. Tuple ternaries now write to scratch memory and merge block reads values back. - Fix abi.encodePacked to properly pack values based on their types instead of 32-byte padding. Returns proper bytes memory format (length prefix + tightly packed data). - Add get_packed_size_from_expr/get_packed_size_from_hir_type for type-based size inference (address=20, bool=1, bytesN=N, etc.) - Update lower_return to handle ternary expressions returning tuples by reading all values from scratch memory. All 34 unifap-v2 tests now pass. Amp-Thread-ID: https://ampcode.com/threads/T-019bf223-95e1-7439-87e4-4ea6fff69ab2 Co-authored-by: Amp <amp@ampcode.com>
…structure (#749) Improve current the codegen (MIR) branch by adding: pass infrastructure, a textual MIR format (printer + parser), the `solar-mir-opt` tool, and an `SCCP` implementation. Builds on top of the existing codegen work in #693 (`feature/codegen-mir`). By adding a way to serialise and deserialise (and to verify) the MIR module, it will make compiler more modular and easier to test individual analysis and passes. In addition to it, by having a Pass Manager, it will be much easier to use some analysis information between individual passes. **Pass manager** `AnalysisPass` and `TransformPass` traits with `AnalysisManager` for caching results by type. All transforms in `evm.rs` (`jump-threading`, `cfg-simplify`, `dce`) now go through the `PassManager` instead of being called directly. **MIR text format** Printer (`--emit=mir`) and hand-written recursive-descent parser covering all instruction kinds. Parser errors show source-line snippets with a caret. Round-trip property tests verify accuracy on every fixture. **solar-mir-opt** Standalone tool for running individual passes. Accepts `.sol` and `.mir` input, supports `--passes a,b,c`, `--print-after-each`, and `--pipeline-default`. **New passes** - `SCCP` (Sparse Conditional Constant Propagation) — Propagates constants through the CFG, folds branches with known conditions, marks unreachable blocks. Resolves #702. - `MIR` validator — checks SSA invariants (`defined-before-use`, pred/succ consistency, phi coverage, etc.) **Liveness / DCE minor fixes** - Precomputed InstId -> ValueId maps replacing O(n) scans in both liveness and DCE - Phi-aware live_out equation for `Value::Phi` nodes - Fixed `sema/emit.rs` panicking on `--emit=bin` via todo!() **Tests** - `.mir` LIT tests under `tests/ui/codegen/mir/` (DCE, CSE, SCCP, jump threading, cfg simplify, phi, switch, pipeline) - `.sol` UI tests under `tests/ui/codegen/` (arithmetic, storage, branches, loops, mappings, ternary, function calls, recursion, etc.) - Round-trip integration tests + validator integration tests on all fixtures - `Mode::Mir` added to the test runner so `cargo uitest` covers both `.sol` and `.mir`
|
Hi @gakonst. I am more than happy to help with bringing this to the level we can merge it with Can we please trigger the CI here? |
Ping :) |
# Conflicts: # Cargo.lock # crates/config/build.rs # crates/sema/src/ty/ty.rs # crates/sema/src/typeck/checker.rs # crates/sema/src/typeck/mod.rs # tests/ui/typeck/function_calls/event_error_context.sol # tests/ui/typeck/function_calls/event_error_context.stderr # tests/ui/typeck/function_implementation_checks.sol # tests/ui/typeck/function_implementation_checks.stderr
|
Thanks @DaniPopes :) I will be working on fixing the CI. |
This is the first part: it had some stale github module references, so it couldn't run the tests.
|
The next important step is to introduce infra for codegen and runtime comparison with |
This is kind of done: #760. I identified some |
Adds `codegen` CI using [solidity-compiler-benchmarks](https://github.com/walnuthq/solidity-compiler-benchmarks) The workflow builds `solar`, compiles the benchmark corpus with both `solc` and `solar`, then checks: - bytecode size - gas usage on Anvil - runtime output equality for the same calls/inputs While trying to build the contracts from the set of contracts I shared, some `codegen` issues occurred. In this PR, I fixed a `codegen` bug: cross-block MIR values were not always spilled before leaving a block, so successor blocks could lose values after the stack model was cleared. The fix improves liveness propagation and preallocates/spills live cross-block values in EVM `codegen`. Besides that, there is `fix(sema): canonicalize bare integer types`, and it should be a separate PR; it fixes `uint/uint256` alias handling, detected in `maple-erc20`.
Continue with fixing `codegen` to avoid mismatches in runtime comparing to `solc`.
Summary
This adds the
solar-codegencrate which provides MIR (Mid-level Intermediate Representation) and EVM bytecode generation for Solar.Test Results
95 tests pass across 6 parallel test suites comparing Solar vs solc:
Per-Project Results
Sample Output
Architecture
MIR Structure
ValueId,InstId,BlockId,FunctionIdindex typesUInt(u16),Int(u16),Address,MemPtr,StoragePtr,CalldataPtr,Function,Bool,FixedBytesInst,Arg,Immediate,Phi,Undef) withImmediateconstantsJump,Branch,Switch,Return,Revert,Stop,SelfDestruct,InvalidLowering (HIR → MIR)
Lowerercontext with storage slot allocation and function loweringCode Generation (MIR → EVM)
EvmCodegenstruct for bytecode generationCompleted Features
{value: X})Optimization Passes (implemented, not yet integrated)
transform/constant_fold.rs)transform/dce.rs)codegen/peephole.rs) - 15+ bytecode patternsKnown Issues (tests skipped, tracked in task list)
Remaining Work