Add statement-level IR with codegen, interpreter, and test suite

.gitignore

@@ -577,3 +577,10 @@ builds/
 install/
 
 *.db
+*.db-shm
+*.db-wal
+mx-workspace/
+.claude/
+.mcp.json
+*_PROMPT.md
+tests/InterpretIR/compile_commands.json

IR_GAPS.md

@@ -0,0 +1,74 @@
+# IR Gaps and Issues
+
+## DONE
+- **MEM opcode** — Unified MEMORY opcode with 70 sub-opcodes (load/store x atomic x LE/BE x 8/16/32/64, bulk memory, string ops, bit-field access, cmpxchg, CONSUME_VA_PARAM).
+- **LAST_VALUE** — Comma operator: evaluates all operands, returns last.
+- **Pointer decrement fix** — `--ptr` correctly emits CONST(-1) with PTR_ADD.
+- **IRObject methods** — source_declaration() and type() implemented.
+- **EnterScopeInst/ExitScopeInst** — Instruction classes with scope() accessor.
+- **Convenience methods** — IRBlock::parent_function(), IRFunction::containing(IRBlock/IRInstruction).
+- **DesignatedInitExpr** — Unwrap to Initializer(). ImplicitValueInitExpr as zero.
+- **Float builtins** — 41 FloatOp sub-opcodes.
+- **Bit-field init fix** — Uses BIT_WRITE with exact bit_offset/bit_width.
+- **Interpreter improvements** — Scope tracking (poison on EXIT_SCOPE), all string/memory ops.
+- **Pointer compound assign fix** — `ptr += n` uses PTR_ADD, not ADD.
+- **PTR_DIFF element_size** — Extracted from pointee type.
+- **_Atomic plain load/store** — Correctly uses atomic ops.
+- **BIT_READ** — Emitted in EmitLoadFromLValue for bit-field reads.
+- **Switch case compensation** — Compensation blocks for switch→case scope crossings.
+- **CFG predecessor fix** — Correct predecessor list updates in compensation.
+- **Database teardown** — Isolated try/catch per step, leaked statement detection.
+- **Sized opcodes** — All integer, pointer, atomic, overflow, and bitwise opcodes are width-specific (_8/_16/_32/_64). Float opcodes have _32/_64 variants. FloatOp and BitwiseOp sub-opcodes carry width. No unsized arithmetic/pointer opcodes remain in the enum. Interpreter is width-correct at all sizes. OpCode is uint8_t (251 values, gap-packed).
+- **ABS opcode** — Moved from BitwiseOp sub-opcode to sized top-level opcode (ABS_8/16/32/64).
+- **EXPECT/ASSUME removed** — Compiler hints with no runtime semantics; not emitted to IR.
+- **Interpreter precision** — Float _32 ops use float precision, _64 use double. All casts (SEXT, ZEXT, TRUNC, int↔float, float↔float, BITCAST) are width-correct. Unsigned operations use correct unsigned types at each width.
+- **Entity ID type safety** — Typed enums in IRBlockId, IRInstructionId, IRStructureId.
+- **IRSwitchCase → IRSwitchCaseStructure** — Switch cases are now IRStructure entities.
+- **ALLOCA sub-opcodes** — AllocaKind: LOCAL, ARG, RETURN, DYNAMIC. DynamicAllocaInst derived from AllocaInst.
+- **EXPRESSION_SCOPE** — New StructureKind for call argument/return allocas.
+- **PARAM_PTR** — Renamed from PARAM_READ. Returns pointer to caller's argument alloca.
+- **RETURN_ADDRESS** — Renamed from RETURN_PTR (for __builtin_return_address).
+- **VA_PACK removed** — Variadic args are regular operands.
+- **VA_ARG → CONSUME_VA_PARAM** — New MemOp sub-opcode.
+- **ArrayToPointerDecay fix** — Uses EmitLValue (address), not EmitRValue (load).
+- **MEMCPY for direct assignment** — `a = b` with lvalue RHS always uses MEMCPY.
+- **Scalar size guards** — IsScalarSize() check before DetermineMemOp; MEMCPY fallback for non-1/2/4/8.
+- **String literal init** — Non-power-of-2 sizes use MEMCPY.
+- **source_statement() assertion fix** — Checks entity ID is StmtId before calling StmtFor.
+- **IRObject::source_declaration() assertion fix** — Checks entity ID is DeclId before calling DeclFor.
+- **Interpreter moved to bin/InterpretIR/** — Separate from Examples.
+- **mx-print-ir human-readable output** — Uses EnumeratorName() for all opcodes, sub-opcodes, kinds.
+- **MX_EXPORT on IR enum EnumeratorName** — All IR enum name functions exported from shared library.
+- **GNU statement expression** — Already handled: SCOPE + emit children + last expr value.
+- **Compound literal** — Already handled: ALLOCA + EmitInitializer + scope-tracked.
+- **EXPRESSION_SCOPE at call sites** — Calls wrapped in EXPRESSION_SCOPE with ALLOCA/ARG for each argument and ALLOCA/RETURN for return value. Scope popped at full-expression boundary.
+- **RETURN_PTR in callee** — Callee emits RETURN_PTR to get pointer to caller's return storage, stores return value into it before RET.
+- **PARAM_PTR without local copy** — Parameters no longer copied into local allocas. PARAM_PTR directly gives pointer to caller's ARG alloca. DeclRefExpr resolves to PARAM_PTR.
+- **CallInst::return_alloca()** — Returns the ALLOCA/RETURN instruction for the return value. `has_return_value()` for void check.
+- **IRObject string literal bytes** — Not needed; content accessible via AST StringLiteral through source_declaration().
+
+## Remaining Gaps
+
+### Codegen
+1. **C++ expressions** — Lambda, new/delete, this, constructors, destructors, etc. emit UNKNOWN. (C-only for now.)
+
+### API
+6. **No IRInstruction::result_type() on base class** — Must downcast to get result type. A base-class method would simplify interpreters/printers. Deferred.
+7. **No Index::ir_functions() enumerator** — Can't iterate all IR functions from an Index.
+
+### Interpreter
+10. **Interpreter is monolithic** — ~1500-line switch in bin/InterpretIR. Plan exists to extract into lib/IR/Interpret/ with ValueFactory/Memory/Driver/Checker policy classes. See docs/InterpreterLibraryPlan.md.
+11. **No multi-path exploration** — Single-path concrete execution only. Need COW memory + fork.
+12. **No call inlining** — Interpreter doesn't step into callees.
+13. **No external function modeling** — malloc/free/memcpy/printf etc. not modeled.
+
+### Codegen / Types
+14. **`__int128` / `_BitInt(N>64)`** — Sized integer opcodes only cover 8/16/32/64-bit widths. Wider types (e.g., `__int128`, `_BitInt(128)`) currently round down to `_64`. These could be decomposed into paired 64-bit operations.
+
+### C23
+15. **`#embed` directive** — C23's `#embed` for embedding binary data. Not handled.
+
+### Documentation
+15. **IR_GAPS.md** — This file (kept up to date).
+16. **docs/IR.md** — Updated.
+17. **docs/InterpreterLibraryPlan.md** — Written.

IR_TODOS.md

@@ -0,0 +1,57 @@
+# IR Implementation Progress
+
+## Plan: Structural IR Entities (robust-plotting-rocket.md)
+
+### Phase 1: FunctionKind + sourceDeclEntityId rename — COMPLETE
+
+### Phase 2: IRStructure entity + scope tracking — COMPLETE
+- [x] StructureKind enum (18 kinds), IRStructure entity, IRStructureId
+- [x] Structure capnp schema, entity providers, pack/unpack
+- [x] Structure generation (PushStructure/PopStructure) for all control flow
+- [x] FUNCTION_SCOPE, nested SCOPE for CompoundStmt, implicit SCOPE for for-init
+- [x] ENTER_SCOPE/EXIT_SCOPE opcodes on all paths (including break/continue/return/goto)
+- [x] Block parentStructureId, scope object association via AssociateObjectWithScope
+- [x] IRFunction::body_scope(), IRBlock::parent_structure() accessors
+- [x] SWITCH_CASE structures for each case/default
+- [ ] IRSwitchCase → IRStructure(SWITCH_CASE) full migration (deferred)
+- [ ] Full Python bindings (stub exists, requires bootstrap regen)
+
+### Phase 3: Control flow region structures — MERGED INTO PHASE 2
+
+### Phase 4: Global initializer functions — COMPLETE
+- [x] GenerateGlobalInit creates synthetic FunctionIR for globals with initializers
+- [x] FunctionKind::GLOBAL_INITIALIZER, sourceDeclEntityId = VarDecl
+- [x] Entry block with ADDRESS_OF → EmitRValue(init) → STORE → RET
+- [x] VarDecl maps to its GLOBAL_INITIALIZER IRFunction
+
+## Known Extensions — PARTIALLY COMPLETE
+- [x] MEMSET (opcode 68): dest, byte_value, size — lowered from memset/builtin calls
+- [x] MEMCPY (opcode 69): dest, src, size — lowered from memcpy/memmove/builtin calls
+- [x] MemsetInst, MemcpyInst instruction class wrappers
+- [ ] VAR_INIT block kind + structure kind (for variable initialization regions)
+
+## Known Issues / Lies Remaining
+1. IRSwitchCase still separate entity type (coexists with SWITCH_CASE structures)
+2. string_bytes() missing on IRObject for string literals
+3. Python bindings are stub only for IRStructure
+4. Global initializer quality depends on EmitRValue handling of all init expressions
+5. **Goto/Duff's device scope compensation**: `goto` that jumps into the middle
+   of a scope bypasses the normal ENTER_SCOPE path. An interpreter following
+   the goto would not see the scope entry for variables in that scope. Similarly,
+   Duff's device-style switch cases can interleave with loop bodies, creating
+   scope entry paths that don't go through ENTER_SCOPE. We need "compensation
+   blocks" — synthetic blocks inserted on goto/case edges that emit the
+   ENTER_SCOPE instructions for any scopes being entered. This is tricky because
+   the label/case might also be reachable from normal control flow (which already
+   has the ENTER_SCOPE), so we can't just add ENTER_SCOPE at the label — we need
+   it on the specific edge. Always using compensation blocks (even for the normal
+   case) would be the simplest correct solution.
+
+## Decisions Made
+- Phase 3 merged into Phase 2
+- SWITCH_CASE structures coexist with IRSwitchCase for backward compat
+- ENTER_SCOPE/EXIT_SCOPE carry IRStructureId extra in entity pool
+- StructureKind embedded in IRStructureId (18 sub_kind offsets)
+- Global initializer uses same EmitRValue path as function body codegen
+- MEMSET/MEMCPY lowered from memset/memcpy/memmove and all __builtin_ variants
+- Goto conservatively exits all scopes to FUNCTION_SCOPE

bin/Bootstrap/PASTA.cpp

@@ -2388,6 +2388,53 @@ MethodListPtr CodeGenerator::RunOnClass(
       make_parent("parent_statement", "MX_VISIT_STMT_LINK", "Stmt");
     }
 
+    // `*::ir()` -- the IR entity corresponding to this AST entity.
+    // For FunctionDecl, returns IRFunction. For Stmt/Expr, returns IRInstruction.
+    // For CaseStmt/DefaultStmt, returns IRStructure (SWITCH_CASE). Etc.
+    if (class_name == "Decl" || class_name == "Stmt") {
+      auto sd = storage.AddMethod("UInt64");  // IR entity ID (any IR kind).
+      auto [cd_getter_name, cd_setter_name, cd_init_name] = NamesFor(sd);
+
+      class_os
+          << "  std::optional<VariantEntity> ir(void) const;\n";
+
+      serialize_inc_os
+          << "  MX_VISIT_ENTITY_ID(" << class_name
+          << ", ir, " << sd << ")\n";
+
+      serialize_cpp_os
+          << "  b." << cd_setter_name << "(es.IREntityId(e));\n";
+
+      lib_cpp_os
+          << "std::optional<VariantEntity> " << class_name << "::ir(void) const {\n"
+          << "  auto raw = impl->reader." << cd_getter_name << "();\n"
+          << "  if (raw == kInvalidEntityId) return std::nullopt;\n"
+          << "  auto vid = EntityId(raw).Unpack();\n"
+          << "  if (auto *p = std::get_if<IRFunctionId>(&vid)) {\n"
+          << "    if (auto ptr = impl->ep->IRFunctionFor(impl->ep, raw)) {\n"
+          << "      return IRFunction(std::move(ptr));\n"
+          << "    }\n"
+          << "  } else if (auto *p = std::get_if<IRBlockId>(&vid)) {\n"
+          << "    if (auto ptr = impl->ep->IRBlockFor(impl->ep, raw)) {\n"
+          << "      return IRBlock(std::move(ptr));\n"
+          << "    }\n"
+          << "  } else if (auto *p = std::get_if<IRInstructionId>(&vid)) {\n"
+          << "    if (auto ptr = impl->ep->IRInstructionFor(impl->ep, raw)) {\n"
+          << "      return IRInstruction(std::move(ptr));\n"
+          << "    }\n"
+          << "  } else if (auto *p = std::get_if<IRObjectId>(&vid)) {\n"
+          << "    if (auto ptr = impl->ep->IRObjectFor(impl->ep, raw)) {\n"
+          << "      return IRObject(std::move(ptr));\n"
+          << "    }\n"
+          << "  } else if (auto *p = std::get_if<IRStructureId>(&vid)) {\n"
+          << "    if (auto ptr = impl->ep->IRStructureFor(impl->ep, raw)) {\n"
+          << "      return IRStructure(std::move(ptr));\n"
+          << "    }\n"
+          << "  }\n"
+          << "  return std::nullopt;\n"
+          << "}\n\n";
+    }
+
     // `Decl::is_definition`
     if (class_name == "Decl") {
       const auto def = storage.AddMethod("Bool");
@@ -4089,6 +4136,9 @@ void CodeGenerator::RunOnClassHierarchies(void) {
       << "#ifndef MX_VISIT_BASE\n"
       << "#  define MX_VISIT_BASE(...)\n"
       << "#endif\n"
+      << "#ifndef MX_VISIT_ENTITY_ID\n"
+      << "#  define MX_VISIT_ENTITY_ID(...)\n"
+      << "#endif\n"
       << "#ifndef MX_VISIT_DECL_LINK\n"
       << "#  define MX_VISIT_DECL_LINK(...)\n"
       << "#endif\n"

bin/Bootstrap/Python.cpp

@@ -3,5 +3,13 @@
 #include <multiplier/Frontend.h>
 #include <multiplier/Index.h>
 
+#include <multiplier/IR/Function.h>
+#include <multiplier/IR/Block.h>
+#include <multiplier/IR/Instruction.h>
+#include <multiplier/IR/Object.h>
+#include <multiplier/IR/OpCode.h>
+#include <multiplier/IR/ObjectKind.h>
+#include <multiplier/IR/BlockKind.h>
+
 #include <multiplier/Re2.h>
 #include <multiplier/Reference.h>

bin/Bootstrap/PythonBindings.py

@@ -919,6 +919,13 @@ class UserToken;
 #include <multiplier/Fragment.h>
 #include <multiplier/Frontend.h>
 #include <multiplier/Index.h>
+#include <multiplier/IR/Function.h>
+#include <multiplier/IR/Block.h>
+#include <multiplier/IR/Instruction.h>
+#include <multiplier/IR/Object.h>
+#include <multiplier/IR/OpCode.h>
+#include <multiplier/IR/ObjectKind.h>
+#include <multiplier/IR/BlockKind.h>
 #include <multiplier/Re2.h>
 #include <multiplier/Reference.h>
 
@@ -1794,6 +1801,11 @@ def wrap(schemas: Iterable[Schema], renamer: Renamer):
   "TemplateParameterList",
   "Macro",
   "Operation",
+  "IRFunction",
+  "IRBlock",
+  "IRInstruction",
+  "IRObject",
+  "IRStructure",
 )
 
 VariantEntitySchema = make_schema_class("VariantEntity", "Entity", Schema)

bin/CMakeLists.txt

@@ -7,3 +7,4 @@
 
 add_subdirectory("Examples")
 add_subdirectory("Index")
+add_subdirectory("InterpretIR")

bin/Examples/CMakeLists.txt

@@ -91,3 +91,4 @@ define_example("mx-print-type-token-graph" "PrintTypeTokenGraph.cpp")
 define_example("mx-find-linked-structures" "FindLinkedStructures.cpp")
 define_example("mx-list-declarations-overlapping-macro-expansion"
   "ListDeclOverlappingMacroExpansions.cpp")
+define_example("mx-print-ir" "PrintIR.cpp")

bin/Examples/Harness.cpp

@@ -15,6 +15,7 @@
 
 DEFINE_uint64(entity_id, mx::kInvalidEntityId, "ID of the entity to harness");
 DEFINE_string(entity_name, "", "Name of the entity to harness");
+DEFINE_bool(deduplicate, true, "Deduplicate like names");
 
 using SeenSet = std::set<mx::PackedFragmentId>;
 using WorkList = std::vector<mx::PackedFragmentId>;
@@ -566,33 +567,35 @@ int main(int argc, char *argv[]) {
   std::unordered_map<std::string, mx::RawEntityId> canon_id;
 
   // Figure out what top-level entities need to be renamed.
-  for (mx::PackedFragmentId frag_id : frags) {
-    for (mx::Decl tld : index.fragment(frag_id)->top_level_declarations()) {
-      tld = tld.canonical_declaration();
-      std::optional<mx::NamedDecl> nd = mx::NamedDecl::from(tld);
-      if (!nd) {
-        continue;
-      }
+  if (FLAGS_deduplicate) {
+    for (mx::PackedFragmentId frag_id : frags) {
+      for (mx::Decl tld : index.fragment(frag_id)->top_level_declarations()) {
+        tld = tld.canonical_declaration();
+        std::optional<mx::NamedDecl> nd = mx::NamedDecl::from(tld);
+        if (!nd) {
+          continue;
+        }
 
-      std::string_view name_view = nd->name();
-      if (name_view.empty()) {
-        continue;
-      }
+        std::string_view name_view = nd->name();
+        if (name_view.empty()) {
+          continue;
+        }
 
-      std::string name(name_view.data(), name_view.size());
-      mx::RawEntityId eid = nd->id().Pack();
-      mx::RawEntityId stored_eid = canon_id.emplace(name, eid).first->second;
-      if (eid != stored_eid) {
-        std::cout << "// Renaming " << name_view << '\n';
-        needs_rename.insert(eid);
-        needs_rename.insert(stored_eid);
+        std::string name(name_view.data(), name_view.size());
+        mx::RawEntityId eid = nd->id().Pack();
+        mx::RawEntityId stored_eid = canon_id.emplace(name, eid).first->second;
+        if (eid != stored_eid) {
+          std::cout << "// Renaming " << name_view << '\n';
+          needs_rename.insert(eid);
+          needs_rename.insert(stored_eid);
+        }
       }
     }
-  }
 
-  // Make sure our original entity isn't subject to renaming.
-  for (mx::Decl redecl : entity->redeclarations()) {
-    needs_rename.erase(redecl.id().Pack());
+    // Make sure our original entity isn't subject to renaming.
+    for (mx::Decl redecl : entity->redeclarations()) {
+      needs_rename.erase(redecl.id().Pack());
+    }
   }
 
   std::cerr
@@ -642,9 +645,11 @@ int main(int argc, char *argv[]) {
         }
 
         std::cout << tag->name();
-        if (mx::RawEntityId eid = tld.canonical_declaration().id().Pack();
-            needs_rename.contains(eid)) {
-          std::cout << '_' << eid;
+        if (FLAGS_deduplicate) {
+          if (mx::RawEntityId eid = tld.canonical_declaration().id().Pack();
+              needs_rename.contains(eid)) {
+            std::cout << '_' << eid;
+          }
         }
         std::cout << ";\n";
       }
@@ -685,7 +690,7 @@ int main(int argc, char *argv[]) {
 
       mx::Decl decl = std::get<mx::Decl>(ent).canonical_declaration();
       mx::RawEntityId eid = decl.id().Pack();
-      if (needs_rename.contains(eid)) {
+      if (FLAGS_deduplicate && needs_rename.contains(eid)) {
         std::cout << '_' << eid;
       }
     }