feat!: Rego -> RVM Compiler and extensive testsuite (microsoft#506)

# RVM compiler test cases

Coverage:
- arithmetic
- arrays
- chained lookups
- comparisons
- comprehensions
- default rules
- destructuring
- function rules
- loops/quantifiers
- multiple entrypoints
- objects/sets
- variables
- negative/edge scenarios such as data/rule conflicts
- virtual data lookups
- etc

 # Modify interpreter and compiled policy for RVM Compilation

- Interpreter::eval_default_rule_for_compiler:
   evaluates a named default rule in isolation - allows compiler to emit a constant value instead of instructions
   for the default value

#  feat: Rego Compiler Scaffolding

- Introduce the rego::compiler module surface and entry point wiring
- Add the core compiler concepts:
  - register allocator
  - scope tracking
  - literal/builtin tables
  - rule worklists
  - instruction emit helpers
  - compiler-specific error types
  - context structs for rules, comprehensions, and loops to support later lowering passes.

# feat: Compile Rules/Queries

- add compiler::compile_from_policy workflow plus rule worklist, entry-point wiring, and recursion checks
- implement query lowering:
  - scheduling-aware statement ordering
  - loop hoisting
  - “every/some” semantics
  - context yields
  -  literal assertions
- finalize Program construction

# feat: Expression Lowering

- add compile_rego_expr and helpers to translate every AST expression into RVM instructions,
- interop with binding plans, comprehensions, and membership checks.
- implement collection literal builders (ArrayCreate, SetCreate, ObjectCreate)
  - dedupe literal keys and handle mixed literal/dynamic fields via instruction data blocks.
- operations:
  - arithmetic/boolean/bin operators
  - membership
  - unary minus
  - set unions/intersections
  - etc
- user-defined and builtin function calls
- reference handling
  - analyse chained refs
  - distinguishe data/input/local roots
  - perform rule dispatch or virtual document lookups
  - emits optimized Index/ChainedIndex instructions.

# feat: Comprehensions & Loops

- shared comprehension emitter
 - wraps array/set/object comprehensions with ComprehensionBegin/End
 - context management
- loop lowering utilities
 - read hoisting metadata
 - emit LoopStart/LoopNext
 - some in lowering
 - every quantifiers
 - index iteration
 - propagate binding plans into stored registers so downstream statements see bound variables.

# feat: Destructuring Lowering

- destructuring planner integration
 - assignment/parameter/loop bindings use hoisted plans instead of re-walking ASTs.
- handle :=, =, wildcard matches, and equality
 - evaluate RHS
 - applying destructuring plans
 - emit assert condition as needed
- support nested array/object destructuring, dynamic keys, and some ... in forms

# test: Shared Testing + RVM Suites

- move YAML test helpers into test_utils.rs and re-export via common.rs for use by interpreter and vm test suites
- comprehensive compiler test suite
  - compiles policies with the new Rego→RVM compiler
  - runs them through RegoVM
  - compares against interpreter behavior
  - supports multiple entry points
  - provides assembly listings
  - filterable YAML suites.

Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>

Author: anakrish

src/compiled_policy.rs

@@ -25,14 +25,29 @@ pub(crate) type InferredResourceTypes = BTreeMap<Ref<Query>, ResourceTypeInfo>;
 /// Wrapper around CompiledPolicyData that holds an Rc reference.
 #[derive(Debug, Clone)]
 pub struct CompiledPolicy {
-    inner: Rc<CompiledPolicyData>,
+    pub(crate) inner: Rc<CompiledPolicyData>,
 }
 
 impl CompiledPolicy {
     /// Create a new CompiledPolicy from CompiledPolicyData.
     pub(crate) fn new(inner: Rc<CompiledPolicyData>) -> Self {
         Self { inner }
     }
+
+    /// Get access to the rules in the compiled policy for downstream consumers like the RVM compiler.
+    pub fn get_rules(&self) -> &Map<String, Vec<Ref<Rule>>> {
+        &self.inner.rules
+    }
+
+    /// Get access to the modules in the compiled policy.
+    pub fn get_modules(&self) -> &Vec<Ref<Module>> {
+        self.inner.modules.as_ref()
+    }
+
+    /// Returns true when the compiled policy should use Rego v0 semantics.
+    pub fn is_rego_v0(&self) -> bool {
+        !self.inner.modules.iter().any(|module| module.rego_v1)
+    }
 }
 
 impl CompiledPolicy {

src/interpreter.rs

@@ -3371,6 +3371,39 @@ impl Interpreter {
         Ok(())
     }
 
+    /// Evaluate a default rule and return the resulting value for compiler consumers.
+    pub fn eval_default_rule_for_compiler(&mut self, rule_path: &str) -> Result<Value> {
+        self.input = Value::Undefined;
+        self.data = Value::Undefined;
+
+        let default_rules = self.compiled_policy.default_rules.get(rule_path).cloned();
+
+        if let Some(rules) = default_rules {
+            for (rule, _) in rules {
+                for module in self.compiled_policy.modules.iter() {
+                    if module.policy.contains(&rule) {
+                        let prev_module = self.set_current_module(Some(module.clone()))?;
+                        let result = self.eval_default_rule(&rule);
+                        self.set_current_module(prev_module)?;
+
+                        if result.is_ok() {
+                            let components: Vec<&str> = rule_path.split('.').skip(1).collect();
+                            let value = Self::get_value_chained(self.data.clone(), &components);
+
+                            if value != Value::Undefined {
+                                return Ok(value);
+                            }
+                        }
+
+                        return result.map(|_| Value::Undefined);
+                    }
+                }
+            }
+        }
+
+        bail!("Could not find default rule for path: {}", rule_path);
+    }
+
     fn update_data(
         &mut self,
         span: &Span,

src/languages/rego/compiler/comprehensions.rs

@@ -0,0 +1,112 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+use super::{CompilationContext, Compiler, ComprehensionType, ContextType, Register, Result};
+use crate::ast::{ExprRef, Query};
+use crate::lexer::Span;
+use crate::rvm::instructions::{ComprehensionBeginParams, ComprehensionMode};
+use crate::rvm::Instruction;
+
+impl<'a> Compiler<'a> {
+    fn compile_comprehension(
+        &mut self,
+        mode: ComprehensionMode,
+        context_type: ComprehensionType,
+        key_expr: Option<&ExprRef>,
+        value_expr: Option<&ExprRef>,
+        query: &Query,
+        span: &Span,
+    ) -> Result<Register> {
+        let result_reg = self.alloc_register();
+        let key_reg = self.alloc_register();
+        let value_reg = self.alloc_register();
+
+        let params_index = self
+            .program
+            .add_comprehension_begin_params(ComprehensionBeginParams {
+                mode,
+                collection_reg: result_reg,
+                result_reg,
+                key_reg,
+                value_reg,
+                body_start: 0,
+                comprehension_end: 0,
+            });
+
+        self.emit_instruction(Instruction::ComprehensionBegin { params_index }, span);
+
+        let body_start = self.program.instructions.len() as u16;
+
+        let context = CompilationContext {
+            context_type: ContextType::Comprehension(context_type),
+            dest_register: result_reg,
+            key_expr: key_expr.cloned(),
+            value_expr: value_expr.cloned(),
+            span: span.clone(),
+            key_value_loops_hoisted: false,
+        };
+        self.push_context(context);
+        self.compile_query(query)?;
+        self.pop_context();
+
+        self.emit_instruction(Instruction::ComprehensionEnd {}, span);
+        let comprehension_end = self.program.instructions.len() as u16;
+
+        self.program
+            .update_comprehension_begin_params(params_index, |params| {
+                params.body_start = body_start;
+                params.comprehension_end = comprehension_end;
+            });
+
+        Ok(result_reg)
+    }
+
+    pub(super) fn compile_array_comprehension(
+        &mut self,
+        term: &ExprRef,
+        query: &Query,
+        span: &Span,
+    ) -> Result<Register> {
+        self.compile_comprehension(
+            ComprehensionMode::Array,
+            ComprehensionType::Array,
+            None,
+            Some(term),
+            query,
+            span,
+        )
+    }
+
+    pub(super) fn compile_set_comprehension(
+        &mut self,
+        term: &ExprRef,
+        query: &Query,
+        span: &Span,
+    ) -> Result<Register> {
+        self.compile_comprehension(
+            ComprehensionMode::Set,
+            ComprehensionType::Set,
+            None,
+            Some(term),
+            query,
+            span,
+        )
+    }
+
+    pub(super) fn compile_object_comprehension(
+        &mut self,
+        key: &ExprRef,
+        value: &ExprRef,
+        query: &Query,
+        span: &Span,
+    ) -> Result<Register> {
+        self.compile_comprehension(
+            ComprehensionMode::Object,
+            ComprehensionType::Object,
+            Some(key),
+            Some(value),
+            query,
+            span,
+        )
+    }
+}