Skip to content

Commit 595029d

Browse files
anakrishCopilot
andcommitted
Introduce Object storage abstraction
Add an opaque Object type for the key→value storage backing Value::Object. It exposes a small set of methods (get, insert, remove, iter, iter_sorted, cursor, serde) and keeps the backing store private, so future representations -- inline small-map, hash-backed, lazy, arena, FFI-callback -- can plug in without touching the call sites that name this type. Nothing in the engine uses Object yet. Value::Object still wraps Rc<BTreeMap<Value, Value>>; the payload swap and call-site migration come in the next PR. Object stands on its own unit tests in the meantime. docs/value/object.md walks through the design, the precedents it follows (serde_json::Map, toml::Table, simdjson DOM), and the concrete workloads the abstraction is meant to unlock. A matching Set abstraction follows in a separate PR. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent acf7f7a commit 595029d

8 files changed

Lines changed: 1123 additions & 4 deletions

File tree

docs/value/object.md

Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,84 @@
1+
# Object
2+
3+
Opaque container for `Value::Object`'s key→value storage, enabling
4+
alternative backends without call-site changes.
5+
6+
## Design
7+
8+
`Object` wraps the storage for a key→value collection of `Value`s and
9+
provides a curated set of methods (`get`, `insert`, `remove`, `iter`,
10+
`iter_sorted`, `cursor`, serde). The backing store is private; callers
11+
never see or pattern-match on it, so the representation can change
12+
without rippling through call sites.
13+
14+
Multiple backends can coexist at runtime. Because the backing store is
15+
private, different `Object` instances in the same process can use
16+
different implementations — e.g., a lazy DB-backed object for `input`,
17+
inline small-map objects for SARIF location records, and a regular
18+
sorted map elsewhere — all interoperating through the same opaque
19+
type. This is stronger than the typical Cargo-feature-selected backend
20+
seen in precedent crates.
21+
22+
Iteration is split intentionally. `iter()` makes no ordering promise,
23+
which lets backends that don't keep entries sorted skip any sort work.
24+
`iter_sorted()` returns entries in `Value` order and is what
25+
serialization and `Ord` rely on for deterministic output. Cursor types
26+
add resumable, incremental traversal for the RVM iteration state
27+
without leaking iterator internals.
28+
29+
`Ord` and `PartialOrd` are defined against `iter_sorted()` rather than
30+
derived from the storage. Two `Object`s built on different backends —
31+
or with different insertion histories — compare equal whenever their
32+
sorted entries match, so changing the backend never changes observable
33+
comparison results.
34+
35+
## Precedents
36+
37+
Other crates that hide storage behind a stable API so the implementation
38+
can change without breaking callers:
39+
40+
- **`serde_json::Map`** — opaque newtype allowing cargo-feature based
41+
swap between `BTreeMap` (canonical order) and `IndexMap` (insertion
42+
order).
43+
- **`toml::Table`** — opaque newtype allowing cargo-feature based swap
44+
between `BTreeMap` and `IndexMap`.
45+
- **`simdjson` DOM** — opaque tree that lazily materializes nodes on
46+
access instead of parsing the whole document up front.
47+
48+
## Use cases
49+
50+
- **SARIF small-object pressure** — SARIF reports contain millions of
51+
small objects (location records, rule references, message arguments),
52+
most with 2-5 keys. A small-map-optimized backend (inline storage
53+
for ≤N entries, heap above) eliminates per-object BTreeMap allocation
54+
for the common case.
55+
56+
- **Kubernetes admission policies** — large, deeply-nested resource
57+
objects (Pod specs, CRDs) where policies typically touch a handful
58+
of paths. A lazy-materializing backend (`LazyObjectProvider` over
59+
the incoming JSON) parses only the accessed subtrees.
60+
61+
- **Azure Policy aliases** — ARM exposes the same logical property
62+
under multiple aliases (e.g. paths like
63+
`Microsoft.Compute/virtualMachines/storageProfile.osDisk.managedDisk.id`).
64+
An alias-aware backend resolves lookups across canonical and alias
65+
forms without rewriting every policy.
66+
67+
- **Azure Policy case-insensitive compare** — ARM property names are
68+
case-preserving but case-insensitive on lookup (`tags.Environment`
69+
and `tags.environment` resolve identically). A case-insensitive
70+
backend centralizes this once at the storage layer instead of at
71+
every comparison site.
72+
73+
- **External data sources**`input` or `data` backed by a database
74+
query, CBOR slice, REST endpoint, or other streaming source via a
75+
`LazyObjectProvider`. Entries materialize on demand; the policy
76+
only pays for what it touches.
77+
78+
- **Eval-time temporaries** — objects constructed during evaluation
79+
(comprehensions, intermediate rule results) on a bumpalo arena.
80+
The whole arena drops at query end with zero per-entry free cost.
81+
82+
- **Host-language interop** — Python dicts or JS objects accessed via
83+
FFI callbacks from the embedding application, without copying into
84+
Rust on every binding boundary.

src/lib.rs

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -177,7 +177,7 @@ pub use utils::limits::{
177177
global_memory_limit, set_global_memory_limit, set_thread_flush_threshold_override,
178178
thread_memory_flush_threshold,
179179
};
180-
pub use value::Value;
180+
pub use value::{Object, Value};
181181

182182
/// Compiled-pattern caches for the `regex.*` and `glob.*` Rego builtins.
183183
///

src/scheduler.rs

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -569,7 +569,7 @@ impl Analyzer {
569569
}
570570
Ok(false)
571571
}
572-
Array { .. } | Object { .. } => Ok(true),
572+
Expr::Array { .. } | Expr::Object { .. } => Ok(true),
573573
_ => Ok(false),
574574
})?;
575575
Ok(true)
@@ -666,7 +666,7 @@ impl Analyzer {
666666
Ok(false)
667667
}
668668
// TODO: key vs value for object binding
669-
Array { .. } | Object { .. } => Ok(true),
669+
Expr::Array { .. } | Expr::Object { .. } => Ok(true),
670670
_ => Ok(false),
671671
})?;
672672
Ok(vars)
@@ -853,7 +853,7 @@ impl Analyzer {
853853
Ok(false)
854854
}
855855
// TODO: Object key/value
856-
Array { .. } | Object { .. } => Ok(true),
856+
Expr::Array { .. } | Expr::Object { .. } => Ok(true),
857857
_ => {
858858
non_vars.push(e.clone());
859859
Ok(false)

src/value.rs renamed to src/value/mod.rs

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,18 @@
1111
clippy::as_conversions
1212
)] // value helpers index paths directly for performance
1313

14+
mod object;
15+
16+
#[cfg(test)]
17+
mod tests;
18+
19+
#[allow(unused_imports)] // surface for downstream PRs
20+
pub use object::{IntoIter, Iter, IterMut, Object};
21+
22+
#[cfg(feature = "rvm")]
23+
#[allow(unused_imports)] // surface for downstream PRs
24+
pub use object::ObjectCursor;
25+
1426
use crate::number::Number;
1527

1628
use alloc::collections::{BTreeMap, BTreeSet};

src/value/object/iter.rs

Lines changed: 148 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,148 @@
1+
// Copyright (c) Microsoft Corporation.
2+
// Licensed under the MIT License.
3+
4+
//! Opaque iterator types for [`Object`].
5+
//!
6+
//! These newtypes wrap the storage backend's iterators so the backend can be
7+
//! swapped without changing any iterator type signatures observed by callers.
8+
9+
use alloc::collections::btree_map;
10+
use core::iter::FusedIterator;
11+
12+
use super::Object;
13+
use crate::value::Value;
14+
15+
/// Owned iterator over `(Value, Value)` entries.
16+
#[derive(Debug)]
17+
pub struct IntoIter {
18+
pub(super) inner: btree_map::IntoIter<Value, Value>,
19+
}
20+
21+
impl Iterator for IntoIter {
22+
type Item = (Value, Value);
23+
#[inline]
24+
fn next(&mut self) -> Option<Self::Item> {
25+
self.inner.next()
26+
}
27+
#[inline]
28+
fn size_hint(&self) -> (usize, Option<usize>) {
29+
self.inner.size_hint()
30+
}
31+
}
32+
33+
impl DoubleEndedIterator for IntoIter {
34+
#[inline]
35+
fn next_back(&mut self) -> Option<Self::Item> {
36+
self.inner.next_back()
37+
}
38+
}
39+
40+
impl ExactSizeIterator for IntoIter {
41+
#[inline]
42+
fn len(&self) -> usize {
43+
self.inner.len()
44+
}
45+
}
46+
47+
impl FusedIterator for IntoIter {}
48+
49+
/// Borrowed iterator over `(&Value, &Value)` entries.
50+
#[derive(Debug, Clone)]
51+
pub struct Iter<'a> {
52+
pub(super) inner: btree_map::Iter<'a, Value, Value>,
53+
}
54+
55+
impl<'a> Iterator for Iter<'a> {
56+
type Item = (&'a Value, &'a Value);
57+
#[inline]
58+
fn next(&mut self) -> Option<Self::Item> {
59+
self.inner.next()
60+
}
61+
#[inline]
62+
fn size_hint(&self) -> (usize, Option<usize>) {
63+
self.inner.size_hint()
64+
}
65+
}
66+
67+
impl<'a> DoubleEndedIterator for Iter<'a> {
68+
#[inline]
69+
fn next_back(&mut self) -> Option<Self::Item> {
70+
self.inner.next_back()
71+
}
72+
}
73+
74+
impl<'a> ExactSizeIterator for Iter<'a> {
75+
#[inline]
76+
fn len(&self) -> usize {
77+
self.inner.len()
78+
}
79+
}
80+
81+
impl<'a> FusedIterator for Iter<'a> {}
82+
83+
/// Borrowed iterator over `(&Value, &mut Value)` entries.
84+
#[derive(Debug)]
85+
pub struct IterMut<'a> {
86+
pub(super) inner: btree_map::IterMut<'a, Value, Value>,
87+
}
88+
89+
impl<'a> Iterator for IterMut<'a> {
90+
type Item = (&'a Value, &'a mut Value);
91+
#[inline]
92+
fn next(&mut self) -> Option<Self::Item> {
93+
self.inner.next()
94+
}
95+
#[inline]
96+
fn size_hint(&self) -> (usize, Option<usize>) {
97+
self.inner.size_hint()
98+
}
99+
}
100+
101+
impl<'a> DoubleEndedIterator for IterMut<'a> {
102+
#[inline]
103+
fn next_back(&mut self) -> Option<Self::Item> {
104+
self.inner.next_back()
105+
}
106+
}
107+
108+
impl<'a> ExactSizeIterator for IterMut<'a> {
109+
#[inline]
110+
fn len(&self) -> usize {
111+
self.inner.len()
112+
}
113+
}
114+
115+
impl<'a> FusedIterator for IterMut<'a> {}
116+
117+
impl IntoIterator for Object {
118+
type Item = (Value, Value);
119+
type IntoIter = IntoIter;
120+
#[inline]
121+
fn into_iter(self) -> Self::IntoIter {
122+
IntoIter {
123+
inner: self.inner.into_iter(),
124+
}
125+
}
126+
}
127+
128+
impl<'a> IntoIterator for &'a Object {
129+
type Item = (&'a Value, &'a Value);
130+
type IntoIter = Iter<'a>;
131+
#[inline]
132+
fn into_iter(self) -> Self::IntoIter {
133+
Iter {
134+
inner: self.inner.iter(),
135+
}
136+
}
137+
}
138+
139+
impl<'a> IntoIterator for &'a mut Object {
140+
type Item = (&'a Value, &'a mut Value);
141+
type IntoIter = IterMut<'a>;
142+
#[inline]
143+
fn into_iter(self) -> Self::IntoIter {
144+
IterMut {
145+
inner: self.inner.iter_mut(),
146+
}
147+
}
148+
}

0 commit comments

Comments
 (0)