|
| 1 | +# Field Selection Module |
| 2 | + |
| 3 | +## Why This Module Exists Before We Have an HTTP Server |
| 4 | + |
| 5 | +The backend is currently a Rust library without an HTTP server. However, the frontend is already written to expect REST endpoints (e.g., `/api/snapshots?fields=epoch,snapshot_hash`). |
| 6 | + |
| 7 | +This module **preemptively implements the validation and allowlisting infrastructure** for sparse fieldsets, so that when the HTTP layer is added, it can use this module immediately without retrofitting security logic later. |
| 8 | + |
| 9 | +## Security Model |
| 10 | + |
| 11 | +This module addresses **semantic access control**, not cryptographic input escaping: |
| 12 | + |
| 13 | +### What It Does ✅ |
| 14 | + |
| 15 | +- **Explicit allowlist per endpoint**: Each endpoint defines exactly which fields can be requested. |
| 16 | +- **Allowlist is a subset**: Adding a new field to a Rust struct does **NOT** automatically expose it via the API. |
| 17 | +- **Client input is never interpolated**: Field names from the client are **only** used as keys in HashMap lookups. The value returned by the HashMap is the only string that reaches any downstream layer. |
| 18 | + |
| 19 | +### What It Does NOT Do ❌ |
| 20 | + |
| 21 | +- **SQL escaping**: This module contains no SQL layer (that comes when the HTTP server is added). |
| 22 | +- **Encoding/sanitization**: Input is validated by allowlist, not by escaping or encoding. |
| 23 | +- **Authorization**: This module does NOT check if the user has permission to request these fields. That's the HTTP layer's job. |
| 24 | + |
| 25 | +## Example: Two Real Domains |
| 26 | + |
| 27 | +### Snapshots Endpoint |
| 28 | + |
| 29 | +Based on `backend/src/event_indexer/dispatch.rs::NormalizedSnapshotSubmitted`: |
| 30 | + |
| 31 | +**Rust struct has:** |
| 32 | +```rust |
| 33 | +pub struct NormalizedSnapshotSubmitted { |
| 34 | + pub epoch: u64, |
| 35 | + pub snapshot_hash: String, |
| 36 | + pub source_data_hash: String, // ← Internal reconciliation state |
| 37 | + pub submitted_at: u64, |
| 38 | + pub submitter: String, // ← Operator identity (privacy concern) |
| 39 | +} |
| 40 | +``` |
| 41 | + |
| 42 | +**Allowlist exposes (only):** |
| 43 | +``` |
| 44 | +epoch |
| 45 | +snapshot_hash |
| 46 | +submitted_at |
| 47 | +``` |
| 48 | + |
| 49 | +**Rationale:** |
| 50 | +- `epoch` and `submitted_at` are audit metadata, safe for public consumption. |
| 51 | +- `snapshot_hash` is the canonical identifier clients need for verification. |
| 52 | +- `source_data_hash` is excluded: it's internal state used for reconciliation, not a public concern. |
| 53 | +- `submitter` is excluded: reveals private operator identity. |
| 54 | + |
| 55 | +**Result:** |
| 56 | +A client requesting `?fields=epoch,submitter` receives a 400 error: |
| 57 | +``` |
| 58 | +field 'submitter' is not available for endpoint 'snapshots' |
| 59 | +``` |
| 60 | + |
| 61 | +### Aggregates Endpoint |
| 62 | + |
| 63 | +Based on `backend/src/reconciliation/spec.rs::OffChainAggregate`: |
| 64 | + |
| 65 | +**Rust struct has:** |
| 66 | +```rust |
| 67 | +pub struct OffChainAggregate { |
| 68 | + pub period: u64, |
| 69 | + pub snapshot_hash: [u8; 32], |
| 70 | + pub source_data_hash: [u8; 32], // ← Internal reconciliation state |
| 71 | +} |
| 72 | +``` |
| 73 | + |
| 74 | +**Allowlist exposes (only):** |
| 75 | +``` |
| 76 | +period |
| 77 | +snapshot_hash |
| 78 | +``` |
| 79 | + |
| 80 | +**Rationale:** |
| 81 | +- Clients request aggregates by their period and snapshot hash. |
| 82 | +- `source_data_hash` is internal; clients don't request by it. |
| 83 | + |
| 84 | +## How to Integrate When the HTTP Server Exists |
| 85 | + |
| 86 | +### Step 1: Add the HTTP Handler |
| 87 | + |
| 88 | +```rust |
| 89 | +use stellar_insights_backend::field_selection::{parse_fields, FieldSelectionError}; |
| 90 | + |
| 91 | +async fn get_snapshots( |
| 92 | + Query(params): Query<QueryParams>, |
| 93 | +) -> Result<Json<SnapshotResponse>, ApiError> { |
| 94 | + // If `?fields=...` is provided, validate and filter |
| 95 | + let selected_fields = if let Some(fields_param) = params.fields { |
| 96 | + parse_fields(&fields_param, "snapshots") |
| 97 | + .map_err(|e| ApiError::BadRequest(e.to_string()))? |
| 98 | + } else { |
| 99 | + // Default to a standard set if no fields requested |
| 100 | + vec!["epoch", "snapshot_hash", "submitted_at"] |
| 101 | + }; |
| 102 | + |
| 103 | + // Now build the response with only selected_fields |
| 104 | + // ... |
| 105 | +} |
| 106 | +``` |
| 107 | + |
| 108 | +### Step 2: Map Errors to HTTP |
| 109 | + |
| 110 | +```rust |
| 111 | +use stellar_insights_backend::field_selection::FieldSelectionError; |
| 112 | + |
| 113 | +impl From<FieldSelectionError> for ApiError { |
| 114 | + fn from(e: FieldSelectionError) -> Self { |
| 115 | + // FieldSelectionError::UnknownField -> 400 Bad Request |
| 116 | + ApiError::BadRequest(e.to_string()) |
| 117 | + } |
| 118 | +} |
| 119 | +``` |
| 120 | + |
| 121 | +### Step 3: Serialize Only Selected Fields |
| 122 | + |
| 123 | +If using `serde_json`, you can dynamically include/exclude fields or use a custom serializer. Or implement a simple projection struct. |
| 124 | + |
| 125 | +## Testing |
| 126 | + |
| 127 | +Two test files validate the security properties: |
| 128 | + |
| 129 | +### `backend/tests/non_allowlisted_field_test.rs` |
| 130 | + |
| 131 | +For each endpoint with an allowlist, this test requests a field that **exists in the Rust struct** but is **deliberately excluded** from the allowlist: |
| 132 | + |
| 133 | +```rust |
| 134 | +// Attempt to request `submitter` from snapshots endpoint |
| 135 | +let result = parse_fields("epoch,submitter", "snapshots"); |
| 136 | +assert_eq!( |
| 137 | + result, |
| 138 | + Err(FieldSelectionError::UnknownField { |
| 139 | + endpoint: "snapshots", |
| 140 | + field: "submitter".to_string(), |
| 141 | + }) |
| 142 | +); |
| 143 | +``` |
| 144 | + |
| 145 | +**This validates:** The allowlist enforces a security boundary, not just rejects truly nonexistent fields. |
| 146 | + |
| 147 | +### `backend/tests/injection_shaped_field_test.rs` |
| 148 | + |
| 149 | +This test attempts a field name that contains SQL injection patterns: |
| 150 | + |
| 151 | +```rust |
| 152 | +// Client tries to inject SQL |
| 153 | +let result = parse_fields(r#"epoch,"; DROP TABLE users; --"#, "snapshots"); |
| 154 | +assert_eq!( |
| 155 | + result, |
| 156 | + Err(FieldSelectionError::UnknownField { |
| 157 | + endpoint: "snapshots", |
| 158 | + field: r#""; DROP TABLE users; --"#.to_string(), |
| 159 | + }) |
| 160 | +); |
| 161 | +``` |
| 162 | + |
| 163 | +**This validates:** The injection attempt is caught at the allowlist lookup stage (HashMap::get returns None), **not** by a SQL sanitizer. The comment in the test explains that this is the correct defense: when the HTTP server adds a SQL layer later, it will never receive this malformed field name because it was rejected at the allowlist layer. |
| 164 | + |
| 165 | +## Design Rationale |
| 166 | + |
| 167 | +### Why Not Auto-Reflect the Struct? |
| 168 | + |
| 169 | +❌ **Bad:** |
| 170 | +```rust |
| 171 | +// ← DON'T DO THIS |
| 172 | +let allowlist = derive_allowlist_from_struct::<NormalizedSnapshotSubmitted>(); |
| 173 | +// Now any new field in the struct is automatically exposed. |
| 174 | +// Adding `submitter` to the struct = it leaks to the API. |
| 175 | +``` |
| 176 | + |
| 177 | +✅ **Good:** |
| 178 | +```rust |
| 179 | +// ← DO THIS |
| 180 | +lazy_static! { |
| 181 | + static ref SNAPSHOTS_ALLOWLIST: HashMap<&'static str, &'static str> = { |
| 182 | + [("epoch", "epoch"), ("snapshot_hash", "snapshot_hash"), ...] |
| 183 | + .iter() |
| 184 | + .copied() |
| 185 | + .collect() |
| 186 | + }; |
| 187 | +} |
| 188 | +// New fields in the struct are NOT exposed until explicitly added to the allowlist. |
| 189 | +``` |
| 190 | + |
| 191 | +### Why Not Just Escape? |
| 192 | + |
| 193 | +❌ **Wrong security model:** |
| 194 | +```rust |
| 195 | +// ← DON'T DO THIS |
| 196 | +let field = client_input; // e.g., "submitter"; DROP TABLE users; --" |
| 197 | +let sql = format!("SELECT {} FROM ...", escape_sql(field)); |
| 198 | +// The field passed validation, so we build the query. |
| 199 | +// escaping prevents injection, but the field was never supposed to exist. |
| 200 | +``` |
| 201 | + |
| 202 | +✅ **Correct security model:** |
| 203 | +```rust |
| 204 | +// ← DO THIS |
| 205 | +let field = client_input; // e.g., "submitter"; DROP TABLE users; --" |
| 206 | +let allowed_fields = get_allowlist("snapshots")?; |
| 207 | +let sql_expr = allowed_fields.get(field)?; // Returns None; request fails at 400 |
| 208 | +// The field never reaches SQL at all. |
| 209 | +``` |
| 210 | + |
| 211 | +The first model escapes at the wrong layer. The second model prevents the problem entirely. |
| 212 | + |
| 213 | +## Future Extensions |
| 214 | + |
| 215 | +### Per-Tenant Allowlists |
| 216 | + |
| 217 | +If you need different field sets per customer/tenant: |
| 218 | + |
| 219 | +```rust |
| 220 | +pub trait AllowlistRegistry { |
| 221 | + fn get_allowlist(&self, endpoint: &str, tenant: &str) -> Option<&'static HashMap<&'static str, &'static str>>; |
| 222 | +} |
| 223 | +``` |
| 224 | + |
| 225 | +### Dynamic Field Aliases |
| 226 | + |
| 227 | +If the Rust field name doesn't match the public API name: |
| 228 | + |
| 229 | +```rust |
| 230 | +static ref SNAPSHOTS_ALLOWLIST: HashMap<&'static str, &'static str> = { |
| 231 | + [ |
| 232 | + ("submittedAt", "submitted_at"), // ← client sees "submittedAt", Rust uses "submitted_at" |
| 233 | + ("epoch", "epoch"), |
| 234 | + ] |
| 235 | + .iter() |
| 236 | + .copied() |
| 237 | + .collect() |
| 238 | +}; |
| 239 | +``` |
| 240 | + |
| 241 | +This already works! The HashMap values can differ from the keys. |
| 242 | + |
| 243 | +--- |
| 244 | + |
| 245 | +**Last updated:** 2026-08-30 |
| 246 | +**Status:** Ready for HTTP layer integration |
0 commit comments