Status: implemented (see the Go package at the repository root) Module:
github.com/BrobridgeOrg/dyfieldsScope: a standalone, dependency-free general-purpose Go library. Build a form definition (schema) programmatically, serialize it to JSON to store or ship to a client, then validate externally supplied values against that same definition.
This document is a rewrite of plasma-backend/docs/dynamic_fields.md (originally a design
proposal for Plasma's internal lib/fieldschema). The difference: this project is a
general-purpose library, bound to no HTTP framework and presuming no module's routes or
error codes. The original's HTTP contract and module-adoption sections are replaced here by a
statement of the library's boundary, and four themes absent from the original are added:
Builder (adding and removing fields dynamically), serialization / deserialization,
cross-field validation, and arrays of objects.
| # | Goal | API |
|---|---|---|
| 1 | Add and remove fields to compose a form definition | Builder / mutations on Schema |
| 2 | Export the definition as JSON and load it back | Schema.MarshalJSON / Parse |
| 3 | Validate externally supplied values against the definition | Compiled.Validate |
All three share one Schema object: it is render instructions, validation rules, and API
documentation at once. They cannot drift, because they are the same data.
This library does not only serve "connection settings" — forms hard-coded by developers. It also serves business forms (registration, checkout, sign-up) and user-built forms (surveys, form builders). That scope has four structural consequences that run through the entire design:
- A schema is runtime data, not a startup constant. An end user may drag it together in a
UI, store it in a database, and change it at any time. So
Compile()errors must be human-readable and locatable, not merely a sentinel for a panic at boot. Field.Namemay be machine-generated (q_7f3a) rather than an identifier a developer typed. Usingnameas an i18n translation key is therefore not enough;labelitself must be able to carry multiple languages (see §13).- Validation is broader. Config forms are mostly single-field range checks; business forms will always hit cross-field validation (confirm password, start/end dates). See §10.
- Arrays of objects are the norm, not an edge case. Order line items, several contacts,
multiple brokers, sub-question groups in a survey — without them, the most common business
forms degrade into a single
jsonfield. See §5.
Non-goals (full list in §14): no UI, no storage, no HTTP, no file transfer.
Schema
├── Version compatibility marker (schema_version)
└── Groups[] field grouping; decides "render it?" and "validate it?"
└── Fields[] a single field; decides how to render, when to appear, how to validate
└── Fields[] (object_list only) the fields of each entry; forms a new scope
Values are separate from the definition. Values are a value document:
{
"values": {
"cluster": "prod",
"brokers": [
{ "$id": "b1", "host": "a.example.com", "port": 9092 },
{ "$id": "b2", "host": "b.example.com", "port": 9092 }
]
},
"secrets": { "brokers.b1.password": "…", "brokers.b2.password": "…" }
}secrets is its own container rather than something you filter key by key — exports, audits,
and logs can dump values wholesale, and the risk of leaking is excluded structurally
rather than depending on the caller remembering to filter. secrets is always a flat
map[string]string, even when values are nested (see §5.4). That makes "no secrets in the
values tree" an invariant you can verify at a glance, instead of a property you would have to
walk two trees to confirm.
Even a caller who just wants "a bunch of fields" with no grouping still gets groups: the
Builder provides a default default group, and AddField() adds straight into it (see §6.1).
That keeps the serialized format singular — no "with groups" and "without groups" variants to
handle.
The type set was derived backwards from real form cases, not forwards from "what a type system ought to look like".
| Class | Type | Go representation | Notes |
|---|---|---|---|
| Scalar | string |
string |
|
integer |
int64 |
Fractions rejected; retry counts, scales, ports | |
number |
float64 |
Ratios, coefficients, weights | |
decimal |
string |
Exact decimal; money, tax rates. float64 loses precision | |
boolean |
bool |
The only type toggle.field may have |
|
| Sensitive | secret |
string |
Lives in the secrets container, never read back |
| Choice | enum |
scalar / []scalar |
multiple decides single or multi select |
| Time | datetime |
string (RFC3339) |
format distinguishes date / time / datetime |
| File | file |
string |
The value is a reference (storage key / URL), not contents |
| Container | list |
[]scalar |
Free-form homogeneous list; order is part of the value |
map |
map[string]scalar |
Arbitrary key/value | |
object_list |
[]object |
Array of objects; see §5 | |
| Escape hatch | json |
any |
Only checked for "is a valid JSON value" |
A value that must be stored and read back but still kept out of an API response or a log — a
national ID, a phone number — is not a secret. It declares mask instead, which shapes only
what Redact writes: "all", "omit", or a rule saying how many characters survive at each
end. See the README.
Nothing marks the result as a mask, and nothing downstream can recognise one: "******6789"
is a string like any other. So the rule keeping a mask from overwriting what it hides is not a
check on the way in, it is that the client sends a patch. Diff produces one from the
document the form was given and the one it holds, so an untouched masked field is not in the
request to begin with, and Apply keeps every value the request does not mention. The masking
rule and the partial-update rule are the same rule seen from two ends — which makes Diff
part of the design rather than a convenience on top of it.
| Need | Type |
|---|---|
| Pick several from fixed options (permissions, tags) | enum + multiple |
| Type in several homogeneous scalars (recipients, tags) | list |
| Arbitrary key/value (headers, env, localized values) | map |
| Several named fields per entry (line items, contacts, brokers) | object_list |
enum + multiple and list do not substitute for each other: one is selection, the other is
input.
Elements of a list / map are scalars, described by items, which is deliberately not a
full Field:
type Items struct {
Type Type `json:"type"` // scalar types only
Format string `json:"format,omitempty"`
Options []Option `json:"options,omitempty"`
Pattern string `json:"pattern,omitempty"`
Min, Max *float64 `json:"min,omitempty"`
MinLength, MaxLength *int `json:"min_length,omitempty"`
}For a map, items describes the value. Keys are constrained separately by key_pattern
(a regexp) on the field — HTTP header names, environment variable names, and label keys all
have format requirements, and without it there would be nowhere to state them. A bad-key error
carries FieldError.Key to say which one.
There is no name, no label, no conditions — elements have no identity, so they need no
scope. Anything that needs named fields and conditions uses object_list (§5); that is where
scope is introduced.
The value of a file field is a reference, not contents. The library checks that the
reference is present and non-empty; the actual upload, storage, and size checks belong to the
caller — those need I/O, which is outside this library.
accept / max_size live in Metadata as declarations for the client and the caller; the
library does not enforce them. This has to be documented, or users will assume the size limit
is already being policed.
Money, tax rates, and exchange rates cannot use float64 (0.1 + 0.2 != 0.3). JSON has no exact
decimal type either, so a decimal value travels as a string ("19.99"); the library
validates that it is a well-formed decimal and compares it against Min / Max. A caller that
needs arithmetic reaches for something like shopspring/decimal.
package dyfields
type Schema struct {
Version int `json:"schema_version"`
Groups []*Group `json:"groups"`
}
type Group struct {
Key string `json:"key"` // a contract once published
Label string `json:"label"`
LabelI18n map[string]string `json:"label_i18n,omitempty"`
Description string `json:"description,omitempty"`
Mode GroupMode `json:"mode"` // always|collapsible|toggleable
DefaultCollapsed bool `json:"default_collapsed,omitempty"` // collapsible only
Toggle *Toggle `json:"toggle,omitempty"` // toggleable only, mandatory
VisibleWhen *Condition `json:"visible_when,omitempty"`
ValidWhen []*Rule `json:"valid_when,omitempty"` // group-level validation, see 10.2
Fields []*Field `json:"fields"`
}
type Toggle struct {
Field string `json:"field"` // name of the switch field; must be boolean
Label string `json:"label"`
Default bool `json:"default"`
}
type Field struct {
Name string `json:"name"` // the value key; a contract once published, never renamed
Type Type `json:"type"`
Format string `json:"format,omitempty"`
Label string `json:"label"`
LabelI18n map[string]string `json:"label_i18n,omitempty"`
Description string `json:"description,omitempty"`
Placeholder string `json:"placeholder,omitempty"`
Required bool `json:"required"` // not allowed on containers, see 11.1
ReadOnly bool `json:"readonly,omitempty"` // always read-only; symmetric with Required
Transient bool `json:"transient,omitempty"` // dropped after successful validation, see 4.3
Default any `json:"default,omitempty"`
Options []Option `json:"options,omitempty"` // enum
Multiple bool `json:"multiple,omitempty"` // multi-select enum
Items *Items `json:"items,omitempty"` // list / map
// object_list only
Fields []*Field `json:"fields,omitempty"` // the fields of each entry; a new scope
MinItems *int `json:"min_items,omitempty"` // also applies to list / map
MaxItems *int `json:"max_items,omitempty"`
Unique []string `json:"unique,omitempty"` // field names unique across entries, see 5.5
KeyPattern string `json:"key_pattern,omitempty"` // map only: constrains the key format
ItemLabel string `json:"item_label,omitempty"` // per-entry title template, e.g. "{host}:{port}"
Layout string `json:"layout,omitempty"` // table|card, a pure UI hint
VisibleWhen *Condition `json:"visible_when,omitempty"`
RequiredWhen *Condition `json:"required_when,omitempty"`
ReadOnlyWhen *Condition `json:"readonly_when,omitempty"`
ValidWhen []*Rule `json:"valid_when,omitempty"`
Pattern string `json:"pattern,omitempty"`
MinLength *int `json:"min_length,omitempty"`
MaxLength *int `json:"max_length,omitempty"`
Min *float64 `json:"min,omitempty"`
Max *float64 `json:"max,omitempty"`
Metadata map[string]any `json:"metadata,omitempty"` // caller-defined, preserved verbatim
}
type Option struct {
Value any `json:"value"`
Label string `json:"label"`
LabelI18n map[string]string `json:"label_i18n,omitempty"`
}MinLength / MaxLength / Min / Max / MinItems / MaxItems are all pointers, because
0 is a meaningful value and must not be confused with "not set".
Entry counts use their own min_items / max_items rather than reusing min_length — if
"string length" and "list length" shared one field, a reader of the schema would have to know
the type before knowing what that number means.
Metadata is where the caller puts things the library has no business knowing
(resource_type, UI hints, permission markers).
format does two things: it tells the client which widget to render, and it tells the
library which built-in validation to apply.
| Class | Values |
|---|---|
| Has built-in validation | email uri hostname ipv4 ipv6 uuid duration (30s) bytesize (512Mi) cron regex (the value itself must compile) |
| Pure UI hint | textarea code markdown color phone password |
datetime only |
date time datetime |
A correction to an earlier judgement: early versions argued that "format takes no part in validation, so the library's rule table cannot drift from the client's". That does not hold — if the library does not validate
uri, every caller writes its own pattern, and that is what really diverges. The right principle is one copy of the rules, in the library; the client's format checks are UX hints only.
An unknown format value is not an error; it degrades to a pure UI hint. Otherwise every new
format the library adds would be a breaking change.
The value namespace is flat (object_list opens a nested one): the keys of values are
Field.Name, and a group's key never appears in a value document.
That buys an important property: moving a field from group A to group B, regrouping, or changing a group's label leaves already-stored values completely untouched — essential for a form builder, where users rearrange the layout constantly.
The cost is that Field.Name must be unique within its level; a group is not a scope. Two
groups each holding a name field is an error, and Compile reports duplicate_name.
"Confirm password" is the sole but sufficient reason for this mechanism: it must exist to be
compared against password, but must never be stored. Without the flag, the caller would
have to remember to delete it from secrets after Validate — and "the caller has to
remember" is exactly the class of problem the secrets container exists to eliminate.
A Transient field exists during validation (otherwise equals_field has nothing to
compare) and is removed from the returned document after it passes. It is not removed when
validation fails, because the caller may need to send the value back to the client.
Because it is dropped from the payload, a transient field is inherently client-only: it never reaches the server at all, and that has consequences on both sides of the wire.
Nothing may make the shape of the document depend on one. visible_when, required_when and
readonly_when — on a field or on a group — are rejected at compile time (transient_ref) when
they read a transient field. The two sides would answer such a condition differently, and the
server's answer is the one that gets persisted: a visible_when reading a transient field marks
its field invisible on every server-side merge, which strips the value from the stored document
on an update that never mentioned it. There is no evaluation order that repairs that, so the
schema is refused instead. The rule is about what a condition reads, not where it is written:
a transient field keeps its own required_when and its own valid_when.
valid_when is the exception, because it only reports a problem and never changes what is
stored — so instead of refusing it, the library simply does not ask it on the server. Apply
reports nothing about a transient field, and nothing about a valid_when rule that reads one or
hangs on one: the value never travels in a patch, so it is never in the stored document, so
Apply could never satisfy a required or an equals_field on it however well the user filled
the form in. A server cannot check what it was never sent — the mirror image of the rule that
keeps a client from checking what it was never shown. Compiled works out which rules those are
once, at compile time (markClientOnly), so no backend has to remember. The value itself is
still visited and still dropped: what Apply throws away is the verdict, not the visit.
A field with Type == TypeSecret always reads and writes the value document's secrets
container, never values. When Redact() produces a version safe to hand out, secrets is
replaced by a secrets_set list:
{ "values": { … }, "secrets_set": ["password", "brokers.b1.password"] }secret is its own type rather than a flag on string: as a flag, "forgot to set it" is a
leak; as a type, it is an error at the type level.
This is the largest extension over the original document. The original listed it under "deliberately not done", on the grounds that conditions would need a notion of scope. Scope is indeed necessary complexity, but it can be defined completely — as long as references are restricted to one direction.
{
"name": "brokers", "type": "object_list", "label": "Broker",
"min_items": 1, "max_items": 16,
"unique": ["host"],
"item_label": "{host}:{port}",
"layout": "table",
"fields": [
{ "name": "host", "type": "string", "format": "hostname", "required": true },
{ "name": "port", "type": "integer", "min": 1, "max": 65535, "default": 9092 },
{ "name": "use_tls", "type": "boolean", "default": false },
{ "name": "password", "type": "secret",
"visible_when": { "field": "use_tls", "is_true": true } }
]
}An entry holds a flat list of fields, with no groups. Groups are a UI partition plus a
toggle mechanism; subdividing a single entry is over-design, and toggleable's "closing clears
it" has no clear meaning inside one entry. Use visible_when to partition.
An entry may contain another object_list (recursion). The rules are identical at every
level, so nothing special is done for it; Compile enforces MaxDepth (5 by default) as a
guard — a schema may come from an end user, so it needs an upper bound.
A condition on a field inside an entry may want to reference three things; only the first two are allowed:
| Reference target | Syntax | Supported |
|---|---|---|
| A sibling field in the same entry | "use_tls" (no prefix, same level by default) |
✅ |
| A field further out | "^.cluster" (one level out, stackable), "$.cluster" (root) |
✅ |
| A field in another entry | — | ❌ never |
The third is where complexity explodes ("the condition on entry 2 refers to which entry?"). Excluding it flattens scope into something simple.
Outer levels cannot reference inner ones either: an inner level has N entries, and "whose
host?" has no answer. That restriction buys something: cross-level references are necessarily
one-directional (inner → outer), so a cross-level cycle is impossible, and the acyclicity
check only has to run within a single level. Compile reports an outward-to-inward reference
as unknown_ref.
When an outer level wants to constrain an inner one, it constrains the object_list field
itself (min_items, unique) rather than referencing a field inside it. That happens to
be the shape the requirement actually has.
Condition resolution is static: the schema structure is known at compile time, so which
field definition ^.cluster points at is fixed; at runtime only "the current entry" gets bound
in. So reference existence and type compatibility are both checkable at compile time.
Every entry carries a $id (a string). It is not a user-defined field; it is a key reserved by
the library.
Why it is needed: an index is not a stable identity. Once the user deletes entry 0, what
was entry 1 becomes brokers[0] — and any external state keyed by index (secrets above all)
silently shifts, handing A's password to B. This is a data-correctness problem, not a
convenience one.
Does the entry contain a secret field? |
$id |
|---|---|
| No | Optional. Validate fills one in and returns it; without one, Apply replaces wholesale |
| Yes | Required, and must be generated by the client. Missing it is a missing_item_id error |
Why the server cannot fill it in when secrets are present: the payload's secret keys are
companions.<$id>.id_number — the client must know the id before it can write that key. By the
time Validate could fill one in, the key is already fixed. So Compile marks such fields
requires_item_id, letting the client learn from the schema that this responsibility is its
own.
The client may generate the id as a uuid or a nanoid; the library only requires that it is non-empty and unique within the list.
Ordinary users never notice $id exists — it is only mandatory in the "each entry has its own
password" case, which is exactly the case that goes wrong without it.
The secrets container stays flat, and its keys are $id paths:
brokers.b1.password
orders.o3.items.i7.license_key // nesting concatenates level by level
Note that secret keys use $id while error paths use the index (brokers[0].password).
They differ on purpose, because their purposes differ: secrets must be stable across edits
(an index changes, a $id does not), while an error message must let a human find the right
row on screen. FieldError carries both: path (index) and item_path (the $id path) —
the latter shares its prefix shape with secret keys, so a client can match it straight against
secrets_set. When nesting, a single item_id cannot hold two levels; it has to be a path.
$id is restricted to [A-Za-z0-9_-]{1,64}. Because secret keys are joined with ., a
$id containing . would make sinks.a.b.token ambiguous between "sink a.b's token" and
"sink a's b.token". uuids and nanoids satisfy this naturally; Validate enforces it (it is
a value, not schema, so Compile cannot see it).
unique: ["host"] means no two entries may repeat host; unique: ["host", "port"] is a
composite uniqueness constraint (the host+port pair does not repeat), not two independent
ones. Two independent constraints would mean two object_list fields — which does not happen
in practice, so no syntax is designed for it.
Values are normalized before comparison (strings trimmed; case-insensitive when format is
email or hostname). Fields hidden within an entry have had their values removed and take no
part in the comparison.
unique may be applied to a secret field ("the same national ID may not register twice"
is a real requirement), but that turns the error message into a leak: a duplicate error
must never contain the duplicated value, only path and item_path. This needs explicit
handling in the implementation — the default "value X is duplicated" message is wrong here.
Each entry runs the §11 convergence procedure independently: compute that entry's visibility (same-level conditions read that entry's values, outer conditions read the parent scope), remove the values of hidden fields, apply defaults, recompute, then validate.
Entries do not interact. The only exceptions are unique and min_items / max_items, which
are constraints on the list as a whole and run after every entry has validated on its own.
Schema is plain data and can be written by hand. In practice, though, it is "some code
decides whether to add this field", so a Builder is provided, separating "what was added" from
"was it added correctly": the Builder only accumulates operations; every consistency check
waits for Build(). Temporarily invalid states are allowed along the way (a condition
referencing a field not yet added, say), or the caller would be forced to care about ordering.
b := dyfields.NewBuilder()
b.SetVersion(1)
// 6.1 No grouping: add fields straight into the default group
b.AddField(dyfields.String("cluster").Label("Cluster").Required())
b.AddField(dyfields.List("recipients").Items(dyfields.ItemString().Format("email")))
b.AddField(dyfields.Map("headers").Items(dyfields.ItemString()))
b.AddField(dyfields.Decimal("amount").Label("Amount").MinDec("0"))
// 6.2 Arrays of objects
brokers := dyfields.ObjectList("brokers").Label("Broker").
MinItems(1).MaxItems(16).Unique("host").ItemLabel("{host}:{port}")
brokers.AddField(dyfields.String("host").Format("hostname").Required())
brokers.AddField(dyfields.Int("port").Range(1, 65535).Default(9092))
brokers.AddField(dyfields.Bool("use_tls").Default(false))
brokers.AddField(dyfields.Secret("password").
VisibleWhen(dyfields.IsTrue("use_tls"))) // same level
brokers.AddField(dyfields.String("sni").
VisibleWhen(dyfields.IsTrue("$.global_tls"))) // references the root
b.AddField(brokers)
// 6.3 Grouping
g := b.Group("tls").Label("TLS / SSL").Toggleable("global_tls", "Enable TLS", false)
g.AddField(dyfields.Enum("ssl_mode").Options(
dyfields.Opt("require", "Require"), dyfields.Opt("verify-ca", "Verify CA")))
// 6.4 Add, edit, remove
b.RemoveField("recipients") // removes across groups, reports whether it hit
b.RemoveGroup("tls") // removes the group's fields with it
b.FieldAt("brokers.port") // fetch a nested field by path and edit it in place
b.RemoveFieldAt("brokers.sni")
schema, err := b.Build() // only now is everything checked| Method | Meaning |
|---|---|
AddField(f) |
Adds to the default group. A duplicate name is recorded and reported by Build() |
Group(key) |
Gets or creates that group (idempotent) |
RemoveField(name) bool / RemoveGroup(key) bool |
Removal |
Field(name) (*Field, bool) |
Fetch the pointer and edit in place |
FieldAt(path) / RemoveFieldAt(path) |
Operate on nested fields by a path like brokers.port |
MoveField(name, groupKey, idx) |
Change group or order (a form builder needs this) |
Extend(other Schema) |
Merge in another schema (same-key groups merge, same-name fields conflict) |
Build() (Schema, error) |
Produce and check |
FromSchema(s) reads an existing schema back into a Builder to keep editing — that is the path
a form builder's "edit an existing form" takes, and together with Parse() it closes the round
trip.
data, err := json.Marshal(schema)
schema, err := dyfields.Parse(data) // deserialize + structural checks
schema, err := dyfields.ParseReader(r)Rules:
- Round-trip stability:
Parse(Marshal(s))is deeply equal tos(includingMetadataand every nested level). A test must guard this — otherwise "store it and read it back" silently changes behaviour. Parseperforms structural checks, not justjson.Unmarshal. Unknown JSON keys are always rejected (DisallowUnknownFields). The exceptions areMetadataand unknownformatvalues (see §4.1).- Field arrays preserve order. Order is render order, part of the definition, so it cannot be stored in a map.
schema_versionmeans whatever the caller decides; the library only preserves and compares it.
c, err := dyfields.Compile(schema)Compile checks whether the schema itself is correct, independent of any values:
- Field names unique within a scope, group keys unique (all entries of a list share one scope)
- Referenced fields exist and the direction is legal (same level or outward; outer
referencing inner →
unknown_ref) - Same-level
visible_whendependencies must not form a cycle — a topological check (cross-level is one-directional, so no cycle is possible there) - A
toggleablegroup must have atoggle, andtoggle.fieldmust not collide with any other field name (see §9.1) - Container types (
list/map/object_list) must not carryrequired(usemin_items, see §11.1) - Both sides of an ordering operator (
gt_field/gte_field/lt_field/lte_field) must be comparable (see §10.1) enummust haveoptions;list/mapmust haveitemswith a scalaritems.typeobject_listmust havefields; nesting must not exceedMaxDepth- Fields named in
uniqueexist among thatobject_list'sfieldsand are scalars - Fields referenced by an
item_labeltemplate exist - An entry containing a
secretmarks its fieldrequires_item_id(see §5.3) Patternandformat: "regex"compile;Min <= Max,MinItems <= MaxItemsDefaultis compatible withType
Only same-level visible_when needs the acyclicity check, because only it feeds back into
visibility; required_when / readonly_when / valid_when are terminal consumers and produce
no loops. The original design applied "must not cycle" to every condition, which was too strict.
Because a schema may come from an end user (see §1.1), Compile returns a locatable list
rather than a single error string:
type SchemaError struct {
Path string `json:"path"` // brokers.password
Group string `json:"group,omitempty"`
Code string `json:"code"` // duplicate_name|unknown_ref|cycle|missing_items|depth_exceeded|…
Reason string `json:"reason"`
}Compiled is read-only and concurrency-safe (regexps pre-compiled, dependency graph already
built), so it can be held for the life of the process.
mode |
Behaviour |
|---|---|
always |
Always shown, always validated |
collapsible |
Folded by default; contents still exist and are still validated |
toggleable |
Controlled by a switch field; when off, the whole group does not exist |
The switch itself (toggle.field) is a real field, present in values, of type boolean,
and always kept and validated.
It is implicitly defined by the toggle declaration; it need not — and may not — be
declared again in any group's fields. Its type is fixed to boolean, and label and default
live inside toggle. What Compile checks is that it does not collide with another field
name, not that it exists somewhere.
This rule was forced by the three-way correspondence: if the switch had to be declared inside some group, the UI would draw it twice (once in the header, once in the body), and the payload could not say which group it belongs to. Semantically the switch belongs to the group it controls — it just is not cleared along with it.
When the switch is false:
- The group is not validated (
requiredincluded — none of it counts) - Existing values of the group's fields are removed from
values/secrets - (The client accordingly does not render the group)
Point 2 is deliberate. Leftover values would produce states like "the proxy is off but
proxy_url is still there", where the data no longer tells you the actual behaviour, and where
turning it back on would restore values the user believed they had cleared. Off means
cleared, one meaning only. The cost is retyping after an off/on cycle — the right direction
for this trade-off.
Removal is recursive: a closed group may contain an object_list, whose entries may
contain further object_lists, any level of which may hold secrets. Every $id path under
that subtree must have its secrets cleared too — implemented as a batch delete by secret-key
prefix. This is exactly what the flat secrets container buys; a nested mirror of the value tree
would need the tree walk written out a second time here.
No separate mechanism. Use one selector field plus several groups carrying visible_when:
{ "key": "auth", "mode": "always", "fields": [
{ "name": "auth_type", "type": "enum", "options": [
{"value":"oauth","label":"OAuth"}, {"value":"sigv4","label":"AWS SigV4"} ] } ] },
{ "key": "auth_oauth", "mode": "always",
"visible_when": { "field": "auth_type", "equals": "oauth" }, "fields": [ … ] },
{ "key": "auth_sigv4", "mode": "always",
"visible_when": { "field": "auth_type", "equals": "sigv4" }, "fields": [ … ] }Deliberately a small, complete set — enough for real requirements, but not an expression language.
Leaves (compared against a constant):
{ "field": "ssl_mode", "equals": "verify-ca" }
{ "field": "ssl_mode", "in": ["verify-ca", "verify-full"] }
{ "field": "ssl_mode", "not_in": ["disable"] }
{ "field": "use_proxy", "is_true": true }
{ "field": "service_name", "is_empty": true }Leaves (compared against another field) — required by general-purpose forms, absent from the original design:
{ "field": "password_confirm", "equals_field": "password" }
{ "field": "end_at", "gt_field": "start_at" }
{ "field": "max", "gte_field": "min" }Field references on both sides follow the scope rules of §5.2 (no prefix = same level, ^. =
one level out, $. = root).
Comparison is type-aware, driven by the field's Type rather than always comparing strings:
| Type | Compared as |
|---|---|
datetime |
Along the time axis (only the relevant part when format is date / time) |
integer / number / decimal |
Numerically (decimal exactly, never through float) |
string |
Lexicographically |
| Anything else | Ordering operators are rejected; Compile errors |
equals_field / not_equals_field work on every scalar type.
Combinators: { "all_of": [ … ] }, { "any_of": [ … ] }, { "not": { … } }
The Go side has matching constructors:
dyfields.Equals("ssl_mode", "verify-ca")
dyfields.In("ssl_mode", "verify-ca", "verify-full")
dyfields.IsTrue("use_proxy")
dyfields.EqualsField("password_confirm", "password")
dyfields.GtField("end_at", "start_at")
dyfields.AllOf(a, b) // AnyOf / NotA hidden field's value counts as "not set", so condition chains converge predictably.
The first three slots control "render it?", "require it?", and "may it change?" — none of
them can say "filling it in that way is simply wrong". Confirm-password and start < end are
the two most common validations in general-purpose forms, hence valid_when:
type Rule struct {
When *Condition `json:"when"` // must hold, otherwise an error is reported
Reason string `json:"reason,omitempty"` // default message
Code string `json:"code,omitempty"` // key for i18n
}{ "name": "password_confirm", "type": "secret", "valid_when": [
{ "when": { "field": "password_confirm", "equals_field": "password" },
"code": "password_mismatch", "reason": "the two passwords do not match" } ] }Groups have valid_when too, for cross-field group constraints; the error is attached to the
group:
{ "key": "contact", "valid_when": [
{ "when": { "any_of": [
{ "field": "email", "is_empty": false },
{ "field": "phone", "is_empty": false } ] },
"code": "contact_required", "reason": "provide at least one of phone or email" } ] }"At least one of three" could be forced through
required_when(each field saying "required when the other two are empty"), but the error would land on all three fields at once, and the conditions are O(n²) handwritten. Group-levelvalid_whenis where this belongs.
Inside an object_list entry, valid_when sees only that entry's values (plus whatever
outer values the scope rules reach). The only cross-entry constraint is unique, and that is a
list-level declaration, not a condition.
The four slots differ as follows:
| Slot | When it does not hold |
|---|---|
visible_when |
Not rendered, not validated, not stored |
required_when |
Still shown, merely not required |
readonly_when |
Shown read-only; Apply() rejects a change to the value |
valid_when |
Validation fails, a FieldError is reported |
func (c *Compiled) Visible(doc ValueDocument) VisibleSet
func (c *Compiled) Validate(doc ValueDocument) (ValueDocument, FieldErrors)
func (c *Compiled) Apply(current ValueDocument, patch Patch) (ValueDocument, FieldErrors)
func (c *Compiled) Redact(doc ValueDocument) PublicDocument
// ImpactOf reports which paths would have their values removed if this patch were applied.
// A caller uses it to prompt for confirmation before submitting; see 11.5.
func (c *Compiled) ImpactOf(current ValueDocument, patch Patch) []stringA patch has to say three things about a secret: leave it, set it, clear it. But
ValueDocument.Secrets is a map[string]string, which can only say two — the empty string is
already taken by "leave it" (see §12), and no string is left to mean "clear it".
type Patch struct {
Values map[string]any `json:"values"`
Secrets map[string]*string `json:"secrets,omitempty"`
}The pointer makes JSON null (clear) and "" (leave) two different values at the type level.
Forcing ValueDocument into the role of a patch would mean inventing a sentinel string — and a
sentinel string will one day be somebody's real password.
Containers (list / map / object_list) express "at least N entries" with min_items and
never with required — the two would say the same thing, and when two mechanisms say the
same thing somebody will always be confused about which wins. Compile rejects required on a
container outright rather than quietly picking one.
readonly_when means "reject a change", and that needs a baseline. Validate(doc) receives
one document with no "before" to compare against, so it cannot tell.
Hence the rule: read-only checking happens only in Apply, never in Validate. This has to
be stated in the API documentation — otherwise a caller will assume running Validate blocked
tampering with read-only fields, and that is a security assumption.
The client does not submit a readonly field, so its value has only two sources: Default,
or the caller writing it in after Validate.
The library does not do computed fields — "amount due = ticket price × headcount" needs an
expression language, which §14 explicitly excludes. The correct use for such a field is: declare
it readonly, have the caller compute it and write it into the document, and let the client
merely display it. This is a boundary, not an omission.
Changing one field can turn an entire subtree invisible, taking its values and secrets with it.
Change a sink's type from kafka to http, for instance, and every mapping's partition_key
below it (visible_when: ^.type == kafka) becomes invalid at once.
This is the correct behaviour (invisible means non-existent), but the scale is not what a
user expects — one dropdown wipes out a whole level of data. The design does not change, but
this is a case where the caller must provide a UI warning, and that is what ImpactOf()
exists for. Confirming beforehand beats discovering afterwards that the data is gone.
Validate's order (the order is itself a contract, because it decides whether a hidden field's
default counts):
- Compute visibility from the current values (group toggle → group
visible_when→ fieldvisible_when) - Apply
Defaultto visible fields with no value supplied - If applying defaults changed visibility, go back to 1 (same-level
visible_whenis acyclic, so this converges; bounded by the field count of the scope) - Only after convergence, remove the values of every invisible field
Removal has to wait for convergence (the original ordering removed at 2 and applied defaults at 3, which was wrong). In the first pass a default not yet applied makes the field read as "no value", so anything conditioned on it is judged invisible — and if you delete right then, by the time the next pass finds it should have been visible, the value is gone.
The starkest example is a toggle with
default: true: the document holds the whole group's values but omits the toggle key, so pass one reads the toggle asfalse→ the group is cleared → only then istrueapplied → pass two finds the group "required but empty". The user's submitted data was eaten by validation itself.The fix: the loop only ever adds (applies defaults), never subtracts; visibility iterates a table to a fixed point; and while evaluating conditions, any field already judged invisible reads as empty — so the document the loop sees is the same one that results after clearing. Once converged, the invisible values are deleted for real.
- Per field, check type,
required/required_when,format,Pattern, length and range,items - On an
object_list: recurse steps 1–7 for each entry (outer conditions read the parent scope), then checkmin_items/max_items/unique - Evaluate
valid_whenfor visible fields and groups (cross-field, so it must follow step 5) - Any key not declared in the schema is reported as an error (
$id/$deletedexcepted). This check stops at the boundary of ajsonvalue — otherwise thejsonescape hatch could never pass validation, which is the same as not having it. Insidejson, the only check is "is a valid JSON value" - Once everything passes, remove the values of
Transientfields (not removed on failure, see §4.3)
Point 8 matches Parse's attitude: unknown keys are rejected, not ignored. Silent ignoring
makes a mistyped setting look like it took effect.
type FieldError struct {
Path string `json:"path"` // brokers[2].host / contact (group level)
Field string `json:"field,omitempty"` // leaf name, host
Group string `json:"group,omitempty"`
ItemPath string `json:"item_path,omitempty"` // $id path: sinks.s_k1.mappings.m_3
Key string `json:"key,omitempty"` // which key of a map
Code string `json:"code"` // required|type|format|pattern|range|length|
// min_items|max_items|duplicate|unknown_field|readonly|custom
Reason string `json:"reason"`
}
type FieldErrors []FieldError // implements error; Len()==0 means it passedPath is essential: reporting only host for "the third broker's host is invalid" leaves a
user staring at a ten-row table reading "format error" with no idea which row.
Every error is reported at once, not returned at the first one. Let the user fix everything in one round instead of fixing one and trying again.
Apply is patch semantics, not a wholesale replace. The patch type is Patch rather than
ValueDocument; see §11.0 for why:
| Case | Behaviour |
|---|---|
| Key absent from the patch | Keep the current value |
| Key carries a new value | Overwrite |
Key carries null |
Delete |
list / map carries a new value |
Replaced wholesale, no element-level merge |
object_list carries a new value |
Aligned by $id, see below |
A secret carries "" |
Keep the current value (not clear it) |
A field's readonly_when holds and the value changed |
Report a readonly error |
| The owning group is switched off / hidden | Removed (see §9.1) |
Replacing list / map wholesale is deliberate: an element-level merge needs its own patch
syntax (insert at which position? delete which key?), which is a separate specification and not
worth complicating Apply for.
object_list cannot be replaced wholesale, because secrets inside an entry are never read
back — the client simply cannot assemble a complete package. So entries are aligned by $id:
| An entry in the patch | Behaviour |
|---|---|
Its $id exists in current |
Recursively apply patch semantics to that entry (unmentioned fields keep their old values, an empty secret is kept) |
Its $id does not exist / no $id given |
Treated as a new entry (a missing $id is generated; a missing $id on an entry holding secrets is an error) |
{"$id": "…", "$deleted": true} |
Delete that entry, recursively including every secret in its subtree |
A $id present in current but absent from the patch |
Kept (consistent with scalars) |
Why deletion must be explicit: an earlier version specified "absent from the patch means deleted". But the most natural dirty-tracking a client framework offers is to send only the entries that changed — which would silently delete every other entry along with its secrets. The user renames one field and the entire mapping table disappears.
Semantics When the client gets it wrong Severity Absence means delete Unsent entries silently deleted, data lost Fatal, irreversible Absence means keep, plus $deletedForgot the marker → something that should have been deleted was not The user sees it immediately The cost is a tombstone list on the client. That cost lands on "a few more lines of state management" rather than on "the user's data is gone".
The empty-string rule for secrets accommodates the client: a secret field cannot be read back,
so the client shows a "already set" placeholder and submits an empty string when the user leaves
it alone. If an empty string meant clear, users would wipe their own passwords on every save. To
actually clear a secret, send null.
Field.NameandGroup.Keyare contracts once published and must never be renamed- Compatible changes: adding optional fields, adding groups, adding enum options, relaxing validation
- Breaking changes: removing fields, changing
type, making an optional field required, tightening validation, renaming - A breaking change bumps
schema_version; the migration path is the caller's responsibility (the library performs no automatic migration)
i18n comes in two situations that need two mechanisms:
| Situation | Where the schema comes from | Approach |
|---|---|---|
| Developer-authored form | Code | label is a fallback; the client uses Name / Key as translation keys |
| User-built form | A database | Name is machine-generated (q_7f3a) with no translation table to look up → carry the languages directly in label_i18n |
Validation messages uniformly use FieldError.Code as the translation key.
- HTTP / storage / UI / file transfer. Input and output are Go values and
[]byte. The original's dynamic options (options_source/OptionsResolver) require network calls and stay out of the core; if wanted, a separatedyfields/httpxsub-package can carry them while the core stays dependency-free. - Cross-entry references (a field in entry 2 referencing entry 1). This is the one thing
object_listexcludes, and it is what flattens scope into something simple. The only cross-entry constraint offered isunique. - Outer referencing inner. The inner level has N entries and "which one?" has no answer. To
constrain the inner level, constrain the
object_listitself (min_items/unique). - Groups inside an entry. Partition with
visible_when;toggleable's "off means cleared" has no clear meaning within a single entry. - Cascading selects (country → province → city). Needs dynamic options, already excluded.
- Display-only elements (headings, dividers, explanatory blocks). They do not get mixed into
Field, or the validation logic would have to check "this one has no value" everywhere. - Cross-schema references. Conditions resolve only within their own schema.
- A full expression language. The set in §10 is deliberately closed.
- Computed fields ("amount = unit price × headcount"). Needs an expression language. Use a
readonlyfield plus the caller writing the value in; see §11.2. - Automatic schema migration. The version is only a marker; how to migrate is the caller's business.
- Concurrent merging. Optimistic locking for two people editing one document belongs to the
caller.
$idmakes an entry-by-entry merge technically possible, but the conflict-resolution strategy is not this library's to decide.
| Step | Contents | Delivers |
|---|---|---|
| 1 | Types + JSON round-trip (Schema / Group / Field / Items / Condition / Rule, Parse) |
Goal 2 |
| 2 | Compile + SchemaError (including scope and reference-direction checks) |
Definition errors become locatable and displayable |
| 3 | Visible + Validate (single level) + FieldErrors |
The skeleton of goal 3 |
| 4 | object_list: recursive validation, $id, unique, path-accurate errors |
Goal 3 |
| 5 | Builder / GroupBuilder / DSL / FieldAt / Extend / FromSchema |
Goal 1 |
| 6 | Apply (with $id alignment) + Redact |
Secret and update semantics |
Steps 1–4 come first, because the Builder is only a convenient way to produce a Schema;
wrapping a DSL around it after the shape of Schema and the validation semantics have settled
avoids reworking the interface repeatedly.
But the scope syntax and $id had to be settled on day one, even though they are only
implemented at step 4 — they are baked into the serialized format and the shape of secret keys,
so changing them later is a breaking change that also touches stored data.
Types can ship in batches: batch 1 string / integer / number / boolean / enum /
secret / json; batch 2 list / map / datetime / object_list; batch 3 decimal /
file.
Two scenario documents serve as acceptance targets for the implementation:
| Document | Covers | Acceptance |
|---|---|---|
example-walkthrough.md |
Registration form: schema / payload / UI three-way correspondence | That payload must pass Validate untouched |
example-pipeline.md |
Data pipeline: two levels of nesting, ^. scope, the Apply editing path |
Applying that patch must produce the result the document lists |
Six lines the tests must cover: deep round-trip equality (nesting included); one case for each
of Compile's rejection reasons; Validate's behaviour for "hidden fields are neither
validated nor stored" and "every error reported at once"; per-entry visibility and path
location for object_list; secrets staying aligned after Apply deletes a middle entry
(the only reason $id exists — without this test, it may as well not); and entries the patch
does not mention are not deleted (the reason $deleted exists, see §12).