Add EEP for native records

[bjorng]

No description provided.

[josevalim]

Thank you @bjorng and @garazdawi! The proposal is well written and building on top of records does a beautiful job of keeping the language changes minimal. Removing the undefined as default for fields is another great change, as well as having the optional of having "required fields" (which must be given on creation).

My only criticism to the proposal is that it doesn't discuss maps at all. For the last ~10 years, we have been using maps as replacements for records, and while native records aim to ease the migration between "records -> native records", there are no paths between "maps -> native records". Perhaps this is intentional, you are not expecting anyone to migrate between maps and native records, but then we have to deal with the side-effect that, as native records improve, a lot of code (specially in Elixir) will be forever suboptimal.

If I am allowed to spitball a bit, I'd love if "native records" were implemented as "named maps" or "record maps" behind the scenes. The proposal would stay roughly the same, the only differences are that:

• Accessing a field in any record could use the existing map syntax and just work:

  Expr#Field

• Updating any record could use the existing map syntax and just work:

  Expr#{Field1=Expr1, ..., FieldN=ExprN}

You could also support the proposed #_{...} syntax and it would additionally check it is a record and not a "plain map". is_map/1 would obviously return true for records but you could do a more precise check with is_record.

Regarding the key-destructive map operations, such as maps:put/3 or maps:remove/2, I'd make them fail unless you "untag" the map (or you could allow them to succeed but the tag would be removed at the end, which I find too subtle).

Overall, the goal would be to effectively unify records and maps, removing the decade-old question "records or maps". This would also provide a path for Erlang and Elixir to unify behind the same constructs, so I'd love to hear your opinions.

[josevalim]

Perhaps worth adding more context to the errors, such as {badrecord, {default_missing, key}}. I know format_error can be used (and have additional side-band information attached), but there may be a benefits in being upfront about it too?

This would also allow distinguishing from other errors below, such as {badrecord, not_found}, etc.

[essen]

More context definitely better, even if it gives only one missing key out of several.

[lpil]

Yes please! I think more fine-grained errors would would be greatly beneficial for newcomers especially.

[bjorng]

I've changed the error reason to {novalue,Field} (where novalue is a new error reason).

[dgud]

Accessing a field in any record could use the existing map syntax and just work:
Expr#Field

In erlang there is no existing map syntax for accessing a single field today,
that is still missing, was never implemented I guess because maps can have arbitrary terms as keys.

I guess it could be added for "literal" fields?
And it is really annoying to have to use the add record_name when accessing fields in records today, e.g. Rec#record_name.field

[josevalim]

@dgud yes! I keep forgetting that it was part of EEP but not implemented.

And it is really annoying to have to use the add record_name when accessing fields in records today, e.g. Rec#record_name.field

I am assuming that, as long as you pattern match on the named record when the variable is defined, the compiler would be able to optimize "unamed" field access and updates?

[bjorng]

I am assuming that, as long as you pattern match on the named record when the variable is defined, the compiler would be able to optimize "unamed" field access and updates?

Yes, that should be possible.

[richcarl]

See the discussion in erlang/otp#9174 on a possible x[y] notation - @michalmuskala suggested M#[Field].

[essen]

Great work!

Would it make sense to have finer grained control on who can do what? For example restrict creation to the defining module while still providing access to fields; or read-only fields outside the defining module. Probably doesn't matter for 95% of use cases I reckon.

[essen]

More context definitely better, even if it gives only one missing key out of several.

[essen]

This creates a new use case because we could have two records v1 and v2 that share some fields, and use this syntax to access the common fields. It probably needs to be highlighted.

[bjorng]

there are no paths between "maps -> native records". Perhaps this is intentional

We didn't really think about migrations from maps. Your suggestion seems reasonable. We will discuss this in the OTP team.

[lpil]

What a cool EEP! Thank you! I've a handful of questions

Language interop

One aspect of this which the document does not touch on that I think could be highly impactful for the BEAM ecosystem is language interop. Today each major language has a different preference for fixed key-value data structures:

• Erlang: maps and records
• Elixir: maps with a special field containing a module atom
• Gleam: records

This creates some degree of friction when calling code from other BEAM languages. If they all were to largely use native records then this friction go away, making interop between languages would be a much better experience.

I'm not immediately seeing any problems for Gleam as we use records there, but it seems like it would be more challenging for Elixir there we use maps.

Adoption within existing OTP modules

It seems that in an ideal world that native records would be the ubiquitous data structure, once they are available. Would existing Erlang/OTP modules be updated to work with them?

Functions that expect classic records, tagged tuples, and maps could have new function clauses added to handle native records in a backwards compatible way, unless I am mistaken. It seems that due to not being compatible with maps or tuples there would be very little ability to update existing functions to return native records.

Is there something we could do here? Or is the expectation that Erlang/OTP code will use different data structures depending on how old it is?

Construction syntax

Is the only difference between the native and classic record syntaxes the # character in the name of the definition? This seems like it will be very error prone, and also hard for less familiar people to debug as the definition will be accepted by the Erlang compiler, but their attempts to construct the record will fail.

---

Thank you all!

[lpil]

Yes please! I think more fine-grained errors would would be greatly beneficial for newcomers especially.

[lpil]

Is there a performance difference when using this syntax compared to using the non-anonymous syntax?

Are there situations in which one cannot use the anonymous syntax?

[bjorng]

Is there a performance difference when using this syntax compared to using the non-anonymous syntax?

Yes, certain optimizations we are thinking about implementing will not be applied when using anonymous syntax.

Are there situations in which one cannot use the anonymous syntax?

No.

[lpil]

Tuple records can be constructed with field names (using the #blah{a=1,b=2} syntax) or without field names (using the {blah, 1, 2}) syntax. Do native records have a field-nameless #blah{1, 2} construction syntax?

[bjorng]

No.

The nameless syntax for tuple records only works because traditional records are implemented using tuples.

[lpil]

Is the expectation that native records will have inferior performance to tuple records?

[bjorng]

We don't know yet, and it also depends on exactly which operations we are talking about. For example, updating multiple elements in a small map (no more than 32 elements) is quite efficient because it can be done in one pass, and similarly matching out multiple elements is also efficient because it can be done in one pass. Accessing one element at the time is more expensive for a map than for a tuple record.

In the first implementation to be released, native records will be implemented similar to how small maps are implemented, with similar performance. We have an idea for an optimization that would make it faster than maps.

[bjorng]

Functions that expect classic records, tagged tuples, and maps could have new function clauses added to handle native records in a backwards compatible way, unless I am mistaken. It seems that due to not being compatible with maps or tuples there would be very little ability to update existing functions to return native records.

Yes. If a function returns a tuple record, all we can do is to create a new function that returns a native record.

Or is the expectation that Erlang/OTP code will use different data structures depending on how old it is?

To some extent, yes. I think that is already the case.

If they all were to largely use native records then this friction go away, making interop between languages would be a much better experience.

Agreed.

[lpil]

Thank you @bjorng.

To confirm: all fields must have names? There are no positional fields?

Can one define a record which does not have any fields?

-record #none{}.
-record #some{value = term()}.
-type option() :: #none{} | #some{}

[bjorng]

To confirm: all fields must have names?

Yes.

Can one define a record which does not have any fields?

Yes.

[dgud]

Great work!

Would it make sense to have finer grained control on who can do what? For example restrict creation to the defining module while still providing access to fields; or read-only fields outside the defining module. Probably doesn't matter for 95% of use cases I reckon.

We are trying to limit the scope here to try to get it in to 29.

Hmm, this would require some additional syntax, 'private' | 'protected' | 'public' (borrowing from ets access types), personally
I don't think this is necessary nor wanted, Opaqueness on fields have been up for discussion but dropped for now at, implementation and reflection reasons.

[RaimoNiskanen]

@lpil wrote:

Is the only difference between the native and classic record syntaxes the # character in the name of the definition? This seems like it will be very error prone, ...

It is a different format, not only the # character, it is like record creation:

-record(foo, {a, b, c}).
%% vs.
-record #foo{a, b, c}.

So there is also a comma and parentheses that are different.

[tsloughter]

I don't see the concern raised that we'll now have record, native records and maps. I know they all serve different purposes, but the differences are slight and new users definitely get confused about which to use. I don't think it should be a blocker to adding a new data structure that can help improve devex or performance when choosing the right one but it does worry me that this can't replace records.

Related to not replacing records, I think -record being overloaded is confusing. To be clear, I don't like the idea of -native_record either.

[essen]

Hmm, this would require some additional syntax, 'private' | 'protected' | 'public' (borrowing from ets access types), personally I don't think this is necessary nor wanted, Opaqueness on fields have been up for discussion but dropped for now at, implementation and reflection reasons.

Right it's something that likely doesn't have to be done at runtime, but it is important to consider at least in the documentation part, as internal fields definitely shouldn't be documented. Read-only fields can be marked as such easily in the text. But this depends on what the documentation will look like I suppose.

[bjorng]

Update: there can now be two distinct errors when Rec#rec.field fails: {badrecord,Term} or {badfield,field}.

[RaimoNiskanen]

@tsloughter wrote:

Related to not replacing records, I think -record being overloaded is confusing. ...

"Not replacing records" is really phrased: not "Replacing all tuple record usage scenarios.", and "Replace most tuple-record usages without having to update anything but the declaration.".

With that in mind I think overloading -record may be more good than bad.

[tsloughter]

@RaimoNiskanen unless it outright replaces, 100%, named tuple records I think using -record will cause additional confusion to the confusion that will already exist from there being 2 types of records and maps.

I take it there are a million reasons -type shouldn't and can't also define a record:

-type #pair(A, B) :: #pair{
                           first :: A,
                           second :: B
                          }.

At first I wanted #{} but that is already used by maps!

I can concede there is no good alternative to -record... except maybe -frame (I kid, I kid). But making the times that tuple records are needed be as small as possible may be important though, to not have Erlang grow its reputation of confusing. Probably ets usage is the biggest one there.

[lpil]

I do share the concern that the similar syntax is confusing, and the difference being 2 characters of punctuation makes it challenging to differentiate between the two when reviewing code.

I can concede there is no good alternative to -record... except maybe -frame

struct was the one option that came to mind.

-struct #state{
  values = [] :: list(number()),
  avg = 0.0 :: float()
}.

Elixir does already have "defstruct", though that construct does seem very similar to this proposal in design and purpose.

[potatosalad]

Question: Behavior when adding a field across distributed nodes

Consider this scenario:

Node A (old code):

-record #state{count = 0, name = "default"}.
State = #state{count = 5, name = "server1"}

Node B (new code):

-record #state{count = 0, name = "default", version = 1}.

When Node A sends a State record to Node B:

1. Reading works fine: Node B can read State#state.count and State#state.name since those fields exist in the record value.
2. Reading the new field fails: State#state.version will raise {badfield, version} because the field doesn't exist in the record value (it was created with the old definition).
3. Pattern matching is unclear: Can Node B do #state{version = V} = State? Based on the spec: "Pattern matching fails if the pattern references a FieldK and the native-record value does not contain this field." This would fail.
4. Updating appears problematic: Can Node B do State#state{version = 1} to add the missing field? The EEP states: "A native-record value is updated according to its native-record definition" and "An update operation fails with a {badfield,FN} error if the native-record update expression references the field FN which is not
   defined (in the structure definition)."

Issue: This seems to check against the definition on Node B, not the fields in the value from Node A. It's unclear whether the update would:

• Succeed and add the version field to the record value
• Fail because version doesn't exist in the value
• Create a new record with all three fields (losing the old value's identity)

[potatosalad]

Question: Field renaming across nodes

Consider this scenario where a field is renamed:

Node A (old code):

-record #user{id, username, city}.
User = #user{id = 1, username = "alice", city = "Stockholm"}

Node B (new code - username renamed to name):

-record #user{id, name, city}.

When Node A sends User to Node B:

1. The record value still contains username: The field names are captured when the record is created, so the value has fields [id, username, city].
2. Node B cannot read the new field: User#user.name raises {badfield, name} because the value doesn't have a name field.
3. Node B CAN still read the old field: User#user.username should work because field access doesn't consult the definition—only the value. But is this problematic
   because username isn't in Node B's definition?
4. Pattern matching breaks: case User of #user{name = N} -> N end fails because name doesn't exist in the value.

Issue: Field renames appear to be breaking changes in distributed systems.

The EEP states:

to perform read operations on native-record values — accessing native-record fields and pattern matching over native-record values — the runtime does not consult the current native-record definition.

This means reading works purely based on the value's fields, not the definition. So:

• Old nodes can't read renamed fields (they use the old name)
• New nodes can't read renamed fields (the value has the old name, pattern matching fails)
• Updating is impossible (the definition has the new name, the value has the old name)

[potatosalad]

Question: Removing a field

Consider this scenario:

Node A (old code):

-record #config{host, port, legacy_timeout}.
Config = #config{host = "localhost", port = 8080, legacy_timeout = 5000}

Node B (new code - legacy_timeout removed):

-record #config{host, port}.

When Node A sends Config to Node B:

1. The removed field still exists in the value: The record value contains [host, port, legacy_timeout] because that's what Node A created.
2. Reading removed fields: Based on the spec, Config#config.legacy_timeout should still work on Node B because field access doesn't consult the definition—it only checks if the field exists in the value. This means:

• Code on Node B can accidentally read fields that "don't exist" in its definition
• Linters/dialyzer on Node B would flag legacy_timeout as an error, but it works at runtime
• Might this create a confusing situation for developers?

3. Pattern matching works: case Config of #config{host = H, port = P} -> ... end works fine (only matching fields that exist in both value and definition).

Issue: The interaction between field access (which ignores definition) and pattern matching (which may or may not check definition) is unclear.

[richcarl]

I really don't like the idea of allowing free #r or #m:r as in is_record(X, #r), because those fragments do not have any semantic meaning on their own, but they will have to be handled as something that may occur in any expression position. I think the rationale further below is wrong in this case: it's much worse to introduce pseudo-expressions that have no defined meaning outside a specific context. And you won't be using that syntax for the API functions records:create(Module, Name, ....) etc. anyway.

[bjorng]

I really don't like the idea of allowing free #r or #m:r as in is_record(X, #r)

It turns out you are not alone in don't liking it. After some discussions, we have decided to abandon that syntax. Instead, is_record/3 will be overloaded to test for a record in another module.

[richcarl]

Something like is_subtype or is_compatible, maybe.

[josevalim]

I think subtype and compatible would make me think the type of the fields are the same and are also checked. Perhaps has_record_fields but there are no guards starting with has_. I am not sure how to phrase it using is_.

[richcarl]

What is the motivation for this matching on non exported records by name?

[dgud]

To have the possibility to use them in an api as socket for example where you want to be able to match it but not the content. Today with opauge types, you have to wrap them in a two tuple {socket, OpagueStaff}.

[bjorng]

It turns out that I had failed to fully update the EEP regarding when a record definition is consulted (or not). In our internal meetings we had decided that the definition for a native record is only used when creating a record. All update and read operations will only refer to the value of record (which includes the names, whether it was exported at value-creation time, the name of the record, and of course all values).

I've now pushed another commit to hopefully make that clearer.

[michalmuskala]

If the definition is never consulted on update, how do you upgrade values to the new definition? Do you have to explicitly deconstruct & reconstruct and keep version of the code doing that for all possible values that are live in the system?
With the previous design, upgrading a value (given all new fields have a default) was potentially as simple as OldValue#record{}, and could handle all sorts of versions.
AFAIK the intention of the previous design was to define update in terms of creation + read from the existing value for fields that weren't explicitly provided

[bjorng]

If the definition is never consulted on update, how do you upgrade values to the new definition? Do you have to explicitly deconstruct & reconstruct and keep version of the code doing that for all possible values that are live in the system?

Yes. Upgrades must be done explicitly.

With the previous design, upgrading a value (given all new fields have a default) was potentially as simple as OldValue#record{}, and could handle all sorts of versions.

While the design was simple, we discovered that the actual implementation was far from simple, especially when more than one node is involved (or even when loading from the disk native records written by a previous instance of the runtime system). Also, we are not sure that is easy for the user to understand and handle all the implications of automatic upgrades of native records.