Proposal: Redo RecordConfig

[tlimoncelli]

Proposal: Redo RecordConfig

RecordConfig is the struct that stores everything about a DNS record: The label (foo.example.com), the rtype ("A" or "CNAME" or etc) and data ("1.2.3.4") but also metadata, the "blob" used to compare 2 records, and so on.

RecordConfig was created more than 10 years ago, when we didn't know as much
about building good Go code, and Go didn't have Generics. Oh, and miekg/dns
v2 didn't exist (v2 has support for non-standard types, unknown types, RDATA
structs, and more).

I propose we modernize RecordConfig to use less memory and be more CPU-efficient.
Luckily we can do this in a way that both old and new code can co-exist
during the transition. However, once we begin the change, we need to complete
it fully so we don't end up with 2 ways to do everything.

Problems to solve

1. RecordConfig uses a lot of memory. We store fields for all record
   types all the time. For example, the struct has fields for all the LOC
   and DS types, even if we're storing an "A" record. An "A" record
   is stored as an ASCII string ("1.2.3.4") instead of using a binary format
   (4 bytes).

2. Types are stored as string, which is slow. We store "CNAME"
   instead of 5 (the RFC code for CNAME). Therefore, the code is constantly
   doing slow string comparions instead of fast int comparisons.

3. Fields are named badly. We didn't know what we didn't know. If
   we were making a new system today, we'd use names like .Owner not .Name.
   We'd have more efficient ways of storing the IDN and Unicode version of strings.

4. There's little consistency about how to do things. Providers had
   to guess how to build RecordConfig data. We really should standardize
   on a few ways: create from a RFC1038-style string, create from a miekg/dns/rdata
   struct, or from an type + []any. (For now we'll call these NewRecFromString(),
   NewRecFromStruct(), and NewRecFromRaw(), i.e. the NewRec*() functions).

I suspect that large sites with thousands of DNS records would
see a performance improvement. Providers would be easier to write.

Proposal

1. Add the new fields to RecordConfig{} that we desire:

   1. The type represented as an uint16 (stored in .TypeNum)
   2. The fields represented as a rdata.RDATA interface (stored in .F)
   3. Retain TTL, NameRaw, NameIDN, NameUnicode; NameFQDN, NameFQDNUnicode, Original, Metadata, ZonefilePartial
   4. All other fields will be considered "legacy" and will be marked for depreciation

2. How to deal with legacy code?

Luckily we know if a RecordConfig is legacy with a simple comparison.
If .F == nil, it is legacy.

Dealing with record data downloaded from providers:

Existing providers generate RecordConfig records with .F == nil.

Once the DNS data is gathered, we can iterate over all records
and, if .F == nil, use the legacy fields to populate the .F field.

We will convert providers to use the MakeRec*() functions until the conversion
step is no longer needed.

Dealing with record data that comes from dnsconfig.js:

This is similar. The current code creates RecordConfig structs
with .F == nil except for the DS, RP and CLOUDFLAREAPI_SINGLE_REDIRECT.
We'll use ".F == nil" to determine if conversion is needed.
We'll migrate old code to use the new fields until the conversion is no longer needed.

What about custom types?

Custom types like CLOUDFLAREAPI_SINGLE_REDIRECT will be assigned a codepoint
out of the unassigned space and registered so that they work like official types.

What about unknown types?

Unknown record types aren't handled very well right now. We can do better,
but only after the conversion is complete.

Risks

The biggest risk is that we do half the conversion then run out of energy.
The code will be confusing to update because there will be two ways to do everything.

Implementation Plan

Some thoughts:

We can't convert everything all at once.

It's probably best to start with the end of the pipeline and work backwards.

What needs to be converted:

1. The "Populate .F from legacy fields" conversions.
2. Migrate the back-end features (the diff2 engine, "preview" and "push")
3. the "Populate legacy fields from .F" conversions (only needed for "DS")
4. Migrate individual providers
5. Migrate from dns v1 to dns v2.

Maybe instead of fixing .F, we should add a new field (DATA rdata.RDATA).
This gives us a clean break, and we don't have to deal with fixing .F.

Converting all the providers is difficult. The technical work is easy.
The hard part is the testing, especially since I don't have test
accounts on all providers. I could make PRs and ask maintainers
to do the testing, but what do we do for maintainers that don't respond?
Merge and pray?

Is this worth it?

Is this worth it?

Is this a needed fix or am I just optimizing something that is "good enough"? Is this inspired by a real need or just by the fact that miekg/dns v2 is shipping and I want to take advantage of the new features?

I've made 3-4 failed attempts at reworking how we manage records. I've learned a lot and identified a lot of cruft and tech debt that I want to get rid of. Does any of that matter?

Most of what I've learned is how to do this all incrementally rather than start a Netscape--esque "second version syndrome" project. Knowing how to do something does not mean that the thing should be done.