fix(enrichment tables): preserve memory enrichment table state on reload by esensar · Pull Request #25547 · vectordotdev/vector

esensar · 2026-06-01T12:51:24Z

Summary

While working on #25143, it was brought to my attention that reload was not handled for memory tables (#25143 (comment)) - that is, new components were generated, that were not attached to the tables that were queried. This PR resolves that by making these tables take over the state of previous components.

Vector configuration

enrichment_tables:
  memory_table:
    type: memory
    ttl: 60
    flush_interval: 5
    inputs: ["cache_generator"]


sources:
  demo_logs_test:
    type: "demo_logs"
    format: "json"

transforms:
  demo_logs_processor:
    type: "remap"
    inputs: ["demo_logs_test"]
    source: |
      . = parse_json!(.message)
      user_id = get!(., path: ["user-identifier"])

      existing, err = get_enrichment_table_record("memory_table", { "key": user_id })

      if err == null {
        . = existing.value
        .source = "cache"
      } else {
        .referer = parse_url!(.referer)
        .referer.host = encode_punycode!(.referer.host)
        .source = "transform"
      }      

  cache_generator:
    type: "remap"
    inputs: ["demo_logs_processor"]
    source: |
      existing, err = get_enrichment_table_record("memory_table", { "key": get!(., path: ["user-identifier"]) })
      if err != null {
        data = .
        . = set!(value: {}, path: [get!(data, path: ["user-identifier"])], data: data)
      } else {
        . = {}
      }      

sinks:
  console:
    inputs: ["demo_logs_processor"]
    target: "stdout"
    type: "console"
    encoding:
      codec: "json"

How did you test this PR?

Ran vector with the above configuration and the --watch-config flag. Changed TTL a couple of times and Vector properly reloaded and kept state, observed by seeing cached output data, instead of newly generated.

Change Type

Is this a breaking change?

Yes
No

Does this PR include user facing changes?

Yes. Please add a changelog fragment based on our guidelines.
No. A maintainer will apply the no-changelog label to this PR.

References

Notes

Please read our Vector contributor resources.
Do not hesitate to use @vectordotdev/vector to reach out to us regarding this PR.
Some CI checks run only after we manually approve them.
- We recommend adding a pre-push hook, please see this template.
- Alternatively, we recommend running the following locally before pushing to the remote branch:
  - make fmt
  - make check-clippy (if there are failures it's possible some of them can be fixed with make clippy-fix)
  - make test
After a review is requested, please avoid force pushes to help us review incrementally.
- Feel free to push as many commits as you want. They will be squashed into one before merging.
- For example, you can run git merge origin master and git push.
If this PR introduces changes Vector dependencies (modifies Cargo.lock), please
run make build-licenses to regenerate the license inventory and commit the changes (if any). More details on the dd-rust-license-tool.

Sponsored by Quad9

Related: vectordotdev#25143 Related: vectordotdev#25143 (comment)

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6583fd70e4

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

pront · 2026-06-01T17:56:45Z

Thanks @esensar! Per our new policy I will come back to this once codex comments are resolved.

esensar · 2026-06-02T13:58:54Z

While resolving this I found that there were quite a few typos and missed cases for reload in enrichment tables that have source/sink. I think I covered them all now.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0044cc7f03

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

…t types

pront

Thank you for this contribution. Can please add a couple of test cases, e.g. take_state_preserves_data and reload_preserves_state?

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1ed0732c66

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f4cb301f9e

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

esensar · 2026-06-08T13:25:01Z

I am adding a topology reload test to confirm this, but I am having some issues with it. For some reason tables are missing on reload, that is TableRegistry::finish_load doesn't store tables properly during the test (it works fine in runtime)

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 961e72fe3b

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

esensar · 2026-06-08T15:26:55Z

I am adding a topology reload test to confirm this, but I am having some issues with it. For some reason tables are missing on reload, that is TableRegistry::finish_load doesn't store tables properly during the test (it works fine in runtime)

I have added the test - it works when run individually, but fails when run together with others - I assume because ENRICHMENT_TABLES is static and shared between tests. Not sure how to best resolve this...

pront · 2026-06-12T19:42:53Z

I am adding a topology reload test to confirm this, but I am having some issues with it. For some reason tables are missing on reload, that is TableRegistry::finish_load doesn't store tables properly during the test (it works fine in runtime)

I have added the test - it works when run individually, but fails when run together with others - I assume because ENRICHMENT_TABLES is static and shared between tests. Not sure how to best resolve this...

@esensar no worries, I will take a closer look and make suggestions.

pront · 2026-06-16T13:55:49Z

Updated this branch with a fix. Can you try the tests now?

esensar · 2026-06-16T16:01:57Z

Updated this branch with a fix. Can you try the tests now?

Thanks, that fixed it!

pront

Edit:

This makes memory table state preservation across reload unconditional when the table key remains the same. I can imagine users who treat reload as a cache invalidation point after changing the table’s producing logic or cached value schema.

This is currently breaking the existing reload behavior. Thoughts on adding an explicit reload behavior config option? Or preserving the current behavior and making this opt-in?

esensar · 2026-06-17T08:16:26Z

Edit:

This makes memory table state preservation across reload unconditional when the table key remains the same. I can imagine users who treat reload as a cache invalidation point after changing the table’s producing logic or cached value schema.

This is currently breaking the existing reload behavior. Thoughts on adding an explicit reload behavior config option? Or preserving the current behavior and making this opt-in?

Okay, that makes sense. Maybe a flag for state preservation on reload, or like you said reload behavior with just these 2 options for now so that we can expand it in the future if some other interesting behavior comes up.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 12939a04e6

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector · 2026-06-17T11:06:01Z

+                .chain(to_change_table_sources.iter())
+            {
                debug!(component_id = %key, "Changing source.");

                self.remove_outputs(key);


Reattach consumers of changed table sources

When a memory enrichment table source is changed, this now removes and rebuilds that source output, but unchanged downstream transforms/sinks are never reattached because reattach_severed_inputs only looks at diff.sources.to_change and diff.transforms.to_change. If only the memory table config changes while a sink still reads from the same source_key, the reload succeeds but the sink is left disconnected from the rebuilt fanout, so subsequent exported records are dropped before reaching it.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-06-17T11:06:01Z

+                let mut prev_state = None;
+                if !self.diff.enrichment_tables.is_added(name)
+                    && table_outer.inner.wants_previous_state()
+                {
+                    prev_state = ENRICHMENT_TABLES.extract_state(&table_name);
+                }


Keep memory state when a reload is rolled back

Because load_enrichment_tables swaps ENRICHMENT_TABLES during the tentative build before healthchecks/build errors can roll the reload back, skipping extract_state for the default ClearState path can destroy the current memory table even when the new config is rejected. For example, a reload that changes a memory table and also has an invalid sink builds an empty table here, fails later, then restores the old config from that already-cleared registry, so accepted records disappear despite the failed reload.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-06-17T11:06:01Z

    fn needs_reload(&self) -> bool {
-        false
+        true


Rebuild the memory sink when forcing reloads

With the default ClearState behavior, returning true here makes every config reload rebuild the registry's memory table even when the table itself is unchanged, but diff.enrichment_tables does not mark the table sink changed in that case. After an unrelated reload, transforms read from the new empty table while the still-running memory sink keeps writing to the old instance, so all subsequent inserts become invisible until the table is explicitly changed or Vector restarts.

Useful? React with 👍 / 👎.

fix(enrichment): preserve memory enrichment table state on reload

6583fd7

Related: vectordotdev#25143 Related: vectordotdev#25143 (comment)

esensar requested a review from a team as a code owner June 1, 2026 12:51

github-actions Bot added the domain: topology Anything related to Vector's topology code label Jun 1, 2026

Add changelog entry

58b908e

esensar mentioned this pull request Jun 1, 2026

feat(enrichment tables): add cuckoo filter to memory table #25143

Open

9 tasks

chatgpt-codex-connector Bot reviewed Jun 1, 2026

View reviewed changes

Comment thread src/topology/running.rs Outdated

Comment thread src/topology/builder.rs Outdated

esensar changed the title ~~fix(enrichment): preserve memory enrichment table state on reload~~ fix(enrichment tables): preserve memory enrichment table state on reload Jun 2, 2026

esensar added 2 commits June 2, 2026 13:19

Restore old state when building the new table

04110a3

Fix reload handling for enrichment tables

0e0952c

Merge branch 'master' into fix/memory-table-reload-state

0044cc7

chatgpt-codex-connector Bot reviewed Jun 2, 2026

View reviewed changes

Comment thread src/topology/builder.rs Outdated

esensar added 3 commits June 3, 2026 06:01

Add additional types for enrichment tables which act as more componen…

a696cb7

…t types

Change stateful flag check with sink check

664d2ae

Merge branch 'master' into fix/memory-table-reload-state

bf3173f

pront reviewed Jun 4, 2026

View reviewed changes

Comment thread lib/vector-vrl/enrichment/src/lib.rs Outdated

Comment thread lib/vector-vrl/enrichment/src/lib.rs Outdated

Comment thread lib/vector-vrl/enrichment/src/lib.rs Outdated

esensar added 3 commits June 5, 2026 11:28

Remove unused enrichment error

d36e743

Simplify state extraction from tables

ed83146

Remove another unused error from enrichment

1ed0732

chatgpt-codex-connector Bot reviewed Jun 5, 2026

View reviewed changes

Comment thread lib/vector-vrl/enrichment/src/test_util.rs Outdated

esensar added 2 commits June 5, 2026 12:26

Add a test case for memory table state preservation

4ab2ce3

Remove obsolete trait methods from enrichment tests

f4cb301

chatgpt-codex-connector Bot reviewed Jun 5, 2026

View reviewed changes

Comment thread src/topology/builder.rs

Prevent needless clone when extracting state from enrichment tables

e9f9738

Merge branch 'master' into fix/memory-table-reload-state

961e72f

chatgpt-codex-connector Bot reviewed Jun 8, 2026

View reviewed changes

Comment thread src/enrichment_tables/memory/table.rs

Comment thread src/topology/running.rs

Add topology test for enrichment table state extraction

79eb2b1

esensar requested a review from pront June 8, 2026 15:27

Merge branch 'master' into fix/memory-table-reload-state

2fdd859

pront mentioned this pull request Jun 12, 2026

fix(topology): serialize enrichment table registry loads #25615

Merged

9 tasks

Merge branch 'master' into fix/memory-table-reload-state

b05daa4

pront approved these changes Jun 16, 2026

View reviewed changes

Add a configuration option for state preservation for memory table

12939a0

chatgpt-codex-connector Bot reviewed Jun 17, 2026

View reviewed changes

Conversation

esensar commented Jun 1, 2026

Summary

Vector configuration

How did you test this PR?

Change Type

Is this a breaking change?

Does this PR include user facing changes?

References

Notes

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

pront commented Jun 1, 2026

Uh oh!

esensar commented Jun 2, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

pront left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

esensar commented Jun 8, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

esensar commented Jun 8, 2026

Uh oh!

pront commented Jun 12, 2026

Uh oh!

pront commented Jun 16, 2026

Uh oh!

esensar commented Jun 16, 2026

Uh oh!

pront left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

esensar commented Jun 17, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Jun 17, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Jun 17, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Jun 17, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

pront left a comment •

edited

Loading