[Feature] Support TTL on Short Term Memory

[avx-wlauer]

Search before asking

• I searched in the issues and found nothing similar.

Description

Support controlling TTL on Short Term memory.

Short term memory is backed by Flink's standard MapState facility. Unlike Sensory memory, Short term memory is supposed to last beyond the current call and be available for future operations on the key. The current implementaiton doesn't specify a TTL on the MapState that underlies the short term memory, but in order to control the size of retained state for a long running pipeline, specifying this is important. A new API should be added that allows specifying a StateTtlConfig for the memory, which would get passed down and used when creating the underlying MapState.

Sensory memory doesn't need this facility, as it isn't supposed to last beyond the current call. And Long term memory is handled differently. It also likely needs a way to specify TTL, but the mechanism should be completely different due to the architecture.

Are you willing to submit a PR?

• I'm willing to submit a PR!

[wenjin272]

Thanks for offering the contribution, @avx-wlauer. For short-term memory, TTL is indeed crucial. I've assigned you to the ticket.