[storage] Draft for Envelope v2 #3853
Conversation
muhamadazmy
force-pushed
the
pr3853
branch
4 times, most recently
from
75eea1e to
7464a02
Compare
|
@tillrohrmann in case you want to take a look the interesting parts are in the v2.rs file in this PR |
crates/types/src/message.rs
Outdated
| } | ||
|
|
||
| #[derive(Debug, Clone, Copy, bilrost::Message)] | ||
| pub struct MessageIndexRecrod { |
| { | ||
| $crate::storage::decode::decode_serde(buf, kind).map_err(|err| { | ||
| ::tracing::error!(%err, "{} decode failure (decoding {})", kind, stringify!($name)); | ||
| $crate::storage::tracing::error!(%err, "{} decode failure (decoding {})", kind, stringify!($name)); |
There was a problem hiding this comment.
not quite sure why is change needed.
There was a problem hiding this comment.
Using this macro in a crate that doesn't have tracing as a dependency will fail. I had this issue when i was trying to use the macro in the wal-protocol crate which does not depend on tracing.
So I had 2 options:
- add
tracingas a dependency to the crate. Only for the sake of this macro. The dependency will not be "used" in the crate anywhere else. - use a better macro hygiene and use tracing from the declaring crate. Which what is done here.
This way users of the macros will not need to pull extra dependencies (tracing in this case)
| &self, | ||
| buf: &mut ::bytes::BytesMut, | ||
| ) -> Result<(), $crate::storage::StorageEncodeError> { | ||
| bytes::BufMut::put(buf, $crate::storage::encode::encode_bilrost(self)); |
There was a problem hiding this comment.
This potentially is better served if we pass the BufMut implementation to the encoder, perhaps by adding another variant (encode_bilrost_buf(buf)). Albeit, it's hard to tell which approach would yield better performance.
There was a problem hiding this comment.
Not all bilrost encode variants (for example encode_contiguous which is used here) can make use of the BufMut. But we can pass it anyway and leave the implementation of the encode_bilrost function to decide what to do.
crates/types/src/message.rs
Outdated
| pub index: MessageIndex, | ||
| } | ||
|
|
||
| bilrost_storage_encode_decode!(MessageIndexRecrod); |
There was a problem hiding this comment.
Perhaps the type should be named after the wal command (i.e. TruncateOutboxRequest) which btw needs to contain the PartitionId.
There was a problem hiding this comment.
Agree! As for the partition-id this record message was mapped 1:1 from v1 without much thought about the payload it carries. I think the reason it doesn't have the partition id is that that's part of the envelope Header (destination)
| #[bilrost(3)] | ||
| dedup: Dedup, | ||
| #[bilrost(4)] | ||
| keys: Keys, |
There was a problem hiding this comment.
I wonder if this is actually needed. On the read path, we ship the keys directly from bifrost (one layer up from the envelope). On the write-path, the appender needs the input record to implement HasRecordKeys which could be implemented on a wrapper type used as input to the appender. Essentially making the input type different than output type to save a few bytes on each record.
crates/wal-protocol/src/v2.rs
Outdated
| impl WithPartitionKey for Header { | ||
| fn partition_key(&self) -> PartitionKey { | ||
| match self.destination { | ||
| Destination::None => unimplemented!("expect destinationt to be set"), |
| #[bilrost(1)] | ||
| source: Source, | ||
| #[bilrost(2)] | ||
| destination: Destination, |
There was a problem hiding this comment.
destination is potentially unnecessary unless you see a clear use-case for it.
crates/wal-protocol/src/v2.rs
Outdated
| { | ||
| match kind { | ||
| StorageCodecKind::FlexbuffersSerde => { | ||
| todo!("implement loading from envelop V1") |
There was a problem hiding this comment.
| todo!("implement loading from envelop V1") | |
| todo!("implement loading from envelope V1") |
| record! { | ||
| @name=PurgeJournal, | ||
| @kind=RecordKind::PurgeJournal, | ||
| @payload=PurgeInvocationRequest |
There was a problem hiding this comment.
is it true that it has the exact same payload as PurgeInvocation?
There was a problem hiding this comment.
That's what v1 does. Yes.
# Summary - Separate header, keys, and other envelope related information from the actual payload of the envelope - Ability to decode envelope header without decoding the full envelope - Support different encoding for the payload (records) (flexbuffers, or bilrost) - Delay decoding of payload until needed # TODO: - [ ] Decode v1 envelope into v2 - [ ] Use Envelope v2 in code
2 participants
[storage] Draft for Envelope v2
Summary
information from the actual payload of the envelope
TODO:
Stack created with Sapling. Best reviewed with ReviewStack.