Generate meaningful segment ids using UUIDv7

Closes #971.

Segment ids were generated as random UUIDv4. Those 16 bytes carried no
information: they were not sortable and told you nothing about when a
segment was created.

This switches non-test segment id generation to UUIDv7 (RFC 9562), which
embeds a 48-bit millisecond creation timestamp in the most significant
bits followed by random bits. The result is still universally unique and
still lock-free to generate from any indexing thread (the properties the
original UUIDv4 was chosen for), while additionally being:

- chronologically sortable: `SegmentId`'s `Ord` is a byte comparison, so
  ids now sort by creation time.
- self-describing: the new `SegmentId::creation_time()` recovers the
  embedded timestamp for debugging.

This is fully backward compatible. The on-disk format is unchanged (a
32-char hex uuid). Ids from older indices are UUIDv4 and keep working;
`creation_time()` returns `None` for them (and for the autoincrement ids
used in tests).

Note: this deliberately does not encode a hostname/label or merge origin
into the id. As discussed in #971, that information is a better fit for
segment attributes on `SegmentMeta` (#999) than for the id itself, and
avoids the unresolved question of how labels should be combined on merge.

Author: Divyesh-k

CHANGELOG.md

@@ -38,6 +38,7 @@ Tantivy 0.26 (Unreleased)
 - Make `Language` hashable [#2763](https://github.com/quickwit-oss/tantivy/pull/2763)(@philippemnoel)
 - Improve `space_usage` reporting for JSON fields and columnar data [#2761](https://github.com/quickwit-oss/tantivy/pull/2761)(@PSeitz-dd)
 - Split `Term` into `Term` and `IndexingTerm` [#2744](https://github.com/quickwit-oss/tantivy/pull/2744) [#2750](https://github.com/quickwit-oss/tantivy/pull/2750)(@PSeitz-dd @PSeitz)
+- Generate segment ids as time-ordered UUIDv7, making them chronologically sortable and exposing `SegmentId::creation_time` [#971](https://github.com/quickwit-oss/tantivy/issues/971)(@Divyesh-k)
 
 ## Performance
 - **Aggregation**

Cargo.toml

@@ -35,7 +35,7 @@ serde = { version = "1.0.219", features = ["derive"] }
 serde_json = "1.0.140"
 fs4 = { version = "0.13.1", optional = true }
 levenshtein_automata = "0.2.1"
-uuid = { version = "1.0.0", features = ["v4", "serde"] }
+uuid = { version = "1.0.0", features = ["v4", "v7", "serde"] }
 crossbeam-channel = "0.5.4"
 rust-stemmers = { version = "1.2.0", optional = true }
 downcast-rs = "2.0.1"

src/index/segment_id.rs

@@ -16,6 +16,13 @@ use uuid::Uuid;
 /// by a UUID which is used to prefix the filenames
 /// of all of the file associated with the segment.
 ///
+/// Segments created by tantivy use a UUIDv7, which embeds the segment's
+/// creation time in its most significant bits. As a result segment ids sort
+/// chronologically and the creation time can be recovered through
+/// [`SegmentId::creation_time`]. Ids read from older indices (created before
+/// this change) are UUIDv4 and remain fully supported; for those
+/// [`SegmentId::creation_time`] returns `None`.
+///
 /// In unit test, for reproducibility, the `SegmentId` are
 /// simply generated in an autoincrement fashion.
 #[derive(Clone, Copy, PartialEq, Eq, Hash, Serialize, Deserialize)]
@@ -40,7 +47,14 @@ fn create_uuid() -> Uuid {
 
 #[cfg(not(test))]
 fn create_uuid() -> Uuid {
-    Uuid::new_v4()
+    // UUIDv7 embeds a 48-bit millisecond creation timestamp in its most significant
+    // bits, followed by random bits. This keeps segment ids universally unique and
+    // lock-free to generate from any indexing thread (like the previous v4), while
+    // additionally making them:
+    //   - chronologically sortable (ids sort by creation time), and
+    //   - self-describing (the creation time can be recovered, see
+    //     `SegmentId::creation_time`).
+    Uuid::now_v7()
 }
 
 impl SegmentId {
@@ -74,6 +88,24 @@ impl SegmentId {
     pub fn from_uuid_string(uuid_string: &str) -> Result<SegmentId, SegmentIdParseError> {
         FromStr::from_str(uuid_string)
     }
+
+    /// Returns the creation time embedded in the segment id, if available.
+    ///
+    /// Segments created by recent versions of tantivy use a UUIDv7, whose most
+    /// significant bits encode the millisecond timestamp at which the id (and
+    /// hence the segment) was created. This can be handy when investigating an
+    /// index: it tells you when each segment was produced without relying on
+    /// filesystem timestamps.
+    ///
+    /// Returns `None` for segment ids that do not carry a timestamp, i.e. ids
+    /// from older indices (UUIDv4) and the autoincrement ids used in tests.
+    pub fn creation_time(&self) -> Option<std::time::SystemTime> {
+        // `get_timestamp` returns `Some` only for UUID versions that carry a
+        // timestamp (v7 here); it is `None` for v4.
+        let timestamp = self.0.get_timestamp()?;
+        let (secs, nanos) = timestamp.to_unix();
+        Some(std::time::UNIX_EPOCH + std::time::Duration::new(secs, u32::from(nanos)))
+    }
 }
 
 /// Error type used when parsing a `SegmentId` from a string fails.
@@ -139,4 +171,33 @@ mod tests {
         // one extra char
         assert!(SegmentId::from_uuid_string("a5c4dfcbdfe645089129e308e26d5523b").is_err());
     }
+
+    #[test]
+    fn test_creation_time_none_for_v4() {
+        // A legacy UUIDv4 id (version nibble `4`) carries no timestamp.
+        let v4 = SegmentId::from_uuid_string("a5c4dfcbdfe645089129e308e26d5523").unwrap();
+        assert!(v4.creation_time().is_none());
+    }
+
+    #[test]
+    fn test_creation_time_some_for_v7() {
+        use std::time::{Duration, SystemTime};
+
+        use uuid::Uuid;
+
+        // Build a UUIDv7 directly so this test does not depend on the (test-only)
+        // autoincrement id generation used elsewhere.
+        let before = SystemTime::now();
+        let seg = SegmentId(Uuid::now_v7());
+        let after = SystemTime::now();
+
+        let created = seg
+            .creation_time()
+            .expect("a v7 segment id must expose a creation time");
+
+        // v7 has millisecond precision, so allow a small slack around the window.
+        let slack = Duration::from_millis(1);
+        assert!(created >= before - slack);
+        assert!(created <= after + slack);
+    }
 }