A new directory layout of redis service

[ClSlaid]

I've been implementing Redis service and found the current design of directory tree layout is incorrect and make implementing inelegant. So I'm here to repropose a new way of representing directories in Redis service.

A new struct RedisEntry will be introduced:

pub(crate) RedisEntry {
    /// Sub-directories of the current entry.
    ///
    /// for files, it will be an 0-lengthed vector.
    pub children: Vec<String>,
}

Standing alone from metadata, the relationship between directories and their children will be stored in two additional dimensions of keys, p for the parent directory and ch for children:

+------------------------+
|Directory               |
|          +-------------+----+
|Name: /   |             |    v
|SKey: v0:ch:/           |    0) /home/
+------------------------+


+------------------------+
|Directory               |
|              +---------+----+
|Name: /home/  |         |    v
|SKey: v0:ch:/home/      |    0) /home/monika/
+------------------------+


+-------------------------+
|Directory                |
|                     +---+---+
|Name: /home/monika/  |   |   v
|SKey: v0:ch:/home/monika/|   0) /home/monika/poem.txt
+-------------------------+   1) /home/monika/music/


+---------------------------------+    +-------------------------------+
|File                             |    |Directory                      |
|                                 |    |                               |
|Name: /home/monika/poem.txt      |    |Name: /home/monika/music/      |
|/* File has no children */       |    |SKey: v0:ch:/home/monika/music/|
+---------------------------------+    +-------------------------------+

• SKey: the children of the current directory, will be stored in a SET.

Listing

For listing directories, just SSCAN through it, nice and correct.

// print all sub-directories of `PATH`

let s_key = format!("v0:ch:{}", PATH);
let mut con = client.new_async_connection().await?;
let mut it = con.sscan::<&str, String>(s_key).await?;

while let Some(child) = it.next_item().await {
    println!("get sub-dir: {}", child);
}

Create

On creating a file or directory, the backend should create its all parent directories if not present.

// create a file under `PATH`
let mut path = std::path::PathBuf::new(PATH);
let mut con = client.new_async_connection().await?;

while let Some(parent) = path.parent() {
    let (p, c): (String, String) = (parent.display(), path.display());
    let to_children = format!("v0:ch:{}", p);
    con.sadd(to_children, c).await?;
    path = parent;
}

Delete

On deleting a file or directory, the backend should remove the entry from its parent's SET, and remove all children of entry.

Recursion may be needed here.

async fn remove_entry(con: &mut redis::aio::AsyncConnection, entry: String) {
    let skey = format!("v0:ch:{}", entry);
    let it = con.sscan::<&str, String>(skey).await?;
    while let Some(child) = it.next_item().await {
        remove_entry(&mut con, child).await;
    }
    if let Some(parent) = std::PathBuf::new(entry).parent() {
        let p: String = parent.display();
        let parent_skey = format!("v0:ch:{}", p);
        let _ = con.srem(parent_skey, entry).await;
    }
    // remove metadata and content
}

Pros

• making listing correct.
• making implement listing easier

Cons

• make deleting and creating more complicated
• draws back performance of deleting and creating

[Xuanwo]

Nice idea! It's a general problem that exists in all in-memory indexes: memory services, ImmutableIndexLayer, and yes, redis services.

Our current design is a bit like an ext file system:

• v0:m:/ -> metadata of /
• v0:ch:/ -> names of / (a SET of names inside /)
• v0:c:/file -> content of a file

Please update the RFC before going further.