The default serialization format for Redis is RESP2, which serializes values of the following Crystal types:

• String
• Int64
• Nil
• Redis::Error (returned as results by Redis::Commands::Deferred types, but Redis::Commands::Immediate types raise an exception when they encounter it)
• Array of any of these types, including other arrays

Something I've been wanting to do for a while is update this client to RESP3, which adds the following types:

• Bool
• Float64
• BigInt
• Hash that can hold any Redis value in keys and values, similar to arrays
• Set which, like arrays and hashes, hold any Redis value

There are a couple other types that seemingly have no semantic difference in Crystal but exist to have a different encoding over the wire:

• Verbatim strings
  • I honestly don't understand the reason this exists but it includes a 3-byte "encoding" prefix that, as far as I can tell, only supports txt or mkd (markdown)
• Bulk errors
  • Still a Redis::Error type, but the error message can span multiple lines
  • It's the error's equivalent to the Redis bulk string encoding
• Null gets its own serialization — in RESP2 it's encoded as a string or array with a negative size

The protocol also includes a couple other constructs that would need to be handled at a higher level than the parser because they are not the results of a Redis command:

• attributes
  • precede a command's reply with additional information that appears to be intended to be attached to the connection, similar to Postgres server parameters
  • encoded identically to hashes (which Redis calls "maps" despite having a data type called a hash 🙃) but use a different byte marker
• pushes
  • out-of-band data whose purpose I don't really understand — maybe pub/sub?
  • the docs mention setting up a callback to handle pushes
  • encoded identically to arrays with a different byte marker

Implementation

From an implementation perspective, most of this can be implemented inside Parser without having to do anything else. However, Commands::Immediate and Commands::Deferred types will also need to accommodate the "attributes" and "push" types.

A couple open questions:

1. Would "attributes" and "push" types be considered Redis::Values?
   • Right now, the types returned by Parser#read and the types in Redis::Value are the same set of types. Do we stick with that or only consider Redis::Value types to be result types you would receive when running a Redis command?
2. When receiving attributes, how do we access them?
   • The docs seem to indicate that they should be attached to the connection, but how do you get them if you're using a Client, which has a pool of clients? Or a Cluster/ReplicationClient, which both use a pool of Clients (so Connection is at least two degrees of separation)?
   • We don't currently expose a first-class method of getting a connection from any of the higher-level clients. Client delegates everything to a Connection, but you're not guaranteed to get the same Connection twice in a row and you're never directly interacting with it. Cluster runs multiple Clients based on the key you're operating on, but getting attributes doesn't involve a key and it doesn't even give you the Client, let alone the Connection. ReplicationClient can at least let you choose which Client you're using (because it has 2 and one of them is read-only), but that still doesn't yield a Connection.
   • Maybe all of these need some sort of with_connection(& : Connection ->) method? Trivial for Client (and, by extension, ReplicationClient), but Cluster would need it to be something like with_connection(key : String, & : Connection ->) to even know which server to yield a connection for. Totally open to suggestions on pros and cons of this approach.
3. For pushes, I assume we could do something like Connection#on_push. I also assume Client#on_push would do something like DB::Database#setup_connection (it uses the same connection pool) to call Connection#on_push. Then Cluster and ReplicationClient would call Client#on_push for all the Client instances they manage.

Upgrade path

This is, unfortunately, a backwards-incompatible change. Not only would a case ... in construct on the result of a redis.run command no longer compile once this is implemented, but several Redis commands return different types when RESP3 is enabled. For example: HGETALL returns an array with RESP2 and a hash with RESP3, so the hgetall method override in Commands::Immediate would be updated to return a Hash.

While this shard is still pre-1.0, that's just an oversight. I meant to bump to 1.0 a while ago. This shard is used in multiple popular Crystal shards, such as cable and mosquito which I think warrants a 1.0 version number. We'll need to work with projects that depend on this shard to plan an update. It's entirely possible that they won't need to change much, but it's important to confirm that.