Data at Rest Encryption

[Bob-The-Marauder]

Data at Rest Encryption

Issue

Riak as a whole does not encrypt data at rest.
https://www.tiot.jp/riak-docs/riak/cs/2.1.1/cookbooks/faqs/riak-cs/#does-riak-cs-encrypt-data-at-rest
https://quabase.sei.cmu.edu/mediawiki/index.php/Riak_Security_Features

Problem

The problem has three fronts:

• Many competing systems are offering encryption of data at rest i.e. marketing.
• Industries are starting to make encryption of data at rest a requirement i.e. usage viability.
• Hackers are becoming better at accessing file systems where non-encrypted data is at risk i.e. security.

Potential Pitfalls

• Partitions are moved between servers so an encryption key would have to be standardised between all nodes in the cluster to allow encryption and decryption i.e. if a partition is encrypted by one node with one key and then transferred to a node with a different key, it is now useless.
• As Riak runs inside an ErlangVM, the encryption key would most probably need to be stored in plain text format.
• Being open source, it should not be long before a potential attacker learns the location of the encryption key, copies it and is able to use it to decrypt data again.
• Keeping the encryption key on one server and sharing with the cluster at run time works until the server that stores the key goes offline.

Security Concepts

Security works in three ways:

• Who you are
• What you have
• What you know

"Who you are" is based on the user you are accessing the server as. If the attacker is root or riak then they can just ask Riak for the full database in human readable form, so let's ignore this one.

"What you have" would apply to the encryption key. Anybody with the encryption key and a bit of technical know how should be able to retrieve human readable data from the file system (data directory) of Riak even if it is only partial data.

"What you know" would be where real users can be differentiated from attackers. Real users will know things such as bucket names and key names. Attackers are thieves of opportunity and will be unlikely to know this information especially if all they have to work with is the file system of a single node.

Proposed Solution

Combining the above Security Concepts, we can see that the only difference between an attacker and a genuine user is the knowledge of the bucket and key names. I would like to propose the following solution to encrypting data at rest within Riak KV:

• Share an encryption key between all nodes in the cluster (probably user specified text file saved per node).
• Have an option in riak.conf to allow encryption to be used (some people might not want it yet).
• Have a start up check and possibly a polled interval that makes sure all nodes in the cluster are using the same key and have encryption turned on.
• When writing data, use the encryption key as discussed but salt the key with the bucket name and the key name to encrypt the value stored.
• When reading data, decrypt with the same salted encryption key which should work as the user will be providing the bucket name and key name.
• Buckets are tagged as encrypted so that we can check what percentage of the data is encrypted

By salting the key with the bucket and key names, it would render the plain encryption key nearly worthless to a potential attacker as they would have to brute force all potential bucket/key name combinations to access the data.

Legacy and Migration

Assuming that the above functionality can be implemented, users may wish to encrypt an already existing Riak install or may decide they no longer wish to keep their data encrypted and wish to decrypt everything. I propose that a default read/re-write mode be created. This mode would check whether data is meant to be encrypted or not and every time Riak reads data that is not of the desired encryption type, it re-writes the exact same data but in the desired encryption type.

Through this method, users with high load clusters could wait for their data to naturally become encrypted/decrypted or users in a hurry could run a script to read every single bucket to force the encryption. Just in case clusters are subject to performance spikes, potentially add a user definable config flag e.g. 70% where read/re-write is temporarily disabled while it waits for the cluster to be under less than 70% load before resuming.

We might want to add an output to a command such as riak-admin cluster status that would include the "Percentage encrypted" information for the cluster.

Additional Security

As security is all about layers, it might be worth looking at storing the key with extremely limited permissions i.e. to access the encryption key you would have to be the riak user or preferably just root but this would depend on how easy it would be to get the ErlangVM to access said encryption key.

[Licenser]

To be blunt this seems like a lot of work for no result. If the encryption key lives on a text file on the host, copying the filesystem means copying the key and all needed information as:

• The key is on the same node - an attacker has it
• The bucket names need to be stored unencrypted for bucket listing/bucket metadata (or encrypted with the key only but as we have the key it's the same thing) - an attacker has them
• The key names have to be stored unencrypted (or encrypted with key + bucket name but since we got those on the last two steps it's the same thing)

The only possible solution for this I see is having a user provide the encryption key on request, it still lives in memory for a while so a crafty attacker would be able to obtain it but at least it'll be a bit safer. This could be somewhat mitigated by using asymetric encyption, i.e. have a public key assigned to a bucket (that lives on the nodes w/o issue) that would mean writes do not need an additional key, and either return encrypted data (which would mean reworking how repair & CRDTs work) or providing the secret key on reads (which introduces another security risk).

[Licenser]

When it comes to just encryption at rest, filesystem-level encryption will probably do a better job then what we can come up with, those systems are often build by experts and audited for security.

[Bob-The-Marauder]

I think the point has been missed here. The idea of salting the encryption key with the bucket name and key name is to render the encryption key alone useless.

From Riak's point of view, it doesn't care whether your bucket is called "myBucket" or "as789AW!233fh84325qhfdsuy9876", so encrypting the bucket name with the encryption key salted with the bucket name is a perfectly valid solution. This way, the attacker cannot retrieve the bucket name from the file system without significant brute force work or some other non-Riak related method to get the list of bucket and key names a different way.

This method does have the caveat that you cannot ask Riak for a list of buckets and keys but even Basho said that this should not be done in a production environment:
https://www.tiot.jp/riak-docs/riak/kv/2.2.3/developing/api/http/list-buckets/
https://www.tiot.jp/riak-docs/riak/kv/2.2.3/developing/api/http/list-keys/

I do not argue that having filesystem-level encryption would be beneficial. However, security is all about layers and if the attack is against a live server where the attacker successfully gains riak or root then filesystem-level encryption will be worthless. Thinking about it, with the bucket and key names encrypted, the attacker would not be able to run a bucket list or key list to retrieve the buckets or keys as the returned lists would be the encrypted names. Of course, the attacker could brute force them but still, this would further slow them down and that is the entire idea of layered security. Nothing is totally secure but every step to hinder the attacker is worthwhile.

At the end of the day, we need to look at this with a regard to the future suitability of Riak for corporate customers which means we need to address the problem listed at the start of the post:

• Many competing systems are offering encryption of data at rest i.e. marketing.
• Industries are starting to make encryption of data at rest a requirement i.e. usage viability.
• Hackers are becoming better at accessing file systems where non-encrypted data is at risk i.e. security.

Riak already has a surprisingly wide array of security options https://www.tiot.jp/riak-docs/riak/kv/2.2.3/using/security/basics/ which have been nicely summarised in this blog post https://gist.github.com/lucperkins/8920660 and, in the corporate world, where ticking boxes is often as far as senior management looks towards IT security, having this feature available would fill that requirement nicely.

It may be a lot of work that many people do not see as a priority right now but as regulations start rolling out, I am not sure whether Riak will be able to survive outside of obscurity for long if its only remaining users are hobbyists and one or two die hard corporate fans.

My 2¢ - better to encrypt the data at the middleware layer - if it were confidential I'd never even send it to the storage system in the first place.