Improper Shutdowns can lead to permanent write locks (until proper restart)

[jhmullen]

When Riak is improperly shut down or its process is killed, the cleanup processes that release lock files are not triggered. If another OS PID has been created that shares Riak's old process id, and Riak is started again, the subsequent checks in Riak will see the original OS PID that was written to the write lock file is still active, and will not release the lock (even though the process id in question does not refer to Riak anymore).

This can be replicated by:

• Starting Riak.
• Shutting down Riak improperly by issuing a kill on the Riak process.
• Starting a process (that is not Riak) that uses the same Pid as the killed Riak process.
• Starting Riak.
• Attempting a PUT.

Could the operation that checks for the OS PID's existence also confirm that it is in fact a beam.smp process, to lessen the likelihood of this stuck lock file?

For additional context, see the following:
https://basho.zendesk.com/agent/#/tickets/3873
https://basho.zendesk.com/agent/#/tickets/5336

[evanmcc]

Recording some discussion on the mailing list and chat re: this issue:
Locks should never survive a system restart, so it's safe to sweep for them and delete what you find when riak starts.

It's hard to encode this given bitcask's slightly weird application semantics, but we should be able to fix for 2.0.

[TJC]

I hit this again recently; in some testing of virtual machines which were subject to sudden shutdowns.
Took a while before I figured out what the problem was.

Would it be an easy fix to just add to the init scripts something like
find $RIAK_DIR -name "bitcask.*.lock" -delete

prior to starting riak?

[evanmcc]

Hi Toby,
Yes, that works, but it isn't safe. The problem here is that the
semantics of backend locking are ill-defined. We need to work out what
guarantees we want to provide and actually work to provide them.

Deleting the locks on startup helps one case, but hurts at least one other.

The guarantees that we try to provide are:

1. No two erlang processes in the same VM can have write access to the
   same cask at the same time.
2. No two OS processes can access the same cask at the same time.

So, 1 is reasonable, but 2 has some problems. We try to provide it,
but it isn't actually the guarantee that we want. The guarantee that
we want is: no two riaks are running on the same data directory.
There's a soft-guarantee of this in that nodenames must be unique, and
that only one process can bind a particular port, but it's still
possible to make it happen via aggressive misconfiguration.

IMO, at the bitcask level guarantee 1 is all we should try to provide.
It should be up to the container application (riak, or whatever
else might use bitcask) to decide what variation of guarantee 2 it
wants to provide and provide it. The question that then remains is:
how does riak provide this guarantee, and how does it communicate it
to the bitcask application instance it is running.

It's possible that I am overthinking this and bitcask + leveldb should
be trying to provide all of these guarantees on their own, but that
smells wrong to me (and is also hard to make portable).

Given where we stand right now, if one were to wrap riak start in some
sort of script that tried to prevent this, I would do this:

killall -9 beam.smp  # make sure that nothing that could possibly
access bitcask is running
find $RIAK_DIR -name "bitcask.*.lock" -delete  # then clean up

[evanmcc]

This is sufficiently covered by basho/riak#535, closing.