Add auto-repair to fully sync the store (#144)

* Allow self-repair of the store

Per issue #23, any error while processing a branch of the dag results in that
branch never being processed again, as we have the blocks in the blockstore and we
assume that every block we have has been processed.

This PR adds a repair goroutine that runs whenever the crdt-store is marked as dirty.

The crdt-store gets marked as dirty when errors have occurred when "processing
nodes" or "sending new jobs" (fetching nodes from the network usually).

This task, which runs every RepairInterval, starts walking down the DAG from
the heads and retriggers re-processing for every node that was not processed.

It does not stop until the bottom has been reached.

In order to know which DAG nodes have been processed and which ones haven't we
need to explicitally store the list of processed nodes (which is different
from the list of nodes that we have). A node becomes processed as soon as its
delta has been merged.

Repair can be an expensive operation thus we should avoid marking the
datastore as dirty easily, which perhaps is the case when we shutdown while
processing things.

Once repair reaches the bottom, the dirty bit is cleared.

Datastores in previous versions will reprocess the store on first run, as they
need to mark all the nodes in the DAG as processed.

* DAGSyncer can become a DAGService

It is no longer necessary to check if a block is in the blockstore, as we keep
track separately.

* Allow disabling auto-repair when interval set to 0

* Add public Repair method and improve repair logic and logging

* Improve node processing logic and queue caching for deduplication

I believe nodes could be removed from the queue by sendJobs
and then re-added to the queue before they were processed, resulting
in several workers reprocessing the same nodes potentially.

* Update globaldb example

* Enable processing of multiple heads in parallel

Now that we have improved processing logic to make branch-processing not step
constantly on each other's feet, we can re-enable processing multiple heads in
parallel and take advantage of the multiple workers which are normally idling
as only one head is processed at a time.

* Keep improving the repair process and the syncing

Ensure that re-processed branches are setting the right heads.

Tighten up logic to avoid duplication of processed nodes.

* Do not rebroadcast anything if there are no heads to rebroadcast

* Fix repair notifications

* Add helper functions to the processedBlock namespace

* Repair: Just keep a full set of visited nodes while walking

In memory, unfortunately.

Author: hsanjuan

README.md

@@ -28,8 +28,8 @@ the height of the Merkle-CRDT node in which the key was introduced.
     [libp2p PubSub](https://godoc.org/github.com/libp2p/go-libp2p-pubsub) and
     the provided
     [`PubsubBroadcaster`](https://godoc.org/github.com/ipfs/go-ds-crdt#PubSubBroadcaster).
-  * A user-defined `DAGSyncer` component to publish and retrieve Merkle DAGs
-    to the network. For example, you can use
+  * A user-defined "dag syncer" component (`ipd.DAGService)` to publish and
+    retrieve Merkle DAGs to the network. For example, you can use
     [IPFS-Lite](https://github.com/hsanjuan/ipfs-lite) which casually
     satisfies this interface.