This is a demo app that implements Instagram style database sharding by id as described here:
instagram-engineering.tumblr.com/post/10853187575/sharding-ids-at-instagram
I forked the Octopus gem and allowed for multiple logical databases to share a single connection pool. Users and their data are partititioned up front into a logical shard. Shards can replicated and moved to new physical servers as needed to scale.
User info is replicated to HBase for analytics and offline processing. It is also sent to ElasticSearch for searching across shards.
The replication and indexing is done asynchronously through a Redis queue using Sidekiq.
The application is set up to use Fig + Docker to manage containers and dependencies.
Ensure that you have Docker (www.docker.com/) and Fig (www.fig.sh/) installed. Then, pull the code and run ‘fig run web rake bootstrap:install` to bootstrap your containers.
Start all application services with ‘fig up`, then visit localhost:3000 to view the app.