InstaDynamo: Decentralized Image Storage System

Group Members

• Aman Hassan (2021CS50607)
• Brian Sajeev Kattikat (2021CS50609)

For instructions on how to setup and run the applications locally or in deployment mode, please refer to SETUP.

Technical Stack Summary

• Language: Python 3.x
• Asynchronous I/O: asyncio, aiohttp, aiofiles
• Web Framework: FastAPI, Uvicorn (ASGI Server)
• Cryptography: cryptography (Fernet symmetric encryption), passlib (Bcrypt), hashlib (SHA-256)
• Database: SQLite (default), PostgreSQL (supported via connection string), SQLAlchemy ORM
• Data Structures: sortedcontainers (SortedDict for Consistent Hashing)
• Frontend: Jinja2 Templates, WebSockets, Vanilla JS

Project Overview

InstaDynamo is a distributed, decentralized image storage and retrieval system. It mimics the architecture of Amazon's Dynamo key-value store to provide high availability. The system allows users to upload images, which are encrypted, hashed to generate a lookup key, and distributed across a ring of storage nodes using consistent hashing.

The project is divided into three networked applications:

1. Backend Application: Handles user auth, serves the UI, and routes requests.
2. Dynamo Node: The unit of storage. Multiple nodes form a ring.
3. Control Panel: Admin dashboard to visualize the ring and manage nodes.

Component Summaries

1. Backend Application (/backend)

Acts as the entry point for end-users. It serves the HTML frontend using Jinja2 templates and exposes API endpoints for user management. It does not store the actual image data. It encrypts the image, calculates the hash key, and forwards the encrypted data to the Dynamo Nodes.

2. Dynamo Node (/dynamo_node)

This is the core storage engine. Each running instance represents a physical node in the Dynamo ring. It implements consistent hashing to determine which keys it is responsible for. Nodes communicate with each other to handle data replication and ring membership changes.

3. Dynamo Control Panel (/dynamo_control_panel)

A centralized management interface. It allows administrators to manually add new nodes to the cluster and visualize the current state of the ring (physical and virtual nodes) via WebSockets.

Detailed Component Architecture & API

Backend Application

The backend abstracts the distributed storage from the user. It acts as a coordinator.

Key API Endpoints:

• POST /auth/*: Standard authentication routes (signup, login). Uses bcrypt for password hashing.
• POST /image/upload: Reads the raw image bytes, generates a SHA-256 hash (the Key), encrypts the content using Fernet, and contacts the storage layer to PUT the data.
• GET /image/{key}: Validates the key against the local metadata DB. If valid, fetches encrypted bytes from the Dynamo Node, decrypts them, and returns the image.

Dynamo Node

This component implements the distributed systems logic. It maintains a HashRing class using a SortedDict to map keys to nodes. It handles Virtual Nodes to distribute load evenly.

Key API Endpoints:

• POST /upload & GET /fetch/{key}: Basic I/O operations for the node's local filesystem (./store).
• POST /invite_node: Sent by the Control Panel to an existing node. This tells the existing node to contact a new node and begin the protocol to add it to the ring.
• POST /join_ring: The receiving node reconstructs its local HashRing based on provided metadata and initializes connections.
• POST /ring_transfer: Triggered during a node join. Existing nodes calculate which keys they no longer own (because the new node took that range) and transfer files to the new node.

Dynamo Control Panel

The control panel provides visibility into the ring. It maintains a connection pool to nodes to monitor status but does not store data.

Key API Endpoints:

• POST /add_node: Connects to a target node. If the ring is empty, it initializes it. If the ring exists, it triggers the invite protocol on a peer.
• WS /admin_dashboard: A WebSocket endpoint that pushes the current list of physical_nodes and virtual_nodes to the frontend client every 5 seconds.

Dynamo Paper Implementation Status

This project implements core concepts from the SOSP 2007 Dynamo paper, with specific simplifications.

Implemented Features

• Consistent Hashing: Data is partitioned across nodes using a consistent hashing ring.
• Virtual Nodes: The HashRing class implements virtual nodes to handle load distribution.
• Replication: The system supports a configurable N (replication factor). The preference list includes the primary node and N-1 successors.
• Decentralization: Nodes operate as peers. Data transfer and state updates happen between nodes.
• Availability: Writes are accepted even during topology changes via pending_transfers queues.

Omissions and Simplifications

• Versioning (Vector Clocks): The paper uses vector clocks for causality. This project currently relies on Last Write Wins (LWW).
• Merkle Trees: The paper uses Merkle trees for background anti-entropy. This project uses explicit transfer endpoints during node join events.
• Gossip Protocol: The paper uses gossip for failure detection. This project uses explicit HTTP endpoints triggered by the control panel or joining nodes.
• Sloppy Quorum: The code has logic for pending transfers, but strict R + W < N quorum logic is not fully enforced in the configuration.

Challenges & Future Work

Challenges

• Node Synchronization: Ensuring all nodes update their HashRing views simultaneously during a join event without a central coordinator like Zookeeper.
• Data Migration: Implementing the logic where a new node "steals" keys from existing nodes. Identifying which keys moved and transferring them without downtime was complex.

Future Work

• Vector Clocks: Replace overwrite logic with vector clocks to handle concurrent updates.
• Read-Repair: Implement logic to update stale replicas when a read operation detects a version mismatch.
• Background Anti-Entropy: Add Merkle trees to detect and fix data consistency issues in the background.
• Automatic Failure Detection: Add a heartbeat mechanism to automatically remove dead nodes from the ring.