Using the client

Simple Hello World

File file = new File(path); //Can be as big as you want
HalladoopClient doop = new HalladoopClient(nameNodeAddress, 8080);
doop.write(fileName, path); //Breaks it into many chunks and replicates 3 times
File returned = doop.read(path); //Gets the first replicas of the blocks and builds your file and returns it

Process Diagrams

Writes

##Reads

##Heartbeats

Namenode Rest Api

Register

• Registers a dataNode with the nameNode.
• Returns an Id that the node will be identified by in the cluster.

• URL

  /register/

• Method:

  POST

• Data Params

  • node_ip
  • total_disk_space_mb
  • available_disk_space_mb

• Sample Call:

  {
    "node_ip": "1.1.1.1",
    "total_disk_space_mb": 1337,
    "available_disk_space_mb": 1337
  }

• Success Response:

  • Code: 201 CREATED
    
  • Content:

  {
    "node_id": 1
  }

Heartbeat

• Updates the nameNode of the dataNode's status.
• Returns an ack may include delete or replicate operations to be executed by the dataNode.

• URL

  /heartbeat/

• Method:

  POST

• Data Params

  • node_id
  • available_disk_space_mb
  • block_manifest

• Sample Call:

  {
    "node_id": 1337,
    "available_disk_space_mb": 10,
    "block_manifest": ["block1", "block2", "block3"]
  }

• Success Response:

  • Code: 200 OK
    
  • Content:

  {
    "delete": ["block1","block2","block3"],
    "replicate": [
      {"block_id": "block123", "nodes": ["1.2.3.4", "2.3.4.5"]}
    ]
  }

Finalize

• Informs the nameNode of a completed block write by the client or a completed replicate operation from a dataNode.

• URL

  /finalize/

• Method:

  POST

• Data Params

  • block_id
  • nodes

• Sample Call:

  {
    "block_id": "block123",
    "nodes": [1,2]
  }

• Success Response:

  • Code: 200 OK
    
  • Content:

  {
    "delete": ["block1","block2","block3"],
    "replicate": [
      {"block_id": "block123", "nodes": ["1.2.3.4", "2.3.4.5"]}
    ]
  }

Write

• Client request to dataNode to prepare to add a file to the virtual filesystem.
• Returns a list of dataNodes that will recieve the file's blocks.

• URL

  /write/

• Method:

  POST

• Data Params

  • file_path
  • num_blocks

• Sample Call:

  {
    "file_path": "/dir/another_dir/text.txt",
    "num_blocks": 12,
  }

• Success Response:

  • Code: 201 CREATED
    
  • Content:

  {
   "nodes": [
     {"node_id": 1, "node_ip": "1.1.1.1"},
     {"node_id": 2, "node_ip": "2.2.2.2"},
     {"node_id": 3, "node_ip": "3.3.3.3"}
   ]
  }

• Error Response:

  • Code: 403 FORBIDDEN
    Content: { error : "File Exists." }

Read

• Client requests the locations of all blocks for a given file.
• Returns a list block_id's mapped to a list of dataNodes that contain the block.

• URL

  /read/{file_path}

• Method:

  GET

• Sample Call:

  /read/dir/another_dir/text.txt

• Success Response:

  • Code: 200 OK
    
  • Content:

  {
  "manifest": [
     {"block_id": "123", "nodes": ["1.2.3.4.", "4.3.2.1","8.8.8.8"]}
   ]
  }

• Error Response:

  • Code: 404 NOT FOUND
    Content: { error : "File not found." }

Delete

• Client requests that a file be deleted from the virtual filesytem.
• Block deletes will be enacted via heartbeat responses.

• URL

  /delete/{file_path}

• Method:

  DELETE

• Sample Call:

  /delete/dir/another_dir/text.txt

• Success Response:

  • Code: 200 OK