CS739.P1.md

CS739-Sp23 (P1): Guidelines and Tests

Overview

The outcome of this project is an AFS-style distributed file system, referred as wiscAFS. Specifically, we will focus on three aspects of wiscAFS: Functionality, Consistency, and Durability. Our evaluation will be based on the combination of these three, with 50% devoted to functionality, 30% to consistency, and 20% to durability.

We require wiscAFS to be implemented based on an open source FUSE-based file system UnreliableFS; you will need to add all the AFS features required in the project description atop. We think it will simplify the work to understand how to use FUSE, and the code is quite clean for learning purposes. We also rely on the fault injection feature to explore some more advanced topics in wiscAFS. You can use programming language among rust/golang/C/C++ for the server of wiscAFS.

To demonstrate the functionality of wiscAFS, it is required to support filebench, a widely used file system benchmark. We select a set of workloads to support. You will need to report and interpret the numbers that the benchmark tool generate so as to get sense of my file system performs well (or performs fine). Furthermore, we select two real applications (qemu and parallel build) for wiscAFS to support such that we are confident that wiscAFS is a quite functional file system.

To showcase that your wiscAFS works well in maintaining data consistency, you need to show that it passes good test cases. To help you start, we provide a testing toolkit that allows for precise control over the coordination of two concurrent clients, including a starting test case. You will need to pass the provided test cases and design your own test cases. Your tests need to cover the important semantics, such as flush on close, last writer win, and consistent in the presence of a client or server crash. The next (more advanced) step of this part can be one of the two options; you either understand more of the nature of the concurrent writes (i.e., last writer wins) or learn to provide primitives on top of the AFS semantics to better serve applications' needs.

Finally, we illustrate the complexity to persist the data locally (as specified by AFS) by ALICE-reproduce, which employs a cache layer and mechanisms in wiscAFS to reproduce the possible corruptions that a kernel file system may result in under application workloads and power-offs.

Functionality (50%)

To testify the functionality, we will use Filebench to stress single operations and synthetic combinations of operations first. We also require wiscAFS to support two important kinds of applications (virtual machine and make).

Filebench (40%)

1. Build and Install filebench.
2. Workloads
   • Workloads can be found in this directory.
   • Filebench, as the name suggested, is a widely-used file system benchmark. So please report all the numbers generated by each workload. You will need to present the numbers and explain them in the project presentation and reports.

• Note:
  • Filebench requires to disable ASLR to run: echo 0 > /proc/sys/kernel/randomize_va_space.
  • $MOUNT_DIR refers to where your file system is mounted, for example /workspace/fuse_mnt/wiscAFS/.
  • You may find this set_dir.py useful to set $MOUNT_DIR/bench into those .f files in a batch (please invoke the script in that directory and supply the absolute path, e.g., mine is /workspace/fuse_mnt/wiscAFS/bench).
  • Please run each workload at least three times and get the average.
  • Please run each workload long enough such that the number is stable during each workload run.
  • Please try to understand what each workload is measuring, as we are likely to ask, and it will be in your report.
  • We will not consider the absolute performance of wiscAFS as much in our assessment, as explaining the performance difference between different workloads is more important in this part.
    • But if you indeed make it blazingly fast, feel free to show us and present how you achieve the fastest ever version.

Applications (10%)

1. Xv6 and Qemu

   • Execute the following commands sequence in $MOUNT_DIR (or a subdirectory).

     git clone https://github.com/mit-pdos/xv6-public.git
     cd xv6-public
     make qemu-nox #(You may need to install qemu for this).

2. Parallel Build

   • Execute the following commands sequence in $MOUNT_DIR (or a subdirectory)

     git clone https://github.com/google/leveldb.git
     cd leveldb
     git submodule update --init
     mkdir build
     cd build
     cmake ..
     make clean
     make -j $N_THREADS # -> Number of threads

   • Vary $N_THREADS and report the time used for make -j $N_THREADS

Consistency (30%)

Each group needs to finish the part of Basic Cache Consistency, and then pick one out of A and B to delve a bit deeper. We expect you to pick one option that you like more (or simply easier); of course, you are welcome to do both.

Basic cache consistency adhere to the AFS semantics (20%)

We provide a simple testing toolkit for you to start with. The testing toolkit implements the coordination of two concurrent clients while running workloads so that we can easily control the relative orders of each operation to understand if the concurrent execution is handled correctly (aka. consistent). We will focus on flush on close and last writer wins. We also expect the file system is consistent after a client or server process crash (i.e., pkill).

1. The testing toolkit is provided here.
2. Setup
   • Configure environment variables in 739p1.env and put it into $HOME in both of the hosts; we tested using root.
3. Try the provided test case
   • Put the python scripts including test1_clientB.py into the right directory in the host specified by CS739_CLIENT_B.
     • Look at this code to see if you want to adjust it or not.
   • In the host specified by CS739_CLIENT_A, do python test1_clientA.py; it will start the execution of this script, which invokes test1_clientB.py in the other host.
4. After you understand a bit of how that simple coordination works, come up with more test cases (at least 3 more) to stress different challenging scenarios.
   • Finally, pass the tests and explain them.
     • The test results are recorded in a temp file, see this.
   • We encourage you to design good (harsh) test cases, as it would be great to integrate them into next year's provided cases of this class.
5. Add the testing of the cases of a client machine crash and server process crash, in whatever format you would like. It's ok to use pkill to kill the wiscAFS (local or remote) process.

• We provide such for you to start with, yet feel free to do your own if you would like. If so, please ensure the similar level of precision and logging of intermediate results to understand what is going on.
  • Hopefully this gives some sense of what is distributed coordination.
• We tried to write comments for you to understand the code, please read and feel free to ask questions.

Option-A: Who is the winner? ([10%])

We will explore the un-determinism in distributed systems (e.g., network transfer) more in this part by injecting delays and exploring the probability of winners.

1. Start two clients who issue requests towards one file, and inject delay of flush by the fault type errinj_slowdown. In the wiscAFS server side, report who is the winner and the time interval between the arrivals or processing time of the two operations.
2. Vary the slowdown and run the experiments for a lot of times until the probability is stable (Recall some basic statistics for testing the probability of flipping a coin?).
3. Measure the basic RPC latency between your client and server. Note please vary the message size.
4. Explain the conditions that how much slow down injected will dramatically change the probability of the winner, otherwise, the probability indicates the network randomness, which you shall also present what kind of the network randomness is like. Change the write size and see if the conclusion changes.

Option-B: Cross-file atomicity ([10%])

We will try to understand CACHE CONSISTENCY IS NOT A PANACEA¹, so consider read the paragraph if reading the whole chapter takes too much time.

1. Implement a new type of error (it's not an error actually), virtue_atomic_group, to specify that a series of filenames (or a directory) will be updated atomically.
   • You will need some mechanism like file lock in the server side, for example, grab the lock of several files, flush the updates, and release the locks.
     • Feel free to come up with other mechanisms.
     • Make sure to explain it clearly and validate the correctness of the design.
2. Design workloads to show
   • Without such feature, concurrent writes from two clients can interleave with each other.
     • For example, a database may store indexes in a file, write-ahead-log in a file, and data in another file. Then if the three files' updates from different wiscAFS clients are not atomically applied, the database is likely to be corrupted and the users of the database will be clearly unhappy.
   • By specifying the set of files as atomic group, the several files are updated as a whole.

Durability (20%)

In this part, to understand the subtle issues around durability in the local file system, we will do ALICE-Reproduce. The idea is to reproduce the symptoms observed in the ALICE paper. The requirements are:

1. Implement two new types of error errinj_alice_reorder and errnj_alice_delay in UnreliableFS.
   • From the high level, these faults require you to add a thin caching layer (or some kind of pending operation queue structure) before the operations are issued to the local host file system.
   • errinj_alice_reorder: the file operations will be reordered and then batched to the local host file system.
   • errinj_alice_delay: the file operation will be delayed before the request is sent to the local host file system.
   • It's up to you to decide the policy of delay and reorder. The goal here is to emulate the Persistence Properties of one of the file system in the table.
     • You don't need to implement all of the properties, only the ones that sufficient for you to reproduce the symptoms (e.g., corruption) under the workload you choose is adequate.
2. Come up with one simple workload that is vulnerable (i.e., making assumptions of the atomicity or ordering of the file operations), run the workload towards your wiscAFS, and crash wiscAFS (local) after the workload finished.
   • You can use one of the workloads in Figure 4 of the paper and simplify it.
   • First ensure that the output of the workload is correct without crashing wiscAFS (local), then run the workload with crashing the wiscAFS (local), and finally show a corruption happens because of the violation.
   • The policy design and the workload can be quite simple (i.e., easy to implement), as long as it shows the symptoms.

Footnotes

1. ASIDE: CACHE CONSISTENCY IS NOT A PANACEA (Page 6, Chapter 50, AFS, OSTEP) ↩