wip

Author: wenkokke

README.md

@@ -297,7 +297,7 @@ The worst-case in-memory size of an LSM-tree is $`O(n)`$.
   `confWriteBufferAlloc`. Regardless of write buffer allocation
   strategy, the size of the write buffer may never exceed 4GiB.
 
-  `AllocNumEntries maxEntries`  
+  `AllocNumEntries maxEntries`
   The maximum size of the write buffer is the maximum number of entries
   multiplied by the average size of a key–operation pair.
 
@@ -309,11 +309,11 @@ The worst-case in-memory size of an LSM-tree is $`O(n)`$.
   filter allocation strategy, which is determined by the `TableConfig`
   parameter `confBloomFilterAlloc`.
 
-  `AllocFixed bitsPerPhysicalEntry`  
+  `AllocFixed bitsPerPhysicalEntry`
   The number of bits per physical entry is specified as
   `bitsPerPhysicalEntry`.
 
-  `AllocRequestFPR requestedFPR`  
+  `AllocRequestFPR requestedFPR`
   The number of bits per physical entry is determined by the requested
   false-positive rate, which is specified as `requestedFPR`.
 
@@ -336,12 +336,12 @@ The worst-case in-memory size of an LSM-tree is $`O(n)`$.
   described in reference to the size of the database in [*memory
   pages*](https://en.wikipedia.org/wiki/Page_%28computer_memory%29 "https://en.wikipedia.org/wiki/Page_%28computer_memory%29").
 
-  `OrdinaryIndex`  
+  `OrdinaryIndex`
   An ordinary index stores the maximum serialised key for each memory
   page. The total in-memory size of all indexes is proportional to the
   average size of one serialised key per memory page.
 
-  `CompactIndex`  
+  `CompactIndex`
   A compact index stores the 64 most significant bits of the minimum
   serialised key for each memory page, as well as 1 bit per memory page
   to resolve clashes, 1 bit per memory page to mark overflow pages, and
@@ -361,7 +361,7 @@ is the in-memory write buffer and all following levels are sequences of
 on-disk runs. Each level has a maximum size. The maximum size of the
 write buffer is determined by the configuration parameter
 `confWriteBufferAlloc`. The maximum size of every other level $`l`$ is
-$`l \times T \times B`$. The constant $`B`$ refers to the write buffer
+$`l \cdot T \cdot B`$. The constant $`B`$ refers to the write buffer
 size and the constant $`T`$ refers to the size ratio. (See
 [Performance](#performance "#performance").)
 

lsm-tree.cabal

@@ -185,15 +185,78 @@ description:
   The total size of an LSM-tree must not exceed \(2^{41}\) physical entries.
   Violation of this condition /is/ checked and will throw a 'TableTooLargeError'.
 
-  === Fine-tuning #fine_tuning#
+  === Fine-tuning Table Layout #fine_tuning#
+
+  The configuration parameters @confMergePolicy@, @confMergeSchedule@, @confSizeRatio@, and @confWriteBufferAlloc@ affect the way in which the table organises its data.
+  To understand what effect these parameters have, one must have a basic understand of how an LSM-tree stores its data.
+  An LSM-tree stores key–operation pairs, which pair a key with an operation such as an @Insert@ with a value or a @Delete@.
+  These key–operation pairs are organised into /runs/, which are sequences of key–operation pairs sorted by their key.
+  Runs are organised into /levels/, which are unordered sequences or runs.
+  Levels are organised hierarchically.
+  Level 0 is kept in memory, and is referred to as the /write buffer/.
+  All subsequent levels are stored on disk, with each run stored in its own file.
+  The following shows an example LSM-tree layout, with each run as a boxed sequence of keys and each level as a row.
 
+  \[
+  \begin{array}{l:l}
+  \text{Level}
+  &
+  \text{Data}
+  \\
+  0
+  &
+  \fbox{\(\texttt{4}\,\_\)}
+  \\
+  1
+  &
+  \fbox{\(\texttt{1}\,\texttt{3}\)}
+  \quad
+  \fbox{\(\texttt{2}\,\texttt{7}\)}
+  \\
+  2
+  &
+  \fbox{\(\texttt{0}\,\texttt{2}\,\texttt{3}\,\texttt{4}\,\texttt{5}\,\texttt{6}\,\texttt{8}\,\texttt{9}\)}
+  \end{array}
+  \]
+
+  The data in an LSM-tree is /partially sorted/: only the key–operation pairs within each run are sorted and deduplicated.
+  As a rule of thumb, keeping more of the data sorted means lookup operations are faster but update operations are slower.
+
+  The configuration parameters @confMergePolicy@, @confSizeRatio@, and @confWriteBufferAlloc@ directly affect the table layout.
+  Let \(B\) refer to the value of @confWriteBufferAlloc@.
+  Let \(T\) refer to the value of @confiSizeRatio@.
+  The write buffer can contain at most \(B\) entries.
+  The size ratio \(T\) determines the ratio between the maxmimum number of entries in each level.
+  For instance, if \(B = 2\) and \(T = 2\), then
+
+  \[
+  \begin{array}{l:l}
+  \text{Level}   & \text{Maximum Size}
+  \\
+  0              & B \cdot T^0 = 2
+  \\
+  1              & B \cdot T^1 = 4
+  \\
+  2              & B \cdot T^2 = 8
+  \\
+  \ell           & B \cdot T^\ell
+  \end{array}
+  \]
+
+  The @confSizeRatio@ determines the /size ratio/ between subsequent levels.
+  The write buffer can contain at most \(B\) entries.
+  This limit is determined by the @TableConfig@ parameter .
+  Level 1 can contain at
+
+
+  In the example below, each run is shown as a boxed sequence of keys, e.g., \(\fbox{\(\texttt{1}\,\texttt{3}\)}\), without showing the operation.
   An LSM-tree stores its data in a partially-sorted structure.
   The key–operation pairs are stored in /runs/, which are sorted sequences of key–operation pairs.
   The runs are organised in /levels/.
   The 0th level is the in-memory write buffer and all following levels are sequences of on-disk runs.
   Each level has a maximum size.
   The maximum size of the write buffer is determined by the configuration parameter @confWriteBufferAlloc@.
-  The maximum size of every other level \(l\) is \(l \times T \times B\).
+  The maximum size of every other level \(l\) is \(l \cdot T \cdot B\).
   The constant \(B\) refers to the write buffer size and the constant \(T\) refers to the size ratio.
   (See [Performance](#performance).)