Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: morpho-org/morpho-data-structures
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: v0.3
Choose a base ref
...
head repository: morpho-org/morpho-data-structures
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: main
Choose a head ref

Commits on Mar 22, 2023

  1. Copy the full SHA
    872081c View commit details

Commits on Apr 4, 2023

  1. Merge pull request #41 from morpho-labs/fix-halmos-ci

    Update CI to install the latest micro version of Python
    MathisGD authored Apr 4, 2023
    Copy the full SHA
    57696ef View commit details

Commits on Apr 20, 2023

  1. Copy the full SHA
    c0aa088 View commit details
  2. fix: format and Natspec

    makcandrov committed Apr 20, 2023
    Copy the full SHA
    66ee9e2 View commit details
  3. fix: harmonize comments

    makcandrov committed Apr 20, 2023
    Copy the full SHA
    75eda42 View commit details

Commits on Apr 21, 2023

  1. fmt: add spacing

    makcandrov committed Apr 21, 2023
    Copy the full SHA
    efc7ee6 View commit details

Commits on Apr 23, 2023

  1. perf: create variables

    makcandrov committed Apr 23, 2023
    Copy the full SHA
    7924d7e View commit details

Commits on Apr 24, 2023

  1. Copy the full SHA
    5f56075 View commit details
  2. test: remove files

    Tristan22400 committed Apr 24, 2023
    Copy the full SHA
    cc49916 View commit details
  3. fix: remplace address(0)

    Tristan22400 committed Apr 24, 2023
    Copy the full SHA
    b224193 View commit details
  4. Copy the full SHA
    75cde36 View commit details
  5. fix: update dependencies

    Tristan22400 committed Apr 24, 2023
    Copy the full SHA
    c42809c View commit details
  6. Merge pull request #115 from morpho-dao/fix/drop-npm-vulnerable-depen…

    …dencies
    
    fix: drop vulnerable dependencies
    julien-devatom authored Apr 24, 2023
    Copy the full SHA
    a1239ae View commit details
  7. Merge pull request #113 from morpho-dao/perf/align-heaps-logic

    Align `HeapOrdering` with `ThreeHeapOrdering` logic and syntax
    MathisGD authored Apr 24, 2023
    Copy the full SHA
    ba30cb2 View commit details

Commits on Apr 25, 2023

  1. Update test/TestRedBlackBinaryTreeOptimized.sol

    Co-authored-by: Merlin Egalite <44097430+MerlinEgalite@users.noreply.github.com>
    Signed-off-by: Tristan Martin - Morpho  <39400008+Tristan22400@users.noreply.github.com>
    Tristan22400 and MerlinEgalite authored Apr 25, 2023
    Copy the full SHA
    d691d32 View commit details
  2. Update test/TestRedBlackBinaryTreeOptimized.sol

    Co-authored-by: Merlin Egalite <44097430+MerlinEgalite@users.noreply.github.com>
    Signed-off-by: Tristan Martin - Morpho  <39400008+Tristan22400@users.noreply.github.com>
    Tristan22400 and MerlinEgalite authored Apr 25, 2023
    Copy the full SHA
    2ca79b2 View commit details
  3. Update test/TestRedBlackBinaryTreeOptimized.sol

    Co-authored-by: Merlin Egalite <44097430+MerlinEgalite@users.noreply.github.com>
    Signed-off-by: Tristan Martin - Morpho  <39400008+Tristan22400@users.noreply.github.com>
    Tristan22400 and MerlinEgalite authored Apr 25, 2023
    Copy the full SHA
    3f74ef9 View commit details
  4. Update test/TestRedBlackBinaryTreeOptimized.sol

    Co-authored-by: Merlin Egalite <44097430+MerlinEgalite@users.noreply.github.com>
    Signed-off-by: Tristan Martin - Morpho  <39400008+Tristan22400@users.noreply.github.com>
    Tristan22400 and MerlinEgalite authored Apr 25, 2023
    Copy the full SHA
    325cbc0 View commit details
  5. Update test/TestRedBlackBinaryTreeOptimized.sol

    Co-authored-by: Merlin Egalite <44097430+MerlinEgalite@users.noreply.github.com>
    Signed-off-by: Tristan Martin - Morpho  <39400008+Tristan22400@users.noreply.github.com>
    Tristan22400 and MerlinEgalite authored Apr 25, 2023
    Copy the full SHA
    dedd130 View commit details
  6. Update test/TestRedBlackBinaryTreeOptimized.sol

    Co-authored-by: Merlin Egalite <44097430+MerlinEgalite@users.noreply.github.com>
    Signed-off-by: Tristan Martin - Morpho  <39400008+Tristan22400@users.noreply.github.com>
    Tristan22400 and MerlinEgalite authored Apr 25, 2023
    Copy the full SHA
    f4feff3 View commit details
  7. Update test/TestRedBlackBinaryTreeOptimized.sol

    Co-authored-by: Merlin Egalite <44097430+MerlinEgalite@users.noreply.github.com>
    Signed-off-by: Tristan Martin - Morpho  <39400008+Tristan22400@users.noreply.github.com>
    Tristan22400 and MerlinEgalite authored Apr 25, 2023
    Copy the full SHA
    a0a0aac View commit details
  8. fix: merge conflict

    Tristan22400 committed Apr 25, 2023
    Copy the full SHA
    aa84ecd View commit details
  9. fix: merge conflic

    Tristan22400 committed Apr 25, 2023
    Copy the full SHA
    a9cad3c View commit details
  10. Copy the full SHA
    1672543 View commit details
  11. fix: add test

    Tristan22400 committed Apr 25, 2023
    Copy the full SHA
    dba6d60 View commit details
  12. Apply suggestions from code review

    Co-authored-by: Merlin Egalite <44097430+MerlinEgalite@users.noreply.github.com>
    Signed-off-by: Tristan Martin - Morpho  <39400008+Tristan22400@users.noreply.github.com>
    Tristan22400 and MerlinEgalite authored Apr 25, 2023
    Copy the full SHA
    c84694d View commit details
  13. Copy the full SHA
    c26fdda View commit details
  14. Copy the full SHA
    ff3d396 View commit details
  15. fix: format and Natspec

    makcandrov authored and QGarchery committed Apr 25, 2023
    Copy the full SHA
    397990a View commit details
  16. fix: harmonize comments

    makcandrov authored and QGarchery committed Apr 25, 2023
    Copy the full SHA
    af5c4e4 View commit details
  17. fmt: add spacing

    makcandrov authored and QGarchery committed Apr 25, 2023
    Copy the full SHA
    2735adb View commit details
  18. perf: create variables

    makcandrov authored and QGarchery committed Apr 25, 2023
    Copy the full SHA
    4ec54aa View commit details
  19. Copy the full SHA
    c2d0b41 View commit details

Commits on Apr 26, 2023

  1. fix: comment

    makcandrov committed Apr 26, 2023
    Copy the full SHA
    d940a8a View commit details
  2. Copy the full SHA
    80e5e52 View commit details
  3. Copy the full SHA
    0229d35 View commit details
  4. Copy the full SHA
    1acbbea View commit details
  5. Merge branch 'main' into update-document-heap-ordering

    Signed-off-by: Tristan Martin - Morpho  <39400008+Tristan22400@users.noreply.github.com>
    Tristan22400 authored Apr 26, 2023
    Copy the full SHA
    83ab9c5 View commit details
  6. Copy the full SHA
    d879ebe View commit details
  7. Copy the full SHA
    bcf4af2 View commit details
  8. fix: update yarn

    Tristan22400 committed Apr 26, 2023
    Copy the full SHA
    da807ae View commit details
  9. fix: files' formatting

    Tristan22400 committed Apr 26, 2023
    Copy the full SHA
    054af06 View commit details

Commits on Apr 27, 2023

  1. Apply suggestions from code review

    Co-authored-by: makcandrov <makcandrov@proton.me>
    Signed-off-by: Tristan Martin - Morpho  <39400008+Tristan22400@users.noreply.github.com>
    Tristan22400 and makcandrov authored Apr 27, 2023
    Copy the full SHA
    5e22440 View commit details
  2. Copy the full SHA
    64170ce View commit details
  3. Copy the full SHA
    d7e5799 View commit details
  4. Copy the full SHA
    cc43c82 View commit details
  5. Copy the full SHA
    4f84b37 View commit details
  6. Copy the full SHA
    9dddbaa View commit details
  7. Copy the full SHA
    3a6354f View commit details
  8. Copy the full SHA
    c5ffae8 View commit details
Showing with 4,446 additions and 11,506 deletions.
  1. +15 −11 .github/workflows/certora.yml
  2. +6 −1 .github/workflows/ci-foundry.yml
  3. +25 −0 .github/workflows/forge-fmt.yml
  4. +3 −0 .gitmodules
  5. +1 −12 .husky/pre-commit
  6. +0 −18 .prettierrc.json
  7. +15 −78 README.md
  8. +23 −0 certora/Makefile
  9. +0 −150 certora/applyHarnessFIFO.patch
  10. +25 −0 certora/applyHarnessFifo.patch
  11. +28 −139 certora/applyHarnessSimple.patch
  12. +12 −0 certora/confs/DllFifo.conf
  13. +12 −0 certora/confs/DllSimple.conf
  14. +98 −0 certora/helpers/MockDllFifo.sol
  15. +86 −0 certora/helpers/MockDllSimple.sol
  16. +0 −23 certora/makefile
  17. +0 −11 certora/scripts/dll-fifo.sh
  18. +0 −11 certora/scripts/dll-simple.sh
  19. +0 −11 certora/scripts/sanity-fifo.sh
  20. +0 −11 certora/scripts/sanity-simple.sh
  21. +413 −0 certora/specs/DllFifo.spec
  22. +394 −0 certora/specs/DllSimple.spec
  23. +0 −335 certora/specs/dll-fifo.spec
  24. +0 −316 certora/specs/dll-simple.spec
  25. +0 −7 certora/specs/sanity.spec
  26. +1 −0 foundry.toml
  27. +0 −37 hardhat.config.ts
  28. +1 −0 lib/halmos-cheatcodes
  29. +6 −27 package.json
  30. +1 −5 src/BucketDLL.sol
  31. +55 −95 src/DoubleLinkedList.sol
  32. +225 −211 src/Heap.sol
  33. +216 −244 src/HeapOrdering.sol
  34. +25 −39 src/LogarithmicBuckets.sol
  35. +273 −303 src/RedBlackBinaryTree.sol
  36. +252 −279 src/RedBlackBinaryTreeOptimized.sol
  37. +203 −233 src/ThreeHeapOrdering.sol
  38. +0 −1,192 test-ts/instruction.json
  39. +0 −50 test-ts/mocks/RedBlackBinaryTreeMock.sol
  40. +0 −42 test-ts/mocks/RedBlackBinaryTreeOptimizedMock.sol
  41. +0 −70 test-ts/test-print-tree.ts
  42. +0 −139 test-ts/test-red-black-binary-tree.ts
  43. +75 −88 test/TestBucketDLL.t.sol
  44. +20 −14 test/TestCommonHeapOrdering.t.sol
  45. +3 −3 test/TestDoubleLinkedList.t.sol
  46. +215 −250 test/TestHeap.t.sol
  47. +143 −0 test/TestHeapInvariant.t.sol
  48. +5 −3 test/TestHeapOrdering.t.sol
  49. +16 −16 test/TestHeapOrderingInvariant.t.sol
  50. +33 −27 test/TestLogarithmicBuckets.t.sol
  51. +2 −2 test/TestLogarithmicBucketsGas.t.sol
  52. +13 −17 test/TestLogarithmicBucketsInvariant.t.sol
  53. +190 −0 test/TestRedBlackBinaryTreeOptimized.sol
  54. +0 −102 test/TestStressHeap.t.sol
  55. +5 −3 test/TestThreeHeapOrdering.t.sol
  56. +65 −21 test/TestThreeHeapOrderingInvariant.t.sol
  57. +4 −0 test/helpers/Random.sol
  58. +10 −5 test/helpers/RandomHeap.sol
  59. +15 −17 test/mocks/BucketDLLMock.sol
  60. +58 −0 test/mocks/HeapMock.sol
  61. +25 −26 test/mocks/HeapOrderingMock.sol
  62. +18 −16 test/mocks/LogarithmicBucketsMock.sol
  63. +24 −25 test/mocks/ThreeHeapOrderingMock.sol
  64. +4 −9 test/mocks/{ → interfaces}/IHeapOrderingMock.sol
  65. +1,089 −6,762 yarn.lock
26 changes: 15 additions & 11 deletions .github/workflows/certora.yml
Original file line number Diff line number Diff line change
@@ -2,12 +2,23 @@ name: Certora

on:
push:
branches:
- main
pull_request:
workflow_dispatch:

jobs:
verify:
runs-on: ubuntu-latest

strategy:
fail-fast: false

matrix:
conf:
- DllFifo
- DllSimple

steps:
- uses: actions/checkout@v3

@@ -23,20 +34,13 @@ jobs:
run: |
wget https://github.com/ethereum/solidity/releases/download/v0.8.17/solc-static-linux
chmod +x solc-static-linux
sudo mv solc-static-linux /usr/local/bin/solc8.17
sudo mv solc-static-linux /usr/local/bin/solc-0.8.17
- name: Verify rule ${{ matrix.script }}
run: |
echo "key length" ${#CERTORAKEY}
bash certora/scripts/${{ matrix.script }} --solc solc8.17
make -C certora munged-simple
make -C certora munged-fifo
certoraRun certora/confs/${{ matrix.conf }}.conf
env:
CERTORAKEY: ${{ secrets.CERTORAKEY }}

strategy:
fail-fast: false
max-parallel: 4

matrix:
script:
- dll-fifo.sh
- dll-simple.sh
7 changes: 6 additions & 1 deletion .github/workflows/ci-foundry.yml
Original file line number Diff line number Diff line change
@@ -5,6 +5,7 @@ on:
branches:
- main
pull_request:
workflow_dispatch:

jobs:
tests:
@@ -14,6 +15,10 @@ jobs:
with:
submodules: recursive

- uses: actions/setup-python@v4
with:
python-version: "3.11"

- uses: actions/setup-node@v3
with:
node-version: 16
@@ -42,4 +47,4 @@ jobs:
run: python3 -m pip install --upgrade halmos

- name: Run halmos
run: halmos --function testProve --loop 4
run: halmos --function testProve --loop 4 --solver-timeout-assertion 0
25 changes: 25 additions & 0 deletions .github/workflows/forge-fmt.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
name: Forge format

on:
push:
branches:
- main
pull_request:

jobs:
forge-fmt:
name: Formatting
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v3
with:
submodules: recursive

- name: Install Foundry
uses: onbjerg/foundry-toolchain@v1
with:
version: nightly

- name: Check formatting
run: forge fmt --check
3 changes: 3 additions & 0 deletions .gitmodules
Original file line number Diff line number Diff line change
@@ -5,6 +5,9 @@
path = lib/forge-std
url = https://github.com/MathisGD/forge-std
branch = feat/real-gas-consumption
[submodule "lib/halmos-cheatcodes"]
path = lib/halmos-cheatcodes
url = https://github.com/a16z/halmos-cheatcodes
[submodule "lib/openzeppelin-contracts"]
path = lib/openzeppelin-contracts
url = https://github.com/OpenZeppelin/openzeppelin-contracts
13 changes: 1 addition & 12 deletions .husky/pre-commit
Original file line number Diff line number Diff line change
@@ -2,15 +2,4 @@

. "$(dirname "$0")/_/husky.sh"

STAGED_TS_FILES=$(git diff --staged --diff-filter=d --name-only | grep '\.ts$' | xargs)
STAGED_SOL_FILES=$(git diff --staged --diff-filter=d --name-only | grep '\.sol$' | xargs)

if [ -n "$STAGED_SOL_FILES" ]; then
yarn prettier --config .prettierrc.json --write $STAGED_SOL_FILES
git add $STAGED_SOL_FILES
fi

if [ -n "$STAGED_TS_FILES" ]; then
yarn prettier --config .prettierrc.json --write $STAGED_TS_FILES
git add $STAGED_TS_FILES
fi
yarn lint-staged
18 changes: 0 additions & 18 deletions .prettierrc.json

This file was deleted.

93 changes: 15 additions & 78 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,106 +1,43 @@
# Morpho Data Structures 🦋

This repository contains the data structures that can be used for the Morpho's matching engine 🦋.
The goal is to compare them in terms of security and gas consumption to find the best solution for the protocol and its users.

# Data structures

The data structures we implement and modified are based on public works of amazing developers. We thank them for what they have done 🙏

You can refer to the following table for the complexity of some data structures.

![complexities](https://devopedia.org/images/article/17/7752.1513922040.jpg)
This repository contains the data structures that are used in Morpho Optimizer's matching engine.
The data structures are built to be secure and gas efficient.

## Double Linked List

The current implementation of the double linked list is based on this [article](https://hackernoon.com/a-linked-list-implementation-for-ethereum-deep-dive-oy9432pa) written by Alberto Cuesta Cañada. You can find the repository [here](https://github.com/HQ20/contracts/tree/master/contracts/lists). Note that the code has been modified to meet our own needs and to allow us to sort the first accounts of the double linked list. Our implementation is not a generalised one.
What you can with it:
The current implementation of the double-linked list is based on this [article](https://hackernoon.com/a-linked-list-implementation-for-ethereum-deep-dive-oy9432pa) written by Alberto Cuesta Cañada. You can find the repository [here](https://github.com/HQ20/contracts/tree/master/contracts/lists). Note that the code has been modified to meet our own needs and to allow us to sort the first accounts of the double-linked list. Our implementation is not a generalized one.
What you can do with it:

- Insert an address sorted by a value passed along this address.
- Insert (and its value) before an account.

## Red Black Binary Tree

A [Red Black Binary Tree](https://en.wikipedia.org/wiki/Red%E2%80%93black_tree) is a kind of binary tree that allows insertion / deletion / search in `O(log(n))`.
A [Red Black Binary Tree](https://en.wikipedia.org/wiki/Red%E2%80%93black_tree) is a kind of binary tree that allows insertion/deletion/search in `O(log(n))`.
Our implementation is a modified version of the [OrderStatisticsTree repository](https://github.com/rob-Hitchens/OrderStatisticsTree) written by [Rob Hitechn](https://github.com/rob-Hitchens) which is also based on [BokkyPooBahsRedBlackTreeLibrary repository](https://github.com/bokkypoobah/BokkyPooBahsRedBlackTreeLibrary) written by [bokkypoobah](https://github.com/bokkypoobah).
Our modified version makes keys unique items instead of just (key, value) unique pairs.

In order to manipulate a binary tree and visualize how it manages to stay balanced, this [tool](https://www.cs.usfca.edu/~galles/visualization/RedBlack.html) is very useful. You can also find [here](http://ion.uwinnipeg.ca/~ychen2/advancedAD/Red-black%20Tree.pdf) the pseudo-code logic of the tree's function.

## Heap based ordering

This implementation is based on a heap data structure, and refines it by adding an unsorted portion after it. This gives us an approximation of a heap, and thus operations are performed in constant time. The main entry point is the `update` function, calling internally either `insert`, `increase`, `decrease` or `remove`.

## Other data structures

Other data structures may be explored in the future and we are open to any suggestions or optimisation of current implementations ⚡️

# Contributing

In this section, you will find some guidelines to read before contributing to the project.

## Setup

Update git submodules:

```
git submodule update --init --recursive
```

Run yarn:
This implementation is based on a heap data structure and refines it by adding an unsorted portion after it. This gives us an approximation of a heap, and thus operations are performed in constant time.

```
yarn
```
At least the first `maxSortedUsers / 2` addresses are sorted in the Heap. To keep constant time operation, we divide by two the size of the Heap once the size overtakes the `maxSortedUsers` number. It means that we remove all leaves of the heap.

## Testing
The choice of this implementation is explained by the desire to store a maximum of high-value nodes in the heap to use them for peer-to-peer matching.
Indeed, a naive implementation that would remove the tail and insert new values at `maxSortedUsers` (once the heap is full), would end up concentrating all new values on the same single path from the leaf to the root node because the `shiftUp` function will be always called from the same node. The risk is that low-value nodes stay in the Heap and that all the liquidity will be concentrated on the path from the leaf of index 'maxSortedUsers' to the root.
Removing all the leaves enables the protocol to remove low-value nodes and to call the `shiftUp` function at different locations in the Heap. This process is meant to keep a maximum of liquidity available in the heap for peer-to-peer lending.
The main entry point is the `update` function, calling internally either `insert`, `increase`, `decrease` or `remove`.

The tests can be run with [foundry](https://github.com/foundry-rs/foundry).

For the `RedBlackBinaryTree`, you can run the tests with hardhat with `yarn test`.

## Creating issues and PRs

Guidelines for creating issues and PRs:

- Issues must be created and labelled with relevant labels (type of issues, high/medium/low priority, etc.).
- Nothing should be pushed directly to the `main` branch.
- Pull requests must be created before and branch names must follow this pattern: `feat/<feature-name>`, `test/<test-name>` or `fix/<fix-name>`. `docs`, `ci` can also be used. The goal is to have clear branches names and make easier their management.
- PRs must be labelled with the relevant labels.
- Issues must be linked to PRs so that once the PR is merged related issues are closed at the same time.
- Reviewers must be added to the PR.
- For commits, we use the bitmoji VS Code extension 🙃

## Before merging a PR

Before merging a PR:

- PR must have been reviewed by reviewers. The must deliver a complete report on the smart contracts (see the section below).
- Comments and requested changes must have been resolved.
- PR must have been approved by every reviewers.
- CI must pass.

## Code Formatting

We use prettier with the default configuration mentionned in the [Solidity Prettier Plugin](https://github.com/prettier-solidity/prettier-plugin-solidity).
We recommend developers using VS Code to set their local config as below:

```
{
"editor.formatOnSave": true,
"solidity.formatter": "prettier",
"editor.defaultFormatter": "esbenp.prettier-vscode"
}
```

In doing so the code will be formatted on each save.
## Other data structures

We use Husky hook to format code before being pushed to any remote branch to enforce coding style among all developers.
Other data structures may be explored in the future and we are open to any suggestions or optimization of current implementations ⚡️

# Audits

The code concerning the [heap based ordering data-structure](./contracts/HeapOrdering.sol) has been audited by [Omniscia](https://omniscia.io) and the report can be found [online](https://omniscia.io/reports/morpho-heap-ordering-structure/) or in the file [Morpho_Omniscia](./audits/Morpho_Omniscia.pdf).

# Questions
# Licenses

For any question you can send an email to [merlin@mopho.best](mailto:merlin@morpho.best) 😊
The data structures are licensed under `AGPL-3.0-only`, see [LICENSE](./LICENSE).
23 changes: 23 additions & 0 deletions certora/Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
help:
@echo "usage: two possible munging, either simple DLL of DLL with FIFO"

munged-simple: $(wildcard ../src/*.sol) applyHarnessSimple.patch
@rm -rf munged-simple
@cp -r ../src munged-simple
@patch -p0 -d munged-simple < applyHarnessSimple.patch

record-simple:
diff -ruN ../src munged-simple | sed -E 's,\.\./src/|munged-simple/,,g' | sed -E 's,((\-\-\-|\+\+\+) [^[:space:]]*).*,\1,' > applyHarnessSimple.patch

munged-fifo: $(wildcard ../src/*.sol) applyHarnessFifo.patch
@rm -rf munged-fifo
@cp -r ../src munged-fifo
@patch -p0 -d munged-fifo < applyHarnessFifo.patch

record-fifo:
diff -ruN ../src munged-fifo | sed -E 's,\.\./src/|munged-fifo/,,g' | sed -E 's,((\-\-\-|\+\+\+) [^[:space:]]*).*,\1,' > applyHarnessFifo.patch

clean:
rm -rf munged-simple munged-fifo

.PHONY: help clean record-simple record-fifo # do not add munged folders here, as it is useful to protect munged edits
Loading