1.0.4 Release Candidate 2 (#465)

* Implement logging abstraction (#371)

Fixes #290

* Fix pre-commit to only check staged files

* Implement account ownership check and fix paging (#383)

Fixes #222

* Remove the github action package signing step

This will be done elsewhere.

* include searched_all in error response of tx (#407)

* helper function for subscribe to ensure cleanup (#402)

* Add closed to header for all paths of ledger_data (#416)

Fixes #219

* Add custom error for malformed owner and request (#417)

Fixes #274

* Use custom malformedAddress error in ledger_entry (#419)

Fixes #272

* Return lgrIdxsInvalid error for ledger_max_index less than ledger_min_index (#339)

Fixes #263

* Update headers to use #pragma once

* Add custom error for malformed request (#414)

Fixes #276

* Return srcCurMalformed on invalid taker_pays in book_offers (#413)

Fixes #267

* Fix source_location issue on MacOSX and Debug build (#431)

Fixes #428

* Implement always adding git ref to version string (#430)

Fixes #427

* add connection counting (#433)

* Fix malformed output format over ws rpc (#426)

Fixes #405

* Remove branch name from version string (#437)

Fixes a bug from #430

* Implement cli parsing using boost::po (#436)

Fixes #367

* Update documentation and config with ssl_cert_file and ssl_key_file (#443)

Fixes #424

* Fix gateway balances to match rippled output (#441)

Fixes #271

* Update README and example config to describe start_sequence (#438)

Fixes #250

* Add copyright to top of each source file (#444)

Fixes #411

* Increase file descriptor limit (#449)

* Update readme with more log configurations (#447)

Fixes #446

* Document dos_guard in example config. Log when client surpasses rate limit (#451)

* Add unit tests for DOSGuard (#453)

Fixes #452

* Build macOS and Ubuntu 22.04 (#456)

build release/x.y.z branches

* Add time measurement profiler (#458)

Rebase

* Match format to rippled error code (#461)

Fixes #263

* Change error message to match rippled (#463)

Fixes #263

* Add requests limit to DosGuard (#462)

Fixing #448

* Set version to 1.0.4-rc2

Co-authored-by: Alex Kremer <akremer@ripple.com>
Co-authored-by: CJ Cobb <46455409+cjcobb23@users.noreply.github.com>
Co-authored-by: Francis Mendoza <francissamuelmendoza7@gmail.com>
Co-authored-by: cyan317 <120398799+cindyyan317@users.noreply.github.com>

Author: 5 people

.githooks/pre-commit

@@ -2,14 +2,16 @@
 
 exec 1>&2
 
-# format all relevant sources
-find src unittests -type f \( -name '*.cpp' -o -name '*.h' -o -name '*.ipp' \) -print0 | xargs -0 clang-format -i
+# paths to check and re-format
+sources="src unittests"
+formatter="clang-format -i"
 
-# check how many lines differ
-lines=$(git diff src unittests | wc -l)
+first=$(git diff $sources)
+find $sources -type f \( -name '*.cpp' -o -name '*.h' -o -name '*.ipp' \) -print0 | xargs -0 $formatter
+second=$(git diff $sources)
+changes=$(diff <(echo "$first") <(echo "$second") | wc -l | sed -e 's/^[[:space:]]*//')
 
-# check if there is any updated files
-if [ "$lines" != "0" ]; then
+if [ "$changes" != "0" ]; then
     cat <<\EOF
 
                                    WARNING 

.github/actions/sign/action.yml

@@ -1,9 +1,9 @@
 name: Build Clio
 on:
   push:
-    branches:  [master, release, develop, develop-next]
+    branches:  [master, release/*, develop, develop-next]
   pull_request:
-    branches:  [master, release, develop, develop-next]
+    branches:  [master, release/*, develop, develop-next]
   workflow_dispatch:
 
 jobs:
@@ -29,6 +29,7 @@ jobs:
           - suffix: rpm
             image: rippleci/clio-rpm-builder:2022-09-17
             script: rpm
+
     container:
       image: ${{ matrix.type.image }}
 
@@ -67,59 +68,63 @@ jobs:
           name: clio_tests-${{ matrix.type.suffix }}
           path: ${{ github.workspace }}/clio_tests
 
-  sign:
-    name: Sign packages
-    needs: build_clio
-    runs-on: ubuntu-20.04
-    if: github.ref == 'refs/heads/master' || github.ref == 'refs/heads/release' || github.ref == 'refs/heads/develop'
-    env:
-      GPG_KEY_B64: ${{ secrets.GPG_KEY_B64 }}
-      GPG_KEY_PASS_B64: ${{ secrets.GPG_KEY_PASS_B64 }}
+  build_dev:
+    name: ${{ matrix.os.name }} test
+    needs: lint
+    continue-on-error: ${{ matrix.os.experimental }}
     strategy:
       fail-fast: false
       matrix:
-        type:
-          - suffix: deb
-            image: ubuntu:20.04
-            script: dpkg
-          # - suffix: rpm
-          #   image: centos:7
-          #   script: rpm
-    container:
-      image: ${{ matrix.type.image }}
-    steps:
-      - uses: actions/checkout@v3
-      - name: Install dpkg-sig
-        run: |
-          apt-get update && apt-get install -y dpkg-sig gnupg
-      - name: Get package artifact
-        uses: actions/download-artifact@v3
-        with:
-          name: clio_${{ matrix.type.suffix }}_packages
-
-      - name: find packages
-        run: find . -name "*.${{ matrix.type.suffix }}"
-
-      - name: Sign packages
-        uses: ./.github/actions/sign
-
+        os:
+          - name: ubuntu-22.04
+            experimental: true
+          - name: macos-11
+            experimental: true
+          - name: macos-12
+            experimental: false
+    runs-on: ${{ matrix.os.name }}
 
-      - name: Verify the signature
-        run: |
-          set -e
-          for PKG in $(ls *.deb); do
-            gpg --verify "${PKG}"
-          done
-
-      - name: Get short SHA
-        id: shortsha
-        run: echo "::set-output name=sha8::$(echo ${GITHUB_SHA} | cut -c1-8)"
-
-      - name: Artifact signed packages
-        uses: actions/upload-artifact@v2
-        with:
-          name: signed-clio-deb-packages-${{ steps.shortsha.outputs.sha8 }}
-          path: ${{ github.workspace }}/*.deb
+    steps:
+    - uses: actions/checkout@v3
+      with:
+        path: clio
+
+    - name: Check Boost cache
+      id: boost
+      uses: actions/cache@v3
+      with:
+        path: boost
+        key: ${{ runner.os }}-boost
+
+    - name: Build boost
+      if: steps.boost.outputs.cache-hit != 'true'
+      run: |
+        curl -s -OJL "https://boostorg.jfrog.io/artifactory/main/release/1.77.0/source/boost_1_77_0.tar.gz"
+        tar zxf boost_1_77_0.tar.gz
+        mv boost_1_77_0 boost
+        cd boost
+        ./bootstrap.sh
+        if [[ ${{ matrix.os.name }} =~ mac ]];then
+          mac_flags='cxxflags="-std=c++14"'
+        fi
+        ./b2 ${mac_flags}
+
+    - name: install deps
+      run: |
+        if [[ ${{ matrix.os.name }} =~ mac ]];then
+            brew install pkg-config protobuf openssl ninja cassandra-cpp-driver bison
+        elif [[ ${{matrix.os.name }} =~ ubuntu ]];then
+            sudo apt-get -y install git pkg-config protobuf-compiler libprotobuf-dev libssl-dev wget build-essential doxygen bison flex autoconf clang-format
+        fi
+
+    - name: Build clio
+      run: |
+        export BOOST_ROOT=$(pwd)/boost
+        cd clio
+        cmake -B build
+        if ! cmake --build build -j$(nproc); then
+          echo '# 🔥${{ matrix.os.name }}🔥 failed!💥' >> $GITHUB_STEP_SUMMARY
+        fi
 
   test_clio:
     name: Test Clio

.github/workflows/build.yml

@@ -0,0 +1,11 @@
+include(CheckIncludeFileCXX)
+
+check_include_file_cxx("source_location" SOURCE_LOCATION_AVAILABLE)
+if(SOURCE_LOCATION_AVAILABLE) 
+    target_compile_definitions(clio PUBLIC "HAS_SOURCE_LOCATION")
+endif()
+
+check_include_file_cxx("experimental/source_location" EXPERIMENTAL_SOURCE_LOCATION_AVAILABLE)
+if(EXPERIMENTAL_SOURCE_LOCATION_AVAILABLE)     
+    target_compile_definitions(clio PUBLIC "HAS_EXPERIMENTAL_SOURCE_LOCATION")
+endif()

CMake/deps/SourceLocation.cmake

@@ -10,7 +10,8 @@ if(NOT googletest_POPULATED)
   add_subdirectory(${googletest_SOURCE_DIR} ${googletest_BINARY_DIR} EXCLUDE_FROM_ALL)
 endif()
 
-target_link_libraries(clio_tests PUBLIC clio gtest_main)
+target_link_libraries(clio_tests PUBLIC clio gmock_main)
+target_include_directories(clio_tests PRIVATE unittests)
 
 enable_testing()
 

CMake/deps/gtest.cmake

@@ -11,6 +11,7 @@ ExecStart=@CLIO_INSTALL_DIR@/bin/clio_server @CLIO_INSTALL_DIR@/etc/config.json
 Restart=on-failure
 User=clio
 Group=clio
+LimitNOFILE=65536
 
 [Install]
-WantedBy=multi-user.target
+WantedBy=multi-user.target

CMake/install/clio.service.in

@@ -1 +1,6 @@
-set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wno-narrowing -Wall -Werror -Wno-dangling-else")
+target_compile_options(clio  
+  PUBLIC -Wall
+         -Werror 
+         -Wno-narrowing
+         -Wno-deprecated-declarations 
+         -Wno-dangling-else)

CMake/settings.cmake

@@ -21,19 +21,18 @@ if(NOT GIT_COMMIT_HASH)
   find_package(Git)
   if(Git_FOUND)
       execute_process(COMMAND ${GIT_EXECUTABLE} rev-parse --short HEAD
-          OUTPUT_STRIP_TRAILING_WHITESPACE OUTPUT_VARIABLE gch)
-      if(gch)
-          set(GIT_COMMIT_HASH "${gch}")
-          message(STATUS "Git commit: ${GIT_COMMIT_HASH}")
-          add_definitions(-DCLIO_GIT_COMMIT_HASH="${GIT_COMMIT_HASH}")
+          OUTPUT_STRIP_TRAILING_WHITESPACE OUTPUT_VARIABLE git-ref)
+      if(git-ref)
+          set(BUILD "${git-ref}")
+          message(STATUS "Build version: ${BUILD}")
+          add_definitions(-DCLIO_BUILD="${BUILD}")
       endif()
   endif()
 endif() #git
 if(PACKAGING)
   add_definitions(-DPKG=1)
 endif()
 
-
 add_library(clio)
 target_compile_features(clio PUBLIC cxx_std_20)
 target_include_directories(clio PUBLIC src)
@@ -45,6 +44,7 @@ include(CMake/ClioVersion.cmake)
 include(CMake/deps/rippled.cmake)
 include(CMake/deps/Boost.cmake)
 include(CMake/deps/cassandra.cmake)
+include(CMake/deps/SourceLocation.cmake)
 
 target_sources(clio PRIVATE
   ## Main
@@ -99,19 +99,23 @@ target_sources(clio PRIVATE
   src/rpc/handlers/Subscribe.cpp
   # Server
   src/rpc/handlers/ServerInfo.cpp
-  # Utility
+  # Utilities
   src/rpc/handlers/Random.cpp
-  src/util/Taggable.cpp
-  src/config/Config.cpp)
+  src/config/Config.cpp
+  src/log/Logger.cpp
+  src/util/Taggable.cpp)
 
 add_executable(clio_server src/main/main.cpp)
 target_link_libraries(clio_server PUBLIC clio)
 
 if(BUILD_TESTS)
   add_executable(clio_tests
-    unittests/RPCErrors.cpp 
-    unittests/config.cpp 
-    unittests/main.cpp)
+    unittests/RPCErrors.cpp
+    unittests/Backend.cpp
+    unittests/Logger.cpp
+    unittests/Config.cpp
+    unittests/ProfilerTest.cpp
+    unittests/DOSGuard.cpp)
   include(CMake/deps/gtest.cmake)
 endif()
 

CMakeLists.txt

@@ -22,7 +22,7 @@ from which data can be extracted. The rippled node does not need to be running o
 
 ## Building
 
-Clio is built with CMake. Clio requires at least GCC-11 (C++20), and Boost 1.75.0 or later.
+Clio is built with CMake. Clio requires at least GCC-11/clang-14.0.0 (C++20), and Boost 1.75.0.
 
 Use these instructions to build a Clio executable from the source. These instructions were tested on Ubuntu 20.04 LTS.
 
@@ -72,6 +72,36 @@ server is running
 to the IP of your Clio server. This entry can take the form of a comma-separated list if
 you are running multiple Clio nodes.
 
+
+In addition, the parameter `start_sequence` can be included and configured within the top level of the config file. This parameter specifies the sequence of first ledger to extract if the database is empty. Note that ETL extracts ledgers in order and that no backfilling functionality currently exists, meaning Clio will not retroactively learn ledgers older than the one you specify. Choosing to specify this or not will yield the following behavior:
+- If this setting is absent and the database is empty, ETL will start with the next ledger validated by the network. 
+- If this setting is present and the database is not empty, an exception is thrown.
+
+In addition, the optional parameter `finish_sequence` can be added to the json file as well, specifying where the ledger can stop.
+
+To add `start_sequence` and/or `finish_sequence` to the config.json file appropriately, they will be on the same top level of precedence as other parameters (such as `database`, `etl_sources`, `read_only`, etc.) and be specified with an integer. Here is an example snippet from the config file:
+
+```json
+"start_sequence": 12345,
+"finish_sequence": 54321
+```
+
+The parameters `ssl_cert_file` and `ssl_key_file` can also be added to the top level of precedence of our Clio config. `ssl_cert_file` specifies the filepath for your SSL cert while `ssl_key_file` specifies the filepath for your SSL key. It is up to you how to change ownership of these folders for your designated Clio user. Your options include:
+- Copying the two files as root somewhere that's accessible by the Clio user, then running `sudo chown` to your user
+- Changing the permissions directly so it's readable by your Clio user
+- Running Clio as root (strongly discouraged)
+
+An example of how to specify `ssl_cert_file` and `ssl_key_file` in the config:
+
+```json
+"server":{
+    "ip": "0.0.0.0",
+    "port": 51233
+},
+"ssl_cert_file" : "/full/path/to/cert.file",
+"ssl_key_file" : "/full/path/to/key.file"
+```
+
 Once your config files are ready, start rippled and Clio. It doesn't matter which you
 start first, and it's fine to stop one or the other and restart at any given time.
 
@@ -152,9 +182,33 @@ You must:
 ## Logging
 Clio provides several logging options, all are configurable via the config file and are detailed below.
 
-`log_level`: The minimum level of severity at which the log message will be outputted.
+`log_level`: The minimum level of severity at which the log message will be outputted by default.
 Severity options are `trace`, `debug`, `info`, `warning`, `error`, `fatal`. Defaults to `info`.
 
+`log_format`: The format of log lines produced by clio. Defaults to `"%TimeStamp% (%SourceLocation%) [%ThreadID%] %Channel%:%Severity% %Message%"`.
+Each of the variables expands like so
+- `TimeStamp`: The full date and time of the log entry
+- `SourceLocation`: A partial path to the c++ file and the line number in said file (`source/file/path:linenumber`)  
+- `ThreadID`: The ID of the thread the log entry is written from
+- `Channel`: The channel that this log entry was sent to
+- `Severity`: The severity (aka log level) the entry was sent at
+- `Message`: The actual log message
+
+`log_channels`: An array of json objects, each overriding properties for a logging `channel`. 
+At the moment of writing, only `log_level` can be overriden using this mechanism.
+
+Each object is of this format:
+```json
+{
+    "channel": "Backend",
+    "log_level": "fatal"
+}
+``` 
+If no override is present for a given channel, that channel will log at the severity specified by the global `log_level`.
+Overridable log channels: `Backend`, `WebServer`, `Subscriptions`, `RPC`, `ETL` and `Performance`.
+
+> **Note:** See `example-config.json` for more details. 
+
 `log_to_console`: Enable/disable log output to console. Options are `true`/`false`. Defaults to true.
 
 `log_directory`: Path to the directory where log files are stored. If such directory doesn't exist, Clio will create it. If not specified, logs are not written to a file.
@@ -170,6 +224,11 @@ rotate the current log file. Defaults to 12 hours.
 Note, time-based log rotation occurs dependently on size-based log rotation, where if a
 size-based log rotation occurs, the timer for the time-based rotation will reset.
 
+`log_tag_style`: Tag implementation to use. Must be one of:
+- `uint`: Lock free and threadsafe but outputs just a simple unsigned integer 
+- `uuid`: Threadsafe and outputs a UUID tag
+- `none`: Don't use tagging at all
+
 ## Cassandra / Scylla Administration
 
 Since Clio relies on either Cassandra or Scylla for its database backend, here are some important considerations: