1.102.1

Author: cjdsellers

.flake8

@@ -24,7 +24,7 @@ exclude =
    dist
    docs
    noxfile.py
-ignore = E225, E227, E252, E999, D100, D101, D102, D104, D200
+ignore = E225, E227, E252, E999
 max-complexity = 10
 max-line-length = 150
 statistics = True
@@ -35,11 +35,3 @@ statistics = True
 # E227 missing whitespace around bitwise or shift operator (picks up pointers)
 # E252 spaces around default argument assignment (incorrect syntax)
 # E999 SyntaxError: invalid syntax (cimport seen as invalid syntax)
-
-# Documentation ignores (will be addressed)
-# -----------------------------------------
-# D100 Missing docstring in public module (todo: picks up entire test suite)
-# D101 Missing docstring in public class  (todo: picks up entire test suite)
-# D102 Missing docstring in public class  (todo: picks up entire test suite)
-# D104 Missing docstring in public package (todo to complete)
-# D200 One-line docstring should fit on one line with quotes (conflicts with Codacy)

.github/workflows/test.yml

@@ -18,7 +18,6 @@ jobs:
       fail-fast: false
       matrix:
         os: [ ubuntu-latest, macos-latest ]
-        rust: [ stable ]
         python-version: [ 3.7, 3.8, 3.9 ]
     name: test - Python ${{ matrix.python-version }} (${{ matrix.os }})
     runs-on: ${{ matrix.os }}

CONTRIBUTING.md

@@ -16,11 +16,7 @@ To contribute, the following steps should be followed;
   your local machine. This will automatically run various checks, auto-formatters
   and linting tools. Further information can be found here https://pre-commit.com/.
 
-- Install the latest version of isort `pip install -U isort` (used in the pre-commit).
-
-- Before committing it's a good practice to run `pre-commit run --all-files` yourself.
-
-- It's recommended you install _Redis_ with a default configuration, so that integration
+- It's recommended you install Redis using the default configuration, so that integration
   tests will pass on your machine.
 
 - Open a pull request (PR) on the `develop` branch with a summary comment.

README.md

@@ -23,6 +23,9 @@ NautilusTrader is an open-source, high-performance, production-grade algorithmic
 providing quantitative traders with the ability to backtest portfolios of automated trading strategies
 on historical data with an event-driven engine, and also deploy those same strategies live.
 
+NautilusTrader is AI/ML first, designed to deploy models for algorithmic trading strategies developed
+using the Python ecosystem - within a highly performant and robust Python native environment.
+
 The platform aims to be universal, with any REST/FIX/WebSocket API able to be integrated via modular
 adapters. Thus the platform can handle high-frequency trading operations for any asset classes
 including FX, Equities, Futures, Options, CFDs and Crypto - across multiple venues simultaneously.
@@ -36,14 +39,36 @@ including FX, Equities, Futures, Options, CFDs and Crypto - across multiple venu
 - **Multi-venue:** Multiple venue capabilities facilitate market making and statistical arbitrage strategies.
 - **AI Agent Training:** Backtest engine fast enough to be used to train AI trading agents (RL/ES).
 
-## Values
+## Why NautilusTrader?
 
-- Reliability
-- Performance
-- Testability
-- Modularity
-- Maintainability
-- Scalability
+One of the key value propositions of NautilusTrader is that it addresses the challenge of keeping
+the research/backtest environment consistent with the production live trading environment.
+
+Normally research and backtesting may be conducted in Python (or other suitable language), with
+trading strategies traditionally then needing to be reimplemented in C++/C#/Java or other statically
+typed language(s). The reasoning here is to enjoy the performance a compiled language can offer,
+along with the tooling and support which has made these languages historically more suitable for
+large enterprise systems.
+
+The value of NautilusTrader here is that this re-implementation step is circumvented, as the
+platform was designed from the ground up to hold its own in terms of performance and quality.
+
+Python has simply caught up in performance (via Cython offering C-level speed) and general tooling,
+making it a suitable language for implementing a large system such as this. The benefit being
+that a Python native environment can be offered, suitable for professional quantitative traders and
+hedge funds.
+
+## Why Python?
+
+Python was originally created decades ago as a simple scripting language with a clean straight
+forward syntax. It has since evolved into a fully fledged general purpose object-oriented
+programming language. Not only that, Python has become the _de facto lingua franca_ of data science,
+machine learning, and artificial intelligence.
+
+The language out of the box is not without its drawbacks however, especially in the context of
+implementing large systems. Cython has addressed a lot of these issues, offering all the advantages
+of a statically typed language, embedded into Pythons rich ecosystem of software libraries and
+developer/user communities.
 
 ## What is Cython?
 
@@ -55,6 +80,15 @@ The project heavily utilizes Cython to provide static type safety and increased
 for Python through C [extension modules](https://docs.python.org/3/extending/extending.html). The vast majority of the production Python code is actually
 written in Cython, however the libraries can be accessed from both pure Python and Cython.
 
+## Values
+
+- Reliability
+- Performance
+- Testability
+- Modularity
+- Maintainability
+- Scalability
+
 ## Documentation
 
 The documentation for the latest version of the package is available at _readthedocs_.
@@ -189,29 +223,42 @@ exchanges.
 
 ## Development
 
-For development of the Python codebase, we recommend using the PyCharm _Professional_ edition IDE, as
-it interprets Cython syntax. Alternatively, you could use Visual Studio Code with a Cython extension.
+For development we recommend using the PyCharm _Professional_ edition IDE, as it interprets Cython
+syntax. Alternatively, you could use Visual Studio Code with a Cython extension.
 
 `poetry` is the preferred tool for handling all Python package and dev dependencies.
 
 > https://python-poetry.org/
 
+`pre-commit` is used to automatically run various checks, auto-formatters and linting tools
+at commit.
+
+> https://pre-commit.com/
+
 #### Environment Setup
 
 The following steps are for Unix-like systems, and only need to be completed once.
 
-1. Install the Cython package by running:
+1. Install the pre-commit package:
+
+        pip install pre-commit
+
+2. Install the Cython package by running:
 
         pip install -U Cython==3.0a6
 
-2. Install `poetry` by running:
+3. Install `poetry` by running:
 
         curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python -
 
-3. Then install all Python package dependencies, and compile the Rust libs and Python C extensions by running:
+4. Then install all Python package dependencies, and compile the C extensions by running:
 
         poetry install
 
+5. Setup the pre-commit hook which will then run automatically at commit:
+
+        pre-commit run --all-files
+
 #### Builds
 
 Following any changes to `.pyx` or `.pxd` files, you can re-compile by running:

docs/source/developer_guide/coding_standards.rst

@@ -1,21 +1,88 @@
 Coding Standards
 ================
 
-- PEP-8 and variations
-- Cython specifics
-- NumPy docstrings
-- flake8 usage and ignores
-- pre-commit hook setup and usage
-- isort usage
-- darglint usage
-
 Code Style
 ----------
-
 `Black` is a PEP-8 compliant opinionated formatter.
 
 > https://github.com/psf/black
 
-We philosophically agree with `Black`, however it does not currently run over
-Cython code. So you could say we are "handcrafting towards" `Blacks` stylistic
-conventions.
+We philosophically agree with the `Black` formatting style, however it does not
+currently run over Cython code. So you could say we are "handcrafting towards"
+`Blacks` stylistic conventions.
+
+The current codebase can be used as a guide for formatting guidance.
+
+- For longer lines of code, and when passing more than a couple of arguments -
+it's common to take a new line which aligns at the next logical indent (rather
+than attempting a hanging alignment off an opening parenthesis).
+
+- The closing parenthesis should be located on a new line, aligned at the logical
+indent.
+
+- Also ensure multiple hanging parameters or arguments end with a comma `,`::
+
+    LongCodeLine(
+        some_arg1,
+        some_arg2,
+        some_arg3,  # <-- ending comma
+    )
+
+
+PEP-8
+-----
+The codebase generally follows the PEP-8 style guide.
+
+One notable departure is that Python `truthiness` is not always taken advantage
+of to check if an argument is `None`, or if a collection is empty/has elements.
+
+There are two reasons for this;
+
+1- Cython can generate more efficient C code from `is None` and `is not None`,
+rather than entering the Python runtime to check the `PyObject` truthiness.
+
+2- As per the `Google Python Style Guide` it's discouraged to use truthiness to
+check if an argument is/is not None, when there is a chance an unexpected object
+could be passed into the function or method which will yield an unexpected
+truthiness evaluation - which could result in a logical error type bug.
+
+_"Always use if foo is None: (or is not None) to check for a None value.
+E.g., when testing whether a variable or argument that defaults to None was set
+to some other value. The other value might be a value that’s false in a boolean
+context!"_
+
+> https://google.github.io/styleguide/pyguide.html
+
+Having said all of this there are still areas of the codebase which aren't as
+performance-critical where it is safe to use Python truthiness to check for None,
+or if a collection is empty/has elements.
+
+We welcome all feedback on where the codebase departs from PEP-8 for no apparent
+reason.
+
+NumPy Docstrings
+----------------
+The NumPy docstring syntax is used throughout the codebase. This needs to be
+adhered to consistently to ensure the docs build correctly during pushes to the
+`master` branch.
+
+> https://numpydoc.readthedocs.io/en/latest/format.html
+
+Flake8
+------
+`flake8` is utilized to lint the codebase. Current ignores can be found in the
+`.flake8` config file, the majority of which are required so that valid Cython
+code is not picked up as flake8 failures.
+
+Cython
+------
+Ensure that all functions and methods returning `void` or a primitive C type
+(such as `bint`, `int`, `double`) include the `except *` keyword in the signature.
+
+This will ensure Python exceptions are not ignored, but instead are `bubbled up`
+to the caller as expected.
+
+More information on Cython syntax and conventions can be found by reading the
+Cython docs.
+
+> https://cython.readthedocs.io/en/latest/index.html

docs/source/developer_guide/developing_adapters.rst

@@ -1,2 +1,4 @@
 Developing Adapters
 ===================
+
+WIP

docs/source/developer_guide/environment.rst

@@ -1,2 +1,46 @@
 Environment
 ===========
+
+For development we recommend using the PyCharm _Professional_ edition IDE, as it
+interprets Cython syntax. Alternatively, you could use Visual Studio Code with
+a Cython extension.
+
+`poetry` is the preferred tool for handling all Python package and dev dependencies.
+
+> https://python-poetry.org/
+
+`pre-commit` is used to automatically run various checks, auto-formatters and linting tools
+at commit.
+
+> https://pre-commit.com/
+
+Setup
+-----
+The following steps are for Unix-like systems, and only need to be completed once.
+
+1. Install pre-commit:
+
+        pip install pre-commit
+
+2. Install the Cython package by running:
+
+        pip install -U Cython==3.0a6
+
+3. Install `poetry` by running:
+
+        curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python -
+
+4. Then install all Python package dependencies, and compile the C extensions by running:
+
+        poetry install
+
+5. Setup the pre-commit hook which will then run automatically at commit:
+
+        pre-commit run --all-files
+
+Builds
+------
+
+Following any changes to `.pyx` or `.pxd` files, you can re-compile by running:
+
+    python build.py

docs/source/developer_guide/overview.rst renamed to docs/source/developer_guide/packaged_data.rst

@@ -1,18 +1,6 @@
-Overview
-========
-
-We recommend the PyCharm Professional edition IDE as it interprets Cython syntax.
-Unfortunately the Community edition will not interpret Cython syntax.
-
-> https://www.jetbrains.com/pycharm/
-
-To run code from source, first compile the C extensions for the package::
-
-    python build.py
-
-
 Packaged Data
--------------
+=============
+
 Various data is contained internally in the `tests/test_kit/data` folder.
 
 Libor Rates

docs/source/developer_guide/testing.rst

@@ -1,17 +1,59 @@
 Testing
 =======
 
-To run tests::
+The test suite is divided into broad categories of tests including;
+    - Unit tests
+    - Integration tests
+    - Acceptance/System tests
+    - Performance tests
+
+The performance tests are not run as part of the CI pipeline, but exist to aid
+development of performance-critical components.
+
+Tests can be run using either Pytest or the Nox tool.
+
+If you're using PyCharm then tests should run directly by right clicking on the
+respective folder (or top level tests folder) and clicking 'Run pytest'.
+
+Alternatively you can use the `pytest .` command from the root level `tests`
+folder, or the other sub folders.
+
+Nox
+---
+Nox sessions are defined within the `noxfile.py`, to run various test collections.
+
+To run unit tests with nox::
 
     nox -s tests
 
 
-If you have `redis-server` up you can run integration tests::
+If you have `redis-server` up you can run integration tests with nox::
 
     nox -s integration_tests
 
 
-- Philosophy
-- Unit Tests
-- Integration Tests
-- Acceptance Tests
+Or both unit and integration tests::
+
+    nox -s tests_with_integration
+
+Mocks
+-----
+Unit tests will often include other components acting as mocks. The intent of
+this is to simplify the test suite to avoid extensive use of a mocking framework,
+although `MagicMock` objects are currently used in particular cases.
+
+Code Coverage
+-------------
+Code coverage output is generated using `coverage` and reported using `codecov`.
+
+High test coverage is a goal for the project however not at the expense of
+appropriate error handling, or causing "test induced damage" to the architecture.
+
+There are currently areas of the codebase which are 'impossible' to test unless
+there is a change to the production code. For example the last condition check
+of an if-else block which would catch an unrecognized value, these should be
+left in place in case there is a change to the production code - which these
+checks could then catch.
+
+Other 'design-time' exceptions may also be impossible to test for, and so 100%
+test coverage is not the ultimate goal.