Skip to content

Commit 31663bd

Browse files
committed
modified call to multiprocessing.Pool;
now it creates min(len(policies), cpu_count) processes. This fixes #177 and replaces #179. In general it does not hurt, previous behavior was to always create cpu_count() processes, some were probably unused.
1 parent f45f39c commit 31663bd

File tree

2 files changed

+35
-1
lines changed

2 files changed

+35
-1
lines changed

cvxportfolio/simulator.py

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,7 @@
4646
import sys
4747
import time
4848
from itertools import starmap
49+
from os import cpu_count
4950
from pathlib import Path
5051

5152
import numpy as np
@@ -770,7 +771,9 @@ def backtest_many(
770771
if (not parallel) or len(policies) == 1:
771772
result = list(starmap(self._worker, zip_args))
772773
else:
773-
with Pool(initializer=_mp_init, initargs=(Lock(),)) as p:
774+
with Pool(
775+
processes=min(len(policies), cpu_count()),
776+
initializer = _mp_init, initargs = (Lock(),)) as p:
774777
result = p.starmap(self._worker, zip_args)
775778

776779
return list(result)

docs/manual.rst

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -405,6 +405,7 @@ by providing a single objective and a single list of constraints, and specifying
405405
the ``planning_horizon`` argument to, for example, 2. This simply copies the terms
406406
for each step of planning horizon.
407407

408+
.. _disk-access:
408409

409410
Disk Access
410411
-----------
@@ -485,6 +486,34 @@ internet access. If you run the test suite (by ``python -m cvxportfolio.tests``)
485486
on a computer without internet access you should see a few tests, mostly
486487
in the ``test_data.py`` module, failing, but most of the test suite will run.
487488

489+
Parallel back-testing
490+
---------------------
491+
492+
You can run multiple back-tests in parallel wit the :meth:`MarketSimulator.backtest_many`
493+
method. It takes a list of policies and returns the corresponding list of
494+
:class:`cvxportfolio.result.BacktestResult`. Also
495+
:meth:`MarketSimulator.optimize_hyperparameters` uses the same approach,
496+
to search over the space of hyper-parameters efficiently.
497+
498+
.. note::
499+
500+
It is not recommended to run multiple Cvxportfolio programs at the same time,
501+
unless you are careful to not :ref:`access on-disk storage <disk-access>`. If
502+
you want to run many back-tests at the same time, you should run a single program
503+
with :meth:`MarketSimulator.backtest_many`.
504+
505+
.. note::
506+
507+
If your Cvxportfolio program uses custom objects, for example
508+
:doc:`a forecaster <examples/user_provided_forecasters>`,
509+
and in that you call complex third party libraries, like machine-learning
510+
ones, parallel back-testing can be problematic. You should in those cases
511+
make sure to :ref:`initialize and finalize <execution-model>` all resources you use.
512+
Cvxportfolio supports the ``multiprocess`` `parallel execution library
513+
<https://multiprocess.readthedocs.io/en/latest/>`_, which may help in such
514+
cases. Simply install ``multiprocess`` in the Python environment to make
515+
Cvxportfolio use it.
516+
488517

489518
CVXPY
490519
-----
@@ -556,6 +585,8 @@ parallel back-tests, ....
556585
In the next section we explain how Cvxportfolio policy objects work,
557586
which is useful to know when extending them.
558587

588+
.. _execution-model:
589+
559590
Policy execution model
560591
----------------------
561592

0 commit comments

Comments
 (0)