Skip to content

Commit c4c4ec9

Browse files
committed
docs: add threading vs. fork documentation
Bug-Url: #1213
1 parent 70498b7 commit c4c4ec9

File tree

2 files changed

+55
-0
lines changed

2 files changed

+55
-0
lines changed

docs/index.rst

+1
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@ Usage
2121

2222
usage
2323
asyncio
24+
threading
2425
iproute
2526
ndb
2627
fixtures

docs/threading.rst

+54
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
.. _threading:
2+
3+
Using library in threaded environments
4+
======================================
5+
6+
Network namespaces
7+
------------------
8+
9+
In order to run a separate socket in network namespace `A` while keeping
10+
the Python process in namespace `B`, the library performs these steps:
11+
12+
1. spawn a child process
13+
2. execute `netns.setns()` in the child
14+
3. create a socket
15+
4. send the file descriptor using `socket.send_fds()` back to the parent
16+
5. terminate the child process
17+
6. create a socket with `socket(fileno=...)`
18+
19+
As a result, in the parent process we get a socket that belongs to another
20+
network namespace, but we can use it natively like any other socket, both in
21+
sync and async way.
22+
23+
Start a child: os.fork()
24+
------------------------
25+
26+
By default, pyroute2 uses `os.fork()` to create the child. When using it
27+
in multithreaded processes, no threads will be recreated, and the child
28+
will only continue the thread where `os.fork()` was called in. This leaves
29+
the GC in a corrupted state.
30+
31+
This should not be an issue since the socket creation routine stops the GC,
32+
and doesn't rely on any shared data. But still there is some risk.
33+
34+
That's why pyroute2 provides a configuration option
35+
`config.child_process_mode`. By default it is `"fork"`, but you can change
36+
the option to `"mp"`, and then pyroute2 will use `multiprocessing` to
37+
create and control the child process.
38+
39+
Start a child: multiprocessing
40+
------------------------------
41+
42+
Using `multiprocessing` may or may not rely on `os.fork()`, it depends on
43+
`multiprocessing.set_start_method()`. In Python version < 3.14 the default
44+
is `"fork"`, but starting with 3.14, the default is `"spawn"`.
45+
46+
Using `"spawn"` is safer, but significantly slower. Beside of that, `"spawn"`
47+
implies additional limitations by pickling the target method and its
48+
arguments. This prevents passing lambdas as the target, and make impossible
49+
to pass the `libc` instance to the child process.
50+
51+
The `multiprocessing` start method is out of scope for pyroute2, thus no
52+
way to set it using `config.child_process_mode`, where you can specify
53+
only `"mp"`. You have to run `multiprocessing.set_start_method()` by
54+
yourself somewhere else in your program then.

0 commit comments

Comments
 (0)