Skip to content

Commit cf3f02c

Browse files
fcostaoliveirafcostaoliveira
committed
Merge remote-tracking branch 'origin/master' into ubsan
2 parents e267e39 + 045f75b commit cf3f02c

File tree

7 files changed

+275
-125
lines changed

7 files changed

+275
-125
lines changed

‎.github/workflows/tsan.yml‎

Lines changed: 90 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,90 @@
1+
name: TSAN (ThreadSanitizer)
2+
3+
# Data race detection using ThreadSanitizer
4+
# This workflow builds memtier_benchmark with TSAN enabled and runs
5+
# tests to detect data races and threading issues.
6+
#
7+
# NOTE: TSAN currently detects known data races in the codebase.
8+
# This workflow is informational and will not fail the build.
9+
# See: https://github.com/google/sanitizers/issues/1716 for TSAN/ASLR issues
10+
11+
on: [push, pull_request]
12+
13+
jobs:
14+
test-with-thread-sanitizer:
15+
runs-on: ubuntu-latest
16+
name: Data race detection (TSAN)
17+
continue-on-error: true # Don't fail build on known races
18+
steps:
19+
- name: Checkout code
20+
uses: actions/checkout@v4
21+
22+
- name: Install build dependencies
23+
run: |
24+
sudo apt-get -qq update
25+
sudo apt-get install -y \
26+
build-essential \
27+
autoconf \
28+
automake \
29+
pkg-config \
30+
libevent-dev \
31+
zlib1g-dev \
32+
libssl-dev
33+
34+
- name: Build with Thread Sanitizer
35+
run: |
36+
autoreconf -ivf
37+
./configure --enable-thread-sanitizer
38+
make -j
39+
40+
- name: Verify TSAN is enabled
41+
run: |
42+
ldd ./memtier_benchmark | grep tsan
43+
echo "✓ ThreadSanitizer is linked"
44+
45+
- name: Setup Python
46+
uses: actions/setup-python@v2
47+
with:
48+
python-version: '3.10'
49+
architecture: x64
50+
51+
- name: Install Python test dependencies
52+
run: pip install -r ./tests/test_requirements.txt
53+
54+
- name: Install Redis
55+
run: |
56+
curl -fsSL https://packages.redis.io/gpg | sudo gpg --dearmor -o /usr/share/keyrings/redis-archive-keyring.gpg
57+
echo "deb [signed-by=/usr/share/keyrings/redis-archive-keyring.gpg] https://packages.redis.io/deb $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/redis.list
58+
sudo apt-get -qq update
59+
sudo apt-get install redis
60+
sudo service redis-server stop
61+
62+
- name: Increase connection limit
63+
run: |
64+
sudo sysctl -w net.ipv4.tcp_fin_timeout=10
65+
sudo sysctl -w net.ipv4.tcp_tw_reuse=1
66+
ulimit -n 40960
67+
68+
- name: Generate TLS test certificates
69+
run: ./tests/gen-test-certs.sh
70+
71+
- name: Test OSS TCP with TSAN
72+
timeout-minutes: 15
73+
run: |
74+
# Use setarch to disable ASLR (workaround for TSAN on kernel 6.6+)
75+
# Use suppression file to ignore known benign races
76+
export TSAN_OPTIONS="suppressions=$(pwd)/tsan_suppressions.txt"
77+
setarch `uname -m` -R bash -c './tests/run_tests.sh'
78+
79+
- name: Test OSS TCP TLS with TSAN
80+
timeout-minutes: 15
81+
run: |
82+
export TSAN_OPTIONS="suppressions=$(pwd)/tsan_suppressions.txt"
83+
setarch `uname -m` -R bash -c 'TLS=1 ./tests/run_tests.sh'
84+
85+
- name: Test OSS-CLUSTER TCP with TSAN
86+
timeout-minutes: 15
87+
run: |
88+
export TSAN_OPTIONS="suppressions=$(pwd)/tsan_suppressions.txt"
89+
setarch `uname -m` -R bash -c 'OSS_STANDALONE=0 OSS_CLUSTER=1 ./tests/run_tests.sh'
90+

‎DEVELOPMENT.md‎

Lines changed: 142 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,142 @@
1+
# Development Guide
2+
3+
This document provides information for developers working on memtier_benchmark.
4+
5+
## Building from Source
6+
7+
### Prerequisites
8+
9+
The following libraries are required for building:
10+
11+
* libevent 2.0.10 or newer.
12+
* OpenSSL (unless TLS support is disabled by `./configure --disable-tls`).
13+
14+
The following tools are required:
15+
* autoconf
16+
* automake
17+
* pkg-config
18+
* GNU make
19+
* GCC C++ compiler
20+
21+
### CentOS/Red Hat Linux 7 or newer
22+
23+
Use the following to install prerequisites:
24+
```
25+
$ sudo yum install autoconf automake make gcc-c++ \
26+
zlib-devel libmemcached-devel libevent-devel openssl-devel
27+
```
28+
29+
### Ubuntu/Debian
30+
31+
Use the following to install prerequisites:
32+
33+
```
34+
$ sudo apt-get install build-essential autoconf automake \
35+
libevent-dev pkg-config zlib1g-dev libssl-dev
36+
```
37+
38+
### macOS
39+
40+
To build natively on macOS, use Homebrew to install the required dependencies:
41+
42+
```
43+
$ brew install autoconf automake libtool libevent pkg-config [email protected]
44+
```
45+
46+
When running `./configure`, if it fails to find libssl it may be necessary to
47+
tweak the `PKG_CONFIG_PATH` environment variable:
48+
49+
```
50+
PKG_CONFIG_PATH=`brew --prefix [email protected]`/lib/pkgconfig ./configure
51+
```
52+
53+
### Building and Installing
54+
55+
After downloading the source tree, use standard autoconf/automake commands:
56+
57+
```
58+
$ autoreconf -ivf
59+
$ ./configure
60+
$ make
61+
$ sudo make install
62+
```
63+
64+
## Testing
65+
66+
The project includes a basic set of integration tests.
67+
68+
### Integration Tests
69+
70+
Integration tests are based on [RLTest](https://github.com/RedisLabsModules/RLTest), and specific setup parameters can be provided
71+
to configure tests and topologies (OSS standalone and OSS cluster). By default the tests will be ran for all common commands, and with OSS standalone setup.
72+
73+
To run all integration tests in a Python virtualenv, follow these steps:
74+
75+
$ mkdir -p .env
76+
$ virtualenv .env
77+
$ source .env/bin/activate
78+
$ pip install -r tests/test_requirements.txt
79+
$ ./tests/run_tests.sh
80+
81+
To understand what test options are available simply run:
82+
83+
$ ./tests/run_tests.sh --help
84+
85+
### Memory Leak Detection with Sanitizers
86+
87+
memtier_benchmark supports building with AddressSanitizer (ASAN) and LeakSanitizer (LSAN) to detect memory errors and leaks during testing.
88+
89+
To build with sanitizers enabled:
90+
91+
$ ./configure --enable-sanitizers
92+
$ make
93+
94+
To run tests with leak detection:
95+
96+
$ ASAN_OPTIONS=detect_leaks=1 ./tests/run_tests.sh
97+
98+
If memory leaks or errors are detected, tests will fail with detailed error messages showing the location of the issue.
99+
100+
To verify ASAN is enabled:
101+
102+
$ ldd ./memtier_benchmark | grep asan
103+
104+
### Undefined Behavior Detection with UBSan
105+
106+
memtier_benchmark supports building with UndefinedBehaviorSanitizer (UBSan) to detect undefined behavior such as integer overflows, null pointer dereferences, and alignment issues.
107+
108+
To build with UBSan enabled:
109+
110+
$ ./configure --enable-ubsan
111+
$ make
112+
113+
To run tests with undefined behavior detection:
114+
115+
$ UBSAN_OPTIONS=print_stacktrace=1:halt_on_error=1 ./tests/run_tests.sh
116+
117+
UBSan can be combined with ASAN for comprehensive testing:
118+
119+
$ ./configure --enable-sanitizers --enable-ubsan
120+
$ make
121+
122+
**Note:** UBSan can be used together with ASAN/LSAN, but not with ThreadSanitizer (TSAN).
123+
124+
### Data Race Detection with Thread Sanitizer
125+
126+
memtier_benchmark supports building with ThreadSanitizer (TSAN) to detect data races and threading issues.
127+
128+
To build with Thread Sanitizer enabled:
129+
130+
$ ./configure --enable-thread-sanitizer
131+
$ make
132+
133+
To run tests with race detection (requires disabling ASLR on kernel 6.6+):
134+
135+
$ TSAN_OPTIONS="suppressions=$(pwd)/tsan_suppressions.txt" setarch `uname -m` -R ./tests/run_tests.sh
136+
137+
To verify TSAN is enabled:
138+
139+
$ ldd ./memtier_benchmark | grep tsan
140+
141+
**Note:** TSAN and ASAN are mutually exclusive and cannot be used together. A suppression file (`tsan_suppressions.txt`) is provided to ignore known benign data races that do not affect correctness.
142+

‎README.md‎

Lines changed: 1 addition & 123 deletions
Original file line numberDiff line numberDiff line change
@@ -55,129 +55,7 @@ brew install memtier_benchmark
5555

5656
### Installing from source
5757

58-
#### Prerequisites
59-
60-
The following libraries are required for building:
61-
62-
* libevent 2.0.10 or newer.
63-
* OpenSSL (unless TLS support is disabled by `./configure --disable-tls`).
64-
65-
The following tools are required
66-
* autoconf
67-
* automake
68-
* pkg-config
69-
* GNU make
70-
* GCC C++ compiler
71-
72-
#### CentOS/Red Hat Linux 7 or newer
73-
74-
Use the following to install prerequisites:
75-
```
76-
$ sudo yum install autoconf automake make gcc-c++ \
77-
zlib-devel libmemcached-devel libevent-devel openssl-devel
78-
```
79-
80-
#### Ubuntu/Debian
81-
82-
Use the following to install prerequisites:
83-
84-
```
85-
$ sudo apt-get install build-essential autoconf automake \
86-
libevent-dev pkg-config zlib1g-dev libssl-dev
87-
```
88-
89-
#### macOS
90-
91-
To build natively on macOS, use Homebrew to install the required dependencies:
92-
93-
```
94-
$ brew install autoconf automake libtool libevent pkg-config [email protected]
95-
```
96-
97-
When running `./configure`, if it fails to find libssl it may be necessary to
98-
tweak the `PKG_CONFIG_PATH` environment variable:
99-
100-
```
101-
PKG_CONFIG_PATH=`brew --prefix [email protected]`/lib/pkgconfig ./configure
102-
```
103-
104-
#### Building and installing
105-
106-
After downloading the source tree, use standard autoconf/automake commands:
107-
108-
```
109-
$ autoreconf -ivf
110-
$ ./configure
111-
$ make
112-
$ sudo make install
113-
```
114-
115-
#### Testing
116-
117-
The project includes a basic set of integration tests.
118-
119-
120-
**Integration tests**
121-
122-
123-
Integration tests are based on [RLTest](https://github.com/RedisLabsModules/RLTest), and specific setup parameters can be provided
124-
to configure tests and topologies (OSS standalone and OSS cluster). By default the tests will be ran for all common commands, and with OSS standalone setup.
125-
126-
127-
To run all integration tests in a Python virtualenv, follow these steps:
128-
129-
$ mkdir -p .env
130-
$ virtualenv .env
131-
$ source .env/bin/activate
132-
$ pip install -r tests/test_requirements.txt
133-
$ ./tests/run_tests.sh
134-
135-
To understand what test options are available simply run:
136-
137-
$ ./tests/run_tests.sh --help
138-
139-
140-
**Memory leak detection with sanitizers**
141-
142-
143-
memtier_benchmark supports building with AddressSanitizer (ASAN) and LeakSanitizer (LSAN) to detect memory errors and leaks during testing.
144-
145-
To build with sanitizers enabled:
146-
147-
$ ./configure --enable-sanitizers
148-
$ make
149-
150-
To run tests with leak detection:
151-
152-
$ ASAN_OPTIONS=detect_leaks=1 ./tests/run_tests.sh
153-
154-
If memory leaks or errors are detected, tests will fail with detailed error messages showing the location of the issue.
155-
156-
To verify ASAN is enabled:
157-
158-
$ ldd ./memtier_benchmark | grep asan
159-
160-
161-
**Undefined behavior detection with UBSan**
162-
163-
164-
memtier_benchmark supports building with UndefinedBehaviorSanitizer (UBSan) to detect undefined behavior such as integer overflows, null pointer dereferences, and alignment issues.
165-
166-
To build with UBSan enabled:
167-
168-
$ ./configure --enable-ubsan
169-
$ make
170-
171-
To run tests with undefined behavior detection:
172-
173-
$ UBSAN_OPTIONS=print_stacktrace=1:halt_on_error=1 ./tests/run_tests.sh
174-
175-
UBSan can be combined with ASAN for comprehensive testing:
176-
177-
$ ./configure --enable-sanitizers --enable-ubsan
178-
$ make
179-
180-
**Note:** UBSan can be used together with ASAN/LSAN, but not with ThreadSanitizer (TSAN).
58+
For detailed instructions on building from source, running tests, and using sanitizers for development, see [DEVELOPMENT.md](DEVELOPMENT.md).
18159

18260
## Using Docker
18361

‎config_types.h‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,7 @@
2727

2828
#include <vector>
2929
#include <string>
30+
#include <atomic>
3031

3132
struct config_range {
3233
int min;
@@ -100,7 +101,7 @@ struct server_addr {
100101
struct addrinfo *m_server_addr;
101102
struct addrinfo *m_used_addr;
102103
int m_resolution;
103-
int m_last_error;
104+
std::atomic<int> m_last_error; // Atomic to prevent data race between resolve() and get_connect_info()
104105
};
105106

106107
#define KEY_PLACEHOLDER "__key__"

0 commit comments

Comments
 (0)