Skip to content

Commit 7d3d63e

Browse files
pbhandar2meta-codesync[bot]
pbhandar2
authored and
meta-codesync[bot]
committed
Remove old build scripts in doc
Summary: Delete the mention of old build scripts in the doc Reviewed By: rlyerly Differential Revision: D105227000 fbshipit-source-id: 4eeb816dd65b7d598d6e97f024290d59cba4531d
1 parent c99f331 commit 7d3d63e

2 files changed

Lines changed: 126 additions & 201 deletions

File tree

‎BUILD.md‎

Lines changed: 109 additions & 195 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,12 @@
11
# Building CacheLib
22

3+
CacheLib uses `getdeps.py` for building, which is used by many of Meta's OSS tools. This script will download and build all of the necessary dependencies first, and will then invoke cmake to build CacheLib. This ensures that you build with relevant versions of all dependent libraries, taking into account what versions are installed locally on your system.
4+
35
## Dependencies
46

5-
CacheLib depends on multiple libraries and programs.
6-
Some are available as system packages, and others need
7-
to be build from source.
7+
CacheLib depends on multiple libraries and programs. Some are available as system packages, and others need to be built from source.
88

9-
The primary dependecies are:
9+
The primary dependencies are:
1010

1111
* a C++20 compiler (tested with GCC, CLANG)
1212
* [https://cmake.org/](CMake)
@@ -26,260 +26,174 @@ These dependencies further require multiple libraries:
2626
* And many more libraries, commonly available as installable packages, e.g:
2727
`boost`, `libevent`, `lz4`, `snappy`, `zlib`, `ssl`, `libunwind`, `libsodium`
2828

29-
Currently, some dependencies can be easily installed using the system's
30-
package manager (e.g. `dnf`/`yum`/`apt`), while others need to be rebuild
31-
from source code.
32-
33-
34-
## Build Script
29+
## Building with getdeps.py
3530

36-
CacheLib provides a build script which prepares and installs all
37-
dependencies and prerequisites, then builds CacheLib.
38-
The build script has been tested to work on CentOS 8,
39-
Ubuntu 18.04, and Debian 10.
31+
### Step 1 - Clone the repository
4032

4133
```sh
42-
git clone https://github.com/facebookincubator/CacheLib
34+
git clone https://github.com/facebook/CacheLib
4335
cd CacheLib
44-
./contrib/build.sh -d -j -v
45-
46-
# The resulting library and executables:
47-
./build-cachelib/cachebench/cachebench --help
4836
```
4937

50-
Re-running `./contrib/build.sh` will update CacheLib and its dependencies
51-
to their latest versions and rebuild them.
38+
### Step 2 - Install system dependencies
5239

53-
The build script supports the following options:
40+
You can install system dependencies to save building them:
5441

5542
```sh
56-
$ ./contrib/build.sh -h
57-
CacheLib dependencies builder
58-
59-
usage: build.sh [-BdhijOStv]
60-
61-
options:
62-
-d build with DEBUG configuration
63-
(default is RELEASE with debug information)
64-
-h This help screen
65-
-j build using all available CPUs ('make -j')
66-
(default is to use single CPU)
67-
-O skip OS package installation (apt/yum/dnf)
68-
-S skip git-clone/git-pull step
69-
-t build tests
70-
(default is to skip tests if supported by the package)
71-
-v verbose build
43+
sudo python3 ./build/fbcode_builder/getdeps.py install-system-deps --recursive cachelib
7244
```
7345

46+
If you'd like to see the packages before installing them:
7447

75-
## Build Process Details
76-
77-
The build process involves the following steps.
78-
These steps can be run manually for troubleshooting and/or
79-
adapting the build to a new system. The wrapper script `./contrib/build.sh`
80-
performs them one by one:
48+
```sh
49+
python3 ./build/fbcode_builder/getdeps.py install-system-deps --dry-run --recursive cachelib
50+
```
8151

82-
#### Step 1 - System packages
52+
On platforms without system dependencies, `getdeps.py` will download and build them for you during the build step.
8353

84-
Installs the suitable tools and packages for the operating
85-
system flavor and version (e.g. Debian 10).
86-
This step requires `sudo`, and uses one of the following scripts:
87-
`contrib/prerequisites-centos8.sh`,
88-
`contrib/prerequisites-debian10.sh`,
89-
`contrib/prerequisites-ubuntu18.sh`.
54+
### Step 3 - Build CacheLib
9055

91-
For Debian/Ubuntu it is a simple matter of running `apt-get` with
92-
a known list of packages. For CentOS, the script first adds
93-
the `Power Tools` repository (required for some of the packages).
56+
`getdeps.py` currently requires python 3.6+ to be on your path.
9457

95-
It is safe to re-run these scripts - if the required packages are
96-
already installed, the script will terminate quickly.
58+
```sh
59+
# Build, using system dependencies if available
60+
python3 ./build/fbcode_builder/getdeps.py --allow-system-packages build cachelib
61+
```
9762

63+
This command will:
64+
1. Download and build all necessary dependencies (folly, fbthrift, wangle, fizz, etc.)
65+
2. Build CacheLib with the appropriate configuration
66+
3. Install the built artifacts to a scratch directory
9867

99-
#### Step 2 - Update Git-Submodules
68+
The build may take several minutes on the first run as it compiles all dependencies. Subsequent builds will be much faster as only changed components are rebuilt.
10069

101-
The CacheLib project includes several library as git-submodules
102-
(folly,fbthrift,wangle,fizz).
103-
Due to the way internal facebook projects are
104-
converted to git and exported to github, the updating process
105-
is slightly more complicated than a simple `git submodule update`.
70+
## Build Configurations
10671

107-
The script `./contrib/update-submodules.sh` performs the required steps
108-
to synchronize the required git revisions.
72+
By default, `getdeps.py` builds in Release mode with debug information. To change the configuration, pass `--build-type` (e.g., `--build-type Debug`) to the `build` command. Run `python3 ./build/fbcode_builder/getdeps.py build --help` for the full list of build options.
10973

110-
It is safe to re-run the `update-submodules.sh` script - it will simply
111-
pull the latest changes (if any).
74+
## Running Tests
11275

76+
By default `getdeps.py` will build the tests for CacheLib. To run them:
11377

114-
#### Step 3 - Build libraries from source code
78+
```sh
79+
python3 ./build/fbcode_builder/getdeps.py --allow-system-packages test cachelib
80+
```
11581

116-
Downloads the latest source code version of the following libraries,
117-
builds and installs them (using `sudo`):
118-
`googleflags`, `googlelog`, `sparsemap`, `fmt`, `xxhash`, `folly`, `fizz`,
119-
`wangle`, `fbthrift`.
82+
This will run the full test suite for CacheLib. Individual tests can be run from the build directory if needed.
12083

121-
In some cases the operating system has a pre-packaged version of some
122-
of these libraries, but they are too old. In these cases the library
123-
is rebuilt from source code.
84+
## Locating Build Output
12485

125-
Building each library is performed using the following script:
86+
`getdeps.py` installs build artifacts to a scratch directory. To find where CacheLib was installed:
12687

12788
```sh
128-
$ ./contrib/build-package.sh -h
129-
CacheLib dependencies builder
130-
131-
usage: build-package.sh [-BdhijStv] NAME
132-
133-
options:
134-
-B skip build step
135-
(default is to build with cmake & make)
136-
-d build with DEBUG configuration
137-
(default is RELEASE with debug information)
138-
-h This help screen
139-
-i install after build using 'sudo make install'
140-
(default is to build but not install)
141-
-j build using all available CPUs ('make -j')
142-
(default is to use single CPU)
143-
-S skip git-clone/git-pull step
144-
(default is to get the latest source)
145-
-t build tests
146-
(default is to skip tests if supported by the package)
147-
-v verbose build
148-
149-
NAME: the dependency to build supported values are:
150-
zstd, xxhash,
151-
googlelog, googleflags, googletest,
152-
fmt, sparsemap,
153-
folly, fizz, wangle, fbthrift,
154-
cachelib
89+
python3 ./build/fbcode_builder/getdeps.py show-inst-dir cachelib
15590
```
15691

157-
All the required packages use `cmake`, and will be built in a new subdirectory
158-
named `build-[PACKAGE]` (e.g. running `./contrib/build-package.sh fmt` will
159-
create the `build-fmt` subdirectory).
92+
The install directory contains:
93+
94+
* `bin/` -- executables (e.g., `cachebench`, `cachebench-util`)
95+
* `lib/` and `lib64/` -- library files (e.g., `libcachelib_allocator.so`)
96+
* `include/` -- header files for CacheLib
97+
* `test_configs/` -- sample CacheBench configurations
16098

161-
Example:
162-
Running the command `./contrib/build-package.sh -i -j -d -t fmt`
163-
is equivalent to the following commands:
99+
For example, to run `cachebench`:
164100

165101
```sh
166-
cd cachelib/external
167-
git clone https://github.com/fmtlib/fmt.git
168-
cd ../..
169-
mkdir build-fmt
170-
cmake ../cachelib/external/fmt -DCMAKE_BUILD_TYPE=Debug
171-
make -j
172-
sudo make install
102+
INST_DIR=$(python3 ./build/fbcode_builder/getdeps.py show-inst-dir cachelib)
103+
$INST_DIR/bin/cachebench --help
104+
$INST_DIR/bin/cachebench --json_test_config $INST_DIR/test_configs/simple_test.json
173105
```
174106

175-
#### Step 4 - Build CacheLib
107+
## Development Workflow
176108

177-
Building CacheLib is identical to installing packages (above),
178-
with the exception of system-wide installation - cachelib is *not* installed
179-
by the `build.sh` wrapper script.
109+
When working on CacheLib itself (e.g., tweaking caching algorithms or adding features to `cachebench`), the following is recommended:
180110

181-
To build cachelib, run:
111+
1. Make your changes to the source code in the `cachelib/` directory
112+
2. Rebuild CacheLib:
182113

183-
`./contrib/build-package.sh -j -d -v cachelib`.
114+
```sh
115+
python3 ./build/fbcode_builder/getdeps.py --allow-system-packages build cachelib
116+
```
184117

185-
To install the cachelib files, either add `-i` to the `build-package.sh` script,
186-
or manually install with:
118+
3. Run relevant tests to verify your changes:
187119

188-
```sh
189-
$ cd build-cachelib
190-
$ sudo make install
191-
```
120+
```sh
121+
python3 ./build/fbcode_builder/getdeps.py --allow-system-packages test cachelib
122+
```
192123

193-
The installed files will be:
194-
* Header files in `/usr/local/include/cachelib/`
195-
* Library files in `/usr/local/lib/libcachelib_*.so`
196-
* `cachebench` and `cachebench-util` executables in `/usr/local/bin`.
124+
4. Locate the updated binaries:
197125

126+
```sh
127+
INST_DIR=$(python3 ./build/fbcode_builder/getdeps.py show-inst-dir cachelib)
128+
$INST_DIR/bin/cachebench --help
129+
```
198130

131+
Since `getdeps.py` uses an incremental build system, rebuilding after small changes is typically very fast as only the modified components and their dependents are recompiled.
199132

200-
## Development Cycle
133+
## Updating to Latest Version
201134

202-
When working on CacheLib itself (e.g. tweaking caching algorithms or adding
203-
features to `cachebench`), the following is recommended:
135+
Facebook's internal development cycle tightly couples CacheLib with its dependencies (e.g., folly, fbthrift, wangle, fizz), and all are frequently updated. In particular, the folly library does not provide a stable API, and using mismatched versions can cause compilation errors.
204136

205-
* Run `./contrib/build.sh -j -d -v` to install dependencies and
206-
build `cachelib`.
207-
* The resulting cachelib files will be stored in the `build-cachelib`
208-
subdirectory.
209-
* Modify source code files in `./cachelib/`
210-
* Rebuild the modified files in `build-cachelib` using `make`.
137+
To update to the latest version:
211138

212-
Example:
139+
1. Pull the latest changes:
213140

214-
```sh
215-
$ ./contrib/build.sh -d -j -v
216-
[... after build is complete ...]
141+
```sh
142+
git pull origin main
143+
```
144+
145+
2. Rebuild with getdeps.py:
146+
147+
```sh
148+
python3 ./build/fbcode_builder/getdeps.py --allow-system-packages build cachelib
149+
```
150+
151+
The `getdeps.py` script automatically handles updating and rebuilding all dependencies to compatible versions. This ensures that you always have a consistent set of libraries that work together.
217152

218-
$ cd build-cachelib
219-
$ make
220-
[... cachelib and cachebench are rebuild ...]
153+
If you encounter build issues after pulling, try cleaning the build artifacts and rebuilding:
221154

222-
$ touch touch ../cachelib/cachebench/main.cpp
223-
$ make
224-
[... cachelib and cachebench are rebuild ...]
155+
```sh
156+
# getdeps.py will automatically rebuild dependencies as needed
157+
python3 ./build/fbcode_builder/getdeps.py --allow-system-packages build cachelib
225158
```
226159

160+
## Troubleshooting
227161

228-
## Updating to latest source code version
162+
### Build Failures
229163

230-
Facebook internal development cycle tightly couples
231-
cachelib with its dependencies (e.g. folly, fbthrifth, wangle, fizz),
232-
and all are frequently updated.
233-
Particularly, the folly library does not provide stable API,
234-
and using mismatched version can cause compilation errors.
164+
If you encounter build failures:
235165

236-
Therefore, it is recommended to *always* update (and rebuild)
237-
all dependencies before updating cachelib. That is,
238-
a simple `git pull` for cachelib alone can often lead to failed builds.
166+
1. Ensure you have a C++20 compatible compiler (GCC 10+ or Clang 10+)
167+
2. Make sure CMake 3.14 or newer is installed
239168

240-
Running the `contrib/build.sh` script takes care of first updating
241-
and rebuilding all dependencies, and then updating and rebuilding cachelib.
242-
Use the `-O` option to skip the installation of system packages, e.g.
243-
`./contrib/build.sh -d -v -j -O`.
169+
### Missing System Dependencies
244170

245-
As all dependencies use `git/cmake/make`, rebuilding the same code (if there
246-
were no updates) will be very fast.
171+
If system dependency installation fails:
247172

173+
1. Try the dry-run option to see what packages would be installed:
174+
```sh
175+
python3 ./build/fbcode_builder/getdeps.py install-system-deps --dry-run --recursive cachelib
176+
```
248177

249-
## Downloading the source code without building
178+
2. Install the packages manually using your system's package manager
250179

251-
The default `build.sh` wrapper script requires internet connection
252-
(for package installation and github updates).
180+
3. Re-run the build with `--allow-system-packages` to use the manually installed dependencies
253181

254-
For special build circumstances where internet connection is not available,
255-
it is possible to download the source code on one machine, then copy it
256-
and build it on another.
182+
### Platform Support
257183

258-
Use `build-package.sh -B` option to only download the latest source code
259-
(using `git clone/git pull`) without building.
184+
`getdeps.py` has been tested on:
185+
- Ubuntu 18.04, 20.04, 22.04
186+
- CentOS 8
187+
- Debian 10, 11
260188

261-
Example:
262-
```sh
263-
./contrib/build-package.sh -B googlelog
264-
./contrib/build-package.sh -B googleflags
265-
./contrib/build-package.sh -B googletest
266-
./contrib/build-package.sh -B fmt
267-
./contrib/build-package.sh -B sparsemap
268-
./contrib/build-package.sh -B folly
269-
./contrib/build-package.sh -B fizz
270-
./contrib/build-package.sh -B wangle
271-
./contrib/build-package.sh -B fbthrift
272-
./contrib/build-package.sh -B cachelib
273-
```
189+
For other platforms, you may need to install dependencies manually and use `--allow-system-packages`.
274190

275-
Will download the latest source code of all libraries under
276-
the `./cachelib/external` subdirectory.
191+
## Legacy Build Scripts
277192

278-
Then the entire build tree can be copied to another machine
279-
(one that does not have internet connectivity).
280-
CacheLib can then be build be adding the `-S` option to `build.sh`
281-
(meaning: skip the `git clone/git pull` step):
193+
**Note**: The legacy build scripts (`./contrib/build.sh`, `./contrib/build-package.sh`, and related scripts) are deprecated and no longer maintained. Please use `getdeps.py` as documented above for all builds.
282194

283-
```sh
284-
$ ./contrib/build.sh -d -j -v -S
285-
```
195+
## Additional Resources
196+
197+
- [Installation Guide](https://cachelib.org/docs/installation/) - Detailed installation instructions
198+
- [CacheBench Documentation](https://cachelib.org/docs/Cache_Library_User_Guides/Cachebench_Overview) - Guide to using CacheBench
199+
- [GitHub Repository](https://github.com/facebook/CacheLib) - Source code and issue tracker

0 commit comments

Comments
 (0)