Development Processes

This document describes two things:

• Local Builds
• Merging Upstream Changes

Local Builds

1. First, check out the relevant repos:

   • Tokutek/tokumxse
   • Tokutek/ft-index
   • Tokutek/jemalloc

2. Create a symlink for jemalloc and build ft-index. We want to "install" the result somewhere.

   1. For a debug build:

      cd ft-index
      ln -s $PWD/../jemalloc third_party/
      mkdir dbg
      cd dbg
      cmake \
          -D CMAKE_BUILD_TYPE=Debug \
          -D USE_VALGRIND=ON \
          -D TOKU_DEBUG_PARANOID=ON \
          -D BUILD_TESTING=OFF \
          -D CMAKE_INSTALL_PREFIX=$PWD/install \
          ..
      make install

   2. For a release build:

      cd ft-index
      ln -s $PWD/../jemalloc third_party/
      mkdir opt
      cd opt
      cmake \
          -D CMAKE_BUILD_TYPE=Release \
          -D USE_VALGRIND=OFF \
          -D TOKU_DEBUG_PARANOID=OFF \
          -D BUILD_TESTING=OFF \
          -D CMAKE_INSTALL_PREFIX=$PWD/install \
          ..
      make install

3. Next, make a symlink from inside the tokumxse repo to the "install directory" where the built ft-index libs live:

   1. For a debug build:

      cd tokumxse
      ln -s $PWD/../ft-index/dbg/install src/third_party/tokuft

   2. For a release build:

      cd tokumxse
      ln -s $PWD/../ft-index/opt/install src/third_party/tokuft

4. Finally, invoke scons with the magic options:

   scons --extrapath=$PWD/src/third_party/tokuft --tokuft --allocator=jemalloc mongod mongos mongo

   If you like, you can add --dbg=[on|off] --opt=[on|off] or -jN to scons. --mute is also quite nice.

5. You can run some C++ unit tests with scons:

   scons --extrapath=$PWD/src/third_party/tokuft --tokuft --allocator=jemalloc smokeCppUnittests

6. You can also run some javascript tests with smoke.py once mongod, mongos, and mongo are built:

   python buildscripts/smoke.py --storageEngine=tokuft jsCore

   Apart from jsCore you can also run other suites, refer to this map for all of them.

Merging Upstream Changes

Upstream MongoDB manages multi-version development by maintaining separate branches for each stable minor version (v2.4, v2.6, v3.0), as well as an active development branch (master), and they cherry-pick any changes that need to hit multiple branches. We do the same for master and v3.0. About once a day, check for upstream changes (on branches master and v3.0), and if there are some, go ahead and merge them to tokumxse:

1. If you haven't set it up already, create an upstream remote in your git checkout of tokumxse:

   git remote add upstream https://github.com/mongodb/mongo

2. Fetch new commits from upstream:

   git fetch upstream

3. Make sure your master and v3.0 branches are up to date:

   git checkout master
   git pull
   git checkout v3.0
   git pull

4. Into each branch, merge the corresponding upstream branch:

   git checkout master
   git merge upstream/master
   git checkout v3.0
   git merge upstream/v3.0

5. Try building. If they've removed or added part of the API, you should notice with a failed compile. You can also watch the src/mongo/db/storage directory for interesting looking changes that might break things.

   Additionally, there's a mechanism that makes sure that the "assert ids" are unique. You may see a build fail with a message like this:

   DUPLICATE IDS: 28610
     ./src/mongo/db/storage/tokuft/tokuft_init.cpp:75:fassertFailedNoTrace(28610);
     ./src/mongo/db/repl/replication_coordinator_impl.cpp:2275:fassert(28610, cbh.getStatus());
   next id to use:28620
   

   In this case, this means that upstream, they've added a new assert id that conflicts with one we had already. To avoid conflicts, we should change ours, so in the above example, you'd go to tokuft_init.cpp and change 28610 to 28620. Then commit the change on either master or v3.0, and make sure you cherry-pick it to the other branch as well. For example, this particular conflict was fixed by this commit on master and this commit on v3.0.