Linking with tcmalloc

[danpovey]

This is to propose modifying the build system to link with Google's tcmalloc.
I think it's just a question of adding something like -ltcmalloc_minimal or -ltcmalloc to the link line. I notice that packages for this exist: for example, on Debian, there is the package
libtcmalloc-minimal4
@LvHang found that for decoding, linking in tcmalloc could give of the order of 10% speedup.
I'd rather not make it a mandatory dependency though, as there may be systems where it's not convenient to install. Also I believe it doesn't release memory to the system so in principle there may be situations with long-running jobs, where it could end up being more of a memory hog than regular malloc. But I hope this won't happen too much in practice.

[kkm000]

FWIW, There is also tbbmalloc that comes with MKL.

Lattice decoder would probably benefit from an arena-type allocator, as it constantly churns a lot of small allocations. I noticed long ago that it's very sensitive to the allocator used (before switching to tbb, I noticed the performance varied much even between Windows versions).

[danpovey]

I guess it should be easy to compare them. @LvHang if you configure on our grid right now, it should give you MKL by default, so your path may be set up so that if you just add -ltbbmalloc to the right place in kaldi.mk, it would use that allocator.

[kkm000]

I've always used -ltbbmalloc_proxy to replace the existing allocator, I am not sure if -ltbbmalloc would cut it. And then, libtbbmalloc.so is loaded at run time by the proxy, so there won't be a path to it baked into the executables by the linker -- or does the proxy looks for it in the same directory it is loaded from? I'll figure out the build switches. I use it on Windows mainly.

[danpovey]

Thanks!

…

On Thu, Apr 25, 2019 at 7:41 PM kkm (aka Kirill Katsnelson) < ***@***.***> wrote: I've always used -ltbbmalloc_proxy to replace the existing allocator, I am not sure if -ltbbmalloc would cut it. And then, libtbbmalloc.so is loaded at run time by the proxy, so there won't be a liker path baked into the executables -- or does the proxy looks for it in the same directory it is loaded from? I'll figure out the build switches. I use it on Windows mainly. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#3271 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAZFLOYYZ7AKICNOL3ZWCCDPSI6RTANCNFSM4HIRDY7A> .

[kkm000]

As a quick hack for the test, the following worked for me. Add these switches to the end of LD_LIBS near the end of kaldi.mk:

-L/opt/intel/tbb/lib/intel64/gcc4.7  -Wl,-rpath,/opt/intel/tbb/lib/intel64/gcc4.7 -l:libtbbmalloc_proxy.so.2

and probably in the path.sh, add

export LD_LIBRARY_PATH=/opt/intel/tbb/lib/intel64/gcc4.7

I in fact did this, instead of the latter, to run tests:

~/work/kaldi2/src/matrix$ LD_LIBRARY_PATH=/opt/intel/tbb/lib/intel64/gcc4.7 make test

gcc4.7 directory is for "gcc 4.7 or any later version", looks confusing at first.

Apparently, DL_RPATH in the executable does not function the same way LD_LIBRARY_PATH does. While the -rpath switch certainly adds the directory to DL_RPATH (can be seen with readelf -d matrix-lib-test), this does not help the proxy load libtbbmalloc.so:

$ LD_LIBRARY_PATH=/opt/intel/tbb/lib/intel64/gcc4.7 ./matrix-lib-test &>/dev/null && echo ok
ok
$ ./matrix-lib-test
./matrix-lib-test: error while loading shared libraries: libtbbmalloc.so.2: cannot open shared object file: No such file or directory

Looks like DL_RPATH attribute is only used for immediate dependencies, and does not apply transitively.

tcmalloc is one piece, so should not require the LD_LIBRARY_PATH. Generally, I do not expect the two to differ much, but we can look at both of them and see.

[LvHang]

I tried to test the speed with different malloc libraries on clsp grid (b11) for standard decoder and combined decoder. I used the standard configuration "beam=15.0, lattice-beam=8.0, max-active=7000, cost-scale=4.0 in combined decoder". The results are as follows.

(1089 utts)	standard_decoder	combined_decoder	standard_decoder_tcmalloc	combined_decoder_tcmalloc	standard_decoder_tbbmalloc	combined_decoder_tbbmalloc
round1	370.405	421.555	313.966	308.197	343.058	331.282
round2	368.33	429.58	317.931	305.551	340.232	329.117
round3	369.668	438.61	315.62	306.652	342.111	330.425
average	369.4676667	429.915	315.839	306.8	341.8003333	330.2746667

From the results, both tcmalloc and tbbmalloc imprvoe the speed of decoders. The tcmalloc looks better. I'm not sure whether it depends on OS platform. You can get more information from https://software.intel.com/en-us/node/506096, and it introduces which kind of functions will be replaced.

Dan, if you want to try it on clsp grid.
For "tcmalloc", you can get it from /home/hlyu/gperf/lib. Add -ltcmalloc to LDLIBS in kaldi.mk and add the LD_LIBRARY_PATH.
For "tbbmalloc", you can get if from /opt/intel/compilers_and_libraries_2019.2.187/linux/tbb/lib/intel64_lin/gcc4.7.
Add
-L/opt/intel/compilers_and_libraries_2019.2.187/linux/tbb/lib/intel64_lin/gcc4.7 -Wl,-rpath,/opt/intel/compilers_and_libraries_2019.2.187/linux/tbb/lib/intel64_lin/gcc4.7 -l:libtbbmalloc_proxy.so.2 to LDLIBS in kaldi.mk and add the LD_LIBRARY_PATH.

There are some weird problems I cannot fully understand.

1. From my knowledge, -Wl,-rpath,librarypath, which add a directory to the runtime library search path, can be used to substitute "adding librarypath to LD_LIBRARY_PATH". I tried to just use one of them, it works. But if I only use "-Wl,-rpath", it will be slower than use them together. [I haven't count the case which uses LD_LIBRARY_PATH only.]（400s vs 340s in standard decoder, 410s vs 330s in combined decoder). I guess it relates to the searching order of dynamic library path.
2. Normally, we link a public dyanmic library whose name is "libabc.so" with the switch -labc. But, for tbbmalloc, I cannot use -ltbbmalloc_proxy which is mentioned in its website (https://software.intel.com/en-us/node/506097). But -l:libtbbmalloc_proxy.so.2 works.

[danpovey]

Hang, I'd rather rely on the system package, we have /usr/lib/libtcmalloc_minimal.so. See if just -ltcmalloc_minimal works. (just -ltcmalloc should work too, I think, on our grid, without relying on the one in your homedir). If we include this, I'd like to have the check_dependencies.sh script suggest system packages to install; but we wouldn't make this one mandatory, since it's just an optimization.

…

On Wed, May 1, 2019 at 2:06 AM LvHang ***@***.***> wrote: I tried to test the speed with different malloc libraries on clsp grid (b11) for standard decoder and combined decoder. I used the standard configuration "beam=15.0, lattice-beam=8.0, max-active=7000, cost-scale=4.0 in combined decoder". The results are as follows. (1089 utts) standard_decoder combined_decoder standard_decoder_tcmalloc combined_decoder_tcmalloc standard_decoder_tbbmalloc combined_decoder_tbbmalloc round1 370.405 421.555 313.966 308.197 343.058 331.282 round2 368.33 429.58 317.931 305.551 340.232 329.117 round3 369.668 438.61 315.62 306.652 342.111 330.425 average 369.4676667 429.915 315.839 306.8 341.8003333 330.2746667 From the results, both tcmalloc and tbbmalloc imprvoe the speed of decoders. The tcmalloc looks better. I'm not sure whether it depends on OS platform. You can get more information from https://software.intel.com/en-us/node/506096, and it introduces which kind of functions will be replaced. Dan, if you want to try it on clsp grid. For "tcmalloc", you can get it from /home/hlyu/gperf/lib. Add -ltcmalloc to LDLIBS in kaldi.mk and add the LD_LIBRARY_PATH. For "tbbmalloc", you can get if from /opt/intel/compilers_and_libraries_2019.2.187/linux/tbb/lib/intel64_lin/gcc4.7 . Add -L/opt/intel/compilers_and_libraries_2019.2.187/linux/tbb/lib/intel64_lin/gcc4.7 -Wl,-rpath,/opt/intel/compilers_and_libraries_2019.2.187/linux/tbb/lib/intel64_lin/gcc4.7 -l:libtbbmalloc_proxy.so.2 to LDLIBS in kaldi.mk and add the LD_LIBRARY_PATH. There are some weird problems I cannot fully understand. 1. From my knowledge, -Wl,-rpath,librarypath, which add a directory to the runtime library search path, can be used to substitute "adding librarypath to LD_LIBRARY_PATH". I tried to just use one of them, it works. But if I only use "-Wl,-rpath", it will be slower than use them together. [I haven't count the case which uses LD_LIBRARY_PATH only.]（400s vs 340s in standard decoder, 410s vs 330s in combined decoder). I guess it relates to the searching order of dynamic library path. 2. Normally, we link a public dyanmic library whose name is "libabc.so" with the switch -labc. But, for tbbmalloc, I cannot use -ltbbmalloc_proxy which is mentioned in its website ( https://software.intel.com/en-us/node/506097). But -l:libtbbmalloc_proxy.so.2 works. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#3271 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAZFLOYWR4JD3GWRRDV2VKLPTEXOLANCNFSM4HIRDY7A> .

[LvHang]

Dan, I found that both "libtcmalloc.so.4" and "libtcmalloc_minimal.so.4" are exist in /usr/bin on clsp grid. So both -ltcmalloc and -ltcmalloc_minimal work. After compiling, I use command ldd <binary> (for example: ldd latgen-faster-mapped) to check whether the dynamic library is linked in a right way. It's ok. After that, I did three rounds speed testing. I didn't see too much speed difference between "tcmalloc" and "tcmalloc_minimal", just 2-3s, it should be noise. Similar with previous tests, standard decoder takes about 315s and the combined decoder takes 308s on mini-librispeech 1089 utterances test-set.

[danpovey]

We should definitely include the tcmalloc stuff, but that's something best done by @kkm if he has time (or possibly @jtrmal), as they understand the various issues relating to the installation scripts. Thus, I think it's better if we just leave the tcmalloc part of this issue aside and wait for one of those guys to do what's needed. However, I do have a suggestion to speed up the current decoder by removing the slowness of new/delete on Tokens. Instead of `new Token` and `delete Tok`, you can have inline functions like the following: inline Token *NewToken(BaseFloat tot_cost, BaseFloat extra_cost, ForwardLinkT *links, Token *next, Token *backpointer) { if (deleted_toks_) { ... } else { return new Token(...); } } inline void DeleteToken(Token *tok) { // put at head of list deleted_toks_. } You would of course need to make changes to the constructor and destructor. It would be great if you could do some experiments about this. Right now, with the combined decoder, the speed improvements, while nice, are not really enough to justify the hassle of switching over (esp. for users of Kaldi who may have modified the decoder code somehow). So I suggest to focus on the other decoder idea about fast LM rescoring.

…

On Wed, May 1, 2019 at 6:11 PM LvHang ***@***.***> wrote: Dan, I found that both "libtcmalloc.so.4" and "libtcmalloc_minimal.so.4" are exist in /usr/bin on clsp grid. So both -ltcmalloc and -ltcmalloc_minimal work. After compiling, I use command ldd <binary> (for example: ldd latgen-faster-mapped) to check whether the dynamic library is linked in a right way. It's ok. After that, I did three rounds speed testing. I didn't see too much speed difference between "tcmalloc" and "tcmalloc_minimal", just 2-3s, it should be noise. Similar with previous tests, standard decoder takes about 315s and the combined decoder takes 308s on mini-librispeech 1089 utterances test-set. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#3271 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAZFLO3LPMQJDCOOD74U5QTPTIIR5ANCNFSM4HIRDY7A> .

[LvHang]

Ok, I will go to the A* decoder thread. I will try the inline function sometime.

[kkm000]

@danpovey, this is an excellent idea! These are the small allocations that I was referring to.

I will look into building with tcmalloc.

@LvHang, Thanks for timing the allocators, very revealing. Both allocators are build on pretty much the same idea; I think the difference is caused by their default tuning. If you have your test rig ready, it would be very interesting to see the peak amount of memory used with TBB vs tsalloc. You can use a tool like /usr/bin/time -v to get a report.

I would rather go with tcmalloc, as it is simpler to bolt on (comes in one moving piece vs. two in tbbmalloc). If its avarice for memory is too extreme, we'll find a way to tame it (possibly at a certain expense of performance).

From my knowledge, -Wl,-rpath,librarypath, which add a directory to the runtime library search path, can be used to substitute "adding librarypath to LD_LIBRARY_PATH".

More or less. If library or binary X depends on library A.so, and A.so depends on B.so, then -rpath pointing to A.so inside X will make A.so load, but it won't be able to load its dependency B.so (unless it's in ldconfig cache of paths, or somewhere on the LD_LIBRARY_PATH). In other words, -rpath stored in X acts as if it was prepended to LD_LIBRARY_PATH for the immediate dependencies of X, but not transitively for their own dependencies.

But if I only use "-Wl,-rpath", it will be slower than use them together. [I haven't count the case which uses LD_LIBRARY_PATH only.]（400s vs 340s in standard decoder, 410s vs 330s in combined decoder). I guess it relates to the searching order of dynamic library path.

I think you must be right. It's possible that some different libraries are loaded in different cases, maybe not even related to the allocator?

Normally, we link a public dyanmic library whose name is "libabc.so" with the switch -labc. But, for tbbmalloc, I cannot use -ltbbmalloc_proxy which is mentioned in its website (https://software.intel.com/en-us/node/506097). But -l:libtbbmalloc_proxy.so.2 works.

The reason for this is MKL only installs the runtime libraries of the allocator. -ltbbmalloc_proxy looks for the file named libtbbmalloc_proxy.so (and then for .a if not found). We do not have that file. When you install the full TBB package with headers and the kitchen sink, there is a link libtbbmalloc_proxy.so -> libtbbmalloc_proxy.so.2, so that -ltbbmalloc_proxy works. The linker follows all links until it finds a real file, and bakes a dependency on that file into the executable, i.e. libtbbmalloc_proxy.so.2 in this case.

[LvHang]

@kkm000 Thank you for your distinct explanation.

Normally, we link a public dyanmic library whose name is "libabc.so" with the switch -labc. But, for tbbmalloc, I cannot use -ltbbmalloc_proxy which is mentioned in its website (https://software.intel.com/en-us/node/506097). But -l:libtbbmalloc_proxy.so.2 works

Actually, if we add a soft link libtbbmalloc_proxy.so which points to the libtbbmalloc_proxy.so.2, it works. Last time, I forgot to change the -L switch, after I added a soft link in another directory. I'm afraid my head was broken at that time. Ha...

[kkm000]

tcmalloc documentation