Crash in IvectorExtractorInfo::Init()

[Veango]

when I init the Kaldi ,

        OnlineIvectorExtractionConfig ivector_extraction_opts;
        ivector_extraction_opts.splice_config_rxfilename = model_path_str_ + "/ivector/splice.conf";
        ivector_extraction_opts.cmvn_config_rxfilename = model_path_str_ + "/ivector/online_cmvn.conf";
        ivector_extraction_opts.lda_mat_rxfilename = model_path_str_ + "/ivector/final.mat";
        ivector_extraction_opts.global_cmvn_stats_rxfilename = model_path_str_ + "/ivector/global_cmvn.stats";
        ivector_extraction_opts.diag_ubm_rxfilename = model_path_str_ + "/ivector/final.dubm";
        ivector_extraction_opts.ivector_extractor_rxfilename = model_path_str_ + "/ivector/final.ie";
        ivector_extraction_opts.max_count = 100;
        feature_info_.use_ivectors = true;
        feature_info_.ivector_extractor_info.Init(ivector_extraction_opts);

it will crash like this:

please tell me how can I fix it

[kkm000]

Could be a mixup of files, but it better should not just crash. Cannot decode as reported, though.

Could you post the backtrace from a debug build? Also,

1. Pictures are much less useful that the output of the bt command. If you look carefully, backtrace lines are all cut short.
2. What was the signal name or number that was caught? Any error messages from the BLAS library on stdout? MKL often prints one before crashing.
3. What BLAS library do you compile against (MKL, OpenBLAS...)?

This info will be helpful.

[jtrmal]

what do you mean by "init kaldi", which code/program are you using? Is it something homebrewn? y.

…

On Tue, Sep 13, 2022 at 5:14 PM Veango ***@***.***> wrote: when I init the Kaldi , it will crash like this: [image: 1] <https://user-images.githubusercontent.com/77492855/189939267-33ce957f-636d-4688-bd35-f389d74fcabf.png> [image: 2] <https://user-images.githubusercontent.com/77492855/189939305-dda30b50-97b2-42ad-aa59-1d1c6d551dfc.png> please tell me how can I fix it — Reply to this email directly, view it on GitHub <#4791>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACUKYX6GDFQKWVVWMEACG73V6CK43ANCNFSM6AAAAAAQLRPLDY> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[Veango]

@kkm000

I used openblas，the code runs in iOS ，is Kaldi not support iOS?

[Veango]

another info is when I call make -C matrix test, it got some error like:
numpy-array-test.cc:82:15: error: 'system' is unavailable: not available on iOS
rc = std::system("python3 -c "import numpy; "
...

[kkm000]

'system' is unavailable: not available on iOS

Are you in fact building the whole of Kaldi on iOS, or is it a nonsensical message? You are likely much better off cross-compiling.

What is your platform? MacOS has its own linalg framework, I forgot its name (@jtrmal help me please, "accel" or something?), use it instead. It's configurable. Especially with their latest accelerator in M1/M2, it must use the accelerator. I've seen iOS used only on devices no beefier than the iPad. Does it have an accelerator for linear algebra? Does OpenBLAS support it? In any case, you will be much better off fully utilizing hardware stuff which is usually a-plenty on device SoCs, which speeds up a limited set of operation manifold, frees up CPU and reduces power consumption.

[does] Kaldi not support iOS?

A gray area. There are too many things that may be called iOS, and too many hardware variants it runs on. Since there's no one around with iOS experience, rather no, not in the full sense of "support." Do use the kaldi-help list, search its archives, post a question if nothing comes up. maybe you'll find a kindred soul.

For the crash, let's start off by removing threading from the picture: set threading to 1 globally, kaldi::g_num_threads = 1; before using any Kaldi functions. i-vector extractor initialization is multithreaded by default, and you see it in the stack.

kaldi/src/util/kaldi-thread.h

Lines 58 to 65 in 727e454

	namespace kaldi {
	extern int32 g_num_threads; // Maximum number of threads (for programs that
	// use threads, which is not many of them, e.g. the SGMM update program does.
	// This is 8 by default. You can change this on the command line, where
	// used, with --num-threads. Programs that think they will use threads
	// should register it with their ParseOptions, as something like:
	// po.Register("num-threads", &g_num_threads, "Number of threads to use.");

The default of 8 is (statically, at compile time) set in util/kaldi-thread.cc:25 and then used to initialize sequencer options a few times through src/ivector/ivector-extractor.cc.

If this has no effect, running the same init code on an x64 Linux or Windows, to make sure we don't have a case of broken or mismatched extractor files. There are a few.

[Veango]

Thank you for your reply,I cross-compiled openblas clapack openfst and kaldi，i ever used the same code in android ， it hasn't any problem. But i used it in iPhone platform, it crash,so i don't know witch step i was wrong

[kkm000]

Looks like an OpenBLAS issue. As I mentioned, the Apple's own solution, Accelerate, includes BLAS routines. And its docs mention that it's available in iOS. By all means, use it instead of OpenBLAS. It is supposed to use all hardware acceleration there is on the platform, and in the optimal way. For a device, it is especially important, as it not only does it faster but saves battery, too (assuming there is hardware available; I know little about iPhones and friends, but mid-line to high-end modern devices just have to have it).

Try to figure out why it is not selected by configure automatically. Maybe we just don't handle iOS cross correctly?

kaldi/src/configure

Lines 1029 to 1080 in 727e454

	elif [ "`uname`" == "Darwin" ]; then
	# Check for Darwin first, because we later call uname -o (for Cygwin)
	# which crashes on Darwin.
	echo "On Darwin: Checking for Accelerate framework ..."
	if [ ! -e /System/Library/Frameworks/Accelerate.framework ]; then
	failure "Need the Accelerate framework to compile on Darwin."
	fi
	OSX_VER=$(sw_vers | grep ProductVersion | awk '{print $2}' | awk '{split($0,a,"."); print a[1] "." a[2]; }')
	OSX_VER_NUM=$(echo $OSX_VER | sed 's/\./ /g' | xargs printf "%d%02d")
	echo "Configuring for OS X version $OSX_VER ..."
	if [ $OSX_VER_NUM -ge 1005 ]; then
	if [ "$MATHLIB" == "CLAPACK" ]; then
	if [ -z "$CLAPACKROOT" ]; then
	failure "Must specify the location of CLAPACK with --clapack-root option (and it must exist)"
	fi
	if [ ! -f ../tools/CLAPACK/clapack.h ]; then
	failure "could not find file ../tools/CLAPACK/clapack.h"
	fi
	if [ ! -d "$CLAPACKROOT" ]; then
	failure "The directory $CLAPACKROOT does not exist"
	fi
	# Also check for cblas.h and f2c.h
	echo "Using CLAPACK libs from $CLAPACKROOT as the linear algebra library."
	echo "CLAPACKROOT = $CLAPACKROOT" >> kaldi.mk
	if [ ! -f makefiles/darwin_clapack.mk ]; then
	failure "makefiles/darwin_clapack.mk not found."
	fi
	cat makefiles/darwin_clapack.mk >> kaldi.mk
	echo "Warning (CLAPACK): this part of the configure process is not properly tested and may not work."
	echo "Successfully configured for Darwin with CLAPACK libs from $CLAPACKROOT"
	else
	cat makefiles/darwin.mk >> kaldi.mk
	fi
	else
	failure "Mac OS X version '$OSX_VER' is not supported."
	fi
	if [ $OSX_VER_NUM == 1011 ]; then
	echo "**BAD WARNING**: You are using OS X El Capitan. Some versions of this OS"
	echo "**BAD WARNING**: have a bug in the BLAS implementation that affects Kaldi."
	echo "**BAD WARNING**: After compiling, cd to matrix/ and type 'make test'. The"
	echo "**BAD WARNING**: test will fail if the problem exists in your version. "
	echo "**BAD WARNING**: Eventually this issue will be fixed by system updates from"
	echo "**BAD WARNING**: Apple. Unexplained crashes with reports of NaNs will"
	echo "**BAD WARNING**: be caused by this bug, but some recipes will (sometimes) work."
	sleep 1; echo -n .; sleep 1; echo -n .; sleep 1; echo .
	fi
	echo "Successfully configured for Darwin with Accelerate framework."
	$use_cuda && configure_cuda
	elif [ "`uname -o`" == "Cygwin" ]; then

[kkm000]

If you can find a simple way to fix configure so that it picks up the correct library, we'd certainly welcome a patch. But if you find that you need to bend all the way backwards to make configure work for you, I want to share how I productize Kaldi. For training and stuff on Linux, I of course use the standard workflow, configure + make. But for compiling the product, I select necessary files from Kaldi into a library (usually static, it allows for advanced optimizations such as PGO for the final build, but I don't know what's the best for your case), and simply build it. You need to define one preprocessor variable (otherwise defined in a generated base/version.h file), namely KALDI_VERSION (to any value, it's used only in diagnostic messages) in the command line to compiler. All compile flags and dependency libraries you can figure out from your platform SDK. You can also catch Kaldi output using the hook routine in base/kaldi-error.h, so that they are logged your way instead of being output to stderr.

[kkm000]

OpenFST also has a logging hook in log.h; that's separate from Kaldi, you need to call it too if you want to hook up their logging.

[Veango]

You mean i shouldn't use openblas and clapack,just only compile openfst and kaldi,don't set the MATHLIB，and it will use the system's accelerate if it's no problem?

[kkm000]

No, this is not exactly what I said.

i shouldn't use openblas and clapack

Correct, use iOS native Accelerator framework.

don't set the MATHLIB, and it will use the system's accelerate

During configure? I don't know, possibly. Try it. It may or may not work. If it doesn't. try to figure out and easy fix to configure, it's hand-coded, not generated. If it happens too tricky to get right, you may do it differently. You also may do it this way to get a greater control over build options, namely:

just only compile openfst and kaldi

As an alternative to configure, you may compile the runtime-only files your own way. I do it both to OpenFST and Kaldi, normally. This is what I do for production builds. You'll have to write whatever build files your build system/IDE requires. But it is the most flexible option. Mind the KALDI_VERSION. I can't recall any other tripping points.

Optionally, regardless of the way you build, hook logging (see kaldi-error.h). You'll probably need it for diagnostics at some point. This applies to all platforms, not only iOS. Kaldi also has SetVerbosityLevel(), beyond INFO, WARNING and ERROR, from 0 to 5, 5 is insane amount of logging. When you don't do verbose logs, set to 0 (default), it's expensive to generate, so don't simply generate and discard it. INFO and above are always logged.