got signal=11 on dlclose with external engine

[guyzohar1]

I am hitting a segfault when FIO tries to close an external lib
at:
void free_ioengine(struct thread_data *td)
{
dprint(FD_IO, "free ioengine %s\n", td->io_ops->name);

if (td->eo && td->io_ops->options) {
	options_free(td->io_ops->options, td->eo);
	free(td->eo);
	td->eo = NULL;
}

if (td->io_ops_dlhandle) {
	dlclose(td->io_ops_dlhandle);
	td->io_ops_dlhandle = NULL;
}

td->io_ops = NULL;

}

io_ops_dlhandle does not hold a pointer to the lib handler resulting a seg fault

i have added a stack trace to my engine:
[bt]: (1) /lib/x86_64-linux-gnu/libc.so.6(+0x3ef20) [0x7f1072d89f20]]
[bt]: (2) /lib64/ld-linux-x86-64.so.2(+0x1f316) [0x7f1073f6b316]
[bt]: (3) /lib64/ld-linux-x86-64.so.2(_dl_exception_create+0x28) [0x7f1073f63868]
[bt]: (4) /lib/x86_64-linux-gnu/libc.so.6(_dl_signal_error+0x31) [0x7f1072eb2251]
[bt]: (5) /lib64/ld-linux-x86-64.so.2(+0x16ae1) [0x7f1073f62ae1]
[bt]: (6) /lib/x86_64-linux-gnu/libc.so.6(_dl_catch_exception+0x6f) [0x7f1072eb22df]
[bt]: (7) /lib/x86_64-linux-gnu/libc.so.6(_dl_catch_error+0x2f) [0x7f1072eb236f]
[bt]: (8) /lib/x86_64-linux-gnu/libdl.so.2(+0x1735) [0x7f107313d735]
[bt]: (9) /lib/x86_64-linux-gnu/libdl.so.2(dlclose+0x33) [0x7f107313d0b3]
[bt]: (10) /fio(close_ioengine+0xa4) [0x55beeb246f54]
[bt]: (11)/fio(+0x7c48b) [0x55beeb29948b]
[bt]: (12)/fio(+0x7fe75) [0x55beeb29ce75]
[bt]: (13) /fio(fio_backend+0xe3) [0x55beeb29cf73]
[bt]: (14) /fio(main+0x93) [0x55beeb2365d3]
[bt]: (15) /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7f1072d6cb97]
[bt]: (16) /fio(_start+0x2a) [0x55beeb23662a]

Is this a known issue?
what am I doing wrong here?

Thanks

[sitsofe]

@guyzohar1 can you make this happen with a small and simple ioengine so we can reproduce the issue here?

[guyzohar1]

during method:
int td_io_init(struct thread_data *td)
{
int ret = 0;

if (td->io_ops->init) {
	ret = td->io_ops->init(td);
	if (ret)
		log_err("fio: io engine %s init failed.%s\n",
			td->io_ops->name,
			td->o.iodepth > 1 ?
			" Perhaps try reducing io depth?" : "");
	else
		td->io_ops_init = 1;
	if (!td->error)
		td->error = ret;
}

return ret;

}

before calling init method we have these values:
td->io_ops_data void * 0x0
td->io_ops_dlhandle void * 0x55555590ed30
td struct thread_data * 0x7fffd513d000

when entering the method, these values changes, and somehow we get a swap between them:
td->io_ops_data void * 0x55555590ed30
td->io_ops_dlhandle void * 0x0
td struct thread_data * 0x7fffd513d000

because i am assigning a new data to io_ops_data, the pointer address changes:
td->io_ops_data void * 0x555555911420
td->io_ops_dlhandle void * 0x0
td struct thread_data * 0x7fffd513d000

and when returning from the init method, these pointer gets swapped again, but the handler lost the prev value:
td->io_ops_data void * 0x0
td->io_ops_dlhandle void * 0x555555911420
td struct thread_data * 0x7fffd513d000

and when arriving at the end to free the handler at free_ioengine:
if (td->io_ops_dlhandle) {
dlclose(td->io_ops_dlhandle);
td->io_ops_dlhandle = NULL;
}

we have this:
td->io_ops_data void * 0x0
td->io_ops_dlhandle void * 0x555555911420
td struct thread_data * 0x7fffd513d000

and we hit segfault

have you seen an issue like this? maybe I am compiling my shared object incorrectly ?

[yu-zou]

Confirm the same issue when using an external engine.

[axboe]

Should be fixed in current -git, can you try?