Crash on Async Mode / GIL release #1383

choandrew · 2021-02-03T02:16:27Z

choandrew
Feb 3, 2021

Bug Report

I can make perspective reliably segfault in a couple of seconds.
If you have a view with a computed column , and then you derive a table with an index from that view (ie SEGFAULT_TABLE = perspective.Table(view.to_arrow(), index="index")
and then update it really fast, it will either:

segfault
throw this error:

Traceback (most recent call last):
  File "/home/user/conda/envs/env/lib/python3.8/site-packages/tornado/ioloop.py", line 741, in _run_callback
    ret = callback()
  File "/home/user/conda/envs/env/lib/python3.8/site-packages/perspective/table/_state.py", line 70, in call_process
    pool._process()
  File "/home/user/conda/envs/env/lib/python3.8/site-packages/perspective/table/table.py", line 537, in _update_callback
    callback["callback"](port_id=port_id, cache=cache)
  File "/home/user/conda/envs/env/lib/python3.8/site-packages/perspective/table/view.py", line 617, in _wrapped_on_update_callback
    callback(port_id, cache["row_delta"])
  File "test2.py", line 122, in update_function
    table.update(delta)
  File "/home/user/conda/envs/tt/lib/python3.8/site-packages/perspective/table/table.py", line 316, in update
    self._table = make_table(
perspective.table.libbinding.PerspectiveCppError: Specified index `

Steps to Reproduce:

In short:

    MANAGER = perspective.PerspectiveManager()
    columns = {"index": int, "num1": int, "num2": float}

    BASE_TABLE = perspective.Table(columns, index="index")

    MANAGER.host_table("test_table1", BASE_TABLE)

    view = BASE_TABLE.view(
        computed_columns=[
            {
                "column": "mult",
                "computed_function_name": "*",
                "inputs": ["num1", "num2"],
            },
        ]
    )
    SEGFAULT_TABLE = perspective.Table(view.to_arrow(), index="index")
    view.on_update(updater(SEGFAULT_TABLE), mode="row")
    MANAGER.host_table("test_table2", SEGFAULT_TABLE)

    # the addition of an index to the segfault table makes it bad
    # the below table would be ok
    # TABLE = perspective.Table(view.to_arrow())

Environment:

python 3.8
perspective-python 0.6
linux with conda

Code Template to Reproduce

import asyncio
import threading

import tornado
import tornado.web
import tornado.ioloop
import tornado.gen

import perspective


def perspective_thread(MANAGER):
    loop = tornado.ioloop.IOLoop()
    MANAGER.set_loop_callback(loop.add_callback)
    loop.start()


class MainHandler(tornado.web.RequestHandler):
    def get(self):
        html = """<!DOCTYPE html>
<html>

<head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
    <title>test</title>

    <meta name="viewport"
        content="width=device-width, initial-scale=1, maximum-scale=1, minimum-scale=1, user-scalable=no">
        <script src="https://cdn.jsdelivr.net/npm/@finos/perspective-viewer"></script>
        <script src="https://cdn.jsdelivr.net/npm/@finos/perspective-viewer-datagrid"></script>
        <script src="https://cdn.jsdelivr.net/npm/@finos/perspective-viewer-d3fc"></script>
        <script src="https://cdn.jsdelivr.net/npm/@finos/perspective/dist/umd/perspective.js"></script>

</head>
<style>
    perspective-viewer {
        position: absolute;
        top: 0;
        left: 0;
        right: 0;
        bottom: 0;
    }
</style>

<body>
    <perspective-viewer id="viewer" , plugin="datagrid"></perspective-viewer>
    <script>
        websocket_url = 'ws://localhost:8080/websocket'
        table_name = 'test_table2'

        window.addEventListener('WebComponentsReady', async function () {
            const viewer = document.getElementById('viewer');
            const websocket = perspective.websocket(websocket_url);
            const table = websocket.open_table(table_name);
            viewer.load(table);
            viewer.toggleConfig();
        });
    </script>
</body>

</html>"""
        self.write(html)


def make_app(MANAGER,):
    handlers = [
        (r"/", MainHandler,),
        (
            r"/websocket",
            perspective.PerspectiveTornadoHandler,
            {"manager": MANAGER, "check_origin": True},
        ),
    ]
    app = tornado.web.Application(handlers)
    return app


async def main():
    MANAGER = perspective.PerspectiveManager()
    columns = {"index": int, "num1": int, "num2": int}

    BASE_TABLE = perspective.Table(columns, index="index")

    MANAGER.host_table("test_table1", BASE_TABLE)

    view = BASE_TABLE.view(
        computed_columns=[
            {
                "column": "mult",
                "computed_function_name": "*",
                "inputs": ["num1", "num2"],
            },
        ]
    )
    SEGFAULT_TABLE = perspective.Table(view.to_arrow(), index="index")
    view.on_update(updater(SEGFAULT_TABLE), mode="row")
    MANAGER.host_table("test_table2", SEGFAULT_TABLE)

    # the addition of an index to the segfault table makes it bad
    # the below table would be ok
    # TABLE = perspective.Table(view.to_arrow())

    thread = threading.Thread(target=perspective_thread, args=(MANAGER,))
    thread.daemon = True
    thread.start()

    app = make_app(MANAGER)
    app.listen(8080)

    i = 0

    BASE_TABLE.update([{"index": i, "num1": i, "num2": 2 * i}])
    while True:
        await asyncio.sleep(0.01)
        BASE_TABLE.update([{"index": i - 1, "num1": i + 1, "num2": 12}])
        BASE_TABLE.update([{"index": i, "num1": i, "num2": 2 * i}])
        i += 1


def updater(table):
    def update_function(port, delta):
        table.update(delta)

    return update_function


if __name__ == "__main__":
    asyncio.run(main(), debug=True)

Answered by sc1f

Apr 15, 2021

In async mode, once you have started the Perspective thread, all operations that touch the Perspective table has to be on that thread. In this example, calling BASE_TABLE.update() from the main thread is not threadsafe and can cause issues in a non-deterministic way.

To call update safely, use MANAGER._loop_callback so that the update operation happens on the Perspective thread and not the main thread:

MANAGER._loop_callback(BASE_TABLE.update, {"index": i, "num1": i, "num2": 2 * i}])

Additionally, if you are able to get the coredumps for the segfaulting process that will help us narrow down where the issue is. I think the earlier example you raised could be fixed with calling update through

View full answer

sc1f · 2021-02-03T05:28:02Z

sc1f
Feb 3, 2021

Thanks for the repro!

I just ran it as written against current master and it does not segfault/throw any errors and the table does continue to update; ~~I am in the process of rebuilding on 0.6.0 and seeing where it fails, because I am curious/might have missed something in the repro.~~

Running against a build on v0.6.0, the example also does not segfault/break for me on Python 3.7, MacOS. From the front-end, I can see "test_table2" being updated with new values, including the computed values.

0 replies

choandrew · 2021-02-03T14:10:19Z

choandrew
Feb 3, 2021
Author

Hey Jun, thanks for the message.

This bug might have to do with my computer being slow or me using conda.

I changed the code

# this might work if added towards the bottom of the example above
    while True:
        BASE_TABLE.update([{"index": i - 1, "num1": i + 1, "num2": 12}])
        BASE_TABLE.update([{"index": p, "num1": p, "num2": 2 * p} for p in range(i)])
        i += 1

If this also does not segfault, I can just wait and see if this will just start working in the next update.

For what it's worth, I also ran gdb python coredump and then backtrace and here is what outputted:

#0  pymalloc_alloc.isra.1 (nbytes=40) at /tmp/build/80754af9/python_1596467742699/work/Objects/obmalloc.c:1460
#1  _PyObject_Malloc (nbytes=40, ctx=0x0) at /tmp/build/80754af9/python_1596467742699/work/Objects/obmalloc.c:1622
#2  PyObject_Malloc () at /tmp/build/80754af9/python_1596467742699/work/Objects/obmalloc.c:685
#3  0x00007fd9670d399a in _PyLong_New () at /tmp/build/80754af9/python_1596467742699/work/Objects/longobject.c:275
#4  0x00007fd9670d420a in x_add () at /tmp/build/80754af9/python_1596467742699/work/Objects/longobject.c:3166
#5  0x00007fd9670bb951 in binary_op1 (op_slot=0, w=0x7fd957cb5310, v=0x7fd957cb7090) at /tmp/build/80754af9/python_1596467742699/work/Objects/abstract.c:808
#6  PyNumber_Add () at /tmp/build/80754af9/python_1596467742699/work/Objects/abstract.c:958
#7  0x00007fd967143b40 in fill_time (nsec=437386769, sec=1612359430, index=9, v=0x7fd957d4e040) at /tmp/build/80754af9/python_1596467742699/work/Modules/posixmodule.c:2101
#8  _pystat_fromstructstat (st=0x7ffe8489d300) at /tmp/build/80754af9/python_1596467742699/work/Modules/posixmodule.c:2170
#9  posix_do_stat.isra.153 (follow_symlinks=<optimized out>, dir_fd=<optimized out>, path=0x7ffe8489d290) at /tmp/build/80754af9/python_1596467742699/work/Modules/posixmodule.c:2277
#10 os_stat_impl () at /tmp/build/80754af9/python_1596467742699/work/Modules/posixmodule.c:2615
#11 os_stat () at /tmp/build/80754af9/python_1596467742699/work/Modules/clinic/posixmodule.c.h:71
#12 0x00007fd9670ca8f4 in cfunction_vectorcall_FASTCALL_KEYWORDS () at /tmp/build/80754af9/python_1596467742699/work/Objects/methodobject.c:437
#13 0x00007fd96716acef in _PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7fd969249340, callable=0x7fd966ec53b0)
    at /tmp/build/80754af9/python_1596467742699/work/Include/cpython/abstract.h:127
#14 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x7fd968249b30) at /tmp/build/80754af9/python_1596467742699/work/Python/ceval.c:4963
#15 _PyEval_EvalFrameDefault () at /tmp/build/80754af9/python_1596467742699/work/Python/ceval.c:3469
#16 0x00007fd967111bdc in _PyEval_EvalCodeWithName () at /tmp/build/80754af9/python_1596467742699/work/Python/ceval.c:4298
#17 0x00007fd967112cc5 in _PyFunction_Vectorcall.localalias.355 () at /tmp/build/80754af9/python_1596467742699/work/Objects/call.c:435
#18 0x00007fd96716acef in _PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7fd9690f2d88, callable=0x7fd95f8a5280)
    at /tmp/build/80754af9/python_1596467742699/work/Include/cpython/abstract.h:127
#19 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x7fd968249b30) at /tmp/build/80754af9/python_1596467742699/work/Python/ceval.c:4963
#20 _PyEval_EvalFrameDefault () at /tmp/build/80754af9/python_1596467742699/work/Python/ceval.c:3469
#21 0x00007fd967112355 in _PyEval_EvalCodeWithName () at /tmp/build/80754af9/python_1596467742699/work/Python/ceval.c:4298
#22 0x00007fd967112d68 in _PyFunction_Vectorcall.localalias.355 () at /tmp/build/80754af9/python_1596467742699/work/Objects/call.c:435
#23 0x00007fd9670ef6c5 in _PyObject_Vectorcall (kwnames=<optimized out>, nargsf=<optimized out>, args=0x7fd957d448c0, callable=0x7fd95f8a4280)
    at /tmp/build/80754af9/python_1596467742699/work/Include/cpython/abstract.h:127
#24 method_vectorcall () at /tmp/build/80754af9/python_1596467742699/work/Objects/classobject.c:60
#25 0x00007fd96716741e in _PyObject_Vectorcall (kwnames=0x7fd95f344c40, nargsf=<optimized out>, args=<optimized out>, callable=0x7fd92b170ec0)
    at /tmp/build/80754af9/python_1596467742699/work/Include/cpython/abstract.h:127
#26 call_function (kwnames=0x7fd95f344c40, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=<optimized out>) at /tmp/build/80754af9/python_1596467742699/work/Python/ceval.c:4963
#27 _PyEval_EvalFrameDefault () at /tmp/build/80754af9/python_1596467742699/work/Python/ceval.c:3515
#28 0x00007fd967111bdc in _PyEval_EvalCodeWithName () at /tmp/build/80754af9/python_1596467742699/work/Python/ceval.c:4298
#29 0x00007fd967112cc5 in _PyFunction_Vectorcall.localalias.355 () at /tmp/build/80754af9/python_1596467742699/work/Objects/call.c:435
#30 0x00007fd96716acef in _PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7fd8dc0231e0, callable=0x7fd95f3453a0)
    at /tmp/build/80754af9/python_1596467742699/work/Include/cpython/abstract.h:127
#31 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x7fd968249b30) at /tmp/build/80754af9/python_1596467742699/work/Python/ceval.c:4963
#32 _PyEval_EvalFrameDefault () at /tmp/build/80754af9/python_1596467742699/work/Python/ceval.c:3469
#33 0x00007fd967111bdc in _PyEval_EvalCodeWithName () at /tmp/build/80754af9/python_1596467742699/work/Python/ceval.c:4298
#34 0x00007fd967112cc5 in _PyFunction_Vectorcall.localalias.355 () at /tmp/build/80754af9/python_1596467742699/work/Objects/call.c:435
#35 0x00007fd9670ef6c5 in _PyObject_Vectorcall (kwnames=<optimized out>, nargsf=<optimized out>, args=0x7fd95b8871e8, callable=0x7fd95f34b5e0)
    at /tmp/build/80754af9/python_1596467742699/work/Include/cpython/abstract.h:127
#36 method_vectorcall () at /tmp/build/80754af9/python_1596467742699/work/Objects/classobject.c:60
#37 0x00007fd96716acef in _PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7fd95b8871f0, callable=0x7fd95b99c980)
    at /tmp/build/80754af9/python_1596467742699/work/Include/cpython/abstract.h:127
#38 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x7fd968249b30) at /tmp/build/80754af9/python_1596467742699/work/Python/ceval.c:4963
#39 _PyEval_EvalFrameDefault () at /tmp/build/80754af9/python_1596467742699/work/Python/ceval.c:3469
#40 0x00007fd9671120a9 in _PyEval_EvalCodeWithName () at /tmp/build/80754af9/python_1596467742699/work/Python/ceval.c:4298
#41 0x00007fd967113278 in _PyFunction_Vectorcall (kwnames=0x0, nargsf=140571581883584, stack=0x7fd957cb7390, func=0x7fd95f34bc10)
    at /tmp/build/80754af9/python_1596467742699/work/Objects/call.c:435
#42 _PyObject_FastCallDict () at /tmp/build/80754af9/python_1596467742699/work/Objects/call.c:96
#43 0x00007fd9671135be in _PyObject_Call_Prepend () at /tmp/build/80754af9/python_1596467742699/work/Objects/call.c:887
#44 0x00007fd9671136fa in slot_tp_init () at /tmp/build/80754af9/python_1596467742699/work/Objects/typeobject.c:6787
#45 0x00007fd9670c6cbc in type_call (kwds=0x0, args=0x7fd957d26c70, type=<optimized out>) at /tmp/build/80754af9/python_1596467742699/work/Objects/typeobject.c:994
#46 _PyObject_MakeTpCall () at /tmp/build/80754af9/python_1596467742699/work/Objects/call.c:159
#47 0x00007fd96716b29a in _PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7fd95b8875c8, callable=0x7fd968377c90)
    at /tmp/build/80754af9/python_1596467742699/work/Include/cpython/abstract.h:125
#48 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x7fd968249b30) at /tmp/build/80754af9/python_1596467742699/work/Python/ceval.c:4963
#49 _PyEval_EvalFrameDefault () at /tmp/build/80754af9/python_1596467742699/work/Python/ceval.c:3469
#50 0x00007fd967112355 in _PyEval_EvalCodeWithName () at /tmp/build/80754af9/python_1596467742699/work/Python/ceval.c:4298
#51 0x00007fd967112d68 in _PyFunction_Vectorcall.localalias.355 () at /tmp/build/80754af9/python_1596467742699/work/Objects/call.c:435
#52 0x00007fd9670ef852 in _PyObject_Vectorcall (kwnames=0x7fd957cb7160, nargsf=5, args=0x7fd957cb7450, callable=0x7fd95f242c10)
    at /tmp/build/80754af9/python_1596467742699/work/Include/cpython/abstract.h:127
#53 method_vectorcall () at /tmp/build/80754af9/python_1596467742699/work/Objects/classobject.c:89
#54 0x00007fd9670c660f in PyVectorcall_Call () at /tmp/build/80754af9/python_1596467742699/work/Objects/call.c:199
#55 0x00007fd9670c678e in PyObject_Call () at /tmp/build/80754af9/python_1596467742699/work/Objects/call.c:227
#56 0x00007fd967167e06 in do_call_core (kwdict=0x7fd957cba180, callargs=0x7fd957cb9900, func=0x7fd92b170e80, tstate=<optimized out>)
    at /tmp/build/80754af9/python_1596467742699/work/Python/ceval.c:5010
#57 _PyEval_EvalFrameDefault () at /tmp/build/80754af9/python_1596467742699/work/Python/ceval.c:3559
#58 0x00007fd967112355 in _PyEval_EvalCodeWithName () at /tmp/build/80754af9/python_1596467742699/work/Python/ceval.c:4298
#59 0x00007fd967112d1e in _PyFunction_Vectorcall.localalias.355 () at /tmp/build/80754af9/python_1596467742699/work/Objects/call.c:435
#60 0x00007fd967166468 in _PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7fd95b887b88, callable=0x7fd95f242b80)
    at /tmp/build/80754af9/python_1596467742699/work/Include/cpython/abstract.h:127
#61 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x7fd968249b30) at /tmp/build/80754af9/python_1596467742699/work/Python/ceval.c:4963
#62 _PyEval_EvalFrameDefault () at /tmp/build/80754af9/python_1596467742699/work/Python/ceval.c:3486
#63 0x00007fd96710b722 in gen_send_ex (closing=0, exc=0, arg=0xffffffff00000000, gen=0x7fd957d24840) at /tmp/build/80754af9/python_1596467742699/work/Objects/genobject.c:222
#64 _PyGen_Send () at /tmp/build/80754af9/python_1596467742699/work/Objects/genobject.c:292
#65 0x00007fd967167a3c in _PyEval_EvalFrameDefault () at /tmp/build/80754af9/python_1596467742699/work/Python/ceval.c:2053
#66 0x00007fd96710b5dc in gen_send_ex (closing=0, exc=0, arg=<optimized out>, gen=0x7fd95e9ec940) at /tmp/build/80754af9/python_1596467742699/work/Objects/genobject.c:222
#67 _PyGen_Send () at /tmp/build/80754af9/python_1596467742699/work/Objects/genobject.c:292
#68 0x00007fd95f2c9349 in task_step_impl (exc=<optimized out>, task=0x7fd92baaa720) at /usr/local/src/conda/python-3.8.5/Modules/_asynciomodule.c:2641
#69 task_step (exc=0x0, task=0x7fd92baaa720) at /usr/local/src/conda/python-3.8.5/Modules/_asynciomodule.c:2934
#70 task_wakeup (o=<optimized out>, task=0x7fd92baaa720) at /usr/local/src/conda/python-3.8.5/Modules/_asynciomodule.c:2971
#71 TaskWakeupMethWrapper_call () at /usr/local/src/conda/python-3.8.5/Modules/_asynciomodule.c:1791
#72 0x00007fd9670c6ddf in _PyObject_MakeTpCall () at /tmp/build/80754af9/python_1596467742699/work/Objects/call.c:159
#73 0x00007fd96719a772 in context_run () at /tmp/build/80754af9/python_1596467742699/work/Python/context.c:634
#74 0x00007fd9670ca8f4 in cfunction_vectorcall_FASTCALL_KEYWORDS () at /tmp/build/80754af9/python_1596467742699/work/Objects/methodobject.c:437
#75 0x00007fd9670c660f in PyVectorcall_Call () at /tmp/build/80754af9/python_1596467742699/work/Objects/call.c:199
#76 0x00007fd9671060c6 in PyCFunction_Call () at /tmp/build/80754af9/python_1596467742699/work/Objects/call.c:774
#77 0x00007fd96716ba8b in do_call_core (kwdict=0x0, callargs=0x7fd95a406d80, func=0x7fd95a46fd60, tstate=<optimized out>) at /tmp/build/80754af9/python_1596467742699/work/Python/ceval.c:4983
#78 _PyEval_EvalFrameDefault () at /tmp/build/80754af9/python_1596467742699/work/Python/ceval.c:3559
#79 0x00007fd967112bfb in function_code_fastcall (globals=<optimized out>, nargs=1, args=<optimized out>, co=<optimized out>) at /tmp/build/80754af9/python_1596467742699/work/Objects/call.c:283
#80 _PyFunction_Vectorcall.localalias.355 () at /tmp/build/80754af9/python_1596467742699/work/Objects/call.c:410
#81 0x00007fd967166468 in _PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7fd969255e88, callable=0x7fd95f34bb80)
    at /tmp/build/80754af9/python_1596467742699/work/Include/cpython/abstract.h:127
#82 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x7fd968249b30) at /tmp/build/80754af9/python_1596467742699/work/Python/ceval.c:4963
#83 _PyEval_EvalFrameDefault () at /tmp/build/80754af9/python_1596467742699/work/Python/ceval.c:3486
#84 0x00007fd967112bfb in function_code_fastcall (globals=<optimized out>, nargs=1, args=<optimized out>, co=<optimized out>) at /tmp/build/80754af9/python_1596467742699/work/Objects/call.c:283
#85 _PyFunction_Vectorcall.localalias.355 () at /tmp/build/80754af9/python_1596467742699/work/Objects/call.c:410
#86 0x00007fd967166468 in _PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7fd957d64ee0, callable=0x7fd95f246280)
    at /tmp/build/80754af9/python_1596467742699/work/Include/cpython/abstract.h:127
#87 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x7fd968249b30) at /tmp/build/80754af9/python_1596467742699/work/Python/ceval.c:4963
#88 _PyEval_EvalFrameDefault () at /tmp/build/80754af9/python_1596467742699/work/Python/ceval.c:3486
#89 0x00007fd967112bfb in function_code_fastcall (globals=<optimized out>, nargs=1, args=<optimized out>, co=<optimized out>) at /tmp/build/80754af9/python_1596467742699/work/Objects/call.c:283
#90 _PyFunction_Vectorcall.localalias.355 () at /tmp/build/80754af9/python_1596467742699/work/Objects/call.c:410
#91 0x00007fd967166468 in _PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7fd957d64768, callable=0x7fd95f242700)
    at /tmp/build/80754af9/python_1596467742699/work/Include/cpython/abstract.h:127
#92 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x7fd968249b30) at /tmp/build/80754af9/python_1596467742699/work/Python/ceval.c:4963
#93 _PyEval_EvalFrameDefault () at /tmp/build/80754af9/python_1596467742699/work/Python/ceval.c:3486
#94 0x00007fd967112bfb in function_code_fastcall (globals=<optimized out>, nargs=2, args=<optimized out>, co=<optimized out>) at /tmp/build/80754af9/python_1596467742699/work/Objects/call.c:283
#95 _PyFunction_Vectorcall.localalias.355 () at /tmp/build/80754af9/python_1596467742699/work/Objects/call.c:410
#96 0x00007fd967166468 in _PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7fd969223648, callable=0x7fd95f242790)
    at /tmp/build/80754af9/python_1596467742699/work/Include/cpython/abstract.h:127
#97 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x7fd968249b30) at /tmp/build/80754af9/python_1596467742699/work/Python/ceval.c:4963
#98 _PyEval_EvalFrameDefault () at /tmp/build/80754af9/python_1596467742699/work/Python/ceval.c:3486
#99 0x00007fd967112355 in _PyEval_EvalCodeWithName () at /tmp/build/80754af9/python_1596467742699/work/Python/ceval.c:4298
#100 0x00007fd967112d68 in _PyFunction_Vectorcall.localalias.355 () at /tmp/build/80754af9/python_1596467742699/work/Objects/call.c:435
#101 0x00007fd96716741e in _PyObject_Vectorcall (kwnames=0x7fd966e63820, nargsf=<optimized out>, args=<optimized out>, callable=0x7fd95f23fee0)
    at /tmp/build/80754af9/python_1596467742699/work/Include/cpython/abstract.h:127
#102 call_function (kwnames=0x7fd966e63820, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=<optimized out>) at /tmp/build/80754af9/python_1596467742699/work/Python/ceval.c:4963
#103 _PyEval_EvalFrameDefault () at /tmp/build/80754af9/python_1596467742699/work/Python/ceval.c:3515
#104 0x00007fd967111bdc in _PyEval_EvalCodeWithName () at /tmp/build/80754af9/python_1596467742699/work/Python/ceval.c:4298
#105 0x00007fd967112ac4 in PyEval_EvalCodeEx () at /tmp/build/80754af9/python_1596467742699/work/Python/ceval.c:4327
#106 0x00007fd967112aec in PyEval_EvalCode (co=co@entry=0x7fd966dda0e0, globals=globals@entry=0x7fd966ed6c00, locals=locals@entry=0x7fd966ed6c00)
    at /tmp/build/80754af9/python_1596467742699/work/Python/ceval.c:718
#107 0x00007fd9671b46d4 in run_eval_code_obj () at /tmp/build/80754af9/python_1596467742699/work/Python/pythonrun.c:1125
#108 0x00007fd9671dffa4 in run_mod () at /tmp/build/80754af9/python_1596467742699/work/Python/pythonrun.c:1147
#109 0x00007fd9671e2851 in PyRun_FileExFlags () at /tmp/build/80754af9/python_1596467742699/work/Python/pythonrun.c:1063
#110 0x00007fd9671e2a41 in PyRun_SimpleFileExFlags () at /tmp/build/80754af9/python_1596467742699/work/Python/pythonrun.c:428
#111 0x00007fd9671e2b5d in PyRun_AnyFileExFlags () at /tmp/build/80754af9/python_1596467742699/work/Python/pythonrun.c:86
#112 0x00007fd9671e3678 in pymain_run_file (cf=0x7ffe8489f738, config=0x7fd968248be0) at /tmp/build/80754af9/python_1596467742699/work/Modules/main.c:387
#113 pymain_run_python (exitcode=0x7ffe8489f730) at /tmp/build/80754af9/python_1596467742699/work/Modules/main.c:612
#114 Py_RunMain () at /tmp/build/80754af9/python_1596467742699/work/Modules/main.c:691
#115 0x00007fd9671e37ff in pymain_main () at /tmp/build/80754af9/python_1596467742699/work/Modules/main.c:1099
#116 0x00007fd9671e3869 in Py_BytesMain () at /tmp/build/80754af9/python_1596467742699/work/Modules/main.c:1123
#117 0x00007fd9667a0b15 in __libc_start_main () from /lib64/libc.so.6
#118 0x00007fd9671853a3 in _start () at ../sysdeps/x86_64/elf/start.S:103

0 replies

sc1f · 2021-02-03T14:53:01Z

sc1f
Feb 3, 2021

From debugging past segfaults, when Perspective is concretely involved in the segfault there should be a function signature/header from Perspective in the trace (even on non-debug builds). Even though this segfault only contains a trace from CPython, it is more than likely that it was triggered by something in Perspective's C++/Python bindings, such as a broken refcount/GIL unlock (based on this StackOverflow answer and similar GitHub issue).

I wonder if this may be due to the GIL release on async mode - try removing the perspective_thread so that Perspective runs without unlocking the GIL, and if the segfault does not happen on your machine then we have at least narrowed it down to a specific area of code. I tried running it with the for p in range(i) added and while the speed of the updates blocked me from getting the front-end HTML rendered, adding a print statement in each iteration of the while loop showed that the loop was running just fine.

0 replies

choandrew · 2021-02-03T15:05:51Z

choandrew
Feb 3, 2021
Author

You are correct. I removed the GIL release/async mode lines

    thread = threading.Thread(target=perspective_thread, args=(MANAGER,))
    thread.daemon = True
    thread.start()

and the difference is immediate. This runs forever and no longer segfaults.

I will cease to do the GIL release for now.

0 replies

sc1f · 2021-02-03T15:29:46Z

sc1f
Feb 3, 2021

That makes sense - any attempt at accessing a ref-counted object while the GIL is released would cause a segfault, and that seems to be clearly what's happening here. @texodus this seems pretty easy to narrow down if we can repro it on our Macs - I can't think of any staggering differences btwn OSX and Linux off the top of my head that would affect this sort of behaviour.

0 replies

choandrew · 2021-04-15T22:00:46Z

choandrew
Apr 15, 2021
Author

@sc1f

I tried using the GIL release / async mode again on 0.6.2 because I am trying to squeeze out a little bit more performance.
There is still a crash, although there is an actual error log this time.

The program crashes around 1/3 the time, sometimes immediately, sometimes after 45 seconds.

Environment:

python 3.8.5
perspective-python 0.6.2
linux with conda

Error Messages

There are a couple different possible error messages

Traceback (most recent call last):
  File "GIL_crash.py", line 143, in <module>
    asyncio.run(main(domain, port, table_name), debug=True)
  File "/home/acho/conda/envs/ps/lib/python3.8/asyncio/runners.py", line 43, in run
    return loop.run_until_complete(main)
  File "/home/acho/conda/envs/ps/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "GIL_crash.py", line 133, in main
    BASE_TABLE.update(
  File "/home/acho/conda/envs/ps/lib/python3.8/site-packages/perspective/table/table.py", line 353, in update
    self._table = make_table(
perspective.table.libbinding.PerspectiveCppError: touching uninited object
terminate called without an active exception
Aborted (core dumped)

sometimes these errors can pop up:

*** Error in `*** Error in `python': double free or corruption (!prev): 0x00007f6bb0002910 ***
python': realloc(): invalid pointer: 0x*** Error in `00007f6ba8000498 ***
*** Error in `python': realloc(): invalid old size: 0x00007f6ba800e3c0 ***
Aborted (core dumped)

*** Error in `python': realloc(): invalid old size: 0x00007f0778028d80 ***
*** Error in `python': *** Error in `python': realloc(): invalid next size: 0x*** Error in `python': realloc(): invalid pointer: 0x00007f07b80025d0 ***
*** Error in `python': corrupted double-linked list: 0x00007f07a8001970 ***
Aborted (core dumped)

*** Error in `python': realloc(): invalid pointer: 0x00007f1160019330 ***
Aborted (core dumped)

Minimum Reproduction Code:

import asyncio
import threading

import tornado
import tornado.web
import tornado.ioloop
import tornado.gen

import perspective


class MainHandler(tornado.web.RequestHandler):
    def initialize(self, domain: str, port: int, table_name: str):
        self.domain = domain
        self.port = port
        self.table_name = table_name

    def get(self):
        print("USER CONNECTED")
        html = f"""<!DOCTYPE html>
<html>

<head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
    <title>test</title>

    <meta name="viewport"
        content="width=device-width, initial-scale=1, maximum-scale=1, minimum-scale=1, user-scalable=no">
        <script src="https://cdn.jsdelivr.net/npm/@finos/perspective-viewer"></script>
        <script src="https://cdn.jsdelivr.net/npm/@finos/perspective-viewer-datagrid"></script>
        <script src="https://cdn.jsdelivr.net/npm/@finos/perspective-viewer-d3fc"></script>
        <script src="https://cdn.jsdelivr.net/npm/@finos/perspective/dist/umd/perspective.js"></script>

</head>

<style>
    perspective-viewer {{
        position: absolute;
        top: 0;
        left: 0;
        right: 0;
        bottom: 0;
    }}
</style>

<body>
    <perspective-viewer id="viewer" plugin="datagrid"></perspective-viewer>
    <script>
        websocket_url = 'ws://{self.domain}:{self.port}/websocket'
        table_name = '{self.table_name}'

        window.addEventListener('WebComponentsReady', async function () {{
            const websocket = perspective.websocket(websocket_url);
            const worker = perspective.worker();

            const server_table = websocket.open_table(table_name);
            const server_view = server_table.view();
            const index = await server_table.get_index();
            const table = worker.table(server_view, {{index: index}});

            document.getElementById('viewer').load(table);

        }});
    </script>
</body>

</html>"""
        self.write(html)


def make_app(
    MANAGER: perspective.PerspectiveManager, domain: str, port: int, table_name: str
):
    handlers = [
        (
            r"/websocket",
            perspective.PerspectiveTornadoHandler,
            {"manager": MANAGER, "check_origin": True},
        ),
        (r"/", MainHandler, {"domain": domain, "port": port, "table_name": table_name}),
    ]
    app = tornado.web.Application(handlers)
    return app


def perspective_thread(manager: perspective.PerspectiveManager):
    loop = tornado.ioloop.IOLoop()
    manager.set_loop_callback(loop.add_callback)
    loop.start()


def updater(table):
    def update_function(port, delta):
        table.update(delta)

    return update_function


async def main(domain: str, port: int, table_name: str):

    columns = {"index": int, "num1": int, "num2": int}
    MANAGER = perspective.PerspectiveManager()
    BASE_TABLE = perspective.Table(columns, index="index")

    thread = threading.Thread(target=perspective_thread, args=(MANAGER,))
    thread.daemon = True
    thread.start()

    app = make_app(MANAGER, domain, port, table_name)
    app.listen(port)
    print(f"Using port {port}")

    i = 0

    BASE_TABLE.update([{"index": i, "num1": i, "num2": 2 * i}])
    while True:
        await asyncio.sleep(0.3)
        BASE_TABLE.update([{"index": i, "num1": i, "num2": 2 * i}])
        BASE_TABLE.update(
            [{"index": p, "num1": p % 3, "num2": 2 * p + i} for p in range(i)]
        )
        i += 1


if __name__ == "__main__":
    domain = "localhost"
    port = 8080
    table_name = "base_table"
    asyncio.run(main(domain, port, table_name), debug=True)

0 replies

sc1f · 2021-04-15T22:21:46Z

sc1f
Apr 15, 2021

In async mode, once you have started the Perspective thread, all operations that touch the Perspective table has to be on that thread. In this example, calling BASE_TABLE.update() from the main thread is not threadsafe and can cause issues in a non-deterministic way.

To call update safely, use MANAGER._loop_callback so that the update operation happens on the Perspective thread and not the main thread:

MANAGER._loop_callback(BASE_TABLE.update, {"index": i, "num1": i, "num2": 2 * i}])

Additionally, if you are able to get the coredumps for the segfaulting process that will help us narrow down where the issue is. I think the earlier example you raised could be fixed with calling update through _loop_callback also, although I did not spot that fix the first time around - my bad.

0 replies

texodus · 2021-04-16T05:16:12Z

texodus
Apr 16, 2021
Maintainer

Slight correction - don't use manager._loop_callback (as the underscore prefix implies, this property is private), rather keep & re-use the reference to loop.add_callback initially supplied to the set_loop_callback(). In 0.7.0, you can use manager.call_loop() here as well.

0 replies

choandrew · 2021-04-16T13:58:49Z

choandrew
Apr 16, 2021
Author

Ah yes. You guys are completely correct.
I just misunderstood the async mode docs and I see where the code is wrong.

I changed the update to be called within the tornado loop callback and the program no longer segfaults.

That is my bad, thank you for the help and for working on this terrific project.

0 replies

sc1f · 2021-04-16T14:10:10Z

sc1f
Apr 16, 2021

No problem - the docs could be clearer on our side as well, so thank you for bringing this issue up. call_loop will be in 0.7.0 and that'll make the requirement to call all Perspective methods on the Perspective thread much clearer, I think.

0 replies

Crash on Async Mode / GIL release #1383

Uh oh!

choandrew Feb 3, 2021

Bug Report

Steps to Reproduce:

Environment:

Code Template to Reproduce

Replies: 10 comments

Uh oh!

Uh oh!

sc1f Feb 3, 2021

Uh oh!

choandrew Feb 3, 2021 Author

Uh oh!

sc1f Feb 3, 2021

Uh oh!

choandrew Feb 3, 2021 Author

Uh oh!

sc1f Feb 3, 2021

Uh oh!

Uh oh!

choandrew Apr 15, 2021 Author

Environment:

Error Messages

Minimum Reproduction Code:

Uh oh!

sc1f Apr 15, 2021

Uh oh!

texodus Apr 16, 2021 Maintainer

Uh oh!

choandrew Apr 16, 2021 Author

Uh oh!

sc1f Apr 16, 2021

choandrew
Feb 3, 2021

sc1f
Feb 3, 2021

choandrew
Feb 3, 2021
Author

sc1f
Feb 3, 2021

choandrew
Feb 3, 2021
Author

sc1f
Feb 3, 2021

choandrew
Apr 15, 2021
Author

sc1f
Apr 15, 2021

texodus
Apr 16, 2021
Maintainer

choandrew
Apr 16, 2021
Author

sc1f
Apr 16, 2021