Cancelled error with langgraph runs

[kedarsp-informa]

I have an inconsistent issue as CancelledError().
I have a langgraph and the error seems to be within langgraph library.
I am also using asyncio in my code as:

workflow.add_node(RAG, lambda state: asyncio.run(generate_with_rag(state=state, generation_rag_chain=generation_rag_chain)))
workflow.add_node(STORE_CHAT, lambda state: asyncio.run(store_chat(state=state, memory=memory)))

and

if chat_history:
        task1 = asyncio.create_task(router_chain.ainvoke({"question": question, "chat_history": state['chat_history']}))
        task2 = asyncio.create_task(rephrase_chain.ainvoke({"question": question, "chat_history": state['chat_history']}))
        source, follow_up_question = await asyncio.gather(task1, task2)
    else:
        source = await asyncio.shield(router_chain.ainvoke({"question": question, "chat_history": state['chat_history']}))
        follow_up_question = question

The below is the error trace:
_CancelledError()Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/langgraph/pregel/init.py", line 1189, in astream
done, inflight = await asyncio.wait(
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/asyncio/tasks.py", line 418, in wait
return await _wait(fs, timeout, return_when, loop)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/asyncio/tasks.py", line 525, in wait
await waiter
asyncio.exceptions.CancelledError
During handling of the above exception, another exception occurred:
Traeback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/langgraph/pregel/init.py", line 1132, in astream
async with AsyncPregelLoop(
asyncio.exceptions.CancelledError
If anyone has encountered this error before or any idea on this would be helpful. Thanks

[delorenzotommaso]

I get the same error when execution time reaches about 60s

[tommasodelorenzo]

Hi, nope. It is a graph called from a fastAPI endpoint deployed as an azure container app. Logging everything to LangSmith

[vikyw89]

same, i'm experiencing the same thing on latest langgraph, cancelled error when ainvoke last about 60s. And i'm not using asyncio run, I do use asyncer asyncify inside the node, not to invoke the langgraph

[vikyw89]

I think it's internal langgraph timeout

[vikyw89]

Cancelled by cancel scope 7fea082e1520

[vikyw89]

Tried with await sleep, couldnt replicate this one...

[nfcampos]

Hi @kedarsp-informa you shouldn't use asyncio.run in add_node we have native support for async functions in nodes. Using asyncio.run would create a new event loop for each node, which is very inefficient and can cause issues such as cancellation not propagating from graph to node etc

[NicholasMicali]

I get the same error when attempting to run a subprocess to install a linter on startup. I am using langgraph cloud. Unsure about what "shouldn't use asyncio.run in add_node" means. this is a very simple graph, which also successfully runs my subprocess to install a linter, but fails afterwards. Deploys successfully without this line:

subprocess.run(["bash", "build_linter.sh"], check=True)

Registering graph with id 'agent'
12/17/2024, 1:46:06 PM
Starting metadata loop
12/17/2024, 1:46:07 PM
Starting cron scheduler
12/17/2024, 1:46:07 PM
Starting 10 background workers
12/17/2024, 1:46:07 PM
Application startup complete.
12/17/2024, 1:46:07 PM
Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)
12/17/2024, 1:46:07 PM
Shutting down background workers
12/17/2024, 1:46:07 PM
Cancelling remote graphs
12/17/2024, 1:46:07 PM
Traceback (most recent call last):
File "/api/langgraph_api/lifespan.py", line 37, in lifespan
File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 700, in lifespan
await receive()
File "/usr/local/lib/python3.11/site-packages/uvicorn/lifespan/on.py", line 137, in receive
return await self.receive_queue.get()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/asyncio/queues.py", line 158, in get
await getter
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 693, in lifespan
async with self.lifespan_context(app) as maybe_state:
File "/usr/local/lib/python3.11/contextlib.py", line 231, in aexit
await self.gen.athrow(typ, value, traceback)
File "/api/langgraph_api/lifespan.py", line 41, in lifespan
File "/usr/local/lib/python3.11/site-packages/langgraph_storage/database.py", line 175, in stop_pool
await _pg_pool.close()
File "/usr/local/lib/python3.11/site-packages/psycopg_pool/pool_async.py", line 474, in close
await agather(*workers, timeout=timeout)
File "/usr/local/lib/python3.11/site-packages/psycopg_pool/_acompat.py", line 142, in agather
await asyncio.wait_for(asyncio.shield(wait), timeout=timeout)
File "/usr/local/lib/python3.11/asyncio/tasks.py", line 489, in wait_for
return fut.result()
^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/psycopg_pool/pool_async.py", line 627, in worker
task = await q.get()
^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/asyncio/queues.py", line 158, in get
await getter
asyncio.exceptions.CancelledError

[n-sviridenko]

@nfcampos I have this issue too - wrote to support and shared traces. In my case there are no asyncio.run calls - only asyncio.gather (but that one has to be for parallelisation). The rest is async/await.

[n-sviridenko]

Could this issue be related to having no listeners on the stream (as the run it purely a background one)? It seems to occur only under those circumstances on LangGraph Cloud.

[codebreach]

We are also facing this and are not doing any asyncio.run in nodes. Its getting a timeout from redis which is caused by cancelled error

  File "/usr/local/lib/python3.11/site-packages/coredis/connection.py", line 584, in fetch_push_message
    await self._read_flag.wait()
  File "/usr/local/lib/python3.11/asyncio/locks.py", line 213, in wait
    await fut
DEFAULT 2025-02-12T08:32:30.866416Z asyncio.exceptions.CancelledError
DEFAULT 2025-02-12T08:32:30.866422Z The above exception was the direct cause of the following exception:
DEFAULT 2025-02-12T08:32:30.866440Z TimeoutError
DEFAULT 2025-02-12T08:32:30.866445Z During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/langgraph_storage/ops.py", line 2029, in listen_for_cancellation
    event = await pubsub.listen()
DEFAULT 2025-02-12T08:32:30.866457Z ^^^^^^^^^^^^^^^^^^^^^
DEFAULT 2025-02-12T08:32:30.866462Z File "/usr/local/lib/python3.11/site-packages/coredis/commands/pubsub.py", line 322, in listen
DEFAULT 2025-02-12T08:32:30.866466Z response = await self.parse_response(block=True)
DEFAULT 2025-02-12T08:32:30.866472Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
DEFAULT 2025-02-12T08:32:30.866476Z File "/usr/local/lib/python3.11/site-packages/coredis/commands/pubsub.py", line 242, in parse_response
DEFAULT 2025-02-12T08:32:30.866480Z return await asyncio.wait_for(
DEFAULT 2025-02-12T08:32:30.866485Z ^^^^^^^^^^^^^^^^^^^^^^^
DEFAULT 2025-02-12T08:32:30.866488Z File "/usr/local/lib/python3.11/asyncio/tasks.py", line 452, in wait_for
DEFAULT 2025-02-12T08:32:30.866492Z return await fut
DEFAULT 2025-02-12T08:32:30.866495Z ^^^^^^^^^
DEFAULT 2025-02-12T08:32:30.866500Z File "/usr/local/lib/python3.11/site-packages/coredis/commands/pubsub.py", line 216, in _execute
DEFAULT 2025-02-12T08:32:30.866504Z return await command(*args)
DEFAULT 2025-02-12T08:32:30.866509Z ^^^^^^^^^^^^^^^^^^^^
DEFAULT 2025-02-12T08:32:30.866514Z File "/usr/local/lib/python3.11/site-packages/coredis/connection.py", line 586, in fetch_push_message
DEFAULT 2025-02-12T08:32:30.866519Z raise TimeoutError
DEFAULT 2025-02-12T08:32:30.866526Z coredis.exceptions.TimeoutError

[edgarrmondragon]

I'm running a similar issue with the prebuilt ReAct agent:

CancelledError('Cancelled by cancel scope fffef75353a0', <Task cancelled name='Task-107483' coro=<AsyncExitStack.__aexit__() done, defined at /usr/local/lib/python3.12/contextlib.py:707>>)Traceback (most recent call last):

  File "/usr/local/lib/python3.12/site-packages/langgraph/pregel/__init__.py", line 2007, in astream
    async for _ in runner.atick(

  File "/usr/local/lib/python3.12/site-packages/langgraph/pregel/runner.py", line 495, in atick
    done, inflight = await asyncio.wait(
                     ^^^^^^^^^^^^^^^^^^^

  File "/usr/local/lib/python3.12/asyncio/tasks.py", line 464, in wait
    return await _wait(fs, timeout, return_when, loop)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/local/lib/python3.12/asyncio/tasks.py", line 550, in _wait
    await waiter

asyncio.exceptions.CancelledError: Cancelled by cancel scope fffef75353a0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "/usr/local/lib/python3.12/site-packages/langgraph/pregel/__init__.py", line 1960, in astream
    async with AsyncPregelLoop(
               ^^^^^^^^^^^^^^^^

  File "/usr/local/lib/python3.12/site-packages/langgraph/pregel/loop.py", line 1103, in __aexit__
    return await exit_task
           ^^^^^^^^^^^^^^^

  File "/usr/local/lib/python3.12/contextlib.py", line 707, in __aexit__
    async def __aexit__(self, *exc_details):

asyncio.exceptions.CancelledError: ('Cancelled by cancel scope fffef75353a0', <Task cancelled name='Task-107483' coro=<AsyncExitStack.__aexit__() done, defined at /usr/local/lib/python3.12/contextlib.py:707>>)

My environment:

langchain_core_version: "0.3.34"
langchain_version: "0.3.18"
library: "langsmith"
platform: "Linux-6.1.112-122.189.amzn2023.aarch64-aarch64-with-glibc2.36"
py_implementation: "CPython"
runtime: "python"
runtime_version: "3.12.9"
sdk: "langsmith-py"
sdk_version: "0.3.8"

Unfortunately I've not been to consistently reproduce it.

[ehsanahmadkhan525]

I have the similar issue

CancelledError('Cancelled by cancel scope 3e9aadaf5ed0', <Task cancelled name='Task-393' coro=<AsyncExitStack.aexit() done, defined at /usr/local/lib/python3.11/contextlib.py:698>>)Traceback (most recent call last):

File "/usr/local/lib/python3.11/site-packages/langgraph/pregel/init.py", line 2080, in astream
async for _ in runner.atick(

File "/usr/local/lib/python3.11/site-packages/langgraph/pregel/runner.py", line 495, in atick
done, inflight = await asyncio.wait(
^^^^^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.11/asyncio/tasks.py", line 428, in wait
return await _wait(fs, timeout, return_when, loop)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.11/asyncio/tasks.py", line 535, in _wait
await waiter

asyncio.exceptions.CancelledError: Cancelled by cancel scope 3e9aadaf5ed0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File "/usr/local/lib/python3.11/site-packages/langgraph/pregel/init.py", line 2033, in astream
async with AsyncPregelLoop(

File "/usr/local/lib/python3.11/site-packages/langgraph/pregel/loop.py", line 1103, in aexit
return await exit_task
^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.11/contextlib.py", line 698, in aexit
async def aexit(self, *exc_details):

asyncio.exceptions.CancelledError: ('Cancelled by cancel scope 3e9aadaf5ed0', <Task cancelled name='Task-393' coro=<AsyncExitStack.aexit() done, defined at /usr/local/lib/python3.11/contextlib.py:698>>)

[ehsanahmadkhan525]

I have the similar issue

CancelledError('Cancelled by cancel scope 3e9aadaf5ed0', <Task cancelled name='Task-393' coro=<AsyncExitStack.aexit() done, defined at /usr/local/lib/python3.11/contextlib.py:698>>)Traceback (most recent call last):

File "/usr/local/lib/python3.11/site-packages/langgraph/pregel/init.py", line 2080, in astream
async for _ in runner.atick(

File "/usr/local/lib/python3.11/site-packages/langgraph/pregel/runner.py", line 495, in atick
done, inflight = await asyncio.wait(
^^^^^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.11/asyncio/tasks.py", line 428, in wait
return await _wait(fs, timeout, return_when, loop)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.11/asyncio/tasks.py", line 535, in _wait
await waiter

asyncio.exceptions.CancelledError: Cancelled by cancel scope 3e9aadaf5ed0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File "/usr/local/lib/python3.11/site-packages/langgraph/pregel/init.py", line 2033, in astream
async with AsyncPregelLoop(

File "/usr/local/lib/python3.11/site-packages/langgraph/pregel/loop.py", line 1103, in aexit
return await exit_task
^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.11/contextlib.py", line 698, in aexit
async def aexit(self, *exc_details):

asyncio.exceptions.CancelledError: ('Cancelled by cancel scope 3e9aadaf5ed0', <Task cancelled name='Task-393' coro=<AsyncExitStack.aexit() done, defined at /usr/local/lib/python3.11/contextlib.py:698>>)

[ldorigo]

We're facing a similar issue.

CancelledError: null
  File "/app/projects/chat/.venv/lib/python3.12/site-packages/langgraph/pregel/__init__.py", line 1413, in astream
    async for _ in runner.atick(
  File "/app/projects/chat/.venv/lib/python3.12/site-packages/langgraph/pregel/runner.py", line 126, in atick
    done, _ = await asyncio.wait(
  File "asyncio/tasks.py", line 464, in wait
    return await _wait(fs, timeout, return_when, loop)
  File "asyncio/tasks.py", line 550, in _wait
    await waiter

CancelledError: Cancelled by cancel scope 7fc5742666c0
  File "/app/projects/chat/.venv/lib/python3.12/site-packages/langgraph/pregel/__init__.py", line 1382, in astream
    async with AsyncPregelLoop(
  File "/app/projects/chat/.venv/lib/python3.12/site-packages/langgraph/pregel/loop.py", line 827, in __aexit__
    return await asyncio.shield(

Similarly to what's reported above; it seems to sometimes trigger when a node takes more than 1 minute to execute. Doesn't seem to do so reliably though. We don't use asyncio.run(). @nfcampos this discussion is marked as resolved but it is not - should we create a new one?

[apthagowda97]

Yes even I am seeing this issue? Even I am not using asyncio. @nfcampos

[hetavsundial]

I had been getting a similar with prebuilt react agent for a while. Adding a step timeout to the compile graph solved this for me.

graph = create_react_agent(...)
graph.step_timeout = 300 # in seconds

[ldnvnbl]

I tried，It still has Cancelled error

[ehsanahmadkhan525]

Here is how I increased the timeout for my langgraph code.

async def get_message_stream(self, message: Optional[str], base64_image: Optional[str], thread_id: str):
"""
Stream the response from the LLM using the graph with a global timeout
"""
import asyncio
config = {
"configurable": {
"thread_id": thread_id
}
# We'll use asyncio.wait_for instead of the config timeout
}

    # Prepare message content
    if base64_image:
        message_content = [
            {"type": "text", "text": message or ""},  # Use empty string if message is None
            {"type": "image_url", "image_url": {"url": base64_image}}
        ]
    else:
        message_content = message or ""  # Use empty string if message is None

    # This is the original streaming function
    async def generate_internal():
        seen_content = False
        async for chunk in self.graph.astream_events(
        {"messages": [
            {
                "role": "system",
                "content": self.SYSTEM_PROMPT + "Current time: " + datetime.utcnow().isoformat()
            },
            {
                "role": "user",
                "content": message_content
            }

        ]},
            config,
            stream_mode="messages",
            version="v1"
        ):

            if chunk["event"] == "on_chat_model_stream":
                data_chunk = chunk["data"]["chunk"]
                content = data_chunk.content
                
                # Output content chunks
                if content:
                    seen_content = True
                    yield f"data: {content}\n\n"
                # Only send usage metadata at the end (when content is empty and we've seen content before)
                elif seen_content:
                    usage_data = None
                    if hasattr(data_chunk, 'usage_metadata') and data_chunk.usage_metadata:
                        usage_data = data_chunk.usage_metadata
                    elif hasattr(data_chunk, 'response_metadata') and data_chunk.response_metadata and 'usage_metadata' in data_chunk.response_metadata:
                        usage_data = data_chunk.response_metadata['usage_metadata']
                    
                    if usage_data:
                        yield f"input_tokens: {usage_data['input_tokens']}\n\n"
                        yield f"output_tokens: {usage_data['output_tokens']}\n\n"


    # This is the wrapper function that adds the global timeout
    async def generate():
        # Set a global timeout of 15 minutes for the entire streaming operation
        # This acts as a safety net in case individual step timeouts don't trigger
        global_timeout = 900  # 15 minutes in seconds
        logging.info(f"Setting global timeout of {global_timeout} seconds for streaming operation")
        
        try:
            # Create the internal generator
            internal_generator = generate_internal()
            
            # Use a cleaner approach with asyncio.wait_for
            # We'll iterate through the generator and apply a timeout to each item
            while True:
                try:
                    # Wait for the next item with a timeout
                    item = await asyncio.wait_for(
                        internal_generator.__anext__(),
                        timeout=global_timeout
                    )
                    yield item
                except StopAsyncIteration:
                    # Generator is exhausted
                    break
                except asyncio.TimeoutError:
                    # Timeout occurred while waiting for the next item
                    error_message = "Operation timed out after waiting too long for a response"
                    logging.error(f"Timeout exceeded: {error_message}")
                    yield f"data: {error_message}\n\n"
                    break
        except Exception as e:
            # Handle other exceptions
            error_message = f"An error occurred: {str(e)}"
            logging.error(f"Error in streaming: {error_message}")
            yield f"data: {error_message}\n\n"
            
    return generate()

[hinthornw]

You may also want to try setting the env var BG_JOB_ISOLATED_LOOPS=true.

Some cancellations may be caused because your instance is being restarted due to failed health checks. A common cause of failed health checks is that you have some synchronous processes blocking the main event loop and slowing the server down.

We have made a lot of improvements to address this but you may want to keep this in mind if you are seeing inexplicable cancellations in your code today. We are working to add even more isolation in the future.

[ldnvnbl]

I am facing a similar issue.

CancelledError('Cancelled by cancel scope ffff78169e90', <Task cancelled name='Task-523786' coro=<AsyncExitStack.__aexit__() done, defined at /usr/local/lib/python3.13/contextlib.py:716>>)Traceback (most recent call last):


  File "/app/.venv/lib/python3.13/site-packages/langgraph/pregel/__init__.py", line 2313, in astream
    async for _ in runner.atick(
    ...<7 lines>...
            yield o


  File "/app/.venv/lib/python3.13/site-packages/langgraph/pregel/runner.py", line 495, in atick
    done, inflight = await asyncio.wait(
                     ^^^^^^^^^^^^^^^^^^^
    ...<3 lines>...
    )
    ^


  File "/usr/local/lib/python3.13/asyncio/tasks.py", line 451, in wait
    return await _wait(fs, timeout, return_when, loop)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^


  File "/usr/local/lib/python3.13/asyncio/tasks.py", line 537, in _wait
    await waiter


asyncio.exceptions.CancelledError: Cancelled by cancel scope ffff78169e90



During handling of the above exception, another exception occurred:



Traceback (most recent call last):


  File "/app/.venv/lib/python3.13/site-packages/langgraph/pregel/__init__.py", line 2265, in astream
    async with AsyncPregelLoop(
               ~~~~~~~~~~~~~~~^
        input,
        ^^^^^^
    ...<12 lines>...
        debug=debug,
        ^^^^^^^^^^^^
    ) as loop:
    ^


  File "/app/.venv/lib/python3.13/site-packages/langgraph/pregel/loop.py", line 1136, in __aexit__
    return await exit_task
           ^^^^^^^^^^^^^^^


  File "/usr/local/lib/python3.13/contextlib.py", line 716, in __aexit__
    async def __aexit__(self, *exc_details):


asyncio.exceptions.CancelledError: ('Cancelled by cancel scope ffff78169e90', <Task cancelled name='Task-523786' coro=<AsyncExitStack.__aexit__() done, defined at /usr/local/lib/python3.13/contextlib.py:716>>)

This is my call code:

    graph = workflow.compile()
    graph.step_timeout = 300


    async def stream_response():
        tools_result_map = {}  # tool_call_id -> tool_result
        result = graph.astream_events({"messages": langchain_messages}, config={"recursion_limit": 100, "configurable": {"workdir": workdir, "project_id": project_id, "uid": uid}}, stream_mode="messages", version="v2")

.....

Can anyone help me fix this?

[ehsanahmadkhan525]

Try this as mentioned by @hinthornw

The LangGraph Server supports specific environment variables for configuring a deployment.

Set this env variable

Set BG_JOB_ISOLATED_LOOPS to True to execute background runs in an isolated event loop separate from the serving API event loop.

This environment variable should be set to True if the implementation of a graph/node contains synchronous code. In this situation, the synchronous code will block the serving API event loop, which may cause the API to be unavailable. A symptom of an unavailable API is continuous application restarts due to failing health checks.

Defaults to False.

https://langchain-ai.github.io/langgraph/cloud/reference/env_var/?h=environ

[yesh-harmonic]

I am facing the same issue when I the execution reaches above 60 second. All the code is asynchronous using asyncio.to_thread and I have already set up BG_JOB_ISOLATED_LOOPS to be True on my Langgraph cloud deployment.

Anyone got any ideas on what I could try next?

[hinthornw]

Is the server being restarted because of a failed healthcheck? You'd be able to tell if the see startup logs like "Starting Postgres runtime with " , etc. in places you wouldn't want them to be

[apthagowda97]

I am facing the same issue when I the execution reaches above 60 second. All the code is asynchronous using asyncio.to_thread and I have already set up BG_JOB_ISOLATED_LOOPS to be True on my Langgraph cloud deployment.

Anyone got any ideas on what I could try next?

Is this only available in Langgraph Platform??

[yesh-harmonic]

@hinthornw No, but i do see this error pop up a lot in my logs

[ERROR] Exception in callback functools.partial(<bound method PollerCompletionQueue._handle_events of <grpc._cython.cygrpc.PollerCompletionQueue object at 0x7b2ccc23c040>>, <uvloop.Loop running=True closed=False debug=False>)
handle: <Handle functools.partial(<bound method PollerCompletionQueue._handle_events of <grpc._cython.cygrpc.PollerCompletionQueue object at 0x7b2ccc23c040>>, <uvloop.Loop running=True closed=False debug=False>)>
Traceback (most recent call last):
  File "uvloop/cbhandles.pyx", line 61, in uvloop.loop.Handle._run
  File "src/python/grpcio/grpc/_cython/_cygrpc/aio/completion_queue.pyx.pxi", line 147, in grpc._cython.cygrpc.PollerCompletionQueue._handle_events
BlockingIOError: [Errno 11] Resource temporarily unavailable

[ldnvnbl]

The same to me, have you fix this ?

[GaryFail]

I'm facing the same issue, after weeks of development work and finally getting it deployed into an AKS cluster I'm now seeing this error. I've never experienced this locally and neither has any of the team after vigorous testing - just as it's about to get released :-(

The graph runs all the way through and then at the end I hit this error about 90% of the time.

"('Cancelled by cancel scope 7fdc90710c80', <Task cancelled name='Task-74196' coro=<AsyncExitStack.aexit() done, defined at /usr/local/lib/python3.12/contextlib.py:707>>)"

A bit more information:

Exception in callback functools.partial(<bound method FuturesDict.on_done of {<Task finished name='create' coro=<arun_with_retry() done, defined at /app/venv/lib/python3.12/site-packages/langgraph/pregel/retry.py:105> result={'messages': ...challenging.'}>: PregelExecutableTask(name='create',

This is in a sub-graph which also has a sub-graph. It doesn't always happen but happens enough to not to be able to use it in production.

The more I look at this the more it points to a client timeout from the calling code, i.e. our BFF. One for next week I think, if I get to the bottom of it I'll update this.

[shuaiqidezhong]

My situation is the same as yours, may I ask if you have any good solutions now?

[LastMonopoly]

any update?

[GaryFail]

So my issue was resolved by setting the ingress on my BFF in kubernetes to have a connection-draining-timeout of 300 and a request-timeout of 300. I spoke to Microsoft Support about this and they were happy that this was the solution. This had to be done with the helm chart as you can't set it via the portal.

ingress:
  enabled: true
  annotations:
    kubernetes.io/ingress.class: azure/application-gateway
    appgw.ingress.kubernetes.io/connection-draining: "true"
    appgw.ingress.kubernetes.io/connection-draining-timeout: "300"
    appgw.ingress.kubernetes.io/request-timeout: "300"

[shuaiqidezhong]

所以我的问题解决了，方法是将 Kubernetes 中 BFF 的入口设置为 300 的连接超时和 300 的请求超时。我联系了微软支持团队，他们很高兴能解决这个问题。这必须通过 Helm Chart 来完成，因为无法通过门户进行设置。

ingress:
  enabled: true
  annotations:
    kubernetes.io/ingress.class: azure/application-gateway
    appgw.ingress.kubernetes.io/connection-draining: "true"
    appgw.ingress.kubernetes.io/connection-draining-timeout: "300"
    appgw.ingress.kubernetes.io/request-timeout: "300"

Before this, I encountered this problem 100%. I set the timeout in nginx and temporarily solved the issue, but it appeared again two days later, although it was an intermittent occurrence, which is really a headache.

[jeremySrgt]

We got the same error in our project, we are using fastAPI, not the langgraph platform
the error doesn't show when running our API locally, which is strange and hard to debug

[GaryFail]

Can I ask what your deployment looks like? We have a FastAPI deployed into an AKS cluster. It's called from a .NET API also running in the cluster.

If I port-forward onto the .NET API pod and call the end-point which calls the FastAPI endpoint it works fine, however if I make the call through our application which routes the request through Azure Application Gateway I get the error most of the time. This is leading me to think the issue is with the configuration of the Application Gateway.

[jeremySrgt]

Our FastAPI is deployed on Heroku and called mostly by our next js app. so it's a pretty simple setup

[GaryFail]

I'm convinced our problem is with Azure Application Gateway, after making a few changes it seems that the connection no longer closes. I wonder if there is something similar with Heroku.

In the code that streams the event_stream I put this error handler in:

except asyncio.CancelledError:
    logging.error("Client disconnected during request.")
    raise  # Re-raise the exception to let FastAPI cancel cleanly.

From what I can tell this is thrown if the client closes the connection. From my logs I could see this error being logged and then perhaps 10-20 seconds later the error was being thrown by LangGraph, obviously it was trying to do something after the connection was closed when it had finished whatever it was doing at the time, i.e. waiting for an LLM response etc. or something similar.

If you put this error handling in and see if you see this error logged prior to the exception thrown from LangGraph it may give you an idea of where / when the error is happening.

[gcalabria]

I'm facing the same problem as you guys. I am also working with a fastAPI backend + a react frontend. The error seems only to occur when I switch windows from my IDE (cursor) to my browser (arc). This is probably terminating the client connection before the stream ends...

[GaryFail]

Yeah it definitely sounds like the case. That was the cause of my troubles, albeit from how my ingress was configured in Azure.