Cancelled error with langgraph runs #1601

kedarsp-informa · 2024-09-04T09:43:53Z

kedarsp-informa
Sep 4, 2024

I have an inconsistent issue as CancelledError().
I have a langgraph and the error seems to be within langgraph library.
I am also using asyncio in my code as:

workflow.add_node(RAG, lambda state: asyncio.run(generate_with_rag(state=state, generation_rag_chain=generation_rag_chain)))
workflow.add_node(STORE_CHAT, lambda state: asyncio.run(store_chat(state=state, memory=memory)))

and

if chat_history:
        task1 = asyncio.create_task(router_chain.ainvoke({"question": question, "chat_history": state['chat_history']}))
        task2 = asyncio.create_task(rephrase_chain.ainvoke({"question": question, "chat_history": state['chat_history']}))
        source, follow_up_question = await asyncio.gather(task1, task2)
    else:
        source = await asyncio.shield(router_chain.ainvoke({"question": question, "chat_history": state['chat_history']}))
        follow_up_question = question

The below is the error trace:
_CancelledError()Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/langgraph/pregel/init.py", line 1189, in astream
done, inflight = await asyncio.wait(
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/asyncio/tasks.py", line 418, in wait
return await _wait(fs, timeout, return_when, loop)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/asyncio/tasks.py", line 525, in wait
await waiter
asyncio.exceptions.CancelledError
During handling of the above exception, another exception occurred:
Traeback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/langgraph/pregel/init.py", line 1132, in astream
async with AsyncPregelLoop(
asyncio.exceptions.CancelledError
If anyone has encountered this error before or any idea on this would be helpful. Thanks

Answered by nfcampos

Sep 21, 2024

Hi @kedarsp-informa you shouldn't use asyncio.run in add_node we have native support for async functions in nodes. Using asyncio.run would create a new event loop for each node, which is very inefficient and can cause issues such as cancellation not propagating from graph to node etc

View full answer

delorenzotommaso · 2024-09-12T20:59:49Z

delorenzotommaso
Sep 12, 2024

I get the same error when execution time reaches about 60s

6 replies

tommasodelorenzo Sep 21, 2024

Hi, nope. It is a graph called from a fastAPI endpoint deployed as an azure container app. Logging everything to LangSmith

vikyw89 Dec 22, 2024

same, i'm experiencing the same thing on latest langgraph, cancelled error when ainvoke last about 60s. And i'm not using asyncio run, I do use asyncer asyncify inside the node, not to invoke the langgraph

vikyw89 Dec 22, 2024

I think it's internal langgraph timeout

vikyw89 Dec 22, 2024

Cancelled by cancel scope 7fea082e1520

vikyw89 Dec 22, 2024

Tried with await sleep, couldnt replicate this one...

nfcampos · 2024-09-21T16:01:17Z

nfcampos
Sep 21, 2024
Maintainer

Hi @kedarsp-informa you shouldn't use asyncio.run in add_node we have native support for async functions in nodes. Using asyncio.run would create a new event loop for each node, which is very inefficient and can cause issues such as cancellation not propagating from graph to node etc

0 replies

NicholasMicali · 2024-12-17T22:29:46Z

NicholasMicali
Dec 17, 2024

I get the same error when attempting to run a subprocess to install a linter on startup. I am using langgraph cloud. Unsure about what "shouldn't use asyncio.run in add_node" means. this is a very simple graph, which also successfully runs my subprocess to install a linter, but fails afterwards. Deploys successfully without this line:

subprocess.run(["bash", "build_linter.sh"], check=True)

Registering graph with id 'agent'
12/17/2024, 1:46:06 PM
Starting metadata loop
12/17/2024, 1:46:07 PM
Starting cron scheduler
12/17/2024, 1:46:07 PM
Starting 10 background workers
12/17/2024, 1:46:07 PM
Application startup complete.
12/17/2024, 1:46:07 PM
Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)
12/17/2024, 1:46:07 PM
Shutting down background workers
12/17/2024, 1:46:07 PM
Cancelling remote graphs
12/17/2024, 1:46:07 PM
Traceback (most recent call last):
File "/api/langgraph_api/lifespan.py", line 37, in lifespan
File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 700, in lifespan
await receive()
File "/usr/local/lib/python3.11/site-packages/uvicorn/lifespan/on.py", line 137, in receive
return await self.receive_queue.get()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/asyncio/queues.py", line 158, in get
await getter
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 693, in lifespan
async with self.lifespan_context(app) as maybe_state:
File "/usr/local/lib/python3.11/contextlib.py", line 231, in aexit
await self.gen.athrow(typ, value, traceback)
File "/api/langgraph_api/lifespan.py", line 41, in lifespan
File "/usr/local/lib/python3.11/site-packages/langgraph_storage/database.py", line 175, in stop_pool
await _pg_pool.close()
File "/usr/local/lib/python3.11/site-packages/psycopg_pool/pool_async.py", line 474, in close
await agather(*workers, timeout=timeout)
File "/usr/local/lib/python3.11/site-packages/psycopg_pool/_acompat.py", line 142, in agather
await asyncio.wait_for(asyncio.shield(wait), timeout=timeout)
File "/usr/local/lib/python3.11/asyncio/tasks.py", line 489, in wait_for
return fut.result()
^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/psycopg_pool/pool_async.py", line 627, in worker
task = await q.get()
^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/asyncio/queues.py", line 158, in get
await getter
asyncio.exceptions.CancelledError

0 replies

n-sviridenko · 2025-01-23T16:56:57Z

n-sviridenko
Jan 23, 2025

@nfcampos I have this issue too - wrote to support and shared traces. In my case there are no asyncio.run calls - only asyncio.gather (but that one has to be for parallelisation). The rest is async/await.

0 replies

n-sviridenko · 2025-01-26T10:51:35Z

n-sviridenko
Jan 26, 2025

Could this issue be related to having no listeners on the stream (as the run it purely a background one)? It seems to occur only under those circumstances on LangGraph Cloud.

0 replies

codebreach · 2025-02-12T09:55:59Z

codebreach
Feb 12, 2025

We are also facing this and are not doing any asyncio.run in nodes. Its getting a timeout from redis which is caused by cancelled error

  File "/usr/local/lib/python3.11/site-packages/coredis/connection.py", line 584, in fetch_push_message
    await self._read_flag.wait()
  File "/usr/local/lib/python3.11/asyncio/locks.py", line 213, in wait
    await fut
DEFAULT 2025-02-12T08:32:30.866416Z asyncio.exceptions.CancelledError
DEFAULT 2025-02-12T08:32:30.866422Z The above exception was the direct cause of the following exception:
DEFAULT 2025-02-12T08:32:30.866440Z TimeoutError
DEFAULT 2025-02-12T08:32:30.866445Z During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/langgraph_storage/ops.py", line 2029, in listen_for_cancellation
    event = await pubsub.listen()
DEFAULT 2025-02-12T08:32:30.866457Z ^^^^^^^^^^^^^^^^^^^^^
DEFAULT 2025-02-12T08:32:30.866462Z File "/usr/local/lib/python3.11/site-packages/coredis/commands/pubsub.py", line 322, in listen
DEFAULT 2025-02-12T08:32:30.866466Z response = await self.parse_response(block=True)
DEFAULT 2025-02-12T08:32:30.866472Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
DEFAULT 2025-02-12T08:32:30.866476Z File "/usr/local/lib/python3.11/site-packages/coredis/commands/pubsub.py", line 242, in parse_response
DEFAULT 2025-02-12T08:32:30.866480Z return await asyncio.wait_for(
DEFAULT 2025-02-12T08:32:30.866485Z ^^^^^^^^^^^^^^^^^^^^^^^
DEFAULT 2025-02-12T08:32:30.866488Z File "/usr/local/lib/python3.11/asyncio/tasks.py", line 452, in wait_for
DEFAULT 2025-02-12T08:32:30.866492Z return await fut
DEFAULT 2025-02-12T08:32:30.866495Z ^^^^^^^^^
DEFAULT 2025-02-12T08:32:30.866500Z File "/usr/local/lib/python3.11/site-packages/coredis/commands/pubsub.py", line 216, in _execute
DEFAULT 2025-02-12T08:32:30.866504Z return await command(*args)
DEFAULT 2025-02-12T08:32:30.866509Z ^^^^^^^^^^^^^^^^^^^^
DEFAULT 2025-02-12T08:32:30.866514Z File "/usr/local/lib/python3.11/site-packages/coredis/connection.py", line 586, in fetch_push_message
DEFAULT 2025-02-12T08:32:30.866519Z raise TimeoutError
DEFAULT 2025-02-12T08:32:30.866526Z coredis.exceptions.TimeoutError

0 replies

edgarrmondragon · 2025-02-19T00:36:42Z

edgarrmondragon
Feb 19, 2025

I'm running a similar issue with the prebuilt ReAct agent:

CancelledError('Cancelled by cancel scope fffef75353a0', <Task cancelled name='Task-107483' coro=<AsyncExitStack.__aexit__() done, defined at /usr/local/lib/python3.12/contextlib.py:707>>)Traceback (most recent call last):

  File "/usr/local/lib/python3.12/site-packages/langgraph/pregel/__init__.py", line 2007, in astream
    async for _ in runner.atick(

  File "/usr/local/lib/python3.12/site-packages/langgraph/pregel/runner.py", line 495, in atick
    done, inflight = await asyncio.wait(
                     ^^^^^^^^^^^^^^^^^^^

  File "/usr/local/lib/python3.12/asyncio/tasks.py", line 464, in wait
    return await _wait(fs, timeout, return_when, loop)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/local/lib/python3.12/asyncio/tasks.py", line 550, in _wait
    await waiter

asyncio.exceptions.CancelledError: Cancelled by cancel scope fffef75353a0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "/usr/local/lib/python3.12/site-packages/langgraph/pregel/__init__.py", line 1960, in astream
    async with AsyncPregelLoop(
               ^^^^^^^^^^^^^^^^

  File "/usr/local/lib/python3.12/site-packages/langgraph/pregel/loop.py", line 1103, in __aexit__
    return await exit_task
           ^^^^^^^^^^^^^^^

  File "/usr/local/lib/python3.12/contextlib.py", line 707, in __aexit__
    async def __aexit__(self, *exc_details):

asyncio.exceptions.CancelledError: ('Cancelled by cancel scope fffef75353a0', <Task cancelled name='Task-107483' coro=<AsyncExitStack.__aexit__() done, defined at /usr/local/lib/python3.12/contextlib.py:707>>)

My environment:

langchain_core_version: "0.3.34"
langchain_version: "0.3.18"
library: "langsmith"
platform: "Linux-6.1.112-122.189.amzn2023.aarch64-aarch64-with-glibc2.36"
py_implementation: "CPython"
runtime: "python"
runtime_version: "3.12.9"
sdk: "langsmith-py"
sdk_version: "0.3.8"

Unfortunately I've not been to consistently reproduce it.

1 reply

ehsanahmadkhan525 Mar 9, 2025

I have the similar issue

CancelledError('Cancelled by cancel scope 3e9aadaf5ed0', <Task cancelled name='Task-393' coro=<AsyncExitStack.aexit() done, defined at /usr/local/lib/python3.11/contextlib.py:698>>)Traceback (most recent call last):

File "/usr/local/lib/python3.11/site-packages/langgraph/pregel/init.py", line 2080, in astream
async for _ in runner.atick(

File "/usr/local/lib/python3.11/site-packages/langgraph/pregel/runner.py", line 495, in atick
done, inflight = await asyncio.wait(
^^^^^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.11/asyncio/tasks.py", line 428, in wait
return await _wait(fs, timeout, return_when, loop)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.11/asyncio/tasks.py", line 535, in _wait
await waiter

asyncio.exceptions.CancelledError: Cancelled by cancel scope 3e9aadaf5ed0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File "/usr/local/lib/python3.11/site-packages/langgraph/pregel/init.py", line 2033, in astream
async with AsyncPregelLoop(

File "/usr/local/lib/python3.11/site-packages/langgraph/pregel/loop.py", line 1103, in aexit
return await exit_task
^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.11/contextlib.py", line 698, in aexit
async def aexit(self, *exc_details):

asyncio.exceptions.CancelledError: ('Cancelled by cancel scope 3e9aadaf5ed0', <Task cancelled name='Task-393' coro=<AsyncExitStack.aexit() done, defined at /usr/local/lib/python3.11/contextlib.py:698>>)

ehsanahmadkhan525 · 2025-03-09T18:54:59Z

ehsanahmadkhan525
Mar 9, 2025

I have the similar issue

CancelledError('Cancelled by cancel scope 3e9aadaf5ed0', <Task cancelled name='Task-393' coro=<AsyncExitStack.aexit() done, defined at /usr/local/lib/python3.11/contextlib.py:698>>)Traceback (most recent call last):

File "/usr/local/lib/python3.11/site-packages/langgraph/pregel/init.py", line 2080, in astream
async for _ in runner.atick(

File "/usr/local/lib/python3.11/site-packages/langgraph/pregel/runner.py", line 495, in atick
done, inflight = await asyncio.wait(
^^^^^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.11/asyncio/tasks.py", line 428, in wait
return await _wait(fs, timeout, return_when, loop)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.11/asyncio/tasks.py", line 535, in _wait
await waiter

asyncio.exceptions.CancelledError: Cancelled by cancel scope 3e9aadaf5ed0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File "/usr/local/lib/python3.11/site-packages/langgraph/pregel/init.py", line 2033, in astream
async with AsyncPregelLoop(

File "/usr/local/lib/python3.11/site-packages/langgraph/pregel/loop.py", line 1103, in aexit
return await exit_task
^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.11/contextlib.py", line 698, in aexit
async def aexit(self, *exc_details):

asyncio.exceptions.CancelledError: ('Cancelled by cancel scope 3e9aadaf5ed0', <Task cancelled name='Task-393' coro=<AsyncExitStack.aexit() done, defined at /usr/local/lib/python3.11/contextlib.py:698>>)

0 replies

ldorigo · 2025-03-11T15:47:32Z

ldorigo
Mar 11, 2025

We're facing a similar issue.

CancelledError: null
  File "/app/projects/chat/.venv/lib/python3.12/site-packages/langgraph/pregel/__init__.py", line 1413, in astream
    async for _ in runner.atick(
  File "/app/projects/chat/.venv/lib/python3.12/site-packages/langgraph/pregel/runner.py", line 126, in atick
    done, _ = await asyncio.wait(
  File "asyncio/tasks.py", line 464, in wait
    return await _wait(fs, timeout, return_when, loop)
  File "asyncio/tasks.py", line 550, in _wait
    await waiter

CancelledError: Cancelled by cancel scope 7fc5742666c0
  File "/app/projects/chat/.venv/lib/python3.12/site-packages/langgraph/pregel/__init__.py", line 1382, in astream
    async with AsyncPregelLoop(
  File "/app/projects/chat/.venv/lib/python3.12/site-packages/langgraph/pregel/loop.py", line 827, in __aexit__
    return await asyncio.shield(

Similarly to what's reported above; it seems to sometimes trigger when a node takes more than 1 minute to execute. Doesn't seem to do so reliably though. We don't use asyncio.run(). @nfcampos this discussion is marked as resolved but it is not - should we create a new one?

1 reply

apthagowda97 Mar 21, 2025

Yes even I am seeing this issue? Even I am not using asyncio. @nfcampos

hetavsundial · 2025-03-17T18:29:19Z

hetavsundial
Mar 17, 2025

I had been getting a similar with prebuilt react agent for a while. Adding a step timeout to the compile graph solved this for me.

graph = create_react_agent(...)
graph.step_timeout = 300 # in seconds

0 replies

ehsanahmadkhan525 · 2025-03-21T18:57:14Z

ehsanahmadkhan525
Mar 21, 2025

Here is how I increased the timeout for my langgraph code.

async def get_message_stream(self, message: Optional[str], base64_image: Optional[str], thread_id: str):
"""
Stream the response from the LLM using the graph with a global timeout
"""
import asyncio
config = {
"configurable": {
"thread_id": thread_id
}
# We'll use asyncio.wait_for instead of the config timeout
}

    # Prepare message content
    if base64_image:
        message_content = [
            {"type": "text", "text": message or ""},  # Use empty string if message is None
            {"type": "image_url", "image_url": {"url": base64_image}}
        ]
    else:
        message_content = message or ""  # Use empty string if message is None

    # This is the original streaming function
    async def generate_internal():
        seen_content = False
        async for chunk in self.graph.astream_events(
        {"messages": [
            {
                "role": "system",
                "content": self.SYSTEM_PROMPT + "Current time: " + datetime.utcnow().isoformat()
            },
            {
                "role": "user",
                "content": message_content
            }

        ]},
            config,
            stream_mode="messages",
            version="v1"
        ):

            if chunk["event"] == "on_chat_model_stream":
                data_chunk = chunk["data"]["chunk"]
                content = data_chunk.content
                
                # Output content chunks
                if content:
                    seen_content = True
                    yield f"data: {content}\n\n"
                # Only send usage metadata at the end (when content is empty and we've seen content before)
                elif seen_content:
                    usage_data = None
                    if hasattr(data_chunk, 'usage_metadata') and data_chunk.usage_metadata:
                        usage_data = data_chunk.usage_metadata
                    elif hasattr(data_chunk, 'response_metadata') and data_chunk.response_metadata and 'usage_metadata' in data_chunk.response_metadata:
                        usage_data = data_chunk.response_metadata['usage_metadata']
                    
                    if usage_data:
                        yield f"input_tokens: {usage_data['input_tokens']}\n\n"
                        yield f"output_tokens: {usage_data['output_tokens']}\n\n"


    # This is the wrapper function that adds the global timeout
    async def generate():
        # Set a global timeout of 15 minutes for the entire streaming operation
        # This acts as a safety net in case individual step timeouts don't trigger
        global_timeout = 900  # 15 minutes in seconds
        logging.info(f"Setting global timeout of {global_timeout} seconds for streaming operation")
        
        try:
            # Create the internal generator
            internal_generator = generate_internal()
            
            # Use a cleaner approach with asyncio.wait_for
            # We'll iterate through the generator and apply a timeout to each item
            while True:
                try:
                    # Wait for the next item with a timeout
                    item = await asyncio.wait_for(
                        internal_generator.__anext__(),
                        timeout=global_timeout
                    )
                    yield item
                except StopAsyncIteration:
                    # Generator is exhausted
                    break
                except asyncio.TimeoutError:
                    # Timeout occurred while waiting for the next item
                    error_message = "Operation timed out after waiting too long for a response"
                    logging.error(f"Timeout exceeded: {error_message}")
                    yield f"data: {error_message}\n\n"
                    break
        except Exception as e:
            # Handle other exceptions
            error_message = f"An error occurred: {str(e)}"
            logging.error(f"Error in streaming: {error_message}")
            yield f"data: {error_message}\n\n"
            
    return generate()

0 replies

hinthornw · 2025-04-01T21:10:53Z

hinthornw
Apr 1, 2025
Maintainer

You may also want to try setting the env var BG_JOB_ISOLATED_LOOPS=true.

Some cancellations may be caused because your instance is being restarted due to failed health checks. A common cause of failed health checks is that you have some synchronous processes blocking the main event loop and slowing the server down.

We have made a lot of improvements to address this but you may want to keep this in mind if you are seeing inexplicable cancellations in your code today. We are working to add even more isolation in the future.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cancelled error with langgraph runs #1601

{{title}}

Replies: 12 comments 8 replies

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

Select a reply

Cancelled error with langgraph runs #1601

Replies: 12 comments · 8 replies

nfcampos Sep 21, 2024 Maintainer

hinthornw Apr 1, 2025 Maintainer

Replies: 12 comments 8 replies

nfcampos
Sep 21, 2024
Maintainer

hinthornw
Apr 1, 2025
Maintainer