Wrong API Key Spawns Phantom Inference Pipeline Process

[yeldarby]

Search before asking

• I have searched the Inference issues and found no similar bug report.

Bug

If you pass the wrong API key to /inference_pipelines/initialise it will create a zombie process that spits out all sorts of errors indefinitely.

Logs:

[12/26/24 14:52:42] ERROR    Could not handle Command. request_id=02820ce1-9dcb-4dd3-b233-f75c3a74f4a1, error=Could not find requested Roboflow resource. Check that the provided  inference_pipeline_manager.py:517
                             dataset and version are correct, and check that the provided Roboflow API key has the correct permissions., error_type=ErrorType.NOT_FOUND,                                            
                             public_error_message=Requested Roboflow resources (models / workflows etc.) not available or wrong API key used.                                                                       
                             Traceback (most recent call last):                                                                                                                                                     
                               File "/Volumes/Code/inference/inference/core/roboflow_api.py", line 87, in wrapper                                                                                                   
                                 return function(*args, **kwargs)                                                                                                                                                   
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                   
                               File "/Volumes/Code/inference/inference/core/roboflow_api.py", line 499, in get_workflow_specification                                                                               
                                 response = _get_from_url(url=api_url)                                                                                                                                              
                                            ^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                              
                               File "/Volumes/Code/inference/inference/core/roboflow_api.py", line 599, in _get_from_url                                                                                            
                                 api_key_safe_raise_for_status(response=response)                                                                                                                                   
                               File "/Volumes/Code/inference/inference/core/utils/requests.py", line 15, in api_key_safe_raise_for_status                                                                           
                                 response.raise_for_status()                                                                                                                                                        
                               File "/Users/yeldarb/venvs/inference-devel/lib/python3.11/site-packages/requests/models.py", line 1024, in raise_for_status                                                          
                                 raise HTTPError(http_error_msg, response=self)                                                                                                                                     
                             requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://api.roboflow.com/roboflow-docs/workflows/clip-frames?api_key=*******                                       
                                                                                                                                                                                                                    
                             The above exception was the direct cause of the following exception:                                                                                                                   
                                                                                                                                                                                                                    
                             Traceback (most recent call last):                                                                                                                                                     
                               File "/Volumes/Code/inference/inference/core/interfaces/stream_manager/manager_app/inference_pipeline_manager.py", line 175, in                                                      
                             _initialise_pipeline                                                                                                                                                                   
                                 self._inference_pipeline = InferencePipeline.init_with_workflow(                                                                                                                   
                                                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                   
                               File "/Volumes/Code/inference/inference/core/utils/function.py", line 35, in wrapper                                                                                                 
                                 return func(*args, **kwargs)                                                                                                                                                       
                                        ^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                       
                               File "/Volumes/Code/inference/inference/core/interfaces/stream/inference_pipeline.py", line 596, in init_with_workflow                                                               
                                 workflow_specification = get_workflow_specification(                                                                                                                               
                                                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                               
                               File "/Volumes/Code/inference/inference/core/roboflow_api.py", line 100, in wrapper                                                                                                  
                                 error_handler(error)                                                                                                                                                               
                               File "/Volumes/Code/inference/inference/core/roboflow_api.py", line 73, in <lambda>                                                                                                  
                                 404: lambda e: raise_from_lambda(                                                                                                                                                  
                                                ^^^^^^^^^^^^^^^^^^                                                                                                                                                  
                               File "/Volumes/Code/inference/inference/core/roboflow_api.py", line 63, in raise_from_lambda                                                                                         
                                 raise exception_type(message) from inner_error                                                                                                                                     
                             inference.core.exceptions.RoboflowAPINotNotFoundError: Could not find requested Roboflow resource. Check that the provided dataset and version are                                     
                             correct, and check that the provided Roboflow API key has the correct permissions.                                                                                                     
                    ERROR    Could not handle Command. request_id=d0ab647dc91a42ef92faa560c88a5051, error=None, error_type=ErrorType.OPERATION_ERROR, public_error_message=Cannot  inference_pipeline_manager.py:517
                             retrieve InferencePipeline status. Try again later - Inference Pipeline not initialised.                                                                                               
                             NoneType: None                                                                                                                                                                         
[12/26/24 14:52:42] ERROR    Error with command handling raised by InferencePipeline Manager. error_type=not_found error_class=RoboflowAPINotNotFoundError error_message=Could not find stream_manager_client.py:342
                             requested Roboflow resource. Check that the provided dataset and version are correct, and check that the provided Roboflow API key has the correct                                     
                             permissions.                                                                                                                                                                           
Traceback (most recent call last):
  File "/Volumes/Code/inference/inference/core/interfaces/http/http_api.py", line 265, in wrapped_route
    return await route(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Volumes/Code/inference/inference/core/interfaces/http/http_api.py", line 1387, in initialise
    return await self.stream_manager_client.initialise_pipeline(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Volumes/Code/inference/inference/core/interfaces/stream_manager/api/stream_manager_client.py", line 132, in initialise_pipeline
    response = await self._handle_command(command=command)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Volumes/Code/inference/inference/core/interfaces/stream_manager/api/stream_manager_client.py", line 213, in _handle_command
    dispatch_error(error_response=response)
  File "/Volumes/Code/inference/inference/core/interfaces/stream_manager/api/stream_manager_client.py", line 348, in dispatch_error
    raise ERRORS_MAPPING[error_type](
inference.core.interfaces.stream_manager.api.errors.ProcessesManagerNotFoundError: Error with command handling raised by InferencePipeline Manager. Error type: not_found. Details: Could not find requested Roboflow resource. Check that the provided dataset and version are correct, and check that the provided Roboflow API key has the correct permissions.
INFO:     127.0.0.1:59145 - "POST /inference_pipelines/initialise HTTP/1.1" 404 Not Found
[12/26/24 14:52:43] ERROR    Could not handle Command. request_id=7ffd4a3afd21479098ee9a5fc5f5e882, error=None, error_type=ErrorType.OPERATION_ERROR, public_error_message=Cannot  inference_pipeline_manager.py:517
                             retrieve InferencePipeline status. Try again later - Inference Pipeline not initialised.                                                                                               
                             NoneType: None                                                                                                                                                                         
[12/26/24 14:52:44] ERROR    Could not handle Command. request_id=0f268db1ab6845d99055293492e6fb00, error=None, error_type=ErrorType.OPERATION_ERROR, public_error_message=Cannot  inference_pipeline_manager.py:517
                             retrieve InferencePipeline status. Try again later - Inference Pipeline not initialised.                                                                                               
                             NoneType: None                                                                                                                                                                         
[12/26/24 14:52:45] ERROR    Could not handle Command. request_id=f4788cc9e67a431292d3a056ae1d30ca, error=None, error_type=ErrorType.OPERATION_ERROR, public_error_message=Cannot  inference_pipeline_manager.py:517
                             retrieve InferencePipeline status. Try again later - Inference Pipeline not initialised.                                                                                               
                             NoneType: None                                                                                                                                                                         
[12/26/24 14:52:46] ERROR    Could not handle Command. request_id=6bb10427411245e5bd6f3c85bbdba14b, error=None, error_type=ErrorType.OPERATION_ERROR, public_error_message=Cannot  inference_pipeline_manager.py:517
                             retrieve InferencePipeline status. Try again later - Inference Pipeline not initialised.                                                                                               
                             NoneType: None                                                                                                                                                                         
[12/26/24 14:52:47] ERROR    Could not handle Command. request_id=3fa4944af1214028b175977df5a8e871, error=None, error_type=ErrorType.OPERATION_ERROR, public_error_message=Cannot  inference_pipeline_manager.py:517
                             retrieve InferencePipeline status. Try again later - Inference Pipeline not initialised.                                                                                               
                             NoneType: None  

Environment

• Inference installed from main branch
• MacOS

Minimal Reproducible Example

Using a public Workflow that doesn't require an API Key but passing an invalid one:

from inference_sdk import InferenceHTTPClient

client = InferenceHTTPClient(
    api_url="http://localhost:9001", # use local inference server
    api_key="invalid" # optional to access your private data and models
)

result = client.start_inference_pipeline_with_workflow(
    video_reference=[0],
    workspace_name="roboflow-docs",
    workflow_id="clip-frames",
    max_fps=max_fps,
    workflows_parameters={
        "prompt": "blurry",
        "threshold": 0.16
    }
)

Additional

No response

Are you willing to submit a PR?

• Yes I'd like to help by submitting a PR!

[farhan0167]

Hi, is anyone working on this? I'd love to make a contribution if this is still unassigned

[grzegorz-roboflow]

Hi @farhan0167 , I have not started working on this issue yet so please feel free to give it a try! Thanks!

[farhan0167]

hi @yeldarby , so I tried running the minimum reproducible as is:

from inference_sdk import InferenceHTTPClient

client = InferenceHTTPClient(
    api_url="http://localhost:9001", # use local inference server
    api_key="invalid" # optional to access your private data and models
)

result = client.start_inference_pipeline_with_workflow(
    video_reference=[0],
    workspace_name="roboflow-docs",
    workflow_id="clip-frames",
    max_fps=max_fps,
    workflows_parameters={
        "prompt": "blurry",
        "threshold": 0.16
    }
)

and I'm getting the following error which I believe is what you are expecting:

InferenceSDKDeprecationWarning: decorate is experimental: Video processing in inference server is under development. Breaking changes are possible.
Traceback (most recent call last):
  File "/Users/farhanishraq/Downloads/No_Code_AI/inference/inference_sdk/http/client.py", line 94, in decorate
    return function(*args, **kwargs)
  File "/Users/farhanishraq/Downloads/No_Code_AI/inference/inference_sdk/http/client.py", line 1850, in start_inference_pipeline_with_workflow
    response.raise_for_status()
  File "/Users/farhanishraq/Downloads/No_Code_AI/inference/venv/lib/python3.10/site-packages/requests/models.py", line 1024, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: http://localhost:9001/inference_pipelines/initialise

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/farhanishraq/Downloads/No_Code_AI/inference/a.py", line 24, in <module>
    result = client.start_inference_pipeline_with_workflow(
  File "/Users/farhanishraq/Downloads/No_Code_AI/inference/inference_sdk/utils/decorators.py", line 56, in wrapper
    return func(*args, **kwargs)
  File "/Users/farhanishraq/Downloads/No_Code_AI/inference/inference_sdk/http/client.py", line 104, in decorate
    raise HTTPCallErrorError(
inference_sdk.http.errors.HTTPCallErrorError: HTTPCallErrorError(description='401 Client Error: Unauthorized for url: http://localhost:9001/inference_pipelines/initialise', api_message='Unauthorized access to roboflow API - check API key and make sure the key is valid for workspace you use. Visit https://docs.roboflow.com/api-reference/authentication#retrieve-an-api-key to learn how to retrieve one.',status_code=401)

Environment

• Same as the one's you've listed
• Python 3.10

Is there anything that I'm missing that I could try out? Is there another public workflow that I could try out that gives you the error?

[grzegorz-roboflow]

Hi @farhan0167 , do you get more output?

@yeldarby suggests we should see infinite chain of lines like the one below:

[12/26/24 14:52:47] ERROR    Could not handle Command. request_id=3fa4944af1214028b175977df5a8e871, error=None, error_type=ErrorType.OPERATION_ERROR, public_error_message=Cannot  inference_pipeline_manager.py:517
                             retrieve InferencePipeline status. Try again later - Inference Pipeline not initialised.                                                                                               
                             NoneType: None  

[farhan0167]

@grzegorz-roboflow yeah I actually tried some different examples, but I wasn't able to reproduce the error with an infinite chain of logs. If i don't provide an api key for the following for example:

from inference_sdk import InferenceHTTPClient

client = InferenceHTTPClient(
    api_url="http://localhost:9001", # use local inference server
    #api_key="<YOUR API KEY>" # optional to access your private data and models
)

result = client.run_workflow(
    workspace_name="roboflow-docs",
    workflow_id="model-comparison",
    images={
        "image": "https://media.roboflow.com/workflows/examples/bleachers.jpg"
    },
    parameters={
        "model1": "yolov8n-640",
        "model2": "yolov11n-640"
    }
)

Then i get a normal response back but when i uncomment the line, and re-run it, I get only:

Traceback (most recent call last):
  File "/Users/farhanishraq/Downloads/No_Code_AI/inference/venv/lib/python3.11/site-packages/inference_sdk/http/client.py", line 94, in decorate
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/farhanishraq/Downloads/No_Code_AI/inference/venv/lib/python3.11/site-packages/inference_sdk/http/client.py", line 1558, in run_workflow
    return self._run_workflow(
           ^^^^^^^^^^^^^^^^^^^
  File "/Users/farhanishraq/Downloads/No_Code_AI/inference/venv/lib/python3.11/site-packages/inference_sdk/http/client.py", line 1630, in _run_workflow
    api_key_safe_raise_for_status(response=response)
  File "/Users/farhanishraq/Downloads/No_Code_AI/inference/venv/lib/python3.11/site-packages/inference_sdk/http/utils/requests.py", line 21, in api_key_safe_raise_for_status
    response.raise_for_status()
  File "/Users/farhanishraq/Downloads/No_Code_AI/inference/venv/lib/python3.11/site-packages/requests/models.py", line 1024, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: http://localhost:9001/roboflow-docs/workflows/model-comparison

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/farhanishraq/Downloads/No_Code_AI/test/a.py", line 42, in <module>
    result = client.run_workflow(
             ^^^^^^^^^^^^^^^^^^^^
  File "/Users/farhanishraq/Downloads/No_Code_AI/inference/venv/lib/python3.11/site-packages/inference_sdk/http/client.py", line 104, in decorate
    raise HTTPCallErrorError(
inference_sdk.http.errors.HTTPCallErrorError: HTTPCallErrorError(description='401 Client Error: Unauthorized for url: http://localhost:9001/roboflow-docs/workflows/model-comparison', api_message='Unauthorized access to roboflow API - check API key and make sure the key is valid for workspace you use. Visit https://docs.roboflow.com/api-reference/authentication#retrieve-an-api-key to learn how to retrieve one.',status_code=401)

[grzegorz-roboflow]

Hi @farhan0167 , I fixed the infinite loop in #983 (please have a look if you'd like to understand how I reproduced the error), I think we still have good area for improvement here when it comes to returned error, would you like to look into this?

[farhan0167]

I'd be interested in taking a look at this. And I see why I wasnt seeing the infinite loop. I was actually not looking at the logs for the server. Before I get started, could you explain how you got your webcam running? Having a hard time running my webcam using the example here

from inference_sdk import InferenceHTTPClient
import atexit
import time

max_fps = 4

client = InferenceHTTPClient(
    api_url="http://localhost:9001", # use local inference server
    #api_key="" # optional to access your private data and models
)

# Start a stream on an rtsp stream
result = client.start_inference_pipeline_with_workflow(
    video_reference=0,
    workspace_name="test-coqbp",
    workflow_id="detect-and-classify",
    max_fps=max_fps,
    workflows_parameters={
        "prompt": "blurry", # change to look for something else
        "threshold": 0.16
    }
)

I get something like this:

    self._start()
  File "/app/inference/core/interfaces/camera/video_source.py", line 593, in _start
    raise SourceConnectionError(
inference.core.interfaces.camera.exceptions.SourceConnectionError: Cannot connect to video source under reference: 0
INFO:     172.17.0.1:63514 - "GET /inference_pipelines/933aee1f-8244-45ed-a23c-06e7c67f4e80/consume HTTP/1.1" 200 OK
INFO:     172.17.0.1:63516 - "GET /inference_pipelines/933aee1f-8244-45ed-a23c-06e7c67f4e80/consume HTTP/1.1" 200 OK
INFO:     172.17.0.1:63532 - "GET /inference_pipelines/933aee1f-8244-45ed-a23c-06e7c67f4e80/consume HTTP/1.1" 200 OK
INFO:     172.17.0.1:63542 - "GET /inference_pipelines/933aee1f-8244-45ed-a23c-06e7c67f4e80/consume HTTP/1.1" 200 OK
INFO:     172.17.0.1:63550 - "GET /inference_pipelines/933aee1f-8244-45ed-a23c-06e7c67f4e80/consume HTTP/1.1" 200 OK
INFO:     172.17.0.1:63564 - "GET /inference_pipelines/933aee1f-8244-45ed-a23c-06e7c67f4e80/consume HTTP/1.1" 200 OK
INFO:     172.17.0.1:63568 - "GET /inference_pipelines/933aee1f-8244-45ed-a23c-06e7c67f4e80/consume HTTP/1.1" 200 OK
INFO:     172.17.0.1:63576 - "GET /inference_pipelines/933aee1f-8244-45ed-a23c-06e7c67f4e80/consume HTTP/1.1" 200 OK
INFO:     172.17.0.1:63590 - "GET /inference_pipelines/933aee1f-8244-45ed-a23c-06e7c67f4e80/consume HTTP/1.1" 200 OK
INFO:     172.17.0.1:63606 - "GET /inference_pipelines/933aee1f-8244-45ed-a23c-06e7c67f4e80/consume HTTP/1.1" 200 OK
INFO:     172.17.0.1:63614 - "GET /inference_pipelines/933aee1f-8244-45ed-a23c-06e7c67f4e80/consume HTTP/1.1" 200 OK
INFO:     172.17.0.1:63616 - "GET /inference_pipelines/933aee1f-8244-45ed-a23c-06e7c67f4e80/consume HTTP/1.1" 200 OK

before it stops. Not sure if there's an issue that already addresses this or if this should be its own issue. But as soon as I understand how everything's working, I can look into the error thing.

[grzegorz-roboflow]

Hi @farhan0167, I start inference pipeline container:

docker build \
    -t roboflow/roboflow-inference-server-cpu:test \
    -f docker/dockerfiles/Dockerfile.onnx.cpu \
        .

docker run \
    --rm \
    -p 9001:9001 \
    -p 9002:9002 \
    -e NOTEBOOK_ENABLED=true \
    -e ENABLE_STREAM_API=true \
    -e LOG_LEVEL=DEBUG \
    -e LMM_ENABLED=True \
    --name=hosted_inference \
    -v ./inference:/app/inference \
    -v ./docker/config/cpu_http.py:/app/cpu_http.py \
    roboflow/roboflow-inference-server-cpu:test

Then, in my browser, I visit app.roboflow.com and once authenticated I navigate to workflows and in there I select option to connect to my localhost.

Hope this helps!

[grzegorz-roboflow]

The original issue of pipeline not terminated was resolved, I'm reopening this since we can add improvement related to this issue - when pipeline fails we can send more descriptive error message back to the UI

[grzegorz-roboflow]

Closing since the issue is resolved