Skip to content

Kserve TorchserveModel can't handle torch's auth #3301

Open
@AntPeixe

Description

@AntPeixe

🐛 Describe the bug

Kserve's TorchserveModel load doesn't take into consideration the fact that torchserve has auth enabled by default.

Error logs

INFO:kserve:Copying contents of /mnt/models/model-store to local
INFO:root:Loading model .. 1 of 10 tries..
INFO:root:The model layoutlm-pytorch-model is not ready
INFO:root:Sleep 30 seconds for load layoutlm-pytorch-model..
...
...
...
INFO:root:Loading model .. 2 of 10 tries..
2024-08-29T10:16:01,232 [INFO ] epollEventLoopGroup-3-1 ACCESS_LOG - /127.0.0.1:50126 "GET /models/model?customized=false HTTP/1.1" 400 4

Installation instructions

Using docker image pytorch/torchserve-kfs:0.11.1 to serve a model

Model Packaging

Model was packaged using torch-model-archiver and the same mar file would work on the older pytorch/torchserve-kfs:0.8.0

config.properties

No response

Versions

Copied from within docker image pytorch/torchserve-kfs:0.11.1

------------------------------------------------------------------------------------------
Environment headers
------------------------------------------------------------------------------------------
Torchserve branch:

torchserve==0.11.1
torch-model-archiver==0.11.1

Python version: 3.9 (64-bit runtime)
Python executable: /home/venv/bin/python

Versions of relevant python libraries:
captum==0.6.0
numpy==1.24.3
pillow==10.3.0
psutil==5.9.8
requests==2.32.0
requests-oauthlib==2.0.0
torch==2.3.0+cpu
torch-model-archiver==0.11.1
torch-workflow-archiver==0.2.14
torchaudio==2.3.0+cpu
torchserve==0.11.1
torchtext==0.18.0+cpu
torchvision==0.18.0+cpu
transformers==4.42.4
wheel==0.42.0
torch==2.3.0+cpu
torchtext==0.18.0+cpu
torchvision==0.18.0+cpu
torchaudio==2.3.0+cpu

Java Version:


OS: Ubuntu 20.04.6 LTS
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0
Clang version: N/A
CMake version: N/A

Environment:
library_path (LD_/DYLD_):

Repro instructions

  • create model mar file
  • create config.properties file and DO NOT disable auth
  • run the image pytorch/torchserve-kfs:0.11.1 - should fail to load as the management request doesn't include auth token

Possible Solution

Read key_file.json and use the token in the request

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions