[BUG CLIENT]: GCP client - can't send images

[romantsovmike]

Python -VV

Python 3.10.16 | packaged by conda-forge | (main, Dec  5 2024, 14:16:10) [GCC 13.3.0]

Pip Freeze

agate==1.9.1
aiofiles==23.2.1
aiohappyeyeballs==2.4.6
aiohttp==3.11.13
aiohttp-retry==2.8.3
aiosignal==1.3.2
aiosqlite==0.21.0
alembic==1.14.1
annotated-types==0.7.0
anyio==4.8.0
apache-airflow==2.9.3
apache-airflow-providers-common-compat==1.5.0
apache-airflow-providers-common-io==1.5.0
apache-airflow-providers-common-sql==1.23.0
apache-airflow-providers-fab==1.5.3
apache-airflow-providers-ftp==3.12.2
apache-airflow-providers-google==14.0.0
apache-airflow-providers-http==5.2.0
apache-airflow-providers-imap==3.8.2
apache-airflow-providers-smtp==2.0.0
apache-airflow-providers-sqlite==4.0.0
apispec==6.8.1
argcomplete==3.5.3
asgiref==3.8.1
asn1crypto==1.5.1
asttokens==3.0.0
async-timeout==4.0.3
attrs==25.1.0
babel==2.17.0
backoff==2.2.1
backports.tarfile==1.2.0
beautifulsoup4==4.13.3
blinker==1.9.0
build==1.2.2.post1
CacheControl==0.14.2
cachelib==0.13.0
cachetools==5.5.2
cattrs==24.1.2
certifi==2025.1.31
cffi==1.17.1
chardet==5.2.0
charset-normalizer==3.4.1
cleo==2.1.0
click==8.1.8
clickclick==20.10.2
cloud-sql-python-connector==1.17.0
cloudpickle==3.1.1
colorama==0.4.6
colorlog==4.8.0
comm==0.2.2
ConfigUpdater==3.2
connexion==2.14.2
contourpy==1.3.1
crashtest==0.4.1
cron-descriptor==1.4.5
croniter==6.0.0
cryptography==44.0.2
cycler==0.12.1
daff==1.3.46
db-dtypes==1.4.1
dbt-adapters==1.7.0
dbt-common==1.10.0
dbt-core==1.8.8
dbt-extractor==0.5.1
dbt-postgres==1.9.0
dbt-semantic-interfaces==0.5.1
debugpy==1.8.12
decorator==5.2.1
deepdiff==7.0.1
Deprecated==1.2.18
dill==0.3.9
distlib==0.3.9
dnspython==2.7.0
docstring_parser==0.16
dulwich==0.22.8
elastic-transport==8.17.0
elasticsearch==8.17.1
email_validator==2.2.0
eval_type_backport==0.2.2
exceptiongroup==1.2.2
executing==2.2.0
fastjsonschema==2.21.1
filelock==3.17.0
findpython==0.6.2
Flask==2.2.5
Flask-AppBuilder==4.5.3
Flask-Babel==2.0.0
Flask-Caching==2.3.1
Flask-JWT-Extended==4.7.1
Flask-Limiter==3.10.1
Flask-Login==0.6.3
Flask-Session==0.5.0
Flask-SQLAlchemy==2.5.1
Flask-WTF==1.2.2
fonttools==4.56.0
frozendict==2.4.6
frozenlist==1.5.0
fsspec==2025.2.0
gcloud-aio-auth==5.3.2
gcloud-aio-bigquery==7.1.0
gcloud-aio-storage==9.3.0
gcsfs==2025.2.0
google-ads==25.2.0
google-analytics-admin==0.23.4
google-api-core==2.24.1
google-api-python-client==2.162.0
google-auth==2.38.0
google-auth-httplib2==0.2.0
google-auth-oauthlib==1.2.1
google-cloud-aiplatform==1.82.0
google-cloud-alloydb==0.4.3
google-cloud-appengine-logging==1.6.0
google-cloud-artifact-registry==1.15.1
google-cloud-audit-log==0.3.0
google-cloud-automl==2.16.1
google-cloud-batch==0.17.34
google-cloud-bigquery==3.30.0
google-cloud-bigquery-connection==1.18.1
google-cloud-bigquery-datatransfer==3.19.0
google-cloud-bigquery-storage==2.28.0
google-cloud-bigtable==2.29.0
google-cloud-build==3.31.0
google-cloud-compute==1.26.0
google-cloud-container==2.56.0
google-cloud-core==2.4.2
google-cloud-datacatalog==3.25.1
google-cloud-dataflow-client==0.8.16
google-cloud-dataform==0.5.16
google-cloud-dataplex==2.7.1
google-cloud-dataproc==5.18.0
google-cloud-dataproc-metastore==1.18.1
google-cloud-dlp==3.28.0
google-cloud-iam==2.18.1
google-cloud-kms==3.4.0
google-cloud-language==2.17.0
google-cloud-logging==3.11.4
google-cloud-managedkafka==0.1.7
google-cloud-memcache==1.12.0
google-cloud-monitoring==2.27.0
google-cloud-orchestration-airflow==1.17.0
google-cloud-os-login==2.17.0
google-cloud-pubsub==2.28.0
google-cloud-redis==2.18.0
google-cloud-resource-manager==1.14.1
google-cloud-run==0.10.16
google-cloud-secret-manager==2.23.1
google-cloud-spanner==3.52.0
google-cloud-speech==2.31.0
google-cloud-sqlcommenter==2.0.0
google-cloud-storage==2.19.0
google-cloud-storage-transfer==1.16.0
google-cloud-tasks==2.19.1
google-cloud-texttospeech==2.25.0
google-cloud-translate==3.20.1
google-cloud-videointelligence==2.16.0
google-cloud-vision==3.10.0
google-cloud-workflows==1.17.0
google-crc32c==1.6.0
google-re2==1.1.20240702
google-resumable-media==2.7.2
googleapis-common-protos==1.68.0
greenlet==3.1.1
grpc-google-iam-v1==0.14.0
grpc-interceptor==0.15.4
grpcio==1.70.0
grpcio-gcp==0.2.2
grpcio-status==1.62.3
gunicorn==23.0.0
h11==0.14.0
httpcore==1.0.7
httplib2==0.22.0
httpx==0.28.1
idna==3.10
imageio==2.37.0
immutabledict==4.2.1
importlib-metadata==6.11.0
importlib_resources==6.5.2
inflection==0.5.1
installer==0.7.0
ipykernel==6.29.5
ipython==8.33.0
ipywidgets==8.1.5
isodate==0.6.1
itsdangerous==2.2.0
jaraco.classes==3.4.0
jaraco.context==6.0.1
jaraco.functools==4.1.0
jedi==0.19.2
jeepney==0.9.0
Jinja2==3.1.5
jmespath==1.0.1
joblib==1.4.2
json-merge-patch==0.2
jsondiff==2.2.1
jsonpath-python==1.0.6
jsonschema==4.23.0
jsonschema-specifications==2024.10.1
jupyter_client==8.6.3
jupyter_core==5.7.2
jupyterlab_widgets==3.0.13
kaleido==0.2.1
keyring==25.6.0
keyrings.google-artifactregistry-auth==1.1.2
kfp==2.12.1
kfp-pipeline-spec==0.6.0
kfp-server-api==2.4.0
kiwisolver==1.4.8
kubernetes==30.1.0
lazy-object-proxy==1.10.0
lazy_loader==0.4
leather==0.4.0
limits==4.0.1
linkify-it-py==2.0.3
lockfile==0.12.2
Logbook==1.5.3
looker-sdk==25.2.0
Mako==1.3.9
markdown-it-py==3.0.0
MarkupSafe==3.0.2
marshmallow==3.26.1
marshmallow-oneofschema==3.1.1
marshmallow-sqlalchemy==0.28.2
mashumaro==3.15
matplotlib==3.10.1
matplotlib-inline==0.1.7
mdit-py-plugins==0.4.2
mdurl==0.1.2
methodtools==0.4.7
minimal-snowplow-tracker==0.0.2
mistralai==1.5.2
more-itertools==10.6.0
msgpack==1.1.0
multidict==6.1.0
mypy-extensions==1.0.0
narwhals==1.29.0
nest-asyncio==1.6.0
networkx==3.4.2
nltk==3.9.1
numpy==1.26.4
oauthlib==3.2.2
opencv-python==4.11.0.86
opentelemetry-api==1.27.0
opentelemetry-exporter-otlp==1.27.0
opentelemetry-exporter-otlp-proto-common==1.27.0
opentelemetry-exporter-otlp-proto-grpc==1.27.0
opentelemetry-exporter-otlp-proto-http==1.27.0
opentelemetry-proto==1.27.0
opentelemetry-sdk==1.27.0
opentelemetry-semantic-conventions==0.48b0
ordered-set==4.1.0
packaging==24.2
pandas==2.1.4
pandas-gbq==0.28.0
parsedatetime==2.6
parso==0.8.4
pathspec==0.12.1
pbs-installer==2025.2.12
pendulum==2.1.2
pexpect==4.9.0
pg8000==1.30.5
pillow==10.4.0
pkginfo==1.12.1.2
platformdirs==4.3.6
plotly==6.0.0
pluggy==1.5.0
poetry==2.1.1
poetry-core==2.1.1
prison==0.2.1
prometheus-client==0.15.0
prompt_toolkit==3.0.50
propcache==0.3.0
proto-plus==1.26.0
protobuf==4.25.6
psutil==7.0.0
psycopg2-binary==2.9.10
ptyprocess==0.7.0
pure_eval==0.2.3
py_proto==1.1.19
pyarrow==19.0.1
pyasn1==0.6.1
pyasn1_modules==0.4.0
pycparser==2.22
pydantic==2.10.6
pydantic_core==2.27.2
pydata-google-auth==1.9.1
Pygments==2.19.1
PyJWT==2.10.1
pyOpenSSL==25.0.0
pyparsing==3.2.1
pyproject_hooks==1.2.0
PySocks==1.7.1
python-daemon==3.1.2
python-dateutil==2.9.0.post0
python-dotenv==1.0.1
python-nvd3==0.16.0
python-slugify==8.0.4
pytimeparse==1.1.8
pytz==2025.1
pytzdata==2020.1
PyYAML==6.0.2
pyzmq==26.2.1
RapidFuzz==3.12.2
referencing==0.36.2
regex==2024.11.6
requests==2.32.3
requests-futures==1.0.2
requests-oauthlib==2.0.0
requests-toolbelt==1.0.0
rfc3339-validator==0.1.4
rich==13.9.4
rich-argparse==1.7.0
rpds-py==0.23.1
rsa==4.9
scikit-image==0.25.2
scikit-learn==1.2.2
scipy==1.15.2
scramp==1.4.5
seaborn==0.13.2
SecretStorage==3.3.3
sentry-sdk==2.22.0
setproctitle==1.3.5
shapely==2.0.7
shellingham==1.5.4
six==1.17.0
sniffio==1.3.1
soupsieve==2.6
SQLAlchemy==1.4.54
sqlalchemy-bigquery==1.12.1
SQLAlchemy-JSONField==1.0.2
sqlalchemy-spanner==1.9.0
SQLAlchemy-Utils==0.41.2
sqlglot==26.7.0
sqlparse==0.5.3
stack-data==0.6.3
starlette==0.46.0
tabulate==0.9.0
tenacity==9.0.0
termcolor==2.5.0
text-unidecode==1.3
threadpoolctl==3.5.0
tifffile==2025.2.18
tomli==2.2.1
tomlkit==0.13.2
tornado==6.4.2
tqdm==4.67.1
traitlets==5.14.3
trove-classifiers==2025.2.18.16
typer==0.15.2
typing-inspect==0.9.0
typing_extensions==4.12.2
tzdata==2025.1
uc-micro-py==1.0.3
unicodecsv==0.14.1
universal_pathlib==0.2.6
uritemplate==4.1.1
urllib3==1.26.20
virtualenv==20.29.2
wcwidth==0.2.13
websocket-client==1.8.0
Werkzeug==2.2.3
widgetsnbextension==4.0.13
wirerope==1.0.0
wrapt==1.17.2
WTForms==3.2.1
yarl==1.18.3
zipp==3.21.0
zstandard==0.23.0

Reproduction Steps

1. Create GCP client:

from mistralai_gcp import MistralGoogleCloud

client = MistralGoogleCloud()

2. Send request to model containing image:

res = client.chat.complete(
    model="mistral-small-2503",
    messages= [
        {
            "content": [
                {"type": "text", "text": "Describe this image"},
                {"type": "image_url", "image_url": "https://tripfixers.com/wp-content/uploads/2019/11/eiffel-tower-with-snow.jpeg"}
            ],
            "role": "user"
        }
    ],
)

3. Receive the error:

ValidationError: 2 validation errors for Unmarshaller
body.0.user.content.str
  Input should be a valid string [type=string_type, input_value=[{'type': 'text', 'text':...-tower-with-snow.jpeg'}], input_type=list]
    For further information visit https://errors.pydantic.dev/2.10/v/string_type
body.0.user.content.list[tagged-union[TextChunk,ReferenceChunk]].1
  Input tag 'image_url' found using <lambda>() does not match any of the expected tags: 'text', 'reference' [type=union_tag_invalid, input_value={'type': 'image_url', 'im...l-tower-with-snow.jpeg'}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.10/v/union_tag_invalid

Expected Behavior

It should work the same way as the base client - it accepts image_url key, not only text.

Additional Context

No response

Suggested Solutions

I guess, there is a problem in ContentChunkTypedDict class for the GCP client - it's not the same as for the base client.

[GaspardBT]

Hey 👋
Thanks for opening this issue, we haven't updated the gcp client to support images (yet), we planned to do it soon.
Sorry for the delay.
We will update this issue once available

[romantsovmike]

@GaspardBT thanks for the quick response
Could you suggest any workaround for now if I need to process images and use gcp authentication?

[GaspardBT]

Sorry I don't have an easy workaround for this 🙇
The fix is in this PR which we plan to merge later today 👍