Skip to content

[module][langchain_mlrun] First version#963

Merged
Eyal-Danieli merged 57 commits intomlrun:developmentfrom
guy1992l:langchain_mlrun
Feb 15, 2026
Merged

[module][langchain_mlrun] First version#963
Eyal-Danieli merged 57 commits intomlrun:developmentfrom
guy1992l:langchain_mlrun

Conversation

@guy1992l
Copy link
Member

@guy1992l guy1992l commented Jan 8, 2026

Created the module langchain_mlrun - MLRun's integration with LangChain, allowing to auto-trace and monitor LangChain and LangGraph based code.

  • Added the module itself - fully documented and sphinx ready.
  • Added about 76 tests for 100% of the current functionality.
  • Added an end-to-end notebook with brief explanations and tutorial.
  • Supports both v3io (MLRun Enterprise) and kafka (MLRun CE) - @tomerbv 👏.
  • Wrote TODOs for future work.

yonishelach and others added 30 commits March 8, 2023 11:01
[Build] Fix html links, add <function>.html as source in documentation
[XGB-Custom] Fix test artifact key name
[XGB-Serving][XGB-Test][XGB-Trainer] Fix tests - artifact key
* [Build] Install python 3.9 when testing

* [Build] Update python version in CI

* .
* [Build] Build with python 3.9

* .
* [Noise-reduction] Add new function to hub

* fix test

* added multiprocessing and silence removal to function
* delete EOS functions

* bring back validate_great_expectations

* bring back load_dataset
Eyal-Danieli and others added 5 commits April 10, 2025 09:10
@review-notebook-app
Copy link

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.


Powered by ReviewNB

@guy1992l guy1992l requested a review from Copilot January 8, 2026 22:50
@guy1992l guy1992l requested review from Eyal-Danieli and omermaim and removed request for Copilot January 8, 2026 22:50
Copy link
Member

@Eyal-Danieli Eyal-Danieli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

very nice, had few comments

:returns: An MLRun model endpoint monitoring client.
"""
if mlrun.mlconf.is_ce_mode():
return _KafkaMLRunEndPointClient(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

missing monitoring_broker and monitoring_topic params

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

its a "hachana lemazgan"... there is a TODO basically anywhere related to Kafka usage.

Copy link
Member

@Eyal-Danieli Eyal-Danieli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

plus add copyright

@guy1992l guy1992l requested a review from Eyal-Danieli January 11, 2026 21:30
Copy link
Contributor

@omermaim omermaim left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good to me

@@ -0,0 +1,23 @@
apiVersion: v1
categories:
- langchain
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Eyal-Danieli, which categories do we support?
Should it be on general for now?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@guy1992l do we have an option to deploy the agent code into the user cluster?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In general, we have a closed mapping of the categories that you can see in the UI (https://www.mlrun.org/hub/) but your question is in place as we should add it to the contribution guidelines (I'll open a ticket). As for the new categories within this PR, we already added them (will be available once this PR is merged).

@@ -0,0 +1,3 @@
pytest
langchain
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Worth pointing out to a specific version, to make sure the function do not change between package releases

@@ -0,0 +1,899 @@
{
Copy link
Collaborator

@GiladShapira94 GiladShapira94 Jan 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can generate the stream path by using the project name of the user or this env MLRUN_ACTIVE_PROJECT in addition, model monitoring always uses the project container so we can just set these values as default.

In this example, the user does not create any serving function, so we can just set the value under the hood if the user does not provide one.

I think we can remove this line - This is a temporary workaround until custom endpoint creation support is added to MLRun.

Once MLRun will add this feature, we will update the notebook


Reply via ReviewNB

@@ -0,0 +1,899 @@
{
Copy link
Collaborator

@GiladShapira94 GiladShapira94 Jan 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. We should use the same requirements.txt file. If it should be a different file I think it would be better to create an application_requirements.txt
  2. Maybe worth to write another step that invoke the chain after the application deployed


Reply via ReviewNB

tomerbv and others added 4 commits February 9, 2026 23:17
* docstring syntax fixes

* Add initial Kafka support for MLRun CE
Implement _KafkaMLRunEndPointClient with KafkaProducer
Add kafka_broker and kafka_topic to MLRunTracerClientSettings
Add Kafka parameters to setup_langchain_monitoring()
Update notebook to auto-detect CE/Enterprise mode
Add kafka-python, orjson, uuid-utils to requirements.txt

* added KAFKA_BROKER and DatastoreProfileTDEngine for registering kafka and tsdb profiles

* clear output

* use get_kafka_topic for the project name
remove "raises:" docstring
added kafka flush
added s3fs to requirements.txt

* adaptive notebook for local execution
- AWS_ENDPOINT_URL_S3 env variable in deployment
- port forwarding scripts

* Replace kafka_broker/kafka_topic with stream_profile_name
 - Update _KafkaMLRunEndPointClient to use DatastoreProfileKafkaStream
 - Fetch Kafka config (broker, topic, SASL, SSL) from registered profile
 - Auto-retrieve stream_profile_name from model monitoring credentials
 - Update MLRunTracerClientSettings with new stream_profile_name field
 - Update setup_langchain_monitoring() to use profile-based config
 - Update notebook to use simplified API

* Add get_kafka_stream_profile_name() utility to auto-detect Kafka profile

* Move dependencies from requirements.txt to item.yaml

* generalize env variables

* Deploy monitoring app once instead of redeploy
Unify profile variable naming for CE and Enterprise modes

* _KafkaMLRunEndPointClient constructor - keep project handling NONE able with parent class handling in

* revert removal of raise docstring
enforce usage of stream_profile_name

* change stream_profile_name to kafka_stream_profile_name
revert ValueError message

* Revert ValueError message
Added mechanism to flush stream upon root run (instead of each monitor call)

* ensure self._mlrun_client is not None before flush

* revert double deploy

* Add configurable kafka_linger_ms for hybrid flush control

Adds kafka_linger_ms parameter to control message delivery timing:
 - Explicit flush mode (linger_ms=0, default): flush after each root run
 - Kafka-managed mode (linger_ms>0): Kafka controls delivery timing

The flush() method now handles the mode internally - it's a no-op when Kafka-managed mode is enabled, keeping the tracer code simple.

* fix double deploy in notebook (set the in-cluster MinIO endpoint before deploying)

* Simplify Kafka flush behavior and enable batching by default
- Always flush at end of root run (removed conditional linger_ms check)
- Set default kafka_linger_ms to 500ms for message batching
- Simplify KafkaProducer initialization (pop bootstrap_servers instead of lambda)

* remove local module import from notebook
remove redundant requirements from item.yaml

* move env variables setup to top of the notebook
@Eyal-Danieli Eyal-Danieli merged commit 5519171 into mlrun:development Feb 15, 2026
3 checks passed
Eyal-Danieli added a commit to Eyal-Danieli/functions that referenced this pull request Feb 16, 2026
* [Build] Fix html links, Add <function>.html as source in documentation

* Update CI temporarily and update index

* [XGB-Custom] Fix test artifact key name

* [XGB-Serving][XGB-Test][XGB-Trainer] Fix tests - artifact key

* [Build] Install python 3.9 when testing (mlrun#618)

* [Build] Update python version in CI (mlrun#620)

* [Build] Install python 3.9 when testing

* [Build] Update python version in CI

* .

* Revert "[Build] Update python version in CI (mlrun#620)" (mlrun#621)

This reverts commit 0cd1f15.

* Revert "[Build] Install python 3.9 when testing (mlrun#618)" (mlrun#619)

This reverts commit 3301415.

* [Build] Build with python 3.9 (mlrun#622)

* [Build] Build with python 3.9

* .

* [onnx utils] update onnx utils packages

* [Noise-reduction] Add new function to hub (mlrun#765)

* [Noise-reduction] Add new function to hub

* fix test

* added multiprocessing and silence removal to function

* delete `load_dask` (mlrun#822)

* [feature selection] update function yaml

* [feature selection] update function yaml

* Revert "[onnx utils] update onnx utils packages"

This reverts commit 8872798.

* [feature selection] update function yaml

* [feature selection] update function yaml

* Delete unsupported functions from the hub (mlrun#824)

* delete EOS functions

* bring back validate_great_expectations

* bring back load_dataset

* Update feature_selection/test_feature_selection.py

Co-authored-by: Eyal Danieli <eyald@iguazio.com>

* Update item.yaml

* Align to master branch (mlrun#826)

* [Category] Fix and add categories to functions (mlrun#808)

* [Category] Fix and add categories to functions

* bump version in structured

* test is not valid in huggingface_serving

* Fix duplicated footer

* Fix duplicated footer

* revert python version change as it will be done in another PR

* comments

* comments

* Bump python:3.6 to python:3.9 (mlrun#810)

* [Describe] Align describe to new pandas version (mlrun#812)

* [Describe] Align describe to new pandas version

* minor test fix

* update mlrun version

* add dask to requirements

* remove dask

* update numpy version

* debug

* debug

* debug

* remove dask tests

* remove debug code

* [get_offline_features] Updated to mlrun 1.6.3 (mlrun#813)

* [Feature-selection] Replace matplotlib with plotly (mlrun#815)

* Iguazio-cicd user token updated

Iguazio-cicd user token updated in repo secrets:
https://github.com/mlrun/functions/settings/secrets/actions
MARKETPLACE_ACCESS_TOKEN_V3
new token gh...Zmf was set around April

* forcing iguazio-cicd auth

forcing iguazio-cicd to deal with Author identity unknown

* checkout@v3 to v4 and echo

* [Mlflow_utils] - mlflow model server (mlrun#811)

* mlflow server

* small fix to test

* small fixes to ms and nb

* small fixes to mlrun version

* update requirements lightgbm

* added req

* Added xgboost to req

---------

Co-authored-by: Avi Asulin <34214569+aviaIguazio@users.noreply.github.com>

* [Mlflow] Remove mlflow tag  (mlrun#825)

* remove mlflow tag

* remove mlflow tag

---------

Co-authored-by: Avi Asulin <34214569+aviaIguazio@users.noreply.github.com>

* align feature_selection yaml

---------

Co-authored-by: Avi Asulin <34214569+aviaIguazio@users.noreply.github.com>
Co-authored-by: Yonatan Shelach <92271540+yonishelach@users.noreply.github.com>
Co-authored-by: rokatyy <ekaterina_molchanova@mckinsey.com>
Co-authored-by: Katerina Molchanova <35141662+rokatyy@users.noreply.github.com>
Co-authored-by: nashpaz123 <44337075+nashpaz123@users.noreply.github.com>
Co-authored-by: ZeevRispler <73653682+ZeevRispler@users.noreply.github.com>

* set `navigation_with_keys` to False (mlrun#829)

* remove xgb and churn functions (mlrun#830)

* [Batch Infer V2] Adjust function to 1.7 (mlrun#832)

* adjust batch infer v2

* update docs in NB

* bring back deprecated params and add warn (mlrun#834)

* fix PyYAML loading (mlrun#837)

* [text to audio generator] Replaced bark with openai tts models (mlrun#836)

* [Text to audio generator] Add speech engine (mlrun#838)

* [text to audio generator] Replaced bark with openai tts models

* [text to audio generator] Fix base url env var

* fix version

* Add speech engine

* after review

* [auto-trainer] update test requirements (mlrun#839)

* [Build] Fix html links, Add <function>.html as source in documentation

* Update CI temporarily and update index

* [XGB-Custom] Fix test artifact key name

* [XGB-Serving][XGB-Test][XGB-Trainer] Fix tests - artifact key

* [Build] Install python 3.9 when testing (mlrun#618)

* [Build] Update python version in CI (mlrun#620)

* [Build] Install python 3.9 when testing

* [Build] Update python version in CI

* .

* Revert "[Build] Update python version in CI (mlrun#620)" (mlrun#621)

This reverts commit 0cd1f15.

* Revert "[Build] Install python 3.9 when testing (mlrun#618)" (mlrun#619)

This reverts commit 3301415.

* [Build] Build with python 3.9 (mlrun#622)

* [Build] Build with python 3.9

* .

* Update requirements.txt

* [Feature Selection] Fix deprecated `get_offline_features`  (mlrun#844)

* fix feature_selection

* fix feature_selection

* fix feature_selection nb

* update yaml name

* fix test

* fix test

* limit torchaudio for unit test (mlrun#845)

* Update requirements.txt (mlrun#843)

* [Open Archive] Fix arbitrary file vulnerability  (mlrun#847)

* fix arbitrary file vulnerability

* fix arbitrary file vulnerability

* fix test

* [open_archive] Add traversal attack test  (mlrun#849)

* add traversal test

* add traversal test

* add traversal test

* first version

* run upg to pydantic v2

* added kafka and mlrun-ce code preparation

* Eyal review

* LangChain MLRun Integration with Kafka Support (CE Mode) (mlrun#1)

* docstring syntax fixes

* Add initial Kafka support for MLRun CE
Implement _KafkaMLRunEndPointClient with KafkaProducer
Add kafka_broker and kafka_topic to MLRunTracerClientSettings
Add Kafka parameters to setup_langchain_monitoring()
Update notebook to auto-detect CE/Enterprise mode
Add kafka-python, orjson, uuid-utils to requirements.txt

* added KAFKA_BROKER and DatastoreProfileTDEngine for registering kafka and tsdb profiles

* clear output

* use get_kafka_topic for the project name
remove "raises:" docstring
added kafka flush
added s3fs to requirements.txt

* adaptive notebook for local execution
- AWS_ENDPOINT_URL_S3 env variable in deployment
- port forwarding scripts

* Replace kafka_broker/kafka_topic with stream_profile_name
 - Update _KafkaMLRunEndPointClient to use DatastoreProfileKafkaStream
 - Fetch Kafka config (broker, topic, SASL, SSL) from registered profile
 - Auto-retrieve stream_profile_name from model monitoring credentials
 - Update MLRunTracerClientSettings with new stream_profile_name field
 - Update setup_langchain_monitoring() to use profile-based config
 - Update notebook to use simplified API

* Add get_kafka_stream_profile_name() utility to auto-detect Kafka profile

* Move dependencies from requirements.txt to item.yaml

* generalize env variables

* Deploy monitoring app once instead of redeploy
Unify profile variable naming for CE and Enterprise modes

* _KafkaMLRunEndPointClient constructor - keep project handling NONE able with parent class handling in

* revert removal of raise docstring
enforce usage of stream_profile_name

* change stream_profile_name to kafka_stream_profile_name
revert ValueError message

* Revert ValueError message
Added mechanism to flush stream upon root run (instead of each monitor call)

* ensure self._mlrun_client is not None before flush

* revert double deploy

* Add configurable kafka_linger_ms for hybrid flush control

Adds kafka_linger_ms parameter to control message delivery timing:
 - Explicit flush mode (linger_ms=0, default): flush after each root run
 - Kafka-managed mode (linger_ms>0): Kafka controls delivery timing

The flush() method now handles the mode internally - it's a no-op when Kafka-managed mode is enabled, keeping the tracer code simple.

* fix double deploy in notebook (set the in-cluster MinIO endpoint before deploying)

* Simplify Kafka flush behavior and enable batching by default
- Always flush at end of root run (removed conditional linger_ms check)
- Set default kafka_linger_ms to 500ms for message batching
- Simplify KafkaProducer initialization (pop bootstrap_servers instead of lambda)

* remove local module import from notebook
remove redundant requirements from item.yaml

* move env variables setup to top of the notebook

* renamed env vars

* fix test

* gilad review

---------

Co-authored-by: yonishelach <yonatanshelach@gmail.com>
Co-authored-by: Yoni Shelach <92271540+yonishelach@users.noreply.github.com>
Co-authored-by: Avi Asulin <avia@iguazio.com>
Co-authored-by: Eyal Danieli <eyald@iguazio.com>
Co-authored-by: Avi Asulin <34214569+aviaIguazio@users.noreply.github.com>
Co-authored-by: rokatyy <ekaterina_molchanova@mckinsey.com>
Co-authored-by: Katerina Molchanova <35141662+rokatyy@users.noreply.github.com>
Co-authored-by: nashpaz123 <44337075+nashpaz123@users.noreply.github.com>
Co-authored-by: ZeevRispler <73653682+ZeevRispler@users.noreply.github.com>
Co-authored-by: daniels290813 <78727943+daniels290813@users.noreply.github.com>
Co-authored-by: Tomer Weitzman <81749152+tomerbv@users.noreply.github.com>
Eyal-Danieli added a commit that referenced this pull request Feb 16, 2026
* Update scikit-learn to version 1.5    (#967)

* updated scikit-learn~=1.5
fixes and patches for new scikit-learn version
changes in item.yaml and regenerate function.yaml

* remove filename

* remove numpy import

* revert sklearn.metrics monkey patch
fix _get_dataframe to handle list/dict before accessing artifact_url
added feature name preservation logic in predict function

* revert mlrun version

* revert get_or_create_project

* revert scikit-learn version

* scikit-learn==1.5.2
mlrun v 1.10

* scikit-learn==1.4.2

* revert scikit-learn<1.4.0

* scikit-learn~=1.5

* mlrun 1.10 with scikit-learn<1.4.0

* scikit-learn strict v~=1.5.2
added skip for test_train in test_auto_trainer.py

* revert sklearn_classifier.py changes
change XGBRegressor to LGBMRegressor

* added xgboost.XGBRegressor, xgboost.XGBClassifier and lightgbm.LGBMClassifier models to test

* [module][langchain_mlrun] First version (#963)

* [Build] Fix html links, Add <function>.html as source in documentation

* Update CI temporarily and update index

* [XGB-Custom] Fix test artifact key name

* [XGB-Serving][XGB-Test][XGB-Trainer] Fix tests - artifact key

* [Build] Install python 3.9 when testing (#618)

* [Build] Update python version in CI (#620)

* [Build] Install python 3.9 when testing

* [Build] Update python version in CI

* .

* Revert "[Build] Update python version in CI (#620)" (#621)

This reverts commit 0cd1f15.

* Revert "[Build] Install python 3.9 when testing (#618)" (#619)

This reverts commit 3301415.

* [Build] Build with python 3.9 (#622)

* [Build] Build with python 3.9

* .

* [onnx utils] update onnx utils packages

* [Noise-reduction] Add new function to hub (#765)

* [Noise-reduction] Add new function to hub

* fix test

* added multiprocessing and silence removal to function

* delete `load_dask` (#822)

* [feature selection] update function yaml

* [feature selection] update function yaml

* Revert "[onnx utils] update onnx utils packages"

This reverts commit 8872798.

* [feature selection] update function yaml

* [feature selection] update function yaml

* Delete unsupported functions from the hub (#824)

* delete EOS functions

* bring back validate_great_expectations

* bring back load_dataset

* Update feature_selection/test_feature_selection.py

Co-authored-by: Eyal Danieli <eyald@iguazio.com>

* Update item.yaml

* Align to master branch (#826)

* [Category] Fix and add categories to functions (#808)

* [Category] Fix and add categories to functions

* bump version in structured

* test is not valid in huggingface_serving

* Fix duplicated footer

* Fix duplicated footer

* revert python version change as it will be done in another PR

* comments

* comments

* Bump python:3.6 to python:3.9 (#810)

* [Describe] Align describe to new pandas version (#812)

* [Describe] Align describe to new pandas version

* minor test fix

* update mlrun version

* add dask to requirements

* remove dask

* update numpy version

* debug

* debug

* debug

* remove dask tests

* remove debug code

* [get_offline_features] Updated to mlrun 1.6.3 (#813)

* [Feature-selection] Replace matplotlib with plotly (#815)

* Iguazio-cicd user token updated

Iguazio-cicd user token updated in repo secrets:
https://github.com/mlrun/functions/settings/secrets/actions
MARKETPLACE_ACCESS_TOKEN_V3
new token gh...Zmf was set around April

* forcing iguazio-cicd auth

forcing iguazio-cicd to deal with Author identity unknown

* checkout@v3 to v4 and echo

* [Mlflow_utils] - mlflow model server (#811)

* mlflow server

* small fix to test

* small fixes to ms and nb

* small fixes to mlrun version

* update requirements lightgbm

* added req

* Added xgboost to req

---------

Co-authored-by: Avi Asulin <34214569+aviaIguazio@users.noreply.github.com>

* [Mlflow] Remove mlflow tag  (#825)

* remove mlflow tag

* remove mlflow tag

---------

Co-authored-by: Avi Asulin <34214569+aviaIguazio@users.noreply.github.com>

* align feature_selection yaml

---------

Co-authored-by: Avi Asulin <34214569+aviaIguazio@users.noreply.github.com>
Co-authored-by: Yonatan Shelach <92271540+yonishelach@users.noreply.github.com>
Co-authored-by: rokatyy <ekaterina_molchanova@mckinsey.com>
Co-authored-by: Katerina Molchanova <35141662+rokatyy@users.noreply.github.com>
Co-authored-by: nashpaz123 <44337075+nashpaz123@users.noreply.github.com>
Co-authored-by: ZeevRispler <73653682+ZeevRispler@users.noreply.github.com>

* set `navigation_with_keys` to False (#829)

* remove xgb and churn functions (#830)

* [Batch Infer V2] Adjust function to 1.7 (#832)

* adjust batch infer v2

* update docs in NB

* bring back deprecated params and add warn (#834)

* fix PyYAML loading (#837)

* [text to audio generator] Replaced bark with openai tts models (#836)

* [Text to audio generator] Add speech engine (#838)

* [text to audio generator] Replaced bark with openai tts models

* [text to audio generator] Fix base url env var

* fix version

* Add speech engine

* after review

* [auto-trainer] update test requirements (#839)

* [Build] Fix html links, Add <function>.html as source in documentation

* Update CI temporarily and update index

* [XGB-Custom] Fix test artifact key name

* [XGB-Serving][XGB-Test][XGB-Trainer] Fix tests - artifact key

* [Build] Install python 3.9 when testing (#618)

* [Build] Update python version in CI (#620)

* [Build] Install python 3.9 when testing

* [Build] Update python version in CI

* .

* Revert "[Build] Update python version in CI (#620)" (#621)

This reverts commit 0cd1f15.

* Revert "[Build] Install python 3.9 when testing (#618)" (#619)

This reverts commit 3301415.

* [Build] Build with python 3.9 (#622)

* [Build] Build with python 3.9

* .

* Update requirements.txt

* [Feature Selection] Fix deprecated `get_offline_features`  (#844)

* fix feature_selection

* fix feature_selection

* fix feature_selection nb

* update yaml name

* fix test

* fix test

* limit torchaudio for unit test (#845)

* Update requirements.txt (#843)

* [Open Archive] Fix arbitrary file vulnerability  (#847)

* fix arbitrary file vulnerability

* fix arbitrary file vulnerability

* fix test

* [open_archive] Add traversal attack test  (#849)

* add traversal test

* add traversal test

* add traversal test

* first version

* run upg to pydantic v2

* added kafka and mlrun-ce code preparation

* Eyal review

* LangChain MLRun Integration with Kafka Support (CE Mode) (#1)

* docstring syntax fixes

* Add initial Kafka support for MLRun CE
Implement _KafkaMLRunEndPointClient with KafkaProducer
Add kafka_broker and kafka_topic to MLRunTracerClientSettings
Add Kafka parameters to setup_langchain_monitoring()
Update notebook to auto-detect CE/Enterprise mode
Add kafka-python, orjson, uuid-utils to requirements.txt

* added KAFKA_BROKER and DatastoreProfileTDEngine for registering kafka and tsdb profiles

* clear output

* use get_kafka_topic for the project name
remove "raises:" docstring
added kafka flush
added s3fs to requirements.txt

* adaptive notebook for local execution
- AWS_ENDPOINT_URL_S3 env variable in deployment
- port forwarding scripts

* Replace kafka_broker/kafka_topic with stream_profile_name
 - Update _KafkaMLRunEndPointClient to use DatastoreProfileKafkaStream
 - Fetch Kafka config (broker, topic, SASL, SSL) from registered profile
 - Auto-retrieve stream_profile_name from model monitoring credentials
 - Update MLRunTracerClientSettings with new stream_profile_name field
 - Update setup_langchain_monitoring() to use profile-based config
 - Update notebook to use simplified API

* Add get_kafka_stream_profile_name() utility to auto-detect Kafka profile

* Move dependencies from requirements.txt to item.yaml

* generalize env variables

* Deploy monitoring app once instead of redeploy
Unify profile variable naming for CE and Enterprise modes

* _KafkaMLRunEndPointClient constructor - keep project handling NONE able with parent class handling in

* revert removal of raise docstring
enforce usage of stream_profile_name

* change stream_profile_name to kafka_stream_profile_name
revert ValueError message

* Revert ValueError message
Added mechanism to flush stream upon root run (instead of each monitor call)

* ensure self._mlrun_client is not None before flush

* revert double deploy

* Add configurable kafka_linger_ms for hybrid flush control

Adds kafka_linger_ms parameter to control message delivery timing:
 - Explicit flush mode (linger_ms=0, default): flush after each root run
 - Kafka-managed mode (linger_ms>0): Kafka controls delivery timing

The flush() method now handles the mode internally - it's a no-op when Kafka-managed mode is enabled, keeping the tracer code simple.

* fix double deploy in notebook (set the in-cluster MinIO endpoint before deploying)

* Simplify Kafka flush behavior and enable batching by default
- Always flush at end of root run (removed conditional linger_ms check)
- Set default kafka_linger_ms to 500ms for message batching
- Simplify KafkaProducer initialization (pop bootstrap_servers instead of lambda)

* remove local module import from notebook
remove redundant requirements from item.yaml

* move env variables setup to top of the notebook

* renamed env vars

* fix test

* gilad review

---------

Co-authored-by: yonishelach <yonatanshelach@gmail.com>
Co-authored-by: Yoni Shelach <92271540+yonishelach@users.noreply.github.com>
Co-authored-by: Avi Asulin <avia@iguazio.com>
Co-authored-by: Eyal Danieli <eyald@iguazio.com>
Co-authored-by: Avi Asulin <34214569+aviaIguazio@users.noreply.github.com>
Co-authored-by: rokatyy <ekaterina_molchanova@mckinsey.com>
Co-authored-by: Katerina Molchanova <35141662+rokatyy@users.noreply.github.com>
Co-authored-by: nashpaz123 <44337075+nashpaz123@users.noreply.github.com>
Co-authored-by: ZeevRispler <73653682+ZeevRispler@users.noreply.github.com>
Co-authored-by: daniels290813 <78727943+daniels290813@users.noreply.github.com>
Co-authored-by: Tomer Weitzman <81749152+tomerbv@users.noreply.github.com>

* [onnx_utils] updated pytorch due to security vulnerability (#968)

* Update onnx_utils dependencies and improve test robustness

  - Upgrade PyTorch (2.6.0 → 2.8.0) and TorchVision (0.21.0 → 0.23.0) for
    better compatibility and performance
  - Update MLRun version requirement to 1.10.0 in item.yaml
  - Bump function version to 1.4.0

  Test improvements:
  - Add environment variable validation (MLRUN_DBPATH, MLRUN_ARTIFACT_PATH)
  - Add conditional test skipping based on tf2onnx availability
  - Fix cleanup function to properly remove test artifacts (model.pt,
    model_modules_map.json, onnx_model.onnx, etc.)
  - Update deprecated artifact_path parameter to output_path
  - Add explicit project context to all MLRun function calls
  - Fix PyTorch test artifact path construction

* Add conftest fixture for test environment and update notebook to PyTorch demo

- Centralize test setup/cleanup in conftest autouse fixture
- Rewrite notebook demo from Keras to a working PyTorch MobileNetV2 example

* deleted iguazio credentials

* Remove conftest.py and inline fixtures into test_onnx_utils.py

Move onnx_project and test_environment fixtures directly into the test
file to reduce unnecessary indirection for a single test module.

---------

Co-authored-by: Tomer Weitzman <81749152+tomerbv@users.noreply.github.com>
Co-authored-by: guy1992l <83535508+guy1992l@users.noreply.github.com>
Co-authored-by: yonishelach <yonatanshelach@gmail.com>
Co-authored-by: Yoni Shelach <92271540+yonishelach@users.noreply.github.com>
Co-authored-by: Avi Asulin <avia@iguazio.com>
Co-authored-by: Avi Asulin <34214569+aviaIguazio@users.noreply.github.com>
Co-authored-by: rokatyy <ekaterina_molchanova@mckinsey.com>
Co-authored-by: Katerina Molchanova <35141662+rokatyy@users.noreply.github.com>
Co-authored-by: nashpaz123 <44337075+nashpaz123@users.noreply.github.com>
Co-authored-by: ZeevRispler <73653682+ZeevRispler@users.noreply.github.com>
Co-authored-by: daniels290813 <78727943+daniels290813@users.noreply.github.com>
Co-authored-by: Omer Mimon <81911093+omermaim@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants