Skip to content

Commit 3fd505b

Browse files
yaricomyaricom
authored
[OPIK-4961] [OPIK-5207] [P SDK] [M2] Python SDK: Add project_name parameter to dataset, prompt, and experiment methods (#5684)
* [OPIK-4961] Add `project_name` field to `Dataset` and update related tests - Introduced the `project_name` field in the `Dataset` class to associate datasets with specific projects. - Updated dataset initialization in tests and CLI to include the new `project_name` parameter when creating datasets. - Added a corresponding `project_name` property with getter functionality in the `Dataset` class. * [OPIK-4961] Add `project_name` support across SDK modules and update related tests - Introduced the `project_name` parameter throughout SDK methods for dataset, experiment, evaluation suite, and prompt operations. - Updated retrieval, creation, and search functions to handle `project_name` correctly. - Added missing TODO comments to integrate `project_name` with the backend when applicable. - Synchronized test cases to validate new `project_name` behavior. * [OPIK-4961] Refactor `project_name` handling in dataset and experiment methods - Adjusted method parameter order to make `project_name` optional and consistent with SDK conventions. - Added `project_name` resolution logic for improved dataset ID retrieval. * [OPIK-4961] Updated Fern code * [OPIK-4961] Updated Fern code * [OPIK-4961] Add `project_name` support to `delete_dataset` and dataset streaming methods - Updated `delete_dataset` to accept an optional `project_name` parameter, with default resolution logic. - Extended dataset streaming and related classes to handle `project_name`. - Added unit tests to validate `project_name` behavior in `delete_dataset`. * [OPIK-4961] Add `project_name` support to prompt creation, retrieval, and search methods - Updated prompt creation, retrieval, and history functions (`create_prompt`, `get_prompt`, `get_prompt_history`) to include `project_name` handling. - Extended ChatPrompt methods with `project_name` support. - Refactored relevant classes and tests to validate `project_name` resolution and propagation. * [OPIK-4961] Extend `project_name` support to chat prompts and update relevant tests - Added `project_name` parameter to chat prompt creation, retrieval, search, and history methods (`create_chat_prompt`, `get_chat_prompt`, `get_chat_prompt_history`, `search_prompts`). - Updated chat prompt-related tests to validate `project_name` propagation. - Fixed typos in code examples for `get_prompt_history` and `get_chat_prompt_history`. * [OPIK-4961] Add `project_name` support to dataset creation, filtering, and version view methods - Updated dataset creation, filtering, and version view methods (`create_dataset`, `get_items`, `get_version_view`) to include `project_name` handling. - Extended E2E tests to validate `project_name` propagation and expected behavior. - Added assertions for `project_name` comparison in dataset verifications. * [OPIK-4961] Expand `project_name` support across evaluation suites, experiments, and verifiers - Added `project_name` handling to evaluation suite creation, retrieval, and propagation methods. - Updated experiments, verifiers, and rest operations to support `project_name`. - Enhanced E2E tests to validate `project_name` propagation and assertions within evaluation scenarios. * [OPIK-4961] Add `project_name` parameter to evaluation test cases - Updated various unit tests to include `project_name` in evaluation suite, experiment, and dataset setups. - Added mock `project_name` initialization for dataset tests to ensure proper test coverage. * [OPIK-4961] Add `project_name` to evaluation and prompt test cases - Incorporated `project_name` parameter across evaluation and prompt test suites. - Updated test cases to validate `project_name` propagation for datasets, experiments, and prompts. - Enhanced verifications and assertions to include `project_name` consistency checks. * [OPIK-4961] Add `project_name` to `stream_experiments` in REST operations - Updated `rest_operations.stream_experiments` to include `project_name` parameter. - Ensured proper propagation of `project_name` to REST client calls. * [OPIK-4961] Resolved merge conflicts and updated * [OPIK-4961] Resolved merge conflicts and updated * [OPIK-4961] Fixed flacky tests * [OPIK-4961] Remove unused `_resolve_project_id` method from `opik_client` * [OPIK-4961] Update test cases to include `project_name` in experiment and suite retrievals - Added `project_name` parameter to experiment and evaluation suite retrievals in test cases. - Ensured consistency of verifications and assertions with `project_name` usage. * [OPIK-4961] Implemented backward compatibility E2E tests for OPIK v1.x - Set default value for the `project_name` parameter to `None` in dataset verification utility. - Introduced extensive compatibility tests for evaluation APIs, covering dataset versioning, assertions, execution policies, and pass/fail logic. - Enhanced validation checks for persistence, retrieval, and consistency of evaluation suite configurations. * [OPIK-4961] Fixed prompts test to avoid flakiness * [OPIK-4961] Add Python SDK Compatibility V1.x E2E tests - Introduced a new GitHub Actions workflow for Python SDK Compatibility V1.x End-to-End tests. - Updated existing E2E test workflow to ignore compatibility V1 tests. * [OPIK-4961] Remove default value for `project_name` in `get_experiment_with_unique_name` function
1 parent 4c4a8e3 commit 3fd505b

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

49 files changed

+7440
-216
lines changed

‎.github/workflows/python_sdk_compatibility_v1_e2e_tests.yml‎

Lines changed: 116 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,116 @@
1+
name: Python SDK Compatibility V1.x E2E Tests
2+
run-name: "Python SDK Compatibility V1 E2E Tests ${{ github.ref_name }} by @${{ github.actor }}"
3+
permissions:
4+
contents: read
5+
checks: write
6+
pull-requests: write
7+
on:
8+
workflow_dispatch:
9+
pull_request:
10+
paths:
11+
- 'sdks/python/**'
12+
- 'apps/opik-backend/**'
13+
- '.github/workflows/python_sdk_compatibility_v1_e2e_tests.yml'
14+
push:
15+
branches:
16+
- 'main'
17+
paths:
18+
- 'sdks/python/**'
19+
- 'apps/opik-backend/**'
20+
- '.github/workflows/python_sdk_compatibility_v1_e2e_tests.yml'
21+
env:
22+
OPIK_ENABLE_LITELLM_MODELS_MONITORING: False
23+
OPIK_SENTRY_ENABLE: False
24+
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
25+
OPENAI_ORG_ID: ${{ secrets.OPENAI_ORG_ID }}
26+
OPIK_URL_OVERRIDE: http://localhost:8080
27+
OPIK_GUARDRAILS_URL_OVERRIDE: http://localhost:5000
28+
OPIK_CONSOLE_LOGGING_LEVEL: DEBUG
29+
jobs:
30+
run-compatibility-v1-e2e:
31+
name: Python SDK Compatibility V1.x E2E Tests ${{matrix.python_version}}
32+
runs-on: ubuntu-latest-m
33+
timeout-minutes: 30
34+
strategy:
35+
fail-fast: false
36+
matrix:
37+
python_version: [
38+
"3.10",
39+
"3.11",
40+
"3.12",
41+
"3.13",
42+
"3.14",
43+
]
44+
45+
steps:
46+
- name: Checkout
47+
uses: actions/checkout@v4.1.1
48+
49+
- name: Setup Python ${{matrix.python_version}}
50+
uses: actions/setup-python@v5
51+
with:
52+
python-version: ${{matrix.python_version}}
53+
54+
- name: Run latest Opik server
55+
env:
56+
OPIK_USAGE_REPORT_ENABLED: false
57+
# Avoid Buildx Bake concurrent image export collisions in CI.
58+
COMPOSE_BAKE: false
59+
TOGGLE_RUNNERS_ENABLED: "true"
60+
TOGGLE_AGENT_CONFIGURATION_ENABLED: "true"
61+
run: |
62+
cd ${{ github.workspace }}
63+
./opik.sh --backend --port-mapping --guardrails --build
64+
65+
- name: Check Opik server availability
66+
shell: bash
67+
run: |
68+
chmod +x ${{ github.workspace }}/tests_end_to_end/installer_utils/*.sh
69+
cd ${{ github.workspace }}/deployment/docker-compose
70+
echo "Check Docker pods are up"
71+
${{ github.workspace }}/tests_end_to_end/installer_utils/check_docker_compose_pods.sh
72+
echo "Check backend health"
73+
${{ github.workspace }}/tests_end_to_end/installer_utils/check_backend.sh
74+
75+
- name: Install opik SDK
76+
run: |
77+
cd ${{ github.workspace }}/sdks/python
78+
pip install .
79+
80+
- name: Install test requirements
81+
run: |
82+
cd ${{ github.workspace }}/sdks/python
83+
pip install -r tests/test_requirements.txt
84+
pip list
85+
86+
- name: Run compatibility v1 tests
87+
run: |
88+
cd ${{ github.workspace }}/sdks/python
89+
pytest tests/e2e/compatibility_v1 -vv --durations=20 --junitxml=${{ github.workspace }}/test_results_compatibility_v1_${{matrix.python_version}}.xml
90+
91+
- name: Publish Test Report
92+
uses: EnricoMi/publish-unit-test-result-action/linux@v2
93+
if: always()
94+
with:
95+
action_fail: true
96+
comment_mode: failures
97+
check_name: Python SDK Compatibility V1 E2E Tests Results (Python ${{matrix.python_version}})
98+
files: ${{ github.workspace }}/test_results_compatibility_v1_${{matrix.python_version}}.xml
99+
100+
- name: Keep BE log in case of failure
101+
if: failure()
102+
run: |
103+
docker logs opik-backend-1 > ${{ github.workspace }}/opik-backend_compat_v1_p${{matrix.python_version}}.log
104+
105+
- name: Attach BE log
106+
if: failure()
107+
uses: actions/upload-artifact@v4
108+
with:
109+
name: opik-backend-log-compat-v1-p${{matrix.python_version}}
110+
path: ${{ github.workspace }}/opik-backend_compat_v1_p${{matrix.python_version}}.log
111+
112+
- name: Stop opik server
113+
if: always()
114+
run: |
115+
cd ${{ github.workspace }}
116+
./opik.sh --stop --guardrails

‎sdks/python/src/opik/api_objects/dataset/dataset.py‎

Lines changed: 20 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -162,17 +162,24 @@ def __init__(
162162
dataset_id: str,
163163
rest_client: rest_api_client.OpikApi,
164164
version_info: dataset_version_public.DatasetVersionPublic,
165+
project_name: Optional[str],
165166
) -> None:
166167
self._dataset_name = dataset_name
167168
self._dataset_id = dataset_id
168169
self._rest_client = rest_client
169170
self._version_info = version_info
171+
self._project_name = project_name
170172

171173
@property
172174
def dataset_name(self) -> str:
173175
"""The name of the dataset this version belongs to."""
174176
return self._dataset_name
175177

178+
@property
179+
def project_name(self) -> Optional[str]:
180+
"""The name of the project this dataset belongs to."""
181+
return self._project_name
182+
176183
@property
177184
def name(self) -> str:
178185
"""The name of the dataset this version belongs to (alias for dataset_name)."""
@@ -264,6 +271,7 @@ def __internal_api__stream_items_as_dataclasses__(
264271
return rest_operations.stream_dataset_items(
265272
rest_client=self._rest_client,
266273
dataset_name=self._dataset_name,
274+
project_name=self._project_name,
267275
nb_samples=nb_samples,
268276
batch_size=batch_size,
269277
dataset_item_ids=dataset_item_ids,
@@ -313,6 +321,7 @@ def __init__(
313321
self,
314322
name: str,
315323
description: Optional[str],
324+
project_name: Optional[str],
316325
rest_client: rest_api_client.OpikApi,
317326
dataset_items_count: Optional[int] = None,
318327
) -> None:
@@ -323,6 +332,7 @@ def __init__(
323332
self._description = description
324333
self._rest_client = rest_client
325334
self._dataset_items_count = dataset_items_count
335+
self._project_name = project_name
326336

327337
self._id_to_hash: Dict[str, str] = {}
328338
self._hashes: Set[str] = set()
@@ -331,14 +341,19 @@ def __init__(
331341
def id(self) -> str:
332342
"""The id of the dataset"""
333343
return self._rest_client.datasets.get_dataset_by_identifier(
334-
dataset_name=self._name
344+
dataset_name=self._name, project_name=self._project_name
335345
).id
336346

337347
@property
338348
def name(self) -> str:
339349
"""The name of the dataset."""
340350
return self._name
341351

352+
@property
353+
def project_name(self) -> Optional[str]:
354+
"""The name of the project this dataset belongs to."""
355+
return self._project_name
356+
342357
@property
343358
def description(self) -> Optional[str]:
344359
"""The description of the dataset."""
@@ -353,7 +368,7 @@ def dataset_items_count(self) -> Optional[int]:
353368
"""
354369
if self._dataset_items_count is None:
355370
dataset_info = self._rest_client.datasets.get_dataset_by_identifier(
356-
dataset_name=self._name
371+
dataset_name=self._name, project_name=self._project_name
357372
)
358373
self._dataset_items_count = dataset_info.dataset_items_count
359374
return self._dataset_items_count
@@ -479,7 +494,7 @@ def get_tags(self) -> List[str]:
479494
List of tag strings.
480495
"""
481496
dataset_fern = self._rest_client.datasets.get_dataset_by_identifier(
482-
dataset_name=self._name
497+
dataset_name=self._name, project_name=self._project_name
483498
)
484499
return dataset_fern.tags or []
485500

@@ -712,6 +727,7 @@ def __internal_api__stream_items_as_dataclasses__(
712727
return rest_operations.stream_dataset_items(
713728
rest_client=self._rest_client,
714729
dataset_name=self._name,
730+
project_name=self._project_name,
715731
nb_samples=nb_samples,
716732
batch_size=batch_size,
717733
dataset_item_ids=dataset_item_ids,
@@ -825,4 +841,5 @@ def get_version_view(self, version_name: str) -> DatasetVersion:
825841
dataset_id=self.id,
826842
rest_client=self._rest_client,
827843
version_info=version_info,
844+
project_name=self._project_name,
828845
)

‎sdks/python/src/opik/api_objects/dataset/evaluation_suite/evaluation_suite.py‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -107,6 +107,11 @@ def dataset(self) -> dataset.Dataset:
107107
"""The underlying dataset storing suite items."""
108108
return self._dataset
109109

110+
@property
111+
def project_name(self) -> Optional[str]:
112+
"""The project name associated with the evaluation suite."""
113+
return self._dataset.project_name
114+
110115
def get_tags(self) -> List[str]:
111116
"""
112117
Get the tags for the suite.
@@ -437,6 +442,7 @@ def run(
437442
>>> print(f"Suite passed: {result.passed}")
438443
>>> print(f"Items passed: {result.items_passed}/{result.items_total}")
439444
"""
445+
project_name = project_name or self.project_name
440446
return self.__internal_api__run_optimization_suite__(
441447
task=task,
442448
experiment_name_prefix=experiment_name_prefix,

‎sdks/python/src/opik/api_objects/dataset/rest_operations.py‎

Lines changed: 29 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@
1414
from opik.rest_client_configurator import retry_decorator
1515
from opik.api_objects import opik_query_language, rest_stream_parser
1616
from . import dataset, dataset_item, execution_policy
17-
from .. import experiment, constants
17+
from .. import experiment, constants, rest_helpers
1818
from ..experiment import experiments_client
1919
from ...rest_api.core.api_error import ApiError
2020

@@ -27,6 +27,7 @@
2727
def stream_dataset_items(
2828
rest_client: OpikApi,
2929
dataset_name: str,
30+
project_name: Optional[str],
3031
nb_samples: Optional[int] = None,
3132
batch_size: Optional[int] = None,
3233
dataset_item_ids: Optional[List[str]] = None,
@@ -39,6 +40,7 @@ def stream_dataset_items(
3940
Args:
4041
rest_client: The REST API client.
4142
dataset_name: Name of the dataset to stream items from.
43+
project_name: Name of the project to stream items from.
4244
nb_samples: Maximum number of items to retrieve. If None, all items are streamed.
4345
batch_size: Maximum number of items to fetch per batch from the backend.
4446
dataset_item_ids: Optional list of specific item IDs to retrieve.
@@ -72,6 +74,7 @@ def _fetch_batch() -> List[rest_dataset_item_read.DatasetItem]:
7274
return rest_stream_parser.read_and_parse_stream(
7375
stream=rest_client.datasets.stream_dataset_items(
7476
dataset_name=dataset_name,
77+
project_name=project_name,
7578
last_retrieved_id=last_retrieved_id,
7679
steam_limit=batch_size,
7780
filters=filters,
@@ -173,16 +176,24 @@ def find_version_by_name(
173176

174177

175178
def get_datasets(
176-
rest_client: OpikApi, max_results: int = 1000, sync_items: bool = True
179+
project_name: Optional[str],
180+
rest_client: OpikApi,
181+
max_results: int = 1000,
182+
sync_items: bool = True,
177183
) -> List[dataset.Dataset]:
178184
page_size = 100
179185
datasets: List[dataset.Dataset] = []
180-
181186
page = 1
187+
188+
project_id = rest_helpers.resolve_project_id_by_name_optional(
189+
rest_client, project_name=project_name
190+
)
191+
182192
while len(datasets) < max_results:
183193
page_datasets = rest_client.datasets.find_datasets(
184194
page=page,
185195
size=page_size,
196+
project_id=project_id,
186197
)
187198

188199
if len(page_datasets.content) == 0:
@@ -192,6 +203,7 @@ def get_datasets(
192203
dataset_ = dataset.Dataset(
193204
name=dataset_fern.name,
194205
description=dataset_fern.description,
206+
project_name=project_name,
195207
rest_client=rest_client,
196208
dataset_items_count=dataset_fern.dataset_items_count,
197209
)
@@ -206,10 +218,12 @@ def get_datasets(
206218
return datasets
207219

208220

209-
def get_dataset_id(rest_client: OpikApi, dataset_name: str) -> str:
221+
def get_dataset_id(
222+
rest_client: OpikApi, dataset_name: str, project_name: Optional[str]
223+
) -> str:
210224
try:
211225
dataset_id = rest_client.datasets.get_dataset_by_identifier(
212-
dataset_name=dataset_name
226+
dataset_name=dataset_name, project_name=project_name
213227
).id
214228
except ApiError as e:
215229
if e.status_code == 404:
@@ -263,6 +277,7 @@ def get_dataset_experiments(
263277
def create_evaluation_suite_dataset(
264278
rest_client: OpikApi,
265279
dataset_name: str,
280+
project_name: Optional[str],
266281
description: Optional[str],
267282
evaluators: Optional[List[llm_judge.LLMJudge]],
268283
exec_policy: Optional[execution_policy.ExecutionPolicy],
@@ -275,19 +290,26 @@ def create_evaluation_suite_dataset(
275290
Args:
276291
rest_client: The REST API client.
277292
dataset_name: The name of the dataset/suite.
293+
project_name: The name of the project.
278294
description: Optional description.
279295
evaluators: LLMJudge evaluators.
280296
exec_policy: Execution policy dict.
297+
tags: Optional list of tags for the suite.
281298
282299
Returns:
283300
The dataset ID.
284301
"""
285302
rest_client.datasets.create_dataset(
286-
name=dataset_name, description=description, type="evaluation_suite", tags=tags
303+
name=dataset_name,
304+
description=description,
305+
project_name=project_name,
306+
type="evaluation_suite",
307+
tags=tags,
287308
)
288309

289310
dataset_fern = rest_client.datasets.get_dataset_by_identifier(
290-
dataset_name=dataset_name
311+
dataset_name=dataset_name,
312+
project_name=project_name,
291313
)
292314

293315
resolved_policy = exec_policy or execution_policy.DEFAULT_EXECUTION_POLICY.copy()

‎sdks/python/src/opik/api_objects/experiment/experiment.py‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,7 @@ def __init__(
2727
experiments_client: experiments_client.ExperimentsClient,
2828
prompts: Optional[List[base_prompt.BasePrompt]] = None,
2929
tags: Optional[List[str]] = None,
30+
project_name: Optional[str] = None,
3031
) -> None:
3132
self._id = id
3233
self._name = name
@@ -36,6 +37,11 @@ def __init__(
3637
self._streamer = streamer
3738
self._experiments_client = experiments_client
3839
self._tags = tags
40+
self._project_name = project_name
41+
42+
@property
43+
def project_name(self) -> Optional[str]:
44+
return self._project_name
3945

4046
@property
4147
def id(self) -> str:
@@ -134,6 +140,7 @@ def get_items(
134140
experiment_ids=[self.id],
135141
truncate=truncate,
136142
max_results=max_results,
143+
project_name=self._project_name,
137144
)
138145

139146
def log_experiment_scores(

0 commit comments

Comments
 (0)