Skip to content

BDD tests for removing project in training and testing dataset #232

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
May 21, 2025
Merged

BDD tests for removing project in training and testing dataset #232

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
a030f4c
Add BDD test for deleting training project
JortBergfeld May 15, 2025
90a03c7
Add BDD test for testing dataset
JortBergfeld May 16, 2025
0876c88
Add separate wait for x seconds step
JortBergfeld May 20, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions interactive_ai/tests/e2e/features/project_removal.feature
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,3 +26,16 @@ Feature: project removal
Then the request is rejected
When the user tries to load the image 'cat.jpg'
Then the request is rejected

Scenario: deletion of a project in training
Given an annotated project of type 'detection'
And the user requests to train a model
And a job of type 'train' is running
And the user waits for 10 seconds
When the user tries to delete the project
Then the request is rejected
When the user cancels the job
And the user waits for 5 seconds
And the user deletes the project
And the user tries to load the project
Then the request is rejected
65 changes: 60 additions & 5 deletions interactive_ai/tests/e2e/features/steps/annotation_management.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,10 @@ def _annotate_image_or_frame( # noqa: C901, PLR0912, PLR0915
media_name: str,
labels: list[str],
frame_index: int | None = None,
dataset_id: str | None = None,
) -> None:
if dataset_id is None:
dataset_id = context.dataset_id
annotations_api: AnnotationsApi = context.annotations_api

media_details = context._media_info_by_name[media_name]
Expand Down Expand Up @@ -277,7 +280,7 @@ def _annotate_image_or_frame( # noqa: C901, PLR0912, PLR0915
organization_id=context.organization_id,
workspace_id=context.workspace_id,
project_id=context.project_id,
dataset_id=context.dataset_id,
dataset_id=dataset_id,
image_id=media_details.id,
create_image_annotation_request=CreateImageAnnotationRequest(
media_identifier=media_identifier,
Expand All @@ -289,7 +292,7 @@ def _annotate_image_or_frame( # noqa: C901, PLR0912, PLR0915
organization_id=context.organization_id,
workspace_id=context.workspace_id,
project_id=context.project_id,
dataset_id=context.dataset_id,
dataset_id=dataset_id,
video_id=media_details.id,
frame_index=frame_index,
create_video_frame_annotation_request=CreateVideoFrameAnnotationRequest(
Expand All @@ -302,17 +305,19 @@ def _annotate_image_or_frame( # noqa: C901, PLR0912, PLR0915


def _get_image_or_frame_annotations(
context: Context, media_type: str, media_name: str, frame_index: int | None = None
context: Context, media_type: str, media_name: str, frame_index: int | None = None, dataset_id: str | None = None
) -> list[Annotation]:
annotations_api: AnnotationsApi = context.annotations_api
annotations: list[Annotation] = []
if dataset_id is None:
dataset_id = context.dataset_id

if media_type == "image":
ann_response = annotations_api.get_image_annotation(
organization_id=context.organization_id,
workspace_id=context.workspace_id,
project_id=context.project_id,
dataset_id=context.dataset_id,
dataset_id=dataset_id,
image_id=context._media_info_by_name[media_name].id,
annotation_id="latest",
)
Expand All @@ -321,7 +326,7 @@ def _get_image_or_frame_annotations(
organization_id=context.organization_id,
workspace_id=context.workspace_id,
project_id=context.project_id,
dataset_id=context.dataset_id,
dataset_id=dataset_id,
video_id=context._media_info_by_name[media_name].id,
frame_index=frame_index,
annotation_id="latest",
Expand All @@ -335,6 +340,19 @@ def _get_image_or_frame_annotations(
return annotations


@when("the user annotates the image '{image_name}' in dataset '{dataset_name}' with label '{label_name}'")
def step_when_user_annotates_image_in_dataset_with_custom_label(
context: Context, image_name: str, dataset_name: str, label_name: str
) -> None:
_annotate_image_or_frame(
context=context,
media_type="image",
media_name=image_name,
labels=[label_name],
dataset_id=context._dataset_info_by_name[dataset_name].id,
)


@when("the user annotates the image '{image_name}' with label '{label_name}'")
def step_when_user_annotates_image_with_custom_label(context: Context, image_name: str, label_name: str) -> None:
_annotate_image_or_frame(context=context, media_type="image", media_name=image_name, labels=[label_name])
Expand Down Expand Up @@ -372,6 +390,43 @@ def step_when_user_tries_annotating_image_with_custom_labels(
context.exception = e


@then("the image '{image_name}' in dataset '{dataset_name}' has labels '{raw_expected_label_names}'")
def step_then_image_in_dataset_has_expected_labels(
context: Context, image_name: str, dataset_name: str, raw_expected_label_names: str
) -> None:
annotations = _get_image_or_frame_annotations(
context=context,
media_type="image",
media_name=image_name,
dataset_id=context._dataset_info_by_name[dataset_name].id,
)

expected_label_names = raw_expected_label_names.split(", ")

label_info_by_id = {label.id: label for label in context.label_info_by_name.values()}
found_label_names = [
label_info_by_id[label_id].name for annotation in annotations for label_id in annotation.label_ids
]
expected_label_names_freq = Counter(expected_label_names)
found_label_names_freq = Counter(found_label_names)
assert expected_label_names_freq == found_label_names_freq, (
f"Expected to find labels with the respective frequency: {expected_label_names_freq}, "
f"found instead: {found_label_names_freq}"
)


@then("the image '{image_name}' in dataset '{dataset_name}' has label '{expected_label_name}'")
def step_then_image_in_dataset_has_expected_label(
context: Context, image_name: str, dataset_name: str, expected_label_name: str
) -> None:
step_then_image_in_dataset_has_expected_labels(
context=context,
image_name=image_name,
dataset_name=dataset_name,
raw_expected_label_names=expected_label_name,
)


@then("the image '{image_name}' has labels '{raw_expected_label_names}'")
def step_then_image_has_expected_labels(context: Context, image_name: str, raw_expected_label_names: str) -> None:
annotations = _get_image_or_frame_annotations(context=context, media_type="image", media_name=image_name)
Expand Down
44 changes: 41 additions & 3 deletions interactive_ai/tests/e2e/features/steps/job_monitoring.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
import time
from typing import TYPE_CHECKING

from behave import then
from behave import step
from behave.runner import Context
from static_definitions import JobState, JobType

Expand All @@ -29,7 +29,12 @@
logger = logging.getLogger(__name__)


@then("a job of type '{job_type:w}' is scheduled")
@step("the user waits for {seconds:d} seconds")
def step_wait_seconds(seconds: int) -> None:
time.sleep(seconds)


@step("a job of type '{job_type:w}' is scheduled")
def step_then_job_scheduled(context: Context, job_type: str) -> None:
"""Asserts that the "job_type" is scheduled"""
jobs_api: JobsApi = context.jobs_api
Expand All @@ -45,7 +50,40 @@ def step_then_job_scheduled(context: Context, job_type: str) -> None:
assert found_job_type == expected_job_type, f"Expected job type {expected_job_type}, but found {found_job_type}"


@then("the job completes successfully within {job_timeout:d} minutes")
@step("a job of type '{job_type:w}' is running")
def step_then_job_running(context: Context, job_type: str) -> None:
"""Asserts that the "job_type" is running"""
jobs_api: JobsApi = context.jobs_api
expected_job_type = JobType(job_type)

for _ in range(20):
context.job_info = jobs_api.get_job(
organization_id=context.organization_id,
workspace_id=context.workspace_id,
job_id=context.job_id,
)
if context.job_info.actual_instance.state == JobState.RUNNING:
break
time.sleep(1)
else:
raise RuntimeError(f"A job of type '{job_type}' is not running within 20 seconds")

found_job_type = JobType(context.job_info.actual_instance.type)
assert found_job_type == expected_job_type, f"Expected job type {expected_job_type}, but found {found_job_type}"


@step("the user cancels the job")
def step_when_user_cancels_job(context: Context) -> None:
"""Cancels the job"""
jobs_api: JobsApi = context.jobs_api
jobs_api.cancel_job(
organization_id=context.organization_id,
workspace_id=context.workspace_id,
job_id=context.job_id,
)


@step("the job completes successfully within {job_timeout:d} minutes")
def step_then_job_finishes_before_timeout(context: Context, job_timeout: int) -> None:
"""
Asserts that the jobs finishes within the passed "job_timeout". If not, raises a TimeoutError.
Expand Down
32 changes: 25 additions & 7 deletions interactive_ai/tests/e2e/features/steps/media_management.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@

import cv2
import numpy as np
from behave import given, when
from behave import given, then, when
from behave.runner import Context

if TYPE_CHECKING:
Expand All @@ -28,7 +28,7 @@
logger = logging.getLogger(__name__)


def _create_and_upload_random_image(context: Context, image_name: str) -> None:
def _create_and_upload_random_image(context: Context, image_name: str, dataset_id: str) -> None:
random_image = np.random.randint(0, 256, (100, 100, 3), dtype=np.uint8)
img = Image.fromarray(random_image)

Expand All @@ -42,15 +42,15 @@ def _create_and_upload_random_image(context: Context, image_name: str) -> None:
organization_id=context.organization_id,
workspace_id=context.workspace_id,
project_id=context.project_id,
dataset_id=context.dataset_id,
dataset_id=dataset_id,
file=image_path, # file.read(),
_headers={"Content-Disposition": f'form-data; name="file"; filename="{image_name}"'},
)

context._media_info_by_name[image_name] = upload_image_response


def _create_and_upload_random_video(context: Context, video_name: str) -> None:
def _create_and_upload_random_video(context: Context, video_name: str, dataset_id: str) -> None:
with tempfile.TemporaryDirectory() as temp_dir:
video_path = f"{temp_dir}/{video_name}"

Expand All @@ -71,7 +71,7 @@ def _create_and_upload_random_video(context: Context, video_name: str) -> None:
organization_id=context.organization_id,
workspace_id=context.workspace_id,
project_id=context.project_id,
dataset_id=context.dataset_id,
dataset_id=dataset_id,
file=video_path,
_headers={"Content-Disposition": f'form-data; name="file"; filename="{video_name}"'},
)
Expand All @@ -81,12 +81,12 @@ def _create_and_upload_random_video(context: Context, video_name: str) -> None:

@given("an image called '{image_name}'")
def step_given_image_with_custom_name(context: Context, image_name: str) -> None:
_create_and_upload_random_image(context=context, image_name=image_name)
_create_and_upload_random_image(context=context, image_name=image_name, dataset_id=context.dataset_id)


@given("a video called '{video_name}'")
def step_given_video_with_custom_name(context: Context, video_name: str) -> None:
_create_and_upload_random_video(context=context, video_name=video_name)
_create_and_upload_random_video(context=context, video_name=video_name, dataset_id=context.dataset_id)


@when("the user loads the image '{image_name}'")
Expand All @@ -101,9 +101,27 @@ def step_when_user_loads_image(context: Context, image_name: str) -> None:
)


@when("the user uploads an image called '{image_name}' to dataset '{dataset_name}'")
def step_when_user_uploads_image_to_dataset(context: Context, image_name: str, dataset_name: str) -> None:
dataset_id = context._dataset_info_by_name[dataset_name].id
_create_and_upload_random_image(context=context, image_name=image_name, dataset_id=dataset_id)


@when("the user tries to load the image '{image_name}'")
def step_when_user_tries_loading_image(context: Context, image_name: str) -> None:
try:
step_when_user_loads_image(context=context, image_name=image_name)
except Exception as e:
context.exception = e


@then("the dataset '{dataset_name}' contains an image called '{image_name}'")
def step_then_dataset_contains_image(context: Context, dataset_name: str, image_name: str) -> None:
media_api: MediaApi = context.media_api
context._media_info_by_name[image_name] = media_api.get_image_detail(
organization_id=context.organization_id,
workspace_id=context.workspace_id,
project_id=context.project_id,
dataset_id=context._dataset_info_by_name[dataset_name].id,
image_id=context._media_info_by_name[image_name].id,
)
11 changes: 11 additions & 0 deletions interactive_ai/tests/e2e/features/steps/project_management.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -271,6 +271,9 @@ def _create_project_from_scratch( # noqa: C901
context.project_info = create_project_response # store the whole response for convenience in other steps
context.project_id = create_project_response.id
context.dataset_id = create_project_response.datasets[0].id
context._dataset_info_by_name = {
dataset_info.name: dataset_info for dataset_info in create_project_response.datasets
}
context.label_info_by_name = { # dict[str, CreateProject201ResponsePipelineTasksInnerLabelsInner]
label.name: label for task in create_project_response.pipeline.tasks if task.labels for label in task.labels
}
Expand Down Expand Up @@ -354,6 +357,14 @@ def step_when_user_deletes_project(context: Context) -> None:
)


@when("the user tries to delete the project")
def step_when_user_tries_to_delete_project(context: Context) -> None:
try:
step_when_user_deletes_project(context=context)
except Exception as e:
context.exception = e


@when("the user loads the project")
def step_when_user_loads_project(context: Context) -> None:
projects_api: ProjectsApi = context.projects_api
Expand Down
4 changes: 2 additions & 2 deletions interactive_ai/tests/e2e/features/steps/training.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
# with no express or implied warranties, other than those that are expressly stated
# in the License.

from behave import then, when
from behave import step, then, when
from behave.runner import Context
from geti_client import ModelsApi, TrainJob, TrainModelRequest
from static_definitions import TaskType
Expand Down Expand Up @@ -49,7 +49,7 @@ def _train(
context.job_id = train_response.job_id


@when("the user requests to train a model")
@step("the user requests to train a model")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Imo it's better to avoid generic @step and explicitly mark each step as either a given, a when or a then, because they have different semantics that also reflect in the implementation.
Step "the user requests to train a model" is an action performed by the user, so it should be a @when.
Conversely, @given describes a state (namely, the state of the system at the beginning of the test): such state may be the result of user actions, whereas an action is not a state on its own therefore it doesn't fit the @given semantics.

I suggest to rewrite the following part of your test:

Given an annotated project of type 'detection'
And the user requests to train a model
And a job of type 'train' is running
...

as:

Given an annotated project of type 'detection'
When the user requests to train a model
Then a job of type 'train' is running
...

It doesn't affect readability too much, while it retains a clear distinction between given/when/then steps.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I disagree, the Given keyword indicates that the step is doing something that is a prerequisite for what we want to test, the When is a keyword indicating an action that we want to test, the Then keyword indicates a sentence that tests if the expected behavior of the When action is achieved. Other than these meanings there is no real difference between these steps from a technical point of view, they are all run in the same way and should all pass in the same way.

In this scenario we do not test whether or not the user requests to train a model actually starts a job that trains a model, we only need a job running to lock the project for the deletion. Therefore this step should be called with the Given keyword to indicate to the reader this step is necessary to set the system in a state that we need to check the proper behavior of the deletion of projects.

One limitation of behave is that we cannot mark a step with two different decorators, with two different sentences, @give("the user requested to train a model") vs @when("the user requests to train a model") would be nicer.

def step_when_user_trains_model(context: Context):
_train(context=context)

Expand Down
Loading