Skip to content

Live data preview and node transition throughput on Flink minicluster #8047

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 49 commits into from
May 27, 2025

Conversation

mgoworko
Copy link
Contributor

@mgoworko mgoworko commented May 7, 2025

Describe your changes

  • added listener, that collects live data processed by the scenario
  • the data is stored in new RingBuffer util based on LinkedHashMap with automatic cleaning of old data, samples corresponding to the last N contextIds are stored
  • also added node transition frequency calculation - node transition counts are stored in buckets for each epoch second and calculated in time windows

The data can be fetched using /api/liveData/{scenario_name}

At the moment only for Flink minicluster, because it is a part of the same JVM and we just access the data without synchronizing it with external data storage.

Checklist before merge

  • Related issue ID is placed at the beginning of PR title in [brackets] (can be GH issue or Nu Jira issue)
  • Code is cleaned from temporary changes and commented out lines
  • Parts of the code that are not easy to understand are documented in the code
  • Changes are covered by automated tests
  • Showcase in dev-application.conf added to demonstrate the feature
  • Documentation added or updated
  • Added entry in Changelog.md describing the change from the perspective of a public distribution user
  • Added MigrationGuide.md entry in the appropriate subcategory if introducing a breaking change
  • Verify that PR will be squashed during merge

mgoworko added 2 commits May 12, 2025 15:48
# Conflicts:
#	engine/flink/executor/src/main/scala/pl/touk/nussknacker/engine/process/compiler/StubbedFlinkProcessCompilerDataFactory.scala
@mgoworko mgoworko marked this pull request as ready for review May 13, 2025 09:33
mgoworko added 3 commits May 13, 2025 11:34
# Conflicts:
#	components-api/src/main/scala/pl/touk/nussknacker/engine/ModelConfig.scala
#	engine/flink/executor/src/main/scala/pl/touk/nussknacker/engine/process/compiler/FlinkProcessCompilerDataFactory.scala
@mgoworko mgoworko changed the title Draft, do not review - Flink minicluster live data preview In progress - Flink minicluster live data preview May 13, 2025
@github-actions github-actions bot added the docs label May 18, 2025
@mgoworko mgoworko changed the title In progress - Flink minicluster live data preview Live data preview and node transition frequency on Flink minicluster May 18, 2025
@github-actions github-actions bot added client client main fe ui labels May 18, 2025
@TouK TouK deleted a comment from github-actions bot May 19, 2025
mgoworko added 7 commits May 23, 2025 12:02
# Conflicts:
#	designer/server/src/main/scala/pl/touk/nussknacker/ui/api/ScenarioTestingApiHttpService.scala
#	designer/server/src/main/scala/pl/touk/nussknacker/ui/api/description/scenarioTesting/Dtos.scala
#	designer/server/src/main/scala/pl/touk/nussknacker/ui/api/description/scenarioTesting/ScenarioTestingApiEndpoints.scala
#	designer/server/src/test/scala/pl/touk/nussknacker/ui/api/testing/ScenarioTestingApiHttpServiceSpec.scala
#	designer/server/src/test/scala/pl/touk/nussknacker/ui/api/testing/SchemalessKafkaJsonTypeTests.scala
#	docs-internal/api/nu-designer-openapi.yaml
#	docs/Changelog.md
# Conflicts:
#	components/openapi/src/main/scala/pl/touk/nussknacker/openapi/enrichers/OpenAPIEnricherFactory.scala
@mgoworko mgoworko requested a review from arkadius May 26, 2025 08:16
@mgoworko mgoworko changed the title Live data preview and node transition frequency on Flink minicluster Live data preview and node transition throughput on Flink minicluster May 26, 2025
@mgoworko mgoworko requested a review from arkadius May 26, 2025 18:45
@mgoworko mgoworko merged commit 8c683e9 into staging May 27, 2025
29 of 31 checks passed
@mgoworko mgoworko deleted the live-data branch May 27, 2025 13:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants