Skip to content

[Bug]: Nexmark Python fails to fetch metrics on Flink and Spark runners #24737

Open
@apilloud

Description

@apilloud

What happened?

This works on DirectRunner and Dataflow...

./gradlew --continue -Dorg.gradle.jvmargs=-Xms2g -Dorg.gradle.jvmargs=-Xmx6g -Dorg.gradle.vfs.watch=false -Pdocker-pull-licenses -Pnexmark.runner=:runners:direct-java "-Pnexmark.args=--runner=DirectRunner --generateEventFilePathPrefix=gs://temp-storage-for-perf-tests/nexmark/eventFiles/jenkins-beam_PostCommit_Python_Nexmark_Flink_PR-5/query0- --query=0 --numEvents=100000 --manageResources=false --monitorJobs=true" :sdks:java:testing:nexmark:run
./gradlew -Dorg.gradle.jvmargs=-Xms2g -Dorg.gradle.jvmargs=-Xmx6g -Dorg.gradle.vfs.watch=false -Pdocker-pull-licenses "-Pnexmark.args= --monitor_jobs --big_query_table=nexmark --big_query_dataset=nexmark_PRs --project=apache-beam-testing --resource_name_mode=QUERY_RUNNER_AND_MODE --export_summary_to_big_query --temp_location=gs://temp-storage-for-perf-tests/nexmark --influx_database=beam_test_metrics --influx_host=http://10.128.0.96:8086/ --base_influx_measurement=nexmark_PRs --export_summary_to_influx_db --influx_retention_policy=forever --suite=SMOKE --stream_timeout=60 --runner=FlinkRunner --query=0 --num_events=100000 --input=gs://temp-storage-for-perf-tests/nexmark/eventFiles/jenkins-beam_PostCommit_Python_Nexmark_Flink_PR-5/query0-\*" :sdks:python:apache_beam:testing:benchmarks:nexmark:run

14:51:25 Traceback (most recent call last):
14:51:25 File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
14:51:25 return _run_code(code, main_globals, None,
14:51:25 File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
14:51:25 exec(code, run_globals)
14:51:25 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Nexmark_Flink_PR/src/sdks/python/apache_beam/testing/benchmarks/nexmark/nexmark_launcher.py", line 528, in
14:51:25 launcher.run()
14:51:25 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Nexmark_Flink_PR/src/sdks/python/apache_beam/testing/benchmarks/nexmark/nexmark_launcher.py", line 513, in run
14:51:25 self.run_query(
14:51:25 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Nexmark_Flink_PR/src/sdks/python/apache_beam/testing/benchmarks/nexmark/nexmark_launcher.py", line 312, in run_query
14:51:25 self.publish_performance_influxdb(query_num, perf)
14:51:25 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Nexmark_Flink_PR/src/sdks/python/apache_beam/testing/benchmarks/nexmark/nexmark_launcher.py", line 400, in publish_performance_influxdb
14:51:25 mt = ','.join([measurement] + [k + "=" + v for k, v in tags.items()])
14:51:25 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Nexmark_Flink_PR/src/sdks/python/apache_beam/testing/benchmarks/nexmark/nexmark_launcher.py", line 400, in
14:51:25 mt = ','.join([measurement] + [k + "=" + v for k, v in tags.items()])
14:51:25 TypeError: can only concatenate str (not "NoneType") to str

Issue Priority

Priority: 3 (minor)

Issue Components

  • Component: Python SDK
  • Component: Java SDK
  • Component: Go SDK
  • Component: Typescript SDK
  • Component: IO connector
  • Component: Beam examples
  • Component: Beam playground
  • Component: Beam katas
  • Component: Website
  • Component: Spark Runner
  • Component: Flink Runner
  • Component: Samza Runner
  • Component: Twister2 Runner
  • Component: Hazelcast Jet Runner
  • Component: Google Cloud Dataflow Runner

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions