Bump Spark Cassandra Connector to 3.4.1 #142

vlsi · 2025-02-05T13:25:30Z

Which problem is this PR solving?

[Bug]: New Docker-Image breaks jaeger-operator spark-dependencies-jobs #138

Description of the changes

Previously, Spark Cassandra Connector and Spark Core was using metrics-core 3.x and 4.x respectively. Bumping Cassandra Connector integrates a newer Cassandra Java Driver, so it does not interfere with spark's metrics-core version.

How was this change tested?

CassandraDependenciesJobTest does not seem to work due to renovate-bot/jaegertracing-_-jaeger@b34cbf3

Checklist

I have read https://github.com/jaegertracing/jaeger/blob/master/CONTRIBUTING_GUIDELINES.md
I have signed all commits
I have added unit tests for the new functionality
I have run lint and test steps successfully
- for jaeger: make lint test
- for jaeger-ui: npm run lint and npm run test

yurishkuro · 2025-02-05T14:19:22Z

CassandraDependenciesJobTest does not seem to work due to renovate-bot/jaegertracing-_-jaeger@b34cbf3

Can we add that job to the CI before making changes to Cassandra related code?

vlsi · 2025-02-05T15:47:39Z

Can we add that job to the CI before making changes to Cassandra related code?

@yurishkuro , could you please clarify what exactly do you mean?

The current CI fails, see: https://github.com/jaegertracing/spark-dependencies/actions/runs/13158395388/job/36723797004?pr=142#step:5:29234

due to error: Keyspace 'jaeger_v1_dc1' does not exist

I believe the root cause is the same as in renovate-bot/jaegertracing-_-jaeger@b34cbf3 since I get the same no viable alternative at input '{' (CREATE TYPE IF NOT in jaeger-cassandra-schema logs.

vlsi · 2025-02-05T16:08:42Z

Here's the log for the main branch of spark-dependencies:

Checking if Cassandra is up at cassandra:9042.
WARNING: cqlsh was built against 5.0.2, but this server is 3.11.18.  All features may not work!

WARN: DESCRIBE|DESC was moved to server side in Cassandra 4.0. As a consequence DESCRIBE|DESC will not work in cqlsh '6.2.0' connected to Cassandra '3.11.18', the version that you are connected to. DESCRIBE does not exist server side prior Cassandra 4.0.
Cassandra connection established.
Cassandra version detected: 
3
Generating the schema for the keyspace jaeger_v1_dc1 and datacenter dc1.
Using template file /cassandra-schema/v004-go-tmpl.cql.tmpl with parameters:
    mode = test
    datacenter = dc1
    keyspace = jaeger_v1_dc1
    replication = {'class': 'SimpleStrategy', 'replication_factor': '1'}
    trace_ttl = 172800
    dependencies_ttl = 0
    compaction_window_size = 96
    compaction_window_unit = MINUTES
<stdin>:2:SyntaxException: line 1:30 no viable alternative at input '{' (CREATE KEYSPACE IF NOT EXISTS [{]...)
<stdin>:12:SyntaxException: line 1:26 no viable alternative at input '{' (CREATE TYPE IF NOT EXISTS [{]...)
<stdin>:17:SyntaxException: line 1:26 no viable alternative at input '{' (CREATE TYPE IF NOT EXISTS [{]...)
<stdin>:23:SyntaxException: line 1:26 no viable alternative at input '{' (CREATE TYPE IF NOT EXISTS [{]...)
<stdin>:28:SyntaxException: line 1:26 no viable alternative at input '{' (CREATE TYPE IF NOT EXISTS [{]...)
<stdin>:52:SyntaxException: line 1:27 no viable alternative at input '{' (CREATE TABLE IF NOT EXISTS [{]...)
<stdin>:65:SyntaxException: line 1:27 no viable alternative at input '{' (CREATE TABLE IF NOT EXISTS [{]...)
<stdin>:80:SyntaxException: line 1:27 no viable alternative at input '{' (CREATE TABLE IF NOT EXISTS [{]...)
<stdin>:96:SyntaxException: line 1:27 no viable alternative at input '{' (CREATE TABLE IF NOT EXISTS [{]...)
<stdin>:112:SyntaxException: line 1:27 no viable alternative at input '{' (CREATE TABLE IF NOT EXISTS [{]...)
<stdin>:130:SyntaxException: line 1:27 no viable alternative at input '{' (CREATE TABLE IF NOT EXISTS [{]...)
<stdin>:149:SyntaxException: line 1:27 no viable alternative at input '{' (CREATE TABLE IF NOT EXISTS [{]...)
<stdin>:156:SyntaxException: line 1:26 no viable alternative at input '{' (CREATE TYPE IF NOT EXISTS [{]...)
<stdin>:169:SyntaxException: line 1:27 no viable alternative at input '{' (CREATE TABLE IF NOT EXISTS [{]...)
<stdin>:176:SyntaxException: line 1:27 no viable alternative at input '{' (CREATE TABLE IF NOT EXISTS [{]...)
<stdin>:184:SyntaxException: line 1:27 no viable alternative at input '{' (CREATE TABLE IF NOT EXISTS [{]...)
<stdin>:190:SyntaxException: line 1:27 no viable alternative at input '{' (CREATE TABLE IF NOT EXISTS [{]...)
WARNING: cqlsh was built against 5.0.2, but this server is 3.11.18.  All features may not work!

yurishkuro · 2025-02-05T16:13:13Z

it looks like we may need to upgrade the test to use Cassandra v4 (v3 is EOL anyway)

vlsi · 2025-02-05T16:15:12Z

it looks like we may need to upgrade the test to use Cassandra v4

Frankly, at this point the culprit seems to be that jaeger-cassandra-schema is still trying to use v004-go-tmpl.cql.tmpl template. Altering Cassandra version within the test does not change much.

yurishkuro · 2025-02-05T16:26:38Z

:2:SyntaxException: line 1:30 no viable alternative at input '{' (CREATE KEYSPACE IF NOT EXISTS [{]...)

Were you able to see these errors in the GitHub runner output, or in a local run only?

It seems some parameter to the schema is now required.

vlsi · 2025-02-05T16:28:01Z

Were you able to see these errors in the GitHub runner output, or in a local run only?

Sure I can do it in GitHub CI if you like.

It seems some parameter to the schema is now required.

No I bet "cassandra version autodetect" does not properly work, so it defaults to "the latest .cql.tmpl file in the image" which happends to be v004-go-tmpl.cql.tmpl

vlsi · 2025-02-05T16:30:06Z

Note the output:

Cassandra version detected: 
3
Generating the schema for the keyspace jaeger_v1_dc1 and datacenter dc1.

I believe there's an unexpected newline before 3.
I guess https://github.com/renovate-bot/jaegertracing-_-jaeger/blob/8afbf86a4a359fa535b201e4601cab2d9a0c9881/internal/storage/v1/cassandra/schema/docker.sh#L48-L58 somehow produces an unexpected newline that should be trimmed at least.

yurishkuro · 2025-02-05T16:30:16Z

I couldn't find similar errors in the CI run, it would be good to see them there.

yurishkuro · 2025-02-05T16:34:11Z

version 3 is correct. v4 template is also correct. However v004-go-tmpl.cql.tmpl is a built-in template, meaning the collector is trying to automatically initialize Cassandra instead of expecting an external script to be run (it's a recent change, jaegertracing/jaeger#5922)

Previously, Spark Cassandra Connector and Spark Core was using metrics-core 3.x and 4.x respectively. Bumping Cassandra Connector integrates a newer Cassandra Java Driver, so it does not interfere with spark's metrics-core version. Fixes jaegertracing#138 Signed-off-by: Vladimir Sitnikov <[email protected]>

vlsi · 2025-02-11T08:05:17Z

Unfortunately, this does not fix Docker image as it still fails as follows:

Exception in thread "main" java.io.IOException: Failed to open native connection to Cassandra at {demo-dc1-service.k8ssandra-operator.svc.cluster.local:9042} :: Can't find class DefaultMetricsFactory (specified by advanced.metrics.factory.class)
	at com.datastax.spark.connector.cql.CassandraConnector$.createSession(CassandraConnector.scala:173)
	at com.datastax.spark.connector.cql.CassandraConnector$.$anonfun$sessionCache$1(CassandraConnector.scala:161)
	at com.datastax.spark.connector.cql.RefCountedCache.createNewValueAndKeys(RefCountedCache.scala:32)
	at com.datastax.spark.connector.cql.RefCountedCache.syncAcquire(RefCountedCache.scala:69)
	at com.datastax.spark.connector.cql.RefCountedCache.acquire(RefCountedCache.scala:57)
	at com.datastax.spark.connector.cql.CassandraConnector.openSession(CassandraConnector.scala:81)
	at com.datastax.spark.connector.cql.CassandraConnector.withSessionDo(CassandraConnector.scala:103)
	at com.datastax.spark.connector.util.package$.tableFromCassandra(package.scala:30)
	at com.datastax.spark.connector.rdd.CassandraTableRowReaderProvider.tableDef(CassandraTableRowReaderProvider.scala:51)
	at com.datastax.spark.connector.rdd.CassandraTableRowReaderProvider.tableDef$(CassandraTableRowReaderProvider.scala:51)
	at com.datastax.spark.connector.rdd.CassandraTableScanRDD.tableDef$lzycompute(CassandraTableScanRDD.scala:59)
	at com.datastax.spark.connector.rdd.CassandraTableScanRDD.tableDef(CassandraTableScanRDD.scala:59)
	at com.datastax.spark.connector.rdd.CassandraTableRowReaderProvider.verify(CassandraTableRowReaderProvider.scala:113)
	at com.datastax.spark.connector.rdd.CassandraTableRowReaderProvider.verify$(CassandraTableRowReaderProvider.scala:112)
	at com.datastax.spark.connector.rdd.CassandraTableScanRDD.verify(CassandraTableScanRDD.scala:59)
	at com.datastax.spark.connector.rdd.CassandraTableScanRDD.getPartitions(CassandraTableScanRDD.scala:261)
	at org.apache.spark.rdd.RDD.$anonfun$partitions$2(RDD.scala:294)
	at scala.Option.getOrElse(Option.scala:189)
	at org.apache.spark.rdd.RDD.partitions(RDD.scala:290)
	at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
	at org.apache.spark.rdd.RDD.$anonfun$partitions$2(RDD.scala:294)
	at scala.Option.getOrElse(Option.scala:189)
	at org.apache.spark.rdd.RDD.partitions(RDD.scala:290)
	at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
	at org.apache.spark.rdd.RDD.$anonfun$partitions$2(RDD.scala:294)
	at scala.Option.getOrElse(Option.scala:189)
	at org.apache.spark.rdd.RDD.partitions(RDD.scala:290)
	at org.apache.spark.Partitioner$.$anonfun$defaultPartitioner$4(Partitioner.scala:78)
	at org.apache.spark.Partitioner$.$anonfun$defaultPartitioner$4$adapted(Partitioner.scala:78)
	at scala.collection.immutable.List.map(List.scala:293)
	at org.apache.spark.Partitioner$.defaultPartitioner(Partitioner.scala:78)
	at org.apache.spark.rdd.PairRDDFunctions.$anonfun$groupByKey$6(PairRDDFunctions.scala:636)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
	at org.apache.spark.rdd.RDD.withScope(RDD.scala:410)
	at org.apache.spark.rdd.PairRDDFunctions.groupByKey(PairRDDFunctions.scala:636)
	at org.apache.spark.api.java.JavaPairRDD.groupByKey(JavaPairRDD.scala:561)
	at io.jaegertracing.spark.dependencies.cassandra.CassandraDependenciesJob.run(CassandraDependenciesJob.java:185)
	at io.jaegertracing.spark.dependencies.DependenciesSparkJob.run(DependenciesSparkJob.java:60)
	at io.jaegertracing.spark.dependencies.DependenciesSparkJob.main(DependenciesSparkJob.java:40)
Caused by: java.lang.IllegalArgumentException: Can't find class DefaultMetricsFactory (specified by advanced.metrics.factory.class)
	at com.datastax.oss.driver.internal.core.util.Reflection.resolveClass(Reflection.java:302)
	at com.datastax.oss.driver.internal.core.util.Reflection.buildFromConfig(Reflection.java:235)
	at com.datastax.oss.driver.internal.core.util.Reflection.buildFromConfig(Reflection.java:110)
	at com.datastax.oss.driver.internal.core.context.DefaultDriverContext.buildMetricsFactory(DefaultDriverContext.java:548)
	at com.datastax.oss.driver.internal.core.util.concurrent.LazyReference.get(LazyReference.java:55)
	at com.datastax.oss.driver.internal.core.context.DefaultDriverContext.getMetricsFactory(DefaultDriverContext.java:941)
	at com.datastax.oss.driver.internal.core.session.DefaultSession.<init>(DefaultSession.java:123)
	at com.datastax.oss.driver.internal.core.session.DefaultSession.init(DefaultSession.java:88)
	at com.datastax.oss.driver.api.core.session.SessionBuilder.buildDefaultSessionAsync(SessionBuilder.java:903)
	at com.datastax.oss.driver.api.core.session.SessionBuilder.buildAsync(SessionBuilder.java:817)
	at com.datastax.oss.driver.api.core.session.SessionBuilder.build(SessionBuilder.java:835)
	at com.datastax.spark.connector.cql.DefaultConnectionFactory$.createSession(CassandraConnectionFactory.scala:143)
	at com.datastax.spark.connector.cql.CassandraConnector$.createSession(CassandraConnector.scala:167)
	... 39 more

vlsi · 2025-02-11T08:29:30Z

I believe the root cause is that minimizeJar=true causes the removal of the class:

spark-dependencies/jaeger-spark-dependencies/pom.xml

Line 61 in 3a4b095

If I apply minimizeJar=false, then the jar grows from 189M to 220M, however, the class com/datastax/oss/driver/internal/core/metrics/DefaultMetricsFactory.class gets included into the jar.

@yurishkuro , what do you think of deactivating minimizeJar?

Just for the reference, DefaultMetricsFactory is mentioned in Cassandra Driver's reference.conf: https://github.com/apache/cassandra-java-driver/blob/90612f6758eb0f0ba964daf054f397a47a90a736/core/src/main/resources/reference.conf#L1408, and it looks like there might be many more such references.

We can try adding filter-artifact-include=**, however, there are many modules in Cassandra Java Driver, so it might look like

                <filter>
                  <artifact>com.datastax.oss:*</artifact>
                  <includes>
                    <include>**</include>
                  </includes>
                </filter>

The resulting jar becomes 191M. However, I am not sure if minimizeJar yields significant advantage as it makes the maintenance complicated. It looks like we resort to effectively "keep" the classes for all the dependencies.

yurishkuro · 2025-02-11T12:39:29Z

Agreed. But I also wonder why didn't our CI catch that, probably because we don't do a proper e2e test running the job's code from a container.

yurishkuro mentioned this pull request Feb 5, 2025

Use Cassandra 4.1 in integration test #144

Merged

yurishkuro mentioned this pull request Feb 5, 2025

Pin Jaeger version in Cassandra test #146

Merged

vlsi force-pushed the bump_cassandra branch from 58268a0 to b29c43f Compare February 5, 2025 19:13

yurishkuro approved these changes Feb 5, 2025

View reviewed changes

vlsi force-pushed the bump_cassandra branch 2 times, most recently from 7980971 to 02f6bb1 Compare February 5, 2025 19:33

vlsi force-pushed the bump_cassandra branch from 02f6bb1 to c488bfa Compare February 5, 2025 19:41

Merge branch 'main' into bump_cassandra

50d4722

yurishkuro approved these changes Feb 5, 2025

View reviewed changes

yurishkuro enabled auto-merge (squash) February 5, 2025 23:41

yurishkuro merged commit 3b194ac into jaegertracing:main Feb 5, 2025
4 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bump Spark Cassandra Connector to 3.4.1 #142

Bump Spark Cassandra Connector to 3.4.1 #142

vlsi commented Feb 5, 2025

yurishkuro commented Feb 5, 2025

vlsi commented Feb 5, 2025

vlsi commented Feb 5, 2025

yurishkuro commented Feb 5, 2025

vlsi commented Feb 5, 2025

yurishkuro commented Feb 5, 2025

vlsi commented Feb 5, 2025

vlsi commented Feb 5, 2025

yurishkuro commented Feb 5, 2025

yurishkuro commented Feb 5, 2025

vlsi commented Feb 11, 2025

vlsi commented Feb 11, 2025

yurishkuro commented Feb 11, 2025

Bump Spark Cassandra Connector to 3.4.1 #142

Bump Spark Cassandra Connector to 3.4.1 #142

Conversation

vlsi commented Feb 5, 2025

Which problem is this PR solving?

Description of the changes

How was this change tested?

Checklist

yurishkuro commented Feb 5, 2025

vlsi commented Feb 5, 2025

vlsi commented Feb 5, 2025

yurishkuro commented Feb 5, 2025

vlsi commented Feb 5, 2025

yurishkuro commented Feb 5, 2025

vlsi commented Feb 5, 2025

vlsi commented Feb 5, 2025

yurishkuro commented Feb 5, 2025

yurishkuro commented Feb 5, 2025

vlsi commented Feb 11, 2025

vlsi commented Feb 11, 2025

yurishkuro commented Feb 11, 2025