Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bump Spark Cassandra Connector to 3.4.1 #142

Merged
merged 2 commits into from
Feb 5, 2025

Conversation

vlsi
Copy link
Contributor

@vlsi vlsi commented Feb 5, 2025

Which problem is this PR solving?

Description of the changes

Previously, Spark Cassandra Connector and Spark Core was using metrics-core 3.x and 4.x respectively. Bumping Cassandra Connector integrates a newer Cassandra Java Driver, so it does not interfere with spark's metrics-core version.

How was this change tested?

CassandraDependenciesJobTest does not seem to work due to renovate-bot/jaegertracing-_-jaeger@b34cbf3

Checklist

@yurishkuro
Copy link
Member

CassandraDependenciesJobTest does not seem to work due to renovate-bot/jaegertracing-_-jaeger@b34cbf3

Can we add that job to the CI before making changes to Cassandra related code?

@vlsi
Copy link
Contributor Author

vlsi commented Feb 5, 2025

Can we add that job to the CI before making changes to Cassandra related code?

@yurishkuro , could you please clarify what exactly do you mean?

The current CI fails, see: https://github.com/jaegertracing/spark-dependencies/actions/runs/13158395388/job/36723797004?pr=142#step:5:29234

due to error: Keyspace 'jaeger_v1_dc1' does not exist

I believe the root cause is the same as in renovate-bot/jaegertracing-_-jaeger@b34cbf3 since I get the same no viable alternative at input '{' (CREATE TYPE IF NOT in jaeger-cassandra-schema logs.

@vlsi
Copy link
Contributor Author

vlsi commented Feb 5, 2025

Here's the log for the main branch of spark-dependencies:

Checking if Cassandra is up at cassandra:9042.
WARNING: cqlsh was built against 5.0.2, but this server is 3.11.18.  All features may not work!

WARN: DESCRIBE|DESC was moved to server side in Cassandra 4.0. As a consequence DESCRIBE|DESC will not work in cqlsh '6.2.0' connected to Cassandra '3.11.18', the version that you are connected to. DESCRIBE does not exist server side prior Cassandra 4.0.
Cassandra connection established.
Cassandra version detected: 
3
Generating the schema for the keyspace jaeger_v1_dc1 and datacenter dc1.
Using template file /cassandra-schema/v004-go-tmpl.cql.tmpl with parameters:
    mode = test
    datacenter = dc1
    keyspace = jaeger_v1_dc1
    replication = {'class': 'SimpleStrategy', 'replication_factor': '1'}
    trace_ttl = 172800
    dependencies_ttl = 0
    compaction_window_size = 96
    compaction_window_unit = MINUTES
<stdin>:2:SyntaxException: line 1:30 no viable alternative at input '{' (CREATE KEYSPACE IF NOT EXISTS [{]...)
<stdin>:12:SyntaxException: line 1:26 no viable alternative at input '{' (CREATE TYPE IF NOT EXISTS [{]...)
<stdin>:17:SyntaxException: line 1:26 no viable alternative at input '{' (CREATE TYPE IF NOT EXISTS [{]...)
<stdin>:23:SyntaxException: line 1:26 no viable alternative at input '{' (CREATE TYPE IF NOT EXISTS [{]...)
<stdin>:28:SyntaxException: line 1:26 no viable alternative at input '{' (CREATE TYPE IF NOT EXISTS [{]...)
<stdin>:52:SyntaxException: line 1:27 no viable alternative at input '{' (CREATE TABLE IF NOT EXISTS [{]...)
<stdin>:65:SyntaxException: line 1:27 no viable alternative at input '{' (CREATE TABLE IF NOT EXISTS [{]...)
<stdin>:80:SyntaxException: line 1:27 no viable alternative at input '{' (CREATE TABLE IF NOT EXISTS [{]...)
<stdin>:96:SyntaxException: line 1:27 no viable alternative at input '{' (CREATE TABLE IF NOT EXISTS [{]...)
<stdin>:112:SyntaxException: line 1:27 no viable alternative at input '{' (CREATE TABLE IF NOT EXISTS [{]...)
<stdin>:130:SyntaxException: line 1:27 no viable alternative at input '{' (CREATE TABLE IF NOT EXISTS [{]...)
<stdin>:149:SyntaxException: line 1:27 no viable alternative at input '{' (CREATE TABLE IF NOT EXISTS [{]...)
<stdin>:156:SyntaxException: line 1:26 no viable alternative at input '{' (CREATE TYPE IF NOT EXISTS [{]...)
<stdin>:169:SyntaxException: line 1:27 no viable alternative at input '{' (CREATE TABLE IF NOT EXISTS [{]...)
<stdin>:176:SyntaxException: line 1:27 no viable alternative at input '{' (CREATE TABLE IF NOT EXISTS [{]...)
<stdin>:184:SyntaxException: line 1:27 no viable alternative at input '{' (CREATE TABLE IF NOT EXISTS [{]...)
<stdin>:190:SyntaxException: line 1:27 no viable alternative at input '{' (CREATE TABLE IF NOT EXISTS [{]...)
WARNING: cqlsh was built against 5.0.2, but this server is 3.11.18.  All features may not work!

@yurishkuro
Copy link
Member

it looks like we may need to upgrade the test to use Cassandra v4 (v3 is EOL anyway)

@vlsi
Copy link
Contributor Author

vlsi commented Feb 5, 2025

it looks like we may need to upgrade the test to use Cassandra v4

Frankly, at this point the culprit seems to be that jaeger-cassandra-schema is still trying to use v004-go-tmpl.cql.tmpl template. Altering Cassandra version within the test does not change much.

@yurishkuro
Copy link
Member

:2:SyntaxException: line 1:30 no viable alternative at input '{' (CREATE KEYSPACE IF NOT EXISTS [{]...)

Were you able to see these errors in the GitHub runner output, or in a local run only?

It seems some parameter to the schema is now required.

@vlsi
Copy link
Contributor Author

vlsi commented Feb 5, 2025

Were you able to see these errors in the GitHub runner output, or in a local run only?

Sure I can do it in GitHub CI if you like.

It seems some parameter to the schema is now required.

No I bet "cassandra version autodetect" does not properly work, so it defaults to "the latest .cql.tmpl file in the image" which happends to be v004-go-tmpl.cql.tmpl

@vlsi
Copy link
Contributor Author

vlsi commented Feb 5, 2025

Note the output:

Cassandra version detected: 
3
Generating the schema for the keyspace jaeger_v1_dc1 and datacenter dc1.

I believe there's an unexpected newline before 3.
I guess https://github.com/renovate-bot/jaegertracing-_-jaeger/blob/8afbf86a4a359fa535b201e4601cab2d9a0c9881/internal/storage/v1/cassandra/schema/docker.sh#L48-L58 somehow produces an unexpected newline that should be trimmed at least.

@yurishkuro
Copy link
Member

I couldn't find similar errors in the CI run, it would be good to see them there.

@yurishkuro
Copy link
Member

version 3 is correct. v4 template is also correct. However v004-go-tmpl.cql.tmpl is a built-in template, meaning the collector is trying to automatically initialize Cassandra instead of expecting an external script to be run (it's a recent change, jaegertracing/jaeger#5922)

@vlsi vlsi force-pushed the bump_cassandra branch 2 times, most recently from 7980971 to 02f6bb1 Compare February 5, 2025 19:33
Previously, Spark Cassandra Connector and Spark Core was using metrics-core 3.x and 4.x respectively.
Bumping Cassandra Connector integrates a newer Cassandra Java Driver, so it does not interfere with
spark's metrics-core version.

Fixes jaegertracing#138

Signed-off-by: Vladimir Sitnikov <[email protected]>
@yurishkuro yurishkuro enabled auto-merge (squash) February 5, 2025 23:41
@yurishkuro yurishkuro merged commit 3b194ac into jaegertracing:main Feb 5, 2025
4 checks passed
@vlsi
Copy link
Contributor Author

vlsi commented Feb 11, 2025

Unfortunately, this does not fix Docker image as it still fails as follows:

Exception in thread "main" java.io.IOException: Failed to open native connection to Cassandra at {demo-dc1-service.k8ssandra-operator.svc.cluster.local:9042} :: Can't find class DefaultMetricsFactory (specified by advanced.metrics.factory.class)
	at com.datastax.spark.connector.cql.CassandraConnector$.createSession(CassandraConnector.scala:173)
	at com.datastax.spark.connector.cql.CassandraConnector$.$anonfun$sessionCache$1(CassandraConnector.scala:161)
	at com.datastax.spark.connector.cql.RefCountedCache.createNewValueAndKeys(RefCountedCache.scala:32)
	at com.datastax.spark.connector.cql.RefCountedCache.syncAcquire(RefCountedCache.scala:69)
	at com.datastax.spark.connector.cql.RefCountedCache.acquire(RefCountedCache.scala:57)
	at com.datastax.spark.connector.cql.CassandraConnector.openSession(CassandraConnector.scala:81)
	at com.datastax.spark.connector.cql.CassandraConnector.withSessionDo(CassandraConnector.scala:103)
	at com.datastax.spark.connector.util.package$.tableFromCassandra(package.scala:30)
	at com.datastax.spark.connector.rdd.CassandraTableRowReaderProvider.tableDef(CassandraTableRowReaderProvider.scala:51)
	at com.datastax.spark.connector.rdd.CassandraTableRowReaderProvider.tableDef$(CassandraTableRowReaderProvider.scala:51)
	at com.datastax.spark.connector.rdd.CassandraTableScanRDD.tableDef$lzycompute(CassandraTableScanRDD.scala:59)
	at com.datastax.spark.connector.rdd.CassandraTableScanRDD.tableDef(CassandraTableScanRDD.scala:59)
	at com.datastax.spark.connector.rdd.CassandraTableRowReaderProvider.verify(CassandraTableRowReaderProvider.scala:113)
	at com.datastax.spark.connector.rdd.CassandraTableRowReaderProvider.verify$(CassandraTableRowReaderProvider.scala:112)
	at com.datastax.spark.connector.rdd.CassandraTableScanRDD.verify(CassandraTableScanRDD.scala:59)
	at com.datastax.spark.connector.rdd.CassandraTableScanRDD.getPartitions(CassandraTableScanRDD.scala:261)
	at org.apache.spark.rdd.RDD.$anonfun$partitions$2(RDD.scala:294)
	at scala.Option.getOrElse(Option.scala:189)
	at org.apache.spark.rdd.RDD.partitions(RDD.scala:290)
	at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
	at org.apache.spark.rdd.RDD.$anonfun$partitions$2(RDD.scala:294)
	at scala.Option.getOrElse(Option.scala:189)
	at org.apache.spark.rdd.RDD.partitions(RDD.scala:290)
	at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
	at org.apache.spark.rdd.RDD.$anonfun$partitions$2(RDD.scala:294)
	at scala.Option.getOrElse(Option.scala:189)
	at org.apache.spark.rdd.RDD.partitions(RDD.scala:290)
	at org.apache.spark.Partitioner$.$anonfun$defaultPartitioner$4(Partitioner.scala:78)
	at org.apache.spark.Partitioner$.$anonfun$defaultPartitioner$4$adapted(Partitioner.scala:78)
	at scala.collection.immutable.List.map(List.scala:293)
	at org.apache.spark.Partitioner$.defaultPartitioner(Partitioner.scala:78)
	at org.apache.spark.rdd.PairRDDFunctions.$anonfun$groupByKey$6(PairRDDFunctions.scala:636)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
	at org.apache.spark.rdd.RDD.withScope(RDD.scala:410)
	at org.apache.spark.rdd.PairRDDFunctions.groupByKey(PairRDDFunctions.scala:636)
	at org.apache.spark.api.java.JavaPairRDD.groupByKey(JavaPairRDD.scala:561)
	at io.jaegertracing.spark.dependencies.cassandra.CassandraDependenciesJob.run(CassandraDependenciesJob.java:185)
	at io.jaegertracing.spark.dependencies.DependenciesSparkJob.run(DependenciesSparkJob.java:60)
	at io.jaegertracing.spark.dependencies.DependenciesSparkJob.main(DependenciesSparkJob.java:40)
Caused by: java.lang.IllegalArgumentException: Can't find class DefaultMetricsFactory (specified by advanced.metrics.factory.class)
	at com.datastax.oss.driver.internal.core.util.Reflection.resolveClass(Reflection.java:302)
	at com.datastax.oss.driver.internal.core.util.Reflection.buildFromConfig(Reflection.java:235)
	at com.datastax.oss.driver.internal.core.util.Reflection.buildFromConfig(Reflection.java:110)
	at com.datastax.oss.driver.internal.core.context.DefaultDriverContext.buildMetricsFactory(DefaultDriverContext.java:548)
	at com.datastax.oss.driver.internal.core.util.concurrent.LazyReference.get(LazyReference.java:55)
	at com.datastax.oss.driver.internal.core.context.DefaultDriverContext.getMetricsFactory(DefaultDriverContext.java:941)
	at com.datastax.oss.driver.internal.core.session.DefaultSession.<init>(DefaultSession.java:123)
	at com.datastax.oss.driver.internal.core.session.DefaultSession.init(DefaultSession.java:88)
	at com.datastax.oss.driver.api.core.session.SessionBuilder.buildDefaultSessionAsync(SessionBuilder.java:903)
	at com.datastax.oss.driver.api.core.session.SessionBuilder.buildAsync(SessionBuilder.java:817)
	at com.datastax.oss.driver.api.core.session.SessionBuilder.build(SessionBuilder.java:835)
	at com.datastax.spark.connector.cql.DefaultConnectionFactory$.createSession(CassandraConnectionFactory.scala:143)
	at com.datastax.spark.connector.cql.CassandraConnector$.createSession(CassandraConnector.scala:167)
	... 39 more

@vlsi
Copy link
Contributor Author

vlsi commented Feb 11, 2025

I believe the root cause is that minimizeJar=true causes the removal of the class:

<minimizeJar>true</minimizeJar>

If I apply minimizeJar=false, then the jar grows from 189M to 220M, however, the class com/datastax/oss/driver/internal/core/metrics/DefaultMetricsFactory.class gets included into the jar.

@yurishkuro , what do you think of deactivating minimizeJar?

Just for the reference, DefaultMetricsFactory is mentioned in Cassandra Driver's reference.conf: https://github.com/apache/cassandra-java-driver/blob/90612f6758eb0f0ba964daf054f397a47a90a736/core/src/main/resources/reference.conf#L1408, and it looks like there might be many more such references.

We can try adding filter-artifact-include=**, however, there are many modules in Cassandra Java Driver, so it might look like

                <filter>
                  <artifact>com.datastax.oss:*</artifact>
                  <includes>
                    <include>**</include>
                  </includes>
                </filter>

The resulting jar becomes 191M. However, I am not sure if minimizeJar yields significant advantage as it makes the maintenance complicated. It looks like we resort to effectively "keep" the classes for all the dependencies.

@yurishkuro
Copy link
Member

Agreed. But I also wonder why didn't our CI catch that, probably because we don't do a proper e2e test running the job's code from a container.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants