Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 41 additions & 0 deletions .asf.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

notifications:
commits: [email protected]
issues: [email protected]
pullrequests: [email protected]
jira_options: link worklog

github:
description: "Apache Spark to Apache Cassandra connector"
homepage: https://apache.github.io/cassandra-spark-connector/
enabled_merge_buttons:
squash: false
merge: false
rebase: true
features:
wiki: false
issues: false
projects: false
discussions: false
autolink_jira:
- CASSANDRA
- CASSANALYTICS
protected_branches:
trunk:
required_linear_history: true
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,5 @@ assignees: ''

---

For issue tracking we use [JIRA](https://datastax-oss.atlassian.net/projects/SPARKC)
Datastax managed QA at [Datastax Community](https://community.datastax.com/index.html)
Questions can be posted to the [mailing list](https://groups.google.com/a/lists.datastax.com/forum/#!forum/spark-connector-user).
For issue tracking we use [JIRA](https://issues.apache.org/jira/projects/CASSANALYTICS)
Community QA at [Apache Cassandra Community](https://cassandra.apache.org/_/community.html#discussions)
4 changes: 2 additions & 2 deletions .github/pull_request_template.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,14 +15,14 @@ pursue this particular fix?

Example: "I removed the incorrect assertion which would throw the ThreeIntException. This exception was incorrectly added and the assertion is not actually needed."

Fixes: [Put JIRA Reference HERE](https://datastax-oss.atlassian.net/projects/SPARKC)
Fixes: [Put JIRA Reference HERE](https://issues.apache.org/jira/projects/CASSANALYTICS)

# How Has This Been Tested?

Almost all changes and especially bug fixes will require a test to be added to either the integration or Unit Tests. Any tests added will be automatically run on travis when the pull request is pushed to github. Be sure to run suites locally as well.

# Checklist:

- [ ] I have a ticket in the [OSS JIRA](https://datastax-oss.atlassian.net/projects/SPARKC)
- [ ] I have a ticket in the [JIRA](https://issues.apache.org/jira/projects/CASSANALYTICS)
- [ ] I have performed a self-review of my own code
- [ ] Locally all tests pass (make sure tests fail without your patch)
16 changes: 7 additions & 9 deletions .github/workflows/main.yml
Original file line number Diff line number Diff line change
@@ -1,11 +1,9 @@
name: CI

on:
# Triggers the workflow on push or pull request events but only for the master branch
# Triggers the workflow on push or pull request events but only for the trunk branch
push:
branches: [ b2.5, b3.0, master]
pull_request:
branches: [ b2.5, b3.0, master]

workflow_dispatch:

Expand All @@ -16,15 +14,15 @@ jobs:
fail-fast: false
matrix:
scala: [2.12.19, 2.13.13]
db-version: [3.11.17, 4.0.12, 4.1.4, 5.0-beta1, dse-6.8.44]
db-version: [3.11.19, 4.0.17, 4.1.8, 5.0.4, dse-6.8.44]

steps:
- uses: actions/checkout@v4

- name: ccm pip installation
uses: BSFishy/pip-action@v1
with:
packages: git+https://github.com/riptano/ccm.git@d74db63d75112908a77b6c80757df9343fdc3338
- name: Install ccm via pip
# if cassandra-ccm's trunk breaks this CI, please file a report,
# and temporarily switch this to @cassandra-test or @<sha> where sha is the last known working ccm commit
run: pip install git+https://github.com/apache/cassandra-ccm.git@trunk

- name: Setup Java
uses: actions/setup-java@v4
Expand All @@ -48,4 +46,4 @@ jobs:
if: always()
with:
report_paths: '**/target/test-reports/*.xml'
annotate_only: true
annotate_only: true
48 changes: 25 additions & 23 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,14 +2,14 @@

*Lightning-fast cluster computing with Apache Spark&trade; and Apache Cassandra&reg;.*

[![CI](https://github.com/datastax/spark-cassandra-connector/actions/workflows/main.yml/badge.svg?branch=master)](https://github.com/datastax/spark-cassandra-connector/actions?query=branch%3Amaster)
[![CI](https://github.com/apache/cassandra-spark-connector/actions/workflows/main.yml/badge.svg?branch=trunk)](https://github.com/apache/cassandra-spark-connector/actions?query=branch%3Atrunk)

## Quick Links

| What | Where |
| ---------- |---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Community | Chat with us at [Apache Cassandra](https://cassandra.apache.org/_/community.html#discussions) |
| Scala Docs | Most Recent Release (3.5.1): [Connector API docs](https://datastax.github.io/spark-cassandra-connector/ApiDocs/3.5.1/connector/com/datastax/spark/connector/index.html), [Connector Driver docs](https://datastax.github.io/spark-cassandra-connector/ApiDocs/3.5.1/driver/com/datastax/spark/connector/index.html) |
| Scala Docs | Most Recent Release (3.5.1): [Connector API docs](https://apache.github.io/cassandra-spark-connector/ApiDocs/3.5.1/connector/com/datastax/spark/connector/index.html), [Connector Driver docs](https://apache.github.io/cassandra-spark-connector/ApiDocs/3.5.1/driver/com/datastax/spark/connector/index.html) |
| Latest Production Release | [3.5.1](https://search.maven.org/artifact/com.datastax.spark/spark-cassandra-connector_2.12/3.5.1/jar) |

## News
Expand Down Expand Up @@ -45,17 +45,17 @@ in your Spark applications.
The connector project has several branches, each of which map into different
supported versions of Spark and Cassandra. For previous releases the branch is
named "bX.Y" where X.Y is the major+minor version; for example the "b1.6" branch
corresponds to the 1.6 release. The "master" branch will normally contain
corresponds to the 1.6 release. The "trunk" branch will normally contain
development for the next connector release in progress.

Currently, the following branches are actively supported:
3.5.x ([master](https://github.com/datastax/spark-cassandra-connector/tree/master)),
3.4.x ([b3.4](https://github.com/datastax/spark-cassandra-connector/tree/b3.4)),
3.3.x ([b3.2](https://github.com/datastax/spark-cassandra-connector/tree/b3.3)),
3.2.x ([b3.2](https://github.com/datastax/spark-cassandra-connector/tree/b3.2)),
3.1.x ([b3.1](https://github.com/datastax/spark-cassandra-connector/tree/b3.1)),
3.0.x ([b3.0](https://github.com/datastax/spark-cassandra-connector/tree/b3.0)) and
2.5.x ([b2.5](https://github.com/datastax/spark-cassandra-connector/tree/b2.5)).
3.5.x ([trunk](https://github.com/apache/cassandra-spark-connector/tree/trunk)),
3.4.x ([b3.4](https://github.com/apache/cassandra-spark-connector/tree/b3.4)),
3.3.x ([b3.2](https://github.com/apache/cassandra-spark-connector/tree/b3.3)),
3.2.x ([b3.2](https://github.com/apache/cassandra-spark-connector/tree/b3.2)),
3.1.x ([b3.1](https://github.com/apache/cassandra-spark-connector/tree/b3.1)),
3.0.x ([b3.0](https://github.com/apache/cassandra-spark-connector/tree/b3.0)) and
2.5.x ([b2.5](https://github.com/apache/cassandra-spark-connector/tree/b2.5)).

| Connector | Spark | Cassandra | Cassandra Java Driver | Minimum Java Version | Supported Scala Versions |
|-----------|---------------|----------------------------|-----------------------|----------------------|--------------------------|
Expand Down Expand Up @@ -85,32 +85,32 @@ Currently, the following branches are actively supported:
API documentation for the Scala and Java interfaces are available online:

### 3.5.1
* [Spark-Cassandra-Connector](https://datastax.github.io/spark-cassandra-connector/ApiDocs/3.5.1/connector/com/datastax/spark/connector/index.html)
* [Spark-Cassandra-Connector](https://apache.github.io/cassandra-spark-connector/ApiDocs/3.5.1/connector/com/datastax/spark/connector/index.html)

### 3.5.0
* [Spark-Cassandra-Connector](https://datastax.github.io/spark-cassandra-connector/ApiDocs/3.5.0/connector/com/datastax/spark/connector/index.html)
* [Spark-Cassandra-Connector](https://apache.github.io/cassandra-spark-connector/ApiDocs/3.5.0/connector/com/datastax/spark/connector/index.html)

### 3.4.1
* [Spark-Cassandra-Connector](https://datastax.github.io/spark-cassandra-connector/ApiDocs/3.4.1/connector/com/datastax/spark/connector/index.html)
* [Spark-Cassandra-Connector](https://apache.github.io/cassandra-spark-connector/ApiDocs/3.4.1/connector/com/datastax/spark/connector/index.html)

### 3.3.0
* [Spark-Cassandra-Connector](https://datastax.github.io/spark-cassandra-connector/ApiDocs/3.3.0/connector/com/datastax/spark/connector/index.html)
* [Spark-Cassandra-Connector](https://apache.github.io/cassandra-spark-connector/ApiDocs/3.3.0/connector/com/datastax/spark/connector/index.html)

### 3.2.0
* [Spark-Cassandra-Connector](https://datastax.github.io/spark-cassandra-connector/ApiDocs/3.2.0/connector/com/datastax/spark/connector/index.html)
* [Spark-Cassandra-Connector](https://apache.github.io/cassandra-spark-connector/ApiDocs/3.2.0/connector/com/datastax/spark/connector/index.html)

### 3.1.0
* [Spark-Cassandra-Connector](https://datastax.github.io/spark-cassandra-connector/ApiDocs/3.1.0/connector/com/datastax/spark/connector/index.html)
* [Spark-Cassandra-Connector](https://apache.github.io/cassandra-spark-connector/ApiDocs/3.1.0/connector/com/datastax/spark/connector/index.html)

### 3.0.1
* [Spark-Cassandra-Connector](https://datastax.github.io/spark-cassandra-connector/ApiDocs/3.0.1/connector/com/datastax/spark/connector/index.html)
* [Spark-Cassandra-Connector](https://apache.github.io/cassandra-spark-connector/ApiDocs/3.0.1/connector/com/datastax/spark/connector/index.html)

### 2.5.2
* [Spark-Cassandra-Connector](https://datastax.github.io/spark-cassandra-connector/ApiDocs/2.5.2/connector/#package)
* [Spark-Cassandra-Connector](https://apache.github.io/cassandra-spark-connector/ApiDocs/2.5.2/connector/#package)

### 2.4.2
* [Spark-Cassandra-Connector](http://datastax.github.io/spark-cassandra-connector/ApiDocs/2.4.2/spark-cassandra-connector/)
* [Embedded-Cassandra](http://datastax.github.io/spark-cassandra-connector/ApiDocs/2.4.2/spark-cassandra-connector-embedded/)
* [Spark-Cassandra-Connector](http://apache.github.io/cassandra-spark-connector/ApiDocs/2.4.2/spark-cassandra-connector/)
* [Embedded-Cassandra](http://apache.github.io/cassandra-spark-connector/ApiDocs/2.4.2/spark-cassandra-connector-embedded/)

## Download

Expand Down Expand Up @@ -157,13 +157,13 @@ In [DS320: Analytics with Spark](https://www.youtube.com/watch?v=D6PMEQAfjeU&lis

### Reporting Bugs

New issues may be reported using [JIRA](https://datastax-oss.atlassian.net/browse/SPARKC/). Please include
New issues may be reported using [JIRA](https://issues.apache.org/jira/projects/CASSANALYTICS). Please include
all relevant details including versions of Spark, Spark Cassandra Connector, Cassandra and/or DSE. A minimal
reproducible case with sample code is ideal.

### Mailing List

Questions and requests for help may be submitted to the [user mailing list](https://groups.google.com/a/lists.datastax.com/forum/#!forum/spark-connector-user).
Questions and requests for help may be submitted to the [user mailing list](https://cassandra.apache.org/_/community.html#discussions).


## Q/A Exchange
Expand All @@ -177,13 +177,15 @@ To protect the community, all contributors are required to sign the Apache Softw
[Tips for Developing the Spark Cassandra Connector](doc/developers.md)

Checklist for contributing changes to the project:
* Create a [SPARKC JIRA](https://datastax-oss.atlassian.net/projects/SPARKC/issues)
* Create a [CASSANALYTICS JIRA](https://issues.apache.org/jira/projects/CASSANALYTICS)
* Make sure that all unit tests and integration tests pass
* Add an appropriate entry at the top of CHANGES.txt
* If the change has any end-user impacts, also include changes to the ./doc files as needed
* Prefix the pull request description with the JIRA number, for example: "SPARKC-123: Fix the ..."
* Open a pull-request on GitHub and await review

Old issues from before the donation to the ASF and the Apache Cassandra project can be found in this [SPARKC JIRA](https://datastax-oss.atlassian.net/projects/SPARKC/issues)

## Testing
To run unit and integration tests:

Expand Down
18 changes: 15 additions & 3 deletions connector/src/it/resources/logback.xml
Original file line number Diff line number Diff line change
@@ -1,8 +1,20 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--
~ Copyright DataStax, Inc.
~
~ Please see the included license file for details.
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->

<configuration>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ sealed trait Fixture extends ClusterProvider {
System.setProperty("dse", "true")
System.setProperty("ccm.directory", home)
System.setProperty("ccm.version", "6.8")
System.setProperty("ccm.branch", "master")
System.setProperty("ccm.branch", "trunk")
}

private[cluster] def configs: Seq[CcmConfig]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,12 +16,6 @@
* limitations under the License.
*/

/*
* Copyright DataStax, Inc.
*
* Please see the included license file for details.
*/

package com.datastax.spark.connector.cql

import com.datastax.bdp.spark.ContinuousPagingScanner
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,12 +16,6 @@
* limitations under the License.
*/

/*
* Copyright DataStax, Inc.
*
* Please see the included license file for details.
*/

package com.datastax.spark.connector.rdd

import java.util.concurrent.LinkedTransferQueue
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,12 +16,6 @@
* limitations under the License.
*/

/*
* Copyright DataStax, Inc.
*
* Please see the included license file for details.
*/

package com.datastax.spark.connector.rdd

import java.lang.{Integer => JInteger, String => JString}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,12 +16,6 @@
* limitations under the License.
*/

/*
* Copyright DataStax, Inc.
*
* Please see the included license file for details.
*/

package com.datastax.spark.connector.rdd

import com.datastax.oss.driver.api.core.CqlSession
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,12 +16,6 @@
* limitations under the License.
*/

/*
* Copyright DataStax, Inc.
*
* Please see the included license file for details.
*/

package com.datastax.spark.connector.types

import com.datastax.dse.driver.api.core.`type`.DseDataTypes
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,11 +16,6 @@
* limitations under the License.
*/

/*
* Copyright DataStax, Inc.
*
* Please see the included license file for details.
*/
package com.datastax.bdp.spark

import java.io.IOException
Expand Down
5 changes: 0 additions & 5 deletions connector/src/main/scala/com/datastax/bdp/util/MiscUtil.scala
Original file line number Diff line number Diff line change
Expand Up @@ -16,11 +16,6 @@
* limitations under the License.
*/

/*
* Copyright DataStax, Inc.
*
* Please see the included license file for details.
*/

package com.datastax.bdp.util

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,11 +16,6 @@
* limitations under the License.
*/

/*
* Copyright DataStax, Inc.
*
* Please see the included license file for details.
*/

package com.datastax.bdp.util

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,11 +16,6 @@
* limitations under the License.
*/

/**
* Copyright DataStax, Inc.
*
* Please see the included license file for details.
*/
package com.datastax.spark.connector.rdd

import java.io.IOException
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,11 +16,6 @@
* limitations under the License.
*/

/**
* Copyright DataStax, Inc.
*
* Please see the included license file for details.
*/
package com.datastax.spark.connector.rdd

import java.io.IOException
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,12 +16,6 @@
* limitations under the License.
*/

/*
* Copyright DataStax, Inc.
*
* Please see the included license file for details.
*/

package com.datastax.spark.connector.rdd

import java.lang.{String => JString}
Expand Down
Loading