can not change column type dynamically

[gengjun123]

Please answer these questions before submitting your issue. Thanks!

What version of Cassandra are you using?

Cassandra 3.7

What version of Gocql are you using?

the commit is ca7d339

What did you do?

I delete a column whose type is double, and add a new column with the same name but the type is int. After that, I got lots of error(can not marshal int64 into double) when insert. Then, I restart my application, and the result is ok. I think the cause is statment prepare int the lru cache. But is there approach to support change the column type dynamically without restarting my application. My application is a store middleware.

What did you expect to see?

change the column type dynamically which out restart the application.

What did you see instead?

error: can not marshal int64 into double

[beltran]

I think the cause is statment prepare int the lru cache.

This is correct. Only fix I can think of is to clear the prepared statement cache for a particular table when a table updated event is received.

[alourie]

@beltran I'm not completely sure that's right. I've built the following test to simulate the scenario; the test creates the table with column of type int, inserts a value of 1; drops the column; adds the column with the same name but type double; inserts 1.1. The test passes without any problem and without clearing the cache. I know it's in reverse order compared to the given report, but I think it indicates that there's no inherent problem in the driver to work with changed underlying C* column types; and it has no problem to do so with the working LRU cache.

@gengjun123 Can you provide some code examples from your application? Specifically, how your queries are constructed, how do you drop/add the column, in which order, etc. Also, why are you "changing" the column type while your application is running? This sounds strange to me...

[beltran]

@alourie I don't know why your test doesn't fail. I can repro with this, which does the same as your test but with a different table:

func TestAlterTable(t *testing.T) {
	fmt.Println("In the beginning of WriteFailure")
	policy0 := TokenAwareHostPolicy(RoundRobinHostPolicy())
	cluster := NewCluster("127.0.0.1", "127.0.0.2")
	// cluster.Keyspace = "example"
	cluster.PoolConfig = PoolConfig{
		HostSelectionPolicy: policy0,
	}
	cluster.Consistency = All
	cluster.ProtoVersion = 5
	session, err := cluster.CreateSession()
	if err != nil {
		panic(err)
	}

	fmt.Println("Droping keyspace")
	if err := session.Query(`DROP KEYSPACE IF EXISTS altertest`).Exec(); err != nil {
		log.Fatal(err)
	}

	fmt.Println("Creating keyspace")
	if err := session.Query(`CREATE KEYSPACE altertest WITH replication = { 'class': 'SimpleStrategy', 'replication_factor': '3' }`).Exec(); err != nil {
		log.Fatal(err)
	}

	fmt.Println("Creating table")
	if err := session.Query(`CREATE TABLE IF NOT EXISTS altertest.mytable (thekey text,
		col1 text,
		col2 text,
		PRIMARY KEY (thekey, col1))`).Exec(); err != nil {
		log.Fatal(err)
	}
	time.Sleep(1000 * time.Millisecond)

	fmt.Println("Inserting into table")
	if err := session.Query(`INSERT INTO altertest.mytable (thekey, col1, col2)
	VALUES (?, ?, ?)`, "1", "2", "3").Exec(); err != nil {
		log.Fatal(err)
	}

	fmt.Println("Altering table")
	if err := session.Query(`ALTER TABLE altertest.mytable DROP col2`).Exec(); err != nil {
		log.Fatal(err)
	}

	if err := session.Query(`ALTER TABLE altertest.mytable ADD col2 int`).Exec(); err != nil {
		log.Fatal(err)
	}

	fmt.Println("Executing last insert")
	if err := session.Query(`INSERT INTO altertest.mytable (thekey, col1, col2)
	VALUES (?, ?, ?)`, "1", "2", 3).Exec(); err != nil {
		errWrite, ok := err.(*RequestErrWriteFailure)
		fmt.Println("ok: ", ok)
		if ok {
			fmt.Println("errWite: ", errWrite.ErrorMap)
		}
		log.Fatal(err)
	}
}

It may be related to https://issues.apache.org/jira/browse/CASSANDRA-10786.

[alourie]

@beltran ok, now it's really weird. I'm using your test exactly, just with a different way to initiate the cluster - not creating the cluster object, just all the code till the first DROP replaced with:

session := createSession(t)
defer session.Close()

Then starting ccm cluster and running the test:

~> ccm start
~> go test -run=TestAlter -tags 'integration gocql_debug' -timeout=5m -race -gocql.timeout=60s -runssl -proto=5 -rf=3 -clusterSize=3 -autowait=2000ms -compressor=snappy -gocql.cversion=3.11.3 -cluster=127.0.0.1,127.0.0.3,127.0.0.2 ./...
In the beginning of WriteFailure
2018/09/28 14:04:59 logger.go:27: gocql: Session.handleNodeUp: 127.0.0.3:9042
2018/09/28 14:05:00 logger.go:27: gocql: handling frame: [event schema_change_keyspace change="DROPPED" keyspace="gocql_test"]
2018/09/28 14:05:02 logger.go:27: gocql: handling frame: [event schema_change_keyspace change="CREATED" keyspace="gocql_test"]
2018/09/28 14:05:03 logger.go:27: gocql: Session.handleNodeUp: 127.0.0.1:9042
Droping keyspace
2018/09/28 14:05:04 logger.go:27: gocql: handling frame: [event schema_change change="DROPPED" keyspace="altertest" object="mytable"]
2018/09/28 14:05:05 logger.go:27: gocql: handling frame: [event schema_change_keyspace change="DROPPED" keyspace="altertest"]
Creating keyspace
2018/09/28 14:05:05 logger.go:27: gocql: handling frame: [event schema_change_keyspace change="CREATED" keyspace="altertest"]
Creating table
2018/09/28 14:05:07 logger.go:27: gocql: handling frame: [event schema_change change="CREATED" keyspace="altertest" object="mytable"]
Inserting into table
Altering table
2018/09/28 14:05:09 logger.go:27: gocql: handling frame: [event schema_change change="UPDATED" keyspace="altertest" object="mytable"]
2018/09/28 14:05:10 logger.go:27: gocql: handling frame: [event schema_change change="UPDATED" keyspace="altertest" object="mytable"]
Executing last insert
PASS
ok      github.com/gocql/gocql  12.522s

So we still can't reproduce reliably.

Moreover, I'm thinking that can not marshal int64 into double error is most likely has nothing to do with the underlining C* type as it's coming directly from the marshal.go code, meaning that it couldn't translate some type. That's why I think we need to see the code that actually is doing the inserts to make a more educated guess.

[beltran]

@alourie I believe you will hit it if you create the cluster object like in the test I posted.

OK I believe it's the bug in CASSANDRA-10786 what's causing this. CASSANDRA-10786 was only done for Cassandra 4.0 and protocol V5. The driver fix of CASSANDRA-10786 would break protocol V5 against Cassandra 3.11, I'm a bit reluctant to add this because our current tests travis tests V5 against 3.11 and it doesn't seem that easy to me to change it to Cassandra 4.0.

To be clear, if you're not using Cassandra 4.0 (which hasn't been released) and protocol V5 there's not a fix for this that I'm aware of. Also after CASSANDRA-10786 you'd get an InvalidRequest error at the ALTER with message similar to Cannot re-add a previously dropped column 'col2' of type int, incompatible with previous type text (this is the fix, instead of silent failure that would be discovered later). So this is not really supported.

@gengjun123 you're best shot is not to do this kind of renames. There are more scenarios in which a columns alter may cause problems, there is more information on the CASSANDRA-10786 jira ticket

[alourie]

@gengjun123 I don't see currently that we need to fix anything. I'll close this ticket tomorrow unless you think there's anything else we could've missed. Thanks!

[Zariel]

We should clear the prepared statement cache after getting a schema change event.

…

On Tue, 9 Oct 2018 at 2:37 am, Alex Lourie ***@***.***> wrote: @gengjun123 <https://github.com/gengjun123> I don't see currently that we need to fix anything. I'll close this ticket tomorrow unless you think there's anything else we could've missed. Thanks! — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1186 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAIpaNKsCX82onjOQR_1YFQpZpKBd3Pks5ui_3AgaJpZM4Wm5i6> .

[alourie]

@Zariel we can, I even tried to completely drop it, but it doesn't solve this problem.

[ashetkar]

Faced similar issue while trying to insert into a re-created table with different schema.
Could not find a way to remove the insert statement from stmtsLRU cache or invalidate the cache itself.

This is what I did:

• Started Cassandra in a Docker container
• Ran below code which
  • Creates a keyspace and a table with two columns (int, int)
  • Inserts a row
  • Drops the table and re-creates the keyspace
  • Creates the same table but with column types (int, text)
  • Attempts to insert a row with the earlier insert statement but with different values.

package main

import (
        "fmt"
        "log"
        "time"

        "github.com/gocql/gocql"
)

func main() {
        // Connect to the cluster.
        cluster := gocql.NewCluster("127.0.0.1")
        cluster.Timeout = 12 * time.Second

        // Create the session.
        session, _ := cluster.CreateSession()
        defer session.Close()

        // Set up the keyspace and table.
        createKyStmt := `CREATE KEYSPACE IF NOT EXISTS demo WITH REPLICATION = { 
                                   'class' : 'SimpleStrategy', 
                                   'replication_factor' : 1 };`
        if err := session.Query(createKyStmt).Exec(); err != nil {
                log.Fatal(err)
        }
        fmt.Println("Created keyspace demo")

        createStmt := `CREATE TABLE demo.employee (id int PRIMARY KEY, language int)`
        if err := session.Query(createStmt).Exec(); err != nil {
                log.Fatal(err)
        }
        fmt.Println("Created table demo.employee (int, int)")

        // Insert into the table.
        insertStmt := "INSERT INTO demo.employee(id, language) VALUES (?,?)"
        if err := session.Query(insertStmt, 1, 1).Exec(); err != nil {
                log.Fatal(err)
        }
        fmt.Printf("Inserted data: %s\n", insertStmt)

        dropTbStmt := `DROP TABLE demo.employee`
        if err := session.Query(dropTbStmt).Exec(); err != nil {
                log.Fatal(err)
        }
        fmt.Println("Dropped table demo.employee")

        dropKyStmt := `DROP KEYSPACE demo`
        if err := session.Query(dropKyStmt).Exec(); err != nil {
                log.Fatal(err)
        }
        fmt.Println("Dropped keyspace demo")

        if err := session.Query(createKyStmt).Exec(); err != nil {
                log.Fatal(err)
        }
        fmt.Println("Created keyspace demo again")

        createStmt = `CREATE TABLE demo.employee (id int PRIMARY KEY,
                                                           language text)`
        if err := session.Query(createStmt).Exec(); err != nil {
                log.Fatal(err)
        }
        fmt.Println("Created table demo.employee with a slightly different structure (int, text)")

        if err := session.Query(insertStmt, 1, "test").Exec(); err != nil {
                log.Fatal(err)
        }
        fmt.Printf("Inserted data: %s\n", insertStmt)
}

The insert into the new table fails with error

can not marshal string to int: strconv.ParseInt: parsing "test": invalid syntax
exit status 1

[OleksiienkoMykyta]

Looks like this issue will be eliminated right after the PR with proto v5 support is merged.
#1822
The related Cassandra ticket.
https://issues.apache.org/jira/browse/CASSANDRA-10786
@joao-r-reis should we close it as stale, or leave it until PR is merged?

[joao-r-reis]

Hmm I think I'll close this since we already have an open issue/jira and pr for the protocol v5 change that resolves this