Integration tests fail using Scylla 5.2.x

[vponomaryov]

Please answer these questions before submitting your issue. Thanks!

What version of Scylla or Cassandra are you using?

5.2.1 and 5.2.0

What version of Gocql are you using?

v1.7.3

What version of Go are you using?

1.20.4

What did you do?

I explicitly defined the Scylla version:

SCYLLA_IMAGE=scylladb/scylla:5.2.1

Then ran tests:

./integration.sh integration scylla

And got following result:

==> Running integration scylla tests with args: -gocql.timeout=60s -proto=4 -rf=1 -clusterSize=1 -autowait=2000ms -compressor=snappy -gocql.cversion=3.11.4 -cluster=192.168.100.11
2023/06/08 12:42:51 logger.go:35: error: failed to connect to "[HostInfo hostname=\"192.168.100.11\" connectAddress=\"192.168.100.11\" peer=\"<nil>\" rpc_address=\"192.168.100.11\" broadcast_address=\"192.168.100.11\" preferred_ip=\"<nil>\" connect_addr=\"192.168.100.11\" connect_addr_source=\"connect_address\" port=9042 data_centre=\"datacenter1\" rack=\"rack1\" host_id=\"38da3ff8-79e5-4515-96d6-9edbe943df37\" version=\"v3.0.8\" state=UP num_tokens=1]" due to error: EOF
2023/06/08 12:42:51 logger.go:35: error: failed to connect to "[HostInfo hostname=\"192.168.100.11\" connectAddress=\"192.168.100.11\" peer=\"<nil>\" rpc_address=\"192.168.100.11\" broadcast_address=\"192.168.100.11\" preferred_ip=\"<nil>\" connect_addr=\"192.168.100.11\" connect_addr_source=\"connect_address\" port=9042 data_centre=\"datacenter1\" rack=\"rack1\" host_id=\"38da3ff8-79e5-4515-96d6-9edbe943df37\" version=\"v3.0.8\" state=UP num_tokens=1]" due to error: Keyspace 'invalidKeyspace' does not exist
2023/06/08 12:42:56 logger.go:35: gocql: unable to dial control conn 127.0.0.1:1: dial tcp :0->127.0.0.1:1: connect: connection refused
panic: test timed out after 5m0s
running tests:
        TestBatchObserve (4m38s)

goroutine 54910 [running]:
testing.(*M).startAlarm.func1()
        /usr/local/go/src/testing/testing.go:2241 +0x219
created by time.goFunc
        /usr/local/go/src/time/sleep.go:176 +0x48

goroutine 1 [chan receive, 4 minutes]:
testing.(*T).Run(0xc000103380, {0xbb06fc, 0x10}, 0xbea210)
        /usr/local/go/src/testing/testing.go:1630 +0x82e
testing.runTests.func1(0x0?)
        /usr/local/go/src/testing/testing.go:2036 +0x8e
testing.tRunner(0xc000103380, 0xc00016fb48)
        /usr/local/go/src/testing/testing.go:1576 +0x217
testing.runTests(0xc000140dc0?, {0xf95a40, 0x94, 0x94}, {0x1c?, 0x4a8b19?, 0xf99780?})
        /usr/local/go/src/testing/testing.go:2034 +0x87d
testing.(*M).Run(0xc000140dc0)
        /usr/local/go/src/testing/testing.go:1906 +0xb45
main.main()
        _testmain.go:345 +0x2ea

goroutine 125 [select, 1 minutes]:
github.com/gocql/gocql.(*Session).reconnectDownedHosts(0xc000472000, 0xdf8475800?)
        /home/kiparis/projects/gocql/session.go:374 +0x177
created by github.com/gocql/gocql.(*Session).init
        /home/kiparis/projects/gocql/session.go:321 +0x108c

goroutine 1360 [select]:
github.com/gocql/gocql.(*Conn).heartBeat(0x47943a?, {0xc9b758, 0xc0003614f0})
        /home/kiparis/projects/gocql/conn.go:616 +0x192
created by github.com/gocql/gocql.(*Conn).init
        /home/kiparis/projects/gocql/conn.go:326 +0x78c

goroutine 1381 [select]:
github.com/gocql/gocql.(*writeCoalescer).writeFlusher(0xc000591c80, 0xbea210?)
        /home/kiparis/projects/gocql/conn.go:880 +0x245
created by github.com/gocql/gocql.newWriteCoalescer
        /home/kiparis/projects/gocql/conn.go:785 +0x305

goroutine 1374 [select]:
github.com/gocql/gocql.(*Conn).prepareStatement(0xc0003a8900, {0xc9b790, 0xc000126018}, {0xc0000a9040, 0x38}, {0x0?, 0x0})
        /home/kiparis/projects/gocql/conn.go:1091 +0x44c
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1349 +0x7bf
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
github.com/gocql/gocql.(*Conn).executeBatch(0xc0003a8900, {0xc9b790, 0xc000126018}, 0xc0000d0000)
        /home/kiparis/projects/gocql/conn.go:1425 +0x18ea
created by testing.(*T).Run
        /usr/local/go/src/testing/testing.go:1629 +0x806

goroutine 194 [select]:
github.com/gocql/gocql.(*Conn).heartBeat(0xc00033e300?, {0xc9b758, 0xc0001b4c30})
        /home/kiparis/projects/gocql/conn.go:616 +0x192
created by github.com/gocql/gocql.(*Conn).init
        /home/kiparis/projects/gocql/conn.go:326 +0x78c

goroutine 75 [select]:
github.com/gocql/gocql.(*writeCoalescer).writeFlusher(0xc000386780, 0x30d40?)
        /home/kiparis/projects/gocql/conn.go:880 +0x245
created by github.com/gocql/gocql.newWriteCoalescer
        /home/kiparis/projects/gocql/conn.go:785 +0x305

goroutine 181 [select, 4 minutes]:
github.com/gocql/gocql.(*eventDebouncer).flusher(0xc0000e6b40)
        /home/kiparis/projects/gocql/events.go:42 +0x126
created by github.com/gocql/gocql.newEventDebouncer
        /home/kiparis/projects/gocql/events.go:30 +0x2b7

goroutine 1345 [select]:
github.com/gocql/gocql.(*controlConn).heartBeat(0xc000498940)
        /home/kiparis/projects/gocql/control.go:76 +0x1a9
created by github.com/gocql/gocql.(*controlConn).connect
        /home/kiparis/projects/gocql/control.go:265 +0x2cf

goroutine 161 [IO wait]:
internal/poll.runtime_pollWait(0x7f343867b070, 0x72)
        /usr/local/go/src/runtime/netpoll.go:306 +0x89
internal/poll.(*pollDesc).wait(0xc0001e6a98, 0xc000346000?, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0xbd
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc0001e6a80, {0xc000346000, 0x1000, 0x1000})
        /usr/local/go/src/internal/poll/fd_unix.go:167 +0x405
net.(*netFD).Read(0xc0001e6a80, {0xc000346000, 0x1000, 0x1000})
        /usr/local/go/src/net/fd_posix.go:55 +0x51
net.(*conn).Read(0xc000192060, {0xc000346000, 0x1000, 0x1000})
        /usr/local/go/src/net/net.go:183 +0xb1
bufio.(*Reader).Read(0xc0001b0e40, {0xc00033e348, 0x1, 0x9})
        /usr/local/go/src/bufio/bufio.go:237 +0x4f3
io.ReadAtLeast({0xc97860, 0xc0001b0e40}, {0xc00033e348, 0x1, 0x9}, 0x1)
        /usr/local/go/src/io/io.go:332 +0xde
io.ReadFull(...)
        /usr/local/go/src/io/io.go:351
github.com/gocql/gocql.readHeader({0xc97860?, 0xc0001b0e40?}, {0xc00033e348?, 0x9?, 0x9?})
        /home/kiparis/projects/gocql/frame.go:436 +0xb6
github.com/gocql/gocql.(*Conn).recv(0xc00033e300, {0xc9b758, 0xc0001b4c30})
        /home/kiparis/projects/gocql/conn.go:659 +0x130
github.com/gocql/gocql.(*Conn).serve(0xc0000e6ba0?, {0xc9b758, 0xc0001b4c30})
        /home/kiparis/projects/gocql/conn.go:576 +0x47
created by github.com/gocql/gocql.(*Conn).init
        /home/kiparis/projects/gocql/conn.go:325 +0x6d4

goroutine 1375 [select, 4 minutes]:
github.com/gocql/gocql.(*eventDebouncer).flusher(0xc0005918c0)
        /home/kiparis/projects/gocql/events.go:42 +0x126
created by github.com/gocql/gocql.newEventDebouncer
        /home/kiparis/projects/gocql/events.go:30 +0x2b7

goroutine 182 [select, 4 minutes]:
github.com/gocql/gocql.(*eventDebouncer).flusher(0xc0000e6c00)
        /home/kiparis/projects/gocql/events.go:42 +0x126
created by github.com/gocql/gocql.newEventDebouncer
        /home/kiparis/projects/gocql/events.go:30 +0x2b7

goroutine 128 [select]:
github.com/gocql/gocql.(*writeCoalescer).writeFlusher(0xc0001b1320, 0x30d40?)
        /home/kiparis/projects/gocql/conn.go:880 +0x245
created by github.com/gocql/gocql.newWriteCoalescer
        /home/kiparis/projects/gocql/conn.go:785 +0x305

goroutine 1351 [IO wait]:
internal/poll.runtime_pollWait(0x7f343867b160, 0x72)
        /usr/local/go/src/runtime/netpoll.go:306 +0x89
internal/poll.(*pollDesc).wait(0xc00045c598, 0xc00030c000?, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0xbd
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc00045c580, {0xc00030c000, 0x1000, 0x1000})
        /usr/local/go/src/internal/poll/fd_unix.go:167 +0x405
net.(*netFD).Read(0xc00045c580, {0xc00030c000, 0x1000, 0x1000})
        /usr/local/go/src/net/fd_posix.go:55 +0x51
net.(*conn).Read(0xc0000140b8, {0xc00030c000, 0x1000, 0x1000})
        /usr/local/go/src/net/net.go:183 +0xb1
bufio.(*Reader).Read(0xc0005919e0, {0xc0000c0348, 0x1, 0x9})
        /usr/local/go/src/bufio/bufio.go:237 +0x4f3
io.ReadAtLeast({0xc97860, 0xc0005919e0}, {0xc0000c0348, 0x1, 0x9}, 0x1)
        /usr/local/go/src/io/io.go:332 +0xde
io.ReadFull(...)
        /usr/local/go/src/io/io.go:351
github.com/gocql/gocql.readHeader({0xc97860?, 0xc0005919e0?}, {0xc0000c0348?, 0x9?, 0x9?})
        /home/kiparis/projects/gocql/frame.go:436 +0xb6
github.com/gocql/gocql.(*Conn).recv(0xc0000c0300, {0xc9b758, 0xc00040af50})
        /home/kiparis/projects/gocql/conn.go:659 +0x130
github.com/gocql/gocql.(*Conn).serve(0x47943a?, {0xc9b758, 0xc00040af50})
        /home/kiparis/projects/gocql/conn.go:576 +0x47
created by github.com/gocql/gocql.(*Conn).init
        /home/kiparis/projects/gocql/conn.go:325 +0x6d4

goroutine 77 [select]:
github.com/gocql/gocql.(*Conn).heartBeat(0xc0003a8180?, {0xc9b758, 0xc000388500})
        /home/kiparis/projects/gocql/conn.go:616 +0x192
created by github.com/gocql/gocql.(*Conn).init
        /home/kiparis/projects/gocql/conn.go:326 +0x78c

goroutine 76 [IO wait]:
internal/poll.runtime_pollWait(0x7f343867af80, 0x72)
        /usr/local/go/src/runtime/netpoll.go:306 +0x89
internal/poll.(*pollDesc).wait(0xc0003a0198, 0xc0003d2000?, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0xbd
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc0003a0180, {0xc0003d2000, 0x1000, 0x1000})
        /usr/local/go/src/internal/poll/fd_unix.go:167 +0x405
net.(*netFD).Read(0xc0003a0180, {0xc0003d2000, 0x1000, 0x1000})
        /usr/local/go/src/net/fd_posix.go:55 +0x51
net.(*conn).Read(0xc000380040, {0xc0003d2000, 0x1000, 0x1000})
        /usr/local/go/src/net/net.go:183 +0xb1
bufio.(*Reader).Read(0xc0003866c0, {0xc0003a81c8, 0x1, 0x9})
        /usr/local/go/src/bufio/bufio.go:237 +0x4f3
io.ReadAtLeast({0xc97860, 0xc0003866c0}, {0xc0003a81c8, 0x1, 0x9}, 0x1)
        /usr/local/go/src/io/io.go:332 +0xde
io.ReadFull(...)
        /usr/local/go/src/io/io.go:351
github.com/gocql/gocql.readHeader({0xc97860?, 0xc0003866c0?}, {0xc0003a81c8?, 0x9?, 0x9?})
        /home/kiparis/projects/gocql/frame.go:436 +0xb6
github.com/gocql/gocql.(*Conn).recv(0xc0003a8180, {0xc9b758, 0xc000388500})
        /home/kiparis/projects/gocql/conn.go:659 +0x130
github.com/gocql/gocql.(*Conn).serve(0xc00038e4e0?, {0xc9b758, 0xc000388500})
        /home/kiparis/projects/gocql/conn.go:576 +0x47
created by github.com/gocql/gocql.(*Conn).init
        /home/kiparis/projects/gocql/conn.go:325 +0x6d4

goroutine 70 [select]:
github.com/gocql/gocql.(*controlConn).heartBeat(0xc0001e1100)
        /home/kiparis/projects/gocql/control.go:76 +0x1a9
created by github.com/gocql/gocql.(*controlConn).connect
        /home/kiparis/projects/gocql/control.go:265 +0x2cf

goroutine 1352 [select]:
github.com/gocql/gocql.(*Conn).heartBeat(0x47943a?, {0xc9b758, 0xc00040af50})
        /home/kiparis/projects/gocql/conn.go:616 +0x192
created by github.com/gocql/gocql.(*Conn).init
        /home/kiparis/projects/gocql/conn.go:326 +0x78c

goroutine 129 [IO wait]:
internal/poll.runtime_pollWait(0x7f343867ae90, 0x72)
        /usr/local/go/src/runtime/netpoll.go:306 +0x89
internal/poll.(*pollDesc).wait(0xc0003a0518, 0xc0003e4000?, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0xbd
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc0003a0500, {0xc0003e4000, 0x1000, 0x1000})
        /usr/local/go/src/internal/poll/fd_unix.go:167 +0x405
net.(*netFD).Read(0xc0003a0500, {0xc0003e4000, 0x1000, 0x1000})
        /usr/local/go/src/net/fd_posix.go:55 +0x51
net.(*conn).Read(0xc000380048, {0xc0003e4000, 0x1000, 0x1000})
        /usr/local/go/src/net/net.go:183 +0xb1
bufio.(*Reader).Read(0xc0003867e0, {0xc0003a8348, 0x1, 0x9})
        /usr/local/go/src/bufio/bufio.go:237 +0x4f3
io.ReadAtLeast({0xc97860, 0xc0003867e0}, {0xc0003a8348, 0x1, 0x9}, 0x1)
        /usr/local/go/src/io/io.go:332 +0xde
io.ReadFull(...)
        /usr/local/go/src/io/io.go:351
github.com/gocql/gocql.readHeader({0xc97860?, 0xc0003867e0?}, {0xc0003a8348?, 0x9?, 0x9?})
        /home/kiparis/projects/gocql/frame.go:436 +0xb6
github.com/gocql/gocql.(*Conn).recv(0xc0003a8300, {0xc9b758, 0xc000388b90})
        /home/kiparis/projects/gocql/conn.go:659 +0x130
github.com/gocql/gocql.(*Conn).serve(0x4aa657?, {0xc9b758, 0xc000388b90})
        /home/kiparis/projects/gocql/conn.go:576 +0x47
created by github.com/gocql/gocql.(*Conn).init
        /home/kiparis/projects/gocql/conn.go:325 +0x6d4

goroutine 210 [select]:
github.com/gocql/gocql.(*Conn).heartBeat(0xc0003a8300?, {0xc9b758, 0xc000388b90})
        /home/kiparis/projects/gocql/conn.go:616 +0x192
created by github.com/gocql/gocql.(*Conn).init
        /home/kiparis/projects/gocql/conn.go:326 +0x78c

goroutine 1376 [select, 4 minutes]:
github.com/gocql/gocql.(*eventDebouncer).flusher(0xc000591980)
        /home/kiparis/projects/gocql/events.go:42 +0x126
created by github.com/gocql/gocql.newEventDebouncer
        /home/kiparis/projects/gocql/events.go:30 +0x2b7

goroutine 1383 [select]:
github.com/gocql/gocql.(*Conn).heartBeat(0x47943a?, {0xc9b758, 0xc0004de7d0})
        /home/kiparis/projects/gocql/conn.go:616 +0x192
created by github.com/gocql/gocql.(*Conn).init
        /home/kiparis/projects/gocql/conn.go:326 +0x78c

goroutine 1359 [IO wait]:
internal/poll.runtime_pollWait(0x7f343867b250, 0x72)
        /usr/local/go/src/runtime/netpoll.go:306 +0x89
internal/poll.(*pollDesc).wait(0xc0002d0a18, 0xc0002d2000?, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0xbd
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc0002d0a00, {0xc0002d2000, 0x1000, 0x1000})
        /usr/local/go/src/internal/poll/fd_unix.go:167 +0x405
net.(*netFD).Read(0xc0002d0a00, {0xc0002d2000, 0x1000, 0x1000})
        /usr/local/go/src/net/fd_posix.go:55 +0x51
net.(*conn).Read(0xc000192068, {0xc0002d2000, 0x1000, 0x1000})
        /usr/local/go/src/net/net.go:183 +0xb1
bufio.(*Reader).Read(0xc000317b60, {0xc0003a8948, 0x1, 0x9})
        /usr/local/go/src/bufio/bufio.go:237 +0x4f3
io.ReadAtLeast({0xc97860, 0xc000317b60}, {0xc0003a8948, 0x1, 0x9}, 0x1)
        /usr/local/go/src/io/io.go:332 +0xde
io.ReadFull(...)
        /usr/local/go/src/io/io.go:351
github.com/gocql/gocql.readHeader({0xc97860?, 0xc000317b60?}, {0xc0003a8948?, 0x9?, 0x9?})
        /home/kiparis/projects/gocql/frame.go:436 +0xb6
github.com/gocql/gocql.(*Conn).recv(0xc0003a8900, {0xc9b758, 0xc0003614f0})
        /home/kiparis/projects/gocql/conn.go:659 +0x130
github.com/gocql/gocql.(*Conn).serve(0x47943a?, {0xc9b758, 0xc0003614f0})
        /home/kiparis/projects/gocql/conn.go:576 +0x47
created by github.com/gocql/gocql.(*Conn).init
        /home/kiparis/projects/gocql/conn.go:325 +0x6d4

goroutine 1357 [select]:
github.com/gocql/gocql.(*Session).reconnectDownedHosts(0xc000600800, 0x0?)
        /home/kiparis/projects/gocql/session.go:374 +0x177
created by github.com/gocql/gocql.(*Session).init
        /home/kiparis/projects/gocql/session.go:321 +0x108c

goroutine 1358 [select]:
github.com/gocql/gocql.(*writeCoalescer).writeFlusher(0xc0004f8b40, 0x0?)
        /home/kiparis/projects/gocql/conn.go:888 +0x310
created by github.com/gocql/gocql.newWriteCoalescer
        /home/kiparis/projects/gocql/conn.go:785 +0x305

goroutine 1382 [IO wait]:
internal/poll.runtime_pollWait(0x7f343867ada0, 0x72)
        /usr/local/go/src/runtime/netpoll.go:306 +0x89
internal/poll.(*pollDesc).wait(0xc0001e7698, 0xc0000b8000?, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0xbd
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc0001e7680, {0xc0000b8000, 0x1000, 0x1000})
        /usr/local/go/src/internal/poll/fd_unix.go:167 +0x405
net.(*netFD).Read(0xc0001e7680, {0xc0000b8000, 0x1000, 0x1000})
        /usr/local/go/src/net/fd_posix.go:55 +0x51
net.(*conn).Read(0xc00037a008, {0xc0000b8000, 0x1000, 0x1000})
        /usr/local/go/src/net/net.go:183 +0xb1
bufio.(*Reader).Read(0xc0004f8840, {0xc0000ba048, 0x1, 0x9})
        /usr/local/go/src/bufio/bufio.go:237 +0x4f3
io.ReadAtLeast({0xc97860, 0xc0004f8840}, {0xc0000ba048, 0x1, 0x9}, 0x1)
        /usr/local/go/src/io/io.go:332 +0xde
io.ReadFull(...)
        /usr/local/go/src/io/io.go:351
github.com/gocql/gocql.readHeader({0xc97860?, 0xc0004f8840?}, {0xc0000ba048?, 0x9?, 0x9?})
        /home/kiparis/projects/gocql/frame.go:436 +0xb6
github.com/gocql/gocql.(*Conn).recv(0xc0000ba000, {0xc9b758, 0xc0004de7d0})
        /home/kiparis/projects/gocql/conn.go:659 +0x130
github.com/gocql/gocql.(*Conn).serve(0x47943a?, {0xc9b758, 0xc0004de7d0})
        /home/kiparis/projects/gocql/conn.go:576 +0x47
created by github.com/gocql/gocql.(*Conn).init
        /home/kiparis/projects/gocql/conn.go:325 +0x6d4

goroutine 54909 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc0001e0a50, 0x1a19c)
        /usr/local/go/src/runtime/sema.go:527 +0x14c
sync.(*Cond).Wait(0xc0001e0a40)
        /usr/local/go/src/sync/cond.go:70 +0xa5
github.com/gocql/gocql.(*writeCoalescer).Write(0xc0004f8b40, {0xc001c68900, 0x47, 0x80})
        /home/kiparis/projects/gocql/conn.go:857 +0x39f
github.com/gocql/gocql.(*Conn).Write(0xc0003a8900, {0xc001c68900, 0x47, 0x80})
        /home/kiparis/projects/gocql/conn.go:332 +0x6e
github.com/gocql/gocql.(*framer).finishWrite(0xc001c489c0)
        /home/kiparis/projects/gocql/frame.go:779 +0x63c
github.com/gocql/gocql.(*writePrepareFrame).writeFrame(0xc001d8ede0, 0xc001c489c0, 0x6680)
        /home/kiparis/projects/gocql/frame.go:861 +0xe16
github.com/gocql/gocql.(*Conn).exec(0xc0003a8900, {0xc9b758, 0xc0003614f0}, {0xc981e0, 0xc001d8ede0}, {0x0, 0x0?})
        /home/kiparis/projects/gocql/conn.go:934 +0x4d9
github.com/gocql/gocql.(*Conn).prepareStatement.func2()
        /home/kiparis/projects/gocql/conn.go:1048 +0x293
created by github.com/gocql/gocql.(*Conn).prepareStatement
        /home/kiparis/projects/gocql/conn.go:1035 +0x3a5
exit status 2
FAIL    github.com/gocql/gocql  300.076s

What did you expect to see?

I expect it to work ok like with all other more older Scylla versions 5.1.x, 5.0.x and 4.6.x which provide following result:

==> Running integration scylla tests with args: -gocql.timeout=60s -proto=4 -rf=1 -clusterSize=1 -autowait=2000ms -compressor=snappy -gocql.cversion=3.11.4 -cluster=192.168.100.11
2023/06/08 11:46:57 logger.go:35: error: failed to connect to "[HostInfo hostname=\"192.168.100.11\" connectAddress=\"192.168.100.11\" peer=\"<nil>\" rpc_address=\"192.168.100.11\" broadcast_address=\"192.168.100.11\" preferred_ip=\"<nil>\" connect_addr=\"192.168.100.11\" connect_addr_source=\"connect_address\" port=9042 data_centre=\"datacenter1\" rack=\"rack1\" host_id=\"09183270-1045-4cb8-adde-5aed8578c76b\" version=\"v3.0.8\" state=UP num_tokens=1]" due to error: context canceled
2023/06/08 11:46:57 logger.go:35: error: failed to connect to "[HostInfo hostname=\"192.168.100.11\" connectAddress=\"192.168.100.11\" peer=\"<nil>\" rpc_address=\"192.168.100.11\" broadcast_address=\"192.168.100.11\" preferred_ip=\"<nil>\" connect_addr=\"192.168.100.11\" connect_addr_source=\"connect_address\" port=9042 data_centre=\"datacenter1\" rack=\"rack1\" host_id=\"09183270-1045-4cb8-adde-5aed8578c76b\" version=\"v3.0.8\" state=UP num_tokens=1]" due to error: Keyspace 'invalidKeyspace' does not exist
2023/06/08 11:47:01 logger.go:35: gocql: unable to dial control conn 127.0.0.1:1: dial tcp :0->127.0.0.1:1: connect: connection refused
2023/06/08 11:47:18 logger.go:35: error: failed to connect to "[HostInfo hostname=\"192.168.100.11\" connectAddress=\"192.168.100.11\" peer=\"<nil>\" rpc_address=\"192.168.100.11\" broadcast_address=\"192.168.100.11\" preferred_ip=\"<nil>\" connect_addr=\"192.168.100.11\" connect_addr_source=\"connect_address\" port=9042 data_centre=\"datacenter1\" rack=\"rack1\" host_id=\"09183270-1045-4cb8-adde-5aed8578c76b\" version=\"v3.0.8\" state=UP num_tokens=1]" due to error: context canceled
2023/06/08 11:47:37 logger.go:35: scylla: 192.168.100.11:9042 connection to shard-aware address 192.168.100.11:19042 resulted in wrong shard being assigned; please check that you are not behind a NAT or AddressTranslater which changes source ports; falling back to non-shard-aware port for 5m0s
PASS
ok      github.com/gocql/gocql  43.496s

What did you see instead?

I see constant connection 300s timeouts.

---

If you are having connectivity related issues please share the following additional information

Describe your Cassandra cluster

please provide the following information

• output of 'nodetool status'
• output of 'SELECT peer, rpc_address FROM system.peers'
• rebuild your application with the 'gocql_debug' tag and post the output

nodetool status:

root@d0e0d92882dd:/# nodetool status
nodetool: Unable to connect to Scylla API server: java.net.ConnectException: Connection refused (Connection refused)
See 'nodetool help' or 'nodetool help <command>'.

CQLSH:

root@d0e0d92882dd:/# cqlsh
Connected to  at 192.168.100.11:9042.
[cqlsh 5.0.1 | Cassandra 3.0.8 | CQL spec 3.3.1 | Native protocol v4]
Use HELP for help.
cqlsh> SELECT peer, rpc_address FROM system.peers;

 peer | rpc_address
------+-------------

(0 rows)

With gocql_debug:
gocql_debug_for_issue_126.log

[mykaul]

127.0.0.1:1 looks wrong to me and can explain the connection refused.

[vponomaryov]

127.0.0.1:1 looks wrong to me and can explain the connection refused.

Only Scylla docker image version was changed.
So, it can be said now that the gocql integration tests are not compatible with the latest Scylla releases.

[fruch]

running the same with master, and in 300s it got killed by my kernel:

Jun 08 14:20:13 fruch-laptop kernel: Out of memory: Killed process 3160814 (gocql.test) total-vm:13869872kB, anon-rss:11142664kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:22996

[roydahan]

@avelanarius / @zimnx can someone please take a look at this?

[avelanarius]

I will look into it.

[dkropachev]

This test failure due to the unabling execute prepared statement due to the getting 0x2500 error from the scylla server:
https://github.com/apache/cassandra/blob/7337fc0/doc/native_protocol_v5.spec#L1414-L1417

Same problem from java-driver scylladb/java-driver#124

[fruch]

This test failure due to the unabling execute prepared statement due to the getting 0x2500 error from the scylla server: https://github.com/apache/cassandra/blob/7337fc0/doc/native_protocol_v5.spec#L1414-L1417

Same problem from java-driver scylladb/java-driver#124

that issue in java-driver wasn't fixed, and was fixed on scylla end in some way.
so I'm guessing it's not exactly the same.

[dkropachev]

I have run this test locally on 5.2.3 and 5.2.1 100 times and it did not fail even once.
Also I run whole suite for 10 times and it also did not fail even once.
Which go version are we using there ?

[dkropachev]

I ran it via integration.sh and it did fail, will look at the difference tomorrow

[dkropachev]

Most likely it is driver issue, it's logic does not cover the case when prepared statement cache on scylla is under high pressure which could lead to being unable to run big batches as prepared statements, because scylla is expunging statements while driver preparing them.

If this is true, then all our and upstream drivers have exact same issue, which is not regression and only regression there is that scylla tightened prepared statement buffer.

I have opened issue on the scylla repo to confirm it.
Fast fix would be to increase scylla memory limit in docker-compose.yaml

[avelanarius]

This test failure due to the unabling execute prepared statement due to the getting 0x2500 error from the scylla server: https://github.com/apache/cassandra/blob/7337fc0/doc/native_protocol_v5.spec#L1414-L1417
Same problem from java-driver scylladb/java-driver#124

that issue in java-driver wasn't fixed, and was fixed on scylla end in some way. so I'm guessing it's not exactly the same.

That particular one was fixed by: scylladb/scylladb#10456 (seehttps://github.com/scylladb/scylladb/issues/10440)

[dkropachev]

From scylladb/scylladb#14338 :

Refs #10440 - a similar issue from the past. That issue was fixed by changing the eviction behavior of loading_cache (note that the fix should be improved: #10674).

One possible improvement to prevent infinite spining:

The way it works now (looking at Rust Driver, but others are similar): let's suppose that there is a batch statement of N prepared statements and the driver sends this statement to Scylla. Scylla might respond that some statement in that batch is not prepared (not present in prepared statements cache). The driver will then re-prepare all statements in that batch. However, if the cache is too small, it won't be possible for all statements to be in the cache at the same time.

Improvement:

The driver(s) should not infinite loop, instead impose some limit of how many times it will try to re-prepare (some draft PR from the past: scylladb/java-driver#143)
When Scylla receives a batch of N prepared statements, it should check if the prepared statements cache is large enough and return some error if it's not.

Which confirms my assumption, it is needed to be fixed on gocql side