Skip to content

scylla-bench v0.1.8 got fatal error: runtime: out of memory #89

New issue
New issue
Open
Open
scylla-bench v0.1.8 got fatal error: runtime: out of memory#89
@yarongilor

Description

@yarongilor
yarongilor
opened on Feb 7, 2022

Installation details
Kernel version: 5.11.0-1022-aws
Scylla version (or git commit hash): 4.6.rc5-0.20220203.5694ec189 with build-id f5d85bf5abe6d2f9fd3487e2469ce1c34304cc14
Cluster size: 3 3 3 nodes (i3.2xlarge)
Scylla running with shards number (live nodes):
longevity-counters-multidc-4-6-db-node-ec7ffc0b-1 (34.254.228.100 | 10.0.0.60): 8 shards
longevity-counters-multidc-4-6-db-node-ec7ffc0b-2 (52.19.209.202 | 10.0.3.34): 8 shards
longevity-counters-multidc-4-6-db-node-ec7ffc0b-3 (54.75.1.178 | 10.0.2.127): 8 shards
longevity-counters-multidc-4-6-db-node-ec7ffc0b-4 (52.34.214.217 | 10.0.3.8): 8 shards
longevity-counters-multidc-4-6-db-node-ec7ffc0b-5 (35.88.112.46 | 10.0.2.163): 8 shards
longevity-counters-multidc-4-6-db-node-ec7ffc0b-6 (35.88.221.142 | 10.0.2.18): 8 shards
longevity-counters-multidc-4-6-db-node-ec7ffc0b-7 (44.200.34.43 | 10.0.3.238): 8 shards
longevity-counters-multidc-4-6-db-node-ec7ffc0b-8 (44.195.21.232 | 10.0.0.201): 8 shards
longevity-counters-multidc-4-6-db-node-ec7ffc0b-9 (3.228.17.3 | 10.0.3.20): 8 shards
OS (RHEL/CentOS/Ubuntu/AWS AMI): ami-06df116ea1c70171c ami-0d2bccabf736b9dd0 ami-097f7b2efb7723718 (aws: eu-west-1 us-west-2 us-east-1)

Test: longevity-counters-multidc-test
Test name: longevity_test.LongevityTest.test_custom_time
Test config file(s):

  • longevity-counters-multidc.yaml

Issue description

====================================

scylla-bench got the following OOM with version 0.1.8:
from the loader-set log of scylla-bench-l0-48d0dafb-2e83-46b1-8add-6e7dc9fc431c.log:
(./longevity-counters-multidc-4-6-loader-node-ec7ffc0b-1/scylla-bench-l0-48d0dafb-2e83-46b1-8add-6e7dc9fc431c.log)

2022/02/06 16:51:00 writev tcp 10.0.2.255:39009->35.88.221.142:9042: i/o timeout
fatal error: runtime: out of memory

runtime stack:
runtime.throw({0x6fbe3b, 0x400000})
	/usr/local/go/src/runtime/panic.go:1198 +0x71
runtime.sysMap(0xc5cf800000, 0x1, 0xc000080800)
	/usr/local/go/src/runtime/mem_linux.go:169 +0x96
runtime.(*mheap).grow(0x93c080, 0x43bdef)
	/usr/local/go/src/runtime/mheap.go:1393 +0x225
runtime.(*mheap).allocSpan(0x93c080, 0x1, 0x0, 0x12)
	/usr/local/go/src/runtime/mheap.go:1179 +0x165
runtime.(*mheap).alloc.func1()
	/usr/local/go/src/runtime/mheap.go:913 +0x69
runtime.systemstack()
	/usr/local/go/src/runtime/asm_amd64.s:383 +0x49

goroutine 671 [running]:
runtime.systemstack_switch()
	/usr/local/go/src/runtime/asm_amd64.s:350 fp=0xc00c367820 sp=0xc00c367818 pc=0x462220
runtime.(*mheap).alloc(0x761f50, 0xc0000180b0, 0x68, 0x1)
	/usr/local/go/src/runtime/mheap.go:907 +0x73 fp=0xc00c367870 sp=0xc00c367820 pc=0x426ed3
runtime.(*mcentral).grow(0x2000)
	/usr/local/go/src/runtime/mcentral.go:241 +0x65 fp=0xc00c3678b8 sp=0xc00c367870 pc=0x418865
runtime.(*mcentral).cacheSpan(0x94d048)
	/usr/local/go/src/runtime/mcentral.go:161 +0x69e fp=0xc00c367930 sp=0xc00c3678b8 pc=0x41869e
runtime.(*mcache).refill(0x7febc181d5b8, 0x12)
	/usr/local/go/src/runtime/mcache.go:162 +0xaf fp=0xc00c367980 sp=0xc00c367930 pc=0x4178cf
runtime.(*mcache).nextFree(0x7febc181d5b8, 0x12)
	/usr/local/go/src/runtime/malloc.go:880 +0x85 fp=0xc00c3679c8 sp=0xc00c367980 pc=0x40da85
runtime.mallocgc(0x70, 0x6a3580, 0x1)
	/usr/local/go/src/runtime/malloc.go:1071 +0x4e8 fp=0xc00c367a48 sp=0xc00c3679c8 pc=0x40e108
runtime.newobject(0xc5cf771b90)
	/usr/local/go/src/runtime/malloc.go:1228 +0x27 fp=0xc00c367a70 sp=0xc00c367a48 pc=0x40e627
main.DoReadsFromTable.func1(0xc0003bc990)
	/tmp/sb_install/scylla-bench-0.1.8/modes.go:479 +0x78e fp=0xc00c367c78 sp=0xc00c367a70 pc=0x680bee
main.RunTest(0xc0003bc990, {0x763ad0, 0xc00020c570}, {0x75f8a0, 0x952d80}, 0xc00c367e40)
	/tmp/sb_install/scylla-bench-0.1.8/modes.go:126 +0x1b0 fp=0xc00c367db0 sp=0xc00c367c78 pc=0x67d5d0
main.DoReadsFromTable({0x6f8ac5, 0xd}, 0x0, 0x0, {0x763ad0, 0xc00020c570}, {0x75f8a0, 0x952d80})
	/tmp/sb_install/scylla-bench-0.1.8/modes.go:427 +0x16e fp=0xc00c367ea0 sp=0xc00c367db0 pc=0x6802ce
main.DoCounterReads(0x7ffe23e16d11, 0x0, {0x763ad0, 0xc00020c570}, {0x75f8a0, 0x952d80})
	/tmp/sb_install/scylla-bench-0.1.8/modes.go:384 +0x53 fp=0xc00c367ef0 sp=0xc00c367ea0 pc=0x67f9f3
main.main.func4(0x0, 0x0, {0x75f8a0, 0x952d80})
	/tmp/sb_install/scylla-bench-0.1.8/main.go:527 +0x105 fp=0xc00c367f88 sp=0xc00c367ef0 pc=0x67c7e5
main.RunConcurrently.func1(0x5c)
	/tmp/sb_install/scylla-bench-0.1.8/modes.go:78 +0x66 fp=0xc00c367fc8 sp=0xc00c367f88 pc=0x67d326
main.RunConcurrently·dwrap·2()
	/tmp/sb_install/scylla-bench-0.1.8/modes.go:79 +0x2d fp=0xc00c367fe0 sp=0xc00c367fc8 pc=0x67d28d
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc00c367fe8 sp=0xc00c367fe0 pc=0x464441
created by main.RunConcurrently
	/tmp/sb_install/scylla-bench-0.1.8/modes.go:76 +0x10c

goroutine 1 [runnable]:
github.com/HdrHistogram/hdrhistogram-go.(*Histogram).getCountAtIndex(...)
	/home/centos/go/pkg/mod/github.com/!hdr!histogram/[email protected]/hdr.go:595
github.com/HdrHistogram/hdrhistogram-go.(*iterator).nextCountAtIdx(0x4e714e7c6b18b182, 0x630d02662c804285)
	/home/centos/go/pkg/mod/github.com/!hdr!histogram/[email protected]/hdr.go:662 +0x65
github.com/HdrHistogram/hdrhistogram-go.(*iterator).next(0xc000107530)
	/home/centos/go/pkg/mod/github.com/!hdr!histogram/[email protected]/hdr.go:670 +0x25
github.com/HdrHistogram/hdrhistogram-go.(*rIterator).next(...)
	/home/centos/go/pkg/mod/github.com/!hdr!histogram/[email protected]/hdr.go:683
github.com/HdrHistogram/hdrhistogram-go.(*Histogram).Merge(0xf00000009, 0x4000000000a)
	/home/centos/go/pkg/mod/github.com/!hdr!histogram/[email protected]/hdr.go:177 +0x98
github.com/scylladb/scylla-bench/pkg/results.(*MergedResult).AddResult(0xc07adacba0, {0x0, 0x0, 0x6, 0x6, 0x0, {0x0, 0x0, 0x0}, 0xc09f0c2600, ...})
	/tmp/sb_install/scylla-bench-0.1.8/pkg/results/merged_result.go:53 +0x1b0
github.com/scylladb/scylla-bench/pkg/results.(*TestResults).GetResultsFromThreadsAndMerge(0xc0002e6600)
	/tmp/sb_install/scylla-bench-0.1.8/pkg/results/thread_results.go:60 +0x89
github.com/scylladb/scylla-bench/pkg/results.(*TestResults).GetTotalResults(0xc0002e6600)
	/tmp/sb_install/scylla-bench-0.1.8/pkg/results/thread_results.go:82 +0xcd
main.main()
	/tmp/sb_install/scylla-bench-0.1.8/main.go:530 +0x31f7

goroutine 80 [select]:
github.com/gocql/gocql.(*Conn).heartBeat(0x0, {0x761f18, 0xc00020ebc0})
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/conn.go:602 +0x125
created by github.com/gocql/gocql.(*Conn).init
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/conn.go:313 +0x2af

goroutine 25 [IO wait]:
internal/poll.runtime_pollWait(0x7feb9848b038, 0x72)
	/usr/local/go/src/runtime/netpoll.go:234 +0x89
internal/poll.(*pollDesc).wait(0xc00009c400, 0xc000144000, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc00009c400, {0xc000144000, 0x1000, 0x1000})
	/usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a
net.(*netFD).Read(0xc00009c400, {0xc000144000, 0x407029, 0x6b42a0})
	/usr/local/go/src/net/fd_posix.go:56 +0x29
net.(*conn).Read(0xc000010038, {0xc000144000, 0x123, 0x1})
	/usr/local/go/src/net/net.go:183 +0x45
bufio.(*Reader).Read(0xc00005c300, {0xc000146048, 0x1, 0x0})
	/usr/local/go/src/bufio/bufio.go:227 +0x1b4
io.ReadAtLeast({0x75b9e0, 0xc00005c300}, {0xc000146048, 0x1, 0x9}, 0x1)
	/usr/local/go/src/io/io.go:328 +0x9a
io.ReadFull(...)
	/usr/local/go/src/io/io.go:347
github.com/gocql/gocql.readHeader({0x75b9e0, 0xc00005c300}, {0xc000146048, 0x9, 0x9})
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/frame.go:452 +0xa5
github.com/gocql/gocql.(*Conn).recv(0xc000146000, {0x761f18, 0xc00005a240})
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/conn.go:645 +0xed
github.com/gocql/gocql.(*Conn).serve(0x0, {0x761f18, 0xc00005a240})
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/conn.go:562 +0x2e
created by github.com/gocql/gocql.(*Conn).init
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/conn.go:312 +0x248

goroutine 79 [runnable]:
internal/poll.runtime_pollWait(0x7feb9848a558, 0x72)
	/usr/local/go/src/runtime/netpoll.go:234 +0x89
internal/poll.(*pollDesc).wait(0xc00009cc00, 0xc000250000, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc00009cc00, {0xc000250000, 0x1000, 0x1000})
	/usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a
net.(*netFD).Read(0xc00009cc00, {0xc000250000, 0x407029, 0x6b42a0})
	/usr/local/go/src/net/fd_posix.go:56 +0x29
net.(*conn).Read(0xc00020a0c8, {0xc000250000, 0x40, 0x1})
	/usr/local/go/src/net/net.go:183 +0x45
bufio.(*Reader).Read(0xc0002018c0, {0xc00023a4c8, 0x1, 0x0})
	/usr/local/go/src/bufio/bufio.go:227 +0x1b4
io.ReadAtLeast({0x75b9e0, 0xc0002018c0}, {0xc00023a4c8, 0x1, 0x9}, 0x1)
	/usr/local/go/src/io/io.go:328 +0x9a
io.ReadFull(...)
	/usr/local/go/src/io/io.go:347
github.com/gocql/gocql.readHeader({0x75b9e0, 0xc0002018c0}, {0xc00023a4c8, 0x9, 0x9})
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/frame.go:452 +0xa5
github.com/gocql/gocql.(*Conn).recv(0xc00023a480, {0x761f18, 0xc00020ebc0})
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/conn.go:645 +0xed
github.com/gocql/gocql.(*Conn).serve(0xc0000f00e0, {0x761f18, 0xc00020ebc0})
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/conn.go:562 +0x2e
created by github.com/gocql/gocql.(*Conn).init
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/conn.go:312 +0x248

goroutine 51 [select, 2 minutes]:
github.com/gocql/gocql.(*eventDebouncer).flusher(0xc00019a190)
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/events.go:39 +0x67
created by github.com/gocql/gocql.newEventDebouncer
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/events.go:27 +0x11f

goroutine 52 [select, 54 minutes]:
github.com/gocql/gocql.(*eventDebouncer).flusher(0xc00019a230)
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/events.go:39 +0x67
created by github.com/gocql/gocql.newEventDebouncer
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/events.go:27 +0x11f

goroutine 72 [select]:
github.com/gocql/gocql.(*Conn).heartBeat(0x0, {0x761f18, 0xc00005a380})
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/conn.go:602 +0x125
created by github.com/gocql/gocql.(*Conn).init
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/conn.go:313 +0x2af

goroutine 37 [select]:
github.com/gocql/gocql.(*Conn).heartBeat(0x0, {0x761f18, 0xc00009ab40})
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/conn.go:602 +0x125
created by github.com/gocql/gocql.(*Conn).init
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/conn.go:313 +0x2af

goroutine 26 [select]:
github.com/gocql/gocql.(*Conn).heartBeat(0x0, {0x761f18, 0xc00005a240})
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/conn.go:602 +0x125
created by github.com/gocql/gocql.(*Conn).init
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/conn.go:313 +0x2af

goroutine 35 [select]:
github.com/gocql/gocql.(*writeCoalescer).writeFlusher(0xc0002008a0, 0xc000187480)
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/conn.go:874 +0x1b9
created by github.com/gocql/gocql.newWriteCoalescer
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/conn.go:771 +0x17a

goroutine 36 [runnable]:
internal/poll.runtime_pollWait(0x7feb9848af50, 0x72)
	/usr/local/go/src/runtime/netpoll.go:234 +0x89
internal/poll.(*pollDesc).wait(0xc00009c580, 0xc0000f6000, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc00009c580, {0xc0000f6000, 0x1000, 0x1000})
	/usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a
net.(*netFD).Read(0xc00009c580, {0xc0000f6000, 0x407029, 0x6b42a0})
	/usr/local/go/src/net/fd_posix.go:56 +0x29
net.(*conn).Read(0xc0000960f8, {0xc0000f6000, 0x41, 0x1})
	/usr/local/go/src/net/net.go:183 +0x45
bufio.(*Reader).Read(0xc00008eba0, {0xc0000f8048, 0x1, 0x0})
	/usr/local/go/src/bufio/bufio.go:227 +0x1b4
io.ReadAtLeast({0x75b9e0, 0xc00008eba0}, {0xc0000f8048, 0x1, 0x9}, 0x1)
	/usr/local/go/src/io/io.go:328 +0x9a
io.ReadFull(...)
	/usr/local/go/src/io/io.go:347
github.com/gocql/gocql.readHeader({0x75b9e0, 0xc00008eba0}, {0xc0000f8048, 0x9, 0x9})
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/frame.go:452 +0xa5
github.com/gocql/gocql.(*Conn).recv(0xc0000f8000, {0x761f18, 0xc00009ab40})
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/conn.go:645 +0xed
github.com/gocql/gocql.(*Conn).serve(0x0, {0x761f18, 0xc00009ab40})
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/conn.go:562 +0x2e
created by github.com/gocql/gocql.(*Conn).init
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/conn.go:312 +0x248

goroutine 39 [select]:
github.com/gocql/gocql.(*controlConn).heartBeat(0x0)
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/control.go:70 +0x10d
created by github.com/gocql/gocql.(*controlConn).connect
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/control.go:259 +0x13b

goroutine 75 [select]:
github.com/gocql/gocql.(*Conn).heartBeat(0x0, {0x761f18, 0xc00005a280})
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/conn.go:602 +0x125
created by github.com/gocql/gocql.(*Conn).init
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/conn.go:313 +0x2af

goroutine 529 [select]:
github.com/gocql/gocql.(*Conn).heartBeat(0x0, {0x761f18, 0xc00005be80})
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/conn.go:602 +0x125
created by github.com/gocql/gocql.(*Conn).init
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/conn.go:313 +0x2af

goroutine 86 [select]:
github.com/gocql/gocql.(*Conn).heartBeat(0x0, {0x761f18, 0xc00005a400})
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/conn.go:602 +0x125
created by github.com/gocql/gocql.(*Conn).init
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/conn.go:313 +0x2af

goroutine 69 [select]:
github.com/gocql/gocql.(*Conn).heartBeat(0x0, {0x761f18, 0xc00020e6c0})
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/conn.go:602 +0x125
created by github.com/gocql/gocql.(*Conn).init
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/conn.go:313 +0x2af

goroutine 73 [select]:
github.com/gocql/gocql.(*writeCoalescer).writeFlusher(0xc000201800, 0xc00005a380)
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/conn.go:874 +0x1b9
created by github.com/gocql/gocql.newWriteCoalescer
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/conn.go:771 +0x17a

goroutine 48 [select]:
github.com/gocql/gocql.(*writeCoalescer).writeFlusher(0xc0002014a0, 0x0)
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/conn.go:866 +0x13e
created by github.com/gocql/gocql.newWriteCoalescer
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/conn.go:771 +0x17a

goroutine 98 [select]:
github.com/gocql/gocql.(*writeCoalescer).writeFlusher(0xc00005d140, 0x0)
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/conn.go:866 +0x13e
created by github.com/gocql/gocql.newWriteCoalescer
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/conn.go:771 +0x17a

goroutine 84 [select]:
github.com/gocql/gocql.(*writeCoalescer).writeFlusher(0xc00008f560, 0x0)
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/conn.go:866 +0x13e
created by github.com/gocql/gocql.newWriteCoalescer
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/conn.go:771 +0x17a

goroutine 67 [select]:
github.com/gocql/gocql.(*writeCoalescer).writeFlusher(0xc0002015c0, 0xc00020e700)
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/conn.go:866 +0x13e
created by github.com/gocql/gocql.newWriteCoalescer
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/conn.go:771 +0x17a

goroutine 70 [select]:
github.com/gocql/gocql.(*writeCoalescer).writeFlusher(0xc0002016e0, 0xc00020e6c0)
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/conn.go:866 +0x13e
created by github.com/gocql/gocql.newWriteCoalescer
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/conn.go:771 +0x17a

goroutine 33 [select]:
github.com/gocql/gocql.(*writeCoalescer).writeFlusher(0xc00008f440, 0xc00002df88)
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/conn.go:866 +0x13e
created by github.com/gocql/gocql.newWriteCoalescer
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/conn.go:771 +0x17a

goroutine 74 [runnable]:
internal/poll.runtime_pollWait(0x7feb9848ae68, 0x72)
	/usr/local/go/src/runtime/netpoll.go:234 +0x89
internal/poll.(*pollDesc).wait(0xc00009c680, 0xc00014a000, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc00009c680, {0xc00014a000, 0x1000, 0x1000})
	/usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a
net.(*netFD).Read(0xc00009c680, {0xc00014a000, 0x407029, 0x6b42a0})
	/usr/local/go/src/net/fd_posix.go:56 +0x29
net.(*conn).Read(0xc000010048, {0xc00014a000, 0x3f, 0x1})
	/usr/local/go/src/net/net.go:183 +0x45
bufio.(*Reader).Read(0xc00005c480, {0xc0001461c8, 0x1, 0x0})
	/usr/local/go/src/bufio/bufio.go:227 +0x1b4
io.ReadAtLeast({0x75b9e0, 0xc00005c480}, {0xc0001461c8, 0x1, 0x9}, 0x1)
	/usr/local/go/src/io/io.go:328 +0x9a
io.ReadFull(...)
	/usr/local/go/src/io/io.go:347
github.com/gocql/gocql.readHeader({0x75b9e0, 0xc00005c480}, {0xc0001461c8, 0x9, 0x9})
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/frame.go:452 +0xa5
github.com/gocql/gocql.(*Conn).recv(0xc000146180, {0x761f18, 0xc00005a280})
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/conn.go:645 +0xed
github.com/gocql/gocql.(*Conn).serve(0x0, {0x761f18, 0xc00005a280})
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/conn.go:562 +0x2e
created by github.com/gocql/gocql.(*Conn).init
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/conn.go:312 +0x248

goroutine 99 [runnable]:
internal/poll.runtime_pollWait(0x7feb9848ac98, 0x72)
	/usr/local/go/src/runtime/netpoll.go:234 +0x89
internal/poll.(*pollDesc).wait(0xc000212380, 0xc000238000, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000212380, {0xc000238000, 0x1000, 0x1000})
	/usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a
net.(*netFD).Read(0xc000212380, {0xc000238000, 0x407029, 0x6b42a0})
	/usr/local/go/src/net/fd_posix.go:56 +0x29
net.(*conn).Read(0xc00020a0a8, {0xc000238000, 0x42, 0x1})
	/usr/local/go/src/net/net.go:183 +0x45
bufio.(*Reader).Read(0xc000200cc0, {0xc00023a048, 0x1, 0x0})
	/usr/local/go/src/bufio/bufio.go:227 +0x1b4
io.ReadAtLeast({0x75b9e0, 0xc000200cc0}, {0xc00023a048, 0x1, 0x9}, 0x1)
	/usr/local/go/src/io/io.go:328 +0x9a
io.ReadFull(...)
	/usr/local/go/src/io/io.go:347
github.com/gocql/gocql.readHeader({0x75b9e0, 0xc000200cc0}, {0xc00023a048, 0x9, 0x9})
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/frame.go:452 +0xa5
github.com/gocql/gocql.(*Conn).recv(0xc00023a000, {0x761f18, 0xc00020e640})
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/conn.go:645 +0xed
github.com/gocql/gocql.(*Conn).serve(0x0, {0x761f18, 0xc00020e640})
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/conn.go:562 +0x2e
created by github.com/gocql/gocql.(*Conn).init
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/conn.go:312 +0x248

goroutine 49 [runnable]:
internal/poll.runtime_pollWait(0x7feb9848ad80, 0x72)
	/usr/local/go/src/runtime/netpoll.go:234 +0x89
internal/poll.(*pollDesc).wait(0xc00018c700, 0xc00014e000, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc00018c700, {0xc00014e000, 0x1000, 0x1000})
	/usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a
net.(*netFD).Read(0xc00018c700, {0xc00014e000, 0x407029, 0x6b42a0})
	/usr/local/go/src/net/fd_posix.go:56 +0x29
net.(*conn).Read(0xc000010050, {0xc00014e000, 0x3e, 0x1})
	/usr/local/go/src/net/net.go:183 +0x45
bufio.(*Reader).Read(0xc00005c720, {0xc000146348, 0x1, 0x0})
	/usr/local/go/src/bufio/bufio.go:227 +0x1b4
io.ReadAtLeast({0x75b9e0, 0xc00005c720}, {0xc000146348, 0x1, 0x9}, 0x1)
	/usr/local/go/src/io/io.go:328 +0x9a
io.ReadFull(...)
	/usr/local/go/src/io/io.go:347
github.com/gocql/gocql.readHeader({0x75b9e0, 0xc00005c720}, {0xc000146348, 0x9, 0x9})
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/frame.go:452 +0xa5
github.com/gocql/gocql.(*Conn).recv(0xc000146300, {0x761f18, 0xc00005a300})
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/conn.go:645 +0xed
github.com/gocql/gocql.(*Conn).serve(0x0, {0x761f18, 0xc00005a300})
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/conn.go:562 +0x2e
created by github.com/gocql/gocql.(*Conn).init
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/conn.go:312 +0x248

goroutine 68 [runnable]:
internal/poll.runtime_pollWait(0x7feb9848aac8, 0x72)
	/usr/local/go/src/runtime/netpoll.go:234 +0x89
internal/poll.(*pollDesc).wait(0xc00018c800, 0xc00023e000, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc00018c800, {0xc00023e000, 0x1000, 0x1000})
	/usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a
net.(*netFD).Read(0xc00018c800, {0xc00023e000, 0x407029, 0x6b42a0})
	/usr/local/go/src/net/fd_posix.go:56 +0x29
net.(*conn).Read(0xc00020a0b0, {0xc00023e000, 0x3f, 0x1})
	/usr/local/go/src/net/net.go:183 +0x45
bufio.(*Reader).Read(0xc000200f60, {0xc00023a1c8, 0x1, 0x0})
	/usr/local/go/src/bufio/bufio.go:227 +0x1b4
io.ReadAtLeast({0x75b9e0, 0xc000200f60}, {0xc00023a1c8, 0x1, 0x9}, 0x1)
	/usr/local/go/src/io/io.go:328 +0x9a
io.ReadFull(...)
	/usr/local/go/src/io/io.go:347
github.com/gocql/gocql.readHeader({0x75b9e0, 0xc000200f60}, {0xc00023a1c8, 0x9, 0x9})
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/frame.go:452 +0xa5
github.com/gocql/gocql.(*Conn).recv(0xc00023a180, {0x761f18, 0xc00020e6c0})
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/conn.go:645 +0xed
github.com/gocql/gocql.(*Conn).serve(0x0, {0x761f18, 0xc00020e6c0})
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/conn.go:562 +0x2e
created by github.com/gocql/gocql.(*Conn).init
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/conn.go:312 +0x248

goroutine 82 [runnable]:
internal/poll.runtime_pollWait(0x7feb9848a8f8, 0x72)
	/usr/local/go/src/runtime/netpoll.go:234 +0x89
internal/poll.(*pollDesc).wait(0xc00009c900, 0xc000242000, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc00009c900, {0xc000242000, 0x1000, 0x1000})
	/usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a
net.(*netFD).Read(0xc00009c900, {0xc000242000, 0x407029, 0x6b42a0})
	/usr/local/go/src/net/fd_posix.go:56 +0x29
net.(*conn).Read(0xc00020a0b8, {0xc000242000, 0x3e, 0x1})
	/usr/local/go/src/net/net.go:183 +0x45
bufio.(*Reader).Read(0xc0002010e0, {0xc00023a348, 0x1, 0x0})
	/usr/local/go/src/bufio/bufio.go:227 +0x1b4
io.ReadAtLeast({0x75b9e0, 0xc0002010e0}, {0xc00023a348, 0x1, 0x9}, 0x1)
	/usr/local/go/src/io/io.go:328 +0x9a
io.ReadFull(...)
	/usr/local/go/src/io/io.go:347
github.com/gocql/gocql.readHeader({0x75b9e0, 0xc0002010e0}, {0xc00023a348, 0x9, 0x9})
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/frame.go:452 +0xa5
github.com/gocql/gocql.(*Conn).recv(0xc00023a300, {0x761f18, 0xc00020e700})
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/conn.go:645 +0xed
github.com/gocql/gocql.(*Conn).serve(0x0, {0x761f18, 0xc00020e700})
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/conn.go:562 +0x2e
created by github.com/gocql/gocql.(*Conn).init
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/conn.go:312 +0x248

goroutine 71 [runnable]:
internal/poll.runtime_pollWait(0x7feb9848a9e0, 0x72)
	/usr/local/go/src/runtime/netpoll.go:234 +0x89
internal/poll.(*pollDesc).wait(0xc000212480, 0xc000158000, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000212480, {0xc000158000, 0x1000, 0x1000})
	/usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a
net.(*netFD).Read(0xc000212480, {0xc000158000, 0x407029, 0x6b42a0})
	/usr/local/go/src/net/fd_posix.go:56 +0x29
net.(*conn).Read(0xc000010058, {0xc000158000, 0x41, 0x1})
	/usr/local/go/src/net/net.go:183 +0x45
bufio.(*Reader).Read(0xc00005c9c0, {0xc0001464c8, 0x1, 0x0})
	/usr/local/go/src/bufio/bufio.go:227 +0x1b4
io.ReadAtLeast({0x75b9e0, 0xc00005c9c0}, {0xc0001464c8, 0x1, 0x9}, 0x1)
	/usr/local/go/src/io/io.go:328 +0x9a
io.ReadFull(...)
	/usr/local/go/src/io/io.go:347
github.com/gocql/gocql.readHeader({0x75b9e0, 0xc00005c9c0}, {0xc0001464c8, 0x9, 0x9})
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/frame.go:452 +0xa5
github.com/gocql/gocql.(*Conn).recv(0xc000146480, {0x761f18, 0xc00005a380})
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/conn.go:645 +0xed
github.com/gocql/gocql.(*Conn).serve(0x0, {0x761f18, 0xc00005a380})
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/conn.go:562 +0x2e
created by github.com/gocql/gocql.(*Conn).init
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/conn.go:312 +0x248

goroutine 85 [runnable]:
internal/poll.runtime_pollWait(0x7feb9848abb0, 0x72)
	/usr/local/go/src/runtime/netpoll.go:234 +0x89
internal/poll.(*pollDesc).wait(0xc00009c780, 0xc00015a000, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc00009c780, {0xc00015a000, 0x1000, 0x1000})
	/usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a
net.(*netFD).Read(0xc00009c780, {0xc00015a000, 0x407029, 0x6b42a0})
	/usr/local/go/src/net/fd_posix.go:56 +0x29
net.(*conn).Read(0xc000010060, {0xc00015a000, 0x40, 0x1})
	/usr/local/go/src/net/net.go:183 +0x45
bufio.(*Reader).Read(0xc00005cb40, {0xc000146648, 0x1, 0x0})
	/usr/local/go/src/bufio/bufio.go:227 +0x1b4
io.ReadAtLeast({0x75b9e0, 0xc00005cb40}, {0xc000146648, 0x1, 0x9}, 0x1)
	/usr/local/go/src/io/io.go:328 +0x9a
io.ReadFull(...)
	/usr/local/go/src/io/io.go:347
github.com/gocql/gocql.readHeader({0x75b9e0, 0xc00005cb40}, {0xc000146648, 0x9, 0x9})
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/frame.go:452 +0xa5
github.com/gocql/gocql.(*Conn).recv(0xc000146600, {0x761f18, 0xc00005a400})
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/conn.go:645 +0xed
github.com/gocql/gocql.(*Conn).serve(0x0, {0x761f18, 0xc00005a400})
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/conn.go:562 +0x2e
created by github.com/gocql/gocql.(*Conn).init
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/conn.go:312 +0x248

goroutine 66 [select]:
github.com/gocql/gocql.(*Conn).heartBeat(0x0, {0x761f18, 0xc00005a300})
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/conn.go:602 +0x125
created by github.com/gocql/gocql.(*Conn).init
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/conn.go:313 +0x2af

goroutine 83 [select]:
github.com/gocql/gocql.(*Conn).heartBeat(0x0, {0x761f18, 0xc00020e700})
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/conn.go:602 +0x125
created by github.com/gocql/gocql.(*Conn).init
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/conn.go:313 +0x2af

goroutine 100 [select]:
github.com/gocql/gocql.(*Conn).heartBeat(0x0, {0x761f18, 0xc00020e640})
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/conn.go:602 +0x125
created by github.com/gocql/gocql.(*Conn).init
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/conn.go:313 +0x2af

goroutine 332 [select]:
github.com/gocql/gocql.(*Session).reconnectDownedHosts(0xc0001ba380, 0x0)
	/home/centos/go/pkg/mod/github.com/scylladb/[email protected]/session.go:321 +0x106
created by github.com/gocql/gocql.(*Session).init

====================================

Restore Monitor Stack command: $ hydra investigate show-monitor ec7ffc0b-b344-43db-9777-6db183568e53
Restore monitor on AWS instance using Jenkins job
Show all stored logs command: $ hydra investigate show-logs ec7ffc0b-b344-43db-9777-6db183568e53

Test id: ec7ffc0b-b344-43db-9777-6db183568e53

Logs:
grafana - [https://cloudius-jenkins-test.s3.amazonaws.com/ec7ffc0b-b344-43db-9777-6db183568e53/20220206_172133/grafana-screenshot-longevity-counters-multidc-test-scylla-per-server-metrics-nemesis-20220206_172345-longevity-counters-multidc-4-6-monitor-node-ec7ffc0b-1.png](https://www.google.com/url?q=https://cloudius-jenkins-test.s3.amazonaws.com/ec7ffc0b-b344-43db-9777-6db183568e53/20220206_172133/grafana-screenshot-longevity-counters-multidc-test-scylla-per-server-metrics-nemesis-20220206_172345-longevity-counters-multidc-4-6-monitor-node-ec7ffc0b-1.png%255D(https://cloudius-jenkins-test.s3.amazonaws.com/ec7ffc0b-b344-43db-9777-6db183568e53/20220206_172133/grafana-screenshot-longevity-counters-multidc-test-scylla-per-server-metrics-nemesis-20220206_172345-longevity-counters-multidc-4-6-monitor-node-ec7ffc0b-1.png)&source=gmail-html&ust=1644321554025000&usg=AOvVaw3HDEgKbq2l99B4HNUgaoLU)
grafana - [https://cloudius-jenkins-test.s3.amazonaws.com/ec7ffc0b-b344-43db-9777-6db183568e53/20220206_172133/grafana-screenshot-overview-20220206_172133-longevity-counters-multidc-4-6-monitor-node-ec7ffc0b-1.png](https://www.google.com/url?q=https://cloudius-jenkins-test.s3.amazonaws.com/ec7ffc0b-b344-43db-9777-6db183568e53/20220206_172133/grafana-screenshot-overview-20220206_172133-longevity-counters-multidc-4-6-monitor-node-ec7ffc0b-1.png%255D(https://cloudius-jenkins-test.s3.amazonaws.com/ec7ffc0b-b344-43db-9777-6db183568e53/20220206_172133/grafana-screenshot-overview-20220206_172133-longevity-counters-multidc-4-6-monitor-node-ec7ffc0b-1.png)&source=gmail-html&ust=1644321554025000&usg=AOvVaw1NGtRRtTSjjUYhAvvI-V5Y)
db-cluster - [https://cloudius-jenkins-test.s3.amazonaws.com/ec7ffc0b-b344-43db-9777-6db183568e53/20220206_173419/db-cluster-ec7ffc0b.tar.gz](https://www.google.com/url?q=https://cloudius-jenkins-test.s3.amazonaws.com/ec7ffc0b-b344-43db-9777-6db183568e53/20220206_173419/db-cluster-ec7ffc0b.tar.gz%255D(https://cloudius-jenkins-test.s3.amazonaws.com/ec7ffc0b-b344-43db-9777-6db183568e53/20220206_173419/db-cluster-ec7ffc0b.tar.gz)&source=gmail-html&ust=1644321554025000&usg=AOvVaw2gBqC6DDm6Lrbp1S0efTuN)
loader-set - [https://cloudius-jenkins-test.s3.amazonaws.com/ec7ffc0b-b344-43db-9777-6db183568e53/20220206_173419/loader-set-ec7ffc0b.tar.gz](https://www.google.com/url?q=https://cloudius-jenkins-test.s3.amazonaws.com/ec7ffc0b-b344-43db-9777-6db183568e53/20220206_173419/loader-set-ec7ffc0b.tar.gz%255D(https://cloudius-jenkins-test.s3.amazonaws.com/ec7ffc0b-b344-43db-9777-6db183568e53/20220206_173419/loader-set-ec7ffc0b.tar.gz)&source=gmail-html&ust=1644321554025000&usg=AOvVaw2LBMsL-uiQIHVZLMrXuV0L)
monitor-set - [https://cloudius-jenkins-test.s3.amazonaws.com/ec7ffc0b-b344-43db-9777-6db183568e53/20220206_173419/monitor-set-ec7ffc0b.tar.gz](https://www.google.com/url?q=https://cloudius-jenkins-test.s3.amazonaws.com/ec7ffc0b-b344-43db-9777-6db183568e53/20220206_173419/monitor-set-ec7ffc0b.tar.gz%255D(https://cloudius-jenkins-test.s3.amazonaws.com/ec7ffc0b-b344-43db-9777-6db183568e53/20220206_173419/monitor-set-ec7ffc0b.tar.gz)&source=gmail-html&ust=1644321554025000&usg=AOvVaw0by3rWi6sHGCB5cMkwLy-b)
sct - [https://cloudius-jenkins-test.s3.amazonaws.com/ec7ffc0b-b344-43db-9777-6db183568e53/20220206_173419/sct-runner-ec7ffc0b.tar.gz](https://www.google.com/url?q=https://cloudius-jenkins-test.s3.amazonaws.com/ec7ffc0b-b344-43db-9777-6db183568e53/20220206_173419/sct-runner-ec7ffc0b.tar.gz%255D(https://cloudius-jenkins-test.s3.amazonaws.com/ec7ffc0b-b344-43db-9777-6db183568e53/20220206_173419/sct-runner-ec7ffc0b.tar.gz)&source=gmail-html&ust=1644321554026000&usg=AOvVaw2Ae2VgboaRnnRSgWtjEQEK)

Jenkins job URL

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions