roachtest: large-schema-benchmark/multiregion=false/tables=1000000 failed

[cockroach-teamcity]

roachtest.large-schema-benchmark/multiregion=false/tables=1000000 failed with artifacts on master @ faea13af3929107bed0057c7e8b2abe3efc9ab4d:

Failed with:

(cluster.go:2533).Run: full command output in run_190726.343139957_n10_cockroach-workload-i.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/large-schema-benchmark/multiregion=false/tables=1000000/run_1

Cluster Node to Ip Mapping:

Node	Public IP	Private IP
teamcity-21252188-1773467964-94-n10cpu16-0001	34.73.29.216	10.142.0.81
teamcity-21252188-1773467964-94-n10cpu16-0002	35.231.142.145	10.142.0.70
teamcity-21252188-1773467964-94-n10cpu16-0003	35.196.145.179	10.142.0.99
teamcity-21252188-1773467964-94-n10cpu16-0004	34.148.149.209	10.142.0.71
teamcity-21252188-1773467964-94-n10cpu16-0005	34.139.97.57	10.142.0.87
teamcity-21252188-1773467964-94-n10cpu16-0006	34.139.116.176	10.142.0.73
teamcity-21252188-1773467964-94-n10cpu16-0007	34.24.73.24	10.142.0.67
teamcity-21252188-1773467964-94-n10cpu16-0008	35.231.2.134	10.142.0.62
teamcity-21252188-1773467964-94-n10cpu16-0009	34.75.21.64	10.142.0.68
teamcity-21252188-1773467964-94-n10cpu16-0010	34.23.53.155	10.142.0.100

Parameters:

• arch=amd64
• cloud=gce
• coverageBuild=false
• cpu=16
• diskCount=0
• encrypted=false
• fs=ext4
• localSSD=false
• runtimeAssertionsBuild=false

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

/cc @cockroachdb/sql-foundations

This test on roachdash | Improve this report!

Jira issue: CRDB-61590

[shghasemi]

/investigate

[github-actions]

Investigation did not produce findings. Check the workflow run for details.

[shghasemi]

full command output in run_190726.343139957_n10_cockroach-workload-i.log: COMMAND_PROBLEM: exit status 1
(1) attached stack trace
  -- stack trace:
  | main.(*clusterImpl).Run
  | 	pkg/cmd/roachtest/cluster.go:2533
  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.setupTPCC.func2
  | 	pkg/cmd/roachtest/tests/tpcc.go:365
  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.setupTPCC
  | 	pkg/cmd/roachtest/tests/tpcc.go:370
  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runLargeSchemaIntrospectionBenchmark
  | 	pkg/cmd/roachtest/tests/large_schema_benchmark.go:446
  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerLargeSchemaIntrospectionBenchmark.func1
  | 	pkg/cmd/roachtest/tests/large_schema_benchmark.go:347
  | main.(*testRunner).runTest.func2
  | 	pkg/cmd/roachtest/test_runner.go:1559
  | runtime.goexit
  | 	src/runtime/asm_amd64.s:1693
Wraps: (2) secondary error attachment
  | full command output in run_190726.343139957_n10_cockroach-workload-i.log: COMMAND_PROBLEM: exit status 1
  | (1) attached stack trace
  |   -- stack trace:
  |   | main.(*clusterImpl).RunE
  |   | 	pkg/cmd/roachtest/cluster.go:2577
  |   | main.(*clusterImpl).Run
  |   | 	pkg/cmd/roachtest/cluster.go:2531
  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.setupTPCC.func2
  |   | 	pkg/cmd/roachtest/tests/tpcc.go:365
  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.setupTPCC
  |   | 	pkg/cmd/roachtest/tests/tpcc.go:370
  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runLargeSchemaIntrospectionBenchmark
  |   | 	pkg/cmd/roachtest/tests/large_schema_benchmark.go:446
  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerLargeSchemaIntrospectionBenchmark.func1
  |   | 	pkg/cmd/roachtest/tests/large_schema_benchmark.go:347
  |   | main.(*testRunner).runTest.func2
  |   | 	pkg/cmd/roachtest/test_runner.go:1559
  |   | runtime.goexit
  |   | 	src/runtime/asm_amd64.s:1693
  | Wraps: (2) full command output in run_190726.343139957_n10_cockroach-workload-i.log
  | Wraps: (3) Node 10. Command with error:
  |   | ```
  |   | ./cockroach workload init tpccmultidb --db-list-file=populate_introspection --data-loader=none --fks=false {pgurl:1-9} --db warehouse_0 --warehouses 1
  |   | ```
  |   | <truncated> ... 260314 19:39:46.777277 1 workload/tpcc/tpcc_multi_db.go:451  [-] 19  created 90000/111112 databases/schemas (batch 18, 5000 entries in 2m14.906s)
  |   | I260314 19:42:04.282703 1 workload/tpcc/tpcc_multi_db.go:451  [-] 20  created 95000/111112 databases/schemas (batch 19, 5000 entries in 2m17.505s)
  |   | I260314 19:44:21.281423 1 workload/tpcc/tpcc_multi_db.go:451  [-] 21  created 100000/111112 databases/schemas (batch 20, 5000 entries in 2m16.999s)
  |   | Error: Could not pre-create: retrying txn failed after 50 attempts. original error: pq: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE): "sql txn" meta={id=b655532c key=/Table/3/1/2176/2/1 iso=Serializable pri=0.10351096 epo=50 ts=1773524019.282082583,1 min=1773517461.281705697,0 seq=60002} lock=true stat=PENDING rts=1773524011.996518605,1 gul=1773517461.781705697,0 obs={n1@1773517461.281705697,0 n2@1773517513.751740527,0 n3@1773517461.283956728,0 n4@1773517461.283970147,0 n5@1773517723.714901606,0, ...} isn=1.
  | Wraps: (4) COMMAND_PROBLEM
  | Wraps: (5) exit status 1
  | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *hintdetail.withDetail (4) errors.Cmd (5) *exec.ExitError
Wraps: (3) full command output in run_190726.343139957_n10_cockroach-workload-i.log: COMMAND_PROBLEM: exit status 1
Error types: (1) *withstack.withStack (2) *secondary.withSecondaryError (3) *errutil.leafError

[shghasemi]

Claude's analysis:

This is a transaction retry error (RETRY_SERIALIZABLE) during the large_schema_introspection_benchmark roachtest.

What's happening:

The test is running workload init tpccmultidb to pre-create a large number of databases/schemas (111,112 total). The workload command is executing on node 10 against nodes 1-9. It successfully created ~100,000 databases/schemas but then hit a serialization conflict that couldn't be resolved after 50 retry attempts.

The specific error:

TransactionRetryError: retry txn (RETRY_SERIALIZABLE)
Key details from the transaction metadata:

key=/Table/3/1/2176/2/1 — This is hitting the system.namespace table (Table ID 3), which is the descriptor namespace table used to map names to IDs.
epo=50 — The transaction went through 50 epochs (retry attempts) and gave up.
seq=60002 — The transaction has a very high sequence number, meaning it's a large batch transaction with many operations.
The transaction spans multiple nodes (obs={n1, n2, n3, n4, n5, ...}).
Root cause:

The workload is creating databases/schemas in large batches (5,000 per batch). Each batch is a single transaction writing many entries to system.namespace. At ~100k entries, contention on the namespace table is high enough that the serializable transaction can't commit — other concurrent activity (or even the schema creation's own catalog operations) keeps causing write-write conflicts, and 50 retries aren't enough to get through.

Likely fixes/mitigations:

• Reduce the batch size (fewer entries per transaction = less contention window)
• Increase the max retry count
• Use a lower isolation level if appropriate

[shghasemi]

/investigate

[blathers-crl]

The release-blocker label has been automatically removed because this issue was assigned priority P-2. Only P-0 and P-1 issues are considered release blockers.