Skip to content

Sentry: cmd_recover_txn.go:134: failed indeterminate commit recovery: programming error: timestamp change by implicitly committed transaction: 1773494393.156737897,2->1773494393.007970970,2 (1) Wraps:... #165782

New issue
New issue
Open
Open
Sentry: cmd_recover_txn.go:134: failed indeterminate commit recovery: programming error: timestamp change by implicitly committed transaction: 1773494393.156737897,2->1773494393.007970970,2 (1) Wraps:...#165782
Labels
C-bugCode not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.O-sentryOriginated from an in-the-wild panic report.T-kvKV TeamX-blathers-triagedblathers was able to find an ownerbranch-release-25.4Used to mark GA and release blockers and technical advisories for 25.4
@cockroach-sentry

Description

@cockroach-sentry
cockroach-sentry
opened on Mar 14, 2026

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: https://cockroach-labs.sentry.io/issues/7335012281/?referrer=webhooks_plugin

Panic Message:

cmd_recover_txn.go:134: failed indeterminate commit recovery: programming error: timestamp change by implicitly committed transaction: 1773494393.156737897,2->1773494393.007970970,2
(1)
Wraps: (2)
  | (opaque error wrapper)
  | type name: github.com/cockroachdb/errors/withstack/*withstack.withStack
  | reportable 0:
  |
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver/txnrecovery.(*manager).ResolveIndeterminateCommit
  | 	pkg/kv/kvserver/txnrecovery/manager.go:105
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleIndeterminateCommitError
  | 	pkg/kv/kvserver/replica_send.go:796
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries
  | 	pkg/kv/kvserver/replica_send.go:568
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).SendWithWriteBytes
  | 	pkg/kv/kvserver/replica_send.go:189
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes
  | 	pkg/kv/kvserver/store_send.go:188
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).SendWithWriteBytes
  | 	pkg/kv/kvserver/stores.go:203
  | github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal
  | 	pkg/server/node.go:1668
  | github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch
  | 	pkg/server/node.go:1861
  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func1
  | 	pkg/rpc/context.go:762
  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.ServerInterceptor.func12
  | 	pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:60
  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
  | 	pkg/rpc/context.go:873
  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func3
  | 	pkg/rpc/context.go:182
  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
  | 	pkg/rpc/context.go:873
  | github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.unaryInterceptor
  | 	pkg/rpc/auth.go:97
  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
  | 	pkg/rpc/context.go:873
  | github.com/cockroachdb/cockroach/pkg/rpc.gatewayRequestRecoveryInterceptor
  | 	pkg/rpc/metrics.go:514
  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
  | 	pkg/rpc/context.go:873
  | github.com/cockroachdb/cockroach/pkg/server.newGRPCServer.NewRequestMetricsInterceptor.func4
  | 	pkg/rpc/metrics.go:467
  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
  | 	pkg/rpc/context.go:873
  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1
  | 	pkg/rpc/context.go:139
  | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
  | 	pkg/util/stop/stopper.go:351
  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1
  | 	pkg/rpc/context.go:137
  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
  | 	pkg/rpc/context.go:873
  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func2
  | 	pkg/rpc/context.go:772
  | github.com/cockroachdb/cockroach/pkg/rpc.NewContext.ClientInterceptor.func9
  | 	pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:185
  | github.com/cockroachdb/cockroach/pkg/rpc.getChainUnaryInvoker.func1
  | 	pkg/rpc/context.go:957
  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func3
  | 	pkg/rpc/context.go:842
  | github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.Batch
  | 	pkg/rpc/context.go:965
  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).sendBatch
  | 	pkg/kv/kvclient/kvcoord/transport.go:211
  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).SendNext
  | 	pkg/kv/kvclient/kvcoord/transport.go:189
  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas
  | 	pkg/kv/kvclient/kvcoord/dist_sender.go:2768
  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch
  | 	pkg/kv/kvclient/kvcoord/dist_sender.go:2227
Wraps: (3) failed indeterminate commit recovery
Wraps: (4) assertion failure
Wraps: (5)
  | (opaque error wrapper)
  | type name: github.com/cockroachdb/errors/withstack/*withstack.withStack
  | reportable 0:
  |
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver/batcheval.RecoverTxn
  | 	pkg/kv/kvserver/batcheval/cmd_recover_txn.go:134
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.evaluateCommand
  | 	pkg/kv/kvserver/replica_evaluate.go:543
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.evaluateBatch
  | 	pkg/kv/kvserver/replica_evaluate.go:356
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evaluateWriteBatchWrapper
  | 	pkg/kv/kvserver/replica_write.go:751
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evaluateWriteBatchWithServersideRefreshes
  | 	pkg/kv/kvserver/replica_write.go:716
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evaluateWriteBatch
  | 	pkg/kv/kvserver/replica_write.go:479
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evaluateProposal
  | 	pkg/kv/kvserver/replica_proposal.go:999
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).requestToProposal
  | 	pkg/kv/kvserver/replica_proposal.go:1095
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evalAndPropose
  | 	pkg/kv/kvserver/replica_raft.go:119
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeWriteBatch
  | 	pkg/kv/kvserver/replica_write.go:188
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries
  | 	pkg/kv/kvserver/replica_send.go:512
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).SendWithWriteBytes
  | 	pkg/kv/kvserver/replica_send.go:189
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes
  | 	pkg/kv/kvserver/store_send.go:188
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).SendWithWriteBytes
  | 	pkg/kv/kvserver/stores.go:203
  | github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal
  | 	pkg/server/node.go:1668
  | github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch
  | 	pkg/server/node.go:1861
  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func1
  | 	pkg/rpc/context.go:762
  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.ServerInterceptor.func12
  | 	pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:60
  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
  | 	pkg/rpc/context.go:873
  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func3
  | 	pkg/rpc/context.go:182
  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
  | 	pkg/rpc/context.go:873
  | github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.unaryInterceptor
  | 	pkg/rpc/auth.go:97
  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
  | 	pkg/rpc/context.go:873
  | github.com/cockroachdb/cockroach/pkg/rpc.gatewayRequestRecoveryInterceptor
  | 	pkg/rpc/metrics.go:514
  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
  | 	pkg/rpc/context.go:873
  | github.com/cockroachdb/cockroach/pkg/server.newGRPCServer.NewRequestMetricsInterceptor.func4
  | 	pkg/rpc/metrics.go:467
  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
  | 	pkg/rpc/context.go:873
  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1
  | 	pkg/rpc/context.go:139
  | github.com/cockroachdb/cockroach/pkg/u...
Stacktrace (expand for inline code snippets):

pkg/kv/kvclient/kvcoord/dist_sender.go#L2226-L2228
pkg/kv/kvclient/kvcoord/dist_sender.go#L2767-L2769
pkg/kv/kvclient/kvcoord/transport.go#L188-L190
pkg/kv/kvclient/kvcoord/transport.go#L210-L212
pkg/rpc/context.go#L964-L966
pkg/rpc/context.go#L841-L843
pkg/rpc/context.go#L956-L958
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go#L184-L186
pkg/rpc/context.go#L771-L773
pkg/rpc/context.go#L872-L874
pkg/rpc/context.go#L136-L138
pkg/util/stop/stopper.go#L350-L352
pkg/rpc/context.go#L138-L140
pkg/rpc/context.go#L872-L874
pkg/rpc/metrics.go#L466-L468
pkg/rpc/context.go#L872-L874
pkg/rpc/metrics.go#L513-L515
pkg/rpc/context.go#L872-L874
pkg/rpc/auth.go#L96-L98
pkg/rpc/context.go#L872-L874
pkg/rpc/context.go#L181-L183
pkg/rpc/context.go#L872-L874
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go#L59-L61
pkg/rpc/context.go#L761-L763
pkg/server/node.go#L1860-L1862
pkg/server/node.go#L1667-L1669
pkg/kv/kvserver/stores.go#L202-L204
pkg/kv/kvserver/store_send.go#L187-L189
pkg/kv/kvserver/replica_send.go#L188-L190
pkg/kv/kvserver/replica_send.go#L567-L569
pkg/kv/kvserver/replica_send.go#L795-L797
pkg/kv/kvserver/txnrecovery/manager.go#L104-L106
pkg/rpc/context.go#L771-L773
pkg/rpc/context.go#L872-L874
pkg/rpc/context.go#L136-L138
pkg/util/stop/stopper.go#L350-L352
pkg/rpc/context.go#L138-L140
pkg/rpc/context.go#L872-L874
pkg/rpc/metrics.go#L466-L468
pkg/rpc/context.go#L872-L874
pkg/rpc/metrics.go#L513-L515
pkg/rpc/context.go#L872-L874
pkg/rpc/auth.go#L96-L98
pkg/rpc/context.go#L872-L874
pkg/rpc/context.go#L181-L183
pkg/rpc/context.go#L872-L874
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go#L59-L61
pkg/rpc/context.go#L761-L763
pkg/server/node.go#L1860-L1862
pkg/server/node.go#L1667-L1669
pkg/kv/kvserver/stores.go#L202-L204
pkg/kv/kvserver/store_send.go#L187-L189
pkg/kv/kvserver/replica_send.go#L188-L190
pkg/kv/kvserver/replica_send.go#L511-L513
pkg/kv/kvserver/replica_write.go#L187-L189
pkg/kv/kvserver/replica_raft.go#L118-L120
pkg/kv/kvserver/replica_proposal.go#L1094-L1096
pkg/kv/kvserver/replica_proposal.go#L998-L1000
pkg/kv/kvserver/replica_write.go#L478-L480
pkg/kv/kvserver/replica_write.go#L715-L717
pkg/kv/kvserver/replica_write.go#L750-L752
pkg/kv/kvserver/replica_evaluate.go#L355-L357
pkg/kv/kvserver/replica_evaluate.go#L542-L544
pkg/kv/kvserver/batcheval/cmd_recover_txn.go#L133-L135

pkg/kv/kvclient/kvcoord/dist_sender.go in pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch at line 2227
pkg/kv/kvclient/kvcoord/dist_sender.go in pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas at line 2768
pkg/kv/kvclient/kvcoord/transport.go in pkg/kv/kvclient/kvcoord.(*grpcTransport).SendNext at line 189
pkg/kv/kvclient/kvcoord/transport.go in pkg/kv/kvclient/kvcoord.(*grpcTransport).sendBatch at line 211
pkg/rpc/context.go in pkg/rpc.internalClientAdapter.Batch at line 965
pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.func3 at line 842
pkg/rpc/context.go in pkg/rpc.getChainUnaryInvoker.func1 at line 957
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go in pkg/rpc.NewContext.ClientInterceptor.func9 at line 185
pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.func2 at line 772
pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4 at line 873
pkg/rpc/context.go in pkg/rpc.NewServerEx.func1 at line 137
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunTaskWithErr at line 351
pkg/rpc/context.go in pkg/rpc.NewServerEx.func1.1 at line 139
pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4 at line 873
pkg/rpc/metrics.go in pkg/server.newGRPCServer.NewRequestMetricsInterceptor.func4 at line 467
pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4 at line 873
pkg/rpc/metrics.go in pkg/rpc.gatewayRequestRecoveryInterceptor at line 514
pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4 at line 873
pkg/rpc/auth.go in pkg/rpc.kvAuth.unaryInterceptor at line 97
pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4 at line 873
pkg/rpc/context.go in pkg/rpc.NewServerEx.func3 at line 182
pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4 at line 873
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go in pkg/rpc.NewServerEx.ServerInterceptor.func12 at line 60
pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.func1 at line 762
pkg/server/node.go in pkg/server.(*Node).Batch at line 1861
pkg/server/node.go in pkg/server.(*Node).batchInternal at line 1668
pkg/kv/kvserver/stores.go in pkg/kv/kvserver.(*Stores).SendWithWriteBytes at line 203
pkg/kv/kvserver/store_send.go in pkg/kv/kvserver.(*Store).SendWithWriteBytes at line 188
pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).SendWithWriteBytes at line 189
pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries at line 568
pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).handleIndeterminateCommitError at line 796
pkg/kv/kvserver/txnrecovery/manager.go in pkg/kv/kvserver/txnrecovery.(*manager).ResolveIndeterminateCommit at line 105
pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.func2 at line 772
pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4 at line 873
pkg/rpc/context.go in pkg/rpc.NewServerEx.func1 at line 137
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunTaskWithErr at line 351
pkg/rpc/context.go in pkg/rpc.NewServerEx.func1.1 at line 139
pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4 at line 873
pkg/rpc/metrics.go in pkg/server.newGRPCServer.NewRequestMetricsInterceptor.func4 at line 467
pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4 at line 873
pkg/rpc/metrics.go in pkg/rpc.gatewayRequestRecoveryInterceptor at line 514
pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4 at line 873
pkg/rpc/auth.go in pkg/rpc.kvAuth.unaryInterceptor at line 97
pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4 at line 873
pkg/rpc/context.go in pkg/rpc.NewServerEx.func3 at line 182
pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4 at line 873
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go in pkg/rpc.NewServerEx.ServerInterceptor.func12 at line 60
pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.func1 at line 762
pkg/server/node.go in pkg/server.(*Node).Batch at line 1861
pkg/server/node.go in pkg/server.(*Node).batchInternal at line 1668
pkg/kv/kvserver/stores.go in pkg/kv/kvserver.(*Stores).SendWithWriteBytes at line 203
pkg/kv/kvserver/store_send.go in pkg/kv/kvserver.(*Store).SendWithWriteBytes at line 188
pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).SendWithWriteBytes at line 189
pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries at line 512
pkg/kv/kvserver/replica_write.go in pkg/kv/kvserver.(*Replica).executeWriteBatch at line 188
pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).evalAndPropose at line 119
pkg/kv/kvserver/replica_proposal.go in pkg/kv/kvserver.(*Replica).requestToProposal at line 1095
pkg/kv/kvserver/replica_proposal.go in pkg/kv/kvserver.(*Replica).evaluateProposal at line 999
pkg/kv/kvserver/replica_write.go in pkg/kv/kvserver.(*Replica).evaluateWriteBatch at line 479
pkg/kv/kvserver/replica_write.go in pkg/kv/kvserver.(*Replica).evaluateWriteBatchWithServersideRefreshes at line 716
pkg/kv/kvserver/replica_write.go in pkg/kv/kvserver.(*Replica).evaluateWriteBatchWrapper at line 751
pkg/kv/kvserver/replica_evaluate.go in pkg/kv/kvserver.evaluateBatch at line 356
pkg/kv/kvserver/replica_evaluate.go in pkg/kv/kvserver.evaluateCommand at line 543
pkg/kv/kvserver/batcheval/cmd_recover_txn.go in pkg/kv/kvserver/batcheval.RecoverTxn at line 134

Tags

Tag Value
Command server
Environment v25.4.3
Go Version go1.23.12 X:nocoverageredesign
Platform linux amd64
Distribution CCL
Cockroach Release v25.4.3
Cockroach SHA 71d8536
# of CPUs 16
# of Goroutines 2425

Jira issue: CRDB-61587

Metadata

Metadata

Assignees

No one assigned

    Labels

    C-bugCode not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.O-sentryOriginated from an in-the-wild panic report.T-kvKV TeamX-blathers-triagedblathers was able to find an ownerbranch-release-25.4Used to mark GA and release blockers and technical advisories for 25.4

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions