Bug Report: vttablet with high replication lag start as healthy instead of unhealthy

[L3o-pold]

Overview of the Issue

vttablet was shutdown for maintenance. After 11 hours, we started it again.

With a 11 hours replication lag, it started as healthy then some seconds after that switched to unhealthy (due to replication lag).

We think that vttablet should be started as unhealthy to avoid sending query to this replica.

Reproduction Steps

start a vttablet with a lag greater than the discovery_high_replication_lag_minimum_serving vtgate argument.

Binary Version

mysqlctld version Version: 21.0.3 (Git revision 94fdc736eae8928a8fdde44e9ec9c3bee1868d6f branch 'HEAD') built on Wed Feb 12 16:52:22 UTC 2025 by vitess@buildkitsandbox using go1.23.5 linux/amd64

Operating System and Environment details

docker

Log Fragments

vttablet I0306 08:59:14.831788       1 state_manager.go:740] Replication is healthy

vttablet I0306 08:59:19.833346       1 state_manager.go:735] Going unhealthy due to high replication lag: 10h51m48s

vttablet I0306 09:00:39.848743       1 state_manager.go:740] Replication is healthy

[bluecrabs007]

@L3o-pold We have seen similar behaviour when we had the disable_active_reparents: true flag set on the vttablet. Removing the flag or setting it to false resolved the issue.

[L3o-pold]

We don't have such flag on vttablet

/vt/bin/vttablet 
   --db_app_user=*  
   --db_app_user=*  
   --db_filtered_user=vt_filtered  
   --db_repl_user=*  
   --db_repl_password=*  
   --db_charset=utf8mb4  
   --enable_replication_reporter=true  
   --degraded_threshold=30s  
   --enforce_strict_trans_tables=true  
   --grpc_port=15999  
   --health_check_interval=5s  
   --init_db_name_override=*  
   --init_keyspace=*  
   --init_shard=-  
   --init_tablet_type=replica  
   --logtostderr=true  
   --mycnf_socket_file=/vt/socket/mysql.sock  
   --mysqlctl_socket=/vt/socket/mysqlctl.sock  
   --port=15000  
   --queryserver-config-max-result-size=100000  
   --queryserver-config-pool-size=96  
   --queryserver-config-query-timeout=120s  
   --queryserver-config-stream-pool-size=96  
   --queryserver-config-transaction-cap=300  
   --service_map=grpc-queryservice,grpc-tabletmanager,grpc-updatestream 
   --tablet-path=*
   --tablet_hostname=*
   --topo_global_root=/vitess/*/global 
   --topo_global_server_address=*
   --topo_implementation=etcd2  
   --backup_storage_implementation=s3  
   --backup_engine_implementation=xtrabackup  
   --s3_backup_aws_region=us-east-1  
   --s3_backup_storage_bucket=default  
   --s3_backup_force_path_style=true  
   --s3_backup_aws_endpoint=*
   --s3_backup_storage_root=*
   --xtrabackup_stream_mode=xbstream  
   --xtrabackup_user=vt_dba  
   --restore_from_backup=true  
   --shutdown_grace_period=10s  
   --queryserver-config-query-timeout=120s  
   --queryserver-config-transaction-timeout=120s 
   --queryserver-config-schema-change-signal=true  
   --grpc_max_message_size=67108864  
   --serving_state_grace_period=5s