BigQuery destination crashes with GRPC_CALL_ERROR_TOO_MANY_OPERATIONS on server-side stream restart (segfault, exit code 139)

[sagarswamirao]

AxoSyslog

Version of AxoSyslog

4.22.0

Platform

Kubernetes; GCP; linux;

Issue

Failure

At times it prints this message and Pod restarts.

GRPC_CALL_ERROR_TOO_MANY_OPERATIONS
Check failed: false
Error closing BigQuery stream; error='Closing the stream because server is restarted'

K8 Event Logs:
Last State: Terminated
Reason: Error
Exit Code: 139
Started: Thu, 19 Feb 2026 18:33:08 -0700
Finished: Thu, 19 Feb 2026 23:41:45 -0700
Ready: True

Configuration

@Version: current
@include "scl.conf"

options {
log_fifo_size(100000);
log_msg_size(262144);
stats(freq(10));
time_reopen(10);
use_fqdn(no);
use_dns(no);
chain_hostnames(no);
keep_hostname(yes);
keep_timestamp(yes);
};

source s_internal { internal(); };

destination d_bigquery_rp {
bigquery(
project("${GOOGLE_CLOUD_PROJECT}")
dataset("d_id")
table("rp")
workers(2)
batch-timeout(3000)
batch-lines(20)
schema(
"timestamp" STRING => "${TIMESTAMP}"
"trace_id" STRING => "${TRACE_ID}"
"func_name" STRING => "${FUNC_NAME}"
"level" STRING => "${LOG_LEVEL}"
"request_payload" STRING => "${REQUEST_PAYLOAD}"
"jsonResponse" JSON => "${JSON_RESPONSE}"
)
on-error("drop-property")
);
};

log {
source(s_otlp);
filterx {
declare log = otel_logrecord(${.otel_raw.log});
log.attributes._bigquery_table == "retrieval_prompts";
${TIMESTAMP} = isset(log.attributes.timestamp) ? string(log.attributes.timestamp) : "";
${TRACE_ID} = isset(log.attributes.trace_id) ? string(log.attributes.trace_id) : "";
${FUNC_NAME} = isset(log.attributes.func_name) ? string(log.attributes.func_name) : "";
${LOG_LEVEL} = isset(log.attributes.level) ? string(log.attributes.level) : "";
${REQUEST_PAYLOAD} = isset(log.attributes.request_payload) ? format_json(log.attributes.request_payload) : "";
${JSON_RESPONSE} = isset(log.attributes.jsonResponse) ? format_json(log.attributes.jsonResponse) : null;
};

destination(d_bigquery_rp);
};

• Destination: BigQuery Storage Write API

[sagarswamirao]

Additional Info

AxoSyslog crashes with exit code 139 (segfault) when the BigQuery Storage Write API performs a server-side stream restart. The crash is caused by a race condition in bigquery-worker.cpp between flush() and disconnect() operating concurrently on the same batch_writer object.

Crash logs

E0000 00:00:... call_op_set.h:979] API misuse of type GRPC_CALL_ERROR_TOO_MANY_OPERATIONS observed
F0000 00:00:... call_op_set.h:981] Check failed: false
SYSLOG-NG INTERNAL: warning Error closing BigQuery stream; error='Closing the stream because server is restarted. This is expected and client is advised to reconnect.'
SYSLOG-NG INTERNAL: err Error writing BigQuery batch
SYSLOG-NG INTERNAL: err Error occurred while trying to send a message, trying again; retries='1'
SYSLOG-NG INTERNAL: err Error occurred while trying to send a message, trying again; retries='2'

Root cause

In bigquery-worker.cpp, disconnect() and flush() both operate on the same batch_writer (ClientReaderWriter) without synchronization:

disconnect() calls:

this->batch_writer->WritesDone();   // operation 1
this->batch_writer->Finish();       // operation 2

flush() simultaneously calls:

this->batch_writer->Write(current_batch);        // operation 3
this->batch_writer->Read(&append_rows_response); // operation 4

When the BigQuery server restarts the stream mid-flush, disconnect() is triggered while flush() is still in progress. gRPC detects concurrent operations on the same call object and aborts with GRPC_CALL_ERROR_TOO_MANY_OPERATIONS, causing a hard crash.

Additional bug

In construct_write_stream(), the return value of CreateWriteStream is never checked:

stub->CreateWriteStream(&ctx, create_write_stream_request, &wstream);
// status never checked - silent failure possible

Expected behavior

AxoSyslog should handle server-side stream restarts gracefully by reconnecting without crashing. disconnect() and flush() should be properly synchronized around batch_writer access.

[sagarswamirao]

2026-02-19 23:41:32 - Error closing BigQuery stream; error='Closing the stream because server is restarted. This is expected and client is advised to reconnect. Entity: projects/ms2-dev-469118/datasets/ai_logs/tables/retrieval_prompts/streams/...'
2026-02-19 23:41:32 - Error writing BigQuery batch
2026-02-19 23:41:32 - Error occurred while trying to send a message; retries='1', batch_size='1'
2026-02-19 23:41:45 - API misuse of type GRPC_CALL_ERROR_TOO_MANY_OPERATIONS observed
2026-02-19 23:41:45 - Check failed: false ← crash
2026-02-19 23:41:45 - Error writing BigQuery batch
2026-02-19 23:41:45 - Error occurred while trying to send a message; retries='2', batch_size='1'
2026-02-19 23:41:47 - syslog-ng starting up (pod restarted)