[Bug]: Segfault on pd.DataFrame(samples).dropna() on s390x

[modassarrana89-new]

Problem Statement: Segfault on pd.DataFrame(samples).dropna() on s390x

The samples list coming from RandomDropDeque contains raw dicts that still have a "timestamp" key. On s390x (big-endian), pandas' internal memory layout for mixed-type DataFrames with datetime-parseable string values triggers a different code path in the C extension for type inference. Specifically:
python# pandas sees "timestamp": "2024-01-01T00:00:00Z" in every row
On x86_64: infers as object column, fine
On s390x: tries to parse as datetime64 with big-endian byte alignment & hits a misaligned memory access in the C extension → SIGSEGV

The timestamp field is the trigger — dropna() internally calls DataFrame._consolidate() which tries to find common dtypes across columns, and the datetime parsing path on s390x has a known misaligned access issue in older numpy/pandas builds.

Solution :-

Below fix removes timestamp BEFORE constructing the DataFrame

for row in samples:
    r = row.copy()
    r.pop("timestamp", None)   # ← removes the problematic field first
    clean.append(r)
raw = pd.DataFrame(clean)      # ← DataFrame never sees the timestamp column
raw = raw.dropna()             # ← dropna on homogeneous numeric types, safe

By the time pandas constructs the DataFrame, all columns are numeric — no datetime inference, no big-endian misalignment, no segfault.

Note: Above Fix is applicable ine below scenario as well

1. TTFT preprocessing
2. TPOT preprocessing
3. Ensemble loop (×4)
4. _split_samples_by_queue

Version

main

Steps to Reproduce

1. You need s390x system
2. Bring up training server & predictor server
3. Apply test job

Environment

Relevant log output

[modassarrana89-new]

/assign @modassarrana89-new