feat: add prefill_density engineered feature for per-request compute awareness

[madhugoutham]

Summary

Add prefill_density = prefill_tokens_in_flight / num_request_running as an engineered feature to the training and prediction pipeline.

Problem

The model receives prefill_tokens_in_flight and num_request_running as separate features but cannot distinguish between:

• Pod A: 10,000 tokens across 10 requests (density 1,000, manageable)
• Pod B: 10,000 tokens in 1 request (density 10,000, overwhelmed)

Both pods look identical to the model. Attention cost is O(n^2) per request, so a single 10,000-token request uses significantly more compute than many smaller requests at the same total token count.

XGBoost uses axis-aligned splits and inefficiently approximates ratios between features, requiring many tree levels to capture what a single pre-computed ratio provides directly. This follows the same pattern as the existing effective_input_tokens feature (training_server.py:453), which pre-computes (1 - prefix_cache_score) * input_token_length.

Proposed approach

1. Add prefill_density computation in _prepare_features_with_interaction() (training_server.py:416) and the equivalent in prediction_server.py
2. Guard division by zero: return 0 when num_request_running == 0
3. Add to Go producer feature vector in dataproducer_hooks.go
4. Note: adding a feature changes the model schema. New models trained with this feature are not compatible with old prediction servers. Coordinate rollout.

Note: This feature has not been tested on production data. Production validation is required before drawing conclusions about impact.

[madhugoutham]

/assign