feat: add generation stats to `TextGeneration` trait

[erfanium]

tabby/crates/tabby-inference/src/lib.rs

Lines 23 to 31 in 99d49a9

	#[async_trait]
	pub trait TextGeneration: Sync + Send {
	async fn generate(&self, prompt: &str, options: TextGenerationOptions) -> String;
	async fn generate_stream(
	&self,
	prompt: &str,
	options: TextGenerationOptions,
	) -> BoxStream<String>;
	}

Current TextGeneration trait is simple, but it doesn't tell the statics we need for monitoring and optimizing the inference server.

for example, input_token_length and output_token_length stats are really important to measure the inference server throughput.

My initial idea would be something like this:

pub struct TextGenerationStat {
    pub prompt_tokens_length: u32,
    pub output_tokens_length: Option<u32>, // not available in stream responses
    pub time: Option<u32>, // not available in stream responses
}

pub struct TextGenerationResponse {
    pub text: String,
    pub stat: Option<TextGenerationStat>,
}

pub struct TextGenerationStreamResponse {
    pub stream: BoxStream<String>,
    pub stat: Option<TextGenerationStat>,
}

pub trait TextGeneration: Sync + Send {
    async fn generate(
        &self,
        prompt: &str,
        options: TextGenerationOptions,
    ) -> TextGenerationResponse;
    async fn generate_stream(
        &self,
        prompt: &str,
        options: TextGenerationOptions,
    ) -> TextGenerationStreamResponse;
}

[wsxiaoys]

I think we can already compute stats for input string length / output string length / time. Should be sufficient for tabby's optimization purpose?

[erfanium]

@wsxiaoys yeah but as you know, input string length != input token length. metrics based on string length is not standard.

[wsxiaoys]

Yes - that's why I emphasis that for tabby's optimization purpose :).