[FEATURE] Better use and control over file() agentic agents

[KeithSBB]

Confirmation of Request Source

• I confirm that this feature request is for the xAI SDK Python library (e.g., client features, SDK methods, or documentation) and not for the xAI API itself (e.g., model capabilities or API changes).

Describe the feature you'd like

I've been playing with the file features and been having a lot of difficulty getting grok to do what I want.

Use Case: Using Grok as an integrated Home Assistant AI assistant
Goal: Reduce token usage by loading prompt, smart home context, and tool catalog as files and using file(file_id) to refer to them. files woul only need to be updated infrequently.

1. I can't use system(file(prompt_file_id)) as file can only be used in user()
2. I can't use tools = file(tool_catalog_file_id) in client.chat.create()
3. I can't upload a json file with a description of my smart home devices and entities in a system() message.
4. I can upload a json file with a description of my smart home devices and entities in a user() message, but it never seems to find anything in the uploaded file.

The idea is to provide a server side listing of every IoT devices and entities so that Grok can count, list and use the entity_id and a local client tool call to get current states (temperature, door open, current water rate, etc.). Grok tries to search for these things server side with :

Server side tool call: type: document_search_tool, Name: read_attachment, ars: {"key":"grok_smart_home_context.json","query":"Count the total number of devices or entities that are in the 'Bedroom' area. List their names and domains if possible, and provide the exact count."}

Which does not return anything useful. It seems to be trying to perform a search inside a file using a call which just reads the file? I used Grok 4.1 to help fix this which was unsuccessful. I don't see a way to get more useful information to debug??? Wish they were better debug features.

Additional Context

No response