Recommendations for context-surfaces SDK improvements from this integration (with acceptance criteria)

[alberttwong]

Summary

This demo project currently carries app-level workarounds around the context-surfaces Python SDK (0.0.1).

Based on integration points in this repo, here is a prioritized recommendation list for SDK improvements, with acceptance criteria for each.

1) Add first-class data_source.connection_config support to UnifiedClient.create_context_surface

Problem

• This project bypasses the SDK and uses raw httpx to create surfaces with embedded Redis settings.

Evidence

• scripts/setup_surface.py (_create_surface) manually POSTs /api/v1/context-surfaces with data_source.connection_config.

Acceptance Criteria

• UnifiedClient.create_context_surface(...) supports creating a surface using embedded Redis connection config (addr/username/password/db/tls/etc).
• The SDK request model includes a typed data_source input (or equivalent) that maps to current admin API contract.
• A minimal end-to-end SDK example (docs + test) creates a surface without raw HTTP calls.

2) Expose a public Python API for data-model parsing (instead of CLI-private internals)

Problem

• The project imports _parse_data_model_from_python from context_surfaces.cli.main, which is private and brittle.

Evidence

• scripts/setup_surface.py imports from context_surfaces.cli.main import _parse_data_model_from_python.

Acceptance Criteria

• A public, documented function is available from a stable module path (non-underscore API) for parsing Python ContextModel modules into data_model payloads.
• Existing CLI implementation uses the same public parser internally (single source of truth).
• Backward-compatible deprecation path is provided for private parser consumers.

3) Normalize list_tools JSON schema output so array fields always include valid items

Problem

• App code sanitizes tool definitions to patch missing array item schemas (especially vector/embedding fields).

Evidence

• backend/app/context_surface_service.py (_sanitize_json_schema, _sanitize_tool_definition)
• tests/test_tool_schema_sanitization.py

Acceptance Criteria

• list_tools(...) always returns JSON Schema-valid array definitions with explicit items.
• Vector/embedding tool params consistently return {"type": "array", "items": {"type": "number"}} (or equivalent typed schema).
• Composition keywords (anyOf/oneOf/allOf) preserve array typing correctly.

4) Return a stable, typed tool-result envelope from query_tool (or ship an SDK normalizer helper)

Problem

• Tool responses can arrive in multiple shapes, forcing custom normalization/parsing in app code.

Evidence

• backend/app/context_surface_service.py (_normalize_tool_result_payload and wrapped-content parsing helpers)

Acceptance Criteria

• SDK exposes a canonical result shape (typed model or helper API), regardless of MCP wire response variants.
• Common cases (text JSON, wrapped text content, dict payloads) deserialize to a single predictable structure.
• SDK docs include a recommended pattern for consuming tool results without custom parsers.

5) Improve nullable argument handling for MCP tool calls (null support for optional args)

Problem

• App code strips None values before calls because optional numeric params can be rejected when sent as null.

Evidence

• backend/app/langgraph_agent.py _make_mcp_tool comment: “MCP server rejects null for optional numeric params”.

Acceptance Criteria

• Optional tool params accept explicit null when schema allows it, or SDK clearly auto-omits None before transport.
• Behavior is documented and consistent across numeric/string/array optional fields.
• Regression test covers optional numeric param passed as None in Python.

6) Provide typed tool-definition models for list_tools instead of untyped dicts

Problem

• list_tools currently returns list[dict[str, Any]]; downstream code must sanitize and map manually.

Evidence

• backend/app/context_surface_service.py casts each tool via dict/model_dump and sanitizes schema.

Acceptance Criteria

• SDK provides typed models for tool definitions (name, description, input_schema, etc.).
• list_tools can return typed objects by default (with optional dict conversion for compatibility).
• Type hints and docs enable direct IDE/static-checking usage.

7) Add admin lifecycle helpers (get_surface, ensure_surface, ensure_agent_key) to UnifiedClient

Problem

• Setup scripts reimplement describe/create/reuse flows, status handling, and response formatting.

Evidence

• scripts/setup_surface.py (_describe_surface, _create_surface, _create_agent_key, reuse logic in main).

Acceptance Criteria

• SDK exposes admin helpers for common lifecycle operations:
  • fetch/describe by surface id
  • create-or-reuse by name
  • create-or-reuse agent key by name
• Helper methods return typed responses and clear conflict/not-found semantics.
• Docs include “first-time setup” and “idempotent setup” examples.

8) Add richer typed SDK exceptions (status code, response body, request id, operation metadata)

Problem

• Integrations currently use broad except Exception and stringified errors due limited structured error info.

Evidence

• scripts/setup_surface.py ad-hoc error formatting
• backend/app/langgraph_agent.py broad exception handling for tool execution

Acceptance Criteria

• SDK raises typed exception classes for auth, validation, not-found, conflict, timeout, and transport errors.
• Exceptions expose structured fields (status_code, response_body, request_id, operation name).
• Docs include recommended error-handling patterns and retry guidance.

Expected Outcome

If the SDK covers these gaps, downstream apps can:

• Remove custom schema and response normalizers
• Stop using private CLI internals
• Replace raw HTTP setup code with supported SDK calls
• Produce safer, more diagnosable integration behavior with structured errors

Context

• Project: context-engine-demos
• SDK version in lockfile: context-surfaces==0.0.1

[jeremyplichta]

Reviewed this against the current cloud-context-engine codebase and the latest published Python SDK.

A few concrete findings:

• The latest published context-surfaces is 0.0.3 (the demo repo is still pinned to 0.0.1 today).
• data_source.connection_config support is now present in the typed request models and low-level Python client (ContextSurfacesClient / CreateContextSurfaceRequest), so this part is partially addressed.
• The remaining gaps are still real in the Python SDK surface area:
  • UnifiedClient still does not expose the same data_source creation path
  • the public Python API still lacks a stable file-based data-model parser/export helper outside CLI internals
  • MCP tool discovery is still largely raw dict / Any
  • tool-result normalization is still left to downstream consumers
  • None / null MCP argument handling still benefits from SDK-side omission
  • exception types exist, but status-specific mapping and request metadata are still incomplete
• The generated RQE array schema issue appears to be largely a server-contract/regression-coverage concern now, not primarily a new demo-side workaround problem.
• ensure_agent_key should probably not be implemented literally as originally written, because existing plaintext agent keys are intentionally only returned on create and are not safely reusable by name later.

Bottom line: the next step should be to fix these in the SDK / cloud-context-engine first, then come back and simplify context-engine-demos against the released SDK APIs. Doing more app-level workaround work in this repo would mostly duplicate logic that should live upstream.

I opened a parent epic and focused follow-up issues in redislabsdev/cloud-context-engine to track that SDK-first work:

• Epic: redislabsdev/cloud-context-engine#132
• Child issues: #133, #134, #135, #136, #137, #138, #140

Once those land and a new SDK version is published, this repo should be able to remove most of the current workaround layer instead of growing it.