fix(extractor): don't skip JSON responses carrying natural language (#24)

[santosh2345]

Closes #24.

_is_trivial_response() blanket-skipped any response starting with { or [. So an LLM answer wrapped in JSON — e.g. {"answer": "The user's budget is $20k, they prefer Python"} — was treated as trivial and produced zero extracted beliefs.

Note on the proposed fix

The issue suggested if classify_response_type(text) in ("json", "sql", "code"): return True. That alone doesn't fix the issue's own example: the example is valid JSON, so classify_response_type returns "json" and it would still be skipped. This PR skips json only when it's a pure data payload.

Changes

• Replace the naive startswith('{' / '[') check with classify_response_type:
  • "sql" / "code" → trivial (skip).
  • "json" → trivial only when the payload's string values contain no meaningful prose (< 5 word-like tokens). {"key": "value"} stays trivial; JSON wrapping a real sentence does not.
• Make the code-character density heuristic whitespace-insensitive, so symbol soup like "{ } ( ) = ;" is still caught now that the startswith guard is gone.

Tests

• Regression: JSON-with-prose → not trivial.
• Pure-data JSON ({"key": "value"}, [{"key": "value"}]) → still trivial.
• SQL and code fences → still trivial.
• All existing _is_trivial_response tests kept.

Verification

pytest tests/test_new_features.py tests/test_extractor.py tests/test_extractor_helpers.py → 95 passed. Full suite: 389 passed (2 pre-existing failures are unrelated missing optional deps — flask, litellm).

[abhay-2108]

Excellent work resolving #24. The recursive JSON walk is clean, and the whitespace-insensitive updates to the density check are a great touch.

One small suggestion to make this future-proof:

Internationalization (i18n) Support: The _json_prose_word_count function uses re.findall(r"[A-Za-z]{2,}", obj) to count words. If a developer uses a non-English language (like Spanish, French, Chinese, or Japanese), this regex will under-count or miss the prose, causing those valid JSON responses to be incorrectly flagged as trivial.

Suggested Fix: Using a unicode-aware regex pattern like \w{2,} (or a basic character length count on string values) will make this language-agnostic.