Platform related foundations#728
Open
mc-dorzo wants to merge 175 commits into
Open
Conversation
mc-dorzo
force-pushed
the
feature/emcie-platform
branch
from
441227e to
f4638bf
Compare
mc-dorzo
force-pushed
the
feature/emcie-platform
branch
2 times, most recently
from
ba1a3a0 to
7904739
Compare
mc-dorzo
force-pushed
the
feature/emcie-platform
branch
from
d761194 to
7aa1563
Compare
mc-dorzo
force-pushed
the
feature/emcie-platform
branch
5 times, most recently
from
85676df to
6d648ab
Compare
mc-dorzo
force-pushed
the
feature/emcie-platform
branch
2 times, most recently
from
e8b8dc8 to
9d4c0a4
Compare
mc-dorzo
force-pushed
the
feature/emcie-platform
branch
3 times, most recently
from
d2daa53 to
5ee10ff
Compare
mc-dorzo
force-pushed
the
feature/emcie-platform
branch
from
dedc07a to
6f00c84
Compare
Signed-off-by: Dor Zohar <dor@emcie.co>
…racer - Add _generate_trace_id() method to base Tracer class for consistent trace ID generation - Convert EmcieSpanData to dataclass for cleaner code - Add TypedDict definitions for event structures with proper transformation rules - Implement CompositeTracer that coordinates trace_ids across multiple tracers - Update all tracer implementations to use consistent trace_id generation - Add comprehensive event filtering for EmcieTracer (keep rationale, remove sensitive data) - Ensure mypy type safety across all tracer code Signed-off-by: Dor Zohar <dor@emcie.co>
Signed-off-by: Dor Zohar <dor@emcie.co>
… emcie module with CompositeTracer Signed-off-by: Dor Zohar <dor@emcie.co>
Signed-off-by: Dor Zohar <dor@emcie.co>
Signed-off-by: Dor Zohar <dor@emcie.co>
Signed-off-by: Dor Zohar <dor@emcie.co>
Signed-off-by: Bar Karov <bar@emcie.co> Signed-off-by: Dor Zohar <dor@emcie.co>
Signed-off-by: Dor Zohar <dor@emcie.co>
Introduce Operation.STREAM_LOGS for log streaming endpoints Add check_websocket_permission and authorize_websocket methods to AuthorizationPolicy Implement websocket authorization logic in development and production policies Update AuthorizationException to support WebSocket Prepare for secure websocket access control Signed-off-by: Dor Zohar <dor@emcie.co>
Signed-off-by: Dor Zohar <dor@emcie.co>
Signed-off-by: Dor Zohar <dor@emcie.co>
Signed-off-by: Dor Zohar <dor@emcie.co>
- NLP services: ruff format had wrapped Foo[t](...) calls onto multiple lines, leaving "# type: ignore" on the closing ")" line where mypy ignores the wrong line. Move it onto the line that contains [t]. Affects mistral_service.py and 11 sites in openai_service.py. - HealthReporter tests: HealthReporter and NullHealthReporter now require application_context. Pass an ApplicationContext(instance_id="test"). Affects test_embedding.py, test_generation.py, test_litellm_service.py. - ToolInsights tests: ToolInsights.missing_data / invalid_data are now Mapping[ToolId, Mapping[ToolCallId, Sequence[X]]]. Flatten in the assertions instead of indexing as a flat list. ToolInsights.evaluations similarly nests; construct the test fixture accordingly. Drop a stale "score" kwarg from create_guideline_match calls (no longer a field on the helper). - relational_resolver.py: declare the _tag_store attribute the new develop code paths use. - guidelines.py CompositeGuidelineStore: forward the new "title" parameter so its signature matches the base GuidelineStore. Signed-off-by: Dor Zohar <dor@emcie.co>
Adds an "evaluation" attribute to tc.result, tc.missing, and tc.invalid, and a new tc.skipped event for tool calls the resolver decided to skip (DATA_ALREADY_IN_CONTEXT) — previously invisible in the trace. The label uses a new ToolEvaluation enum (parallel to MatchReason) so the trace value is decoupled from ToolCallEvaluation's internal enum value. Notably ToolCallEvaluation.NEEDS_TO_RUN.value is "success", which would be misleading as a trace label; the new enum exposes "needs_to_run" instead. Signed-off-by: Dor Zohar <dor@emcie.co>
Migrates the five canned-response trace events emitted by CannedResponseGenerator (canrep.preamble_generated, canrep.ttfm, canrep.streaming.ttfm, canrep.draft, canrep.selected) to typed methods on EngineTracer. Same event names, same payloads. Adds an is_fallback: bool attribute on canrep.selected and emits the event at the three previously-silent no-match paths (strict mode with no canreps, selection failure, and chosen-id-not-in-rendered-list). Trace consumers can now distinguish a real selection from a no-match-provider fallback. Also normalizes canrep.draft's insights attribute to always be a list (defaulting to ["N/A"] when absent) — was previously list[str] or the bare string "N/A" depending on the branch. Signed-off-by: Dor Zohar <dor@emcie.co>
Mirrors the recent Tag and Relationship changes: Agent now exposes last_modified_utc alongside creation_utc. AgentDocumentStore bumps 0.5.0 -> 0.6.0 with a v0_5_0 -> v0_6_0 migration that defaults last_modified to the existing creation_utc on legacy documents. Three write paths bump last_modified to "now": - update_agent (any update via AgentUpdateParams) - upsert_tag (a tag added to the agent) - remove_tag (a tag removed from the agent) The synthetic Tag built for an agent_id in app_modules/relationships.py now sources its last_modified_utc from agent.last_modified_utc instead of agent.creation_utc. Signed-off-by: Dor Zohar <dor@emcie.co>
Signed-off-by: Dor Zohar <dor@emcie.co>
Introduces parlant.core.engines.alpha.canned_response_source with a NamedTuple (kind, id) plus the CannedResponseSourceKind enum and a GLOBAL_CANNED_RESPONSE_SOURCE singleton. Lets callers tag each candidate canned response with where it came from — agent, agent tag, global, journey, journey node, guideline, or a tool — so the engine can later emit "what triggered this response" attributes on the trace. No callers wired up yet; that comes in subsequent commits. Signed-off-by: Dor Zohar <dor@emcie.co>
EntityQueries.find_canned_responses_for_context now returns a CannedResponseLookup (canned_responses + sources map) instead of a flat sequence. Each per-tag query path stamps a CannedResponseSource onto every canrep it returned, and a canrep tagged in multiple ways accumulates multiple sources. Source kinds: - AGENT: id = agent.id (Tag.for_agent_id) - AGENT_TAG: id = the agent's specific tag id - GLOBAL: GLOBAL_CANNED_RESPONSE_SOURCE singleton - JOURNEY: id = journey.id (Tag.for_journey_id) - JOURNEY_NODE: id = node id (Tag.for_journey_node_id, journey-node guidelines) - GUIDELINE: id = guideline.id (Tag.for_guideline_id) Updates the two call sites in CannedResponseGenerator and three test sites to read .canned_responses off the new lookup. The TOOL source for transient canreps is wired in a subsequent commit. Signed-off-by: Dor Zohar <dor@emcie.co>
CannedResponseGenerator._get_relevant_canned_responses now returns a CannedResponseLookup. Stored canreps inherit their sources from the EntityQueries lookup; transient canreps from staged tool events get a TOOL source whose id is the producing tool_id (one source per (canrep, tool_id) pair, even if multiple tools surfaced the same response). The non-fallback canrep.selected emit now carries the selected response's triggers via two parallel Sequence[str] attributes: trigger_kinds and trigger_ids. The three fallback paths (which use the NoMatchResponseProvider) keep sources empty — is_fallback=True already signals the origin. EngineTracer.canrep_selected gains an optional sources parameter and omits the trigger_* attributes entirely when the source list is empty. Signed-off-by: Dor Zohar <dor@emcie.co>
ToolId keys aren't JSON-serializable as trace attribute values, so convert missing_data and invalid_data dicts to use string keys via ToolId.to_string(). Update event step assertions to traverse the nested dict structure. Signed-off-by: Dor Zohar <dor@emcie.co>
canrep_selected now takes the full CannedResponse object (matching the glossary_term_loaded/Term convention) and emits last_modified_utc as an attribute, so downstream trace consumers can correlate events against canned-response revisions. Signed-off-by: Dor Zohar <dor@emcie.co>
Wire DEPENDENCY_ANY through RelationshipKindDTO and both kind-mapping helpers, and plumb the Relationship.group_id field end-to-end through RelationshipModel / RelationshipDTO / RelationshipCreationParamsDTO so that DEPENDENCY_ANY relationships are addressable from both the relationships API and the guideline API. Also add the missing OVERLAP case to the guideline kind mapping for parity with the relationships API. Signed-off-by: Dor Zohar <dor@emcie.co>
Tool-returned guidelines arrive via the resolver's ``matches`` argument but are not in ``usable_guidelines``, so the per-call ``guidelines_by_id`` lookup built from the stored set didn't know about them. Downstream sites that did a hard ``[gid]`` lookup (e.g. the dep-failure clear loop) would KeyError for those guidelines. Use setdefault so transient guidelines are added to the map without overwriting the authoritative stored copies when both exist. Signed-off-by: Dor Zohar <dor@emcie.co>
Guards the fix from 31ed27d — confirms that a guideline arriving only via ``matches`` (a transient guideline returned by a tool) doesn't KeyError the resolver's per-call guideline_by_id lookup. Without the hydration in resolve(), the dep-failure-clear loop trips at line 328 with KeyError on the transient gid; with hydration in place the resolver returns the transient match with kind NONE. Signed-off-by: Dor Zohar <dor@emcie.co>
Signed-off-by: Dor Zohar <dor@emcie.co>
The engine calls EngineTracer.matches(...) once per preparation iteration. A guideline that survives unchanged across iterations was producing identical gm.selected / journey.state.selected events each time (and the same shape for ruled_out). Add an _add_match_event_if_new helper that keys a per-trace set on (event_name, sorted-attributes-json) and skips emit when the fingerprint already fired in the current trace. The cache resets when the underlying tracer's trace_id changes, so the dedup is naturally scoped per request. If the resolver's mind changes (e.g. a previously selected guideline becomes ruled_out, or a resolution gets added/removed), the event has a different fingerprint and emits normally. Signed-off-by: Dor Zohar <dor@emcie.co>
The previous instance-level dedup state was shared across all concurrent requests handled by the same EngineTracer singleton. Two requests racing each other would each see the other's trace_id, clear the cache repeatedly, and break dedup for both. Move _match_event_seen and _match_event_trace_id into ContextVars so each asyncio task carries its own copy. Default is None and the set is lazily allocated with .set() — that avoids sharing a single mutable default set across tasks. Signed-off-by: Dor Zohar <dor@emcie.co>
Match LocalTracer's style for ContextVar usage in EngineTracer: - ``import contextvars`` and subscripted ``contextvars.ContextVar[T]`` rather than ``from contextvars import ContextVar`` + a separate ``Optional[T]`` annotation. - Empty defaults (``""`` and ``set()``) instead of ``None``. The default set is never mutated — each task allocates a fresh set and calls ``.set()`` on first encountering a new trace_id, so the shared-mutable-default trap doesn't fire. Net effect: same behavior, slightly tighter code, no spurious ``Optional`` types. Signed-off-by: Dor Zohar <dor@emcie.co>
Mirrors the session-facing ``status: ready`` status event in the trace, so consumers can see when the engine reached a ready state (and at which stage, when supplied). Goes through EngineTracer to keep tracer call sites uniform. Signed-off-by: Dor Zohar <dor@emcie.co>
journey_guideline_projection: _inject_sub_node now takes the full link_metadata mapping instead of separate sub_journey_id / link_id arguments. The injected guideline's journey_node metadata is composed by spreading link_metadata (which already carries link_id, sub_journey_id, and sub_journey_last_modified). Future link-scoped keys flow through without further plumbing. link_id is recovered from the mapping for the namespaced id. tracing: drop the _sub_journey_attrs helper from EngineTracer.matches. Both _emit_selected and _emit_ruled_out now read the journey_node metadata once at the top and gate / extract from the same local, removing a redundant cast and the if/elif fallback for the older sub_journey_last_modified_utc key (it's now always sub_journey_last_modified, sourced from link_metadata). Signed-off-by: Dor Zohar <dor@emcie.co>
254ef47 inlined the old _sub_journey_attrs helper but dropped its guard: it hard-accessed journey_node["sub_journey_last_modified"]. For a sub-journey-linked node whose metadata carries sub_journey_id but not sub_journey_last_modified (older / cached journey projections), _emit_selected / _emit_ruled_out raised KeyError. That aborts the emit loop in matches(), so every journey.state.* (and any remaining gm.*) event for that call disappears — matching the "no journey events at all" symptom. Restore the guard: build the sub-journey attrs as a local dict, including sub_journey_last_modified only when present. Behavior for well-formed metadata is unchanged. Signed-off-by: Dor Zohar <dor@emcie.co>
…lookup" This reverts commit 0676651. Signed-off-by: Dor Zohar <dor@emcie.co>
Reverts the dedup introduced in ce302ee / 7a7f207 / 2bbcb0d. The dedup state lived on the singleton EngineTracer in ContextVars. 2bbcb0d switched the seen-set default from None to a shared ``set()`` and dropped the ``seen is None`` guard; under asyncio task-context inheritance (preparation runs parallel matching via safe_gather, and the tracer's trace_id is inherited by child tasks) the ``else`` branch could hand back that single shared default set. Fingerprints then accumulated globally and never cleared, so once a journey/guideline event fired it was suppressed for the rest of the process — including the first occurrence in later requests. Net effect: journey.state.* (and gm.*) events silently disappeared. In-tracer dedup on a process-lifetime singleton is the wrong place for this. Drop it entirely and go back to emitting one event per preparation iteration (the prior, correct behavior — duplicates across iterations are expected and meaningful: they show the resolver re-evaluating after tool calls). If we want dedup later it should be request-scoped state owned by the engine, not the tracer. Signed-off-by: Dor Zohar <dor@emcie.co>
A journey with no triggers is a sub-journey: it can only be entered via a parent journey's link, never self-activated. _sort_journeys_by_ relevance was ranking the full available_journeys set, so a sub-journey could be picked into high_prob_journeys purely by semantic relevance. That guideline then can't survive pruning anyway: find_guidelines_for_ context skips projecting trigger-less / node_properties-less journeys into all_stored_guidelines, while find_journey_related_guidelines projects unconditionally — so high_prob_journey_related_ids referenced node guidelines that were never in all_stored_guidelines, and the journey silently failed to contribute anything. Restrict the relevance-ranked candidate pool to triggerable journeys. Sub-journeys still count as high-probability when they legitimately appear in journey_paths (entered via their link and active) — that path is unchanged. Signed-off-by: Dor Zohar <dor@emcie.co>
ParlantCloudTracer, ParlantCloudLogger, ParlantCloudMeter now live in parlant/adapters/observability/. Auto-enabled when PARLANT_CLOUD_API_KEY is set — validates against PARLANT_CLOUD_OTEL_URL, then appends to the container's CompositeTracer/CompositeLogger. Zero code changes needed by users — just set the env vars. Signed-off-by: Dor Zohar <dor@emcie.co>
… var _setup_parlant_cloud_observability() now reads project_id from the auth endpoint response instead of requiring a separate PARLANT_PROJECT_ID environment variable. Only PARLANT_CLOUD_API_KEY is needed. Signed-off-by: Dor Zohar <dor@emcie.co>
…loud.py Move ParlantCloudTracer, ParlantCloudLogger, ParlantCloudMeter and configure_container() into a single file at adapters/modules/parlant_cloud.py. Remove the adapters/observability/ directory. Server auto-loads the module when PARLANT_CLOUD_API_KEY is set, reading project_id from the auth response. Signed-off-by: Dor Zohar <dor@emcie.co>
Signed-off-by: Dor Zohar <dor@emcie.co>
ParlantCloudTracer now injects the Parlant-generated trace_id into the OTEL span context so both systems use the same ID. Previously the OTEL SDK generated its own trace_id, causing the collector to store a different ID than what the frontend uses for lookup. Signed-off-by: Dor Zohar <dor@emcie.co>
The OTEL SDK ignores NonRecordingSpan parents with INVALID_SPAN_ID. Using is_remote=True with a valid span_id makes start_span inherit the trace_id from the seeded parent context. Signed-off-by: Dor Zohar <dor@emcie.co>
mc-dorzo
force-pushed
the
feature/emcie-platform
branch
from
b0ca847 to
1191d41
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.