Merge branch 'master' of github.com:Stoobly/stoobly-agent

Author: J3utter

docs/mock-request-matching.md

@@ -4,9 +4,24 @@
 
 How an **incoming proxied request** is matched to a **recorded row** in the local DB for mocks: **hash-based** identity, optional **match rules** that drop hash dimensions, and optional **`compute`** to re-hash stored `raw` when **ignored components** change between record time and the retry path. Custom status codes: **`IGNORE_COMPONENTS = 498`**, **`NOT_FOUND = 499`** ([`custom_response_codes`](../stoobly_agent/app/proxy/constants/custom_response_codes.py)) — not standard HTTP 404/498.
 
-### Remote project key and `compute`
+### Endpoint lookup and `compute`
 
-When **local** + **remote project key**: **`compute='1'`** is attached only if **`retry`** and non-empty **`ignored_components`** after [`eval_request`](../stoobly_agent/app/proxy/mock/eval_request_service.py) ([`COMPUTE`](../stoobly_agent/config/constants/query_params.py)). That widens the ORM query and runs [`filter_requests_by_hashes`](../stoobly_agent/app/models/factories/resource/local_db/helpers/filter_requests_by_hashes_service.py) so stored **`raw`** is re-hashed with the same ignores as the live request.
+When mocking against **local DB**, [`eval_request`](../stoobly_agent/app/proxy/mock/eval_request_service.py) may attach an `endpoint_promise` from either:
+
+- remote project endpoint search via [`search_endpoint`](../stoobly_agent/app/proxy/mock/search_endpoint.py), or
+- OpenAPI endpoint search via [`search_open_api_endpoint`](../stoobly_agent/app/proxy/mock/search_open_api_endpoint.py).
+
+If `endpoint_promise` exists, and the request is on **retry** with non-empty ignored components, **`compute='1'`** is attached ([`COMPUTE`](../stoobly_agent/config/constants/query_params.py)). That widens the ORM query and runs [`filter_requests_by_hashes`](../stoobly_agent/app/models/factories/resource/local_db/helpers/filter_requests_by_hashes_service.py) so stored **`raw`** is re-hashed with the same ignores as the live request.
+
+### Endpoint cache behavior (remote + OpenAPI)
+
+[`endpoint_cache`](../stoobly_agent/app/proxy/mock/endpoint_cache.py) is a singleton used by both search paths. It:
+
+- caches parsed OpenAPI specs by normalized absolute path,
+- caches remote endpoint index calls by `(project_id, index_params)`,
+- merges both layers into one endpoint-id map, where **latest merge wins** on ID collisions,
+- returns OpenAPI-derived ignored components from optional/nondeterministic fields (query/header/body/response-header),
+- prefetches remote endpoints from settings when remote mode is enabled.
 
 ### Hash dimensions
 
@@ -22,7 +37,7 @@ When **local** + **remote project key**: **`compute='1'`** is attached only if *
 flowchart TB
   subgraph prep [Optional prep — handle_request_mock_generic]
     IG{ignore_rules non-empty?}
-    IG -->|yes| MERGE[Merge rewrite → ignored_components]
+    IG -->|yes| MERGE[Merge ignore rules → ignored_components]
     IG -->|no| IG0[Use ignored_components as-is]
     MERGE --> INIT
     IG0 --> INIT
@@ -42,7 +57,7 @@ flowchart TB
 
   subgraph opt_retry [Options inside eval_request on retry]
     CP{Add compute=1?}
-    CP -->|yes| WCOMP["compute=1: local resource AND remote project key AND len(ignored_components) > 0"]
+    CP -->|yes| WCOMP["compute=1: local resource AND endpoint_promise AND retry AND len(ignored_components) > 0"]
     CP -->|no| NOCOMP[Query without compute]
     WCOMP --> RES2[response]
     NOCOMP --> RES2
@@ -87,17 +102,18 @@ flowchart TB
   Strip --> ORM2[where_for coarse candidates]
   ORM2 --> Rows2[candidates]
   Rows2 --> IG[ignored_components from ENDPOINT_PROMISE]
-  IG --> Filt[filter_requests_by_hashes on raw]
-  Filt --> CLR[clear ENDPOINT_PROMISE in query_params]
-  CLR --> Rows3[rows]
+  IG --> CF{ignored_components non-empty?}
+  CF -->|yes| Filt[filter_requests_by_hashes on raw]
+  CF -->|no| Rows3[rows]
+  Filt --> Rows3[rows]
   Rows1 --> Pick[pick row or scenario tiebreak]
   Rows3 --> Pick
   Pick --> Got{row found?}
   Got -->|yes| OK[transform stored response]
   Got -->|no| NFD[no matching row]
   NFD --> RY{retry truthy?}
   RY -->|yes| R499[499 CustomNotFoundResponseBuilder]
-  RY -->|no| EP2{endpoint_promise yields ignores?}
+  RY -->|no| EP2{not retry AND component hashes exist AND endpoint yields ignores?}
   EP2 -->|yes| R498[498 IgnoreComponentsResponseBuilder]
   EP2 -->|no| R499
 ```
@@ -110,6 +126,9 @@ flowchart TB
 |--------|----------|
 | Mock entry, retry, fixtures | [`handle_mock_service.py`](../stoobly_agent/app/proxy/handle_mock_service.py) |
 | Query / hashes / match rules / `compute` | [`eval_request_service.py`](../stoobly_agent/app/proxy/mock/eval_request_service.py) |
+| Endpoint cache + OpenAPI ignored-component derivation | [`endpoint_cache.py`](../stoobly_agent/app/proxy/mock/endpoint_cache.py) |
+| Remote endpoint search adapter | [`search_endpoint.py`](../stoobly_agent/app/proxy/mock/search_endpoint.py) |
+| OpenAPI endpoint search adapter | [`search_open_api_endpoint.py`](../stoobly_agent/app/proxy/mock/search_open_api_endpoint.py) |
 | Local DB lookup, strip columns, not found 498/499 | [`request_adapter.py`](../stoobly_agent/app/models/factories/resource/local_db/request_adapter.py) |
 | Candidate filtering | [`filter_requests_by_hashes_service.py`](../stoobly_agent/app/models/factories/resource/local_db/helpers/filter_requests_by_hashes_service.py) |
 | Hashing | [`hashed_request_decorator.py`](../stoobly_agent/app/proxy/mock/hashed_request_decorator.py) |

stoobly_agent/app/api/configs_controller.py

@@ -31,7 +31,7 @@ def instance(cls):
     # GET /configs/policies
     def policies(self, context):
         settings = Settings.instance()
-        active_mode = settings.proxy.intercept.active
+        active_mode = settings.proxy.intercept.mode
 
         if active_mode in [mode.MOCK, mode.TEST]:
             context.render(

stoobly_agent/app/cli/helpers/iterate_group_by.py

@@ -28,7 +28,7 @@ def iterate_group_by(alias_name: str, trace_context: TraceContext, handler: Call
       _trace_alias.trace_request_id = trace_request.id
       _trace_alias.save()
 
-    trace_context = TraceContext(trace_context.endpoints_resource, trace)
+    trace_context = TraceContext(trace_context.endpoint_cache, trace)
 
     handler(trace_context)
 

stoobly_agent/app/cli/helpers/openapi_endpoint_adapter.py

@@ -1,4 +1,5 @@
 import copy
+import hashlib
 import itertools
 import logging
 import re
@@ -15,13 +16,41 @@
 
 from stoobly_agent.lib.api.interfaces.endpoints import (
   Alias,
-  EndpointShowResponse,
+  OpenApiEndpointShowResponse,
   RequestComponentName,
 )
 from stoobly_agent.lib.utils.python_to_ruby_type import convert_reverse
 
 from .schema_builder import SchemaBuilder
 
+
+def _md5_hex(value: str) -> str:
+  return hashlib.md5(value.encode('utf-8')).hexdigest()
+
+
+def _endpoint_service_id(parsed_url, port_str: str) -> str:
+  hostname = parsed_url.hostname or ""
+  if hostname:
+    hostname = hostname.lower()
+  return _md5_hex(f"{hostname}:{port_str}")
+
+
+def compute_openapi_service_id(hostname: str, port: str) -> str:
+  """MD5 hex of ``{hostname}:{port}`` (hostname lowercased). Used by tests and callers."""
+  hn = (hostname or "").lower()
+  return _md5_hex(f"{hn}:{port}")
+
+
+def compute_openapi_endpoint_id(service_id: str, match_pattern: str, method: str) -> str:
+  """
+  MD5 hex of ``service_id``, ``match_pattern``, and HTTP ``method`` (concatenated).
+
+  ``method`` is included so operations that share the same ``match_pattern`` (e.g. GET vs POST)
+  do not receive the same ``id``.
+  """
+  return _md5_hex(f"{service_id}{match_pattern or ''}{method}")
+
+
 class OpenApiEndpointAdapter():
   def __init__(self, strict_refs=False):
     """
@@ -34,7 +63,7 @@ def __init__(self, strict_refs=False):
     self.spec = None
     self.strict_refs = strict_refs
 
-  def adapt_from_file(self, file_path) -> List[EndpointShowResponse]:
+  def adapt_from_file(self, file_path) -> List[OpenApiEndpointShowResponse]:
     spec = {}
 
     with open(file_path, "r") as stream:
@@ -52,10 +81,9 @@ def adapt_from_file(self, file_path) -> List[EndpointShowResponse]:
 
     return self.adapt(spec)
 
-  def adapt(self, spec: SchemaPath) -> List[EndpointShowResponse]:
+  def adapt(self, spec: SchemaPath) -> List[OpenApiEndpointShowResponse]:
     self.spec = spec  # Store spec for use in __dereference
     endpoints = []
-    endpoint_counter = 0
     components = spec.get("components", {})
     schemas = components.get("schemas", {})
     paths = spec.getkey('paths')
@@ -81,11 +109,8 @@ def adapt(self, spec: SchemaPath) -> List[EndpointShowResponse]:
           if http_method not in path:
             continue
 
-          endpoint_counter += 1
-
           parsed_url = urlparse(url)
-          endpoint: EndpointShowResponse = {}
-          endpoint['id'] = endpoint_counter
+          endpoint: OpenApiEndpointShowResponse = {}
           endpoint['method'] = http_method.upper()
           endpoint['host'] = '-' if parsed_url.netloc == '' else parsed_url.netloc
 
@@ -124,6 +149,10 @@ def adapt(self, spec: SchemaPath) -> List[EndpointShowResponse]:
           else:
             endpoint['port'] = str(parsed_url.port)
 
+          endpoint['service_id'] = _endpoint_service_id(parsed_url, endpoint['port'])
+          mp = endpoint.get('match_pattern') or ''
+          endpoint['id'] = _md5_hex(endpoint['service_id'] + mp + endpoint['method'])
+
           alias_counter = 0
           header_param_counter = 0
           operation = path[http_method]
@@ -465,7 +494,7 @@ def __dereference(self, components: SchemaPath, reference: str, spec: SchemaPath
 
     return current 
 
-  def __convert_literal_component_param(self, endpoint: EndpointShowResponse,
+  def __convert_literal_component_param(self, endpoint: OpenApiEndpointShowResponse,
       required_component_params: List[str], literal_component_params: Union[dict, list],
       component_name: str, literal_component_name: str) -> None:
 
@@ -603,7 +632,7 @@ def __evaluate_servers(self, servers: SchemaPath) -> List[dict]:
 
     return result
 
-  def __parse_responses(self, endpoint: EndpointShowResponse, responses: SchemaPath, components: SchemaPath):
+  def __parse_responses(self, endpoint: OpenApiEndpointShowResponse, responses: SchemaPath, components: SchemaPath):
     for response_code, response_definition in responses.items():
       # Only support status code 200 for now
       if response_code != '200':

stoobly_agent/app/cli/helpers/replay_facade.py

@@ -15,6 +15,7 @@ class ReplayCliOptions(TypedDict):
   group_by: str
   host: str
   lifecycle_hooks_path: str
+  openapi_specification_path: str
   on_response: Callable
   project_key: str
   public_directory_path: str  # Comma-separated list of paths, optionally with origin prefix
@@ -58,6 +59,7 @@ def common_cli_options(self, cli_options: ReplayCliOptions) -> ReplayCliOptions:
       'group_by': cli_options.get('group_by'),
       'host': cli_options.get('host'),
       'lifecycle_hooks_path': cli_options.get('lifecycle_hooks_path'),
+      'openapi_specification_path': cli_options.get('openapi_specification_path'),
       'overwrite': cli_options.get('overwrite'),
       'public_dir_path': cli_options.get('public_dir_path'),
       'request_origin': request_origin.CLI,

stoobly_agent/app/cli/helpers/synchronize_request_service.py

@@ -42,7 +42,7 @@ class SynchronizeRequestService():
   local_db_request_adapter: LocalDBRequestAdapter
 
   def synchronize_request(self, request: Request, endpoint: EndpointShowResponse, lifecycle_hooks = {}):
-    facade = EndpointFacade(None, -1).with_show_response(endpoint)
+    facade = EndpointFacade(-1).with_show_response(endpoint)
     mitmproxy_request = OrmRequestAdapterFactory(request).mitmproxy_request()
 
     # Query Params

stoobly_agent/app/cli/helpers/trace_context_facade.py

@@ -3,17 +3,16 @@
 from typing import List
 
 from stoobly_agent.app.cli.helpers.trace_aliases import parse_aliases
+from stoobly_agent.app.proxy.mock.endpoint_cache import endpoint_cache
 from stoobly_agent.app.proxy.replay.trace_context import TraceContext
 from stoobly_agent.app.settings import Settings
-from stoobly_agent.lib.api.endpoints_resource import EndpointsResource
 from stoobly_agent.lib.orm.trace import Trace
 
 class TraceContextFacade():
 
   def __init__(self, settings: Settings, trace: Trace = None):
     self.__settings = settings
-    self.__endpoints_resource = EndpointsResource(self.__settings.remote.api_url, self.__settings.remote.api_key)
-    self.__trace_context = TraceContext(self.__endpoints_resource, trace)
+    self.__trace_context = TraceContext(endpoint_cache, trace)
 
   @property
   def trace_context(self):

stoobly_agent/app/cli/request_cli.py

@@ -180,6 +180,7 @@ def snapshot_list(**kwargs):
 @click.option('--group-by', help='Repeat for each alias name.')
 @click.option('--host', help='Rewrite request host.')
 @click.option('--lifecycle-hooks-path', help='Path to lifecycle hooks script.')
+@click.option('--openapi-specification-path', help='Path to OpenAPI specification file.')
 @click.option(
   '--log-level', default=logger.WARNING, type=click.Choice(log_levels), 
   help=f'''

stoobly_agent/app/cli/scenario_cli.py

@@ -207,6 +207,7 @@ def snapshot_list(**kwargs):
         Configure which logs to print. Defaults to {logger.WARNING}.
     '''
 )
+@click.option('--openapi-specification-path', help='Path to OpenAPI specification file.')
 @click.option(
   '--output-level', default=test_output_level.PASSED, type=click.Choice([test_output_level.FAILED, test_output_level.SKIPPED, test_output_level.PASSED]),
   help=f'''

stoobly_agent/app/models/factories/resource/local_db/request_adapter.py

@@ -76,9 +76,16 @@ def show(self, request_id: str, **options: RequestShowParams) -> Tuple[RequestSh
   def response(self, **query_params: RequestColumns) -> 'Response':
     self.__adapt_scenario_id(query_params)
 
+    endpoint = None
     request = None
     retry = bool(query_params.get('retry'))
     should_compute = bool(query_params.get(request_query_params.COMPUTE))
+    endpoint_promise = query_params.get(request_query_params.ENDPOINT_PROMISE)
+
+    if endpoint_promise:
+      endpoint = endpoint_promise()
+
+    ignored_components = endpoint.get('ignored_components') if endpoint else []
 
     if not query_params.get('request_id'):
       request_columns = { 'is_deleted': False, **query_params }
@@ -94,8 +101,7 @@ def response(self, **query_params: RequestColumns) -> 'Response':
       # Find most recent matching record
       requests = self.__request_orm.where_for(**request_columns).get()
 
-      if should_compute:
-        ignored_components = self.__ignored_components(query_params.get(request_query_params.ENDPOINT_PROMISE))
+      if should_compute and ignored_components:
         requests = filter_requests_by_hashes(requests, _component_hashes, ignored_components)
 
       if len(requests) > 1 and 'scenario_id' in query_params:
@@ -118,18 +124,20 @@ def response(self, **query_params: RequestColumns) -> 'Response':
         request = None
 
     if not request:
-      endpoint_promise = None
       # Only attempt to provide ignored components when not retrying 
       # and there are component hashes to re-compute. Otherwise matching by hostname and path already
-      if not retry and component_hashes(query_params):
-        endpoint_promise = query_params.get(request_query_params.ENDPOINT_PROMISE)
-      return self.__handle_request_not_found(endpoint_promise) 
+      if endpoint and not retry and component_hashes(query_params):
+        if ignored_components:
+          return IgnoreComponentsResponseBuilder().build(ignored_components)
+      return CustomNotFoundResponseBuilder().build()
 
     response_record = request.response
     if not response_record:
       return CustomNotFoundResponseBuilder().build()
 
     headers = {}
+    if endpoint:
+      headers[custom_headers.MOCK_REQUEST_ENDPOINT_ID] = str(endpoint.get('id'))
     headers[custom_headers.MOCK_REQUEST_ID] = str(request.id)
     headers[custom_headers.MOCK_REQUEST_KEY] = request.key()
     headers[custom_headers.RESPONSE_LATENCY] = str(request.latency)
@@ -140,15 +148,6 @@ def response(self, **query_params: RequestColumns) -> 'Response':
         .transform()
     )
 
-  def __handle_request_not_found(self, endpoint_promise):
-    if endpoint_promise:
-      ignored_components = self.__ignored_components(endpoint_promise)
-
-      if ignored_components:
-        return IgnoreComponentsResponseBuilder().build(ignored_components)
-
-    return CustomNotFoundResponseBuilder().build()
-
   def index(self, **query_params: RequestsIndexQueryParams) -> Tuple[RequestsIndexResponse, int]:
     self.__adapt_scenario_id(query_params)
 
@@ -389,17 +388,4 @@ def __adapt_scenario_id(self, params: dict):
 
     scenario = self.__scenario_orm.find_by(uuid=scenario_id)
     if scenario:
-      params['scenario_id'] = scenario.id
-
-  def __ignored_components(self, endpoint_promise):
-    if not endpoint_promise:
-      return []
-
-    endpoint = endpoint_promise()
-
-    if endpoint:
-      ignored_components = endpoint.get('ignored_components')
-
-      if ignored_components:
-        return ignored_components
-    return []
+      params['scenario_id'] = scenario.id