Doc updates ,extending monocle for python example, scope guide WIP

Signed-off-by: Prasad Mujumdar <prasad@okahu.ai>

Author: prasad-okahu

Monocle_scopes.md

@@ -0,0 +1,26 @@
+# This guide describes various options to set trace and scopes in your application
+
+
+### Set scope programatically in python 
+```python
+from monocle_apptrace import monocle_trace_scope
+...
+with monocle_trace_scope("Conversation"):
+    while True:
+        message = input("How can I help you:")
+        cleaned_message = gaurdrail_chai(message)
+        result = rag_chat_chain.invoke(message)
+```
+The above code will generate two traces (one per chain invocation). All the spans in these traces will have an attribute called `Conversaion` with a unique value.
+```json
+"attributes": {
+    "span.type": "inference",
+    ...
+    "scope.conversation": "0xcb80e6f772968ed50ead80657b09cf52",
+```
+
+### Set scope programatically in typescript
+TBD
+
+### Set scope declaratively
+You can set scope via a configuration file at the method level or to track http header, without having to make changes 

documentation/Extending_monocle.md

@@ -0,0 +1,260 @@
+# Custom Instrumentation with Monocle
+
+This guide demonstrates how to use Monocle to instrument OpenAI and Vector DB interactions, collecting telemetry data to analyze and monitor their performance.
+
+## Overview
+
+The example includes the following components:
+
+- **OpenAI Client (`openai_client.py`)**: A client for interacting with OpenAI's Chat API
+- **Vector Database (`vector_db.py`)**: An in-memory vector database with OpenAI embeddings
+- **Output Processors**: Configuration files that define how to extract and structure telemetry data
+- **Example script**: Shows how to instrument and run the application
+
+## Component Details
+
+### OpenAI Client
+
+`OpenAIClient` is a wrapper around the OpenAI API that provides methods for:
+
+- Making chat completion requests via the `chat()` method
+- Formatting messages for the API using `format_messages()`
+- Handling API responses and errors
+
+```python
+# Initialize client
+client = OpenAIClient()
+
+# Format messages and send to OpenAI
+messages = client.format_messages(
+    system_prompts=["You are a helpful assistant."],
+    user_prompts=["Tell me a joke about programming."]
+)
+response = client.chat(messages=messages, model="gpt-3.5-turbo")
+```
+
+### Vector Database
+
+`InMemoryVectorDB` is a simple vector database implementation that:
+
+- Converts text to vector embeddings using OpenAI's embedding API
+- Stores vectors with associated metadata
+- Performs similarity searches using cosine similarity
+
+```python
+# Initialize vector database
+vector_db = InMemoryVectorDB()
+
+# Store documents
+vector_db.store_text("doc1", "Python is a programming language", {"source": "docs"})
+
+# Search for similar documents
+results = vector_db.search_by_text("programming languages", top_k=2)
+```
+
+## Instrumenting Your Code with Monocle
+
+### 1. Define Output Processors
+
+Output processors define what data to extract from your methods. Two examples are provided:
+
+#### Inference Output Processor
+
+`output_processor_inference.py` defines how to extract data from OpenAI chat completions:
+
+```python
+INFERENCE_OUTPUT_PROCESSOR = {
+    "type": "inference",
+    "attributes": [
+        [
+            # Entity attributes for the provider
+            {
+                "attribute": "type",
+                "accessor": lambda arguments: "openai"
+            },
+            {
+                "attribute": "deployment",
+                "accessor": lambda arguments: arguments['kwargs'].get('model', 'unknown')
+            },
+            # More attributes...
+        ]
+    ],
+    "events": [
+        {
+            "name": "data.input",
+            "attributes": [
+                {
+                    "attribute": "input",
+                    "accessor": lambda arguments: [
+                        msg["content"] 
+                        for msg in arguments['kwargs'].get('messages', [])
+                    ] if isinstance(arguments['kwargs'].get('messages'), list) else []
+                }
+            ]
+        },
+        # More events...
+    ]
+}
+```
+
+#### Vector DB Output Processor
+
+`output_processor_vector.py` defines how to extract data from vector database operations:
+
+```python
+VECTOR_OUTPUT_PROCESSOR = {
+    "type": "retrieval",
+    "attributes": [
+        [
+            # Vector store attributes
+            {
+                "attribute": "name",
+                "accessor": lambda arguments: type(arguments["instance"]).__name__,
+            },
+            # More attributes...
+        ]
+    ],
+    "events": [
+        {
+            "name": "data.input",
+            "attributes": [
+                {
+                    "attribute": "input",
+                    "accessor": lambda arguments: arguments["args"][0] if arguments["args"] else None
+                }
+            ]
+        },
+        # More events...
+    ]
+}
+```
+
+### 2. Accessor Functions
+
+The key to instrumentation is the `accessor` function, which extracts data from method calls:
+
+- `arguments["instance"]`: The object instance (e.g., the OpenAIClient or InMemoryVectorDB)
+- `arguments["args"]`: Positional arguments passed to the method
+- `arguments["kwargs"]`: Keyword arguments passed to the method
+- `arguments["result"]`: The return value from the method call
+
+These give you access to all inputs, outputs, and context of the instrumented methods.
+
+### 3. Configure Instrumentation
+
+Set up Monocle's telemetry system with your output processors:
+
+```python
+from monocle_apptrace.instrumentation.common.wrapper_method import WrapperMethod
+from monocle_apptrace.instrumentation.common.instrumentor import setup_monocle_telemetry
+
+setup_monocle_telemetry(
+    workflow_name="openai.app",
+    wrapper_methods=[
+        WrapperMethod(
+            package="openai_client",           # Module name
+            object_name="OpenAIClient",        # Class name
+            method="chat",                     # Method to instrument
+            span_name="openai_client.chat",    # Span name in telemetry
+            output_processor=INFERENCE_OUTPUT_PROCESSOR
+        ),
+        # More method wrappers...
+    ]
+)
+```
+
+## Running the Example
+
+1. Ensure you have your **OpenAI API key** available:
+
+```bash
+export OPENAI_API_KEY=your_api_key_here
+```
+
+2. Install the required packages:
+
+```bash
+pip install -r requirements.txt
+```
+
+3. Run the example script:
+
+```bash
+python example.py
+# Or use the provided shell script
+./run_example.sh
+```
+
+## Understanding the Telemetry Output
+
+Monocle generates JSON trace files in your directory with names like:
+`monocle_trace_openai.app_<trace_id>_<timestamp>.json`
+
+### Output Format
+
+The trace files contain structured telemetry data:
+
+```json
+{
+    "name": "openai_client.chat",
+    "context": { /* trace context */ },
+    "attributes": {
+        "entity.2.type": "openai",
+        "entity.2.provider_name": "OpenAI",
+        "entity.2.deployment": "gpt-3.5-turbo",
+        "entity.2.inference_endpoint": "https://api.openai.com/v1",
+        "entity.3.name": "gpt-3.5-turbo",
+        "entity.3.type": "model.llm.gpt-3.5-turbo"
+    },
+    "events": [
+        {
+            "name": "data.input",
+            "timestamp": "2025-02-27T10:36:49.985586Z",
+            "attributes": {
+                "input": [
+                    "You are a helpful AI assistant.",
+                    "Tell me a short joke about programming."
+                ]
+            }
+        },
+        {
+            "name": "data.output",
+            "attributes": {
+                "response": "Why do programmers prefer dark mode? Because the light attracts bugs!"
+            }
+        },
+        {
+            "name": "metadata",
+            "attributes": {
+                "prompt_tokens": 26,
+                "completion_tokens": 14,
+                "total_tokens": 40
+            }
+        }
+    ]
+}
+```
+
+### Key Elements
+
+1. **Attributes**: Contains information about the instrumented entity:
+   - Model name and type
+   - Deployment details
+   - API endpoints
+   - Provider information
+
+2. **Events**: Contains captured events during the method execution:
+   - `data.input`: The inputs provided to the method
+   - `data.output`: The response or results from the method
+   - `metadata`: Additional information like token usage
+
+## Customizing for Your Application
+
+To instrument your own code:
+
+1. Create output processors tailored to your methods
+2. Use accessor functions to extract the data you need
+3. Set up telemetry with your method wrappers
+4. Run your application and analyze the generated traces
+
+By customizing the output processors, you can collect exactly the telemetry data you need from any Python method.

documentation/Monocle_User_Guide.md

@@ -210,49 +210,11 @@ Monocle exporters handle storing the trace for future analysis. By default each
 
 ## Using scopes
 Imagine you have a chatbot application that supports a long conversion ie multiple question/answer back and forth between end user and bot. It uses various genAI tech components like LLMs and vector stores. A simple instrumentation will generate a trace per genAI API call (eg invocation of a framework chat or direct OpenAI API). As the app developer or owner, you are more interested in tracking the conversions than just APIs. The scopes in Monocle enables that use case. 
-You can set the scope in application either programatically or declaratively. You can specific a value for scope or Monocle will generate a unique value (GUID) which gives you options to choose what's best suited for your use case. Please see the different API references and configuration reference below for the all the available options.
-TBD!!
+You can set the scope in application either programatically or declaratively. You can specific a value for scope or Monocle will generate a unique value (GUID) which gives you options to choose what's best suited for your use case. Please see the [Monocle scopes guide](Monocle_scopes.md) for the details and examples.
 
-### Set scope for a python
-```python
-from monocle_apptrace import monocle_trace_scope
-...
-with monocle_trace_scope("Conversation"):
-    while True:
-        message = input("How can I help you:")
-        cleaned_message = gaurdrail_chai(message)
-        result = rag_chat_chain.invoke(message)
-```
-The above code will generate two traces (one per chain invocation). All the spans in these traces will have an attribute called `Conversaion` with a unique value.
-```json
-"attributes": {
-    "span.type": "inference",
-    ...
-    "scope.conversation": "0xcb80e6f772968ed50ead80657b09cf52",
-```
-
-### Set scope for a typescript method
-TBD
 
 ## Extending Monocle
-TBD
-
-
-##Monocle Reference
-### Python APIs
-#### Enable tracing
-#### Trace
-#### Scopes
-#### Customization
-
-### Typescript APIs
-#### Enable tracing
-#### Trace
-#### Scopes
-#### Customization
+If you are using a genAI technology that's not yet supported by Monocle out of the box or have you own proparitory code, you can extend monocle to generate traces in the Monocle format. Please refer to [extending monocle guide](Extending_monocle.md) 
 
-### Configuration reference
-#### Scope
-#### Exporters
 
 

documentation/examples/custom/custom_instrumentation/__init__.py

@@ -0,0 +1,3 @@
+from .openai_client import OpenAIClient
+
+__all__ = ['OpenAIClient']