feat: support human in the loop for TS

.changeset/eight-moons-perform.md

@@ -0,0 +1,5 @@
+---
+"@llamaindex/server": patch
+---
+
+feat: support human in the loop for TS

packages/create-llama/templates/types/llamaindexserver/nextjs/package.json

@@ -12,7 +12,7 @@
   "dependencies": {
     "@llamaindex/openai": "~0.4.0",
     "@llamaindex/server": "~0.2.1",
-    "@llamaindex/workflow": "~1.1.3",
+    "@llamaindex/workflow": "~1.1.8",
     "@llamaindex/tools": "~0.0.11",
     "llamaindex": "~0.11.0",
     "dotenv": "^16.4.7",

packages/server/README.md

@@ -8,6 +8,7 @@ LlamaIndexServer is a Next.js-based application that allows you to quickly launc
 - Edit code and document artifacts in an OpenAI Canvas-style UI
 - Extendable UI components for events and headers
 - Built on Next.js for high performance and easy API development
+- Human-in-the-loop (HITL) support, check out the [Human-in-the-loop](https://github.com/run-llama/create-llama/blob/main/packages/server/examples/hitl/README.md) documentation for more details.
 
 ## Installation
 

packages/server/examples/hitl/README.md

@@ -0,0 +1,172 @@
+# Human in the Loop
+
+This example shows how to use the LlamaIndexServer with a human in the loop. It allows you to start CLI commands that are reviewed by a human before execution.
+
+## Getting Started
+
+### Environment Setup
+
+Export your OpenAI API key:
+
+```bash
+export OPENAI_API_KEY=<your-openai-api-key>
+```
+
+### Starting the Server
+
+Run the server in development mode:
+
+```bash
+npx nodemon --exec tsx index.ts --ignore output/*
+```
+
+### Access the Application
+
+Open your browser and go to:
+
+```
+http://localhost:3000
+```
+
+You will see the LlamaIndexServer UI, where you can interact with the HITL agent. Try "List all files in the current directory" and see how the agent pauses and waits for a human response before executing the command.
+
+## How does HITL work?
+
+### Events
+
+The human-in-the-loop approach used here is based on a simple idea: the workflow pauses and waits for a human response before proceeding to the next step.
+
+To do this, you will need to implement two custom events:
+
+- [HumanInputEvent](https://github.com/run-llama/create-llama/blob/main/packages/server/src/utils/hitl/events.ts): This event is used to request input from the user.
+- [HumanResponseEvent](https://github.com/run-llama/create-llama/blob/main/packages/server/src/utils/hitl/events.ts): This event is sent to the workflow to resume execution with input from the user.
+
+In this example, we have implemented these two custom events in [`events.ts`](src/app/events.ts):
+
+- `cliHumanInputEvent` – to request input from the user for CLI command execution.
+- `cliHumanResponseEvent` – to resume the workflow with the response from the user.
+
+```typescript
+export const cliHumanInputEvent = humanInputEvent<{
+  type: "cli_human_input";
+  data: { command: string };
+  response: typeof cliHumanResponseEvent;
+}>();
+
+export const cliHumanResponseEvent = humanResponseEvent<{
+  type: "human_response";
+  data: { execute: boolean; command: string };
+}>();
+```
+
+### UI Component
+
+HITL also needs a custom UI component, that is shown when the LlamaIndexServer receives the `cliHumanInputEvent`. The name of the component is defined in the `type` field of the `cliHumanInputEvent` - in our case, it is `cli_human_input`, which corresponds to the [cli_human_input.tsx](./components/cli_human_input.tsx) component.
+
+The custom component must use `append` to send a message with a `human_response` annotation. The data of the annotation must be in the format of the response event `cliHumanResponseEvent`, in our case, for sending to execute the command `ls -l`, we would send:
+
+```tsx
+append({
+  content: "Yes",
+  role: "user",
+  annotations: [
+    {
+      type: "human_response",
+      data: {
+        execute: true,
+        command: "ls -l", // The command to execute
+      },
+    },
+  ],
+});
+```
+
+This component displays the command to execute and the user can choose to execute or cancel the command execution.
+
+### Workflow Implementation
+
+The workflow is implemented in [`workflow.ts`](src/app/workflow.ts) using LlamaIndex workflows. The workflow handles three main steps:
+
+1. **Initial Request Handling**: When a user input is received, the workflow uses `chatWithTools` to determine if a CLI command should be executed. If so, it emits a `cliHumanInputEvent` to request user permission.
+
+```typescript
+workflow.handle([startAgentEvent], async ({ data }) => {
+  const { userInput, chatHistory = [] } = data;
+
+  const toolCallResponse = await chatWithTools(
+    llm,
+    [cliExecutor],
+    chatHistory.concat({ role: "user", content: userInput }),
+  );
+
+  const cliExecutorToolCall = toolCallResponse.toolCalls.find(
+    (toolCall) => toolCall.name === cliExecutor.metadata.name,
+  );
+
+  const command = cliExecutorToolCall?.input?.command as string;
+  if (command) {
+    return cliHumanInputEvent.with({
+      type: "cli_human_input",
+      data: { command },
+      response: cliHumanResponseEvent,
+    });
+  }
+
+  return summaryEvent.with("");
+});
+```
+
+2. **Human Response Handling**: After receiving human input, the workflow either executes the command or cancels based on the user's choice.
+
+```typescript
+workflow.handle([cliHumanResponseEvent], async ({ data }) => {
+  const { command, execute } = data.data;
+
+  if (!execute) {
+    return summaryEvent.with(`User reject to execute the command ${command}`);
+  }
+
+  const result = (await cliExecutor.call({ command })) as string;
+
+  return summaryEvent.with(
+    `Executed the command ${command} and got the result: ${result}`,
+  );
+});
+```
+
+3. **Final Response**: The workflow generates a final response based on the execution result and streams it back to the user.
+
+### Tools
+
+The CLI executor tool is defined in [`tools.ts`](src/app/tools.ts):
+
+```typescript
+export const cliExecutor = tool({
+  name: "cli_executor",
+  description: "This tool executes a command and returns the output.",
+  parameters: z.object({ command: z.string() }),
+  execute: async ({ command }) => {
+    try {
+      const output = execSync(command, {
+        encoding: "utf-8",
+      });
+      return output;
+    } catch (error) {
+      console.error(error);
+      return "Command failed";
+    }
+  },
+});
+```
+
+## Architecture
+
+The HITL implementation consists of:
+
+1. **Workflow Factory** (`workflow.ts`): Creates and configures the workflow with event handlers
+2. **Events** (`events.ts`): Defines typed events for human input and response
+3. **Tools** (`tools.ts`): Implements the CLI executor tool
+4. **UI Component** (`components/cli_human_input.tsx`): Provides the user interface for human approval
+5. **Server Entry** (`index.ts`): Configures and starts the LlamaIndexServer
+
+This architecture ensures that dangerous operations like CLI command execution require explicit human approval before proceeding.

packages/server/examples/hitl/components/cli_human_input.tsx

@@ -0,0 +1,95 @@
+import { Button } from "@/components/ui/button";
+import { Card, CardContent, CardFooter } from "@/components/ui/card";
+import { JSONValue, useChatUI } from "@llamaindex/chat-ui";
+import React, { FC, useState } from "react";
+import { z } from "zod";
+
+// This schema is equivalent to the CLICommand model defined in events.py
+const CLIInputEventSchema = z.object({
+  command: z.string(),
+});
+type CLIInputEvent = z.infer<typeof CLIInputEventSchema>;
+
+const CLIHumanInput: FC<{
+  events: JSONValue[];
+}> = ({ events }) => {
+  const inputEvent = (events || [])
+    .map((ev) => {
+      const parseResult = CLIInputEventSchema.safeParse(ev);
+      return parseResult.success ? parseResult.data : null;
+    })
+    .filter((ev): ev is CLIInputEvent => ev !== null)
+    .at(-1);
+
+  const { append } = useChatUI();
+  const [confirmedValue, setConfirmedValue] = useState<boolean | null>(null);
+  const [editableCommand, setEditableCommand] = useState<string | undefined>(
+    inputEvent?.command,
+  );
+
+  // Update editableCommand if inputEvent changes (e.g. new event comes in)
+  React.useEffect(() => {
+    setEditableCommand(inputEvent?.command);
+  }, [inputEvent?.command]);
+
+  const handleConfirm = () => {
+    append({
+      content: "Yes",
+      role: "user",
+      annotations: [
+        {
+          type: "human_response",
+          data: {
+            execute: true,
+            command: editableCommand, // Use editable command
+          },
+        },
+      ],
+    });
+    setConfirmedValue(true);
+  };
+
+  const handleCancel = () => {
+    append({
+      content: "No",
+      role: "user",
+      annotations: [
+        {
+          type: "human_response",
+          data: {
+            execute: false,
+            command: inputEvent?.command,
+          },
+        },
+      ],
+    });
+    setConfirmedValue(false);
+  };
+
+  return (
+    <Card className="my-4">
+      <CardContent className="pt-6">
+        <p className="text-sm text-gray-700">
+          Do you want to execute the following command?
+        </p>
+        <input
+          disabled
+          type="text"
+          value={editableCommand || ""}
+          onChange={(e) => setEditableCommand(e.target.value)}
+          className="my-2 w-full overflow-x-auto rounded border border-gray-300 bg-gray-100 p-3 font-mono text-xs text-gray-800"
+        />
+      </CardContent>
+      {confirmedValue === null ? (
+        <CardFooter className="flex justify-end gap-2">
+          <>
+            <Button onClick={handleConfirm}>Yes</Button>
+            <Button onClick={handleCancel}>No</Button>
+          </>
+        </CardFooter>
+      ) : null}
+    </Card>
+  );
+};
+
+export default CLIHumanInput;

packages/server/examples/hitl/index.ts

@@ -0,0 +1,20 @@
+import { OpenAI } from "@llamaindex/openai";
+import { LlamaIndexServer } from "@llamaindex/server";
+import { Settings } from "llamaindex";
+import { workflowFactory } from "./src/app/workflow";
+
+Settings.llm = new OpenAI({
+  model: "gpt-4o-mini",
+});
+
+new LlamaIndexServer({
+  workflow: workflowFactory,
+  uiConfig: {
+    starterQuestions: [
+      "Check status of git in the current directory",
+      "List all files in the current directory",
+    ],
+    componentsDir: "components",
+  },
+  port: 3000,
+}).start();

packages/server/examples/hitl/src/app/events.ts

@@ -0,0 +1,12 @@
+import { humanInputEvent, humanResponseEvent } from "@llamaindex/server";
+
+export const cliHumanInputEvent = humanInputEvent<{
+  type: "cli_human_input";
+  data: { command: string };
+  response: typeof cliHumanResponseEvent;
+}>();
+
+export const cliHumanResponseEvent = humanResponseEvent<{
+  type: "human_response";
+  data: { execute: boolean; command: string };
+}>();

packages/server/examples/hitl/src/app/tools.ts

@@ -0,0 +1,20 @@
+import { execSync } from "child_process";
+import { tool } from "llamaindex";
+import { z } from "zod";
+
+export const cliExecutor = tool({
+  name: "cli_executor",
+  description: "This tool executes a command and returns the output.",
+  parameters: z.object({ command: z.string() }),
+  execute: async ({ command }) => {
+    try {
+      const output = execSync(command, {
+        encoding: "utf-8",
+      });
+      return output;
+    } catch (error) {
+      console.error(error);
+      return "Command failed";
+    }
+  },
+});