[RFC] 130 - Multi-Agent Orchestration (Group Chat) Implementation

[rivertwilight]

This RFC documents the implementation of multi-agent orchestration capabilities in LobeHub, enabling dynamic conversations between multiple AI agents coordinated by an intelligent supervisor system.

Motivation

Traditional AI chat interfaces are limited to single-agent interactions, which constrains the potential for complex problem-solving scenarios that could benefit from multiple specialized agents working collaboratively. This implementation addresses several key use cases:

• Expert collaboration: Different agents with specialized knowledge domains working together
• Perspective diversity: Multiple viewpoints on complex topics
• Dynamic problem-solving: Agents building upon each other's responses
• Workflow automation: Sequential task handoffs between specialized agents

Architecture Overview

• A supervisor will decide who will speak next, and how (DM or public message)
• New entry to create a group chat, consisting of user's existing assistants
• New database entity, lambda, service, model chatGroup for group
• New generateAIGroupChat action for group chatting
• Branching on some existing UI for group-specific logc, and some new group-only componenet

Data Model

/* eslint-disable sort-keys-fix/sort-keys-fix  */
import {
  boolean,
  jsonb,
  pgTable,
  primaryKey,
  text,
  uniqueIndex,
  varchar,
} from 'drizzle-orm/pg-core';
import { createInsertSchema } from 'drizzle-zod';

import { idGenerator, randomSlug } from '@/database/utils/idGenerator';

import { timestamps } from './_helpers';
import { agents } from './agent';
import { users } from './user';

/**
 * Chat groups table for multi-agent conversations
 * Allows multiple agents to participate in a single chat session
 */
export const chatGroups = pgTable(
  'chat_groups',
  {
    id: text('id')
      .primaryKey()
      .$defaultFn(() => idGenerator('chatGroups'))
      .notNull(),
    slug: varchar('slug', { length: 100 })
      .$defaultFn(() => randomSlug(4))
      .unique(),
    title: text('title'),
    description: text('description'),

    /**
     * Group configuration
     */
    config: jsonb('config').$type<{
      maxResponseInRow?: number;
      orchestratorModel?: string;
      orchestratorProvider?: string;
      responseOrder?: 'sequential' | 'natural';
      responseSpeed?: 'slow' | 'medium' | 'fast';
      revealDM?: boolean;
      systemPrompt?: string;
    }>(),

    clientId: text('client_id'),

    userId: text('user_id')
      .references(() => users.id, { onDelete: 'cascade' })
      .notNull(),

    pinned: boolean('pinned').default(false),

    ...timestamps,
  },
  (t) => [uniqueIndex('chat_groups_client_id_user_id_unique').on(t.clientId, t.userId)],
);

export const insertChatGroupSchema = createInsertSchema(chatGroups);

export type NewChatGroup = typeof chatGroups.$inferInsert;
export type ChatGroupItem = typeof chatGroups.$inferSelect;

/**
 * Junction table connecting chat groups with agents
 * Defines which agents participate in each group chat
 */
export const chatGroupsAgents = pgTable(
  'chat_groups_agents',
  {
    chatGroupId: text('chat_group_id')
      .references(() => chatGroups.id, { onDelete: 'cascade' })
      .notNull(),
    agentId: text('agent_id')
      .references(() => agents.id, { onDelete: 'cascade' })
      .notNull(),
    userId: text('user_id')
      .references(() => users.id, { onDelete: 'cascade' })
      .notNull(),

    /**
     * Whether this agent is active in the group
     */
    enabled: boolean('enabled').default(true),

    /**
     * Display or speaking order of the agent in the group
     */
    order: text('order').default('0'),

    /**
     * Role of the agent in the group (e.g., 'moderator', 'participant')
     */
    role: text('role').default('participant'),

    ...timestamps,
  },
  (t) => ({
    pk: primaryKey({ columns: [t.chatGroupId, t.agentId] }),
  }),
);

export type NewChatGroupAgent = typeof chatGroupsAgents.$inferInsert;
export type ChatGroupAgentItem = typeof agents.$inferInsert

Supervisor Prompt

Use XML-pesudo for group member and history info.

      const supervisorPrompt = `
      You are a conversation orchestrator for a group chat with multiple AI agents. Your role is to decide which agents should respond next based on the conversation context.

Rules:
- Return an array of objects where each object has an "id" field for the agent who should respond
- If a response should be a direct message (DM) to a specific member, include a "target" field with the target member ID or "user"
- If no "target" field is provided, the response will be a group message visible to everyone
- If the conversation seems complete, return empty array []
- Your goal is to make the conversation as natural as possible. For example, if user DM to an agent, the agent is likely to respond to the user privately too
- Return ONLY a JSON array of objects, nothing else

Examples: 
- Group responses: [{"id": "agt_01"}]
- DM responses: [{"id": "agt_01", "target": "agt_02"}, {"id": "agt_04", "target": "user"}]
- Mixed responses: [{"id": "agt_01"}, {"id": "agt_02", "target": "user"}]
- Stop conversation: []

<group_role>
${systemPrompt}
</group_role>

${memberDescriptions}

<conversation_history>
${conversationHistory}
</conversation_history>
`;

Agent Prompt

Use a system prompt + history + user prompt combo.
Each history message also includes a <author_name_do_not_include_in_your_response> tag for marking the message author.

  const prompt = `${baseSystemRole}
You are participating in a group chat in real world.

Guidelines:
- Stay in character as ${agentId} (${agentTitle})
- Be concise and natural, behave like a real person
- Engage naturally in the conversation flow
- Your message is automatically sent privately or publicly to the group, so you don't need to mention it, just respond as you would in a real conversation
- Be collaborative and build upon others' responses when appropriate
- Keep your responses concise and relevant to the ongoing discussion
- Each message should no more than 100 words

${membersTag}

${historyTag}
`;
      const userMessage: ChatMessage = {
        id: 'group-user',
        role: 'user',
        content:
          "Now it's your turn to respond. Please respond as this agent would, considering the full conversation history provided above. Directly return the message content, no other text. You do not need add author name or anything else.",
        createdAt: Date.now(),
        updatedAt: Date.now(),
        meta: {},
      };

[rivertwilight]

The test flight version is available here