📚 Documentation: Request for RAG based use case documentation

[SadmanYasar]

💭 Description

Currently in the process of whether or not to choose Appwrite for a project using Refine.dev and Nextjs. It will have a chatbot with RBAC so the response contents should be restricted based on the roles. I looked through the docs but only found basic implementation with OpenAI, Huggingface etc but no docs regarding saving embeddings to the database then converting user query to embedding and finding the nearest match etc. I would like to know if the use case I mentioned is possible using Appwrite. TIA.

👀 Have you spent some time to check if this issue has been raised before?

• I checked and didn't find similar issue

🏢 Have you read the Code of Conduct?

• I have read the Code of Conduct

[ChiragAgg5k]

I think we can implement something that you are looking for like this:

1. generateEmbedding

async function generateEmbedding(text: string): Promise<number[]> {
    const response = await openai.createEmbedding({
        model: 'text-embedding-ada-002',
        input: text,
    });
    return response.data.data[0].embedding;
}

2. storeEmbedding

async function storeEmbedding(text: string, embedding: number[], metadata: any) {
    await databases.createDocument(
        '[DATABASE_ID]',
        '[COLLECTION_ID]',
        'unique()',
        {
            text,
            embedding,
            metadata,
        }
    );
}

3. searchEmbedding

async function searchSimilarEmbeddings(queryEmbedding: number[], limit: number = 5) {
    const response = await databases.listDocuments(
        '[DATABASE_ID]',
        '[COLLECTION_ID]',
        [
            Query.orderBy('embedding', 'ASC'),
            Query.limit(limit),
        ]
    );

    return response.documents.map(doc => ({
        text: doc.text,
        similarity: cosineSimilarity(queryEmbedding, doc.embedding),
        metadata: doc.metadata,
    })).sort((a, b) => b.similarity - a.similarity);
}

function cosineSimilarity(a: number[], b: number[]): number {
    const dotProduct = a.reduce((sum, _, i) => sum + a[i] * b[i], 0);
    const magnitudeA = Math.sqrt(a.reduce((sum, val) => sum + val * val, 0));
    const magnitudeB = Math.sqrt(b.reduce((sum, val) => sum + val * val, 0));
    return dotProduct / (magnitudeA * magnitudeB);
}

The similarity function can be adjusted as per your need.

As for RBAC, appwrite uses concept of Permissions that you can use.