[Collaboration]: Support for AI-driven workflows in Kedro

[ElenaKhaustova]

Description

We are collaborating on a real project that currently uses pythonic workflows without branching, including some GenAI functionality. Our goal is to build a POC within the next 2 weeks that demonstrates how Kedro can support these workflows.

The partner team will provide:

• Access to the data
• Guidance on how to run their current workflows

We will provide:

• A Kedro-based implementation of their workflows
• Best practices on structuring pipelines and organizing code so that they can later take over with the same approach

Functionality needed to be released (blocking for POC):

• LangChainPromptDataset for handling LangChain prompt templates directly: Create LangChainPromptDataset #5089
• Preview Text/Markdown Datasets in Kedro Viz: Preview Text/Markdown Datasets kedro-viz#2343
• Custom colours for nodes/datasets in Kedro Viz: Custom colours for nodes and datasets in Kedro Viz kedro-viz#2480

Possibly depending on their data and preference:

• PDFDataset to manage unstructured PDF data: kedro-datasets: Support PDF dataset for unstructured data kedro-plugins#1180
• LangfusePromptDataset and LangfuseTraceDataset for prompt management + tracing: Integrate with Langfuse for prompt management, tracing and evaluation #5088

Additional context:

During our call, the team mentioned they would like to have Kedro nodes that can define an LLM, prompts, and tools. This aligns with ongoing work around agentic abstractions (see example).

Acceptance criteria:

• POC implemented in Kedro using provided workflows and data
• Demonstration of how Kedro can be extended with nodes for LLMs/prompts/tools setup
• Documentation or examples showing how the team can replicate this approach for future projects