Roadmap: Interactive Pipeline Builder UI

[AliceCima10]

Overview

We want to explore whether building a pipeline builder directly in the UI would help users. The goal is to make pipeline creation easier, faster, and more intuitive—removing the need to start from code. This should lower the initial barrier, improve usability, and shorten the learning curve.

Problem Statement

• New users face a steep learning curve when creating pipelines in Kedro (Flatten the learning curve of Kedro #4892)
• The current process requires starting from code, which may slow adoption and experimentation
• We want to validate whether an interactive, UI-based builder could meaningfully improve this experience

Proposed Approach

We will validate the idea through a limited, time-boxed process:

1. Community Feedback (Phase 0)

• Share the idea with the community (GitHub discussion, Slack)
• Gather feedback on whether an interactive pipeline builder prototype seems valuable
• Use this input to refine the scope and design assumptions

2. Design (Phase 1)

• Define user requirements
• Define acceptance criteria
• Keep scope deliberately small

3. Prototype (Phase 2)

• Build a simple interactive tool (outside Kedro-Viz for now)
• Focus on a single-use, “one-shot” pipeline creation

4. User Testing (Phase 3)

• Share prototype with users
• Collect structured feedback on usability and value

5. Decision (Phase 4)

• Assess findings
• Decide go/no-go on further investment or Kedro-Viz integration

Prototype Context

• Builds on prior work explaining how nodes, config, and pipelines fit together; an interactive way of exploring that concept may help new technical users get up to speed with Kedro faster (Create a high-level 3 minute explanation of how Kedro works #5060)
• Could accelerate onboarding of new technical users
• Modern prototyping tools make building this fast and low cost
• Allow exporting a “starter pipeline template” from the tool

Out of Scope for Prototype

• Direct integration with Kedro-Viz (may be considered later)
• Editing/reloading pipelines
• Writing/editing node code from the UI

Open Questions

• Value: Would an interactive UI for building pipelines improve the learning curve for new users? Would it make faster to build pipelines?
• Scope: What’s the minimum feature set the prototype should include to be useful for testing?
• Adoption: Would the ability to export to a starter template meaningfully increase adoption?

What key metric is this targeting?

• User adoption
  • This will open the user base to a wider number of less-technical users. It will also help technical users learn Kedro faster, and therefore be more likely to adopt

SPIP Questions

Q1. What are you trying to do? Articulate your objectives using absolutely no jargon.

• Create a low-code UI to allow users to create Kedro pipelines more quickly

Q2. What problem is this proposal NOT designed to solve?

• Not experiment tracking or live ops

Q3. How is it done today, and what are the limits of current practice?

• Currently there is no solution for this. Kedro-Viz is read only. Other similar tools have some similar interfaces

Q4. What is new in your approach and why do you think it will be successful?

• The approach uses newly available components such as ReactFlow, and is likely to be successful because other similar tools have taken a similar approach. (Though this does not guarantee success, so progress should be incremental and proven at each stage)

Q5. Who cares? If you are successful, what difference will it make?

• Some existing users may switch to the UI version. New users may adopt Kedro via the new interface. This will increase the number of users of Kedro

Q6. What are the risks?

• This is potentially a large piece of work, which will not be fully testable until a large amount of work is completed. It is not fully clear whether less-technical people will indeed adopt this. The similar interfaces built by similar products may ultimately prove to be a flop. To avoid risks this should be done incrementally, with rapid iterations and testing at each stage

Q7. How long will it take?

• Months/Years

Q8. What are the mid-term and final “exams” to check for success?

• Build a series of incrementally more complex MVPs and test at each stage before proceeding

[datajoely]

Every single big player eventually builds this Palantir and dbt are probably the best examples.

My view is that adoption is dismal and this is usually driven by buyers being different from the users. It placates a part of the sale cycle without actually achieving anything.

The headless version of this is kind of interesting, if we were to expose a whole host of commands like kedro pipeline create for all manner of other components and expose via MCP suddenly you've got a composable system. That's more interesting to me, spending a year building a UI driven builder that looks cool in demos but ultimately won't be adopted is a big risk. Doing something like that feels like experiment tracking all over again.

[datajoely]

https://docs.getdbt.com/docs/cloud/use-canvas

https://www.palantir.com/docs/foundry/pipeline-builder/overview

https://www.databricks.com/blog/announcing-lakeflow-designer-no-code-etl

[deepyaman]

Out of Scope for Prototype

• Writing/editing node code from the UI

I'll write up more thoughts on this later, but—what exactly are users doing with the pipeline builder then? Creating the structure of a pipeline?

I actually can imagine a world where people want to build pipelines from a notebook/pipeline environment. I just don't understand how this is useful without it also being your code editor.

[AliceCima10]

@deepyaman The idea is to build an initial prototype to test whether building pipelines directly from the UI could add value for users. At this stage, we do not want to create a fully functional prototype that allows end-to-end coding of pipelines from the UI, as that would require a significant investment of time and effort before we know if the feature will be adopted.

However, I believe even this first, simplified version of the prototype could provide value to users (though I may be wrong—which is exactly what I want to test). From many conversations with users, and based on past experience (which I think @astrojuanlu can also confirm), one of the biggest challenges in learning Kedro is simply getting started. If users could build a first version of a pipeline through the UI, with the project structure already generated for them, it could significantly lower the learning curve. In addition, several users have mentioned that they begin designing pipelines with tools like Miro or other UI-based platforms, and have expressed a desire to convert those into Kedro projects.

[astrojuanlu]

To note, @jitu5 , @ravi-kumar-pilla and @iamelijahko made a PoC in 2 days https://github.com/jitu5/kedro-builder and it was useful enough for an internal team in its current state.

I do think there's an audience for this, but regardless, maybe the key is to find how to deliver something of value in 1 month instead of 1 year.

[deepyaman]

@AliceCima10 @astrojuanlu What is the nature of this "less technical" user persona that designs pipelines in Miro or some other GUI tool and can't write code for the pipeline structure, but they can write code for the pipeline logic? Are they data scientists?

If they can write code, why is the jump to building pipeline structure from a UI? Why not make it simple to define a pipeline in a single file, where they're authoring the code? It seems with the proposed solution, you're targeting users who can't define the (complex) project and pipeline structure, yet they can understand and maintain it once its been created? Am I missing something?

If this were part of Kedro-Viz and/or Kedro-VSCode, you click on a node to jump to the code and edit it, or you edit code and it's reflected in the builder--may or may not provide sufficient value, but at least I can see the idea. However, these things seem to be explicitly out of scope.

As a side note, there is (or at least was) some potential value in letting less technical users assemble pipelines from components (parametrizable modular pipelines, sometimes available as Brix posts). But, if I understand, this is not overlapping with the above proposal.

[astrojuanlu]

@deepyaman I don't think @AliceCima10 's writeup focuses solely on non-technical users? Copying from the first comment, emphasis mine:

Builds on prior work explaining how nodes, config, and pipelines fit together; an interactive way of exploring that concept may help new technical users get up to speed with Kedro faster

Could accelerate onboarding of new technical users

This will open the user base to a wider number of less-technical users. It will also help technical users learn Kedro faster, and therefore be more likely to adopt

---

Speaking only about myself, I can write code but I often draw the logic before I start coding anyway. I know others do as well.

In any case, the concerns about the value of the feature are valid. This is what should be proven first, right? No amount of speculation on our side is a replacement for that.

[AliceCima10]

@deepyaman Yes, the target is still technical users, with the goal of helping them get started more quickly and easily