Split by LLM

[rowanseymour]

Trying to put it all together and document it to help with adding support on the editor side...

@locals is now a thing in the context. For now the values of locals are limited to text values. The names are limited to snaked names (^[a-z_][a-z0-9_]{0,63}$). @locals.foo_bar_1 etc.

There is an action set_run_local which is a lot like set_run_result but writes a local variable and has three operations (set, increment and clear). increment tries to make it easier to build counters because it assumes zero for undefined values.

{
    "uuid": "ad154980-7bf7-4ab8-8728-545fd6378912",
    "type": "set_run_local",
    "local": "my_var1",
    "value": "sesame",
    "operation": "set"
}

{
    "uuid": "ad154980-7bf7-4ab8-8728-545fd6378912",
    "type": "set_run_local",
    "local": "counter",
    "value": "1",
    "operation": "increment"
}

These don't produce any events because there's nothing for the caller to do and we're trying to keep locals more lightweight than results.

There is a new call_llm action which is completely generic and just takes instructions and input. It saves its output to a local variable called @locals._llm_output. It produces either an llm_called event or an error event. In the latter case it sets @locals._llm_output to <ERROR>.

{
    "uuid": "ad154980-7bf7-4ab8-8728-545fd6378912",
    "type": "call_llm",
    "llm": {
        "uuid": "63998ee7-a7a5-4cc5-be67-c773e1b6b9b1",
        "name": "Claude"
    },
    "instructions": "Categorize the following text as positive or negative",
    "input": "@input.text",
    "output_local": "_llm_output"
}

Instead of creating HTTP logs like we've always done for classifiers, airtime etc this generates an llm_called event like below. If in future we decide we sometimes need HTTP logs for debugging it could be selective (see #1332).

{
    "type": "llm_called",
    "created_on": "2006-01-02T15:04:05Z",
    "llm": {
        "uuid": "14115c03-b4c5-49e2-b9ac-390c43e9d7ce",
        "name": "GPT-4"
    },
    "instructions": "Categorize the following text as Positive or Negative",
    "input": "Please stop messaging me",
    "output": "Positive",
    "tokens_used": 567,
    "elapsed_ms": 123
}

To avoid always storing raw LLM prompts in flow definitions, there's a new function called prompt to return pre-defined prompts that we can iterate on over time.

@(prompt("categorize", array("Positive", "Negative")))

Categorize the following text into one of the following categories and only return 
that category or <CANT> if you can't: Positive, Negative

If that function is used on a split node it can access the node's category names via new context value @node.categories. We've also added the slice(array, start [,end]) expression function to allow trimming off the Other and Failure categories:

@(prompt("categorize", slice(node.categories, 0, -2)))

These pieces can be combined into an "Split By LLM" node which splits on any value in the context (set as the input of the call_llm action):

{
    "uuid": "145eb3d3-b841-4e66-abac-297ae525c7ad",
    "actions": [
        {
            "type": "call_llm",
            "uuid": "3cd8f2db-8429-462e-ab93-8041dd23abf1",
            "llm": {
                "uuid": "1c06c884-39dd-4ce4-ad9f-9a01cbe6c000",
                "name": "Claude"
            },
            "instructions": "@(prompt(\"categorize\", slice(node.categories, 0, -2)))",
            "input": "@results.response_1",
            "output_local": "_llm_output"
        }
    ],
    "router": {
        "type": "switch",
        "categories": [
            {
                "uuid": "6103fa71-6ca9-4300-aec6-929f50fa1ae0",
                "name": "Flights",
                "exit_uuid": "33712037-9861-4d61-9dcb-60d7fffef96a"
            },
            {
                "uuid": "bcb18434-1932-4a38-a4cd-a2c4a70b8e9a",
                "name": "Hotels",
                "exit_uuid": "fdd988ba-34c1-45a8-8413-e89b0a36001e"
            },
            {
                "uuid": "a766ac1a-766f-4ce5-be4c-a24061bfdec0",
                "name": "Other",
                "exit_uuid": "bbcb0d26-a8c4-48f7-9d39-eeb202d09876"
            },
            {
                "uuid": "e86a60b9-6e8e-4150-9ab9-19e6eb7003d9",
                "name": "Failure",
                "exit_uuid": "959d6e4c-658a-49fc-a80d-5ed7df5af640"
            }
        ],
        "default_category_uuid": "a766ac1a-766f-4ce5-be4c-a24061bfdec0",
        "operand": "@locals._llm_output",
        "result_name": "Intent",
        "cases": [
            {
                "uuid": "d2f852ec-7b4e-457f-ae7f-f8b243c49ff5",
                "type": "has_only_text",
                "arguments": [
                    "Flights"
                ],
                "category_uuid": "6103fa71-6ca9-4300-aec6-929f50fa1ae0"
            },
            {
                "uuid": "692926ea-09d6-4942-bd38-d266ec8d3716",
                "type": "has_only_text",
                "arguments": [
                    "Hotels"
                ],
                "category_uuid": "bcb18434-1932-4a38-a4cd-a2c4a70b8e9a"
            },
            {
                "uuid": "73bfb9dc-9bbb-4928-92fd-d8957edf4a92",
                "type": "has_only_text",
                "arguments": [
                    "<ERROR>"
                ],
                "category_uuid": "e86a60b9-6e8e-4150-9ab9-19e6eb7003d9"
            }
        ]
    },
    "exits": [
        {
            "uuid": "33712037-9861-4d61-9dcb-60d7fffef96a",
            "destination_uuid": "8f4aba68-3250-43b6-8409-93ba44092962"
        },
        {
            "uuid": "fdd988ba-34c1-45a8-8413-e89b0a36001e",
            "destination_uuid": "ac3fcd8e-e7bb-4545-865d-39424a8f1d7b"
        },
        {
            "uuid": "bbcb0d26-a8c4-48f7-9d39-eeb202d09876",
            "destination_uuid": "72b177c0-f66f-48fe-bdb5-f5c4383502c0"
        },
        {
            "uuid": "959d6e4c-658a-49fc-a80d-5ed7df5af640"
        }
    ]
}

• The call_llm action could be an addable action in its own right and usable for generating responses if that's your cup of tea.
• I had originally envisaged a single Wait and then Split By LLM node but that was forgetting that waits always come after actions. Having the wait and the split be two different nodes is how classifiers work and so this should be a drop in replacement for the few users still hanging on to classifiers. It also lends itself to using the first wait to split on non-LLM logic and save LLM calls.
• The prompt function gets prompt text from the environment which in turn gets it from the engine which exposes it as an option - so we can have a master store of LLM prompts in mailroom.
• Flow spec version has been bumped to 14.0.0.

[codecov]

Codecov Report

Attention: Patch coverage is 69.56522% with 7 lines in your changes missing coverage. Please review.

Project coverage is 69.14%. Comparing base (6d5d0ef) to head (13ab221).

Files with missing lines	Patch %	Lines
flows/runs/run.go	28.57%	4 Missing and 1 partial ⚠️
flows/actions/set_run_local.go	75.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1319      +/-   ##
==========================================
+ Coverage   69.12%   69.14%   +0.02%     
==========================================
  Files         283      283              
  Lines       21533    21543      +10     
==========================================
+ Hits        14884    14896      +12     
+ Misses       6214     6211       -3     
- Partials      435      436       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[rowanseymour]

<CANT> would also go here into Other if the LLM can't do what you've asked

[rowanseymour]

I forgot that router rules have one operand... so need a single value that describes successful output as well as failure

[ericnewcomer]

Damn @rowanseymour this is really great. Nice work.