Using parallel_chat_structured with custom_solver

[Seviks]

We use parallel_chat_structured to enable the LLM to code certain text data fields. Ideally, we could evaluate the codings using the exact same configuration.

However, when attempting to integrate parallel_chat_structured into a custom solver, we encounter an error. Is there an alternative approach to achieve this, or is this functionality not currently supported?

I am currently using the latest development version (0.1.0.9000).

Error

✖ Solving [10.5s]                                                  
Error in `purrr::map_chr()`:
ℹ In index: 1.
ℹ With name: code.
Caused by error in `c$last_turn`:
! $ operator is invalid for atomic vectors
Hide Traceback
     ▆
  1. ├─are_claude$eval(...)
  2. │ └─self$solve(..., epochs = epochs)
  3. │   └─private$solver(self$get_samples()$input, ...)
  4. │     └─global fn(...)
  5. │       └─purrr::map_chr(res, function(c) c$last_turn()@text)
  6. │         └─purrr:::map_("character", .x, .f, ..., .progress = .progress)
  7. │           ├─purrr:::with_indexed_errors(...)
  8. │           │ └─base::withCallingHandlers(...)
  9. │           ├─purrr:::call_with_cleanup(...)
 10. │           └─.f(.x[[i]], ...)
 11. └─base::.handleSimpleError(...)
 12.   └─purrr (local) h(simpleError(msg, call))
 13.     └─cli::cli_abort(...)
 14.       └─rlang::abort(...)

Reprex (gives error above, works when replacing parallel_chat_structured with parallel_chat)

library(vitals)
library(ellmer)
library(tidyverse)

openai_api_key <- Sys.getenv("key_openai")
anthropic_api_key <- Sys.getenv("key_anthropic")

type_codes <- type_object(
  code = type_string(required = FALSE)
)

# Not working with structured chat for some reason
custom_solver <- function(inputs, ..., solver_chat) {
  ch <- solver_chat$clone()
  res <- ellmer::parallel_chat_structured(ch, as.list(inputs), type_codes)
  list(
    result = purrr::map_chr(res, function(c) c$last_turn()@text), 
    solver_chat = res
  )
}

are_task <-
  Task$new(
  dataset = are[1:2,],
  solver = custom_solver,
  scorer = model_graded_qa(
    scorer_chat = chat_anthropic(api_key=anthropic_api_key, model = "claude-3-7-sonnet-latest"), 
    partial_credit = TRUE
  ),
  name = "An R Eval",
  epochs = 1
)

are_claude <- are_task$clone()
are_claude$eval(solver_chat = chat_anthropic(api_key=anthropic_api_key, model = "claude-sonnet-4-20250514"))

[gary-mu]

I am also encountering the same issue which I filed under tidyverse/ellmer#753

TLDR: I'd like to use parallel_chat_structured to get structured output, but Vitals require both the result & chat object.
My understanding is parallel_chat_structured removes the chat object whereas parallel_chat keeps it.

The key is using parallel_chat_structured but somehow find a way to retrieve the chat object, which I don't know how to figure this out yet.

I am trying an alternative from Chatalot (https://github.com/dylanpieper/chatalot) which is an extension of Ellmer that also provides parallel LLM calling but keeps all chat history, but that does not play nicely with Vitals either.
Because Vitals tries to get chat text from the Chat object via text_contents@text but the chat object from Chatalot with structured output, there is no data in @text object :(

Would be best if Ellmer can support structured output and chat object simultaneoulsy.

[simonpcouch]

Thanks for the issue!

The approach you want here is to return res as the result, and repeats of ch for solver_chat. There's some extra metadata that won't be logged in the eval, but this is just the consequence of parallel_chat_structured() returning only the results rather than Chat objects.

Haven't actually tried to run this, but:

custom_solver <- function(inputs, ..., solver_chat) {
  ch <- solver_chat$clone()
  res <- ellmer::parallel_chat_structured(ch, as.list(inputs), type_codes)
  list(
    result = res, 
    solver_chat = purrr::map(seq_len(inputs), function(i) ch)
  )
}

[gary-mu]

@simonpcouch thank you for the reply.
purrr::map(seq_len(inputs), function(i) ch) Does this mean you are calling the LLM again on the inputs?
If so, wouldn't the output deviate away from the result since they are different LLM calls?

[simonpcouch]

Note that all is passed to that anonymous function is an index, and that the function doesn't actually make use of the index i. This is just a way of making a list of length length(inputs) where each element is the original chat object. :)

[gary-mu]

@simonpcouch I see. So it's just copying the cloned Chat object as many times as the input length that does not have the turns? (Also should the seq_len(inputs) be seq_along(inputs) or seq_len(length(inputs)) ?)

I tried and was seeing the result & solver_chat object with 3 copies of the cloned object in the task object; however the eval still failed with this info:

Error in `` `*tmp*`[[j]] ``:
! subscript out of bounds
    ▆
 1. └─vocab_task$eval(solver_chat = model$clone())
 2.   └─self$log(self$dir)
 3.     └─private$add_working_starts(samples)
 4.       └─vitals:::add_working_start_to_turns(...)

[gary-mu]

Oh and also in vitals, it requires to have text in the last turn (

vitals/R/translate.R

Line 215 in 819e30c

explanation <- scorer_chat$last_turn()@text

), so even if the empty Chat object is returned, it will not have text info and might error out with
! no applicable method for @ applied to an object of class "NULL"?

This is exactly what I encountered when using Chatalot that gives structured output as well as the chat object with turns, but the last turn does not have @text data due to the structured output.

[gary-mu]

I think I have found a very hacky workaround, not sure how brittle it is given the nature of LLM's variability:

Steps:

1. Create a custom solver and call parallel_chat function on your inputs. This should give you a list of Chat object with the same length as the input
2. Create a deep copy of the Chat object from 1 above (the output of 1 is a list of Chat objects which are R6 objects). Use this to assign to the solver_chat variable at the end
3. With the original Chat object, make another call of chat_structured and pass in the the structured types variable. Use this as your final output

My example:

structured_output_solver <- function(inputs, ..., solver_chat) {
  ch <- solver_chat 
  res <- ellmer::parallel_chat(ch, as.list(inputs))
  #This is a deep copy of above which will not add additional turns when calling chat_structured below
  res_deep_clone <- lapply(res, function(x) x$clone(deep = TRUE))
  #Calling chat_structured using the original list of Chat object to get the output in the structure you want
  res_structured <- purrr::map(res, function(x){x$chat_structured(type = type_vocab)})
  #Use your purrr function of choice to get the final result you want
  final_result <- purrr::map_chr(res_structured, 'complexity_score')

  #return the final result and the cloned object to solver_chat
  list(
    result = final_result,
    solver_chat = res_deep_clone
  )
}

Conceptually, this should be fine because:

1. We keep the original chat object to pass to solver_chat. This should already have a lot of information from LLM based on your prompt. Using a deep clone to "freeze" it, so the object don't add additional turns anymore later on when you call LLM again.
2. The chat_structured method calls LLM again essentially just to organize the last conversation (the LLM output) into the structured output you define. This will be added to the original Chat object as another turn

The only downside is we are essentially adding another turn, so LLM might hallucinate and produce the structured result that differs from the previous output.

[Seviks]

Thanks for the issue!

The approach you want here is to return res as the result, and repeats of ch for solver_chat. There's some extra metadata that won't be logged in the eval, but this is just the consequence of parallel_chat_structured() returning only the results rather than Chat objects.

Haven't actually tried to run this, but:

custom_solver <- function(inputs, ..., solver_chat) {
ch <- solver_chat$clone()
res <- ellmer::parallel_chat_structured(ch, as.list(inputs), type_codes)
list(
result = res,
solver_chat = purrr::map(seq_len(inputs), function(i) ch)
)
}

Thank you for your responses. I have tried both suggestions. While I do not fully understand the outcome, I provide my logs for reference (using the example above and replacing custom_solver).

Suggestion from @simonpcouch failed with this info:

Error in `seq_len()`:
! argument must be coercible to non-negative integer
 1. └─are_claude$eval(...)
 2.   └─self$solve(..., epochs = epochs)
 3.     └─private$solver(self$get_samples()$input, ...)
 4.       └─global fn(...)
 5.         └─purrr::map(seq_len(inputs), function(i) ch)
 6.           └─purrr:::map_("list", .x, .f, ..., .progress = .progress)
 7.             └─purrr:::vctrs_vec_compat(.x, .purrr_user_env)

The code from @gary-mu yields this for me:

Error in `purrr::map()`:
ℹ In index: 1.
Caused by error in `req_perform()`:
! HTTP 400 Bad Request.
ℹ messages.2: all messages must have non-empty content except for the optional
  final assistant message [invalid_request_error]

[gary-mu]

@Seviks in @simonpcouch's example, I think seq_len should be replaced with seq_along, see my comment above: #153 (comment)
It will resolve the error you see, but you will see another error as I don't think the solution is right.

The error you got from my code seems to be from this line
res_structured <- purrr::map(res, function(x){x$chat_structured(type = type_vocab)})
where your chat object res (from ellmer::parallel_chat(ch, as.list(inputs)) may be empty
You might want to just try out these two lines independently in your IDE to test out how they look like.

[simonpcouch]

Okay, just took a moment to revisit this. The goal is to assemble a set of solver_chats that can "pass" as having actually been the result of chat() or chat_structured() so that we can take advantage of both structured output and parallelism. Here's a variant of generate() that can handle structured output:

generate_structured <- function(solver_chat = NULL, type = NULL) {
  chat <- solver_chat
  type_spec <- type

  function(inputs, ..., solver_chat = chat, type = type_spec) {
    ch <- solver_chat$clone()
    res <- ellmer::parallel_chat_structured(
      ch,
      as.list(inputs),
      type = type,
      ...
    )

    solver_chats <- purrr::map2(
      inputs,
      res,
      mock_chat,
      solver_chat = ch
    )

    list(
      result = res,
      solver_chat = solver_chats
    )
  }
}

mock_chat <- function(input, result, solver_chat) {
  chat <- solver_chat$clone()

  user_turn <- ellmer::Turn(
    role = "user",
    contents = list(ellmer::ContentText(as.character(input)))
  )

  result_json <- jsonlite::toJSON(result, auto_unbox = TRUE)
  assistant_turn <- ellmer::Turn(
    role = "assistant",
    contents = list(ellmer::ContentText(as.character(result_json)))
  )

  chat$set_turns(list(user_turn, assistant_turn))

  chat
}

Here's an example usage:

library(ellmer)

type_codes <- type_object(
  code = type_string(required = FALSE),
  explanation = type_string(required = FALSE)
)

solver_structured <- generate_structured(
  solver_chat = chat_anthropic(model = "claude-sonnet-4-20250514"),
  type = type_codes
)

task <- Task$new(
  dataset = are[1:2, ],
  solver = solver_structured,
  scorer = model_graded_qa(),
  name = "An R Eval with Structured Output"
)

task$eval()

It's not pretty, but it produces valid logs. :)

I'm wondering if vitals itself should actually supply something like generate_structured(). From one perspective, it's not an easy programming task to get that set up right, and at the same time, we're "faking" the solver_chats and the result won't be nearly as complete as with a typical evaluation.