Benchmarks?

[nigelp]

Hi, has anyone done any actual benchmarks on the value of using these agents in this way, as opposed to just using the standardized AI tools coming out of the major frontier labs at the moment? Benchmarks on cost, efficiency, effectiveness, speed, accuracy, etcetera.

[shesl-meow]

Same question.

And I am curious about: is there any common method to evaluate these sub-agent skill? or we must focus on spec area.

[msitarzewski]

Great question, @nigelp. Honest answer: no formal benchmarks exist, and they'd be tricky to design well.

These agents are specialized system prompts — they shape the AI's persona, workflow, and output format rather than changing the underlying model's capabilities. The value is more about consistency and structure (getting the same thorough security review every time, not forgetting to check accessibility, following a repeatable audit template) than raw performance gains you could capture in a benchmark.

That said, if someone wanted to attempt it, the most meaningful comparison would probably be: "give the same task to a base model vs. the same model with an agent prompt" and evaluate output quality, completeness, and adherence to best practices. More qualitative than quantitative.

If anyone in the community has done informal comparisons, we'd love to hear about them in Discussions.

[antongulin]

@shesl-meow @nigelp @msitarzewski
You can try to use this tool or integrate before adding any skills https://github.com/olgasafonova/SkillCheck-Free
At least it's a check the skill structure to follow standards.