Four skill-doc issues surfaced by eval-driven testing

[ttokttokttok]

Summary

While testing evals against agents loaded with these skills (including our xulux coding agent), we surfaced four concrete issues in the existing skill docs that caused agents to trip, error, or get vague guidance. Each fix below is a small, surgical edit (max one paragraph). We're also proposing an evals/ folder at the repo root so future PRs can prove a behavior gap with a failing eval before changing a skill file.

Net diff: 7 insertions, 1 deletion across the three SKILL.md files (six lines added; one line modified — Step 2's schema-fetch imperative in apify-ultimate-scraper), plus a new evals/ directory (5 files, 115 entries).

---

Issues found and proposed fixes

1. apify-sdk-integration: agents fabricate Actor IDs

Eval prompt (apify-sdk-integration.json, type: "discovery"):

"I need to scrape Instagram profile data. Find me an Apify Actor for it."

What we observed. Agents responded with plausible-but-wrong Actor IDs from memory instead of running apify actors search first. The most illustrative case: ask the agent for a Google Maps scraper and it confidently produces apify/google-maps-scraper — which doesn't exist. The real top result is compass/crawler-google-places (different owner entirely, verified via live apify actors search).

Why it happens. The "Finding the Right Actor" section tells the reader to search/browse but never states what an Actor ID looks like. With no format guidance, the agent pattern-matches on training data and produces a guess that feels right ("anything in Apify Store is probably apify/...").

Fix. One-line note after the store-browse paragraph:

Actor IDs have the form username/actor-name (e.g. apify/instagram-scraper, compass/crawler-google-places) — take them from the search results, don't guess them from memory.

2. apify-ultimate-scraper: no fallback when dynamic search also fails

Eval prompt (apify-ultimate-scraper.json, type: "selection"):

"Scrape listings from a regional classifieds site that isn't a major platform."

What we observed. A fresh agent (subagent run, no eval context, no expected answer shown) said exactly: "I'd guess based on the WAF behavior described in gotchas." — it knew to pivot away from listed Actors but had no rule for which generic crawler to pick. Same response pattern in a separate self-run.

Why it happens. Step 1 covers "platform in the index" and "platform not in the index → search dynamically," but never covers "search also turned up nothing." The information needed to pick (cheerio vs playwright vs camoufox) lives in gotchas.md, but the Step 1 flow doesn't surface it as a decision rule.

Fix. One-paragraph fallback rule keyed on rendering profile:

If dynamic search also returns nothing suitable, fall back to a generic crawler picked by the target site's rendering: apify/cheerio-scraper for static HTML, apify/playwright-scraper for JS-rendered sites, apify/camoufox-scraper for anti-bot/WAF-protected sites (see references/gotchas.md).

3. apify-ultimate-scraper: Step 2 / Step 3 contradict for trivial lookups

Eval prompt (apify-ultimate-scraper.json, type: "run"):

"What's Nike's follower count on Instagram?"

What we observed. Two independent agent runs (one self-run, one fresh subagent) both said they'd "fetch the input schema first" before even attempting this trivial lookup — citing Step 2's blanket "Fetch the input schema dynamically:" instruction. The schema fetch is pure overhead on a query where the input shape (just a username) is obvious from the prompt.

Why it happens. Step 2 says imperatively "Fetch the input schema dynamically:" with no qualification. Step 3 says "skip user preferences for simple lookups, go straight to quick-answer mode." The two rules contradict for trivial lookups; the strict reading of Step 2 wins and forces a redundant call.

Fix. Five-word conditional appended to the Step 2 line:

-Fetch the input schema dynamically:
+Fetch the input schema dynamically, unless you already know the input fields:

Gives the agent a checkable criterion (do I already know this Actor's fields?) without an explicit "Exception:" callout.

4. apify-actor-development: apify create -t rejects manifest id, requires name

Surfaced via. Caught while validating template IDs for our build-path evals. We tried apify create -t ts-crawlee-cheerio (the manifest's id, the most-prominent field) before encoding it in an eval. It didn't work.

What we observed.

$ apify create test -t ts-crawlee-cheerio --skip-dependency-install --skip-git-init
Error: Could not find the selected template: ts-crawlee-cheerio in the list of templates.

$ apify create test -t project_cheerio_crawler_ts --skip-dependency-install --skip-git-init
Success: ✅ Actor 'test' created successfully!

Verified across multiple templates: ts-empty (id) errors; ts_empty (name) succeeds. js-empty (id) errors; project_empty (name) succeeds. Python templates happen to align (python-empty is both id and name), which is why this issue is invisible for Python-only users.

Why it happens. apify create --help points users at the manifest as the source of truth, and the manifest's most prominent field is id — but the CLI accepts only name. The three documented defaults in the current skill (project_empty, ts_empty, python-empty) are all correct name values, but the skill never tells the reader which field of the manifest to use when scaling beyond them. So the docs are technically correct but operationally misleading the moment a user needs a different template.

Fix. One-line clarification in the Template selection section:

For other templates (Crawlee + Cheerio, Playwright, Camoufox, etc.): the -t value is the manifest's name field, not the id field. Look up the name here: https://raw.githubusercontent.com/apify/actor-templates/master/templates/manifest.json

---

Companion: evals/ folder (new)

A small two-layer eval framework that surfaced the issues above. We're including it in the PR but it's logically independent — split into its own commit so it can be cleanly dropped or factored out if you'd rather not adopt evals in this repo.

evals/
├── README.md
├── routing.json                    cross-skill routing (10 evals)
├── apify-actor-development.json    behavioral, type=build|cli (8)
├── apify-sdk-integration.json      behavioral, type=discovery|integrate (9)
└── apify-ultimate-scraper.json     behavioral, type=selection|run|cost|
                                      schema-and-gotchas|workflow:<name> (88)

• Routing evals test which skill the agent picks when only description fields are visible. Failures point at the description to fix.
• Behavioral evals test what an agent does once a given skill is loaded. Each entry has {prompt, expected, type}; workflow evals use type: "workflow:<workflow-name>" mapping 1:1 to references/workflows/*.md so a stale Actor ID in any specific pipeline gets a targeted eval.

All prompts are written as single-shot — no prior-turn assumptions. Counts: 115 evals total.

---

What's deliberately NOT in this PR (transparency)

• A pagination guidance rewrite for apify-sdk-integration (drafted then reverted — the "250K items default" claim is technically wrong, but the impact is low since most datasets don't approach 250K and agent context blows at far smaller sizes regardless; the cleaner improvement we'd layered on was somewhat agent-centric for a doc that also serves human app developers).
• A polling-cadence rule for async runs (drafted then reverted — the specific numbers were invented rather than sourced).
• Filing the apify create -t ergonomic issue against apify-cli. We considered it, concluded the CLI works as designed and the gap is documentation-only, and chose this skill clarification instead.
• The actor-index.md "tier" terminology question (calling compass/..., clockworks/... etc. as apify-tier when by the file's own definition apify means "Apify-maintained"). Real inconsistency but touches every row of the index — out of scope for this PR; happy to discuss separately if you want.

Companion issue (separate, filed against apify-cli)

Unrelated to this PR, we also caught a /v2/store API paging bug while building the search evals: apify actors search --json --limit 1 returns count: 1 with an empty items[], and the deficit grows with --offset. Reproduced against api.apify.com directly (not a CLI bug). Filing separately at apify/apify-cli since there's no public API tracker.

How to review

# Just the skill edits
git diff main -- 'skills/**/SKILL.md'

# The full diff (skill edits + evals folder)
git diff main

[patrikbraborec]

Hi, thanks for the issue. I created fix for apify-actor-development skill: #52.