diff --git a/.agents/skills/graphistry-rest-api/SKILL.md b/.agents/skills/graphistry-rest-api/SKILL.md index 296ab96..5102d50 100644 --- a/.agents/skills/graphistry-rest-api/SKILL.md +++ b/.agents/skills/graphistry-rest-api/SKILL.md @@ -1,6 +1,13 @@ --- name: graphistry-rest-api -description: "Graphistry Hub REST API specialist for auth, upload lifecycle, URL controls, sessions, and sharing safety. Use for curl/requests endpoint guidance independent of SDK choice." +description: > + Graphistry Hub REST API: auth, upload, URL controls, sessions, and sharing. + Use when asked to "call the graphistry API with curl", "get a JWT token from graphistry", + "upload a graph via REST", "graph.html URL parameters", "graphistry session API", + or "share a graph link safely". Also triggers on "/api/v2/", "Bearer token", + "graphistry upload endpoint", or any direct HTTP endpoint question about Graphistry Hub. + Prefer this over pygraphistry when the user explicitly uses curl, requests, or raw HTTP + rather than the Python SDK. --- # Graphistry REST API diff --git a/.agents/skills/graphistry-rest-api/evals/evals.json b/.agents/skills/graphistry-rest-api/evals/evals.json new file mode 100644 index 0000000..3954799 --- /dev/null +++ b/.agents/skills/graphistry-rest-api/evals/evals.json @@ -0,0 +1,47 @@ +{ + "skill_name": "graphistry-rest-api", + "evals": [ + { + "id": 1, + "prompt": "How do I get a JWT token from Graphistry Hub using my username and password via the REST API?", + "expected_output": "Shows POST /api/v2/o/token/ (or equivalent auth endpoint) with username/password body, returns token to use as Bearer in subsequent requests", + "assertions": [ + {"text": "Shows POST to /api/v2/ auth endpoint", "type": "contains"}, + {"text": "Returns a JWT/Bearer token", "type": "contains"}, + {"text": "Uses env vars for username/password, not hardcoded literals", "type": "contains"} + ], + "files": [] + }, + { + "id": 2, + "prompt": "What's the REST API call to upload a graph (edges CSV) to Graphistry Hub and get back a shareable graph URL?", + "expected_output": "Shows POST to /api/v2/upload/edges/ or dataset endpoint with Bearer auth, explains how to get the graph URL from the response", + "assertions": [ + {"text": "Shows upload endpoint /api/v2/upload/ or equivalent", "type": "contains"}, + {"text": "Uses Bearer token auth header", "type": "contains"}, + {"text": "Explains how to construct or retrieve the shareable graph.html URL", "type": "contains"} + ], + "files": [] + }, + { + "id": 3, + "prompt": "I'm embedding a Graphistry graph in an iframe. What URL parameters can I use to disable the toolbar and set the initial zoom level?", + "expected_output": "Shows graph.html URL params: toolbar=false/0, zoom/play settings, references the URL options doc section", + "assertions": [ + {"text": "References graph.html URL parameters", "type": "contains"}, + {"text": "Shows toolbar or play URL param syntax", "type": "contains"} + ], + "files": [] + }, + { + "id": 4, + "prompt": "How do I use the PyGraphistry Python library to register with Graphistry and call .plot() on my DataFrame?", + "expected_output": "Does NOT stay in graphistry-rest-api; routes to pygraphistry-core since this is a Python SDK task, not a raw REST task", + "assertions": [ + {"text": "Routes to pygraphistry-core for Python SDK usage", "type": "contains"}, + {"text": "Does not answer with curl or raw /api/v2/ endpoint calls for the Python SDK question", "type": "negative"} + ], + "files": [] + } + ] +} diff --git a/.agents/skills/graphistry/SKILL.md b/.agents/skills/graphistry/SKILL.md index 5ec1f1a..b6283ab 100644 --- a/.agents/skills/graphistry/SKILL.md +++ b/.agents/skills/graphistry/SKILL.md @@ -1,6 +1,14 @@ --- name: graphistry -description: "Umbrella router for Graphistry workflows across SDK and API surfaces. Use to dispatch between Python SDK, REST API, and (future) JavaScript SDK workflows." +description: > + Umbrella router for Graphistry workflows across SDK and API surfaces. + Use when asked to "use graphistry", "visualize with graphistry", "graph this data in graphistry", + "graphistry plot", or any question mixing Python SDK and REST API concerns. Also triggers on + "graphistry SDK vs API", "graphistry authentication", "how do I share a graphistry graph", + "Graphistry Hub", or any ambiguous graphistry request before the interface is clear. + Routes to pygraphistry for Python SDK tasks, graphistry-rest-api for curl/REST tasks. + Proactively suggest when the user mentions graph visualization or network analysis and + has not yet chosen an interface. --- # Graphistry Router @@ -10,7 +18,7 @@ Use this skill as the shared entrypoint across Graphistry interfaces. ## Route By Interface - Python SDK tasks (`import graphistry`, DataFrame shaping, `.plot()`, `.gfql()` including Cypher/Let/DAG, PyGraphistry notebooks): use `pygraphistry`. - REST API tasks (`curl`, `/api/v2/...`, JWT/Bearer auth, upload endpoints, `graph.html` URL params): use `graphistry-rest-api`. -- JavaScript/TypeScript SDK tasks (`@graphistry/*`, browser/frontend integrations): use `graphistry-js` when available. +- JavaScript/TypeScript SDK tasks (`@graphistry/*`, browser/frontend integrations): use `graphistry-js` if available. ## Mixed Requests - If a request mixes Python SDK and REST endpoints, start with `pygraphistry` and pull in `graphistry-rest-api` for exact endpoint syntax. diff --git a/.agents/skills/graphistry/evals/evals.json b/.agents/skills/graphistry/evals/evals.json new file mode 100644 index 0000000..e3c1f66 --- /dev/null +++ b/.agents/skills/graphistry/evals/evals.json @@ -0,0 +1,47 @@ +{ + "skill_name": "graphistry", + "evals": [ + { + "id": 1, + "prompt": "I have an edges CSV with src and dst columns. How do I load it into Graphistry and get an interactive graph?", + "expected_output": "Routes to pygraphistry skill, shows graphistry.register() + graphistry.edges() + .plot() pattern with env-var credentials", + "assertions": [ + {"text": "Routes to or invokes pygraphistry skill", "type": "contains"}, + {"text": "Shows graphistry.register() with api=3", "type": "contains"}, + {"text": "Uses environment variables for credentials, not hardcoded values", "type": "contains"} + ], + "files": [] + }, + { + "id": 2, + "prompt": "What's the curl command to upload a graph to Graphistry Hub and get back a shareable URL?", + "expected_output": "Routes to graphistry-rest-api skill, shows the /api/v2/upload/ endpoint with Bearer token auth", + "assertions": [ + {"text": "Routes to or invokes graphistry-rest-api skill", "type": "contains"}, + {"text": "References /api/v2/ upload endpoint", "type": "contains"}, + {"text": "Shows Bearer token auth pattern", "type": "contains"} + ], + "files": [] + }, + { + "id": 3, + "prompt": "I want to both query my graph with Cypher AND share the result as a link. Which graphistry tools do I need?", + "expected_output": "Uses graphistry router to identify the mixed-intent request, pulls in pygraphistry-gfql for Cypher and pygraphistry-visualization for sharing", + "assertions": [ + {"text": "Identifies pygraphistry-gfql for Cypher query", "type": "contains"}, + {"text": "Identifies pygraphistry-visualization for sharing", "type": "contains"} + ], + "files": [] + }, + { + "id": 4, + "prompt": "I'm building a networkx graph and want to run Louvain community detection. How do I do that?", + "expected_output": "Does NOT activate graphistry skill; responds using networkx/python-louvain directly since no Graphistry component is needed", + "assertions": [ + {"text": "Does not call graphistry.register() or graphistry.edges()", "type": "negative"}, + {"text": "Answers with networkx or python-louvain community_louvain", "type": "contains"} + ], + "files": [] + } + ] +} diff --git a/.agents/skills/pygraphistry-ai/SKILL.md b/.agents/skills/pygraphistry-ai/SKILL.md index e960e6f..522bbb9 100644 --- a/.agents/skills/pygraphistry-ai/SKILL.md +++ b/.agents/skills/pygraphistry-ai/SKILL.md @@ -1,6 +1,12 @@ --- name: pygraphistry-ai -description: "Apply PyGraphistry graph ML/AI workflows such as UMAP, DBSCAN, embedding-based anomaly analysis, and fit/transform pipelines on nodes or edges. Use for feature-driven exploration, clustering, anomaly triage, and graph-AI notebook workflows." +description: > + PyGraphistry graph ML/AI: UMAP, DBSCAN, embeddings, and anomaly detection workflows. + Use when asked to "run UMAP on my graph", "cluster nodes", "find anomalies in my network data", + "embed nodes", "fit-transform pipeline", "semantic search over graph nodes", or "graph AI". + Also triggers on "graphistry umap", "dbscan clusters", "node embeddings", "featurize", + or "anomaly triage". Proactively suggest when the user has node feature columns and asks + about outliers, clusters, or similarity without yet using UMAP or DBSCAN. --- # PyGraphistry AI diff --git a/.agents/skills/pygraphistry-ai/evals/evals.json b/.agents/skills/pygraphistry-ai/evals/evals.json new file mode 100644 index 0000000..f60ca47 --- /dev/null +++ b/.agents/skills/pygraphistry-ai/evals/evals.json @@ -0,0 +1,46 @@ +{ + "skill_name": "pygraphistry-ai", + "evals": [ + { + "id": 1, + "prompt": "I have a graph with node feature columns (transaction_count, avg_amount, days_active). How do I run UMAP to find clusters and visualize them in Graphistry?", + "expected_output": "Shows g.featurize(kind='nodes').umap() or g.umap(feature_columns=[...]) workflow, then g.plot() with UMAP-derived x/y positions", + "assertions": [ + {"text": "Uses .umap() or .featurize().umap()", "type": "contains"}, + {"text": "Calls .plot() on the UMAP result", "type": "contains"}, + {"text": "References feature columns or featurize()", "type": "contains"} + ], + "files": [] + }, + { + "id": 2, + "prompt": "After running UMAP on my graph, how do I identify and label the outlier nodes as anomalies using DBSCAN?", + "expected_output": "Shows g.dbscan() or g.umap().dbscan() chaining, explains cluster=-1 as anomaly label, shows how to filter and highlight anomalous nodes", + "assertions": [ + {"text": "Uses .dbscan() after umap()", "type": "contains"}, + {"text": "Explains cluster label -1 as outlier/anomaly", "type": "contains"} + ], + "files": [] + }, + { + "id": 3, + "prompt": "I want to do semantic search over my graph nodes - find nodes similar to a given text query using embeddings. How?", + "expected_output": "Shows g.embed() or g.featurize(kind='nodes').embed() and semantic search/nearest-neighbor workflow in PyGraphistry", + "assertions": [ + {"text": "References .embed() or embedding-based search", "type": "contains"}, + {"text": "Shows nearest-neighbor or similarity search pattern", "type": "contains"} + ], + "files": [] + }, + { + "id": 4, + "prompt": "I want to run the Louvain community detection algorithm on my graph without using Graphistry. Just pure networkx.", + "expected_output": "Does NOT activate pygraphistry-ai; answers with python-louvain or networkx community detection directly without graphistry", + "assertions": [ + {"text": "Does not import graphistry or call graphistry.register()", "type": "negative"}, + {"text": "Answers with networkx or community.best_partition", "type": "contains"} + ], + "files": [] + } + ] +} diff --git a/.agents/skills/pygraphistry-connectors/SKILL.md b/.agents/skills/pygraphistry-connectors/SKILL.md index 0f69398..47df71c 100644 --- a/.agents/skills/pygraphistry-connectors/SKILL.md +++ b/.agents/skills/pygraphistry-connectors/SKILL.md @@ -1,6 +1,13 @@ --- name: pygraphistry-connectors -description: "Select and use PyGraphistry connector and plugin workflows for graph databases, SQL/data platforms, SIEM/log sources, and layout/compute plugins. Use when requests involve Neo4j/Neptune/Splunk/Kusto/Databricks/SQL/TigerGraph and similar integrations." +description: > + PyGraphistry connector workflows for external data sources and graph databases. + Use when asked to "connect graphistry to Neo4j", "load from Splunk into graphistry", + "query Kusto/ADX and visualize", "Databricks graph", "TigerGraph with pygraphistry", + "ingest SQL into a graph", or any "graphistry + [external platform]" request. + Also triggers on Neptune, Postgres, BigQuery, Memgraph, or connector/plugin keywords. + Proactively suggest when the user has data in an external system and wants graph visualization + without first loading it into a DataFrame. --- # PyGraphistry Connectors diff --git a/.agents/skills/pygraphistry-connectors/evals/evals.json b/.agents/skills/pygraphistry-connectors/evals/evals.json new file mode 100644 index 0000000..f9e1bab --- /dev/null +++ b/.agents/skills/pygraphistry-connectors/evals/evals.json @@ -0,0 +1,46 @@ +{ + "skill_name": "pygraphistry-connectors", + "evals": [ + { + "id": 1, + "prompt": "How do I load a Neo4j graph into PyGraphistry for visualization? I have nodes and relationships I want to explore.", + "expected_output": "Shows graphistry.neo4j(driver) or g.from_neo4j() connector pattern, covers auth and basic query to extract nodes/edges, then .plot()", + "assertions": [ + {"text": "Uses Neo4j connector (graphistry.neo4j or from_neo4j)", "type": "contains"}, + {"text": "Shows driver/auth setup with env vars", "type": "contains"}, + {"text": "Does not hardcode Neo4j credentials", "type": "negative"} + ], + "files": [] + }, + { + "id": 2, + "prompt": "I'm querying Splunk for network connection events and want to visualize the src/dest IP pairs as a graph in Graphistry. How?", + "expected_output": "Shows Splunk connector or DataFrame ingest path: run SPL query -> load into DataFrame -> graphistry.edges(df, 'src_ip', 'dest_ip').plot()", + "assertions": [ + {"text": "Shows path from Splunk query result to graphistry.edges()", "type": "contains"}, + {"text": "References Splunk SDK, splunklib, or DataFrame ingest approach", "type": "contains"} + ], + "files": [] + }, + { + "id": 3, + "prompt": "How do I connect PyGraphistry to a Databricks table and build a graph from a SQL query?", + "expected_output": "Shows Databricks connector or spark/JDBC approach to read a table into a DataFrame, then bind edges/nodes in PyGraphistry", + "assertions": [ + {"text": "Shows Databricks or Spark DataFrame -> graphistry binding path", "type": "contains"}, + {"text": "Uses env vars for Databricks credentials", "type": "contains"} + ], + "files": [] + }, + { + "id": 4, + "prompt": "I already have my data in a pandas DataFrame with src and dst columns. I just want to make a quick graph - no external DB.", + "expected_output": "Does NOT use pygraphistry-connectors; routes to pygraphistry-core for the direct DataFrame -> graphistry.edges() -> plot() path", + "assertions": [ + {"text": "Routes to pygraphistry-core for direct DataFrame plot", "type": "contains"}, + {"text": "Does not suggest a connector or external data source integration", "type": "negative"} + ], + "files": [] + } + ] +} diff --git a/.agents/skills/pygraphistry-core/SKILL.md b/.agents/skills/pygraphistry-core/SKILL.md index c279caa..36f9cb0 100644 --- a/.agents/skills/pygraphistry-core/SKILL.md +++ b/.agents/skills/pygraphistry-core/SKILL.md @@ -1,6 +1,13 @@ --- name: pygraphistry-core -description: "Core PyGraphistry workflow for authentication, shaping edges/nodes/hypergraphs, and plotting. Use for first-run setup, converting tables to graphs, and producing an initial interactive graph quickly and safely." +description: > + Core PyGraphistry workflow: auth, DataFrame-to-graph shaping, and first interactive plot. + Use when asked to "register graphistry", "get started with pygraphistry", "plot my edges dataframe", + "graphistry.register()", "bind src and dst columns", "make a hypergraph", "materialize nodes", + or any first-graph / ETL-to-plot task. Also triggers on "first graphistry graph", + "graphistry install", "api=3", or questions about graphistry auth credentials. + Proactively suggest when the user is setting up graphistry for the first time or can't + get a basic plot working from a DataFrame. --- # PyGraphistry Core diff --git a/.agents/skills/pygraphistry-core/evals/evals.json b/.agents/skills/pygraphistry-core/evals/evals.json new file mode 100644 index 0000000..32a32ce --- /dev/null +++ b/.agents/skills/pygraphistry-core/evals/evals.json @@ -0,0 +1,47 @@ +{ + "skill_name": "pygraphistry-core", + "evals": [ + { + "id": 1, + "prompt": "I just installed pygraphistry. How do I authenticate against Graphistry Hub and plot my first graph from two DataFrames?", + "expected_output": "Shows graphistry.register(api=3, username=os.environ[...], password=os.environ[...]), .edges()/.nodes() binding, and .plot(). Uses env vars for credentials.", + "assertions": [ + {"text": "Uses graphistry.register(api=3, ...)", "type": "contains"}, + {"text": "Reads credentials from os.environ or os.getenv, not hardcoded", "type": "contains"}, + {"text": "Calls .plot()", "type": "contains"}, + {"text": "Does not hardcode username or password literals", "type": "negative"} + ], + "files": [] + }, + { + "id": 2, + "prompt": "I have a wide event log table with columns actor, target, event_type, timestamp. How do I build a hypergraph from it?", + "expected_output": "Shows graphistry.hypergraph(df, ['actor', 'target', 'event_type']) and hg['graph'].plot()", + "assertions": [ + {"text": "Uses graphistry.hypergraph()", "type": "contains"}, + {"text": "Accesses hg['graph'].plot() or equivalent", "type": "contains"} + ], + "files": [] + }, + { + "id": 3, + "prompt": "My edges DataFrame has null values in the src column and the plot looks wrong. How do I diagnose and fix this?", + "expected_output": "Shows null-check/drop pattern on src/dst columns, suggests materialize_nodes() if node list is missing, confirms column type consistency", + "assertions": [ + {"text": "Addresses null handling in src/dst columns", "type": "contains"}, + {"text": "Mentions materialize_nodes() or node binding", "type": "contains"} + ], + "files": [] + }, + { + "id": 4, + "prompt": "I have a working plot and now want to color nodes by their 'risk_score' column and add badges showing the severity label. How?", + "expected_output": "Does NOT stay in pygraphistry-core; routes to pygraphistry-visualization for encoding/styling work", + "assertions": [ + {"text": "Routes to pygraphistry-visualization for encode_point_color or badge bindings", "type": "contains"}, + {"text": "Does not try to answer encode_point_color inline without referencing the visualization skill", "type": "negative"} + ], + "files": [] + } + ] +} diff --git a/.agents/skills/pygraphistry-gfql/SKILL.md b/.agents/skills/pygraphistry-gfql/SKILL.md index 6e880bd..db1e4b9 100644 --- a/.agents/skills/pygraphistry-gfql/SKILL.md +++ b/.agents/skills/pygraphistry-gfql/SKILL.md @@ -1,6 +1,12 @@ --- name: pygraphistry-gfql -description: "Construct and run GFQL graph queries in PyGraphistry using chain-list syntax OR Cypher strings. Covers pattern matching, hop constraints, predicates, let/DAG bindings, GRAPH constructors, and remote execution. Use when requests involve subgraph extraction, path-style matching, Cypher queries, or GPU/remote graph query workflows." +description: > + Construct and run GFQL graph queries in PyGraphistry using chain-list syntax or Cypher strings. + Use when asked to "query my graph with GFQL", "MATCH pattern in graphistry", "find paths between nodes", + "hop constraints", "let bindings", "GRAPH constructor", or "run Cypher on my graph". + Also triggers on "g.gfql()", "n() e_forward() n()", "chain-list query", "subgraph extraction", + "remote graph query", or "pattern matching in graphistry". Proactively suggest when the user + wants multi-hop traversal or pattern matching on a graph already loaded in PyGraphistry. --- # PyGraphistry GFQL diff --git a/.agents/skills/pygraphistry-gfql/evals/evals.json b/.agents/skills/pygraphistry-gfql/evals/evals.json new file mode 100644 index 0000000..3016b98 --- /dev/null +++ b/.agents/skills/pygraphistry-gfql/evals/evals.json @@ -0,0 +1,47 @@ +{ + "skill_name": "pygraphistry-gfql", + "evals": [ + { + "id": 1, + "prompt": "I want to find all accounts that are within 3 hops of any flagged fraud node in my transaction graph. Show me the GFQL chain-list syntax.", + "expected_output": "Shows g.gfql([n({'flagged': True}), e_forward(min_hops=1, max_hops=3), n()]) or equivalent chain-list pattern", + "assertions": [ + {"text": "Uses g.gfql() with chain-list syntax", "type": "contains"}, + {"text": "Uses e_forward() with min_hops/max_hops or equivalent", "type": "contains"}, + {"text": "Does not use deprecated hop() or chain() methods", "type": "negative"} + ], + "files": [] + }, + { + "id": 2, + "prompt": "Can I write a Cypher query to find people who know someone over 30 years old in my PyGraphistry graph?", + "expected_output": "Shows g.gfql(\"MATCH (p:Person)-[:KNOWS]->(q) WHERE q.age > 30 RETURN p.name\") or equivalent. Explains label__Person column mapping if needed.", + "assertions": [ + {"text": "Uses g.gfql() with a Cypher string argument", "type": "contains"}, + {"text": "Shows WHERE clause with property filter", "type": "contains"}, + {"text": "Mentions label__Person column mapping or recommends WHERE p.type='Person' as simpler alternative", "type": "contains"} + ], + "files": [] + }, + { + "id": 3, + "prompt": "How do I use Let bindings in GFQL to define reusable subgraph patterns as named variables?", + "expected_output": "Shows Let/DAG binding syntax with named pattern variables, explains scoping rules, gives a concrete example", + "assertions": [ + {"text": "Shows Let binding or DAG variable syntax in GFQL", "type": "contains"}, + {"text": "Includes a concrete example with named pattern variable", "type": "contains"} + ], + "files": [] + }, + { + "id": 4, + "prompt": "How do I change the color of nodes in my PyGraphistry plot based on a 'category' column?", + "expected_output": "Does NOT stay in pygraphistry-gfql; this is a visualization task and routes to pygraphistry-visualization for encode_point_color", + "assertions": [ + {"text": "Routes to pygraphistry-visualization for color encoding", "type": "contains"}, + {"text": "Does not answer the styling question with GFQL syntax", "type": "negative"} + ], + "files": [] + } + ] +} diff --git a/.agents/skills/pygraphistry-visualization/SKILL.md b/.agents/skills/pygraphistry-visualization/SKILL.md index 70ddaa3..d002b6d 100644 --- a/.agents/skills/pygraphistry-visualization/SKILL.md +++ b/.agents/skills/pygraphistry-visualization/SKILL.md @@ -1,6 +1,13 @@ --- name: pygraphistry-visualization -description: "Build PyGraphistry visualizations with bindings, encodings, layout controls, static export, and privacy-aware sharing. Use for color/size/icon/badge styling, layout tuning, map/static output, and plot link sharing workflows." +description: > + PyGraphistry visualization: bindings, color/size/icon encodings, layout controls, static export, + and privacy-safe link sharing. Use when asked to "color nodes by type", "set point size", + "add icons to nodes", "layout my graph", "export graph as PNG", "share a graphistry link privately", + or "url_params". Also triggers on "encode_point_color", "bind(point_label=...)", + "settings(url_params=...)", "badge", "static graph image", or "iframe embed". + Proactively suggest when the user has a working plot but wants to customize its appearance + or control link sharing and privacy. --- # PyGraphistry Visualization diff --git a/.agents/skills/pygraphistry-visualization/evals/evals.json b/.agents/skills/pygraphistry-visualization/evals/evals.json new file mode 100644 index 0000000..8d2e3f2 --- /dev/null +++ b/.agents/skills/pygraphistry-visualization/evals/evals.json @@ -0,0 +1,46 @@ +{ + "skill_name": "pygraphistry-visualization", + "evals": [ + { + "id": 1, + "prompt": "I have a working PyGraphistry plot. How do I color nodes by a 'category' column and size them by their 'degree' column?", + "expected_output": "Shows .encode_point_color('category', categorical_mapping={...}) and .encode_point_size('degree') with a concrete example", + "assertions": [ + {"text": "Uses encode_point_color() with categorical_mapping", "type": "contains"}, + {"text": "Uses encode_point_size() or bind(point_size=...)", "type": "contains"} + ], + "files": [] + }, + { + "id": 2, + "prompt": "How do I export my PyGraphistry graph as a static PNG image for a report?", + "expected_output": "Shows .plot(render=False) and screenshot/export approach or graphistry static export API, including any required settings like play=0", + "assertions": [ + {"text": "References static export or screenshot capability", "type": "contains"}, + {"text": "Mentions play:0 or disabling animation for static output", "type": "contains"} + ], + "files": [] + }, + { + "id": 3, + "prompt": "I want to share a Graphistry graph link with a client but only allow viewing, not editing or downloading the data. How do I set that up?", + "expected_output": "Shows privacy mode settings, sharing URL controls, token-safe link generation with privacy: 'private' or url_params for view-only access", + "assertions": [ + {"text": "References privacy mode or private sharing", "type": "contains"}, + {"text": "Shows how to generate a view-only shareable URL", "type": "contains"}, + {"text": "Does not put JWT or credentials in a URL query parameter", "type": "negative"} + ], + "files": [] + }, + { + "id": 4, + "prompt": "I want to find all nodes within 2 hops of a fraud node in my graph. What GFQL pattern should I use?", + "expected_output": "Does NOT stay in pygraphistry-visualization; routes to pygraphistry-gfql since this is a query/traversal task, not a styling task", + "assertions": [ + {"text": "Routes to pygraphistry-gfql for the traversal query", "type": "contains"}, + {"text": "Does not answer with visualization bindings or encode_point_color", "type": "negative"} + ], + "files": [] + } + ] +} diff --git a/.agents/skills/pygraphistry/SKILL.md b/.agents/skills/pygraphistry/SKILL.md index 964c647..07db082 100644 --- a/.agents/skills/pygraphistry/SKILL.md +++ b/.agents/skills/pygraphistry/SKILL.md @@ -1,6 +1,14 @@ --- name: pygraphistry -description: "TOC router for PyGraphistry tasks. Use when a request involves PyGraphistry and you need to choose the right workflow: loading/ETL shaping, visualization/layout/sharing, GFQL queries (Cypher, chain-lists, Let/DAG, GRAPH constructors), AI/UMAP/embed/semantic-search workflows, or connector-specific ingestion." +description: > + TOC router for PyGraphistry Python SDK tasks. Use when asked to "plot a graph", "visualize my edges", + "load a dataframe into graphistry", "import graphistry", "run UMAP on my graph", "query my graph with + GFQL or Cypher", "connect graphistry to Neo4j/Splunk/Kusto", or any Python SDK graph workflow. + Also triggers on "graphistry.register", "g.plot()", ".gfql()", "chain-list", "hypergraph", or + "pygraphistry". Dispatches to pygraphistry-core (auth/ETL/plot), pygraphistry-gfql (queries), + pygraphistry-visualization (styling/sharing), pygraphistry-ai (UMAP/DBSCAN/embeddings), or + pygraphistry-connectors (external DBs). Proactively suggest when the user shares an edges/nodes + DataFrame and asks about graph analysis or visualization. --- # PyGraphistry Router diff --git a/.agents/skills/pygraphistry/evals/evals.json b/.agents/skills/pygraphistry/evals/evals.json new file mode 100644 index 0000000..d8f985e --- /dev/null +++ b/.agents/skills/pygraphistry/evals/evals.json @@ -0,0 +1,47 @@ +{ + "skill_name": "pygraphistry", + "evals": [ + { + "id": 1, + "prompt": "I have a pandas DataFrame with columns 'source', 'target', and 'weight'. How do I turn that into an interactive graph visualization?", + "expected_output": "Routes to pygraphistry-core, shows graphistry.register() + .edges(df, 'source', 'target') + .plot()", + "assertions": [ + {"text": "Routes to pygraphistry-core skill", "type": "contains"}, + {"text": "Uses .edges(df, 'source', 'target') binding", "type": "contains"}, + {"text": "Calls .plot()", "type": "contains"} + ], + "files": [] + }, + { + "id": 2, + "prompt": "How do I write a GFQL query to find all 3-hop paths from fraud nodes to account nodes in my graph?", + "expected_output": "Routes to pygraphistry-gfql, shows chain-list or Cypher syntax for multi-hop traversal", + "assertions": [ + {"text": "Routes to pygraphistry-gfql skill", "type": "contains"}, + {"text": "Shows e_forward() or Cypher MATCH with variable-length path", "type": "contains"}, + {"text": "Uses g.gfql()", "type": "contains"} + ], + "files": [] + }, + { + "id": 3, + "prompt": "I want to run UMAP on my graph nodes using their feature columns and then visualize the clusters", + "expected_output": "Routes to pygraphistry-ai, shows g.umap() or g.featurize().umap() workflow with visualization step", + "assertions": [ + {"text": "Routes to pygraphistry-ai skill", "type": "contains"}, + {"text": "Shows umap() or featurize().umap() call", "type": "contains"} + ], + "files": [] + }, + { + "id": 4, + "prompt": "I need to call the Graphistry Hub REST API with a Bearer token to download graph metadata. What's the endpoint?", + "expected_output": "Does NOT use pygraphistry (Python SDK) skill; routes to graphistry-rest-api for the REST endpoint details", + "assertions": [ + {"text": "Does not suggest graphistry.register() Python SDK for the REST download", "type": "negative"}, + {"text": "Routes to graphistry-rest-api or references /api/v2/ endpoint", "type": "contains"} + ], + "files": [] + } + ] +} diff --git a/CHANGELOG.md b/CHANGELOG.md index b1c4ab6..5131635 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,13 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm ## [Development] +### Added +- **Skills / evals**: Added `evals/evals.json` to all 8 user-facing skills that were missing per-skill evals: `graphistry`, `pygraphistry`, `pygraphistry-ai`, `pygraphistry-connectors`, `pygraphistry-core`, `pygraphistry-gfql`, `pygraphistry-visualization`, `graphistry-rest-api`. Each file includes 3 positive test cases and 1 negative boundary case with assertions. + +### Changed +- **Skills / descriptions**: Updated description frontmatter for all 8 user-facing skills above to include explicit quoted trigger phrases ("Use when asked to..."), secondary trigger patterns ("Also triggers on..."), and proactive suggest clauses - following skill-creator best practices to reduce undertriggering. +- **Docs**: Added `docs/skill-evals-audit-2026-04.md` with full audit findings, priority matrix, and implementation plan. + --- ## [0.4.2 - 2026-03-30] diff --git a/docs/skill-evals-audit-2026-04.md b/docs/skill-evals-audit-2026-04.md new file mode 100644 index 0000000..74e7d1b --- /dev/null +++ b/docs/skill-evals-audit-2026-04.md @@ -0,0 +1,325 @@ +# Skill Evals Audit - April 2026 + +Sanity check of all user-facing skills against skill-creator best practices. +Focus: missing evals, weak trigger phrases, description anti-patterns, tier sizing. + +--- + +## Scope + +Skills audited (all user-facing, `~/.claude/skills/`): + +| Skill | Lines | Has `evals/` | +|---|---|---| +| `graphistry` (router) | 32 | No | +| `pygraphistry` (router) | 48 | No | +| `pygraphistry-ai` | 68 | No | +| `pygraphistry-connectors` | 79 | No | +| `pygraphistry-core` | 84 | No | +| `pygraphistry-visualization` | 168 | No | +| `graphistry-rest-api` | 268 | No (has `references/`) | +| `pygraphistry-gfql` | 232 | No | + +Skills already with evals (not audited): `ad-attack-paths`, `azure`, `cyber-umap-anomaly`, +`defender-incident-graph`, `disk-cleanup`, `extract-skill`, `gmail-*`, `kusto-security-graph`, +`network-segmentation`, `sentinel-graph-hunt`, `threat-hunt-splunk-graph`, `vpc-flow-graph`, +`zeek-network-graph`, `zendesk`. + +--- + +## Best Practices Reference (from skill-creator) + +1. **Evals** - every skill needs `evals/evals.json` with 2-3 positive test cases + 1 negative +2. **Description trigger phrases** - use `"Use when asked to [phrase]"` and `"Also triggers on [phrase]"` patterns with quoted realistic user phrases; be "pushy" (Claude undertriggers by default) +3. **Proactive suggest** - add `"Proactively suggest when [context clue]"` for skills with clear ambient signals +4. **No negative conditions in description** - "Don't use for X" belongs in the skill body, not frontmatter +5. **Tier sizing:** + - Minimal: <100 lines - single SKILL.md only + - Middle: 100-300 lines - SKILL.md + `examples/` + - References tier: 300+ lines - SKILL.md + `references/` dir +6. **Negative assertions** - at least one eval should verify the skill does NOT fire for an adjacent domain + +--- + +## Findings by Skill + +### graphistry (router) - CRITICAL + +**Current description:** +> Umbrella router for Graphistry workflows across SDK and API surfaces. Use to dispatch between Python SDK, REST API, and (future) JavaScript SDK workflows. + +**Issues:** +- No evals +- Description has zero quoted trigger phrases; a user saying "graph my logs with graphistry" or "how do I use the graphistry API" may not reliably trigger this +- "(future) JavaScript SDK workflows" is dead weight - signals a half-finished feature to the model +- No proactive suggest +- At 32 lines it is appropriately minimal, but the description needs more trigger surface + +**Recommended description:** +```yaml +description: > + Umbrella router for Graphistry workflows across SDK and API surfaces. + Use when asked to "visualize with graphistry", "use graphistry", "graph this + data in graphistry", "graphistry plot", or any question mixing Python SDK + and REST API concerns. Also triggers on "graphistry SDK vs API", "how do I + share a graphistry graph", or "graphistry authentication". Routes to + pygraphistry for Python SDK tasks and graphistry-rest-api for curl/REST tasks. + Proactively suggest when the user mentions Graphistry Hub, graph visualization, + or network graph sharing without specifying an interface. +``` + +**Evals needed:** 3 cases: Python SDK intent, REST API intent, ambiguous cross-interface intent; 1 negative for a domain that is NOT graphistry (e.g., networkx-only graph analysis) + +--- + +### pygraphistry (router) - CRITICAL + +**Current description:** +> TOC router for PyGraphistry tasks. Use when a request involves PyGraphistry and you need to choose the right workflow: loading/ETL shaping, visualization/layout/sharing, GFQL queries (Cypher, chain-lists, Let/DAG, GRAPH constructors), AI/UMAP/embed/semantic-search workflows, or connector-specific ingestion. + +**Issues:** +- No evals +- Description uses "Use when a request involves PyGraphistry" - Claude must already know it's PyGraphistry; misses users who say "how do I plot a graph in Python" or "my edges dataframe won't plot" +- No quoted realistic user phrases +- No proactive suggest clause +- Body has good routing logic, but description alone may not pull in the right cases + +**Recommended description:** +```yaml +description: > + TOC router for PyGraphistry Python SDK tasks. Use when asked to "plot a graph", + "visualize my edges", "load a dataframe into graphistry", "import graphistry", + "run UMAP on my graph", "query my graph with GFQL or Cypher", or "connect + graphistry to Neo4j/Splunk/Kusto". Also triggers on "graphistry.register", + "g.plot()", ".gfql()", "chain-list", or "hypergraph". Routes to specialized + sub-skills (pygraphistry-core, pygraphistry-gfql, pygraphistry-ai, etc.). + Proactively suggest when the user shares an edges/nodes DataFrame and asks + about graph analysis or visualization. +``` + +**Evals needed:** 3 cases: ETL+plot, GFQL/Cypher query, UMAP/ML workflow; 1 negative for REST-only request (should route to graphistry-rest-api, not this skill) + +--- + +### pygraphistry-ai - HIGH + +**Current description:** +> Apply PyGraphistry graph ML/AI workflows such as UMAP, DBSCAN, embedding-based anomaly analysis, and fit/transform pipelines on nodes or edges. Use for feature-driven exploration, clustering, anomaly triage, and graph-AI notebook workflows. + +**Issues:** +- No evals +- Passive "Apply ... such as" format - no trigger phrases +- No "Use when asked to..." pattern +- No proactive suggest +- 68 lines - fits minimal tier, no resizing needed + +**Recommended description:** +```yaml +description: > + Apply PyGraphistry graph ML/AI workflows: UMAP, DBSCAN, embeddings, and + anomaly detection. Use when asked to "run UMAP on my graph", "cluster nodes", + "find anomalies in my network data", "embed nodes", "fit-transform pipeline", + or "semantic search over graph nodes". Also triggers on "graphistry umap", + "dbscan clusters", "node embeddings", or "anomaly triage". Proactively suggest + when the user has a node feature table and asks about outliers, clusters, or + similarity. +``` + +**Evals needed:** 3 cases: UMAP run, DBSCAN clustering, anomaly triage; 1 negative for a scikit-learn-only clustering task with no graphistry + +--- + +### pygraphistry-connectors - HIGH + +**Current description:** +> Select and use PyGraphistry connector and plugin workflows for graph databases, SQL/data platforms, SIEM/log sources, and layout/compute plugins. Use when requests involve Neo4j/Neptune/Splunk/Kusto/Databricks/SQL/TigerGraph and similar integrations. + +**Issues:** +- No evals +- "Use when requests involve" is passive; no quoted trigger phrases +- Good connector enumeration but reads like a capabilities list, not a trigger guide +- 79 lines - appropriate minimal tier + +**Recommended description:** +```yaml +description: > + PyGraphistry connector workflows for external data sources and graph databases. + Use when asked to "connect graphistry to Neo4j", "load from Splunk into graphistry", + "query Kusto/ADX and visualize", "Databricks graph", "TigerGraph with pygraphistry", + or "ingest SQL into a graph". Also triggers on Neptune, Postgres, BigQuery, + Memgraph, or any "graphistry + [platform]" request. Proactively suggest when + the user has data in an external system and wants graph visualization. +``` + +**Evals needed:** 3 cases: Neo4j load, Splunk ingest, SQL-to-graph; 1 negative for a direct DataFrame plot with no external connector + +--- + +### pygraphistry-core - HIGH + +**Current description:** +> Core PyGraphistry workflow for authentication, shaping edges/nodes/hypergraphs, and plotting. Use for first-run setup, converting tables to graphs, and producing an initial interactive graph quickly and safely. + +**Issues:** +- No evals +- No quoted trigger phrases; a first-time user saying "how do I get started with graphistry" or "my plot() call fails" may not trigger this +- "Use for first-run setup" is too narrow - this is also the baseline for any ETL + plot task +- 84 lines - appropriate minimal tier + +**Recommended description:** +```yaml +description: > + Core PyGraphistry workflow for auth, DataFrame-to-graph shaping, and first + interactive plot. Use when asked to "register graphistry", "get started with + pygraphistry", "plot my edges dataframe", "graphistry.register()", "bind src + and dst columns", "make a hypergraph", or "materialize nodes". Also triggers + on "first graphistry graph", "graphistry install", "api=3", or any question + about graphistry auth credentials. Proactively suggest when the user is + setting up graphistry for the first time or can't get a basic plot working. +``` + +**Evals needed:** 3 cases: first-time register+plot, ETL shaping, hypergraph build; 1 negative for a GFQL pattern-match query (should route to pygraphistry-gfql) + +--- + +### pygraphistry-gfql - HIGH + +**Current description:** +> Construct and run GFQL graph queries in PyGraphistry using chain-list syntax OR Cypher strings. Covers pattern matching, hop constraints, predicates, let/DAG bindings, GRAPH constructors, and remote execution. Use when requests involve subgraph extraction, path-style matching, Cypher queries, or GPU/remote graph query workflows. + +**Issues:** +- No evals +- Description is technically good but has no quoted trigger phrases +- "Use when requests involve" is passive +- At 232 lines this is in the middle tier but has no `examples/` directory to offload detail +- Body is well-structured; descriptions need trigger phrase upgrade more than body restructuring + +**Recommended description:** +```yaml +description: > + Construct and run GFQL graph queries in PyGraphistry using chain-list syntax + or Cypher strings. Use when asked to "query my graph with GFQL", "MATCH pattern + in graphistry", "find paths between nodes", "hop constraints", "let bindings", + "GRAPH constructor", or "run Cypher on my graph". Also triggers on "g.gfql()", + "n() e_forward() n()", "chain-list query", "subgraph extraction", or "remote + graph query". Proactively suggest when the user wants pattern matching or + multi-hop traversal on a graph already loaded in PyGraphistry. +``` + +**Tier note:** At 232 lines, consider adding an `examples/` directory with the extended Cypher/chain-list quick-reference tables to keep SKILL.md under 200 lines. + +**Evals needed:** 3 cases: chain-list hop query, Cypher MATCH, Let/DAG binding; 1 negative for a visualization-only request (should route to pygraphistry-visualization) + +--- + +### pygraphistry-visualization - MEDIUM + +**Current description:** +> Build PyGraphistry visualizations with bindings, encodings, layout controls, static export, and privacy-aware sharing. Use for color/size/icon/badge styling, layout tuning, map/static output, and plot link sharing workflows. + +**Issues:** +- No evals +- Passive "Use for" pattern; no trigger phrases +- 168 lines - sits in middle tier; has `references/` dir for icon lookup, which is good +- Body content is solid + +**Recommended description:** +```yaml +description: > + PyGraphistry visualization: bindings, color/size/icon encodings, layout + controls, static export, and privacy-safe link sharing. Use when asked to + "color nodes by type", "set point size", "add icons to nodes", "layout my + graph", "export graph as PNG", "share a graphistry link privately", or + "url_params". Also triggers on "encode_point_color", "bind(point_label=...)", + "settings(url_params=...)", "badge", or "static graph image". Proactively + suggest when the user has a working plot but wants to customize its appearance + or control who can see the shared URL. +``` + +**Evals needed:** 3 cases: color encoding, layout + export, privacy sharing; 1 negative for a GFQL query masquerading as a visualization request + +--- + +### graphistry-rest-api - MEDIUM + +**Current description:** +> Graphistry Hub REST API specialist for auth, upload lifecycle, URL controls, sessions, and sharing safety. Use for curl/requests endpoint guidance independent of SDK choice. + +**Issues:** +- No evals +- No quoted trigger phrases; "Use for curl/requests endpoint guidance" is passive +- At 268 lines approaching references tier; already has `references/` dir - good +- "independent of SDK choice" is vague and adds little triggering value + +**Recommended description:** +```yaml +description: > + Graphistry Hub REST API: auth, upload, URL controls, sessions, and sharing. + Use when asked to "call the graphistry API with curl", "get a JWT token from + graphistry", "upload a graph via REST", "graph.html URL parameters", "graphistry + session API", or "share a graph link safely". Also triggers on "/api/v2/", + "Bearer token", "graphistry upload endpoint", or any direct HTTP endpoint + question about Graphistry Hub. Prefer this over pygraphistry when the user + explicitly uses curl, requests, or raw HTTP rather than the Python SDK. +``` + +**Evals needed:** 3 cases: JWT auth flow, graph upload via REST, URL param control; 1 negative for a Python SDK `.plot()` task (should route to pygraphistry-core, not REST) + +--- + +## Priority Matrix + +| Priority | Issue | Skills Affected | Effort | +|---|---|---|---| +| P1 | Missing `evals/evals.json` | All 8 | Medium per skill (3-4 cases each) | +| P2 | No quoted trigger phrases in description | All 8 | Low per skill (rewrite ~3 lines) | +| P3 | No proactive suggest clause | All 8 (routers most critical) | Low (1 line each) | +| P4 | gfql tier sizing (232 lines, no examples/) | pygraphistry-gfql | Low-Medium | +| P5 | Dead-weight content in descriptions | graphistry, graphistry-rest-api | Low | + +--- + +## Implementation Plan + +### Phase 1 - Description updates (fast, low-risk) + +For each skill, update the `description` frontmatter in `.agents/skills//SKILL.md`: +- Add `"Use when asked to [quoted phrase]"` pattern +- Add `"Also triggers on [phrase]"` secondary triggers +- Add `"Proactively suggest when [context]"` where applicable +- Remove passive "Use for" and "Use when requests involve" phrasing +- Remove dead-weight forward-looking notes ("future JS SDK") + +Order: graphistry -> pygraphistry (routers first, highest leverage) -> pygraphistry-core -> pygraphistry-gfql -> pygraphistry-ai -> pygraphistry-connectors -> pygraphistry-visualization -> graphistry-rest-api + +### Phase 2 - Add evals (main effort) + +For each skill, create `.agents/skills//evals/evals.json`: +- 2-3 realistic positive prompts (what a real user would type) +- 1 negative prompt (adjacent domain that should NOT trigger this skill) +- `expected_output` description for each +- `assertions` array with `contains` and `negative` type checks + +Suggested eval themes per skill are listed in each finding above. + +### Phase 3 - Tier review for pygraphistry-gfql + +- Review whether chain-list/Cypher quick-reference tables can move to `examples/` to bring SKILL.md under 200 lines +- Assess if this improves model behavior or is purely cosmetic + +### Phase 4 - Description optimization loop (optional, post-merge) + +Run `scripts/run_loop.py` with trigger evals for the two router skills (graphistry, pygraphistry) +since they have the most triggering surface area. This requires the `claude` CLI. + +--- + +## Acceptance Criteria for PR + +- [ ] All 8 skills have `evals/evals.json` with 3+ cases each (2 positive + 1 negative minimum) +- [ ] All 8 skill descriptions include at least 3 quoted trigger phrases +- [ ] All 8 skill descriptions include a "Proactively suggest when" clause +- [ ] No description uses passive "Use for" or "Use when requests involve" as the only trigger guidance +- [ ] No negative conditions ("Don't use for X") in any description frontmatter +- [ ] `validate_skills.py` passes for all modified skills +- [ ] CHANGELOG entry added under `[Unreleased]`