We tested our own product for three weeks. Here's why we're winding it down.
SupaSkills started with a bet: that curated, quality-scored AI skills make model output better. We spent the last three weeks trying to prove that bet with blind, cross-model benchmarks instead of assuming it. It didn't hold. We're winding the product down, and this is the honest account of what we found — because the measurements are more useful to you than the catalog ever was.
What we tested
We ran ~1,088 skills against Claude Opus 4.8, GPT-5.6, and Gemini Flash. Blind judging, always by a different model family than the one being judged (a model grading its own family inflates results — our old "89% win rate" came from Claude judging Claude, and it did not survive a fair test). Deterministic where we could measure it, pre-registered gates before every run so we couldn't move the goalposts after seeing the numbers.
What we found
Skills as a system prompt don't help a frontier model. On every Claude tier, loading a skill was net-neutral to net-negative. The 2026 models already ship the work discipline the skills were teaching. Independent benchmarks published this year (SWE-Skills-Bench, SkillsBench) found the same thing on strong models: most skills produce zero improvement, some make it worse.
On current facts, a curated catalog is a liability. Our skills carried numbers — tax thresholds, procurement limits, rates. Numbers go stale. On fresh-fact tasks the catalog confidently served outdated values, and a model that simply searched the web beat it every time, at 93–100% correct. We tested whether a skill could fix this by telling the model "don't trust your memory, search here." It couldn't: current models already search on their own when you give them the tool. There was no gap left to fill.
Combining skills made it worse, not better. A single clear prompt beat a stack of three or four skills. More structure injected more defects, not fewer.
The one place skills helped was weak models. On smaller, cheaper models, curated skills did add value. But that's a shrinking market — this year's cheap model is next year's mid-tier, and the value drops as models improve. We watched it shrink mid-test: on the newest cheap model, the advantage was already gone.
What this means for you
If you're on a frontier model, you don't need a skill catalog. Talk to the model, give it search and tools, and correct it as you go. A conversation beats a perfect prompt, and prompt-crafting itself matters far less than it did two years ago.
Skills still have a real, narrow use: bundling executable tools and procedures a model lacks (Anthropic's PDF and spreadsheet skills run code for deterministic results), and weak-model deployments where quality still matters on a budget. That's not the product we built. We built a knowledge catalog, and knowledge is the part frontier models already have.
Why we're telling you this
The AI skills space is full of "these 5 skills 10x your output." We measured it. Most of it doesn't hold, and almost nobody publishing those claims has run a controlled test. We ran ours on our own product and it told us to stop. We'd rather publish that than keep selling something our own numbers don't support.
The catalog is winding down. The measurement work — the methodology, the benchmarks, what we learned about testing AI claims honestly — is the part worth keeping. If that's useful to you, it's why this post exists.
— Max, Kill The Dragon