Claude Code Skills Get Stale. Audit Them Quarterly.

In AI-assisted research, a stale instruction could place a wrong number in an analysis and propagate that error to later documents. The infrastructure accumulates fast: Skills that format tables for a specific journal. Agents that pull data from a federal API and validate it. Hooks that block a document from compiling until every figure has descriptive alt text. Memory entries that preserve a project-specific correction. The practical question is which version-dependent instructions need regression tests.

Why this fails differently in research

A version-dependent instruction creates a possible propagation path: processing error, incorrect estimate, public claim, and downstream use. This page does not document a completed case along that full chain.

A hypothetical propagation path to convert into assertions and release checks.

Consider a hypothetical memory entry that strips a leading zero because an earlier Census endpoint returned a malformed code. If the source format later changes, the instruction could corrupt a merge without raising an error. A regression test should compare the returned schema and merge keys with a frozen fixture before the analysis runs.

A second hypothetical skill drops two header rows because one Bureau of Labor Statistics file included two metadata lines. If a later vintage has one, the same instruction could discard the first observation. A row-count and header-schema assertion would detect that change.

These scenarios are plausible failure modes, not observed peer-review failures. Each scenario can be implemented as a fixture with a known pass and fail condition.

The dumbest model available

Models and tools change across releases. A general capability forecast is not a benchmark for a particular research task. Some skills compensate for a model limitation; others encode durable project rules. Both types should be treated as version-dependent until tested.

Divergence from a bare-model output is a signal to inspect, not proof that the skill is harmful. The relevant comparison is whether each version satisfies the same frozen assertions, not whether their prose or implementation is identical.

Evaluating skills with benchmarks, not vibes

The instinct is to assess skills informally: “it seems to work.” The problem is that “seems to work” does not survive a model update, and in research, incorrect numbers can appear plausible.

A more rigorous approach uses the same logic that applies to empirical research: define a testable claim, run the experiment, compare to a baseline.

Step 1: Define assertions. Each skill should have a set of concrete, checkable outcomes. For an accessibility compliance skill, the assertions are specific: the output document must have a tagged structure, all figures must have alt text, all table header cells must be marked as headers.

For a data-cleaning skill, the assertions are empirical: the output file must have the expected number of rows, the merge key must be unique, the value range must fall within documented bounds. A prompt to generate assertions for an existing skill:

Read the skill file at [path]. List every concrete,
testable outcome this skill is supposed to produce.
For each outcome, write a check that returns pass/fail.
Flag any outcome that depends on assumptions about
the data format or model behavior that might have
changed since the skill was written.

Step 2: Compare skill output to bare-model output. The key test: does the skill still add value over what the model does on its own?

I'm going to give the same task twice. First, handle
it using only these instructions (no skill). Then I'll
run the skill on the same input. Compare the two
outputs and note any differences. Flag cases where
the skill produces worse output than the plain prompt.

If the outputs are equivalent on the frozen assertions, the skill may be redundant for that task and model version. If the skill produces worse scored output, retire or revise it after confirming the comparison across representative fixtures.

Step 3: Track correction rates over time. For skills that do add value, track how often they trigger changes.

The numeric thresholds in this example are configurable. A project should set them from task frequency, error cost, and enough runs to distinguish inactivity from low task demand.

Look at the recent times this skill was run. How many
corrections or changes did it make each time? What is
the trend? If it is consistently catching fewer than
the project's predeclared threshold, flag it for review.
If it caught zero across the predeclared review window,
flag it for a retirement test.

An editing skill might catch eight corrections per document in one period and two in another. That decline could reflect model changes, task mix, or easier documents. A zero count is a review trigger, not by itself proof that the skill has become overhead.

Fixing stale triggers

Skills activate based on trigger descriptions: keywords and phrases that tell the assistant when to invoke them. As projects and data sources evolve, trigger performance can deteriorate through two error types.

False negatives mean the skill does not fire when it should. A slide-building skill triggers on “branded presentation” and “department slides.” But when a colleague asks for “a deck for the advisory board,” the skill does not fire because “deck” and “advisory board” are absent from the trigger list.

False positives mean the skill fires when it should not. A data-pipeline skill triggers on “pull economic data” and “download federal statistics.” It also triggers on “fetch the data,” which is broad enough to fire when working with a local spreadsheet that has nothing to do with federal APIs.

Trigger performance can be measured with labeled prompts. For a slide-building skill, the test set might look like this:

Should activate:

Make me a presentation for the advisory board meeting
Build a deck summarizing the Q2 findings
Create slides for the department retreat
Put together a PowerPoint for the grant review
Draft a slide deck on the pilot program results

Should NOT activate:

Summarize the advisory board meeting notes
Write a memo about the Q2 findings
Create a one-page brief for the department
Draft a report for the grant review
Write up the pilot program results

Running all ten through the assistant and checking which ones trigger the skill identifies trigger errors in both directions.

Maintenance and reproducibility

Published research adds a reproducibility requirement: the infrastructure that produced the results needs to be stable and documented at publication time, even as it keeps evolving for future work. A reproducibility package that includes skill files and memory entries is a snapshot of a specific model-plus-infrastructure configuration, and the maintenance cycle should not retroactively alter the infrastructure behind published results.

The practical approach borrows from Sandve et al.’s Ten Simple Rules for Reproducible Computational Research (PLOS Comput Biol, 2013): tag or archive the skill and memory state at submission time (a git tag works), then continue evolving the working infrastructure. The tag identifies the skill and memory files used for the run. Reproducing the complete configuration also requires the model version, dependencies, data, external services, environment, prompts, and saved outputs. The current version reflects the latest tested configuration.

This creates a clear separation between “what produced this result” (frozen) and “what produces the best result going forward” (evolving). Without that separation, maintaining infrastructure for new work can make the published workflow harder to reproduce.

A maintenance cycle

A quarterly cycle is a proposed default cadence to test and adjust. The protocol has six steps: audit by invocation frequency, benchmark high-use skills against the bare model, test triggers in both directions, prune memory entries whose underlying reason no longer applies, consolidate overlapping skills, and retire hooks that no longer improve scored output.

Audit by invocation frequency. Pull the list of all skills. Flag any skill uninvoked during a predeclared inactivity window, such as 60 days, then distinguish low task frequency, trigger failure, and native-model redundancy with representative fixtures.

Benchmark high-use skills against the bare model. For skills invoked weekly or more, run them against the model on three to five representative tasks without the skill loaded. Compare outputs.

Test triggers on fresh prompts. Five prompts that should activate the skill, five that should not. Check both directions.

Prune memory entries. A prompt to audit them:

Read MEMORY.md and every memory file it references.
For each entry, check: (1) does the file or behavior
it references still exist? (2) is the "why" reason
still valid? (3) is it older than 90 days without
being updated? Flag an entry if (1) is no or (2) is
no. If (3) is yes, flag it for age-based review.
Always flag a referenced file or tool that no longer
exists.

Consolidate overlapping skills. A prompt to find candidates:

Read every skill file in the skills directory. Group
them by what they do. Flag any group where two or more
skills have overlapping trigger descriptions or produce
similar outputs. For each overlap, suggest how to merge
them into a single skill with a mode parameter.

Treat hooks as temporary. A hook that blocks document compilation without alt text on every figure exists because someone once forgot alt text. Once frozen tests show that the model includes required alt text without the hook across representative fixtures, compare the hook’s residual catches with its cost before retiring it.

The maintenance benefit

Each maintenance cycle can remove redundant instructions, incorporate capabilities demonstrated by the current model, and retire outdated patches.

Trusting the processes we built means reassessing them systematically and regularly. Run the audit before a published table is reused in an eligibility formula.