Evaluating Overlapping Plugins in Coding Agents

You install one coding-agent plugin for its PR review workflow, another for its debugging tools, and a third because it contains a skill the first two lack. Then you ask the agent to review a pull request. The result looks more thorough than yesterday’s. Was that the model, the new plugin, or a different review procedure selected from the collection you already had?

That is the part of the plugin ecosystem I find difficult to reason about. Installing overlapping capabilities is understandable. Attributing the resulting behaviour is harder, especially when the model and configuration change between sessions.

A plugin manifest tells you what you installed; it does not tell you whether the resulting agent works better.

There are several steps between those two things. By harness, I mean the application that assembles context, exposes tools and manages the agent’s execution. Its configuration helps determine which installed components can participate in a run:

flowchart TD
 Installed[Installed components] --> Enabled[Enabled for this scope]
 Enabled --> Available[Available in this run]
 Available --> Used[Invoked, loaded or executed]
 Used --> Outcome[Review findings and measured cost]
 Context[Harness, model, repository and tool configuration] -.-> Available
 Context -.-> Used
 Context -.-> Outcome

This is a distinction to investigate, not a promise that every component follows the same lifecycle. A skill selected by a model and a command hook triggered by an event take different routes into the work.

Two Review Skills Can Mean Two Different Reviews#

Suppose our pull request changes a payment service’s retry handler. One installed skill instructs the agent to inspect the diff, trace changed call sites and report only actionable regressions introduced by the patch. Another asks for a broader review: inspect the subsystem, run integration tests and include existing reliability risks relevant to the change.

Both are reasonable procedures. They answer different questions. The first may correctly omit a pre-existing retry-storm risk that the second considers important. If one demands patch-introduced findings only and the other demands relevant existing risks too, their reporting requirements cannot both govern the same findings list unchanged.

Now give both skills descriptions that match “review this PR”. The installation can be perfectly valid while the intended procedure remains ambiguous. The agent might select one, combine parts of both, or need explicit direction. Which happens depends on the harness and the run; it is something to observe, not infer from the finished prose.

This is not evidence that plugin systems lack manifests or dependency handling. Claude Code plugins have manifests, namespaced skills and multiple component types. Its dependency documentation describes resolution and constraints. Its marketplace documentation also supports source references and SHA pinning, with marketplace references distinct from the sources of individual plugins.

Those mechanisms address packaging and identity. Resolving compatible package versions does not establish that two review procedures compose sensibly. Namespacing lets you distinguish the skills; it does not decide which review you wanted. Semantic compatibility and package compatibility are separate questions.

Available Instructions Are Not All Loaded Instructions#

It is tempting to picture every installed plugin being concatenated into one enormous prompt. That is not an accurate general model of skills.

Claude Code skills use progressive disclosure: descriptions support discovery, while full skill instructions are loaded when invoked. Manual-only controls can also keep a skill’s description out of the model’s automatic discovery context. A command hook, by contrast, runs according to configured events and conditions rather than the model deciding that a skill sounds relevant. Deterministic triggering does not guarantee a successful command, but it is a different selection mechanism.

VS Code’s skill documentation likewise describes progressive loading. Codex documents implicit skill invocation as well as explicit invocation. These similarities do not make the harnesses interchangeable: supported components, invocation controls and execution behaviour need checking in each product, including when moving plugins into VS Code.

For our retry review, the useful question is therefore not just “which plugins were installed?” It is which were enabled, which descriptions or tools were available, which skill bodies were loaded, and which commands or tools actually ran. Tool availability is not a tool call. Loading instructions is not proof that the agent followed every step.

Descriptions, loaded content and tool results can all contribute context, but an installation count cannot tell you the token cost. Use reported token usage and available traces. If the harness does not expose a particular loading or selection event, record that limitation rather than inventing a complete execution history. A longer review is not, by itself, evidence of either better coverage or wasted context.

Compare Configurations Against the Same Work#

I would start evaluating the retry review with a same-harness baseline that disables optional plugins while retaining required repository instructions and safety controls. Comparing an unrestricted bare agent with a governed plugin setup changes too many things at once, including what actions the agent is allowed to take.

Run that baseline and the candidate configuration against the same repository revision and task input, with the same exposed model settings, tool permissions and budgets. Start each attempt in a fresh context and restore equivalent workspace state. Otherwise, the second reviewer may inherit the first reviewer’s findings, test output or edits.

Keep a record of the harness version, model identifier, resolved plugin references, enabled components, relevant configuration changes and the repository instructions used. Pin what can be pinned. A marketplace revision alone is not necessarily the resolved revision of every plugin it references. Record observable configuration, not an imaginary hash of hidden system instructions. Hosted model internals and service-side changes are not fully reproducible from the client.

Repeat paired comparisons across representative review tasks, varying run order where practical. A single successful attempt cannot separate configuration effects from run-to-run variation. There is no universal task count: choose coverage and repetition according to the decisions at stake, and report uncertainty alongside the results.

For the retry change, assess whether the review catches an introduced duplicate-payment risk, whether its claims are supported by code and tests, and whether it invents defects. Include safety violations, human effort spent validating or dismissing findings, elapsed time and actual tokens. Report variance as well as averages. Finding more issues is not automatically better if most of them consume reviewer time without improving the decision.

Start with useful additions and removals: the focused review skill alone, the broader one alone, and removing either from the existing setup. Where they overlap, test their interaction. You do not need every possible pair of unrelated plugins. Explicit invocation can help distinguish a procedure’s value from automatic selection quality; both matter if automatic selection is how you normally work.

These are proposed measurements, not results from an experiment I have run. The outcome may favour either skill, their combination, or the baseline. The useful claim would be narrow: this configuration improved these tasks under these conditions, at this cost.

The Source You Trust Can Extend Beyond the Listing#

Evaluation also needs a boundary around what you are willing to execute. A marketplace listing is a discovery mechanism, not necessarily the source repository or the complete set of executable dependencies. Review the actual plugin source and what its components depend on. Curation can help, but it does not replace that inspection; Claude Code’s discovery documentation includes trust warnings for plugins.

The components matter here. A text-only review skill supplies instructions; it is not equivalent to a hook that runs a shell command or an MCP server process with access to local files. Instructions can still influence the agent’s use of permitted tools, so text is not risk-free. But treating every skill as arbitrary executable code obscures the different permissions and trust decisions involved.

Pinning a reviewed revision makes later changes easier to detect. It does not prove the pinned content is safe, nor automatically freeze every downstream dependency. For the broader retry reviewer, inspect what “run integration tests” actually causes to execute and what credentials or network access that environment has. Do that before benchmarking it, not after it produces a persuasive report.

The direction I want from this ecosystem is better visibility into effective per-run configuration and evidence for its contribution. Manifests and resolvers already provide part of the foundation. The remaining question is whether the configured agent makes the work better.

Back at the pull request, that means knowing which review you asked for, which procedure ran, and whether it helped a human decide what to merge. Without that evidence, it is too easy to credit the model for a configuration change, or blame the model for a review procedure you never meant to select.

Further Reading#

• Claude Code plugins, marketplaces and dependency resolution.
• Claude Code skills and plugin discovery and trust.
• VS Code agent skills and agent plugins.
• Building skills for Codex.

Sources checked on 5 September 2026. Product behaviour described here is based on documentation, not measured comparative results.