ARGUS·LAB
repo

adversarial testing lab · agentic payment rails

We sign our findings,
not our opinions.

Argus Agent Test Lab runs black-box adversarial batteries against the SDKs and platforms that move agentic money — x402 payment flows, credit and settlement handling, idempotency guarantees. Every verdict is a sealed record in a public hash chain that anyone can verify in one command.

Explore the registryCommission a round
sealed records, hash-chained
10
sealed records, hash-chained
systems under test
6
systems under test
pre-registered probes
60+
pre-registered probes
findings retracted
0
findings retracted

the registry

A chain of seals, not a list of claims

Each node is an Ed25519-signed record chained to the hash of its predecessor: finding or battery, verdicts, findings, hash-pinned evidence. Select a seal to inspect it.

latest seal

registry intact · 10 records · verify-registry.mjs ↗

argus-seal-00092026-10-28tripl3tr3s/mcp-tool-idempotency (public repo, unmodified)

mcp-tool-idempotency external verification — 4d665ae to v0.2.0: F1/F2/F3 durability windows fixed, ten-window battery 10/10 PASS, suite 91/91 @ tag / 93/93 @ main

verdicts

T1-Before: FAIL (SUT)T1-W0..W6: PASST1-Suite: PASS

chain a5512384 → c26af630 ✓ · Ed25519 · prev_record_hash → chain head

hash-pinned evidence (5)

external-verification-mcp-tool-idempotency-v0.2.0.mdpack1-t1-results.jsonpack1-t1-results-rerun-2026-10-28.jsonpack1-t1-harness.tspvr-t1-mcp-tool-idempotency.md

$ git clone github.com/igor-vii/Argus-Agent-Test-Lab && node tools/registry/verify-registry.mjs → registry intact: 10 records, chain head c26af630…

attestations/records — hash chain explorer
argus-seal-00092026-10-28tripl3tr3s/mcp-tool-idempotency (public repo, unmodified)

mcp-tool-idempotency external verification — v0.2.0 fixes hold (10/10)

The cooperative model working end-to-end: three durability windows (lost response after commit, crash after commit, cross-instance race) reported via coordinated disclosure; the maintainer shipped owner-gated claim leases and released v0.2.0; the lab verified the fix with a ten-window battery — 10/10 PASS, suite 91/91 at tag. Credited in the project's SECURITY.md.

verdicts

T1-Before (4d665ae): FAIL (SUT)T1-W0..W6 (v0.2.0): PASST1-Suite: PASS (91/91)

5 hash-pinned evidence files

findings

  • F1 (high) Lost response after commit: retry re-fires the committed side effect (fixed in v0.2.0).
  • F2 (high) Crash after commit: dead holder's pending row unconditionally reset by the next caller (fixed in v0.2.0).
  • F3 (high) Cross-instance race: claim loser resets the winner's LIVE pending row (fixed in v0.2.0).
mcp-tool-idempotency ↗

$ node tools/registry/verify-registry.mjs → registry intact: 10 records, chain head c26af6309d54c6e2… records ↗

method

Four rules that make a verdict worth something

The lab's credibility is structural: it comes from what a probe must go through before anyone is allowed to call it a failure.

01

Black-box by construction

The SUT is always the published artifact — npm dist, exact pin, unmodified. No source access, no vendor cooperation required. If it can't be shown against the artifact users actually install, it isn't shown.

02

Pre-registered before the run

Required behaviors and verdict rules are fixed in committed matrix files before a single probe fires. No post-hoc reshaping of what a failure means — the rig can't drift toward the conclusion.

03

Offline by default

All traffic on 127.0.0.1 against adversarial stands and honest mock backends modeled from the vendor's own contracts, each decision citing a vendor source line. No vendor-operated service is touched without written authorization.

04

Sealed and hash-chained

Every record is signed with Ed25519 and chained to the previous record's hash. Anyone can verify the whole chain from the public repo in one command. Seals attest run validity and findings completeness — not bug-freeness.

verdict taxonomy

FAIL
Vendor-side failure under the probe — the SDK or platform loses money or integrity under the tested fault.
HOLD
The SDK rejected or limited the fault — a genuine guard fired where the battery expected one.
PASS
The platform held beyond the documented scenarios.
OBSERVED
Observed behavior matches the vendor's published disclosure — an honest-posture result, not a defect.
DX
Developer-experience finding: no direct money-loss path, but the tooling hides a hazard the runtime itself flags.

cases

Every verdict, public and linked

Six fronts so far — from an upstream reference client to payment SDKs carrying real money. Each card links back to sealed records and public threads; nothing on this page exists outside the registry.

x402-foundation

x402 reference client + PR #3920 core · @x402/core 2.28.0

A11 closedA10 open

Four review rounds; one fix verified closed, one still open at PR head.

  • Origin finding (#0000): the reference client signs any server-declared 402 amount.
  • Round 2–4 engagement on issue #3866 with control batteries and delta probes.
  • Cycle-2 at PR head b1a4d8ce: non-atomic amount rejection verified closed; binding race (signer ×2) reproduces.
seals#0000#0003#0004#0008
Issue #3866 PR #3920

Upstream engagement · 5 rounds

BlockRunAI

@blockrun/nano-client · 0.8.0 (npm exact pin)

3 FAIL3 HOLD

A signature over the attacker's payTo, ×50 with no cap, and a README claim its own advice breaks.

  • F1: shape-only payTo check signs authorization over attacker-controlled recipient.
  • F2: no default spend cap — ×50 of quoted price signed on default config.
  • F3: lost-response dead-end; the README's own recovery advice routes into a double charge.
  • Earlier report #15 closed as not planned; the sealed battery answers the closure premise.
seals#0006
Issue #18 Closure #15

Public · awaiting vendor

Nevermined

@nevermined-io/payments · 1.13.0 (npm exact pin)

7 FAIL1 DX3 HOLD

Seven failures in credit and settlement handling — including a double charge on retry.

  • F1: lost-response retry double-charges on the default token; no idempotency key, no settlement query.
  • F2: single-use guard is per-process and reactive — concurrent race delivers a free service.
  • F3: any settle failure except one special case delivers the resource unpaid — with a receipt.
  • The class was already flagged in the vendor's own tracker (#374/#367, open since June); the battery makes it sealed and reproducible.
seals#0007
nevermined-io/payments

Sealed · publication ready

Inngest

inngest + @inngest/test · 4.26.0 / 1.0.0 (npm dist)

0 FAIL2 DX

Thirteen probes, zero failures: behavior matches the vendor's own disclosure on every probe.

  • Commit-then-crash window verified live against the documented A3 model, including supervisor-restart variant.
  • Event-ID dedup holds under a 12-way concurrent race; prescribed mitigations verified effective.
  • Two developer-experience findings in the vendor's test engine; triaged upstream (Linear EXE-2315).
  • Lab close-out with seal #0005 published to the issue thread on closure.
seals#0005
Issue #5091

Closed · close-out published

tripl3tr3s

mcp-tool-idempotency · 4d665ae → v0.2.0

3 fixed10/10 verified

Reported privately, fixed publicly, verified independently — credited in SECURITY.md.

  • Three durability windows: lost response after commit, crash after commit, cross-instance race.
  • Coordinated disclosure before publication; maintainer shipped owner-gated claim leases.
  • Ten-window battery on v0.2.0: 10/10 PASS, suite 91/91 at tag.
seals#0009
mcp-tool-idempotency

Fixed · verified · credited

AgentMiddleware

amw-paypal-agent-demo · live demo (Block D)

REPRODUCED1 high finding

The lab's first live round — and the clean-machine reproduction that set the standard.

  • S10/S11 executed live against a real merchant agent demo with raw evidence captured.
  • Independent clean-machine reproduction from committed sources: verdicts identical.
  • The two-seal discipline (run + reproduction) applied ever since.
seals#0001#0002
amw-paypal-agent-demo

Live round · reproduced

engage

Run your code before your users do

Two working tracks, both ending in the same artifact: a sealed verdict anyone can verify. Engagements begin with written scope; nothing about an engagement changes a verdict.

A

Coordinated production round

You invited reproduction against production; we run it under written scope authorization: funded test wallet, agreed time window on 48 hours' notice, a no-debit/refund arrangement for probe transactions, full request capture shared with you. Deliverable: a sealed production-round report — either clearing the production endpoint, or documenting the exact boundary where defects become live money movement.

from $990 / round

B

Fix verification

Our batteries become your acceptance rig. You ship a patched artifact; we re-run the sealed suite against the published release and seal PASS/FAIL per finding. A clean seal is an independently checkable artifact you can cite to integrators and in your security narrative.

Per-finding sealed verdict

terms of engagement

  • Fixed per-round fee or a short retainer, agreed before any run.
  • Reports and seals remain the lab's to publish; vendor responses get equal space.
  • Production rounds require written scope, a funded test wallet, and a no-debit arrangement.
  • Private resolution is welcome — the seal is not a weapon, it is a receipt.
Start via GitHubOne-pager (PDF)argus.testlab@gmail.com

github.com/igor-vii/Argus-Agent-Test-Lab · issues open

  • §1Praise in public where it is earned — as loudly as failures get stated.
  • §2Break in public where it is broken — with line numbers, not adjectives.
  • §3Vendor responses get equal space alongside every published finding.
  • §4No vendor operates our stands; no engagement buys a verdict.