Creating Acceptance Criteria for AI Coding Agents: Otherwise You Won't Trust Their Code Claims

Right now, when it comes to AI writing code, just getting the code to run is only the first step. The real trap is that when an agent says "I'm done," you have no idea what checks it actually ran. It says tests passed—did they all pass, or did it skip half? It says linting is clean—did it not run at all, or were there really no errors? I've had enough of this guessing game.

ProofRun solves exactly this problem. It doesn't judge whether your code is correct; it acts more like a notary: cryptographically proving which checks were definitely run, on which version of the code, and what the results were. It's essentially an acceptance receipt—whether it ran or not, and how it ran, can't be denied.

I tried it from scratch and wrote down the steps. You don't need to understand cryptography, nor deep technical stuff; you just need to know a bit of git and be able to use the terminal.

First, let's talk about the environment. I used macOS; Linux should be similar, and Windows works with WSL too. Things you need installed: Node.js 18+ (to run ProofRun), git (to check change logs after the agent finishes), and an AI coding agent, such as Claude Code or whatever else you have on hand.

Step 1: Install ProofRun into your project. Open a terminal in the project root and run:

After installation, you'll see new items in node_modules and a new dependency line in package.json. This step usually goes smoothly; if it fails, it's likely because your Node version is too old. Check with node -v, upgrade to 18+, and try again.

Step 2: Create the configuration file. In the project root, create a file named .proofrun.json and write down the checks you want the agent to run before finishing work. Here's what I wrote:

This file tells ProofRun: whenever the agent finishes a task, it must submit the raw output for these three things. type-check is type checking, letting the TypeScript compiler catch any type errors. lint is code style checking for consistent formatting. test runs the test cases.

Step 3: Have ProofRun generate a receipt. Run this command:

Normally, you'll see three lines of results printed in the terminal, each followed by a string like sha256:xxxxx. That's a checksum, think of it as the "fingerprint" of that output content. If even one character differs, the fingerprint changes—it can't be forged. At this point, a new file appears in the project, called proofrun-receipt.json in my case, storing those fingerprints and check names.

Pitfall warning: On the first run, I messed up the path for the lint command in my config, resulting in ESLint couldn't find the config file. This error was due to my incorrect command; ProofRun just reported it faithfully. Ensure these commands work individually in the project root before feeding them to ProofRun. Otherwise, the agent will get stuck retrying because it can't get the "acceptance receipt," wasting quota.

Step 4: Make the AI agent use this receipt. This step is key. Add a file named AGENTS.md in the project (or whatever name your agent reads; Claude Code reads this) with a rule:

Meaning: After finishing work, you MUST run npx proofrun verify, obtain the receipt, and include the receipt file path in the final report. This way, the agent won't make empty claims. Previously, agents would sincerely write "All tests passed," but when I manually ran them, 14 cases failed. They were lazy and skipped time-consuming tests by default. With this rule, they at least have to present the files where proofs were successfully run.

Step 5: Also the most useful usage I found: have the agent commit changes to git after finishing, including the receipt file. This way, looking back later, you can directly see which checks were run against specific commits. Run:

Then in git show or the PR page, you'll see every commit comes with a receipt, containing the raw output and checksums for each check. This is far better than verbal "I tested it," and saves time during code reviews asking "Did you check this?"

Pitfall 2: If the agent lacks permission to run npx proofrun, e.g., in a container environment or missing dependencies, the receipt won't generate. My solution is adding npx proofrun to the whitelist in the agent's permission config so it can execute. Config methods vary by agent; look for settings related to "allowed commands."

Pitfall 3: Agents might alter the receipt file content or generate a fake one. This isn't fully preventable since agents have model capabilities. However, this tool targets "accidental omissions," not "malicious forgery." Preventing the latter requires heavier audit solutions. For my lab scenario, proving "the agent ran it and outputs match" is sufficient.

I used it for about a week. Subjectively: hard to say if code quality improved, but the anxiety of "not knowing if it actually tested" has decreased. Before, I trusted its word upon completion; now I check the receipt first to confirm all three checks genuinely ran before deciding on manual review.

Let me clarify: ProofRun doesn't guarantee your code logic is correct. It proves "these checks were indeed run on this version of the code, with these results." Whether the tests themselves are well-written is another matter.

Next steps after learning this: integrate proofrun verify into CI, forcing receipt generation on every commit, making verification a hard gate rather than voluntary. I'm considering doing this and will write another post once it's smooth.

---

📌 Compiled from Hacker News. Original: https://github.com/yebiguo/ProofRun

Copyright belongs to the original author. This is a compilation and independent analysis based on public reports.