iFixAi grades AI agents on business KPIs, not token counts

The pitch on the iFixAi repository is blunt: existing eval, red-teaming, and observability tools grade an agent on technical competence. Token efficiency. Latency. Prompt-injection resistance. What they do not answer, the README argues, is whether the agent is doing the job it is supposed to do against business KPIs and organizational structure. iFixAi claims to answer that in under 120 seconds.

That is the part worth arguing with. The tooling is real and unusually well-specified for a young repo. The framing is doing more work than the code, and it deserves scrutiny before anyone puts an iFixAi scorecard in a board deck.

What the tool actually does

Strip the marketing and iFixAi is a test harness. You install it with pip install "ifixai[<provider>]", run ifixai setup for an arrow-key wizard, then ifixai run. It ships a CLI, an explicit-flags mode for CI, and a plugin for Claude Code and Codex, plus a /ifixai-skill scaffold that drops into Cursor, VS Code, Windsurf, Cline, Continue, Gemini, and Zed.

The default run executes ten inspections. Three are mandatory minimums: B01 (a role only uses tools it is allowed to use), B08 (the agent cannot be talked into more access than the user has), and P01 (no destructive action for someone not authorized). A run missing those has its overall score withheld. The other seven cover prompt injection, policy-breaking requests, cross-user data leakage, uncertainty reporting, failure honesty, task adherence, and instruction-following on format and length. --full-run expands to all 60, and the suite ladder runs smoke (3), strategic (8), essential (10, the default), core (32), extended (28), and all (60).

The cost math is disclosed, which is rarer than it should be. A default run makes roughly 850 calls to the agent under test and 820 judge calls. A full run makes about 5,500 and 4,700. On OpenRouter list prices, a full suite with Claude Sonnet 4.6 as the single judge lands around $12–18. Two judges, Gemini 2.5 Pro plus GPT-5.4-mini, come in at roughly $10–14 combined.

The cross-vendor judge rule is the real idea

Here is the design decision that separates iFixAi from the pile of agent-eval repos. A grade is only “citable” when a second, independent provider graded the agent. The system under test (SUT) is selected with --provider and --api-key, and that key is always passed explicitly, never read from the environment. The judge is auto-paired from a different provider whose key is present in the environment. The SUT’s own vendor is excluded, so a model never grades itself.

Without a second key you can still run with --eval-mode self, but the output is flagged as self-judged and explicitly not citable. Ties break conservatively: fail beats partial beats pass.

This is the correct instinct, and it is the one most eval vendors avoid because it complicates the sale. Self-grading is the default failure mode of the entire agent-evaluation market. A lab ships a model, ships a benchmark, and reports the score its own model produced. iFixAi makes that structurally impossible in a citable run by requiring two keys from two vendors. It is a small enforcement mechanism with an outsized effect on what the resulting number means.

Where the business-KPI claim strains

The README’s central claim is that iFixAi grades against business KPIs and organizational structure. Look at what the ten default inspections actually check. B01 and P01 read the agent’s tool-authorization layer. The rest are behavioral: does it resist injection, does it admit uncertainty, does it avoid leaking one user’s data to another.

Those are governance and safety properties. They are not KPIs. A KPI is “resolved 80% of support tickets without escalation” or “cut refund processing time by half.” Nothing in the default suite measures whether the agent moved a business metric. The “organizational structure” framing maps onto role-based tool permissions, which is an access-control question, not an org-design one.

That gap is not fatal, and the repo is honest about the mechanics elsewhere. But the marketing line oversells. What iFixAi actually delivers is an operational-assurance scorecard: a structured pass/fail on whether the agent stays inside its lane. That is genuinely useful and genuinely underserved. Calling it KPI-based invites a procurement fight the tool cannot currently win.

The cost-forecasting detail worth stealing

One small feature deserves to be copied across the industry: the plugin names the cost before anything is billed, and the skill shows a free --dry-run before the run proceeds. Agent evaluation has a nasty habit of surprising teams with API bills, because a suite generates far more probes than its headline test count suggests. iFixAi says this out loud. The suite “generates far more probes than its 50-test count,” which is why the cost figure is stable across fixtures.

Self-grading is the default failure mode of the agent-evaluation market. iFixAi makes it structurally impossible in a citable run.

What it means for builders

Three things are true at once. The tool is well-built, cheap to try, and solves a real problem: teams shipping agents into production have almost no way to prove the agent respects role boundaries, and iFixAi gives them a repeatable 10-inspection smoke test for roughly the cost of a couple of full-suite runs.

The framing is ahead of the substance. “Business KPIs and organizational structure” is a promise the default suite does not keep, and buyers who take it literally will be disappointed. The honest description is access-control and behavioral assurance.

The cross-vendor judge rule is the piece the rest of the market should adopt. If a score can be produced by having a model grade itself, it is marketing, not measurement. iFixAi’s insistence on two keys from two vendors is the kind of constraint that makes a number worth citing. The rest of the eval industry has spent two years shipping benchmarks that fail this test. A small open-source repo just made the rule enforceable in a config file.

Watch whether any major lab adopts the two-key standard voluntarily. That, not the 120-second claim, is the number that would matter.