Positioning
Promptfoo is a CLI-first eval and red-team tool that ships YAML test suites runnable against any provider (OpenAI, Anthropic, Google, Bedrock, Azure, local). The product covers offline evals (model-graded, deterministic, classifier-based, embedding similarity, structured-output assertions) and a separate red-team mode that generates adversarial inputs across OWASP-style categories. OpenAI announced the acquisition on March 9, 2026 1, and the repository README now says the project is part of OpenAI and remains open source and MIT licensed 2. The homepage still reports adoption at 156 of the Fortune 500 (as of October 2026) 3.
The format that works best is keeping the YAML suites in your repo, running Promptfoo in CI, and gating PRs on score deltas.
Strengths
- CLI-first ergonomics.
promptfoo eval runs a YAML suite, prints a table, and returns a non-zero exit code on regressions. The CI integration is as simple as a GitHub Actions step.
- Provider-agnostic. OpenAI, Anthropic, Google, Bedrock, Azure, local Llama via Ollama, custom HTTP endpoints, and more all plug in via a small provider config.
- First-class red-team mode.
promptfoo redteam generates adversarial prompts and grades model responses against safety categories. The red-team suite covers prompt injection, jailbreaks, harmful content, and PII leakage 4.
- PR code scanning. A separate GitHub Action scans pull requests for LLM-specific vulnerabilities such as prompt injection, PII exposure, and excessive agency, and posts findings as review comments. Scans run on Promptfoo Cloud by default; keeping your code on your own infrastructure needs the Enterprise On-Prem tier 5.
Limitations
- YAML-heavy. Complex evals with shared fixtures and multi-step pipelines can grow YAML files past the ergonomic point. A pytest-style framework (DeepEval) handles that case more cleanly.
- Limited observability. Promptfoo focuses on offline eval and red-team; it is not a production-observability platform. It does accept OpenTelemetry spans during an eval run, and since 0.122.1 (August 2026) it can also pull traces from Grafana Tempo, Langfuse, or Braintrust 2. Trace assertions such as
trajectory:tool-used, trajectory:tool-sequence, and trace-span-count then grade agent behavior in CI 6. Pair with Langfuse, Phoenix, or LangSmith for production tracing.
- RAG assertions passed silently before 0.123.1. Until September 2026,
answer-relevance, context-faithfulness, context-recall, and context-relevance defaulted to a threshold of 0, so they passed on any score unless you set one. They now default to 0.5. Set thresholds explicitly, and do not trust green RAG results from older runs that omitted them 2.
- OpenAI ownership. The MIT license is preserved, but the project's strategic direction is now under OpenAI. Treat that as a useful signal rather than a problem; reasonable people disagree.
Best fit
CI regression suites for prompt and model changes, structured red-team programs that run nightly, side-by-side prompt comparison across providers, and teams that prefer keeping eval logic in repo-versioned YAML rather than a vendor UI.
Teams still on OpenAI's hosted Evals platform have a deadline: existing evals go read-only on October 31, 2026 and the platform is scheduled to shut down on November 30, 2026 7. OpenAI's own migration guide moves those evals to Promptfoo, with graders becoming assertions and metrics in a config file 8.
Getting started
prompts:
- "You are a helpful assistant. {{question}}"
providers:
- openai:gpt-4o-mini
- anthropic:messages:claude-haiku-4-5
tests:
- vars:
question: "Is the lentil bowl vegan?"
assert:
- type: contains
value: "yes"
- type: llm-rubric
value: "Answer is concise and accurate."
- vars:
question: "Refund policy?"
assert:
- type: similar
value: "30-day refund window"
threshold: 0.7
Run with promptfoo eval. The CLI prints a pass-rate table per provider and writes a JSON report. Use promptfoo redteam for the adversarial mode.
Two things changed since earlier versions of this example. Anthropic provider IDs take the anthropic:messages: prefix in the current docs, and Claude 3.5 Haiku was retired on the Claude API on February 19, 2026 with Haiku 4.5 as the listed replacement 9. The CLI also needs Node.js 22.22 or newer: 0.122.0 (August 2026) dropped Node 20 support 2.
Pricing notes
Promptfoo OSS is free under MIT. The pricing page lists three tiers as of October 2026: Community (free, open source), Enterprise (SSO, centralized dashboards, managed cloud deployment, SLA support), and On-Premise, with the last two custom-priced and sales-led 10. The OSS CLI is genuinely standalone for CI regression suites: provider API costs are the only bill.
Red teaming is where the free tier has a ceiling. Since 0.120.27 (March 2026) the CLI counts red-team probes per calendar month for anyone not logged into Promptfoo Cloud. Every turn of a multi-turn attack counts as a probe, and pointing attack generation at your own model does not exempt you 2. The pricing page lists 10k probes per month for Community 10, while the CLI source sets the cap at 100,000. The two disagree, so check your own usage before building a nightly red-team program on the free tier.
Alternatives
Consider DeepEval for pytest-style ergonomics. Consider Inspect AI for sandboxed and agentic evals. Consider Giskard for AI-specific red-team and SOC 2-compatible reporting. For hosted dashboards, pair with Langfuse, Arize Phoenix, or Braintrust.
Citations and last verified
Verified 2026-10-02 against the Promptfoo homepage, pricing page, tracing, code-scanning, red-team and Anthropic provider docs, the OpenAI acquisition post, the GitHub repository, release notes and probe-limit source, the npm registry, the Anthropic model deprecations page, and OpenAI's Evals deprecation notice and migration guide.