Promptfoo

Testing LLM applications the way software gets tested is the point of promptfoo, an open-source CLI and library for evaluation and red-teaming released under the MIT license. Developers define test cases declaratively and run side-by-side comparisons of prompts, models, and RAG pipelines across providers including OpenAI, Anthropic, Azure, Bedrock, and local models through Ollama, with automated assertions scoring outputs against expected behavior. Beyond quality evals, it performs adversarial red-teaming and vulnerability scanning that probe applications for jailbreaks, prompt injection, and data leakage, and it integrates into CI/CD so regressions and security findings surface in pull requests. Evaluations run locally, so prompts are not sent to any third-party service, and caching, live reload, and shareable web reports speed up iteration. It installs through npm, Homebrew, or pip and runs with no setup via npx. The project reports adoption by teams whose LLM applications collectively serve millions of end users.