Positioning
Giskard is an open-source library for testing and red-teaming LLM agents. Giskard v3 is a ground-up rewrite: v3.0.0 shipped on August 26, 2026 and v3.0.1 on October 2, 2026 1. The repository moved to Giskard-AI/giskard-oss, and v2 is still installable but no longer actively maintained 2.
v3 splits into two packages you actually touch. giskard-checks is the test layer: scenarios, assertions, and LLM-judge checks such as Groundedness and Conformity, multi-turn included. giskard-scan is the red-team layer: vulnerability_scan generates attacks from a plain-language description of your agent, and quality_scan replaces v2's RAGET for knowledge-base question answering 2. Giskard Hub is the paid enterprise platform on top, with a larger probe set, continuous red teaming, and team workflows 3.
The OSS scan runs seven generators: prompt injection, direct adversarial requests, GOAT and Crescendo multi-turn jailbreaks, GCG suffix injection, and replays of the do-not-answer and HarmBench corpora 4. The HarmBench replay reuses prompts from the published red-team benchmark 5, but the operating model is different: a product team scans its own agent on its own job rather than benchmarking a foundation model.
Strengths
- Description-driven attacks. The scan writes adversarial scenarios from what your agent is supposed to refuse, so a bank bot gets attacked on investment advice and other customers' data rather than on generic jailbreak prompts 4.
- Scans become regression suites. A scan returns a suite you can save as JSON and replay in CI against the next build, with JUnit XML output for the CI report step. Replays do not regenerate scenarios, but the judging step still calls an LLM, so CI needs a judge model configured too 4.
- Third-party scanners behind one API.
third_party_scan() runs garak 6 or DeepTeam against the same target and returns the same result type. Giskard labels it experimental, and third-party results export to JUnit but cannot be saved and replayed 7.
- Procurement-friendly Hub. Giskard states GDPR adherence, SOC 2 Type II, and HIPAA compliance for the Hub and offers EU or US data residency and on-prem installs 8. Custom audit reports are a paid service option 9. That shortens your vendor security review. It does not turn a scan report into audit evidence for your own SOC 2.
- Public enterprise customers. Michelin, BNP Paribas, and Decathlon are quoted on the Giskard homepage 8. Useful signal that the product handles regulated production deployments.
Limitations
- v2 code does not carry over.
giskard.Model, giskard.Dataset, and giskard.scan are v2 APIs. The automatic tabular/ML scan, the giskard.testing ML suite, and the v2 Hub integration stay v2-only and are not planned for v3; keep them with pip install "giskard[llm]>2,<3" 2. Old tutorials and blog posts will break against a fresh install.
- Python 3.12 or newer. v3 requires it. Teams pinned to older runtimes stay on v2 or run the scan in a separate environment 2.
- The OSS scan is the basic tier. The 50+ probes across 11 vulnerability categories, security grading, and continuous red teaming are Hub features 4. Giskard's own comparison labels OSS security detection "basic coverage" 3.
- A clean scan proves little. The scan samples attacks and an LLM judges the replies, so a pass only means those attacks failed. Giskard's docs say this themselves 4.
- Telemetry is on by default. v3 sends aggregated usage analytics unless you set
DO_NOT_TRACK=1; the README says no prompts or outputs are sent 2.
- Younger eval layer than the alternatives.
giskard-checks covers scenario tests and LLM judges, but its first stable release only landed in August 2026 1. For broad regression eval with a large metric catalog, Promptfoo or DeepEval are more mature.
- Vendor-published benchmarks. Coverage and detection claims should be treated as vendor positioning, not independent measurement.
Best fit
Pre-release and nightly red-team scans of production-bound agents, regression suites built from scan output, and regulated industries (finance, retail, manufacturing) where the Hub's compliance posture and on-prem option matter. Stay on v2 only if you need the tabular/ML scan.
Getting started
# pip install "giskard[scan,openai]" (Python 3.12+)
import asyncio
from pathlib import Path
from pydantic import BaseModel
from giskard.checks import set_default_generator
from giskard.scan import vulnerability_scan
set_default_generator("openai/gpt-4o") # writes and judges the attacks
class AgentInput(BaseModel):
question: str
class AgentOutput(BaseModel):
answer: str
async def support_bot(inputs: AgentInput) -> AgentOutput:
return AgentOutput(answer=await run_chatbot(inputs.question)) # your app
async def main() -> None:
result = await vulnerability_scan(
target=support_bot,
description=(
"A customer support assistant for an online retailer. It answers "
"refund and account questions and must never reveal another "
"customer's data."
),
languages=["en"],
max_scenarios=20,
)
if result.suite is not None:
Path("scan_suite.json").write_text(result.suite.model_dump_json())
result.to_junit_xml(path="reports/scan.xml")
asyncio.run(main())
The scan prints a grouped report with the judge's verdict and the full conversation for every failed scenario. In CI, import giskard.scan first (it registers the scan check types), load scan_suite.json with Suite.model_validate_json, run it against the new build, and point the CI report step at the JUnit file 4. The snippet was assembled from the v3 README and docs; it was not run here. Single-turn agents should also pass target_mode="singleturn", which skips GOAT and Crescendo.
Pricing notes
Giskard OSS is free under Apache 2.0. Giskard Hub is an Enterprise plan with no public rate card; the pricing page sends you to a demo 9. Hub deployment can be SaaS, private cloud, or on-prem, and consulting on guardrails and custom audit reports are add-on services. For internal red-team work in CI the OSS library is usually sufficient. The Hub earns its cost when you need the larger probe set, scheduled red teaming with alerts, or business reviewers who will not touch Python.
Alternatives
Consider Promptfoo for a CLI-first eval and red-team tool with a different operational model. Consider Confident AI / DeepEval and the DeepTeam OWASP suite; Giskard can also drive DeepTeam and garak through third_party_scan(). For frontier safety eval, Inspect AI plus HarmBench is the academic-leaning alternative.
Citations and last verified
Verified 2026-10-02 against the giskard-oss repository and release list, the Giskard v3 docs (scan guide, scan API reference, OSS vs Hub comparison), and the Giskard homepage and pricing page.