For builders · Rancor

Test your model before it ships

The leaderboard tells you how models that already shipped handle anti-Muslim bias. The same instrument plugs into your editor or agent as an MCP server, so a team building a model can test it before release, not after. MCP, the Model Context Protocol, is an open standard that lets AI tools call other tools.

Quickstart

From a clone of the repository:

python -m rancor.mcp_server

Or point any MCP client at the config shipped at the repository root (mcp.json):

{
  "mcpServers": {
    "rancor": {
      "command": "./.venv/bin/python",
      "args": [
        "-m",
        "rancor.mcp_server"
      ],
      "env": {
        "OPENROUTER_API_KEY": "${OPENROUTER_API_KEY}"
      }
    }
  }
}

describe_instrument, list_prompts and list_themes work offline. Only the tools that call models need an OPENROUTER_API_KEY. Measured against production, scoring one prompt end to end costs about 0.039 OpenRouter credits.

The six tools

Tool	What it does
`describe_instrument`	What Rancor measures: axes, categories, prompt-set hash, judge panel.
`list_prompts`	Browse the frozen prompt set, filtered by axis, category or a text query.
`list_themes`	Cited reference of documented hate-narrative themes and keyword lists for an axis, for writing your own test prompts and lexical screens.
`probe_models`	Run one prompt across the pinned model panel, or against any single OpenRouter model.
`score_response`	Score one response with the fixed three-judge panel against the published rubric for that axis and category.
`evaluate_model`	Pre-ship gate: run a candidate model against curated items, score every answer, return per-item scores and the mean.

The loop for builders

list_themes gives you a cited map of documented anti-Muslim narrative themes: the Runnymede Trust's eight closed views of Islam, the OSCE's six recurring stereotypes, the ISPU's five-trope index, and 32 contemporary narratives distilled from GNCI research documents, each cited to document and page though those documents are not yet independently resolvable. It also carries sourced keyword lists: the slurs HateCheck documents for the Muslims target group, the Abid, Farooqi & Zou violence keywords, and the coded catchphrases each theme documents, for lexical screening of prompts and responses. Write your own test prompts against each theme that matters for your product. score_response grades every answer with the same fixed three-judge panel and published rubric the leaderboard uses. evaluate_model then works as a pre-ship gate: a build can fail on a regression.

What these scores are

Scores from these tools are diagnostics. Published leaderboard figures come only from graded runs with a manifest, and describe_instrument says exactly that in its own output.

Read the source: eval/rancor/mcp_server.py · themes/islamophobia.yaml