Conkurrence

Created By

AlligatorC0der3 months ago

Conkurrence measures whether multiple AI models produce consistent outputs on your evaluation tasks. It tells you which items your AI agrees on and which need human review — using Fleiss' κ, Kendall's W, and bootstrap confidence intervals, the same psychometric methods trusted in clinical research.

# ai-evaluation

# golden-dataset

Overview Content Tools Comments

Overview

ConKurrence

One command. Find out if your AI agrees with itself.

ConKurrence is a statistically validated consensus measurement toolkit for AI evaluation pipelines. It uses multiple AI models as independent raters, measures inter-rater reliability with Fleiss' kappa and bootstrap confidence intervals, and routes contested items to human experts.

Install

npm install -g conkurrence

MCP Server

Use ConKurrence as an MCP server in Claude Desktop or any MCP-compatible client:

npx conkurrence mcp

Claude Desktop Configuration

Add to your claude_desktop_config.json:

{
  "mcpServers": {
    "conkurrence": {
      "command": "npx",
      "args": ["-y", "conkurrence", "mcp"]
    }
  }
}

Claude Code Plugin

/plugin marketplace add AlligatorC0der/conkurrence

Features

Multi-model evaluation — Run your schema against Bedrock, OpenAI, and Gemini models simultaneously
Statistical rigor — Fleiss' kappa with bootstrap confidence intervals, Kendall's W for validity
Self-consistency mode — No API keys needed; uses the host model via MCP Sampling
Schema suggestion — AI-powered schema design from your data
Trend tracking — Compare runs over time, detect agreement degradation
Cost estimation — Know the cost before running

MCP Tools

Tool	Description
`conkurrence_run`	Execute an evaluation across multiple AI raters
`conkurrence_report`	Generate a detailed markdown report
`conkurrence_compare`	Side-by-side comparison of two runs
`conkurrence_trend`	Track agreement over multiple runs
`conkurrence_suggest`	AI-powered schema suggestion from your data
`conkurrence_validate_schema`	Validate a schema before running
`conkurrence_estimate`	Estimate cost and token usage

License

BUSL-1.1 — Business Source License 1.1

Try in Playground

Server Config

{
  "mcpServers": {
    "conkurrence": {
      "command": "npx",
      "args": [
        "-y",
        "conkurrence",
        "mcp"
      ]
    }
  }
}

Project Info

Created At

3 months ago

Updated At

3 months ago

Author Name

AlligatorC0der

Star

Language

License

Recommend Servers

10 days ago

One API for Search, Crawling, and Sitemaps

a year ago

302_sandbox_mcp

@302ai

Create a remote sandbox that can execute code/run commands/upload and download files. 创建远程沙盒，可以执行代码/运行命令/上传下载文件

a year ago

EdgeOne Pages MCP

@TencentEdgeOne

An MCP service designed for deploying HTML content to EdgeOne Pages and obtaining an accessible public URL.

TypeScript

a year ago

Algeria Payments Mcp

@junter1989k-ai

9 days ago

Ivory Coast Payments Mcp

@junter1989k-ai

9 days ago

Serper MCP Server

@garymengcom

A Serper MCP Server

Python

a year ago

Mcp Server Chatsum

@chatmcp

summarize chat message

typescript

a year ago

Sequential Thinking

@modelcontextprotocol

An MCP server implementation that provides a tool for dynamic and reflective problem-solving through a structured thinking process.

a year ago

9 days ago

A Model Context Protocol (MCP) server for image generation and manipulation using fal.ai's Stable Diffusion model.

a year ago

Kazakhstan Payments Mcp

@junter1989k-ai

9 days ago

EverArt

@modelcontextprotocol

AI image generation using various models

a year ago

Google Maps

@modelcontextprotocol

Location services, directions, and place details

a year ago

MiniMax MCP

@MiniMax-AI

Official MiniMax Model Context Protocol (MCP) server that enables interaction with powerful Text to Speech, image generation and video generation APIs.

Python

a year ago

Slack

@modelcontextprotocol

Channel management and messaging capabilities

a year ago

Time

@modelcontextprotocol

A Model Context Protocol server that provides time and timezone conversion capabilities. This server enables LLMs to get current time information and perform timezone conversions using IANA timezone names, with automatic system timezone detection.

5 months ago

Nepal Payments Mcp

@junter1989k-ai

9 days ago

Guatemala Payments Mcp

@junter1989k-ai

9 days ago

Firecrawl Mcp Server

@mendableai

Official Firecrawl MCP Server - Adds powerful web scraping to Cursor, Claude and any other LLM clients.

JavaScript

a year ago

9 days ago

MCP Advisor & Installation - Use the right MCP server for your needs

TypeScript

a year ago

PostgreSQL

@modelcontextprotocol

Read-only database access with schema inspection

a year ago

Uae Payments Mcp

@junter1989k-ai

10 days ago

Australia Payments Mcp

10 days ago

10 days ago

@modelcontextprotocol

A Model Context Protocol server that provides access to Redis databases. This server enables LLMs to interact with Redis key-value stores through a set of standardized tools.

a year ago

9 days ago

9 days ago

9 days ago