{"format":"ble-official-view-v1","schema_version":1,"strings":["Claude 4.5 Opus (high)","Anthropic","mini-SWE-agent 2.0.0; 1 attempt","Gemini 3 Flash (high)","Google DeepMind","MiniMax M2.5 (high)","Minimax","Claude 4.6 Opus","Claude 4.5 Opus (20251101) (medium)","mini-SWE-agent 1.16.0; 1 attempt","Gemini 3 Pro Preview (2025-11-18)","mini-SWE-agent 1.15.0; 1 attempt","GLM 5 (high)","Z-AI","GPT 5.2 (high)","OpenAI","GPT 5.2 Codex","GPT 5.2 (2025-12-11) (high)","mini-SWE-agent 1.17.2; 1 attempt","Claude 4.5 Sonnet (high)","Kimi K2.5 (high)","Moonshot AI","Claude 4.5 Sonnet (20250929)","mini-SWE-agent 1.13.3; 1 attempt","DeepSeek V3.2 (high)","DeepSeek","Gemini 3 Pro","GPT 5.2 (2025-12-11)","Claude 4 Opus (20250514)","mini-SWE-agent 1.0.0; 1 attempt","Claude 4.5 Haiku (high)","GPT 5.1 (2025-11-13) (medium)","GPT 5.1 Codex (medium)","GPT 5 (2025-08-07) (medium)","mini-SWE-agent 1.7.0; 1 attempt","Claude 4 Sonnet (20250514)","Kimi K2 Thinking","MiniMax M2","mini-SWE-agent 1.17.0; 1 attempt","DeepSeek V3.2 Reasoner","deepseek","mini-SWE-agent 1.17.1; 1 attempt","GPT 5 mini (2025-08-07) (medium)","o3 (2025-04-16)","Devstral Small (2512)","mistral","GPT 5 mini","GLM 4.6 (T=1)","Z.ai","Qwen3-Coder 480B/A35B Instruct","Qwen","GLM 4.5 (2025-08-22)","mini-SWE-agent 1.9.1; 1 attempt","Devstral (2512)","Gemini 2.5 Pro (2025-05-06)","Claude 3.7 Sonnet (20250219)","mini-SWE-agent 0.0.0; 1 attempt","o4-mini (2025-04-16)","Kimi K2 Instruct","GPT 4.1 (2025-04-14)","GPT 5 nano (2025-08-07) (medium)","Gemini 2.5 Flash (2025-04-17)","gpt-oss-120b","GPT 4.1 mini (2025-04-14)","GPT 4o (2024-11-20)","Llama 4 Maverick Instruct","Meta","Gemini 2.0 Flash","Llama 4 Scout Instruct","Qwen2.5-Coder 32B Instruct","GPT-6 Astra (Max)","CoT","GPT-6.1 Sol (Max)","Claude Opus 5.5 (High)","GPT-6 Astra (XHigh)","Claude Opus 5.5 (XHigh)","GPT-5.6 Sol (Max)","GPT-6 Astra (High)","GPT-6 Astra (Medium)","Claude Opus 5.5 (Max)","GPT-6.1 Sol (High)","GPT-6.1 Sol (XHigh)","Gemini 4 Argon (High)","Google","Claude Opus 5 (Max)","Claude Fable 5.1 (Max)","Claude Fable 5.1 (XHigh)","GPT-5.6 Sol (XHigh)","GPT-6 Sol (Max)","Claude Fable 5 (Max)","Gemini 3.8 Flash (High)","Claude Fable 5.1 (High)","Claude Fable 5 (XHigh)","Claude Opus 5 (High)","Claude Fable 5 (High)","Claude Opus 5.5 (Medium)","Gemini 4 Argon (Medium)","GPT-6.1 Sol (Medium)","Claude Fable 5.1 (Medium)","GPT-5.6 Sol (High)","GPT-6 Astra (Low)","GPT-5.5 (XHigh)","Gemini 3.7 Flash (High)","GPT-5.5 Pro (High)","Gemini 3 Deep Think (2/26)","GPT-5.5 Pro (XHigh)","GPT-5.6 Terra (Max)","Claude Sonnet 5.5 (XHigh)","GPT-5.4 Pro (XHigh)","GPT-5.5 (High)","Gemini 3.8 Flash (Medium)","Claude Fable 5 (Medium)","Claude Fable 5.1 (Low)","DeepSeek V4.1 Flash 1017 (Max)","GPT-6 Sol (XHigh)","Gemini 3.8 Flash (Low)","Gemini 3.1 Pro (Preview)","Dots3-Note Preview (Max)","Dots Studio","Claude Fable 5 (Low)","GPT-6.1 Sol (Low)","Claude 4.7 (Max)","GPT-5.6 Terra (XHigh)","GPT-5.4 (XHigh)","DeepSeek V4.1 Flash 1017 (Low)","DeepSeek V4.1 Flash (Max)","GPT-5.2 (Refine.)","Refinement; Johan Land","Claude Opus 4.8 (High)","Gemini 3.5 Flash (High)","Claude Opus 4.8 (Medium)","GPT-5.5 (Medium)","Claude Opus 5.5 (Low)","Claude Opus 4.6 (120K, High)","GPT-6 Sol (High)","Claude Opus 4.6 (120K, Max)","DeepSeek V4.1 Flash 1017 (High)","Claude 4.7 (High)","Claude 4.7 (Medium)","DeepSeek V4.1 Flash (High)","GPT-5.4 (High)","Claude Sonnet 5.5 (High)","Grok 4.6 (XHigh)","xAI","GPT-5.6 Sol (Medium)","GPT-5.6 Terra (High)","Claude Opus 4.6 (120K, Medium)","GLM-5.3-Flash (Max)","Grok 4.20 (Reasoning)","Grok 4.6 (High)","Claude Opus 4.6 (120K, Low)","Gemini 3.7 Flash (Medium)","Claude Opus 4.8 (Low)","Claude 4.7 (Low)","DeepSeek V4 Flash 0731 (Max)","Grok 4.7 (XHigh)","Claude Haiku 5.5 (XHigh)","DeepSeek V4 Pro 0813 (Max)","Grok 4.6 (Medium)","DeepSeek V4.1 Flash (Low)","Claude Sonnet 4.6 (High)","Gemini 3.6 Flash (High)","Kimi K3 (Max)","DeepSeek V4 Pro 0813 (High)","GPT-5.6 Luna 2026-07-30 (Max)","GPT-6 Astra (None)","GPT-5.6 Luna (Max)","GPT-6 Luna (Max)","Grok 4.7 (Medium)","Grok 4.7 (High)","Claude Sonnet 4.6 (Max)","GPT-6 Sol (Medium)","Muse Spark Open (XHigh)","Meta AI","DeepSeek V4 Pro 0813 (Low)","DeepSeek V4 Flash 0731 (High)","GPT-5.4 (Medium)","Kimi K3 (High)","GPT-5.2 Pro (High)","Gemini 3 Pro (Refine.)","Refinement; Poetiq","Gemini 3.7 Flash (Low)","GPT-5.2 (XHigh)","Grok 4.5 (High)","Grok 4.5 (Medium)","Gemini 3.6 Flash (Medium)","GLM-5.3-Flash (High)","GPT-5.6 Luna (XHigh)","Muse Spark Open (High)","DeepSeek V4 Flash 0731 (Low)","Gemini 3 Deep Think (Preview) ²","GPT-5.6 Luna 2026-07-30 (XHigh)","GPT-5.2 (High)","GPT-5.6 Sol (Low)","Qwen3.8-27B (XHigh)","Alibaba","Gemini 4 Argon (Low)","GPT-6 Luna (XHigh)","Inkling Small (XHigh)","Thinking Machines","GPT-5.2 Pro (Medium)","Opus 4.5 (Thinking, 64K)","GPT-5.6 Terra (Medium)","Inkling","Gemini 3 Flash Preview (High)","GPT-5.6 Luna 2026-07-30 (High)","GPT-5.5 (Low)","Grok 4.5 (Low)","Inkling Small (High)","Claude Sonnet 5.5 (Medium)","GPT-6 Sol (Low)","GPT-6 Luna (High)","Claude Haiku 5.5 (High)","Gemini 3.6 Flash (Low)","Grok 4 (Refine.)","Refinement; J. Berman","GPT-5.6 Luna (High)","GPT-5.4 (Low)","GLM-5.3-Flash (Low)","Grok 4.6 (Low)","Muse Spark Open (Medium)","GPT-5.2 (Medium)","Claude Sonnet 5.5 (Low)","Beam 1.0 (XHigh)","Reflection","Opus 4.5 (Thinking, 16K)","Qwen3.8-27B (Low)","GLM-5.2","Beam 1.0 (High)","GPT-5.4 Mini (XHigh)","GPT-5.6 Terra (Low)","GPT-5 Pro","GPT-6 Luna (Medium)","GPT-5.1 (Thinking, High)","Grok 4.7 (Low)","Grok 4 (Thinking)","Opus 4.5 (Thinking, 8K)","Inkling Small (Medium)","Claude Sonnet 4.5 (Thinking 32K)","Qwen3.8-27B (Medium)","GPT-5.4 Mini (High)","Gemini 3 Flash Preview (Medium)","Kimi K3 (Low)","Muse Spark Open (Low)","Kimi K2.5","Gemini 3.5 Flash-Lite (High)","GPT-5 (High)","GPT-5.2 (Low)","Claude Haiku 5.5 (Medium)","Gemini 3.5 Flash (Minimal)","Claude Opus 4 (Thinking 16K)","Opus 4.5 (Thinking, None)","GPT-5 (Medium)","GPT-5.6 Luna 2026-07-30 (Medium)","GPT-5.6 Luna (Medium)","Claude Sonnet 4.5 (Thinking 16K)","Claude Sonnet 4.5 (Thinking 8K)","GPT-5.1 (Thinking, Medium)","o3 (High)","Beam 1.0 (Medium)","o4-mini (High)","Claude Sonnet 4 (Thinking 16K)","Claude Sonnet 4.5 (Thinking 1K)","GPT-5.4 Nano (XHigh)","Grok 4 (Fast Reasoning)","Gemini 3.5 Flash-Lite (Medium)","GPT-5.6 Luna (Low)","Inkling Small (Low)","GLM-5","Gemini 2.5 Pro (Thinking 32K)","Minimax M2.5","o3-Pro (High)","CoT + Synthesis","GPT-6 Luna (Low)","Claude Opus 4 (Thinking 8K)","GPT-5 Mini (High)","GPT-5.4 Mini (Medium)","Claude Haiku 4.5 (Thinking 32K)","Deepseek V3.2","Base LLM","GPT-5 Mini (Medium)","Gemini 2.5 Pro (Thinking 16K)","Inkling Small (Minimal)","Claude Sonnet 4.5","GPT-5.6 Luna 2026-07-30 (Low)","GPT-5.4 Nano (High)","Gemini 3 Flash Preview (Minimal)","o3-mini (High)","o3 (Medium)","Gemini 2.5 Pro (Thinking 8K)","Claude Haiku 4.5 (Thinking 16K)","Beam 1.0 (Low)","Claude Haiku 5.5 (Low)","Gemini 3.6 Flash (Minimal)","GPT-5 Nano (High)","Gemini 2.5 Flash (Preview) (Thinking 24K)","o4-mini (Medium)","Gemini 2.5 Flash (Preview) (Thinking 1K)","Claude Sonnet 4 (Thinking 8K)","Gemini 2.5 Flash (Preview) (Thinking 8K)","DeepSeek V4 Flash 0731 (None)","DeepSeek V4.1 Flash 1017 (None)","o3-mini (Medium)","o3-Pro (Low)","o3 (Low)","Gemini 2.5 Flash (Preview) (Thinking 16K)","Nemotron 3.5 Super VL (Medium)","NVIDIA","GPT-5 (Low)","GPT-5.1 (Thinking, Low)","GPT-5.4 Nano (Medium)","o3-Pro (Medium)","Gemini 2.5 Flash (Preview)","Claude Haiku 4.5 (Thinking 8K)","GPT-5 Mini (Minimal)","o4-mini (Low)","GPT-6 Sol (None)","GPT-5.4 Nano (Low)","Gemini 3.5 Flash-Lite (Low)","Nemotron 3.5 Super VL (High)","Qwen3.8-27B (None)","Deepseek R1","Claude Opus 4","Claude Sonnet 4","Codex Mini (Latest)","Claude Haiku 4.5","Claude Haiku 4.5 (Thinking 1K)","DeepSeek V4.1 Flash (None)","Gemini 3 Flash Preview (Low)","Qwen3-235b-a22b Instruct (25/07)","Deepseek R1 (05/28)","GPT-5.4 Mini (Low)","Claude 3.7 (8K)","GPT-5 Nano (Medium)","Claude Sonnet 4 (Thinking 1K)","DeepSeek V4 Pro 0813 (None)","GPT-5.6 Luna 2026-07-30 (None)","Gemini 3.5 Flash-Lite (Minimal)","Nemotron 3.5 Super VL (None)","GPT-5 Mini (Low)","GPT-5.2","o1-mini","GPT-4.5","Gemini 1.5 Pro","Claude 3.7 (16K)","GPT-4.1","GPT-5.1 (Thinking, None)","Grok 3 Mini (Low)","Claude 3.7 (1K)","Claude 3.7","Claude Opus 4 (Thinking 1K)","GPT-4.1-Mini","GPT-4.1-Nano","GPT-4o","GPT-4o-mini","GPT-5 (Minimal)","GPT-5 Nano (Low)","GPT-5 Nano (Minimal)","GPT-6 Luna (None)","Gemini 2.5 Pro (Thinking 1K)","Grok 3","Inkling Small (None)","Llama 4 Maverick","Llama 4 Scout","Magistral Medium","Mistral","Magistral Medium (Thinking)","Magistral Small","o3-mini (Low)","gpt-6-astra_max","","gpt-6.1-sol_max","claude-opus-5-5_max","gpt-6.1-terra_max","claude-fable-5-1_max","gpt-6-sol_max","gpt-5.6-sol_max","claude-sonnet-5-5_max","gpt-5.5-pro_xhigh","claude-fable-5_max","gemini-4-argon_high","gpt-5.6-terra_max","claude-opus-5_max","gpt-5.5_xhigh","gpt-5.4-pro-2026-03-05_xhigh","gpt-5.6-luna_max","muse-wonder-1.0_max","claude-opus-4-8_max","gpt-6-luna_max","gpt-5.4-2026-03-05_xhigh","claude-haiku-5-5_max","qwen3.8-max_xhigh","muse-spark-1.3_xhigh","muse-spark-1.3_max","gpt-5.2-pro-2025-12-11_xhigh","kimi-k3_max","Moonshot","gemini-3.7-flash_high","deepseek-v4.1-flash-1017_max","claude-opus-4-7_max","Muse-Spark-Open-816B-A42B_xhigh","glm-5.3_max","Z.ai (Zhipu AI)","gemini-3.8-flash_high","gpt-5.2-2025-12-11_xhigh","deepseek-v4.1-flash_max","claude-opus-4-6_max","grok-4.6_xhigh","claude-sonnet-5_max","qwen3.8-max-0902_xhigh","deepseek-v4-pro-0813_max","qwen3.7-max","gemini-3.5-flash_high","gemini-3.1-pro-preview","glm-5.2_max","gemini-3.6-flash_high","mistral-large-4-0_high","Mistral AI","deepseek-v4-flash-0731_max","grok-4.5_high","kimi-k2.6","glm-5.3-flash_max","gpt-5-pro-2025-10-06_high","gpt-5-2025-08-07_high","glm-5.2_low","kimi-k2.7-code","grok-4.7_xhigh","gemini-3-flash-preview","gpt-5.4-mini-2026-03-17_xhigh","gpt-5-mini-2025-08-07_high","inkling-small_xhigh","deepseek-v4-pro_max","gpt-5.4-nano-2026-03-17_high","grok-4.20-0309-reasoning","grok-4.3_high","glm-5.2_none","gpt-5.6-luna_low","gpt-5.6-luna_none","qwen3.6-plus","gpt-5-2025-08-07_low","glm-5.1","o4-mini-2025-04-16_high","qwen3.6-27b","claude-opus-4-5-20251101_32K","qwen3.7-plus_none","qwen3.6-27b_none","Inkling_xhigh","o3-2025-04-16_high","qwen3.6-plus_none","qwen3.5-397b-a17b_none","o3-2025-04-16_medium","qwen3.5-397b-a17b","o4-mini-2025-04-16_medium","gemini-3.1-flash-lite_high","gpt-5.5-instant","gemini-3.5-flash-lite_high","glm-5.1_none","gemini-2.5-pro","gpt-5.4-mini-2026-03-17_low","claude-sonnet-4-5-20250929_32K","gemini-3.1-flash-lite_low","qwen3.6-flash_none","gemini-3.1-flash-lite_minimal","gpt-5.4-nano-2026-03-17_low","qwen3.6-35b-a3b_none","gpt-5-nano-2025-08-07_high","o3-2025-04-16_low","qwen3.7-flash","qwen3.7-flash_none","qwen3-max-2025-09-23","o3-mini-2025-01-31_high","gpt-5-2025-08-07_minimal","gpt-5-mini-2025-08-07_low","qwen3.5-flash_none","qwen3.6-35b-a3b","gpt-5.4-mini-2026-03-17_none","qwen3.6-flash","o4-mini-2025-04-16_low","o1-2024-12-17_high","claude-opus-4-1-20250805_32K","o3-mini-2025-01-31_medium","o1-2024-12-17_medium","qwen3.5-flash","o1-2024-12-17_low","gpt-4.1-mini-2025-04-14","gpt-4.1-2025-04-14","gpt-5-mini-2025-08-07_minimal","gpt-5-nano-2025-08-07_low","gpt-5.4-nano-2026-03-17_none","o3-mini-2025-01-31_low","gpt-5-nano-2025-08-07_minimal","gpt-4-turbo-2024-04-09","gpt-4o-mini-2024-07-18","gpt-4o-2024-08-06","gpt-3.5-turbo-0125","gpt-6-astra_high","gpt-6-astra_xhigh","gpt-6-astra_medium","gpt-6-astra_low","gpt-6-astra_none","gpt-5.6-sol_promax","gdm-ai-co-mathematician","Gemini-3.1-Pro","Gemini-2.5-Pro-Exp-03-25","Claude-3.5-Sonnet (2024-10-22)","Gemini-2.0-Flash-exp","Claude-3.5-Sonnet (2024-06-20)","GPT-4o (2024-08-06)","GPT-oss-20B(medium)","Athene-V2-Chat (0-shot)","GPT-4o (2024-05-13)","Gemini-1.5-Pro-002","Athene-V2-Chat","QwQ-32B","Claude-3-Opus","Mistral-Large-Instruct-2411","Llama-3.3-70B-Instruct","Mistral-Large-Instruct-2407","DeepSeek-Chat-V2_5","Qwen2-72B-Chat","Gemini-1.5-Flash-002","magnum-72b-v1","GPT-4-Turbo","DeepSeek-Coder-V2-Instruct","Llama-3.1-70B-Instruct","Llama-3.1-Nemotron-70B-Instruct-HF","Yi-Lightning","01.AI","Claude-3-5-Haiku-20241022","Reflection-Llama-3.1-70B","Gemini-1.5-Flash","EXAONE-3.5-32B-Instruct","Yi-large","Claude-3-Sonnet","Gemma-2-27B-it","Mixtral-8x22B-Instruct-v0.1","Llama-3-70B-Instruct","Phi3-medium-4k","Qwen2-72B-32k","Deepseek-V2-Chat","Llama-3-70B","Qwen1.5-72B-Chat","Llama-3.1-70B","Yi-1.5-34B-Chat","Gemma-2-9B-it","Phi3-medium-128k","MAmmoTH2-8x7B-Plus","Qwen1.5-110B","Jamba-1.5-Large","Mistral-Small-Instruct-2409","GLM-4-9B-Chat","Z AI","GLM-4-9B","Phi-3.5-mini-instruct","Microsoft","Qwen2-7B-Instruct","EXAONE-3.5-7.8B-Instruct","Yi-1.5-9B-Chat","Phi3-mini-4k","Aya-Expanse-32B","Gemma-2-9B","Mistral-Nemo-Instruct-2407","Llama-3.1-8B-Instruct","Phi3-mini-128k","MAmmoTH2-8B-Plus","Mixtral-8x7B-Instruct-v0.1","Yi-34B","Claude-3-Haiku-20240307","Mathstral-7B-v0.1","DeepSeek-Coder-V2-Lite-Instruct","Granite-3.1-8B-Instruct","Mixtral-8x7B-v0.1","Llama-3-8B-Instruct","MAmmoTH2-7B-Plus","Qwen2-7B","Mistral-Nemo-Base-2407","WizardLM-2-8x22B","EXAONE-3.5-2.4B-Instruct","Yi-1.5-6B-Chat","Qwen1.5-14B-Chat","Ministral-8B-Instruct-2410","Staring-7B","c4ai-command-r-v01","Llama-2-70B","OpenChat-3.5-8B","InternMath-20B-Plus","Llama3-Smaug-8B","Llama-3.1-8B","Llama-3-8B","DeepseekMath-7B-Instruct","DeepSeek-Coder-V2-Lite-Base","Aya-Expanse-8B","Gemma-7B","InternMath-7B-Plus","Granite-3.1-8B-Base","Zephyr-7B-Beta","Granite-3.1-2B-Instruct","Granite-3.0-8B-Base","Mistral-7B-v0.1","Mistral-7B-Instruct-v0.2","Mistral-7B-v0.2","Qwen1.5-7B-Chat","Yi-6B-Chat","Neo-7B-Instruct","Yi-6B","Neo-7B","Mistral-7B-Instruct-v0.1","Granite-3.1-3B-A800M-Instruct","Llama-2-13B","Granite-3.1-2B-Base","Llemma-7B","Qwen2-1.5B-Instruct","Qwen2-1.5B","Llama-3.2-3B","Granite-3.0-2B-Base","Granite-3.1-3B-A800M-Base","Llama-2-7B","SmolLM2-1.7B","Qwen2-0.5B-Instruct","Gemma-2B","Qwen2-0.5B","Granite-3.1-1B-A400M-Instruct","Granite-3.1-1B-A400M-Base","Llama-3.2-1B","SmolLM-1.7B","SmolLM2-360M","SmolLM-135M","SmolLM-360M","SmolLM2-135M","Claude-Opus-4-5-20251101 (FC)","Claude-Sonnet-4-5-20250929 (FC)","Gemini-3-Pro-Preview (Prompt)","GLM-4.6 (FC thinking)","Zhipu AI","Grok-4-1-fast-reasoning (FC)","Claude-Haiku-4-5-20251001 (FC)","Gemini-3-Pro-Preview (FC)","o3-2025-04-16 (Prompt)","Grok-4-0709 (Prompt)","Grok-4-0709 (FC)","Moonshotai-Kimi-K2-Instruct (FC)","MoonshotAI","Grok-4-1-fast-non-reasoning (FC)","Command A Reasoning (FC)","Cohere","DeepSeek-V3.2-Exp (Prompt + Thinking)","Gemini-2.5-Flash (FC)","GPT-5.2-2025-12-11 (FC)","GPT-5-mini-2025-08-07 (FC)","xLAM-2-32b-fc-r (FC)","Salesforce","DeepSeek-V3.2-Exp (FC)","GPT-4.1-2025-04-14 (FC)","o4-mini-2025-04-16 (FC)","xLAM-2-70b-fc-r (FC)","Qwen3-235B-A22B-Instruct-2507 (Prompt)","GPT-5-nano-2025-08-07 (FC)","Nanbeige4-3B-Thinking-2511 (FC)","Nanbeige","Gemini-2.5-Flash (Prompt)","GPT-4.1-mini-2025-04-14 (FC)","o4-mini-2025-04-16 (Prompt)","Qwen3-32B (FC)","o3-2025-04-16 (FC)","Qwen3-235B-A22B-Instruct-2507 (FC)","Nanbeige3.5-Pro-Thinking (FC)","Qwen3-32B (Prompt)","xLAM-2-8b-fc-r (FC)","Command A (FC)","BitAgent-Bounty-8B","Bittensor","Arch-Agent-32B","katanemo","GPT-5.2-2025-12-11 (Prompt)","Qwen3-8B (FC)","ToolACE-2-8B (FC)","Huawei Noah & USTC","Qwen3-30B-A3B-Instruct-2507 (FC)","xLAM-2-3b-fc-r (FC)","Qwen3-14B (FC)","Qwen3-8B (Prompt)","GPT-4.1-2025-04-14 (Prompt)","mistral-large-2411 (FC)","Qwen3-14B (Prompt)","Mistral-Medium-2505","Mistral-Medium-2505 (FC)","Llama-4-Maverick-17B-128E-Instruct-FP8 (FC)","Mistral-small-2506 (FC)","Gemini-2.5-Flash-Lite (FC)","Qwen3-30B-A3B-Instruct-2507 (Prompt)","Qwen3-4B-Instruct-2507 (FC)","Qwen3-4B-Instruct-2507 (Prompt)","Arch-Agent-3B","Claude-Opus-4-5-20251101 (Prompt)","GPT-4.1-nano-2025-04-14 (FC)","Mistral-Small-2506 (Prompt)","Arch-Agent-1.5B","Command R7B (FC)","Llama-3.3-70B-Instruct (FC)","mistral-large-2411 (Prompt)","Hammer2.1-7b (FC)","MadeAgents","xLAM-2-1b-fc-r (FC)","Gemma-3-12b-it (Prompt)","GPT-4.1-mini-2025-04-14 (Prompt)","Hammer2.1-3b (FC)","Gemma-3-27b-it (Prompt)","Phi-4 (Prompt)","Qwen3-1.7B (FC)","Llama-4-Scout-17B-16E-Instruct (FC)","Gemini-2.5-Flash-Lite (Prompt)","CoALM-70B","UIUC + Oumi","Hammer2.1-1.5b (FC)","palmyra-x-004 (FC)","Writer","GPT-5-mini-2025-08-07 (Prompt)","Open-Mistral-Nemo-2407 (FC)","GPT-5-nano-2025-08-07 (Prompt)","Amazon-Nova-2-Lite-v1:0 (FC)","Amazon","Granite-3.1-8B-Instruct (FC)","IBM","Falcon3-10B-Instruct (FC)","TII UAE","Granite-3.2-8B-Instruct (FC)","CoALM-8B","Llama-3.1-8B-Instruct (Prompt)","MiniCPM3-4B-FC (FC)","openbmb","Claude-Haiku-4-5-20251001 (Prompt)","Amazon-Nova-Pro-v1:0 (FC)","Claude-Sonnet-4-5-20250929 (Prompt)","GPT-4.1-nano-2025-04-14 (Prompt)","Falcon3-7B-Instruct (FC)","Qwen3-0.6B (FC)","Granite-20b-FunctionCalling (FC)","Qwen3-0.6B (Prompt)","Amazon-Nova-Micro-v1:0 (FC)","RZN-T (Prompt)","Phronetic AI","MiniCPM3-4B (Prompt)","Llama-3.2-3B-Instruct (FC)","Bielik-11B-v2.3-Instruct (Prompt)","SpeakLeash & ACK Cyfronet AGH","Hammer2.1-0.5b (FC)","Gemma-3-4b-it (Prompt)","Open-Mistral-Nemo-2407 (Prompt)","Granite-4.0-350m (FC)","Falcon3-3B-Instruct (FC)","Ministral-8B-Instruct-2410 (FC)","Falcon3-1B-Instruct (FC)","Llama-3.2-1B-Instruct (FC)","Llama-3.1-Nemotron-Ultra-253B-v1 (FC)","Gemma-3-1b-it (Prompt)","Claude Opus 5","max; batch tool; 500 steps","MuseSpark 1.3","high; batch tool; 500 steps","xhigh; batch tool; 500 steps","medium; batch tool; 500 steps","low; batch tool; 500 steps","GPT-5.6 Sol","none; batch tool; 500 steps","Gemini-2.5-Pro","CoT; context 1M","Gemini-2.5-Flash","Qwen3-235B-A22B-Thinking-2507","CoT; context 256k","DeepSeek-R1","CoT; context 128k","Qwen3-235B-A22B-Instruct-2507","o1-preview","DeepSeek-R1-0528","MiniMax-Text-01","MiniMax","CoT; context 4M","Gemini-2.0-Flash-Thinking","Gemini-Exp-1206","CoT; context 2M","GPT-4o (2024-11-20)","Gemini-2.0-Flash","GLM-4.5","Z.ai & Tsinghua","Qwen3-235B-A22B","Qwen3-30B-A3B-Thinking-2507","Qwen3-32B","GLM-4.5-Air","Claude 3.5 Sonnet","CoT; context 200k","GLM-4-Plus","Kimi-K2-Instruct","Qwen2.5-72B","Qwen3-30B-A3B","Mistral Large 24.11","Llama 3.1 70B","Llama 3.3 70B","Qwen2.5-7B","Nemotron 70B","Nvidia","Mistral Large 2","GPT-4o mini","NExtLong 8B","CAS","CoT; context 512k","Command R+","Llama 3.1 8B","video-SALMONN 2+","THU & ByteDance","No subtitles; frames: 10 fps","No subtitles; frames: 1/0.5 fps","AdaReTaKe","HIT & Huawei","No subtitles; frames: 1024","Keye-VL-1.5","Kwai Keye Team","No subtitles; frames: 64","TSPO","XJTU & TeleAI & USTB","No subtitles; frames: 128","InternVL2.5","Shanghai AI Lab","No subtitles; frames: 384","Qwen2-VL","No subtitles; frames: 768","JT-VL-Chat","China Mobile","No subtitles; frames: 1800","LLaVA-Video","Bytedance & NTU S-Lab","Gemini 1.5 Flash","LinVT","Meituan Inc","No subtitles; frames: 120","Aria","Rhymes AI","No subtitles; frames: 256","ViLAMP","RUC & Ant Group","No subtitles; frames: 1 fps","Oryx-1.5","THU & Tencent & NTU","Long-VITA","Tencent & NJU","LLaVA-OneVision","No subtitles; frames: 32","VideoLLaMA 3","No subtitles; frames: 180","QuoTA","XMU","TPO","Stanford University","No subtitles; frames: 96","VideoChat-Flash","Shanghai AI Lab & NJU","No subtitles; frames: 512","No subtitles; frames: 250","ByteVideoLLM","Bytedance","No subtitles; frames: 100","NVILA","LiveCC","NUS Show Lab & ByteDance","No subtitles; frames: 2 fps","MiniCPM-o 2.6","OpenBMB","ReTaKe","VideoLLaMA 2","InternVL2","No subtitles; frames: 16","MiniCPM-V 2.6","No subtitles; frames: 20","GPT-4V","No subtitles; frames: 10","mPLUG-Owl3","TimeMarker","Meituan","VITA 1.5","Tencent Youtu Lab & NJU","Kangaroo","Meituan & UCAS","VITA","Video-XL","SJTU & BAAI","Video-CCAM","QQMM","Long-LLaVA","LongVA","NTU S-Lab","Qwen-VL-Max","No subtitles; frames: 4","InternVL-Chat-V1.5","SliME","CASIA","No subtitles; frames: 8","ShareGemini","Qwen-VL-Chat","Chat-UniVi-v1.5","PKU","ShareGPT4Video","Video-LLaVA","VideoChat2-Mistral","ST-LLM","Keen AgenticOS V1.0","KeenData","Base models: Deepseek，Kimi，Claude","Shadow Harness V8.01","Shadow","Base models: GPT Claude Grok Gemini DeepSeek","CustomGPT.ai Research Lab v44","CustomGPT.ai","Base models: CustomGPT.ai enterprise agent, Claude, Gemini, GPT","https://github.com/adorosario/customgpt-agent","OmniSeek_Preview_v1.1","Alibaba Enterprise Intelligence & HKUST(GZ)","Base models: Qwen3.8 Max, GPT-5.5, Gemini3.7 Flash","Co-Sight Pro v1.0.1","ZTE-AICloud","Base models: ZTE Nebula LLM, Gemini 3.1 Pro, GPT 5.5, Claude Opus 4.7","https://github.com/ZTE-AICloud/Co-Sight","Keen AgenticOS","Base models: Deeepseek ，Kimi，Claude","Shadow Harness V2.01","https://rysx.com.cn/","Shadow Harness V3.01","Shadow Harness V4.01","Base models: GPT Claude Gemini DeepSeek","Shadow Harness V5.01","Base models: GPT Claude Gemini DeepSeek Grok","Shadow Harness V6.01","Shadow Harness V7.01","Shadow Harness V 1.04","Shadow Harness V1.05","Shadow Harness V1.07","Base models: Grok GPT Claude DeepSeek","OPS-Agentic-Search","Alibaba Cloud","Base models: Qwen, Claude 4.6, GPT 5, DeepSeek 3.2, Gemini 3 Pro, Kimi K2.5","https://www.alibabacloud.com/en/product/opensearch?_p_lc=1","SHAPARTNER V1.01","SHAPARTNER","SHAPARTNER V1.02","Shadow Harness V14","Base models: GPT Claude Grok","AuraAi","AI","Base models: Claude","Co-Sight Pro v1.0.0","Base models: ZTE Nebula LLM, claude-opus-4.7, gemini-3.1-pro-preview","Cobra Agent Orch","Dalpha Inc","Base models: GPT 5.5, Gemini 3 Pro, Claude Opus 4.8","CustomGPT.ai Research Lab V42","Base models: CustomGPT.ai Enterprise Agents","CustomGPT.ai Research Lab v41","Base models: Claude, GPT, Gemini","CustomGPT.ai Research Lab v43","https://github.com/adorosario/customgpt-manus-agent","OmniSeek_Preview_v1","SHAPARTNER V11","Shadow Harness V13","Shadow Harness","Base models: Claude GPT Grok","Cobra Agent_v3.1","Base models: GPT 5.5","Cobra Agent_v3.2","Cobra Agent_v3.2.1","Cobra Agent_v3.2.2","Cobra Agent_v3.2.3","Cobra Agent_v3.3","CustomGPT.ai Research Lab v39","Shadow Harness V1.03","Base models: Grok GPT","Shadow Harness V1.06","Shadow Harness V1.09","Shadow Harness V11","Shadow Harness V15","Base models: Claude GPT Gemini","openJiuwen-deepagent","openJiuwen","Base models: GPT5, Gemini-3-Pro","https://gitcode.com/openJiuwen/agent-store/tree/main/community/deepagent","Cobra Agent_2.7","Cobra Agent_v2.7","Cobra Agent_v3.0","Cobra Agent_v3.4","CustomGPT.ai Research Lab v34","CustomGPT.ai Research Lab v35","Lemon","LR AILab of Lenovo CTO Org","Base models: GPT5, Gemini-3-pro, o3","https://github.com/Open-Lemon/LemonAgent","Shadow Harness V1.02","Base models: Claude GPT","CustomGPT.ai Research Lab v33","Harness V8","Harness","Base models: Grok","SHADOW HARNESS V10","SHADOW","Shadow Harness V1.08","Shadow Harness V12","https://rysx.com.cn","Shadow Harness V9","Shapartner V6","Shapartne","Base models: Grok Claude","CustomGPT.ai Research Lab v31","Base models: claude, gemini, gpt","CustomGPT.ai Research Lab v32","Nemotron-ToolOrchestra-0106","Base models: Nemotron-ToolOrchestrator-8B, GPT-5, Claude Opus 4.1, Qwen2.5-Math-72B","https://research.nvidia.com/labs/lpr/ToolOrchestra/","Nemotron-ToolOrchestra-0107","CustomGPT.ai Research Lab v27","https://huggingface.co/Arnav3035/garuda-sql-2b","CustomGPT.ai Research Lab v25","CustomGPT.ai Research Lab v26","CustomGPT.ai Research Lab v19","CustomGPT.ai Research Lab v16","Lemon Agent v1.0.8","Base models: GPT5, o3, Gemini-3-pro","CustomGPT.ai Research Lab v17","Nemotron-ToolOrchestra","Co-Sight_v2.1.0","Base models: ZTE Nebula LLM, Claude Sonnet 4, Gemini 2.5 Pro","Favonis Agent","Favonis","Base models: GPT 6.1 Sol","https://favon.is/","Keen-Agent-v3","Individual","Base models: Kimi + Claude + GPT","https://github.com/qingguang0309/MiroFlow_GAIA_Keendata","Raccoon AI","Base models: Gemini 3 Pro, Claude Opus 4.6, GPT 5.1","https://raccoonai.tech/","Keen-Agent-v4","Mimir v0.9.13","Team Norse","Base models: GPT, Gemini, Claude","https://mimir-paper.vercel.app/","Co-Sight_v2.0.1","CustomGPT.ai Research Lab v11","CustomGPT.ai Research Lab","Co-Sight_v2.0.0","Base models: Claude Sonnet 4, Gemini 2.5 Pro","h2oGPTe Agent v1.6.44","h2o.ai","Base models: sonnet-4-5 + gpt-5","https://h2ogpte.genai.h2o.ai/","CustomGPT.ai Research Lab v5","Skywork Deep Research Agent v2","Skywork AI","Base models: skywork-agent-model, gpt-4.1, claude-3.7-sonnet, gemini-2.5-pro","https://skyworkai.github.io/DeepResearchAgent","CustomGPT.ai Research Lab v2","Keen-Agent-v2","🐦‍🔥 AWorld (Run Instantly)","inclusionAI","Base models: GPT-4o, DeepSeek V3, Claude-Sonnet-4, Gemini-2.5-Pro","https://github.com/inclusionAI/AWorld/tree/main","Mimir","h2oGPTe Agent v1.6.33","Base models: claude-3-7-sonnet-20250219, gemini-2.5-pro-preview-06-05 (extended thinking)","https://id.public.h2o.ai/","AG-GAIA-S7","AG","ARI-CLI V11 0322","Clevi","Base models: cip-5.5-agent, cip-5.5-mm","https://clevi.ai/ko","MiroFlow (Reproducible!)","MiroMind AI","Base models: GPT-5 (agent), o3 (hints, summary), Gemini-2.5-Pro (as tool), Claude (as tool)","https://github.com/MiroMindAI/MiroFlow","h2oGPTe Agent v1.6.32","Base models: claude-3-7-sonnet-20250219, gemini-2.5-pro-preview-06-05","abcdagent v26.09.28","abcd","Base models: gpt-5.5","https://developers.openai.com/api/docs/models/gpt-5","ari-cli with arbiter agent","CLEVI","🦤 AWorld (Run Instantly)","Keen-Agent","remio_gpt5.5","everdent.ai","Base models: GPT-5.5","https://openai.com/zh-Hans-CN/index/introducing-gpt-5-5/","ARI-Self Drive Without Prompt","https://clevi.ai/","h2oGPTe Agent v1.6.27 | March 17 original date","Base models: claude-3-7-sonnet","ARI CLI","Base models: cip-5.5-agent","Agnes AI","Sapiens AI","Base models: Agnes AI","https://agnes-ai.com","Clawdbot","Nora AI","Base models: Claude Opus 4.5","https://docs.clawd.bot","CustomGPT Agent v2.1","Base models: Claude (Opus 4.6 + Sonnet 4.6)","https://github.com/uckmhnds/customgpt-agent","CustomGPT Agent v2","AiDE-Agent_v9","ValueLabs","Base models: Claude-sonnet-4-5","https://www.anthropic.com/news/claude-sonnet-4-5","AiDE-Agent_v7","🦩 AWorld (Run Instantly)","Base models: GPT-4o, DeepSeek V3, Claude-3.7-Sonnet, Gemini 2.5 Pro","Corint v1.0","Futurity","Base models: DeepSeek V4 Pro","https://futurity.work","CustomGPT Agent","AiDE-Agent_v5","Durandal (deepseek-v4-pro)","cloudai.nexus","Base models: DeepSeek-V4-Pro","https://api-docs.deepseek.com","ARI on SVCE with CTA-Subagents","AiDE-Agent_v6","https://agnes-ai.com/","remio","Base models: GPT-5.4","https://openai.com/zh-Hans-CN/index/introducing-gpt-5-4/","Sukuna V0.1.5","Midsphere AI","Base models: gpt-5, gemini-2.5-pro, o3, claude-sonnet-4","https://www.midsphere.ai/","AiDE-Agent_v4","ReAct_ds_fr","imperial college london","Base models: DeepSeek v4 flash","https://api-docs.deepseek.com/","Slai Agent","slai.life","Base models: gpt 5.6 terra","https://openai.com","AiDE-Agent_v3","JoyAgent-Genie-v2","JD Enterprise Intelligence (EI)","Base models: OpenAI, Anthropic","https://github.com/jd-opensource/joyagent-jdgenie","OpenHands_Versa","CMU, All Hands AI","Base models: OpenAI GPT5","https://arxiv.org/abs/2506.03011","Slai Agent Runtime","Base models: GPT-5.6 Terra","https://openai.com/","AiDE-Agent_v8","Trase Agent v0.3","Trase Systems","Base models: Claude, o1, Gemini","https://trasesystems.com","AiDE-Agent_v2","Max-1.1","Finalmoment","Base models: Deepseek-V4-Flash-4.1, Gpt-Luna, Musespark-1.3","https://iris.finalmoment.ai/","OpenHands Versa","Base models: GPT_5","https://github.com/adityasoni9998/OpenHands-Versa","XWork-MultiAgent","iFLYTEK","https://api.deepseek.com","EvolAgent","KLGR123","Base models: GPT, Claude","https://github.com/KLGR123/EvolAgent","JoyAgent-Genie","h2oGPTe Agent v1.6.8","Base models: claude-3-5-sonnet","Aron","FereAI.xyz","Base models: Claude 4.6, Gemini 3.1","https://docs.anthropic.com/en/docs/about-claude/models","Sukuna V0.1.4","https://midsphere.ai/","Base models: GPT-5.6 Terra xhigh","https://slai.life/","🐟 AWorld (Run Instantly)","Ant Group","Evolagent-baseline-v0.1","Sukuna V0.1.3","Base models: gpt-5, gemini-2.5-pro, claude-sonnet-4","Sukuna V0.1.2","AiDE-Agent_v1","Base models: Claude & GPT","Claude Code Ali 2.1.224","AgentEval","Base models: deepseek-v4-flash-0731","https://help.aliyun.com/zh/model-studio/deepseek-api","Sukuna V0.1.1","Base models: gpt-5, gemini-2.5-pro, sonnet-4","AiDE-Agent","exo-unoptimized","heyalexchoi","Base models: grok 4 fast, gemini 2.5 flash","https://heyalexchoi.com/exo/","gemini-3.1-pro-s1-react","Independent","Base models: gemini-pro-preview-3-1","https://arxiv.org/abs/2506.08529","gemini-3.1-pro-s1-react-corrected","exo-vanilla-react","AiDE Agents","Base models: claude-sonnet-4-5","Flywork CLI 0.2.4-beta.1 Corrected","Base models: Qwen3.8-27B-NVFP4","https://qwen.ai/blog?id=qwen3.8","gemini-3.1-pro-s2-PAR","exo-2025-11-12-distilled-with-feedback","Sukuna V0.1","Base models: gemini-2.5-pro, gpt-5, claude-sonnet-4","Ormind v0.2","Ormind","Base models: Claude 3.7, GPT4.1, Gemini 2.5","https://ormind.ai","Federal-KB FedQES (GPT-5.6 Luna)","Federal-KB","Base models: GPT-5.6 Luna (manager, search, retrieval); GPT-4o (vision); Whisper-1 (audio)","https://github.com/kesfc/Federal-KB","OpenHands-Versa","Base models: claude-sonnet-4-20250514","🐳 AWorld (Click to Run Instantly)","Base models: GPT-4o, DeepSeek V3, Claude-3.7-Sonnet","open Deep Research | pass@1","HF 🤗 smolagents","Base models: o1","https://huggingface.co/blog/open-deep-research","Codex CLI 0.149.1 Corrected","https://help.aliyun.com/zh/model-studio/deepseek-v4-flash","h2oGPTe Agent v1.6.27 (Open Weights)","Base models: DeepSeek-R1 & Qwen2.5-VL-72B-Instruct","https://genai.h2o.ai/","Vector Assist","Vector Ventures","Base models: Kimi-K2.5, Minimax","https://ollama.com/library/kimi-k2.5","Auto-Use","Base models: gemini-3.1-pro (openrouter)","https://openrouter.ai/google/gemini-3.1-pro-preview","TapeAgents & BrowserGym","ServiceNow Research","Base models: claude-3.7-sonnet","https://github.com/ServiceNow/TapeAgents/tree/main/examples/gaia_agent","sonnet-4.6-s1-react","rect","person","Base models: qwen","https://github.com/QwenLM/Qwen3","Magentic-UI","MSR AI Frontiers","Base models: o4-mini","https://github.com/microsoft/magentic-ui","my_agent","snikmas","Base models: DeepSeek-V4.1-Flash","https://huggingface.co/spaces/snikmas/my_space/tree/main","Trase Agent v0.2","Base models: Multi-Agent - Gemini Fine-tuned GPT-4o o1-preview","https://www.trasesystems.com/","DynaSaur 🦖","Adobe Research","Base models: GPT-4o (gpt-4o-2024-08-06)","https://github.com/adobe-research/dynasaur","Magentic-1 (o1)","Base models: GPT-4o and o1-preview (varies by agent)","https://github.com/microsoft/autogen/tree/main/python/packages/autogen-magentic-one","ADK-GAIA","SZU-GML","Base models: Gemini","https://github.com/GML-FMGroup/adk-gaia","Trase Agent v0.1","Base models: Fine-tuned GPT-4o","https://trasesystems.com/","Sibyl System v0.2","Baichuan Inc.","Base models: GPT-4o","https://sibyl.wangyulong.io/","GenericAgent","KW","https://github.com/lsdefine/GenericAgent","AgentKB Pass1 Frozen final-301-of-301-20260826","FedAgentKE","Base models: Qwen3-Next-80B-A3B-Instruct (Vertex MaaS)","https://huggingface.co/WeihaoLi","Hugging Face Agents + GPT-4o","Hugging Face 🤗","https://huggingface.co/learn/cookbook/agents","Magentic-1","Multi-Agent Experiment v0.1 (powered by AutoGen)","Base models: GPT-4-turbo","https://aka.ms/gaia_multiagent_v01_march_1st","AgentKB Pass1 Frozen final-301-of-301","FedQES L1 rerun 20260929 - original L2 L3","AgentKB FedQES Pass1 final-301-of-301","FedQES-Test-301-Final-20260903","Base models: Qwen3-Next-80B-A3B-Instruct (Vertex AI MaaS)","DR_agent","Independent researcher","Base models: DeepSeek-V3.2","https://github.com/jacksong-sourse/DR_agent","MAYA_v1","Adya","Base models: Claude-sonnet-4.5","https://adya.ai/maya","UK AI Safety Institute Internal","UK AI Safety Institute","Base models: GPT-4-Turbo","https://www.gov.uk/government/publications/ai-safety-institute-overview/introducing-the-ai-safety-institute","FRIDAY","OS-Copilot","https://os-copilot.github.io/","deepseek-v4-flash","chery-ai","Base models: deepseek-chat","https://api.deepseek.com/v1","haiku-4.5-s1-react","Base models: claude-haiku-4-5","GLM-5.2 smolagents baseline partial-71","1g0rp4vl","Base models: GLM-5.2","https://huggingface.co/zai-org/GLM-5.2","FRIDAY_without_learning","FedAgentKE ExpGraph Test current 238-of-301","Hybrid-Search","Institute of Automation Chinese Academy of Sciences","Base models: Qwen3.6-35B-A3B","https://huggingface.co/Qwen/Qwen3.6-35B-A3B","Prometheus17","2AcreStudios","Base models: Claude Opus 4.6 + Cascading Router","https://github.com/2AcreStudios/prometheus","MAYA","Base models: Claude 4.5 Sonnet","MyGAIA_Agent_v1","Amal Alexander","Base models: smolagents + Mistral Small","https://huggingface.co/amallx1997","WelfareWise","SimplifyNext","Base models: gpt-4.1","https://openai.com/index/gpt-4-1/","GAIA-LLM-Agent (Llama-3-70B)","Northeastern University","Base models: Meta-Llama-3-70B-Instruct-Turbo","https://together.ai/models/meta-llama/Meta-Llama-3-70B-Instruct-Turbo","FedAgentKE ExpGraph Global-only 151-of-301 20260901","Base models: Qwen3-Next-80B-A3B-Instruct","CG05","Base models: ollama","https://www.deepseek.com/","AOne超级连接","天翼云科技有限公司","Base models: DeepSeek、Qwen3","https://www.ctyun.cn/products/accessone","ACE-Qwen3-32B-AWQ-NoSearch","中科大","Base models: Qwen3","https://huggingface.co/Qwen/Qwen3-32B-AWQ","Codex web-assisted Level3 diagnostic","Base models: OpenAI Codex (GPT-5 family)","https://openai.com/codex/","gemma-4-26b-a4b-it","Personal","Base models: Google:Gemma","https://openrouter.ai/google/gemma-4-26b-a4b-it:free","Prometheus","Base models: Claude Opus 4.5 + Cascading Router","https://huggingface.co/2AcreStudios","Prometheus2","https://github.com/2AcreStudios","GPT4 Turbo","GAIA authors","Base models: GPT4","https://openai.com/blog/new-models-and-developer-products-announced-at-devday","Prometheus (Claude + Cascading Router)","Base models: Claude claude-opus-4-5-20251101 + Qwen/QwQ-32B + Groq","https://anthropic.com/claude","personal","AutoGPT4","AutoGPT","Base models: GPT4 + AutoGPT","https://github.com/Significant-Gravitas/AutoGPT","Shenova AI","Neha Bhat","Base models: Qwen/Qwen2.5-1.5B-Instruct","https://huggingface.co/spaces/nehabhat345/shenova-ai","GPT4","https://arxiv.org/abs/2303.08774","Codex CLI 0.149.1","https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731","GAIA-","Self","Base models: deepseek-r1-distill-qwen-7b","https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-7B","GPT3.5","Base models: GPT3","Custom-ReAct-Agent-v1","https://github.com/tu-usuario/tu-agente](https://github.com/tu-usuario/tu-agente","AG-GAIA-1","Base models: Claude 102","gaia_agent_huggingfacet","Base models: llama-3.1","https://github.com/Ppp111ppp111/gaia_agent_huggingface","test-101","TIE","AG-GAIA","AG-GAIA-3","AG-GAIA-5","AG-GAIA-7","Flywork CLI 0.2.4-beta.1","GAIA Agent (Gemini 2.0 Flash)","Karukola Hemanth","Base models: Gemini 2.0 Flash","https://github.com/Karukolahemanth/Hugging-face","Opus 5 (Claude Code) xhigh","anthropic","V2 Full; harness as listed in the model name","Gemini 4 Argon (mini-swe-agent) high","openai","Fable 5.1 (Claude Code) high","Kimi-K3 (mini-swe-agent) max","moonshot","GPT-6-Astra (Codex) high","Muse Wonder 1.0 (mini-swe-agent) xhigh","GPT-6.1-Sol (mini-swe-agent) xhigh","Sonnet 5.5 (mini-swe-agent) xhigh","GLM-5.3 (mini-swe-agent) max","zai","GPT-5.6-sol (Codex) xhigh","Muse Spark Open (mini-swe-agent) xhigh","Gemini 3.8 Flash (mini-swe-agent) high","google","Sonnet 5 (Claude Code) xhigh","GPT-5.6-Terra (Codex) xhigh","Inkling (mini-swe-agent) xhigh","thinkingmachines","Haiku 5.5 (mini-swe-agent) high","gpt-5 (high)","Edit format: diff; Reasoning effort: high; Versions: 0.86.2.dev","gpt-5 (medium)","Edit format: diff; Reasoning effort: medium; Versions: 0.86.2.dev","o3-pro (high)","Edit format: diff; Reasoning effort: high; Versions: 0.85.1.dev","gemini-2.5-pro-preview-06-05 (32k think)","Edit format: diff-fenced; Versions: 0.84.1.dev","gpt-5 (low)","Edit format: diff; Reasoning effort: low; Versions: 0.86.2.dev","o3 (high)","Edit format: diff; Reasoning effort: high; Versions: 0.84.1.dev","grok-4 (high)","Edit format: diff; Reasoning effort: high; Versions: 0.85.2.dev","gemini-2.5-pro-preview-06-05 (default think)","Gemini 2.5 Pro Preview 05-06","Edit format: diff-fenced; Versions: 0.82.4.dev","o3","Edit format: diff; Versions: 0.84.1.dev","DeepSeek-V3.2-Exp (Reasoner)","Edit format: diff; Versions: 0.86.2.dev","Gemini 2.5 Pro Preview 03-25","Edit format: diff-fenced; Versions: 0.81.3.dev","claude-opus-4-20250514 (32k thinking)","Edit format: diff; Versions: 0.83.3.dev","o4-mini (high)","Edit format: diff; Versions: 0.82.1.dev","claude-opus-4-20250514 (no think)","DeepSeek-V3.2-Exp (Chat)","claude-3-7-sonnet-20250219 (32k thinking tokens)","Edit format: diff; Versions: 0.75.1.dev","DeepSeek R1 + claude-3-5-sonnet-20241022","Edit format: architect; Versions: 0.72.3.dev","claude-sonnet-4-20250514 (32k thinking)","claude-3-7-sonnet-20250219 (no thinking)","Edit format: diff; Versions: 0.74.4.dev","Qwen3 235B A22B diff, no think, Alibaba API","Edit format: diff; Versions: 0.82.4.dev","Kimi K2","Edit format: diff; Versions: 0.85.3.dev","DeepSeek R1","Edit format: diff; Versions: 0.71.2.dev","claude-sonnet-4-20250514 (no thinking)","DeepSeek V3 (0324)","Edit format: diff; Versions: 0.78.1.dev","gemini-2.5-flash-preview-05-20 (24k think)","Quasar Alpha","Edit format: diff; Versions: 0.80.5.dev","o3-mini (medium)","Edit format: diff; Versions: 0.72.4.dev","Grok 3 Beta","Edit format: diff; Versions: 0.81.2.dev","Optimus Alpha","gpt-4.1","Edit format: diff; Versions: 0.81.4.dev","claude-3-5-sonnet-20241022","Grok 3 Mini Beta (high)","Edit format: whole; Versions: 0.81.3.dev","DeepSeek Chat V3 (prev)","Edit format: diff; Versions: 0.69.2.dev","gemini-2.5-flash-preview-04-17 (default)","Edit format: diff; Versions: 0.82.3.dev","chatgpt-4o-latest (2025-03-29)","Edit format: diff; Versions: 0.79.3.dev","gemini-2.5-flash-preview-05-20 (no think)","gpt-oss-120b (high)","Qwen3 32B","gemini-exp-1206","Edit format: whole; Versions: 0.69.2.dev","Gemini 2.0 Pro exp-02-05","Edit format: whole; Versions: 0.75.2.dev","Grok 3 Mini Beta (low)","Edit format: whole; Versions: 0.81.2.dev","o1-mini-2024-09-12","gpt-4.1-mini","claude-3-5-haiku-20241022","QwQ-32B + Qwen 2.5 Coder Instruct","Edit format: architect; Versions: 0.75.3.dev","Edit format: diff; Versions: 0.70.1.dev","gemini-2.0-flash-exp","qwen-max-2025-01-25","Edit format: diff; Versions: 0.75.3.dev","gemini-2.0-flash-thinking-exp-01-21","Edit format: diff; Versions: 0.72.2.dev","gpt-4o-2024-11-20","DeepSeek Chat V2.5","Qwen2.5-Coder-32B-Instruct","yi-lightning","command-a-03-2025-quality","Edit format: whole; Versions: 0.77.1.dev","Codestral 25.01","Edit format: whole; Versions: 0.71.2.dev","openhands-lm-32b-v0.1","Edit format: whole; Versions: 0.82.2.dev","gpt-4.1-nano","Edit format: whole; Versions: 0.81.4.dev","gemma-3-27b-it","o3-mini-2025-01-31 (temperature=1, reasoning=medium)","Hard-Instruct; prefill=False; prompted=True","https://openai.com/index/openai-o3-mini/","Claude-3.7-Sonnet-20250219 (temperature=1, length=12800, reasoning=3200)","https://www.anthropic.com/news/claude-3-7-sonnet","o1-2024-12-17 (temperature=1, reasoning=high)","https://openai.com/o1/","o3-mini-2025-01-31 (temperature=1, reasoning=high)","Claude-3.7-Sonnet-20250219","GPT-4.1-2025-04-14","GPT-4.1-Mini-2025-04-14","Quasar-Alpha","https://openrouter.ai/openrouter/quasar-alpha","Grok-3-Mini-Beta (temperature=1, reasoning=low)","https://x.ai/news/grok-3","o3-mini-2025-01-31 (temperature=1, reasoning=low)","Optimus-Alpha","https://openrouter.ai/openrouter/optimus-alpha","Athene-V2-Agent","Hard-Instruct; prefill=True; prompted=True","https://huggingface.co/Nexusflow/Athene-V2-Agent","https://huggingface.co/deepseek-ai/DeepSeek-R1","https://blog.google/technology/google-deepmind/gemini-model-thinking-updates-march-2025/","QwQ-32B (w/ Reasoning)","https://huggingface.co/Qwen/QwQ-32B","o1-2024-12-17 (temperature=1, reasoning=low)","GPT-4-Turbo-2024-04-09","Qwen2.5-Max","https://qwenlm.github.io/blog/qwen2.5-max/","DeepSeek-V3","https://huggingface.co/deepseek-ai/DeepSeek-V3","GPT-4.1-Nano-2025-04-14","https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct","o1-2024-12-17 (temperature=1, reasoning=medium)","DeepSeek-V3-0324","https://huggingface.co/deepseek-ai/DeepSeek-V3-0324","GPT-4o-2024-11-20","https://openai.com/gpt-4o/","https://deepmind.google/technologies/gemini","Llama-4-Maverick","https://huggingface.co/meta-llama/Llama-4-Maverick-17B-128E-Instruct","https://huggingface.co/Qwen/Qwen2.5-Coder-32B-Instruct","o1-Mini-2024-09-12 (temperature=1)","https://openai.com/index/introducing-openai-o1-preview/","https://huggingface.co/Nexusflow/Athene-V2-Chat","DeepSeek-V2.5-1210","Gemini-1.5-Pro-Exp-0827","https://deepmind.google/technologies/gemini/pro","Gemini-2.0-Pro-Exp-02-05","https://blog.google/technology/google-deepmind/gemini-model-updates-february-2025/","Grok-3-Beta","Gemini-Exp-1114","Sky-T1-32B-Preview","https://huggingface.co/NovaSky-AI/Sky-T1-32B-Preview","ChatGPT-4o-latest-20250129","https://chat.openai.com/","Claude-3.5-Haiku-20241022","https://claude.ai/","Claude-3.5-Sonnet-20240620","Claude-3.5-Sonnet-20241022","DeepSeek-Coder-V2-Instruct (2024-07-24)","Sky-T1-32B-Flash","https://huggingface.co/NovaSky-AI/Sky-T1-32B-Flash","DeepSeek-V2-Chat (2024-06-28)","GPT-4o-2024-05-13","https://openai.com/index/hello-gpt-4o/","GPT-4o-2024-08-06","https://openai.com/index/introducing-structured-outputs-in-the-api/","Gemini-1.5-Pro-Exp-0801","Gemma-3-27B-Instruct","https://huggingface.co/google/gemma-3-27b-it","QwQ-32B-Preview","https://huggingface.co/Qwen/QwQ-32B-Preview","https://huggingface.co/deepseek-ai/DeepSeek-Coder-V2-Instruct","Gemini-2.0-Flash-Thinking-Exp-01-21","Gemini-2.0-Flash-Thinking-Exp-1219","Gemini-Exp-1121","Phi-4","https://huggingface.co/microsoft/phi-4","DeepSeek-R1-Distill-Qwen-32B","https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-32B","GPT-4o-mini-2024-07-18","https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/","Gemini-2.0-Flash-001","Grok-3-Mini-Beta (temperature=1, reasoning=high)","Grok-Beta","https://grok.com/","Qwen2.5-Coder-14B-Instruct","https://huggingface.co/Qwen/Qwen2.5-Coder-14B-Instruct","DeepSeek-V2.5","Gemini-2.0-Flash-Exp","https://huggingface.co/meta-llama/Meta-Llama-3.1-70B-Instruct","o1-Preview-2024-09-12 (temperature=1)","Claude-3-Opus-20240229","https://www.anthropic.com/news/claude-3-family","Gemini-1.5-Flash-API-0514","https://deepmind.google/technologies/gemini/flash/","https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct","Llama-3.1-405B-Instruct","https://huggingface.co/meta-llama/Meta-Llama-3.1-405B-Instruct","https://huggingface.co/mistralai/Mistral-Large-Instruct-2407","Qwen2.5-32B-Instruct","https://huggingface.co/Qwen/Qwen2.5-32B-Instruct","DeepSeek-V2-Chat","https://huggingface.co/deepseek-ai/DeepSeek-V2-Chat","Mistral-Small-24B-Instruct-2501","https://huggingface.co/mistralai/Mistral-Small-24B-Instruct-2501","Qwen2.5-72B-Instruct","https://huggingface.co/Qwen/Qwen2.5-72B-Instruct","DeepSeek-R1-Distill-Qwen-14B","https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-14B","Dracarys-Llama-3.1-70B-Instruct","https://huggingface.co/abacusai/Dracarys-Llama-3.1-70B-Instruct","Llama-3.1-Nemotron-70B-Instruct","https://huggingface.co/nvidia/Llama-3.1-Nemotron-70B-Instruct-HF","Qwen2.5-14B-Instruct","https://huggingface.co/Qwen/Qwen2.5-14B-Instruct","Claude-3-Sonnet-20240229","DeepSeek-R1-Distill-Llama-70B","https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Llama-70B","Hermes-2-Theta-Llama-3-70B","https://huggingface.co/NousResearch/Hermes-2-Theta-Llama-3-70B","Qwen2.5-Coder-7B-Instruct","https://huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct","Gemini-1.5-Flash-Exp-0827","Gemini-1.5-Pro-API-0514","Gemini-2.0-Flash-Lite-Preview-02-05","ReflectionCoder-DS-33B","https://huggingface.co/SenseLLM/ReflectionCoder-DS-33B","CodeGeex4-All-9B","https://huggingface.co/THUDM/codegeex4-all-9b","CodeQwen1.5-7B-Chat","https://huggingface.co/Qwen/CodeQwen1.5-7B-Chat","Mixtral-8x22B-Instruct","https://huggingface.co/mistralai/Mixtral-8x22B-Instruct-v0.1","DeepCoder-14B-Preview","https://huggingface.co/agentica-org/DeepCoder-14B-Preview","Dracarys-72B-Instruct","https://huggingface.co/abacusai/Dracarys-72B-Instruct","GPT-3.5-Turbo-0125","https://openai.com/index/new-embedding-models-and-api-updates","Llama-3-70B-Synthia-v3.5","https://huggingface.co/migtissera/Llama-3-70B-Synthia-v3.5","Nxcode-CQ-7B-Orpo","https://huggingface.co/NTQAI/Nxcode-CQ-7B-orpo","OpenCoder-8B-Instruct","https://huggingface.co/infly/OpenCoder-8B-Instruct","DeepSeek-Coder-33B-Instruct","https://huggingface.co/deepseek-ai/deepseek-coder-33b-instruct","Gemma-2-27B-Instruct","https://huggingface.co/google/gemma-2-27b-it","Phi-3.1-Mini-128K-Instruct","https://huggingface.co/microsoft/Phi-3-mini-128k-instruct","WhiteRabbitNeo-33B-v1.5","https://huggingface.co/WhiteRabbitNeo/WhiteRabbitNeo-33B-v1.5","Codestral-22B-v0.1","https://huggingface.co/mistralai/Codestral-22B-v0.1","DeepCoder-14B-Preview (w/ Reasoning, 64k tokens, temperature=0.6)","Llama-4-Scout","https://huggingface.co/meta-llama/Llama-4-Scout-17B-16E-Instruct","Phi-3.5-Mini-Instruct","https://huggingface.co/microsoft/Phi-3.5-mini-instruct","https://huggingface.co/deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct","Athene-70B","https://huggingface.co/Nexusflow/Athene-70B","AutoCoder","https://huggingface.co/Bin12345/AutoCoder","Yi-Large","https://www.lingyiwanwu.com/","GPT-4-0613","Hermes-2-Pro-Llama-3-70B","https://huggingface.co/NousResearch/Hermes-2-Pro-Llama-3-70B","https://huggingface.co/Qwen/Qwen2-72B-Chat","Tess-v2.5.2-Qwen2-72B","https://huggingface.co/migtissera/Tess-v2.5.2-Qwen2-72B","Granite-Code-34B-Instruct","https://huggingface.co/ibm-granite/granite-34b-code-instruct","InverseCoder-CL-7B","https://huggingface.co/wyt2000/InverseCoder-CL-7B","WarriorCoder-6.7B (Reproduced)","https://arxiv.org/abs/2412.17395","C4AI-Command-R-08-2024","https://huggingface.co/CohereForAI/c4ai-command-r-08-2024","https://huggingface.co/CohereForAI/c4ai-command-r-plus","https://huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct","Magicoder-S-DS-6.7B","https://huggingface.co/ise-uiuc/Magicoder-S-DS-6.7B","OpenCodeInterpreter-DS-6.7B","https://huggingface.co/m-a-p/OpenCodeInterpreter-DS-6.7B","Codestral-Mamba","https://huggingface.co/mistralai/mamba-codestral-7B-v0.1","InverseCoder-CL-13B","https://huggingface.co/wyt2000/InverseCoder-CL-13B","Mistral-Small-2402","https://mistral.ai/news/mistral-large/","Phind-CodeLlama-34B-v2","https://huggingface.co/Phind/Phind-CodeLlama-34B-v2","WaveCoder-Ultra-6.7B","https://huggingface.co/microsoft/wavecoder-ultra-6.7b","Artigenz-Coder-DS-6.7B","https://huggingface.co/Artigenz/Artigenz-Coder-DS-6.7B","Qwen1.5-110B-Chat","https://huggingface.co/Qwen/Qwen1.5-110B-Chat","https://huggingface.co/Qwen/Qwen1.5-72B-Chat","Qwen2.5-7B-Instruct","https://huggingface.co/Qwen/Qwen2.5-7B-Instruct","StarCoder2-15B-Instruct-v0.1","https://huggingface.co/bigcode/starcoder2-15b-instruct-v0.1","C4AI-Command-R-Plus-08-2024","https://huggingface.co/CohereForAI/c4ai-command-r-plus-08-2024","CodeLlama-70B-Instruct","https://huggingface.co/codellama/CodeLlama-70b-Instruct-hf","Granite-Code-20B-Instruct","https://huggingface.co/ibm-granite/granite-20b-code-instruct","Mistral-Nemo-12B-Instruct","https://huggingface.co/nv-mistralai/Mistral-Nemo-12B-Instruct","ReflectionCoder-CL-34B","https://huggingface.co/SenseLLM/ReflectionCoder-CL-34B","ReflectionCoder-DS-6.7B","https://huggingface.co/SenseLLM/ReflectionCoder-DS-6.7B","Yi-Coder-9B-Chat","https://huggingface.co/01-ai/Yi-Coder-9B-Chat","InverseCoder-DS-6.7B","https://huggingface.co/wyt2000/InverseCoder-DS-6.7B","https://huggingface.co/mistralai/Mistral-Small-Instruct-2409","Phi-3-Medium-128K-Instruct","https://huggingface.co/microsoft/Phi-3-medium-128k-instruct","Phi-3-Small-128K-Instruct","https://huggingface.co/microsoft/Phi-3-small-128k-instruct","DeepSeek-Coder-6.7B-Instruct","https://huggingface.co/deepseek-ai/deepseek-coder-6.7b-instruct","Gemma-2-9B-Instruct","https://huggingface.co/google/gemma-2-9b-it","Qwen2-57B-A14B","https://huggingface.co/Qwen/Qwen2-57B-A14B","https://huggingface.co/Qwen/Qwen2-7B-Instruct","ReflectionCoder-CL-7B","https://huggingface.co/SenseLLM/ReflectionCoder-CL-7B","https://huggingface.co/01-ai/Yi-1.5-9B-Chat","CodeLlama-13B-Instruct","https://huggingface.co/codellama/CodeLlama-13b-Instruct-hf","OpenCoder-1.5B-Instruct","https://huggingface.co/infly/OpenCoder-1.5B-Instruct","Qwen1.5-32B-Chat","https://huggingface.co/Qwen/Qwen1.5-32B-Chat","AutoCoder-QW-7B","https://huggingface.co/Bin12345/AutoCoder_QW_7B","AutoCoder-S-6.7B","https://huggingface.co/Bin12345/AutoCoder_S_6.7B","CodeLlama-34B-Instruct","https://huggingface.co/codellama/CodeLlama-34b-Instruct-hf","OpenChat-3.6-8B-20240522","https://huggingface.co/openchat/openchat-3.6-8b-20240522","https://huggingface.co/01-ai/Yi-1.5-34B-Chat","CodeGemma-7B-Instruct","https://huggingface.co/google/codegemma-7b-it","Granite-3.0-8B-Instruct","https://huggingface.co/ibm-granite/granite-3.0-8b-instruct","Phi-3-Mini-128K-Instruct","https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct","Llama-3.2-3B-Instruct","https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct","OpenCodeInterpreter-DS-1.3B","https://huggingface.co/m-a-p/OpenCodeInterpreter-DS-1.3B","Qwen2.5-Coder-1.5B-Instruct","https://huggingface.co/Qwen/Qwen2.5-Coder-1.5B-Instruct","Mistral-7B-Instruct-v0.3","https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3","Mistral-Large-2402","https://huggingface.co/01-ai/Yi-1.5-6B-Chat","CodeLlama-7B-Instruct","https://huggingface.co/codellama/CodeLlama-7b-Instruct-hf","DeepSeek-Coder-1.3B-Instruct","https://huggingface.co/deepseek-ai/deepseek-coder-1.3b-instruct","DeepSeek-R1-Distill-Qwen-7B","InternLM2.5-7B-Chat","Shanghai AI Laboratory","https://huggingface.co/internlm/internlm2_5-7b-chat","DeepSeek-R1-Distill-Llama-8B","https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Llama-8B","Granite-3.0-2B-Instruct","https://huggingface.co/ibm-granite/granite-3.0-2b-instruct","Qwen2.5-1.5B-Instruct","https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct","Llama-3.2-1B-Instruct","https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct","Qwen2.5-0.5B-Instruct","https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct","DeepSeek-R1-Distill-Qwen-1.5B","https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B","O1 Mini (Sept 2024)","Chat prompted","O1 Preview (Sept 2024)","GPT 4o (Aug 2024)","DeepSeek-V3 (Nov 2024)","DeepSeek-V2.5 (Nov 2024)","GPT-4-Turbo (April 2024)","Gemini 1.5 Pro 002","GPT 4o Mini (July 2024)","GPT-4 (May 2023)","Grok Beta","Claude Sonnet 3.5 (June 2024)","GPT-4-Turbo (Nov 2023)","claude-3-opus (Mar 2024)","Gemini 1.5 Flash 002","DeepSeek-Coder-33B-instruct","WizardCoder-33B-V1.1","OpenCodeInterpreter-DS-33B","Llama3-70B-instruct","speechless-codellama-34B-v2.0","GPT-3.5-Turbo (Nov 2023)","claude-3-haiku (Mar 2024)","DeepSeek-Coder-7B-instruct-v1.5","XwinCoder-34B","databricks/dbrx-instruct","DeepSeek-Coder-6.7B-instruct","code-millenials-34B","starchat2-15b-v0.1","GPT-3.5 (May 2023)","WizardCoder-Python-34B-V1.0","OpenChat-3.5-7B-0106","WhiteRabbitNeo-33B-v1","Completion","speechless-coder-ds-6.7B","Magicoder-S-CL-7B","claude-3-sonnet (Mar 2024)","Llama3.1-8B-instruct","Mistral Large (Mar 2024)","claude-2 (Mar 2024)","Gemini Pro 1.5","starcoder2-15b-instruct-v0.1","speechless-starcoder2-15b","DeepSeek-Coder-1.3B-instruct","Code-290k-6.7B-Instruct","Phi-3-mini-4k-instruct","Command-R+","dolphin-2.6-mixtral-8x7b","Gemini Pro 1.0","Llama3-8B-instruct","codegemma-7b-it","claude-instant-1 (Mar 2024)","WizardCoder-15B-V1.0","Code-13B","speechless-starcoder2-7b","CodeLlama-70B","Code-33B","speechless-coding-7B-16k-tora","OpenHermes-2.5-Code-290k-13B","CodeLlama-34B","CodeQwen1.5-7B","DeepSeek-Coder-33B-base","WizardCoder-Python-7B-V1.0","phi-2-2.7B","Mistral-codealpaca-7B","speechless-code-mistral-7B-v1.0","DeepSeek-Coder-6.7B-base","MistralHermes-CodePro-7B-v1","StarCoder2-15B","codegemma-7b","SOLAR-10.7B-Instruct-v1.0","CodeLlama-13B","gemma-1.1-7b-it","xDAN-L1-Chat-RL-v1-7B","CodeLlama-7B","StarCoder2-7B","gemma-7b","StarCoder-15B","Llama3-8B-base","CodeGen-16B","Python-Code-13B","CodeT5+-16B","StarCoder2-3B","Zephyr β-7B","CodeGen-6B","CodeT5+-6B","stable-code-3B","DeepSeek-Coder-1.3B-base","Mistral-7B","gemma-7b-it","codegemma-2b","CodeT5+-2B","gemma-2b","CodeGen-2B","StarCoderBase-7B","gemma-1.1-2b-it","CodeGen2-16B","CodeGen2-7B","StarCoderBase-3B","gemma-2b-it","Vicuna-13B","CodeGen2-3B","InCoder-6.7B","SantaCoder-1.1B","StarCoderBase-1B","GPT-J-6B","InCoder-1.3B","Vicuna-7B","CodeGen2-1B","GPT-Neo-2.7B","PolyCoder-2.7B","StableLM-7B","zyte-1B","Narada AI Code","https://narada.ai/blog/introducing-narada-code-operatormini","Deepseek v3.2","WebTactix","https://paper-submission-anoymous.github.io/webtactix_introduction/","OpAgent","https://github.com/codefuse-ai/Oagent","ColorBrowserAgent","ColorBrowserAgent; ite-specific knowledge adaptation & progressive summarization for long-horizon tasks.","https://github.com/MadeAgents/browser-agent","Claude Code + GBOX MCP","GBOX AI","https://gbox.ai/","DeepSky Agent","Narada AI","IBM CUGA","https://github.com/cuga-project/cuga-agent","OpenAI Operator","OpenAI CUA; System card","https://openai.com/index/computer-using-agent/","Jace.AI","https://www.jace.ai/; Note from the developer of the work, see the comment of the cell","GLM-4-32B + Qwen2.5-VL-7B","WebChallenger","https://arxiv.org/abs/2606.10423","WebOperator + GPT-4o","WebOperator","https://arxiv.org/abs/2512.12692","ScribeAgent + GPT-4o","ScribeAgent; ScribeAgent is finetuned with proprietary data","https://arxiv.org/abs/2411.15004","AgentSymbiotic","AgentSymbiotic; Code","https://www.arxiv.org/abs/2502.07942","Learn-by-Interact","Learn-by-interact","https://arxiv.org/abs/2501.10893","AgentOccam-Judge","https://arxiv.org/abs/2410.13825","A3-Qwen3.5-9B","Agent-as-Annotators; Multimodal (AXTree + screenshots). SFT on 2,322 synthetic trajectories from Gemini 3 Pro. Evaluated using GenericAgent/BrowserGym.","https://arxiv.org/abs/2604.07776","WebPilot","WebPilot; No open source code or trajectory released from the work","https://www.arxiv.org/pdf/2408.15978","GUI-API Hybrid Agent","Beyond Browsing; Using both API and GUI","https://arxiv.org/pdf/2410.16464","Agent Workflow Memory","AWM","https://arxiv.org/pdf/2409.07429","SteP","SteP; High-level plans are derived by human","https://arxiv.org/abs/2310.03720","TTI","https://huggingface.co/sjunhongs/tti_webarena/tree/main","BrowserGym + GPT-4","BrowserGym; different observation representation","https://arxiv.org/pdf/2403.07718.pdf","AgentTrek-1.0-32B","AgentTrek","https://agenttrek.github.io/","GPT-4 + Auto Eval","Auto Eval & Refine","https://arxiv.org/abs/2404.06474","GPT-4o + Tree Search","Tree Search for LM Agents","https://jykoh.com/search-agents","AutoWebGLM","https://arxiv.org/abs/2404.03648","NNetNav","NNetscape; LLama 3.1-8B-instruct fine-tuned on NNetNav6k (a newer version of the dataset where the work keeps the best of 3 trajectories for each instruction, where we use a llama 3.1 70b as the reward model). The model is available here: https://huggingface.co/stanfordnlp/llama8b-nnetnav-wa","https://arxiv.org/abs/2410.02907","gpt-4-0613","GPT; when \"not achievable\" hint is not provided","https://arxiv.org/abs/2307.13854","gpt-4o-2024-05-13","GPT; when \"not achievable\" hint is provided","Qwen1.5-72B-Chat (Mixture B fine-tune)","Patel et al + 2024","https://arxiv.org/abs/2405.20309","gpt-3.5-turbo-16k-0613","GPT","Qwen-1.5-chat-72b","Gemini Pro","Llama3-chat-70b","Llama3","Synatra-CodeLLama7b","Synatra","https://arxiv.org/pdf/2409.15637","Lemur-chat-70b","Lemur","https://arxiv.org/abs/2310.06830","Agent Flan","https://arxiv.org/abs/2403.12881","CodeLlama-instruct-34b","Llama2","AgentLM-70b","Agent Tuning","https://arxiv.org/abs/2310.12823","Llama3-chat-8b","CodeAct Agent","CodeAct","AgentLM-13b","Mixtral","AgentLM-7b","FireAct","CodeLlama-instruct-7b","CodeLLama","Gemini 2.5 Flash","Multimodal (SoM); SoM; SGV","https://arxiv.org/abs/2507.11662","GPT-5","Multimodal (SoM); SoM; WALT","https://arxiv.org/abs/2510.01524","GLM-4-32B","Caption-augmented; Structured DOM Representation; WebChallenger","Multimodal (SoM); SoM + Caption + Image; AWorld","Multimodal; AXTree + Image; Agent-as-Annotators","GPT-4o + R-MCTS","Multimodal (SoM); SoM + Caption + Image; ExACT","https://agent-e3.github.io/ExACT/","GPT-4o + Search","Multimodal (SoM); SoM + Caption + Image; Tree Search for LM Agents","GPT-4o + WebDreamer","Multimodal (SoM); SoM + Caption + Image; WebDreamer","https://arxiv.org/abs/2411.06559","GPT-4o + ICAL","Multimodal (SoM); SoM + Caption + Image; ICAL","https://arxiv.org/abs/2406.14596","Multimodal (SoM); SoM + Caption + Image; VisualWebArena","https://arxiv.org/abs/2401.13649","Llama-3-70B + Search","Caption-augmented; AxTree + Caption; Tree Search for LM Agents","Multimodal; AXTree + Caption + Image; VisualWebArena","GPT-4 + BLIP-2-T5XL","Caption-augmented; AxTree + Caption; VisualWebArena","Gemini-Pro-1.5","ViGoRL-7B","Image-only; Image; ViGoRL","https://arxiv.org/abs/2505.23678","Llama-3-70B-Instruct + BLIP-2-T5XL","Qwen2-VL + ICAL","GPT-4","Text-only; AXTree; VisualWebArena","Gemini-Flash-1.5","ViGoRL-3B","Gemini-Pro","Qwen-VL-7B","Gemini-Pro + BLIP-2-T5XL","GPT-3.5 + BLIP-2-T5XL","GPT-3.5 + LLaVa-7B","GPT-3.5","Mixtral-8x7b + BLIP-2-T5XL","Mixtral-8x7B","IDEFICS-80B-Instruct","Llama-2-70B + BLIP-2-T5XL","CogVLM","AGI-AW","AGI-0; Screenshot; [09/2026]: 97.4 --> 100. Trained open source models using our novel training technique on AndroidWorld tasks. Code and model will be open-sourced upon publication.; trials: 1","GPT6 Astra","MobileRun; Screenshot + A11y tree; [09/30/2026] 91.4 --> Updated harness and using GPT-6 Astra [10/6/2025]: 78.4 --> 91.4; details here [09/19/2025]: Updated score from 63.0 --> 78.4. For this run, we used GPT-5 for reasoning in combination with Gemini-2.5-Pro for acting, while continuing to rely on a hybrid method of the Accessibility (A11y) tree plus corresponding screenshots to provide both structural and visual context to the agent.; trials: 1","gpt-4o / gpt-5.4","Kirk Engine; Screenshot + A11y tree; Code is not open-sourced yet.; trials: 1","https://abhishekdh.github.io/kirk-engine/","gpt-4o / gpt-5.6-sol","FluizAI; Screenshot + A11y tree; Code is not open-sourced yet.; trials: 1","https://android-world.fluiz.ai/","gpt-6-sol","agent-qa by Vostride AI; Screenshot + A11y tree; Note: Open-source self-improving QA agent with memory. Code is available at https://github.com/vostride/agent-qa. agent-qa's memory and self-improvement loop were switched off during the benchmark run.; trials: 1","https://vostride.com/benchmark/android-world","Gemini 3.7 Flash, Gemini Robotics ER 2","Artemis; Screenshot + A11y tree; Open-source project available at https://github.com/google/artemis; trials: 1","Gemini 3 flash, Gemini 3 flash lite","Finalrun; Screenshot + A11y tree; 04/2026: Project is open-source at https://github.com/final-run/finalrun-agent 03/2026: Updated results: https://blogs.finalrun.app/finalrun-achieves-the-new-1-sota-mobile-agent-eclipsing-benchmarks-at-97-4 08/2025: The technical doc/approach can be found here: https://github.com/final-run/finalrun-android-world-benchmark; trials: 1","https://drive.google.com/file/d/1NITEkiYb3itSq46I8BYsmzefEddzYbqm/view?usp=sharing","Seed1.8-GUI","MobileUseAgent; Screenshot; trials: 1","askui AndroidVisionAgent, Claude 4.5 Sonnet + Claude 4.0 Sonnet","askui AndroidVisionAgent; Screenshot; trials: 1","gemini 3 pro + sonnet 4.5","AutoDevice; Screenshot; Agent code is available on https://github.com/autodevice/android_world; trials: 1","https://autodevice.io/benchmark/","Gemini-3.5-Flash","Midscene.js; Screenshot; trials: 1","Llama 4-scout, Gemini 2.5 pro, GPT-5 nano","mobile-use; Screenshot + A11y tree; [12/16/2025] 84.5% -> 91.4% [10/1/2025]: 77.6% -> 84.5% [09/12/2025]: Updated score from 74.1% --> 77.6% [08/19/2025]: Initial trajectory labeling issue (MarkorCreateNoteFromClipboard) has been corrected by the authors. They provide Discord support for setup issues. [08/18/2025]: Repo has been reported broken by some users (not independently verified). Some trajectories (e.g., MarkorCreateNoteFromClipboard, BrowserMaze) are labeled as successful by the authors but contain incorrect actions; trials: 1","https://minitap.ai/benchmark","o3 + holo1.5-72b","Surfer 2; Screenshot; Agent is based on https://github.com/hcompai/surfer-h-cli - same architecture, but with modifications to prompt and action space; trials: 1","https://hcompai.github.io/android-world-traces/","Sonnet 4.5 + Sonnet 4","gbox.ai; Screenshot; GBOX uses Claude code as the agent with GBOX mcp. Link to report -> https://github.com/babelcloud/android_world_benchmark/blob/main/GBOX/report.md; trials: 1","https://github.com/babelcloud/android_world_benchmark/tree/main/GBOX/trajectory","AutoGLM-Mobile","AutoGLM-Mobile; Screenshot + A11y tree; [9/26/2025]: Updated score from 75.8 --> 80.2 This is an updated version of AutoGLM-Mobile. In this version, we added more pretraining data and trained the VLM at full image resolution.; trials: 1","LX-GUIAgent","LX-GUIAgent; Screenshot + A11y tree; [9/23/2025]: Updated score from 75.0 --> 79.3. Added trajectories.; trials: 1","Gemini-2.5-Pro+UI-TARS-1.5","AgentProg; Screenshot; Paper: https://arxiv.org/pdf/2512.10371; Code: https://github.com/MobileLLM/AgentProg; trials: 1","MAI-UI-235B-A22B","MAI-UI; Screenshot; trials: ","Qwen2.5-VL-72B + Qwen2.5-VL-7B","K²-Agent; Screenshot; K²-Agent, a hierarchical approach that self-evolves a Qwen2.5-VL-72B for high-level planning and post-trains a Qwen2.5-VL-7B for low-level execution. Our GitHub repository, which contains a detailed description of our approach, can be found here: https://github.com/k2-agent/k2-agent.; trials: 1","https://github.com/k2-agent/k2-agent/blob/main/androidworld-trajectories.zip","Hammer-UI-32B","MobileUse-v2; Screenshot; Make further post-training based on the GUI-Owl-32b model. Optimize the memory and knowledge module of the MobileUse framework.; trials: 1","GUI-Owl-32B","Mobile-Agent-v3; Screenshot; https://github.com/X-PLUG/MobileAgent/tree/main; trials: 1","MAI-UI-32B","MAI-UI-8B","Gemini 2.5 Computer Use","Gemini 2.5 Computer Use; Screenshot; trials: 1","JT-GUIAgent-V2","JT-GUIAgent-V2; Screenshot; trials: 1","GUI-Owl-7B","GUI-Owl-7B; Screenshot; https://github.com/X-PLUG/MobileAgent/tree/main; trials: 1","UI-Venus-Navi-72B","UI-Venus; Screenshot; https://huggingface.co/inclusionAI/UI-Venus-Navi-72B; trials: 1","https://github.com/inclusionAI/UI-Venus/blob/main/vis_androidworld/UI-Venus-androidworld.zip","Qwen2.5-VL-72B","MobileUse; Screenshot; trials: 1","Seed1.5-VL","Seed1.5-VL; Screenshot + A11y tree; trials: 1","JT-GUIAgent-V1","JT-GUIAgent-V1; Screenshot; trials: 1","V-Droid (Llama8B)","V-Droid Paper; A11y tree; Training data consists of apps and tasks from the AndroidWorld benchmark. Code.; trials: 1","Agent S2","Agent S2; Screenshot; trials: 1","MAI-UI-2B","Venus-Navi-7B","UI-Venus; Screenshot; https://huggingface.co/inclusionAI/UI-Venus-Navi-7B; trials: 1","GUI-Explorer; Screenshot + A11y tree; trials: 1","AndroidGen; A11y tree; trials: 1","UI-TARS","UI-TARS; Screenshot; trials: 1","GPT-4o + Aria-UI","Aria-UI; Screenshot; trials: 1","GPT-4o + UGround","UGround; Screenshot; Code for reproduction; trials: 1","ScaleTrack-7B","ScaleTrack; A11y tree; trials: 1","Mirage-1; Screenshot; With Mirage-1-O; uses OS-Atlas grounder; trials: 1","Ponder & Press; Screenshot; Code is not yet open-sourced; trials: 1","GPT-4 Turbo","AndroidWorld; A11y tree; trials: 1","GPT-4o, OS-Atlas-Pro 4B, Qwen2-VL-2B-Instruct","EcoAgent; Screenshot; trials: 1","Qwen-2.5-VL-7B","GUI-Critic-R1; Screenshot + A11y tree; trials: 1","Qwen2-VL-2B (fine-tuned)","InfiGUIAgent; Screenshot; trials: 1","Gemini-SQL2","Google Research & Cloud","Single-model track; self-consistency: Many","ReToolSQL","JPMorganChase","https://arxiv.org/pdf/2608.27796","SiriusAI-SQL-27B","Tencent Data Computing Platform Department","Gemini-SQL (Multitask SFT + Gemini-2.5-Pro)","Google Cloud Databases","https://cloud.google.com/blog/products/databases/how-to-get-gemini-to-deeply-understand-your-database","SiriusAI-SQL-9B","Native mini","Native","Q-SQL","AWS-Quick Science","https://www.notion.so/BIRD-v2-Memory-2bfe8bf30514801ba112f92ff20784e3","Databricks RLVR 32B","Databricks Mosaic Research","Single-model track; self-consistency: Few","https://www.databricks.com/research","Spektr-SQL","Amazon Ads-SpektrBot","https://www.notion.so/BIRD-Submission-SQL-291e8bf3051480afa7b8cfac3aa26b10","Sophon-Text2SQL-32B","ByteDance DataPlatform-LLM","ELG-SQL","AMS CIR","Velldepth-ZenPivot-27B","Alibaba Security","Kwai-AutoSQL-32B","Kuaishou Business Team","Kwai-AutoSQL-14B","Arctic-Text2SQL-R1-32B","Snowflake","https://www.snowflake.com/en/product/ai/ai-research/","flash-sqlite: Qwen3.5-2B Generator + Self-Judge","Alio Leuchtmann","Single-model track; self-consistency: ","Jiayin-Pangu-Text2SQL-14B","Jiayin AI Research Team","FTEL-Text2SQL","FPT Telecom","GPT-5.5-xhigh","Arctic-Text2SQL-R1-14B","Fractal-SQL","Fractal Analytics","https://fractal.ai/","SQLWeaver-32B","Alibaba AI Data","ShepherdSQL","JD Bitcircuit Research","JD-UNION-Text2sql-32B","JD UNION","SIFT-32B","Seek AI","Single-model track; self-consistency: Scale","https://www.seek.ai/research","CrazyData-Text2SQL-32B","Infly-RL-SQL-32B","inftech.ai","Arctic-Text2SQL-R1-7B","Claude Opus 4.6","XiYanSQL-QwenCoder-32B","https://arxiv.org/abs/2507.04701","Arctic-ExCoT-70B","https://arxiv.org/abs/2503.19988","Qwen3.5-2B + StructCoT Distillation","Arctic-ExCoT-32B","Qwen3-Coder-480B-A35B","Align-SQL-7B","Anonymous","Claude 4.5 Sonnet","Align-SQL-3B","Command A","Cohere AI","https://arxiv.org/abs/2504.00698","OneSQL-v0.1-Qwen-32B","onekq.ai","GLM-4.7","SuperSQL","HKUST(GZ)","https://arxiv.org/abs/2406.01265","Struct-SQL","Crater Labs","https://arxiv.org/abs/2512.17053","SFT CodeS-15B","Renmin University of China","https://arxiv.org/abs/2402.16347","Kimi-K2-Thinking","xorazm-text2sql-0.8b","Martin Luther University Halle-Wittenberg","https://huggingface.co/shakha-de/xorazm-text2sql-0.8b","SFT CodeS-7B","Prem-1B-SQL","Prem AI","Claude-2","ChatGPT + CoT","HKU & DAMO","https://arxiv.org/abs/2305.03111","ChatGPT","Codex","sqlpup (7-sample vote)","https://github.com/shivenkk/sqlpup","Palm-2","sqlpup (single pass)","T5-3B","T5-Large","T5-Base","Tianqiong Data Agent + GLM 5.2","Data Computing Platform Department, Tencent","Spider 2.0-Lite","DecisionX Agent","DecisionX AI","https://www.decisionx.ai/","ktx","Kaelio","https://github.com/Kaelio/ktx","DivSkill-SQL","Snowflake AI Research x UCSD","SOMA-SQL","Oracle Cloud Infrastructure, AI Science team","https://arxiv.org/abs/2606.11424","Databao Agent","Databao by JetBrains","https://databao.app/","QUVI-2.3 + Claude-Opus-4.5","DAQUV","https://daquv.com/en","EXA-SQL","MJ Park x MS Sim","nQuery","nGenux","https://www.ngenux.com/","Claude Code Agent + Sonnet","Nodal Data","ProSPy: Profiling-driven SQL-Python Analysis Framework + Claude-4.5-Opus","ReFoRCE + o3","Hao AI Lab x Snowflake","https://arxiv.org/abs/2502.00675","CoFD-SQL + GPT-5","Samsung SDS Research America","CT-ChatBI + o4-mini","CT-ChatBI Team","AutoLink + Deepseek-R1","HUST VLR Lab","https://arxiv.org/pdf/2511.17190","GRASP + Reforce + Gemini-3-Flash + DeepSeek-V3.2","AgenticData + Qwen3","Tsinghua University","https://arxiv.org/abs/2508.05002","AI-DIVE + DeepSeek-V4-Flash","S-Core","https://s-core.co.kr/en/","Spider-Agent + Claude-Sonnet-4.5","Myoseop Sim","Dr.Q","Dr.Q Lab","ReFoRCE + Qwen3","LinkAlign + DeepSeek-R1","https://arxiv.org/abs/2503.18596","RSL-SQL + o3","https://arxiv.org/abs/2411.00073","YAN-Agent + GLM-5","Kuaishou","RSL-SQL + DeepSeek-R1","ReFoRCE + o1-preview","Spider-Agent + Claude-3.7-Sonnet-20250219-Thinking","Spider-Agent + Claude-4-Sonnet-20250514","RSL-SQL + DeepSeek-V3","Spider-Agent + Claude-3.7-Sonnet-20250219","LinkAlign + DeepSeek-V3","Spider-Agent + o3-mini-2025-01-31","Spider-Agent + o1-preview","AI-DIVE + gpt-oss-120B","Spider-Agent + DeepSeek-R1","Spider-Agent + GPT-4o-2024-11-20","Spider-Agent + QwQ-32B","Duo","Spider-Agent + Claude-3.5-Sonnet-20240620","Spider-Agent + Qwen2.5-Coder-32B-Instruct","DailSQL + GPT-4o","CHESS + GPT-4o","DIN-SQL + GPT-4o","GroundCascade (Indeed-UI-32B + KV-Ground-8B)","Training-free geometric consistency cascade; Indeed-UI-32B primary/verifier + conditional KV-Ground-8B helper; fixed 0.5 zoom; 1% self-consistency gate; 4% cross-model agreement gate; greedy decoding; 2.574 average model calls/example.","GroundCascade (Indeed-UI-32B + Indeed-UI-8B)","Training-free geometric consistency cascade; Indeed-UI-32B primary/verifier + conditional Indeed-UI-8B helper; fixed 0.5 zoom; 1% self-consistency gate; 4% cross-model agreement gate; greedy decoding; 2.572 average model calls/example.","Indeed-UI-32B-zoomin","Indeed-UI-8B-zoom-in","KV-Ground-8B + Qwen3.5-27B Consistency Router","Heterogeneous ensemble of KV-Ground-8B-BaseGuiOwl1.5-0315 and Qwen3.5-27B-AWQ-4bit. Both models run independent 2-step zoom-in pipelines (crop_ratio=0.5). Per sample, the model with lower zoom consistency (step-2 prediction closer to crop center = higher confidence) is selected. KV-Ground selected for 64.9% of samples, Qwen3.5-27B for 35.1%.","KV-Ground-GuiOwl1.5-0315-8B-ZoomIn","Holo2-235B-A22B (Agentic)","MAI-UI-32B (MVP)","MVP on attention layer 48; max inference process is 2.","AdaZoom-GUI-4B (cond. zoom-in, refine w/ Qwen3.5-397B)","Conditional zoom-in and refine with Qwen3.5-397B-A17B","KV-Ground-GuiOwl1.5-4B-0228-ZoomIn","Holo2-30B-A3B (Agentic)","MVP_Qwen3VL-32B","MVP crops different regions from original screenshot to get different prediction and then cluster them through k-means. The final result is decided by the centroid coordinate of the largest cluster.","MAI-UI-32B (Zoom In)","Indeed-UI-8B","Indeed-UI-32B","KV-Ground-GuiOwl1.5-0315-8B","Duvo Eye-1 (Holo-3.1-35B-A3B + LoRA)","Holo-3.1-35B-A3B with a LoRA adapter (rank 64, alpha 128, 1 epoch) trained on duvoai/SynthUI (14.9k rows). Single-shot grounding, temperature 0, thinking disabled, guided-JSON {\"x\",\"y\"} in [0,1000] scaled to absolute pixels. Self-reported; per-sample predictions published for rescoring at https://huggingface.co/datasets/duvoai/duvo-eye-1-evals","HuzzleWorld-3B (Zoom In)","Proprietary","MAI-UI-8B (Zoom In)","Holo2-8B (Agentic)","HuzzleWorld-2B (Zoom In)","AdaZoom-GUI-4B (conditional zoom-in)","Holo2-235B-A22B","KV-Ground-Qwen3VL-4B-ZoomIn","UI-Venus-1-5-30B-A3B","Holo2-4B (Agentic)","UI-Venus-1-5-8B","KV-Ground-GuiOwl1.5-0228-4B","HuzzleWorld-3B","Holo2-30B-A3B","MVP_Qwen3VL-8B","GTA1-32B","HuzzleWorld-1B (Zoom In)","Holo1.5-72B","KV-Ground-Qwen3VL-4B","MAI-UI-2B (Zoom In)","UI-Venus-72B","HuzzleWorld-2B","AdaZoom-GUI-4B","UI-TARS-1.5","Seed-1.5-VL","GUI-ARP-7B","GUI-AIMA-3B","Holo2-8B","GTA1-Qwen2.5VL-72B","Holo1.5-7B","BAMI-7B","A test-time inference method for GUI grounding that uses coarse-to-fine focus and candidate selection. Base model: TianXi-Action-7B.","UI-Venus-1-5-2B","Holo2-4B","HuzzleWorld-1B","GTA1-7B","GTA1-Qwen2.5VL-32B","Qwen2.5-VL-72B-Instruct","GUI-Spotlight","A think-with-image GUI visual grounding model.","V2P-7B","TianXi-Action-7B","Holo1.5-3B","Kimi-VL-A3B-Thinking-2506","UI-Venus-7B","GTA1-Qwen2.5VL-7B","Qwen2.5-VL-32B-Instruct","UI-AGILE-7B","UI-AGILE enhances GUI agents through improved training with a Continuous Reward function, and Cropping-based Resampling, and inference with Decomposed Grounding with Selection.","SE-GUI-7B","UI-AGILE-3B","GUI-Actor-2.5VL-7B","GUI-Actor-2.5VL-3B","GUI-Actor-2VL-7B","UI-TARS-72B","GUI-Actor-2VL-2B","GUI-G1","SE-GUI-3B","UI-TARS-7B","UGround-v1-72B","Di-GUI-3B","UGround-V1-7B","ZonUI-3B","UI-TARS-2B","Qwen2.5-VL-7B-Instruct","UGround-V1-2B","Aguvis-7B","OS-Atlas-7B","GPT5-minimal (resized)","Claude (Computer Use)","UGround-7B","Qwen2.5-VL-3B-Instruct","AriaUI (MOE, 3.9B active)","CogAgent (18B)","ShowUI (2B)","GPT5-high (resized)","OS-Atlas-4B","MiniCPM-V (7B)","Qwen2-VL-7B","SeeClick (7B)","Qwen2-VL-72B-Instruct","DreamPRM (o4-mini)","Reason 🧠","VL-Rethinker","Step R1-V-Mini","Kimi-k1.6-preview-20250308","Doubao-pro-1.5","Ovis2_34B","LMM 🖼️","Kimi-k1.5","OpenAI o1","Vision-R1-7B","QVQ-72B-Preview","Qwen2VL-72B","Pixtral Large (124B)","Grok-2","Grok-2 mini","InternVL2-8B-MPO","Ovis1.6-Gemma2-9B","InternVL2-Pro","Chimera-Reasoner-8B","TextGrad (GPT-4o)","TBAC-VLR1-3B-preview","Gemini 1.5 Pro (May 2024)","Phi-4-multimodal-ins","InternVL-Chat-V1.2-Plus","Gemini 1.5 Flash (May 2024)","GPT-4T 2024-04-09","Pixtral-12B","InternLM-XComposer2-VL-7B","Gemini 1.0 Ultra","Grok-1.5V","Gemini 1.5 Pro (Feb 2024)","Claude 3 Opus","GPT-4V (Playground)","Claude 3 Sonnet","InternVL-Chat-V1.2","Math-LLaVA-13B","LLaVA-NeXT-34B","Claude 3 Haiku","Gemini 1.0 Pro","Phi-3-Vision-128K-In","Phi-3.5-Vision 4.2B","Mini-Gemini-HD (Hermes-2-Yi-34B)","Qwen-VL-Plus","SPHINX-MoE","MoE 🤖","Mini-Gemini (Mixtral-8x7B)","MM1-7B-MoE-Chat","MiniCPM-V-2 (2.8B)","MM1-30B-Chat","SPHINX-Plus","SPHINX (V2)","MM1-7B-Chat","SPHINX-Intern2","OmniLMM-12B","Multimodal Bard","LLaVA-NeXT-Vicuna-7B","PoT GPT-4 (Caption+OCR)","Tool 🛠️","CoT ChatGPT (Caption+OCR)","CoT Claude (Caption+OCR)","CoT GPT4 (Caption+OCR)","MM1-3B-MoE-Chat","MM1-3B-Chat","Gemini 1.0 Nano 2","LLaVA-1.5-13B","SPHINX (V1)","Gemini 1.0 Nano 1","PoT ChatGPT (Caption+OCR)","SPHINX-Tiny","LLaVA (LLaMA-2-13B)","InstructBLIP (Vicuna-7B)","LLaVAR","LLaMA-Adapter-V2 (7B)","miniGPT4 (LLaMA-2-7B)","mPLUG-Owl (LLaMA-7B)","IDEFICS (9B-Instruct)","GPT-5.4 (xhigh reasoning, w/ Python) (3rd-party eval)","Full test set (3,040 problems); settings in result name","https://www.kimi.com/blog/kimi-k2-6","Gemini 3.1 Pro (thinking high, w/ Python) (3rd-party eval)","Kimi K2.6 (w/ Python)","GPT-5.4 (xhigh reasoning) (3rd-party eval)","Gemini 3.0 Flash (3rd-party eval)","https://console.volcengine.com/ark/region:ark+cn-beijing/model/detail?Id=doubao-seed-2-0-lite","Doubao-Seed-2.0-lite-0428","Gemini 3.1 Pro (thinking high) (3rd-party eval)","Doubao-Seed-2.0-pro-0215","Qwen3.5-397B-A17B","https://qwen.ai/blog?id=qwen3.5","Kimi K2.6 (no tools)","Gemini 3 Pro (3rd-party eval)","https://huggingface.co/blog/mlabonne/qwen35","Doubao-Seed-2.0-lite-0215","Qwen3.5-122B-A10B","https://huggingface.co/Qwen/Qwen3.5-27B","Qwen3.5-27B","Gemma 4 31B","https://huggingface.co/blog/gemma4","Kimi K2.5 (thinking, w/ Python)","Claude Opus 4.6 (max effort, w/ Python) (3rd-party eval)","https://build.nvidia.com/moonshotai/kimi-k2.5/modelcard","Qwen3.5-35B-A3B","GPT-5.2 (3rd-party eval)","https://arxiv.org/html/2602.02276v1","Gemma 4 26B A4B","Seed-1.8","https://lf3-static.bytednsdoc.com/obj/eden-cn/lapzild-tss/ljhwZthlaukjlkulzlp/research/Seed-1.8-Modelcard.pdf","Seed 1.6-Thinking","Claude Opus 4.5 (3rd-party eval)","Step3-VL-10B (PaCoRe)","https://arxiv.org/abs/2601.09668","Qwen3-VL-235B-A22B-Thinking","https://arxiv.org/abs/2511.21631","Gemini 2.5 Pro","https://deepmind.google/technologies/gemini/pro/","https://openai.com/gpt-5/","GPT-5-mini (2025-08-07) (3rd-party eval)","Claude Opus 4.6 (max effort) (3rd-party eval)","Claude Sonnet 4.5 (3rd-party eval)","Step3-VL-10B (SeRe)","dots.vlm1","https://github.com/rednote-hilab/dots.vlm1","https://github.com/ByteDance-Seed/Seed1.5-VL/","Claude Opus 4.1 (thinking) (3rd-party eval)","Qwen3-VL-235B-A22B-Instruct","GLM-4.5V (106B-A12B)","https://huggingface.co/zai-org/GLM-4.5V","Step-3 (321B-A38B)","https://stepfun.ai/research/en/step3/","InternVL3.5 (241B-A28B)","https://huggingface.co/OpenGVLab/InternVL3_5-241B-A28B","GLM-4.6V (106B-A12B)","https://arxiv.org/abs/2507.01006","MiMo-VL-RL","https://github.com/XiaomiMiMo/MiMo-VL/","MiMo-VL-RL-2508","https://arxiv.org/abs/2506.03569","Qwen3-VL-8B-Thinking","Gemma 4 E4B","Claude 3.7 Sonnet (3rd-party eval, Skywork)","https://huggingface.co/Skywork/Skywork-R1V3-38B","OpenAI o4-mini (3rd-party eval)","https://huggingface.co/Skywork/Skywork-R1V2-38B-AWQ","MiMo-VL-7B-SFT","https://arxiv.org/pdf/2504.07491","https://platform.stepfun.com","InternVL3.5 (30B-A3B)","https://huggingface.co/OpenGVLab/InternVL3_5-30B-A3B","SenseNova V6 Reasoner","https://platform.sensenova.cn/","GLM-4.1V (9B)","https://huggingface.co/zai-org/GLM-4.1V-9B-Thinking","GLM-4.6V-Flash (9B)","InternVL3.5-38B","https://arxiv.org/html/2508.18265v1","AStar-7B (training-free, Qwen2.5-VL-7B)","https://arxiv.org/abs/2502.02339","Ovis2.5-9B","https://arxiv.org/abs/2508.11737","Kimi k1.6 Preview","https://github.com/MoonshotAI/Kimi-k1.5","Decoupled LLM-LMM (Qwen2.5-VL-72B + Qwen3-32B)","https://arxiv.org/abs/2509.23322","Skywork-R1V3-38B","Gemma 4 E2B","InternVL3.5-8B","Open-Vision-Reasoner-7B","https://arxiv.org/abs/2507.05255","Qianfan-VL-70B","https://baidubce.github.io/Qianfan-VL","Skywork-R1V2-38B","https://github.com/SkyworkAI/Skywork-R1V","Doubao-1.5-pro","https://team.doubao.com/zh/special/doubao_1_5_pro","Gemini 2.0 Pro (3rd-party eval)","https://internvl.github.io/blog/2025-04-11-InternVL-3.0/","https://openai.com/index/introducing-gpt-4-5/","Gemma 3 27B (no think)","Keye-VL-8B","GPT-5 (minimal) (3rd-party eval)","VL-Rethinker-72B","https://tiger-ai-lab.github.io/VL-Rethinker/","LLaVA-Critic-R1 (Qwen2.5-VL-7B base)","https://arxiv.org/abs/2509.00676","Vision-R1-7B (contamination-flagged)","https://arxiv.org/abs/2503.06749","InternVL3-78B","https://arxiv.org/abs/2504.10479","Skywork-R1V2-38B-AWQ","INFRL-Qwen2.5-VL-72B-Preview","https://huggingface.co/infly/INFRL-Qwen2.5-VL-72B-Preview","Gemini-2 Flash","ProxyThinker (Qwen2.5-VL-32B + OpenVLThinker-7B expert)","https://arxiv.org/abs/2505.24872","VL-Rethinker-32B","https://arxiv.org/abs/2504.08837","ViCrit-RL-72B","https://arxiv.org/abs/2506.10128","InternVL3.5-4B","Kimi k1.5","https://arxiv.org/pdf/2501.12599","Virgo-72B","https://arxiv.org/abs/2501.01904","Claude3.5-Sonnet","https://www.anthropic.com/news/claude-3-5-sonnet/","AVAR-Thinker-7B","https://arxiv.org/abs/2603.03825","Ovis2.5-2B","InternVL3-14B","Kimi-VL-A3B-Thinking","https://arxiv.org/abs/2504.07491","Phi-4-reasoning-vision-15B (testmini)","https://huggingface.co/microsoft/Phi-4-reasoning-vision-15B","QvQ-72B-Preview","https://qwenlm.github.io/blog/qvq-72b-preview/","URSA-8B + URSA-RM (BoN=32)","https://arxiv.org/abs/2501.04686","InternVL3-38B","ThinkLite-VL-7B","https://arxiv.org/abs/2504.07934","Qianfan-VL-8B","InternVL2.5-78B","https://huggingface.co/OpenGVLab/InternVL2_5-78B","Ovis2-34B","https://github.com/AIDC-AI/Ovis","InternVL2.5-38B","https://huggingface.co/OpenGVLab/InternVL2_5-38B","URSA-8B-PS-GRPO","TBAC-VLR1-7B","https://arxiv.org/abs/2509.03321","LLaVA-Critic-R1 (LLaMA-3.2-11B-V base)","https://platform.openai.com/docs/models/gpt-4-and-gpt-4-turbo","DualMindVLM-7B","https://arxiv.org/abs/2511.16670","MMR1-Math-v0-7B","https://huggingface.co/MMR1/MMR1-Math-v0-7B","Ovis2-16B","R1-Onevision-7B","https://arxiv.org/abs/2503.10615","InternVL3-8B","VOLD (Qwen2.5-VL-3B, text-only RL)","https://arxiv.org/abs/2510.23497","InternVL3-9B","https://huggingface.co/OpenGVLab/InternVL3-9B","Claude3-Opus","https://www.anthropic.com/news/claude-3-family/","MM-Eureka-Qwen-7B","https://arxiv.org/abs/2503.07365","Qwen2.5-VL-DP-7B (MathV-DP)","https://arxiv.org/abs/2507.02804","Vision-SR1-7B","https://arxiv.org/abs/2508.19652","MathGLM-Vision-32B","https://arxiv.org/abs/2409.13729","VLAA-Thinker-Qwen2.5VL-7B","https://arxiv.org/abs/2504.11468","MathCoder-VL-8B","https://github.com/mathllm/MathCoder/","OpenVLThinker-7B","https://arxiv.org/abs/2503.17352","Ovis2-8B (testmini)","Qwen2-VL-72B","https://qwenlm.github.io/blog/qwen2-vl/","LLaVA-OneVision-72B","https://huggingface.co/llava-hf/llava-onevision-qwen2-72b-ov-hf","Qwen2.5-VL-7B","https://huggingface.co/TencentBAC/TBAC-VLR1-3B-preview","R1-VL-7B (contamination-flagged)","https://arxiv.org/abs/2503.12937","X-Reasoner-3B (text-only)","CoT GPT4V","https://arxiv.org/abs/2310.02255","R1-Onevision-3B","https://huggingface.co/openbmb/MiniCPM-V-2_6","InternVL2.5-26B","https://huggingface.co/OpenGVLab/InternVL2_5-26B","GPT4V","InternVL3-2B","https://huggingface.co/OpenGVLab/InternVL3-2B","MathCoder-VL-2B","https://huggingface.co/openbmb/MiniCPM-o-2_6","Kimi-VL (base, non-thinking)","Qwen2.5-VL-3B","Multimath-7B","https://arxiv.org/abs/2409.00147","InternVL2.5-8B","https://huggingface.co/OpenGVLab/InternVL2_5-8B","Mulberry-Qwen2VL-7B (contamination-flagged)","https://arxiv.org/abs/2412.18319","Gemini-1.5 Pro","https://storage.googleapis.com/deepmind-media/gemini/gemini_v1_5_report.pdf","InternVL3-1B","https://huggingface.co/OpenGVLab/InternVL3-1B","MAVIS-7B","https://arxiv.org/abs/2407.08739","Aquila-VL-2B","https://huggingface.co/BAAI/Aquila-VL-2B-llava-qwen","Ovis2-2B (testmini)","Qwen2-VL-DP-7B (MathV-DP)","https://arxiv.org/abs/2312.11805","InternVL-Chat-V1-2-Plus","https://github.com/OpenGVLab/InternVL","https://github.com/HZQ950419/Math-LLaVA","https://github.com/QwenLM/Qwen-VL","InternVL2.5-2B","https://huggingface.co/OpenGVLab/InternVL2_5-2B","InternLM-XComposer2-VL","https://github.com/InternLM/InternLM-XComposer","InternVL2.5-1B","https://huggingface.co/OpenGVLab/InternVL2_5-1B","GPT 4-CoT (caption)","Qwen2-VL-2B","ShareGPT4V-13B","https://sharegpt4v.github.io/","LLaVA-v1.5-13B","https://llava-vl.github.io/","ShareGPT4V-7B","https://github.com/Alpha-VLLM/LLaMA2-Accessory/tree/main/SPHINX","LLaVA-v1.5-7B","Validation; reasoning","o4 mini (high)","Claude 3.7 Sonnet","GPT 4.1 mini","GPT 4.1","GPT 4.5","o1 (high)","Doubao 1.5 Pro","o1","GPT-4o 241120","Qwen2.5-VL 72B","GPT-4o 240513","InternVL3 38B","InternVL3 78B","InternVL3 14B","Qwen2-VL 72B","Qwen2.5-VL 7B","InternVL V2.5 38B","InternVL V2.5 78B","Pixtral 12B","GPT 4.1 nano","InternVL Chat V2.0 Pro","InternVL Chat V2.0 76B","InternVL3 9B","InternVL3 8B","Llama 3.2 Vision 90B","NVLM-D 72B","Molmo 72B","Llava OneVision 72B","InternVL V2.5 26B","Qwen2-VL 7B","GPT-4o Mini","InternVL Chat V2.0 26B","InternVL V2.5 8B","Phi-3.5 Vision","Phi-3 Vision","Qwen2.5-VL 3B","Llama 3.2 Vision 11B","MiniCPM-V2.6 (Upsize+CoT)","InternVL Chat V1.5","GLM 4V 9B","Reka Core","Ovis 1.5 Gemma2 9B","InternVL3 2B","Ovis 1.5 Llama3 8B","Cambrian 34B","MiniCPM-V2.6 (Upsize)","Reka Flash","Molmo 7B D","InternLM XComposer2 4KHD","Mini Gemini HD Yi 34B","InternVL V2.5 4B","MiniCPM-V2.5","Qwen VL Max","Qwen2-VL 2B","Molmo 7B O","VILA 1.5 40B","Llava OneVision 7B","Reka Edge","LLaVA 1.6 Yi 34B","InternVL V2.5 2B","InternVL3 1B","InternVL V2.5 1B","Mini Gemini HD Llama3 8B","CogAgent","InternLM XComposer2","MiniCPM-V2","IDEFICS 2","MolmoE 1B","IDEFICS 2 Chatty","MoAI","DeepSeek VL","DocOwl 1.5 Chat","SPHINX V2","Qwen VL Plus","Llava OneVision 0.5B","UReader","ChartLlama","LLaVA 1.6 Mistral 7B","ChartGemma","ChartAssistant","ChartInstruct-FlanT5","DocOwl 1.5 Omni","ChartInstruct-Llama2","TinyChart","UniChart-ChartQA","TextMonkey","Validation; descriptive","Llama Nemotron Nano VL 8B","Published task-average score (0–100)","InternVL-3-14B","InternVL-3-8B","Ovis2-8B","InternVL2-26B","Step-1V","Claude3.5-sonnet","MiniCPM-o-2.6","SAIL-VL-1.6-8B","Deepseek-VL2-Small","GLM-4V-Plus","GLM-4V-9B","Kimi-VL-A3-16B","Ovis1.6-3B","MiniCPM-V-2.6","InternVL2-8B","LLaVA-OV-7B","Phi-4-MultiModal","Cambrian-1-8B","Molmo-7B","Idefics3-8B","LLaVA-Next-8B","XComposer2-4KHD","Eagle-X5-7B","Deepseek-VL-7B","VILA1.5-8B","Qwen-VL-chat","Qwen-VL","Monkey","CogVLM-chat","DocOwl2","EMU2-chat","Janus-1.3B","Yi-VL-6B","TextHarmony","Kimi-VL-A3B-16B","ViLA1.5-8B","Epoch; anti-abstention prompt (2026-08-27 onward)","gemini-3-flash-preview_high","muse-spark-1.2_xhigh","muse-spark-1.1","qwen3.6-max-preview","claude-opus-4-7_xhigh","grok-4.6_high","gpt-5.1-2025-11-13_high","gpt-5.2-2025-12-11_high","kimi-k2.5","claude-sonnet-5_xhigh","claude-sonnet-4-6_max","gpt-5.2-2025-12-11_low","gpt-5.2-2025-12-11_medium","glm-4.7","claude-sonnet-4-5-20250929_59K","gpt-5.4-mini-2026-03-17_high","qwen3.5-plus","claude-3-opus-20240229","claude-haiku-4-5-20251001_32K","gemma-4-31b-it","gpt-4.1-nano-2025-04-14","Claude Fable 5.1","LiveBench 2.0; 2026-10-25; official weighted overall; reasoning effort: max","GPT-6 Astra","Claude Opus 5.5","GPT-6.1 Sol","Claude Sonnet 5.5","Gemini 4 Argon","LiveBench 2.0; 2026-10-25; official weighted overall; reasoning effort: high","GPT-6.1 Terra","Muse Wonder 1.0","DeepSeek V4.1 Flash 1017","Kimi K3.1 Pro","Grok 4.7","LiveBench 2.0; 2026-10-25; official weighted overall; reasoning effort: xhigh","Claude Haiku 5.5","MiMo V2.6 Pro","Xiaomi","GPT-5.6 Terra","Grok 4.6","MiniMax M3.1 Flash","GPT-6 Luna","Kimi K3","Mistral Large 4 1029","Muse Spark 1.3","GLM-5.3 Flash","Z.AI","Qwen3.8 Max","Gemini 3.8 Flash","Step 5 (Preview)","StepFun","Qwen3.8 27B","Muse Spark Open","Smaug-Nemotron-3.5-NVFP4","Abacus.AI","Nemotron 3.5 Super VL","Nemotron 3.5 Super VL (NVFP4)","Mistral Large 4","GPT-5-2025-08-07 (high)","OpenCompass; Chat","JT-Think","TeleChat-thinking","TeleAI","Kimi-K2.5","Step-3.5-Flash","o4-mini-2025-04-16 (high)","GPT-5-mini-2025-08-07 (high)","Gemini-3-Pro-Preview","o3-2025-04-16","DeepSeek-V3.2-Speciale","MiniMax-M2.5","GPT-5-nano-2025-08-07 (high)","LongCat-Flash-Chat","MiniMax-M2","DeepSeek-V3.2","MiMo-V2-Flash","Qwen3-Next-80B-A3B-Thinking","ERNIE-4.5-300B-A47B-PT","Baidu","gpt-oss-20b (high)","Finix-P1-32B (Thinking)","GLM-4.6","Qwen3-4B-Thinking-2507","Claude Sonnet 4 (Thinking)","GPT-4.1-20250414","Qwen3-Next-80B-A3B-Instruct","Llama4-Maverick-17B-128E-Instruct","Qwen3-32B (Thinking)","Qwen3-8B (Thinking)","Qwen3-30B-A3B-Instruct-2507","Seed-OSS-36B-Instruct","ByteDance","Qwen3-4B-Instruct-2507","ERNIE-4.5-21B-A3B-Thinking","Gemma-3-27B-it","doubao-seed-1.6-thinking-250615","MiniCPM-SALA","Hunyuan-A13B-Instruct","Tencent","Lingzhi-72B-chat","5-shot; publisher README","Telechat2-35B","Spark 4.0-2024-10-14","Qwen2-72B","Jiutian-大模型","JIUTIAN-57B","BlueLM","Youyuanjian","Qwen1.5-72B","Mind GPT","PCI-TransGPT","Qwen1.5-32B","ZW-LM","QuarkLLM","ZhiLu-2-8B","BlueLM-7B","Galaxy","Qwen1.5-7B","KwaiYii-66B","XuanYuan-70B","FanFan-1.5B","Mengzi-7B","KwaiYii-13B","XuanYuan-13B","Qwen-7B","MiLM-6B","ZhiLu-13B","Baichuan-13B","Baichuan","ChatGLM2-6B","MiLM-1.3B","Baichuan-7B","Falcon-40B","LLaMA-65B","ChatGLM-6B","BatGPT-15B","BLOOMZ-7B","Chinese-LLaMA-13B","Bactrian-LLaMA-13B","MOSS-SFT-16B","Claude Opus 4.6 (high)","OpenClaw 2026.4.1","GPT-5.5 (xhigh)","GPT-5.4 (high)","Gemini 3.1 Pro (high)","Claude Sonnet 4.6 (high)","GLM 5.1 (enabled)","OpenClaw 2026.4.1; via OpenRouter, no round-trip reasoning context","Kimi-K2.5 (enabled)","GPT-6 Astra - Provider Adapter (High)","GPT-6 Astra - Provider Adapter (Max)","GPT-6 Astra - Provider Adapter (XHigh)","GPT-6 Astra - Provider Adapter (Medium)","GPT-6 Astra - Provider Adapter (Low)","Gemini 4 Argon - Provider Adapter (Low)","Gemini 4 Argon - Provider Adapter (Medium)","Gemini 4 Argon - Provider Adapter (High)","GPT-6 Astra - Provider Adapter (None)","GPT-6.1 Sol - Provider Adapter (XHigh)","GPT-6.1 Sol - Provider Adapter (Max)","GPT-6.1 Sol - Provider Adapter (High)","GPT-6.1 Sol - Provider Adapter (Medium)","GPT-6.1 Sol - Provider Adapter (Low)","Claude Sonnet 5.5 - Provider Adapter (XHigh)","Claude Sonnet 5.5 - Provider Adapter (High)","Gemini 3.8 Flash - Provider Adapter (High)","Gemini 3.8 Flash - Provider Adapter (Medium)","GPT-6 Sol - Provider Adapter (Max)","Gemini 3.8 Flash - Provider Adapter (Low)","Grok 4.7 - Provider Adapter (High)","GPT-6 Sol - Provider Adapter (XHigh)","Grok 4.7 - Provider Adapter (Medium)","Grok 4.7 - Provider Adapter (XHigh)","GPT-6 Sol - Provider Adapter (High)","GPT-6 Sol - Provider Adapter (Medium)","Claude Sonnet 5.5 - Provider Adapter (Medium)","DeepSeek V4.1 Flash - Provider Adapter (Low)","DeepSeek V4.1 Flash - Provider Adapter (High)","Claude Haiku 5.5 - Provider Adapter (Max)","DeepSeek V4.1 Flash - Provider Adapter (Max)","GPT-6 Sol - Provider Adapter (Low)","GPT-6 Sol - Provider Adapter (None)","Grok 4.7 - Provider Adapter (Low)","Claude Sonnet 5.5 - Provider Adapter (Low)","GPT-6 Luna - Provider Adapter (Max)","Anthropic Opus 4.6 (Max)","GPT-6 Luna - Provider Adapter (XHigh)","DeepSeek V4.1 Flash - Provider Adapter (None)","Claude Haiku 5.5 (Max)","Muse Spark Open - Provider Adapter (XHigh)","GPT-6 Luna - Provider Adapter (High)","GPT-6 Luna - Provider Adapter (Medium)","GPT-6 Luna - Provider Adapter (Low)","Opus 4.7 (High)","Grok 4.20 (Beta Reasoning)","GPT-6 Luna - Provider Adapter (None)","DeepSeek-V4-Pro-Base","Base; 5-shot","https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro","DeepSeek-V4-Flash-Base","DeepSeek-V3.2-Base","Base; 8-shot","Gemini-3.1-Pro High","High","Opus-4.6 Max","Max","DS-V4-Pro Max","K2.6 Thinking","Thinking","GPT-5.4 xHigh","xHigh","GLM-5.1 Thinking","GPT-6-Astra (Codex)","Codex; 93 tasks","Fable 5.1","93 tasks","Fable 5","GPT-5.6-Sol (Codex)","DeepSeek-V4.1-Flash","GLM-5.3","Kimi-K3","Opus 5","Qwen3.8-Max","HY3","Claude Code; v1.1; 106 tasks","GPT-5.6 sol","Codex; v1.1; 106 tasks","Pine Computer","Pine Computer Runtime (PCR); v1.1; 106 tasks","Kimi-CLI; v1.1; 106 tasks","Qwen 3.8 max","Qwen Code; v1.1; 106 tasks","Browser-Use (Open-Source); v1.1; 106 tasks","GPT-5.5-High","OpenHands; 2026-06; 100 tasks","Claude-Opus-4.8-XHigh","GLM-5.2-Max","MiniMax-M3-Adaptive","DeepSeek-V4-Pro-High","Doubao-Seed-2.1-Pro-High","Qwen-3.7-Max-Thinking","Qwen-3.7-Plus-Thinking","DeepSeek-V4-Flash-High","Hunyuan-HY3-Preview-High","Gemini-3.5-Flash-High","Step-3.7-Flash-High","Claude-Opus-4.7","OpenHands; 115 tasks","Claude-Opus-4.6","GPT-5.4","DeepSeek-V4-Pro","GLM-5.1","Kimi-K2.6","Mimo-V2.5-Pro","Qwen3.6-Plus","MiniMax-M2.7","Qwen3.5-397B","Seed-2.0-Pro","https://arxiv.org/html/2605.15846v2#S4.T2.8.1.16","Claude-Opus-4.8","terminus-2; effort xhigh; one attempt; clean 2026-06-25 release","GPT-5.5","terminus-2; effort high; one attempt; clean 2026-06-25 release","MiniMax-M3","terminus-2; thinking adaptive; one attempt; clean 2026-06-25 release","terminus-2; thinking on¹; one attempt; clean 2026-06-25 release","DeepSeek-V4-Flash","Qwen3.7-Max","Kimi-K2.7-Code","Doubao-Seed-2.1-Pro","terminus-2; reasoning split; one attempt; clean 2026-06-25 release","claude-opus-4-7","hermes; V2; 130 tasks; lenient judge rubric; 2026-05-20","gpt-5.5","deepseek-v4-pro","deepseek-v4-flash:free","z-ai/glm-4.5-air:free","minimax-m2.5:free","openclaw; V2; 130 tasks; lenient judge rubric; 2026-05-20","openrouter-owl-alpha","hermes; V2; 130 tasks; strict judge rubric; 2026-05-20","openclaw; V2; 130 tasks; strict judge rubric; 2026-05-20","EchoZ-1.0","UniPat AI","Overall; published Elo","Deepseek-V4-Pro","Grok-4.1-Fast","Minimax-M2.7","Grok-4.2-beta","Minimax-M2.5","Qwen3-Max","Deepseek-V3.2","Qwen3.8-Max (with tools)","Open","Fable 5 (max) (with tools)","GPT-5.6 Sol (max) (with tools)","Kimi K3 (with Python tools)","GPT-5.5 (xhigh) (with tools)","Claude-Opus-4.8 (max) (with tools)","Muse Spark 1.1 (with tools)","Seed-2.1-Pro","Qwen3.7-Plus (with tools)","Kimi-K2.6 (with tools)","Seed-2.0-Pro (high)","GPT-5.4 (xhigh)","Gemini-3.0-Pro","Qwen3VL-235B-Thinking","Qwen3VL-235B-Instruct","Tool-free; three runs; 388 items","https://arxiv.org/html/2601.06521v2#A3.T6","InternVL3.5-241B","OpenGVLab","Qwen3-VL-Plus","GLM4.6V","Qwen3VL-32B-Thinking","Grok-4","MimoVL-7B-RL","Step3","Qwen3VL-4B-Thinking","Claude-4.5-Opus","Qwen3VL-8B-Thinking","KimiVL-A3B","Codex; cc-3d; up to 3 runs; 250 questions","Claude Code; cc-3d; up to 3 runs; 250 questions","Qwen3.8 Max 0902","Qwen3.8 Flash","DeepSeek V4 Flash","Doubao Seed Evolving","Hunyuan-4 Preview","Codex; cc-mcp; up to 3 runs; 250 questions","Claude Code; cc-mcp; up to 3 runs; 250 questions","Claude Code; cc-2d-6; up to 3 runs; 250 questions","Codex; cc-2d-6; up to 3 runs; 250 questions","terminus-2; one attempt; 50 tasks","GPT-5.5-high","Qwen-3.6-Plus","Qwen-3.5-397B-A17B","Anonymous model A","Anonymous A–D panel; 14 tasks; human process grading","Anonymous model D","Anonymous model B","Anonymous model C","GPT 5.6 Sol (Max reasoning)","GPT 6 Astra (Max reasoning)","Claude Opus 5.5 (Adaptive/Max)","Claude Opus 5 (Adaptive/Max)","GPT 5.6 Terra (Max reasoning)","Claude Fable 5.1 (Adaptive/Max)","GPT 6 Sol (Max reasoning)","Claude Fable 5 (Adaptive/Max)","GPT 6 Luna (Max reasoning)","GPT 5.5 (xHigh reasoning)","Gemini 3.8 Flash (High reasoning)","Claude Opus 4.8 (Adaptive/Max)","GPT 5.4 (xHigh reasoning)","Gemini 3.7 Flash (High reasoning)","DeepSeek V4 Pro (Max reasoning)","Grok 4.5 (High reasoning)","Grok","GPT 5.2 (xHigh reasoning)","Kimi K3 (Max reasoning)","Kimi","Gemini 3.5 Flash (High reasoning)","Gemini 3.1 Pro (High reasoning)","Claude Opus 4.7 (Adaptive/Max)","Gemini 3.6 Flash (High reasoning)","Muse Spark 1.3 (xHigh reasoning)","Claude Opus 4.6 (Adaptive/Max)","DeepSeek V4.1 Flash (Max reasoning)","Hy4 Preview (High reasoning)","Hunyuan","DeepSeek V4 Flash Vision (experimental) (High reasoning)","Muse Spark 1.2 (xHigh reasoning)","Muse Spark 1.1 (xHigh reasoning)","Inkling (xHigh reasoning)","Qwen 3.7 Max (Thinking on)","Qwen 3.8 Max (xHigh reasoning)","Muse Glimmer 30B (xHigh reasoning)","GLM 5.3 (Max reasoning)","Kimi K2.5 (Thinking on)","Claude Opus 4.5 (Adaptive/Max)","Gemini 3.5 Flash-Lite (High reasoning)","DeepSeek V4 Flash (Max reasoning)","GLM 5.2 (Max reasoning)","GLM 5.3 Flash (Max reasoning)","DeepSeek V3.2 (High reasoning)","Kimi K2.6 (Thinking on)","DeepSeek V4 Pro (preview) (xHigh reasoning)","MAI Thinking 1","Gemini 4 Argon (High reasoning)","Gemini 3.6 Flash (Medium reasoning)","GPT 5.6 Luna (Max reasoning)","Grok 4.6 (xHigh reasoning)","Qwen 3.8 Flash (xHigh reasoning)","Claude Sonnet 5 (Adaptive/Max)","Qwen 3.5 Plus (Thinking on)","Qwen 3.7 Plus (Thinking on)","Grok 4.7 (xHigh reasoning)","Grok 4.3 (High reasoning)","DeepSeek V4 Flash Vision (experimental) (Max reasoning)","Gemini 3.5 Flash-Lite (Minimal reasoning)","Inkling (High reasoning)","Inkling Small (xHigh reasoning)","Mistral Large 3","GLM 5.2 (xHigh reasoning)","Hy3 (High reasoning)","DeepSeek V4 Flash (High reasoning)","Grok 4.1 (Fast)","GPT 5.2 Codex (xHigh reasoning)","Gemini 3 Flash (High reasoning)","GLM 5 (Auto reasoning)","Claude Sonnet 4.6 (High reasoning)","Gemini 3 Pro (High reasoning)","Nemotron Lightning 3.5 30B A3B (Thinking on)","Nova 2 Pro (High reasoning)","Qwen 3 Max (Thinking on)","Claude Fable 5 (Adaptive/High)","Gemini 3.7 Flash (Medium reasoning)","Claude Fable 5.1 (Adaptive/High)","Gemini 3.8 Flash (Medium reasoning)","Gemini 3.5 Flash (Medium reasoning)","GPT 5.6 Sol (Medium reasoning)","Claude Opus 5 (Adaptive/High)","DeepSeek V4 Pro (High reasoning)","Claude Opus 4.6 (High reasoning)","Claude Opus 4.8 (Adaptive/Default/High)","Claude Opus 4.7 (High reasoning)","Claude Opus 4.8 (High reasoning)","GLM 5.2 (Auto reasoning)","GPT 5.5 (Medium reasoning)","Claude Opus 4.5 (High reasoning)","Muse Spark 1.2 (Auto reasoning)","DeepSeek V4 Pro (preview) (High reasoning)","GPT 5.2 Chat","GPT 5.4 (No reasoning)","Muse Spark 1.1 (Medium reasoning)","Grok 4.6 (High reasoning)","DeepSeek V4 Flash Vision (experimental)","Hy3 (No reasoning)","DeepSeek V4 Flash (preview) (High reasoning)","GPT 5.2 (No reasoning)","Qwen 3 Max (No reasoning)","Grok 4.1 Fast Reasoning (Thinking on)","Muse Glimmer 30B (High reasoning)","Nova 2 Pro (No reasoning)","Gemini 4 Carbon","Agent: Loop; reasoning=high; pass-1","Sonnet 5.5","Agent: Loop; reasoning=max; pass-1","Opus 5.5","DeepSeek-V4.1-Flash-1017","Gemini 3.7 Flash","Agent: Loop; reasoning=xhigh; pass-1","MiMo V2.6 Flash RL","Agent: Loop; reasoning=auto; pass-1","Zhipu","Grok 4.5","Haiku 5.5","Sonnet 5","GPT-6 Sol","MiniMax-M3.1-Flash","GLM-5.3-Flash","Agent: Loop; reasoning=medium; pass-1","Mercury 3","Inception","Opus 4.7","Opus 4.8","Beam","Qwen 3.8 27B","DeepSeek-V4-Pro-0813","Gemini 3.6 Flash","Opus 4.6","Muse Spark 1.2","GPT-5.6 Luna","Sonnet 4.6","Agent: Loop; pass-1","Kimi K2.7 Code","Gemini 3.1 Pro","Muse Spark 1.1","Gemini 3.5 Flash Lite","Gemini 3.5 Flash","Nemotron 3.5 Bolt","Qwen3.5","Nemotron 3 Ultra","GPT-OSS-120B","Agent: Single-Shot Agent; reasoning=high; mean-score","Agent: Single-Shot Agent; reasoning=xhigh; mean-score","Agent: Single-Shot Agent; reasoning=max; mean-score","GPT-5.3 Codex","Gemini 3 Flash","Grok 4.1","Agent: Single-Shot Agent; reasoning=fast; mean-score","Opus 4.5","Grok 4","Agent: Single-Shot Agent; mean-score","Agent: Terminus-2; reasoning=max; pass-1","Agent: Terminus-2; reasoning=xhigh; pass-1","Agent: Terminus-2; reasoning=high; pass-1","Agent: Terminus-2; reasoning=medium; pass-1","Agent: Inspect; reasoning=max; pass-1","Agent: Terminus-2; reasoning=auto; pass-1","Agent: Inspect; reasoning=high; pass-1","Agent: Inspect; reasoning=xhigh; pass-1","Agent: Inspect; pass-1","Agent: Terminus-2; pass-1","Kimi K2.7","Cognition SWE-1.6 Preview","Sonnet 4.5","GPT-5.2 Codex","GPT-5.1 Codex","Agent: proximus; mean@5; 20 hours per task","https://www.frontierswe.com/traces?model=gpt-6-astra","https://www.frontierswe.com/traces?model=claude-opus-5.5","https://www.frontierswe.com/traces?model=claude-sonnet-5.5","https://www.frontierswe.com/traces?model=gemini-4-carbon","https://www.frontierswe.com/traces?model=claude-fable-5.1","https://www.frontierswe.com/traces?model=gemini-4-argon","https://www.frontierswe.com/traces?model=claude-opus-5","Claude Fable 5","https://www.frontierswe.com/traces?model=claude-fable-5","https://www.frontierswe.com/traces?model=claude-haiku-5.5","https://www.frontierswe.com/traces?model=gpt-5.6-sol","https://www.frontierswe.com/traces?model=glm-5.3","https://www.frontierswe.com/traces?model=grok-4.7","https://www.frontierswe.com/traces?model=kimi-k3","https://www.frontierswe.com/traces?model=muse-spark-1.3","https://www.frontierswe.com/traces?model=grok-4.6","https://www.frontierswe.com/traces?model=gemini-3.7-flash","https://www.frontierswe.com/traces?model=gemini-3.8-flash","https://www.frontierswe.com/traces?model=glm-5.3-flash","Qwen3.8-Max-0902","https://www.frontierswe.com/traces?model=qwen3.8-max-0902","https://www.frontierswe.com/traces?model=qwen3.8-max","DeepSeek V4 Flash Vision Exp","https://www.frontierswe.com/traces?model=deepseek-v4-flash-vision-exp","https://www.frontierswe.com/traces?model=muse-spark-1.2","Thinking Machines Lab","https://www.frontierswe.com/traces?model=inkling","claude-fable-5","Agent: Claude Code","Agent: Codex","claude-opus-4-8","Agent: OpenCode","Published task-level pass rate","Claude Opus 4.8","GPT-5.6 (sol)","reasoning=high; eight trials; shared 3451-problem grid","Gemini 3.1 Pro Preview","Agent: OpenHands; skills=true","Agent: Codex; skills=true","Claude","Agent: Claude Code; skills=true","Gemini","Agent: Gemini CLI; skills=true","GLM 5.1","Kimi K2.6","MiniMax M3","DeepSeek V4 Pro","Grok 4.3","GPT-5.4 Mini","MiniMax M2.7","Haiku 4.5","Gemini 3.1 Flash Lite","Agent: OpenHands; skills=false","Agent: Codex; skills=false","Agent: Claude Code; skills=false","Agent: Gemini CLI; skills=false","Agent: FAB v2","https://www.vals.ai/models/anthropic_claude-opus-5-5","https://www.vals.ai/models/google_gemini-4-argon","Wonder Full","https://www.vals.ai/models/meta_muse-wonder-full","https://www.vals.ai/models/anthropic_claude-sonnet-5-5","https://www.vals.ai/models/google_gemini-4-carbon","https://www.vals.ai/models/google_gemini-3.8-flash","Wonder","https://www.vals.ai/models/meta_muse-wonder","DeepSeek V4.1 Flash","https://www.vals.ai/models/deepseek_deepseek-v4.1-flash","https://www.vals.ai/models/anthropic_claude-haiku-5-5","GLM 5.3","https://www.vals.ai/models/zai_glm-5.3","https://www.vals.ai/models/xiaomi_mimo-v2.6-pro","MiMo V2.6 Flash","https://www.vals.ai/models/xiaomi_mimo-v2.6-flash","https://www.vals.ai/models/deepseek_deepseek-v4.1-flash-1017","https://www.vals.ai/models/openai_gpt-6.1-sol","Mistral Large 4 OSS","https://www.vals.ai/models/mistralai_mistral-large-4-oss","Hy4 Preview","https://www.vals.ai/models/tencent_hy4-preview","https://www.vals.ai/models/kimi_kimi-k3.1-pro","https://www.vals.ai/models/mistralai_mistral-large-4","https://www.vals.ai/models/meta_muse-spark-open","GLM 5.3 Flash","https://www.vals.ai/models/zai_glm-5.3-flash","https://www.vals.ai/models/grok_grok-4.7","https://www.vals.ai/models/openai_gpt-6-astra","https://www.vals.ai/models/kimi_kimi-k3","Step 5 Preview","https://www.vals.ai/models/stepfun_step-5-preview","https://www.vals.ai/models/minimax_MiniMax-M3.1-flash","Qwen 3.8 Max 0902","https://www.vals.ai/models/alibaba_qwen3.8-max-0902","DeepSeek V4 Pro 0813","https://www.vals.ai/models/deepseek_deepseek-v4-pro-0813","https://www.vals.ai/models/openai_gpt-6-luna","https://www.vals.ai/models/openai_gpt-6.1-terra","https://www.vals.ai/models/inception_mercury-3","https://www.vals.ai/models/minimax_MiniMax-M3","Beam 1.0","https://www.vals.ai/models/reflection_Beam-1.0","https://www.vals.ai/models/thinkingmachines_inkling","https://www.vals.ai/models/baseten_nemotron-3.5-super-vl-120b-a12b","Inkling Small","https://www.vals.ai/models/thinkingmachines_inkling-small","Agent: Tax Agent Bench","https://www.vals.ai/models/anthropic_claude-fable-5-1","https://www.vals.ai/models/anthropic_claude-opus-5","Muse Spark 1.3 Max","https://www.vals.ai/models/meta_muse_spark_1_3_max","https://www.vals.ai/models/anthropic_claude-fable-5","https://www.vals.ai/models/meta_muse_spark_1_3","https://www.vals.ai/models/grok_grok-4.6","https://www.vals.ai/models/meta_muse_spark_1_2","Qwen 3.8 Max","https://www.vals.ai/models/alibaba_qwen3.8-max","https://www.vals.ai/models/openai_gpt-5.6-sol","Ember-1","https://www.vals.ai/models/fireworks_ember-1","https://www.vals.ai/models/alibaba_qwen3.8-27b","https://www.vals.ai/models/openai_gpt-5.6-terra","https://www.vals.ai/models/anthropic_claude-opus-4-8","Claude Sonnet 5","https://www.vals.ai/models/anthropic_claude-sonnet-5","DeepSeek V4 Flash 0731","https://www.vals.ai/models/deepseek_deepseek-v4-flash-0731","Claude Sonnet 4.6","https://www.vals.ai/models/anthropic_claude-sonnet-4-6","DeepSeek V4","https://www.vals.ai/models/deepseek_deepseek-v4-pro","https://www.vals.ai/models/grok_grok-4.5","https://www.vals.ai/models/openai_gpt-5.6-luna","Claude Opus 4.7","https://www.vals.ai/models/anthropic_claude-opus-4-7","https://www.vals.ai/models/google_gemini-3.7-flash","GPT 5.5","https://www.vals.ai/models/openai_gpt-5.5","https://www.vals.ai/models/google_gemini-3.5-flash","Ling 3.0 Flash Fin","https://www.vals.ai/models/ant_ling-3.0-flash-fin","https://www.vals.ai/models/google_gemini-3.6-flash","Qwen 3.7 Max","https://www.vals.ai/models/alibaba_qwen3.7-max","https://www.vals.ai/models/openai_gpt-6-sol","https://www.vals.ai/models/kimi_kimi-k2.6","Gemini 3 Flash (12/25)","https://www.vals.ai/models/google_gemini-3-flash-preview","MiMo V2.5 Pro","https://www.vals.ai/models/xiaomi_mimo-v2.5-pro","Qwen 3.7 Plus","https://www.vals.ai/models/alibaba_qwen3.7-plus","Gemini 3.1 Pro Preview (02/26)","https://www.vals.ai/models/google_gemini-3.1-pro-preview","GPT 5.4 Mini","https://www.vals.ai/models/openai_gpt-5.4-mini-2026-03-17","MiMo V2.5","https://www.vals.ai/models/xiaomi_mimo-v2.5","Qwen 3.6 Plus","https://www.vals.ai/models/alibaba_qwen3.6-plus","https://www.vals.ai/models/google_gemini-3.5-flash-lite","https://www.vals.ai/models/grok_grok-4.20-0309-reasoning","https://www.vals.ai/models/minimax_MiniMax-M2.7","https://www.vals.ai/models/grok_grok-4.3","Claude Haiku 4.5 (Thinking)","https://www.vals.ai/models/anthropic_claude-haiku-4-5-20251001-thinking","GPT 5.4 Nano","https://www.vals.ai/models/openai_gpt-5.4-nano-2026-03-17","Mistral Medium 3.5","https://www.vals.ai/models/mistralai_mistral-medium-3.5","Gemini 3.1 Flash Lite Preview","https://www.vals.ai/models/google_gemini-3.1-flash-lite-preview","Nemotron 3.5 Lightning","https://www.vals.ai/models/fireworks_nemotron-lightning-3p5-30b-a3b","Mercury 2.5","https://www.vals.ai/models/inception_mercury-2.5","Agent: Valkyrie","Agent: Valkyrie / Harvey LAB","Claude 4.5 Haiku","Average of four safety modules and evaluated languages","https://phare.giskard.ai/models/claude-haiku-4-5-20251001","Claude 4.5 Opus","https://phare.giskard.ai/models/claude-opus-4-5-20251101","Claude 5 Sonnet","https://phare.giskard.ai/models/claude-sonnet-5","https://phare.giskard.ai/models/claude-sonnet-4-5-20250929","Claude 4.1 Opus","https://phare.giskard.ai/models/claude-opus-4-1-20250805","Llama 3.1 405B Instruct OR","https://phare.giskard.ai/models/llama-3.1-405b-instruct-bf16","https://phare.giskard.ai/models/kimi-k2.6","https://phare.giskard.ai/models/qwen3.7-max-preview","https://phare.giskard.ai/models/claude-opus-4-6","https://phare.giskard.ai/models/qwen3.7-plus","Claude 4.6 Sonnet","https://phare.giskard.ai/models/claude-sonnet-4-6","Gemini 3.0 Pro Preview","https://phare.giskard.ai/models/gemini-3-pro","https://phare.giskard.ai/models/gpt-5-mini","GPT 5.1","https://phare.giskard.ai/models/gpt-5.1","GPT 5.2","https://phare.giskard.ai/models/gpt-5.2","https://phare.giskard.ai/models/llama-4-maverick-17b-128e-instruct","GPT 4o","https://phare.giskard.ai/models/gpt-4o","https://phare.giskard.ai/models/gemini-3.1-pro","https://phare.giskard.ai/models/gpt-5.5","https://phare.giskard.ai/models/gpt-4.1-mini","GLM 5.2","https://phare.giskard.ai/models/glm-5.2","GPT 5 nano","https://phare.giskard.ai/models/gpt-5-nano","https://phare.giskard.ai/models/claude-3-7-sonnet-20250219","Claude 3.5 Haiku 20241022","https://phare.giskard.ai/models/claude-3-5-haiku-20241022","Qwen Plus","https://phare.giskard.ai/models/qwen-plus-0125","https://phare.giskard.ai/models/grok-4.3","https://phare.giskard.ai/models/gemini-3.1-flash-lite","Gemma 4","https://phare.giskard.ai/models/gemma-4-31b-it","GPT 5","https://phare.giskard.ai/models/gpt-5","https://phare.giskard.ai/models/gpt-4.1","https://phare.giskard.ai/models/kimi-k2.5","Grok 4 Fast No Reasoning","https://phare.giskard.ai/models/grok-4-fast-non-reasoning","GPT OSS 120B","https://phare.giskard.ai/models/gpt-oss-120b","Qwen 3 Max","https://phare.giskard.ai/models/qwen3-max","Deepseek V3.1","https://phare.giskard.ai/models/deepseek-v3.1","Mistral Small 3.2","https://phare.giskard.ai/models/mistral-small-2506","https://phare.giskard.ai/models/mistral-medium-3.5","https://phare.giskard.ai/models/gemini-2.0-flash","Qwen 3 8B","https://phare.giskard.ai/models/qwen3-8b","https://phare.giskard.ai/models/llama-4-scout-17b-16e-instruct","https://phare.giskard.ai/models/deepseek-v4-flash","https://phare.giskard.ai/models/gemini-2.5-flash","Llama 3.1 8B Instruct","https://phare.giskard.ai/models/llama-3.1-8b-instruct","Magistral Medium Latest","https://phare.giskard.ai/models/magistral-medium-latest","Deepseek V3","https://phare.giskard.ai/models/deepseek-v3","Qwen 2.5 Max","https://phare.giskard.ai/models/qwen2.5-max","https://phare.giskard.ai/models/grok-4","Llama 3.3 70B Instruct OR","https://phare.giskard.ai/models/llama-3.3-70b-instruct","https://phare.giskard.ai/models/mistral-large-3","https://phare.giskard.ai/models/cohere-command-a","Deepseek V3 0324","https://phare.giskard.ai/models/deepseek-v3-0324","https://phare.giskard.ai/models/gemini-2.5-pro","Grok 3 mini","https://phare.giskard.ai/models/grok-3-mini","Qwen 3 30B VL Instruct","https://phare.giskard.ai/models/qwen3-30b-a3b-instruct","Gemini 2.5 Flash Lite","https://phare.giskard.ai/models/gemini-2.5-flash-lite","https://phare.giskard.ai/models/deepseek-v4-pro","https://phare.giskard.ai/models/mistral-large-2407","Deepseek R1 0528","https://phare.giskard.ai/models/deepseek-r1-0528","GPT 4o mini","https://phare.giskard.ai/models/gpt-4o-mini","Mistral Medium Latest","https://phare.giskard.ai/models/mistral-medium-2508","https://phare.giskard.ai/models/gpt-4.1-nano","Gemma 3 27B IT OR","https://phare.giskard.ai/models/gemma-3-27b-it","Gemini 2.0 Flash Lite","https://phare.giskard.ai/models/gemini-2.0-flash-lite","Magistral Small Latest","https://phare.giskard.ai/models/magistral-small-latest","Gemma 3 12B IT OR","https://phare.giskard.ai/models/gemma-3-12b-it","Grok 2","https://phare.giskard.ai/models/grok-2","https://phare.giskard.ai/models/grok-3","Vending-Bench 2; mean final money balance; USD","Muse Wonder Full","Muse Wonder","GPT-5.3-Codex","Claude Opus 4.8 - High","Claude Fable 5 - High","Kimi K3 (Moonshot)","Claude Fable 5 - Low","Claude Opus 4.5","Claude Fable 5 - Max","Kimi K3 (Fireworks)","Grok 4.20","Claude Fable 5 - None","Claude Fable 5 - Medium","Qwen 3.6 Max","Gemini 3.1 Pro Custom Tools","Deepseek V4 Pro","Claude Opus 4.8 - Max","GPT-5.1","Grok 4.1 Fast","Qwen 3.5 35B A3B","Qwen 3.5 27B","Qwen3 Max","Qwen 3.5 Plus","Qwen3 235B A22B Thinking","GPT-OSS-120b","GPT-5 mini","Junie","Resolved rate; official default board; pass@1","Claude Code","Cursor","XiaomiMiMo","DeepSeek-V4 Pro [high]","Qwen3.6-27B","Qwen3.6-35B-A3B","Agent: MSWE-agent","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/java/verified/20250426_MSWE-agent_Gemini-2.5-Pro","Claude-3.7-Sonnet","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/java/verified/20250329_MSWE-agent_Claude-3.7-Sonnet","Agent: MagentLess","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/java/verified/20250329_MagentLess_DeepSeek-R1","Agent: MopenHands","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/java/verified/20250329_MopenHands_Claude-3.7-Sonnet","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/java/verified/20250426_MagentLess_Gemini-2.5-Pro","OpenAI-o1","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/java/verified/20250329_MSWE-agent_OpenAI-o1","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/java/verified/20250329_MagentLess_OpenAI-o1","Claude-3.5-Sonnet(Oct)","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/java/verified/20250329_MSWE-agent_Claude-3.5-Sonnet(Oct)","OpenAI-o3-mini-high","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/java/verified/20250329_MSWE-agent_OpenAI-o3-mini-high","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/java/verified/20250329_MagentLess_Claude-3.5-Sonnet(Oct)","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/java/verified/20250329_MopenHands_Claude-3.5-Sonnet(Oct)","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/java/verified/20250426_MagentLess_Llama-4-Maverick","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/java/verified/20250329_MagentLess_Claude-3.7-Sonnet","Doubao-1.5-thinking","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/java/verified/20250426_MagentLess_Doubao-1.5-thinking","GPT-4o-1120","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/java/verified/20250329_MSWE-agent_GPT-4o-1120","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/java/verified/20250426_MopenHands_Gemini-2.5-Pro","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/java/verified/20250329_MSWE-agent_DeepSeek-V3","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/java/verified/20250426_MSWE-agent_Doubao-1.5-thinking","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/java/verified/20250329_MagentLess_GPT-4o-1120","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/java/verified/20250426_MopenHands_Doubao-1.5-thinking","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/java/verified/20250329_MagentLess_Qwen2.5-72B-Instruct","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/java/verified/20250329_MopenHands_OpenAI-o3-mini-high","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/java/verified/20250329_MSWE-agent_DeepSeek-R1","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/java/verified/20250329_MopenHands_DeepSeek-V3","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/java/verified/20250329_MopenHands_GPT-4o-1120","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/java/verified/20250329_MopenHands_DeepSeek-R1","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/java/verified/20250329_MagentLess_DeepSeek-V3","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/java/verified/20250329_MSWE-agent_Doubao-1.5-pro","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/java/verified/20250426_MSWE-agent_Llama-4-Maverick","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/java/verified/20250426_MopenHands_Llama-4-Maverick","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/java/verified/20250329_MagentLess_Doubao-1.5-pro","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/java/verified/20250329_MagentLess_OpenAI-o3-mini-high","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/java/verified/20250329_MopenHands_OpenAI-o1","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/java/verified/20250329_MopenHands_Qwen2.5-72B-Instruct","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/java/verified/20250329_MSWE-agent_Qwen2.5-72B-Instruct","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/java/verified/20250329_MopenHands_Doubao-1.5-pro","Agent: RepoRepair","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/typescript/verified/20250910_RepoRepair_Claude-3.5-Sonnet(Oct)","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/typescript/verified/20250426_MopenHands_Gemini-2.5-Pro","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/typescript/verified/20250329_MopenHands_Claude-3.5-Sonnet(Oct)","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/typescript/verified/20250426_MagentLess_Gemini-2.5-Pro","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/typescript/verified/20250329_MSWE-agent_Claude-3.7-Sonnet","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/typescript/verified/20250426_MagentLess_Llama-4-Maverick","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/typescript/verified/20250426_MSWE-agent_Gemini-2.5-Pro","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/typescript/verified/20250329_MSWE-agent_Claude-3.5-Sonnet(Oct)","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/typescript/verified/20250426_MopenHands_Llama-4-Maverick","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/typescript/verified/20250426_MagentLess_Doubao-1.5-thinking","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/typescript/verified/20250426_MSWE-agent_Doubao-1.5-thinking","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/typescript/verified/20250329_MagentLess_DeepSeek-V3","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/typescript/verified/20250329_MagentLess_DeepSeek-R1","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/typescript/verified/20250329_MSWE-agent_DeepSeek-R1","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/typescript/verified/20250329_MagentLess_OpenAI-o1","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/typescript/verified/20250426_MopenHands_Doubao-1.5-thinking","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/typescript/verified/20250329_MagentLess_Claude-3.5-Sonnet(Oct)","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/typescript/verified/20250329_MSWE-agent_OpenAI-o3-mini-high","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/typescript/verified/20250426_MSWE-agent_Llama-4-Maverick","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/typescript/verified/20250329_MagentLess_Qwen2.5-72B-Instruct","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/typescript/verified/20250329_MSWE-agent_OpenAI-o1","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/typescript/verified/20250329_MagentLess_Claude-3.7-Sonnet","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/typescript/verified/20250329_MSWE-agent_DeepSeek-V3","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/typescript/verified/20250329_MopenHands_Claude-3.7-Sonnet","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/typescript/verified/20250329_MagentLess_Doubao-1.5-pro","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/typescript/verified/20250329_MagentLess_GPT-4o-1120","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/typescript/verified/20250329_MSWE-agent_Doubao-1.5-pro","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/typescript/verified/20250329_MopenHands_DeepSeek-V3","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/typescript/verified/20250329_MopenHands_DeepSeek-R1","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/typescript/verified/20250329_MSWE-agent_GPT-4o-1120","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/typescript/verified/20250329_MopenHands_OpenAI-o1","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/typescript/verified/20250329_MagentLess_OpenAI-o3-mini-high","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/typescript/verified/20250329_MopenHands_OpenAI-o3-mini-high","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/typescript/verified/20250329_MopenHands_Doubao-1.5-pro","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/typescript/verified/20250329_MopenHands_GPT-4o-1120","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/typescript/verified/20250329_MSWE-agent_Qwen2.5-72B-Instruct","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/typescript/verified/20250329_MopenHands_Qwen2.5-72B-Instruct","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/javascript/verified/20250426_MopenHands_Gemini-2.5-Pro","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/javascript/verified/20250426_MopenHands_Doubao-1.5-thinking","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/javascript/verified/20250426_MagentLess_Gemini-2.5-Pro","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/javascript/verified/20250426_MSWE-agent_Gemini-2.5-Pro","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/javascript/verified/20250426_MagentLess_Doubao-1.5-thinking","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/javascript/verified/20250329_MopenHands_Claude-3.7-Sonnet","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/javascript/verified/20250426_MagentLess_Llama-4-Maverick","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/javascript/verified/20250329_MagentLess_OpenAI-o1","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/javascript/verified/20250329_MSWE-agent_Claude-3.7-Sonnet","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/javascript/verified/20250426_MopenHands_Llama-4-Maverick","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/javascript/verified/20250329_MagentLess_DeepSeek-R1","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/javascript/verified/20250329_MSWE-agent_Claude-3.5-Sonnet(Oct)","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/javascript/verified/20250329_MSWE-agent_OpenAI-o1","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/javascript/verified/20250329_MSWE-agent_OpenAI-o3-mini-high","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/javascript/verified/20250329_MopenHands_OpenAI-o1","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/javascript/verified/20250329_MagentLess_DeepSeek-V3","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/javascript/verified/20250329_MopenHands_OpenAI-o3-mini-high","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/javascript/verified/20250329_MagentLess_OpenAI-o3-mini-high","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/javascript/verified/20250329_MopenHands_DeepSeek-R1","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/javascript/verified/20250329_MSWE-agent_DeepSeek-V3","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/javascript/verified/20250426_MSWE-agent_Llama-4-Maverick","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/javascript/verified/20250329_MagentLess_Claude-3.5-Sonnet(Oct)","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/javascript/verified/20250329_MopenHands_Claude-3.5-Sonnet(Oct)","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/javascript/verified/20250329_MagentLess_Claude-3.7-Sonnet","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/javascript/verified/20250329_MopenHands_GPT-4o-1120","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/javascript/verified/20250426_MSWE-agent_Doubao-1.5-thinking","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/javascript/verified/20250329_MSWE-agent_DeepSeek-R1","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/javascript/verified/20250329_MSWE-agent_Doubao-1.5-pro","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/javascript/verified/20250329_MagentLess_GPT-4o-1120","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/javascript/verified/20250329_MopenHands_DeepSeek-V3","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/javascript/verified/20250329_MagentLess_Doubao-1.5-pro","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/javascript/verified/20250329_MopenHands_Doubao-1.5-pro","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/javascript/verified/20250329_MSWE-agent_GPT-4o-1120","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/javascript/verified/20250329_MagentLess_Qwen2.5-72B-Instruct","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/javascript/verified/20250329_MopenHands_Qwen2.5-72B-Instruct","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/javascript/verified/20250329_MSWE-agent_Qwen2.5-72B-Instruct","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/go/verified/20250426_MopenHands_Gemini-2.5-Pro","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/go/verified/20250426_MSWE-agent_Gemini-2.5-Pro","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/go/verified/20250329_MopenHands_Claude-3.7-Sonnet","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/go/verified/20250329_MopenHands_Claude-3.5-Sonnet(Oct)","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/go/verified/20250426_MopenHands_Doubao-1.5-thinking","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/go/verified/20250426_MagentLess_Gemini-2.5-Pro","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/go/verified/20250329_MSWE-agent_Claude-3.5-Sonnet(Oct)","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/go/verified/20250329_MagentLess_Claude-3.7-Sonnet","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/go/verified/20250426_MopenHands_Llama-4-Maverick","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/go/verified/20250329_MSWE-agent_Claude-3.7-Sonnet","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/go/verified/20250329_MagentLess_DeepSeek-V3","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/go/verified/20250329_MagentLess_Claude-3.5-Sonnet(Oct)","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/go/verified/20250329_MSWE-agent_OpenAI-o1","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/go/verified/20250329_MSWE-agent_DeepSeek-V3","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/go/verified/20250426_MagentLess_Doubao-1.5-thinking","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/go/verified/20250329_MagentLess_OpenAI-o1","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/go/verified/20250426_MSWE-agent_Doubao-1.5-thinking","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/go/verified/20250329_MSWE-agent_OpenAI-o3-mini-high","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/go/verified/20250329_MagentLess_OpenAI-o3-mini-high","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/go/verified/20250329_MagentLess_DeepSeek-R1","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/go/verified/20250329_MopenHands_OpenAI-o1","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/go/verified/20250329_MopenHands_GPT-4o-1120","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/go/verified/20250426_MagentLess_Llama-4-Maverick","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/go/verified/20250329_MagentLess_GPT-4o-1120","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/go/verified/20250426_MSWE-agent_Llama-4-Maverick","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/go/verified/20250329_MSWE-agent_GPT-4o-1120","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/go/verified/20250329_MopenHands_OpenAI-o3-mini-high","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/go/verified/20250329_MSWE-agent_DeepSeek-R1","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/go/verified/20250329_MSWE-agent_Doubao-1.5-pro","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/go/verified/20250329_MagentLess_Doubao-1.5-pro","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/go/verified/20250329_MopenHands_Doubao-1.5-pro","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/go/verified/20250329_MagentLess_Qwen2.5-72B-Instruct","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/go/verified/20250329_MopenHands_Qwen2.5-72B-Instruct","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/go/verified/20250329_MopenHands_DeepSeek-V3","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/go/verified/20250329_MSWE-agent_Qwen2.5-72B-Instruct","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/go/verified/20250329_MopenHands_DeepSeek-R1","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/rust/verified/20250329_MopenHands_Claude-3.7-Sonnet","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/rust/verified/20250426_MopenHands_Gemini-2.5-Pro","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/rust/verified/20250329_MopenHands_Claude-3.5-Sonnet(Oct)","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/rust/verified/20250426_MSWE-agent_Gemini-2.5-Pro","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/rust/verified/20250329_MagentLess_OpenAI-o3-mini-high","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/rust/verified/20250426_MagentLess_Doubao-1.5-thinking","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/rust/verified/20250329_MagentLess_OpenAI-o1","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/rust/verified/20250329_MSWE-agent_Claude-3.5-Sonnet(Oct)","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/rust/verified/20250329_MSWE-agent_Claude-3.7-Sonnet","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/rust/verified/20250329_MagentLess_DeepSeek-R1","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/rust/verified/20250426_MagentLess_Llama-4-Maverick","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/rust/verified/20250329_MSWE-agent_DeepSeek-V3","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/rust/verified/20250329_MagentLess_GPT-4o-1120","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/rust/verified/20250329_MagentLess_Claude-3.7-Sonnet","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/rust/verified/20250426_MagentLess_Gemini-2.5-Pro","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/rust/verified/20250329_MagentLess_Claude-3.5-Sonnet(Oct)","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/rust/verified/20250329_MagentLess_DeepSeek-V3","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/rust/verified/20250426_MopenHands_Llama-4-Maverick","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/rust/verified/20250329_MSWE-agent_OpenAI-o3-mini-high","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/rust/verified/20250329_MopenHands_OpenAI-o3-mini-high","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/rust/verified/20250329_MopenHands_DeepSeek-R1","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/rust/verified/20250329_MopenHands_DeepSeek-V3","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/rust/verified/20250329_MagentLess_Doubao-1.5-pro","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/rust/verified/20250329_MSWE-agent_OpenAI-o1","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/rust/verified/20250426_MSWE-agent_Doubao-1.5-thinking","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/rust/verified/20250426_MopenHands_Doubao-1.5-thinking","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/rust/verified/20250329_MopenHands_GPT-4o-1120","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/rust/verified/20250426_MSWE-agent_Llama-4-Maverick","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/rust/verified/20250329_MopenHands_OpenAI-o1","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/rust/verified/20250329_MagentLess_Qwen2.5-72B-Instruct","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/rust/verified/20250329_MSWE-agent_DeepSeek-R1","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/rust/verified/20250329_MSWE-agent_GPT-4o-1120","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/rust/verified/20250329_MSWE-agent_Doubao-1.5-pro","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/rust/verified/20250329_MopenHands_Qwen2.5-72B-Instruct","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/rust/verified/20250329_MopenHands_Doubao-1.5-pro","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/rust/verified/20250329_MSWE-agent_Qwen2.5-72B-Instruct","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/c/verified/20250426_MSWE-agent_Gemini-2.5-Pro","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/c/verified/20250426_MagentLess_Gemini-2.5-Pro","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/c/verified/20250329_MSWE-agent_Claude-3.7-Sonnet","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/c/verified/20250329_MopenHands_Claude-3.7-Sonnet","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/c/verified/20250329_MagentLess_Claude-3.7-Sonnet","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/c/verified/20250426_MopenHands_Gemini-2.5-Pro","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/c/verified/20250329_MSWE-agent_Claude-3.5-Sonnet(Oct)","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/c/verified/20250426_MagentLess_Doubao-1.5-thinking","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/c/verified/20250426_MopenHands_Doubao-1.5-thinking","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/c/verified/20250426_MagentLess_Llama-4-Maverick","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/c/verified/20250329_MSWE-agent_OpenAI-o1","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/c/verified/20250329_MopenHands_Claude-3.5-Sonnet(Oct)","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/c/verified/20250329_MagentLess_DeepSeek-V3","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/c/verified/20250329_MopenHands_DeepSeek-V3","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/c/verified/20250426_MopenHands_Llama-4-Maverick","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/c/verified/20250329_MagentLess_OpenAI-o1","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/c/verified/20250329_MopenHands_OpenAI-o1","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/c/verified/20250329_MopenHands_DeepSeek-R1","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/c/verified/20250329_MSWE-agent_DeepSeek-V3","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/c/verified/20250329_MSWE-agent_Doubao-1.5-pro","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/c/verified/20250329_MSWE-agent_OpenAI-o3-mini-high","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/c/verified/20250329_MagentLess_OpenAI-o3-mini-high","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/c/verified/20250329_MSWE-agent_GPT-4o-1120","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/c/verified/20250329_MagentLess_GPT-4o-1120","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/c/verified/20250329_MopenHands_OpenAI-o3-mini-high","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/c/verified/20250329_MSWE-agent_Qwen2.5-72B-Instruct","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/c/verified/20250329_MagentLess_Qwen2.5-72B-Instruct","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/c/verified/20250329_MagentLess_Claude-3.5-Sonnet(Oct)","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/c/verified/20250329_MSWE-agent_DeepSeek-R1","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/c/verified/20250329_MagentLess_DeepSeek-R1","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/c/verified/20250329_MagentLess_Doubao-1.5-pro","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/c/verified/20250426_MSWE-agent_Doubao-1.5-thinking","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/c/verified/20250329_MopenHands_Qwen2.5-72B-Instruct","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/c/verified/20250329_MopenHands_Doubao-1.5-pro","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/c/verified/20250329_MopenHands_GPT-4o-1120","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/c/verified/20250426_MSWE-agent_Llama-4-Maverick","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/c++/verified/20250329_MopenHands_Claude-3.7-Sonnet","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/c++/verified/20250329_MopenHands_Claude-3.5-Sonnet(Oct)","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/c++/verified/20250329_MSWE-agent_Claude-3.7-Sonnet","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/c++/verified/20250426_MopenHands_Gemini-2.5-Pro","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/c++/verified/20250426_MSWE-agent_Gemini-2.5-Pro","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/c++/verified/20250329_MSWE-agent_DeepSeek-V3","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/c++/verified/20250329_MopenHands_DeepSeek-V3","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/c++/verified/20250329_MSWE-agent_Claude-3.5-Sonnet(Oct)","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/c++/verified/20250329_MagentLess_GPT-4o-1120","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/c++/verified/20250426_MSWE-agent_Llama-4-Maverick","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/c++/verified/20250329_MopenHands_OpenAI-o3-mini-high","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/c++/verified/20250329_MSWE-agent_DeepSeek-R1","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/c++/verified/20250329_MSWE-agent_Doubao-1.5-pro","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/c++/verified/20250426_MopenHands_Doubao-1.5-thinking","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/c++/verified/20250426_MagentLess_Llama-4-Maverick","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/c++/verified/20250329_MagentLess_OpenAI-o1","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/c++/verified/20250329_MSWE-agent_OpenAI-o3-mini-high","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/c++/verified/20250329_MopenHands_DeepSeek-R1","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/c++/verified/20250426_MSWE-agent_Doubao-1.5-thinking","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/c++/verified/20250329_MagentLess_Claude-3.5-Sonnet(Oct)","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/c++/verified/20250426_MagentLess_Doubao-1.5-thinking","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/c++/verified/20250329_MopenHands_GPT-4o-1120","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/c++/verified/20250329_MSWE-agent_OpenAI-o1","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/c++/verified/20250329_MopenHands_OpenAI-o1","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/c++/verified/20250329_MagentLess_Claude-3.7-Sonnet","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/c++/verified/20250329_MagentLess_DeepSeek-R1","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/c++/verified/20250329_MopenHands_Doubao-1.5-pro","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/c++/verified/20250426_MopenHands_Llama-4-Maverick","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/c++/verified/20250329_MSWE-agent_GPT-4o-1120","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/c++/verified/20250426_MagentLess_Gemini-2.5-Pro","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/c++/verified/20250329_MopenHands_Qwen2.5-72B-Instruct","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/c++/verified/20250329_MagentLess_DeepSeek-V3","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/c++/verified/20250329_MagentLess_OpenAI-o3-mini-high","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/c++/verified/20250329_MagentLess_Qwen2.5-72B-Instruct","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/c++/verified/20250329_MagentLess_Doubao-1.5-pro","https://github.com/multi-swe-bench/experiments/tree/main/evaluation/c++/verified/20250329_MSWE-agent_Qwen2.5-72B-Instruct","Highest reasoning; M4; Mean@3 independent trajectories","Hy4 preview","Seed2.1 Pro","Published Kaggle FACTS protocol","QwenLM","Gemini 3 Flash Preview","Gemini 3.1 Flash-Lite Preview","Gemini 3.5 Flash-Lite","o4 mini","Claude Opus 4.1","Claude Sonnet 4.5 (thinking)","Gemma 3 27B","Mistral Small 3.1","Gemma 3 12B","Grok 4 Fast Reasoning","Gemma 3 4B","Qwen 3 235B A22B Instruct 2506","Agent: τ-bench; retrieval=alltools; reasoning=xhigh; user simulator=gpt-5.2; v1.0.1","Agent: τ-bench; retrieval=alltools; reasoning=max; user simulator=gpt-5.2; v1.0.1","Agent: τ-bench; retrieval=alltools; reasoning=high; user simulator=gpt-5.2; v1.0.1","GPT-5.6-sol","Agent: τ-bench; retrieval=terminal; reasoning=high; user simulator=gpt-5.2; v1.0.1","Agent: τ-bench; retrieval=terminal; reasoning=enabled; user simulator=gpt-5.2; v1.0.1","Grok 4.2","Grok 4 fast","Grok 4.1 fast","Agent: τ-bench; retrieval=qwen_embeddings; reasoning=none; user simulator=gpt-5.2; v1.0.1","Agent: τ-bench; retrieval=text-emb-3-large; reasoning=enabled; user simulator=gpt-5.2; v1.0.1","Gemini-3-flash-preview","Agent: IpaziaHPA; Increased max_steps from 15 to 30; publisher submission declares a modified evaluation protocol","https://huggingface.co/spaces/ServiceNow/browsergym-leaderboard/blob/main/results/IpaziaHPA-Gemini-3-flash-preview/README.md","Agent: GenericAgent; Increased max_steps from 15 to 30; publisher submission declares a modified evaluation protocol","https://huggingface.co/spaces/ServiceNow/browsergym-leaderboard/blob/main/results/GenericAgent-GPT-5/README.md","Claude-4-Sonnet","Agent: GenericAgent","https://huggingface.co/spaces/ServiceNow/browsergym-leaderboard/blob/main/results/GenericAgent-Claude-4-Sonnet/README.md","GPT-5-mini","https://huggingface.co/spaces/ServiceNow/browsergym-leaderboard/blob/main/results/GenericAgent-GPT-5-mini/README.md","Claude-3.5-Sonnet","https://huggingface.co/spaces/ServiceNow/browsergym-leaderboard/blob/main/results/GenericAgent-Claude-3.5-Sonnet/README.md","GPT-o1-mini","https://huggingface.co/spaces/ServiceNow/browsergym-leaderboard/blob/main/results/GenericAgent-GPT-o1-mini/README.md","Qwen3.5-9B","Agent: A3; 330 tasks. Fine-tuned on A3-Synth trajectories via the Agent-as-Annotators framework.","https://huggingface.co/spaces/ServiceNow/browsergym-leaderboard/blob/main/results/A3-Qwen3.5-9B/README.md","GPT-oss-120b","https://huggingface.co/spaces/ServiceNow/browsergym-leaderboard/blob/main/results/GenericAgent-GPT-oss-120b/README.md","o3-mini","https://huggingface.co/spaces/ServiceNow/browsergym-leaderboard/blob/main/results/GenericAgent-o3-mini/README.md","https://huggingface.co/spaces/ServiceNow/browsergym-leaderboard/blob/main/results/GenericAgent-GPT-4o/README.md","Llama-3.1-405b","https://huggingface.co/spaces/ServiceNow/browsergym-leaderboard/blob/main/results/GenericAgent-Llama-3.1-405b/README.md","GPT-5-nano","https://huggingface.co/spaces/ServiceNow/browsergym-leaderboard/blob/main/results/GenericAgent-GPT-5-nano/README.md","GPT-oss-20b","https://huggingface.co/spaces/ServiceNow/browsergym-leaderboard/blob/main/results/GenericAgent-GPT-oss-20b/README.md","AgentTrek-1.0-32b","https://huggingface.co/spaces/ServiceNow/browsergym-leaderboard/blob/main/results/GenericAgent-AgentTrek-1.0-32b/README.md","Llama-3.1-70b","https://huggingface.co/spaces/ServiceNow/browsergym-leaderboard/blob/main/results/GenericAgent-Llama-3.1-70b/README.md","https://huggingface.co/spaces/ServiceNow/browsergym-leaderboard/blob/main/results/GenericAgent-GPT-4o-mini/README.md","Agent: A3; 341 tasks (full benchmark). Fine-tuned on A3-Synth trajectories via the Agent-as-Annotators framework.","Agent: GenericAgent; Increased max_steps from 50 to 100; publisher submission declares a modified evaluation protocol","BasicAgent claude-3.5-sonnet","Original paper; default budget; 3 runs; mean replication score","IterativeAgent claude-3.5-sonnet","BasicAgent o1-high","IterativeAgent o3-mini-high","BasicAgent deepseek-r1","BasicAgent gpt-4o","BasicAgent gemini-2.0-flash","BasicAgent o3-mini-high","Jamba-1.5-large (94B/398B)","RULER v1; 128K context; 13 synthetic tasks","Gemini-1.5-pro,-Text%20and%20images)","Qwen2.5-14B-Instruct-1M (14B)","Qwen3-235B-A22B (235B)","Jamba-1.5-mini (12B/52B)","EXAONE-4.0-32B (32B)","Qwen3-32B (32B)","Qwen3-14B (14B)","Qwen2.5-7B-Instruct-1M (7B)","MegaBeam-Mistral (7B)","GLM4 (9B)","ProLong (8B)","GPT-4-1106-preview","Qwen3-30B-A3B (30B)","Qwen3-8B (8B)","Yi (34B)","Llama3.1 (8B)","Command-R (35B)","Command-R-0824 (32B)","GradientAI/Llama3 (70B)","GradientAI/Llama3 (8B)","InternLM2.5 (7B)","Phi3-mini (3.8B)","Llama3.1 (70B)","Qwen3-4B (4B)","LWM (7B)","Command-R-plus-0824 (104B)","Command-R-plus (104B)","Qwen2 (72B)","Mistral-Large-2411 (123B)","Phi3-medium (14B)","Mixtral-8x7B (12.9B/46.7B)","GLM3 (6B)","Mixtral-8x22B (39B/141B)","FILM-7B (7B)","Mistral-Large-2407 (123B)","Mistral-Nemo","Mistral (7B)","DBRX (36B/132B)","LongAlpaca (13B)","LongChat (7B)","Qwen1.5 (72B)","Together (7B)","Gemini-1.5-Pro","Paper Table 4; Multimodal models","Qwen2.5-Math-72B","Paper Table 4; Text-only models","Qwen2-Math-72B","Qwen2.5-Math-7B","Qwen2-Math-7B","Pixtral-12B-2409","LLaVA-OV(Qwen2-7B)","Oracle; context-last; original 150-question panel","https://arxiv.org/html/2311.11944v1#S5.F2","Oracle; original 150-question panel","https://arxiv.org/html/2311.11944v1#A4.F5","Long Context; original 150-question panel","Claude2","Single Vector Store; original 150-question panel","Long Context; context-last; original 150-question panel","Shared Vector Store; original 150-question panel","Closed Book; original 150-question panel","Agent: ReAct; B2B; Single-turn Setting","gemini-2.5-flash","gemini-2.0-flash","llama4-maverick","llama3.1-405b","gpt-4o","llama3.1-70b","gpt-4o-mini","Agent: ReAct; B2C; Single-turn Setting","Agent: ReAct; B2B; Multi-turn Setting","Agent: ReAct; B2C; Multi-turn Setting","GPT-4V-1106","Agent: Navi; UIA tree + Omniparser; original 154-task paper evaluation","Phi3-V","Agent: Famou-Agent 2.0; 24 hours; mean any-medal rate","Agent: AIBuildAI; 24 hours; mean any-medal rate","Agent: CAIR MARS+; 24 hours; mean any-medal rate","Agent: MLEvolve; 12 hours; mean any-medal rate","Agent: PiEvolve (Fractal AI Research); 24 hours; mean any-medal rate","Deepseek-V3.2-Speciale","Agent: ML-Master 2.0; 24 hours; mean any-medal rate","Agent: CAIR MARS; 24 hours; mean any-medal rate","Agent: PiEvolve (Fractal AI Research); 12 hours; mean any-medal rate","Agent: Leeroo; 24 hours; mean any-medal rate","gpt-5-codex","Agent: Thesis; 24 hours; mean any-medal rate","Agent: CAIR MLE-STAR-Pro-1.5; 24 hours; mean any-medal rate","Agent: Famou-Agent; 24 hours; mean any-medal rate","gpt-5 (low verbosity/effort)","Agent: Operand ensemble; 24 hours; mean any-medal rate","Agent: CAIR MLE-STAR-Pro-1.0; 12 hours; mean any-medal rate","deepseek-r1","Agent: InternAgent; 12 hours; mean any-medal rate","gpt-5","Agent: R&D-Agent; 12 hours; mean any-medal rate","undisclosed","Agent: Neo multi-agent; 36 hours; mean any-medal rate","Agent: AIRA-dojo; 24 hours; mean any-medal rate","o3 + GPT-4.1","Agent: R&D-Agent; 24 hours; mean any-medal rate","Agent: ML-Master; 12 hours; mean any-medal rate","Agent: AIDE; 24 hours; mean any-medal rate","claude-3-5-sonnet-20240620","Agent: OpenHands; 24 hours; mean any-medal rate","llama-3.1-405b-instruct","Agent: MLAB; 24 hours; mean any-medal rate","3.5 Sonnet","Diamond; reasoning=N/A; original paper scaffold","Diamond; reasoning=High; original paper scaffold","Diamond; original paper scaffold; preliminary one-seed run","https://arxiv.org/html/2502.12115v4#A1.T11","Deepseek-R1","Full; reasoning=N/A; original paper scaffold","Full; reasoning=High; original paper scaffold","Full; original paper scaffold; preliminary one-seed run","Agent: OpenClaw; domain skills=on; meta prompt=on; non-safety tasks","Gemini 3.1 Flash-Lite","Browser-Use; May 2026 paper; full 106-task set","https://arxiv.org/html/2605.15777v2#S4.T2.6.1.3","GPT-5.4 High","https://arxiv.org/html/2605.15777v2#S4.T2.6.1.6","https://arxiv.org/html/2605.15777v2#S4.T2.6.1.5","Doubao Seed 2.0 Pro","https://arxiv.org/html/2605.15777v2#S4.T2.6.1.11","GPT-5.5 High","https://arxiv.org/html/2605.15777v2#S4.T2.6.1.4","https://arxiv.org/html/2605.15777v2#S4.T2.6.1.8","Browser-Use; May 2026 paper; 74 text-only tasks","https://arxiv.org/html/2605.15777v2#S4.T2.6.1.14","https://arxiv.org/html/2605.15777v2#S4.T2.6.1.13","Gemini 3.5 Flash High","https://arxiv.org/html/2605.15777v2#S4.T2.6.1.12","https://arxiv.org/html/2605.15777v2#S4.T2.6.1.7","https://arxiv.org/html/2605.15777v2#S4.T2.6.1.15","https://arxiv.org/html/2605.15777v2#S4.T2.6.1.10","https://arxiv.org/html/2605.15777v2#S4.T2.6.1.9","https://arxiv.org/html/2605.15777v2#S4.T2.6.1.16","max; batched tool; 500 steps","max; standard; 500 steps","medium; standard; 500 steps","Kimi 2.6","enabled; standard; 500 steps","Qwen 3.7-Plus","thinking; standard; 500 steps","Terminus-2; 2026-03; 112 tasks","GPT-5.4-High","MiMo-V2-Pro","MiniMax-2.7","OpenHands; 2026-04; 90 tasks","GLM-5.1-Thinking","Claude-Opus-4.7-High","Qwen-3.6-Plus-Thinking","Gemini-3.1-Pro-High","OpenHands; 2026-05; 100 tasks","Kimi-K2.7-Code-Thinking","Kimi-K2.6-Thinking","MiMo-V2.5-Pro-High","Doubao-Seed-2.0-Pro-High","MiniMax-M2.7-Reasoning","Terminus 2; 115 tasks","https://arxiv.org/html/2605.15846v2#S4.T2.8.1.18","https://arxiv.org/html/2605.15846v2#S4.T2.8.1.19","https://arxiv.org/html/2605.15846v2#S4.T2.8.1.20","https://arxiv.org/html/2605.15846v2#S4.T2.8.1.21","https://arxiv.org/html/2605.15846v2#S4.T2.8.1.22","https://arxiv.org/html/2605.15846v2#S4.T2.8.1.23","https://arxiv.org/html/2605.15846v2#S4.T2.8.1.24","https://arxiv.org/html/2605.15846v2#S4.T2.8.1.25","https://arxiv.org/html/2605.15846v2#S4.T2.8.1.26","https://arxiv.org/html/2605.15846v2#S4.T2.8.1.27","https://arxiv.org/html/2605.15846v2#S4.T2.8.1.28","OpenClaw; V1; 153 tasks","Qwen 3.5","claude-opus-4-6","Hermes; V1; 153 tasks","claude-sonnet-4-6","claude-haiku-4-5-20251001","gpt-5.4-2026-03-05","gpt-5.4-mini-2026-03-17","0-shot answer-only; 52 test subjects; API","https://arxiv.org/html/2305.08322v3#S3.T3","Claude-v1.3","Bloomz-mt","BigScience","0-shot answer-only; 52 test subjects; Weights","GLM-130B","Tsinghua","Claude-instant-v1.0","MOSS","Fudan","Chinese-Alpaca-13B","Cui et al.","5-shot answer-only; 52 test subjects; API","https://arxiv.org/html/2305.08322v3#S3.T4","5-shot answer-only; 52 test subjects; Weights","5-shot CoT; 52 test subjects; API","https://arxiv.org/html/2305.08322v3#S3.T5","5-shot CoT; 52 test subjects; Weights","0-shot answer-only; eight hard test subjects; API","https://arxiv.org/html/2305.08322v3#S3.T6","0-shot answer-only; eight hard test subjects; Weights","5-shot answer-only; eight hard test subjects; API","5-shot answer-only; eight hard test subjects; Weights","5-shot CoT; eight hard test subjects; API","5-shot CoT; eight hard test subjects; Weights","0-shot answer-only; 52 validation subjects; API","https://arxiv.org/html/2305.08322v3#S3.T7","0-shot answer-only; 52 validation subjects; Weights","5-shot answer-only; 52 validation subjects; API","5-shot answer-only; 52 validation subjects; Weights","code-davinci-002","8-shot standard; greedy decoding; 1 exemplar order","https://arxiv.org/html/2201.11903v6#A2.T2","PaLM 540B","text-davinci-002","PaLM 62B","PaLM 8B","UL2 20B","text-curie-001","text-babbage-001","text-ada-001","8-shot CoT; greedy decoding; 1 exemplar order","Claude 3.5 Sonnet (20241022)","HELM v1.13.0; 5-shot; 14042 evaluation instances; mean across 57 subjects","https://crfm.stanford.edu/helm/mmlu/latest/","DeepSeek v3","Gemini 1.5 Pro (002)","Claude 3.5 Sonnet (20240620)","Claude 3 Opus (20240229)","Llama 3.1 Instruct Turbo (405B)","Qwen2.5 Instruct Turbo (72B)","Gemini 1.5 Pro (001)","GPT-4 (0613)","Qwen2 Instruct (72B)","Amazon Nova Pro","Palmyra-X-004","GPT-4 Turbo (2024-04-09)","Gemini 1.5 Pro (0409 preview)","Llama 3.2 Vision Instruct Turbo (90B)","Llama 3.1 Instruct Turbo (70B)","Mistral Large 2 (2407)","Gemini 2.0 Flash (Experimental)","GPT-4 Turbo (1106 preview)","Llama 3 (70B)","Yi Large (Preview)","Llama 3.3 Instruct Turbo (70B)","Palmyra X V3 (72B)","PaLM-2 (Unicorn)","Jamba 1.5 Large","Gemini 1.5 Flash (001)","Mixtral (8x22B)","Gemini 1.5 Flash (0514 preview)","Solar Pro","Phi-3 (14B)","Amazon Nova Lite","Qwen1.5 Chat (110B)","GPT-4o mini (2024-07-18)","Claude 3 Sonnet (20240229)","Gemma 2 (27B)","Phi-3 (7B)","Qwen1.5 (32B)","Claude 3.5 Haiku (20241022)","DBRX Instruct","Gemini 1.5 Flash (002)","Claude 3 Haiku (20240307)","Claude 2.1","Qwen2.5 Instruct Turbo (7B)","DeepSeek LLM Chat (67B)","Gemma 2 (9B)","Mixtral (8x7B 32K seqlen)","Amazon Nova Micro","Gemini 1.0 Pro (001)","Jamba 1.5 Mini","Llama 2 (70B)","Command R Plus","PaLM-2 (Bison)","GPT-3.5 Turbo (0613)","Claude Instant 1.2","Mistral Large (2402)","Mistral Small (2402)","Qwen1.5 (14B)","Arctic Instruct","GPT-3.5 Turbo (0125)","Llama 3 (8B)","Gemma (7B)","Jamba Instruct","Mistral NeMo (2402)","Command R","Yi (6B)","Qwen1.5 (7B)","Mistral Instruct v0.3 (7B)","Mistral v0.1 (7B)","Llama 3.2 Vision Instruct Turbo (11B)","Llama 3.1 Instruct Turbo (8B)","Llama 2 (13B)","Llama 2 (7B)","HELM v1.13.0; 5-shot; 1000 evaluation instances","https://crfm.stanford.edu/helm/lite/latest/","Gemini 1.0 Pro (002)","Gemma 2 Instruct (27B)","Claude v1.3","Gemma 2 Instruct (9B)","Palmyra X V2 (33B)","Mistral Medium (2312)","GPT-3.5 (text-davinci-003)","Claude 2.0","Phi-2","LLaMA (65B)","GPT-3.5 (text-davinci-002)","Command","Falcon (40B)","Jurassic-2 Jumbo (178B)","Jurassic-2 Grande (17B)","Command Light","Luminous Supreme (70B)","Luminous Extended (30B)","Falcon (7B)","OLMo (7B)","Luminous Base (13B)","mini-SWE-agent 2.0.0a0; 1 attempt","GLM 5","MiniMax M2.5","DeepSeek V3.2","Direct; context 4M","Direct; context 128k","Direct; context 2M","Direct; context 1M","Direct; context 256k","Direct; context 200k","Direct; context 512k","With subtitles; frames: 10 fps","With subtitles; frames: 1/0.5 fps","With subtitles; frames: 1024","With subtitles; frames: 1800","With subtitles; frames: 768","With subtitles; frames: 384","With subtitles; frames: 64","With subtitles; frames: 128","With subtitles; frames: 1 fps","With subtitles; frames: 256","With subtitles; frames: 120","With subtitles; frames: 96","With subtitles; frames: 2 fps","With subtitles; frames: 180","With subtitles; frames: 512","With subtitles; frames: 32","With subtitles; frames: 250","With subtitles; frames: 100","With subtitles; frames: 10","With subtitles; frames: 20","With subtitles; frames: 16","With subtitles; frames: 4","With subtitles; frames: 8","Doubao-pro-32k","Gemini-1.5-pro","Qwen-Max","moonshot-v1-8k","GPT-4-turbo","Baichuan3-turbo","DeepSeek-67B","Qwen2.5-32B","LLaMA3.1-70B","Doubao-lite-4k","Qwen2.5-14B","DeepSeek-V2-Lite-Chat","InternLM2.5-20B","Yi-1.5-34B","GLM4-9B","InternLM2.5-7B","DeepSeek-7B","Baichuan2-13B","Yi-1.5-9B","LLaMA3.1-8B","Qwen2.5-3B","Yi-1.5-6B","Baichuan2-7B","ChatGLM3-6B","Qwen2.5-1.5B","InternLM2.5-1.8B","V2 Hard; harness as listed in the model name","GPT-6 Astra (Codex) high","GPT-5.6 Terra (Codex) xhigh","GPT-5.6 Sol (Codex) xhigh","Haiku 4.5 (Claude Code) xhigh","Hard-Complete; prefill=False; prompted=True","Hard-Complete; prefill=True; prompted=True","KwaiCoder-23B-A4B-v1","Hard-Complete; prefill=True; prompted=False","https://huggingface.co/Kwaipilot/KwaiCoder-23B-A4B-v1","https://huggingface.co/m-a-p/OpenCodeInterpreter-DS-33B","DeepSeek-Coder-33B-Base","https://huggingface.co/deepseek-ai/deepseek-coder-33b-base","CodeLlama-70B-Base","https://huggingface.co/codellama/CodeLlama-70b-hf","https://huggingface.co/Qwen/CodeQwen1.5-7B","Llama-3-70B-Base","https://huggingface.co/meta-llama/Meta-Llama-3-70B","Mixtral-8x22B-Base","https://huggingface.co/mistralai/Mixtral-8x22B-v0.1","DeepSeek-Coder-6.7B-Base","https://huggingface.co/deepseek-ai/deepseek-coder-6.7b-base","CodeLlama-34B-Base","https://huggingface.co/codellama/CodeLlama-34b-hf","https://huggingface.co/bigcode/starcoder2-15b","https://huggingface.co/deepseek-ai/DeepSeek-Coder-V2-Lite-Base","Granite-Code-20B-Base","https://huggingface.co/ibm-granite/granite-20b-code-base","Granite-Code-34B-Base","https://huggingface.co/ibm-granite/granite-34b-code-base","CodeGemma-7B","https://huggingface.co/google/codegemma-7b","CodeLlama-13B-Base","https://huggingface.co/codellama/CodeLlama-13b-hf","https://huggingface.co/01-ai/Yi-1.5-34B","https://huggingface.co/bigcode/starcoder2-7b","https://huggingface.co/01-ai/Yi-1.5-9B","Granite-Code-8B-Base","https://huggingface.co/ibm-granite/granite-8b-code-base","Granite-Code-8B-Instruct","https://huggingface.co/ibm-granite/granite-8b-code-instruct","CodeGemma-2B","https://huggingface.co/google/codegemma-2b","Llama-3-8B-Base","https://huggingface.co/meta-llama/Meta-Llama-3-8B","CodeLlama-7B-Base","https://huggingface.co/codellama/CodeLlama-7b-hf","Granite-Code-3B-Instruct","https://huggingface.co/ibm-granite/granite-3b-code-instruct","Mistral-7B-v0.3","https://huggingface.co/mistralai/Mistral-7B-v0.3","https://huggingface.co/bigcode/starcoder2-3b","https://huggingface.co/01-ai/Yi-1.5-6B","DeepSeek-Coder-1.3B-Base","https://huggingface.co/deepseek-ai/deepseek-coder-1.3b-base","Granite-Code-3B-Base","https://huggingface.co/ibm-granite/granite-3b-code-base","Full-Instruct; prefill=True; prompted=True","Full-Instruct; prefill=False; prompted=True","Full-Complete; prefill=False; prompted=True","Full-Complete; prefill=True; prompted=True","Full-Complete; prefill=True; prompted=False","AskData","AT&T CDO - DSAIR","Overall track; oracle knowledge: ✔️","https://arxiv.org/abs/2505.19988","GrainSQL","Sarim Chaudhry, Purdue University","DataGallery-Text2SQL","Huawei 2012 Labs","https://datagallery.cn/benchmark","SiriusAI-SQL","Agentar-Scale-SQL","https://arxiv.org/abs/2509.24403","Sber Text2SQL","SberData Research","Xiaomi Text2SQL","Xiaomi ITP & Data","RAS","Adya AI","DeepEye","https://arxiv.org/pdf/2603.28889","MarkovSQL","DeepEye-SQL","https://arxiv.org/abs/2510.17586","Amazon Ads - SpektrBot","https://bird-glass-cc9.notion.site/Spektr-SQL-BIRD-Benchmark-Submission-37da39e6358b806bae30f66ee09fd934","LongData-SQL","LongShine AI Research","AxisSQL","UST","GT-ChatBI-SQL","MR Tech","Zhiwen-Lingsi-Agent","China Telecom, TeleAI","MIC -SQL","DataSpace-Text2SQL","Institute of Dataspace, Hefei \nComprehensive National Science Center-IDATA","JoyDataAgent-SQL","JD JoyWork & JD-CHO & JDT-RMC","RS","CHASE-SQL + Gemini","Google Cloud","https://arxiv.org/abs/2410.01943","xiaoyi-text-to-sql","wenyuai","RED-SQL","South China Normal University","Sinovatio-SQL","Sinovatio AI Lab","TCDataAgent-SQL","Tencent Cloud","Contextual-SQL","Contextual AI","XiYan-SQL","CYAN-SQL","Tencent Cloud / Fudan University","DB-SQL","MCR-SQL + Qwen2.5-Coder-32B-Instruct","MRS-Agent","AIC-Lab, Chungbuk National University","Blender-SQL+Qwen3.8-27B","DAMENG","JT-SQLAgent","CMCC, JIUTIAN Research","flash-sqlite: Qwen3.5-2B Generator + Self-Judge (SC@32)","CSC-SQL + XiYanSQL-QwenCoder-32B-2412","Wuhan University of Technology + University of Science and Technology of China","https://arxiv.org/abs/2505.13271","Kwai-AutoSQL","ExSL + granite-34b-code","IBM Research AI","Reasoning-SQL 14B","Google Cloud / Stanford","https://arxiv.org/pdf/2503.23157","GenaSQL","Gena Co.","https://arxiv.org/abs/2505.14174","OpenSearch-SQL, v2 + GPT-4o","https://arxiv.org/abs/2502.14913","EGV-SQL + Qwen3.6-27B","Tampere University","OmniSQL-32B","Renmin University of China + ByteDance Infra Lab","https://arxiv.org/abs/2503.02240","Distillery + GPT-4o","Distyl AI Research","https://arxiv.org/abs/2408.07702","Share + GPT-5","HKU","https://arxiv.org/abs/2506.00391","CSC-SQL + Qwen2.5-Coder-7B-Instruct","LEAF-SQL","JXUFE","Queryosity","Conrad Labs","CHESS IR + CG + UT","Stanford","https://arxiv.org/abs/2405.16755","SLM-SQL + Qwen2.5-Coder-1.5B-Instruct","https://arxiv.org/abs/2507.22478","Alpha-SQL + Qwen2.5-Coder-32B-Instruct","https://arxiv.org/abs/2502.17248","Insights AI","Uber Freight","PURPLE + RED + GPT-4o","Fudan University + Transwarp Technology","wmyaccept-SQL","Independent Researcher","GSR","PB-SQL, GPT-4o","Seoul National University","TC-SQL","mindflow.ai","RECAP + Gemini","ByteBrain","ByteDance Infra Lab","RSL-SQL + GPT-4o","Qwen3.5-2B + StructCoT Distillation (SC@7)","PRIME-SQL","OmniSQL-7B","ExSL + granite-20b-code","AskData + GPT-4o","Overall track; oracle knowledge: ","CHESS IR + SS + CG","Elysian-SQL + gpt-oss-120b","E-SQL + GPT-4o","Bilkent University","Arcwise + GPT-4o","Arcwise","NucliOS","MathCo","RSL-SQL + DeepSeek-v2","MCS-SQL + GPT-4","Dunamu","https://arxiv.org/abs/2405.07467","SCL-SQL","Xffuture","OpenSearch-SQL,v1 + GPT-4","SFT CodeS-15B + SQLFixAgent","Soochow University","PURPLE + GPT-4o","MSL-SQL + DeepSeek-V2.5","Wuhan University of Technology","EBA-SQL + GPT-4","Sense","GRA-SQL","Tencent CDP-youpu","SLM-SQL + Qwen2.5-Coder-0.5B-Instruct","{Chat2Query} (GPT-4 + data entity modeling) (PingCAP)","PingCAP","Dubo-SQL, v1","Mercator Technologies","DTS-SQL + DeepSeek 7B","University of Alberta","https://arxiv.org/abs/2402.01117","E-SQL + GPT-4o mini","MAC-SQL + GPT-4","BUAA & Tencent","https://arxiv.org/abs/2312.11242","TA-SQL + GPT-4","https://arxiv.org/abs/2405.15307","DAIL-SQL + GPT-4","Alibaba Group","https://arxiv.org/abs/2308.15363","DIN-SQL + GPT-4","https://arxiv.org/abs/2304.11015","Interactive-T2S","Peking University","https://dl.acm.org/doi/10.1145/3746252.3761052","Open-SQL","Genloop's Sentinel Agent v2 Pro","Genloop","Spider 2.0-Snow","https://genloop.ai/","usenative.ai","QUVI-3 + Gemini-3-pro-preview","TCDataAgent-SQL with Contextual Scaling Engine","Tencent Cloud Big Data","Prism Swarm with Deepthink + Claude-Sonnet-4.5","Paytm","https://github.com/paytm/prism","Genloop's Sentinel Agent v2","Ask Data with Relational Knowledge Graph","AT&T CDO & RelationalAI","QUVI-3 + Claude-Opus-4.6","ByteBrain-Agent","ByteDance Infra System Lab","MAESTRO — Multi-Agent Engine for Semantic T2S Reasoning & Orchestration","IntelligentOps Tigers","Genloop's Sentinel Agent v1.5","Deepinsight Agent","AiCheng Agent","alibaba_cfo_tech","Omni 2.0","Prism Swarm + Claude-Sonnet-4.5","LingXi Agent + Claude-Sonnet-4.5","Arctic-FLEX","Snowflake AI Research","Sophon-Agent","ByteDance DataPlatform LLM","APEX-SQL","LIGHTSPEED","https://arxiv.org/pdf/2602.16720","QiSi-SQL + Deepseek3.2","Ant Group Tech Risk","PExA","Bloomberg - AI Engineering group","DIA (Data Intelligence Agents)","C3 AI","https://c3.ai/","Chicory AI Agent + Claude Sonnet 4.5 + Opus 4.5 Judge","Chicory AI","https://app.chicory.ai","Omni 1.0","360 Labs","SSDAT + GPT-5","FlexSQL + gpt-oss-120b","https://arxiv.org/abs/2605.02815","WindAgent + Claude-4-Sonnet","MeiTuan AI For FinData","PAI-DataSurfer Agent","Alibaba Cloud Computing Platform","AutoLink + DeepSeek-R1","https://arxiv.org/abs/2511.17190","GD-TTS Agent + GPT-5.4-mini","Gagandeep Singh","PGV-Agent + GLM-5","Yiwei, Bangguo","Meituan-agent","Meituan FinData Intelligence","KDGCCloud-KCILab + Qwen3-Max","KDGCCloud-KCILab Team","AgenticView + GPT-5-mini","Griffith BigData Lab, Griffith University","CT-ChatBI + Qwen3-Max","Chat2DB-Agent + Claude-4-Sonnet","Chat2DB","ReFoRCE + DeepSeek-V3","Spider-Agent + Qwen3-Coder-Plus","Bot-UTAI","Spider-Agent + Qwen3-Coder","Spider-Agent + o1-2024-12-17","Spider-Agent + Claude-3.5-Sonnet-20241022","AWS ProServe","Spider-Agent + Gemini-2.0-Pro","CollideNL2SQL + GPT-4o","Collide Tech","Spider-Agent + DeepSeek-V3","Dail-SQL + GPT-4o","SignalPilot Agent","signalpilot.ai","Spider 2.0-DBT","https://github.com/SignalPilot-Labs/SignalPilot/","Shadowfax-DBT-Agent + GPT-5","Shadowfax Data","Spider-Agent-Extended + GPT-5","Woods, A. '25.","USTC-KCIL DBT Agent + GPT-5.4","USTC-KCIL","Chicory AI Agent + Claude Sonnet 4.5","Spider-Agent-DBT + GPT-5.4","Symbiote Agent + GPT-5","DAQUV_QUVI_Agent","Spider-Agent + GPT-4.1","Spider-Agent + GPT-4o","Spider-Agent + o3-mini","Spider-Agent + o3","0-shot; publisher README","DiMind","云天天书","GPT-5 (high)","Vanilla Gaia2; Meta reference agent","Claude-4-Sonnet Thinking","GPT-5 (low)","DeepSeek-v3.1 Terminus","DeepSeek-v3.1","Kimi-K2","GPT-5 (minimal)","Qwen3-235B-thinking","GPT-OSS 120B (high)","Qwen3-235B","Llama 3.3 70B Instruct","Base; 5-shot; Qwen2.5 release","https://qwenlm.github.io/blog/qwen2.5-llm/#qwen25-base-language-model-evaluation","Llama-3-405B","Mixtral-8x22B","Gemma2-27B","Gemma2-9B","Llama3-8B","Gemma2-2.6B","Qwen2.5-0.5B","Base; 4-shot; Qwen2.5 release","Agent: Loop; reasoning=high; mean-score","Agent: Loop; reasoning=max; mean-score","Agent: Loop; reasoning=xhigh; mean-score","reasoning=high; held-out mean rubric score","o3 Pro","Gemini 2.5 Flash (On)","held-out mean rubric score","Opus 4.1","Gemini 2.5 Pro (On)","Vending-Bench v1; mean final net worth; USD","Vending-Bench v1; mean final net worth; USD; source label=Kimi K2 Thinking (Moonshot API)","Gemini Robotics-ER 1.5","Gemini 2.5 Pro (preview-03-25)","Vending-Bench v1; mean final net worth; USD; source label=Kimi K2 Thinking (3rd party API)","Claude 3.5 Haiku","Gemini 2.0 Pro","Agent: Tau half-duplex; BM25 grep; reasoning=max; user simulator=GPT-5.2 low; judge=Gemini 3.5 Flash; 97 tasks × 3","Agent: Tau half-duplex; BM25 grep; reasoning=high; user simulator=GPT-5.2 low; judge=Gemini 3.5 Flash; 97 tasks × 3","Agent: Tau half-duplex; BM25 grep; reasoning=xhigh; user simulator=GPT-5.2 low; judge=Gemini 3.5 Flash; 97 tasks × 3","Agent: Function Calling; prompting","Agent: ReAct","Agent: Function Calling","claude-3.5-sonnet","Agent: Act","claude-3-sonnet","llama3.1-8b","Table 1; four-component average; Grounding v2","GPT o3","Claude 4.5 Sonnet Thinking","Claude 4 Sonnet","GPT o4 mini","Grok 4 Fast","OpenAI o3-mini-low","Original author leaderboard; main problems","OpenAI o3-mini-high","OpenAI o3-mini-medium","OpenAI o1-preview","Claude3.5-Sonnet (new)","Deepseek-Coder-v2","Deepseek-v3","Claude3-Sonnet","Llama-3.1-405B-Chat","OpenAI o1-mini","Qwen2-72B-Instruct","Llama-3-70B-Chat","Llama-3.1-70B-Chat","Seed 2.1 Pro","64 main problems; background provided; pass@1; two-environment OR grading","DeepSeek V4 Pro Preview","DeepSeek V4 Flash Preview","Kaggle BrowseComp v1; Score task v1; 1 attempt; latest run; version gemini-3.1-pro-preview","https://www.kaggle.com/benchmarks/openai/browsecomp/leaderboard","Kaggle BrowseComp v1; Score task v1; 1 attempt; latest run; version gpt-5-2025-08-07","Kaggle BrowseComp v1; Score task v1; 1 attempt; latest run; version gemini-3-flash-preview","Gemini 2.5 Pro Preview","Kaggle BrowseComp v1; Score task v1; 1 attempt; latest run; version gemini-2.5-pro-preview-06-05","Kaggle BrowseComp v1; Score task v1; 1 attempt; latest run; version o1-2024-12-17","Kaggle BrowseComp v1; Score task v1; 1 attempt; latest run; version gpt-4.1-2025-04-14","Kaggle BrowseComp v1; Score task v1; 1 attempt; latest run; version o4-mini-2025-04-16","Kaggle BrowseComp v1; Score task v1; 1 attempt; latest run; version deepseek-r1-0528","deepseek-v3.1","Kaggle BrowseComp v1; Score task v1; 1 attempt; latest run; version deepseek-v3.1","Gemini 2.5 Flash Preview","Kaggle BrowseComp v1; Score task v1; 1 attempt; latest run; version gemini-2.5-flash-preview-05-20","Grok 3 Mini","Kaggle BrowseComp v1; Score task v1; 1 attempt; latest run; version grok-3-mini","Kaggle BrowseComp v1; Score task v1; 1 attempt; latest run; version gpt-oss-120b","o3 mini","Kaggle BrowseComp v1; Score task v1; 1 attempt; latest run; version o3-mini-2025-01-31","Kaggle BrowseComp v1; Score task v1; 1 attempt; latest run; version deepseek-v3","Kaggle BrowseComp v1; Score task v1; 1 attempt; latest run; version grok-3","gpt-oss-20b","Kaggle BrowseComp v1; Score task v1; 1 attempt; latest run; version gpt-oss-20b","Kaggle BrowseComp v1; Score task v1; 1 attempt; latest run; version gemini-3.1-flash-lite-preview","Kaggle BrowseComp v1; Score task v1; 1 attempt; latest run; version claude-opus-4-20250514","Kaggle BrowseComp v1; Score task v1; 1 attempt; latest run; version claude-opus-4-1-20250805","Kaggle BrowseComp v1; Score task v1; 1 attempt; latest run; version grok-2-1212","Kaggle BrowseComp v1; Score task v1; 1 attempt; latest run; version claude-3-5-sonnet-20241022","o1 mini","Kaggle BrowseComp v1; Score task v1; 1 attempt; latest run; version o1-mini-2024-09-12","Kaggle BrowseComp v1; Score task v1; 1 attempt; latest run; version claude-3-7-sonnet-20250219","Kaggle BrowseComp v1; Score task v1; 1 attempt; latest run; version gpt-4o-2024-08-06","Kaggle BrowseComp v1; Score task v1; 1 attempt; latest run; version claude-sonnet-4-20250514","Codestral","Kaggle BrowseComp v1; Score task v1; 1 attempt; latest run; version codestral-2501","GPT-3.5 Turbo","Kaggle BrowseComp v1; Score task v1; 1 attempt; latest run; version gpt-3.5-turbo-1106","Kaggle BrowseComp v1; Score task v1; 1 attempt; latest run; version gpt-4o-mini-2024-07-18","Kaggle BrowseComp v1; Score task v1; 1 attempt; latest run; version gemini-2.0-flash-001","Kaggle BrowseComp v1; Score task v1; 1 attempt; latest run; version gemma-3-27b-it","Kaggle BrowseComp v1; Score task v1; 1 attempt; latest run; version mistral-large-2411","Kaggle BrowseComp v1; Score task v1; 1 attempt; latest run; version gemma-3-12b-it","Mixtral 8x22B","Kaggle BrowseComp v1; Score task v1; 1 attempt; latest run; version open-mixtral-8x22b-2404","Qwen 3 235B A22B Thinking","Kaggle BrowseComp v1; Score task v1; 1 attempt; latest run; version qwen3-235b-a22b-thinking-2507","Kaggle BrowseComp v1; Score task v1; 1 attempt; latest run; version gemini-1.5-flash-002","Kaggle BrowseComp v1; Score task v1; 1 attempt; latest run; version gemini-1.5-pro-002","Kaggle BrowseComp v1; Score task v1; 1 attempt; latest run; version gemma-3-4b-it","Granite 4.0 Small","Kaggle BrowseComp v1; Score task v1; 1 attempt; latest run; version granite-4.0-h-small","Kaggle BrowseComp v1; Score task v1; 1 attempt; latest run; version claude-3-5-haiku-20241022","Ministral 8B","Kaggle BrowseComp v1; Score task v1; 1 attempt; latest run; version ministral-8b-2410","Ministral 3B","Kaggle BrowseComp v1; Score task v1; 1 attempt; latest run; version ministral-3b-2410","Gemini 1.5 Flash 8B","Kaggle BrowseComp v1; Score task v1; 1 attempt; latest run; version gemini-1.5-flash-8b-001","gpt-6-astra","codex; reasoning=max","https://rsi-benchmark.com/tasks/biomedical-tinyclip-data-curation#results","gpt-6.1-sol","claude-opus-5-5","claude-code; reasoning=max","claude-sonnet-5-5","qwen3.8-max","terminus-2; reasoning=max (= xhigh)","kimi-k3","terminus-2; reasoning=max","glm-5.3","deepseek-v4.1-flash","Mistral Large","Standalone","GPT-5.4-mini","GPT-5.3","Claude-Fable-5","Qwen3.7-Plus","DeepSeek-v4-Pro","DeepSeek-v4-Flash","GB/T-Reviewer","Qwen3-14B-AWQ","GLM-4.7-Flash","gpt-oss-20B","Gemma-4-12B","Qwen3.5-4B","MedGemma-1.5-4B","GPT-5.6","gpt-5.6-sol","Qwen3.6 27B","Base","SFT w/o CoT","CoT SFT","GPT-6-Astra","mini-SWE-agent · highest reasoning effort · 181 graded","GPT-5.6-Sol","mini-SWE-agent · highest reasoning effort · 254 graded","Claude-Fable-5.1","mini-SWE-agent · highest reasoning effort · 223 graded","Claude-Opus-5","mini-SWE-agent · highest reasoning effort · 256 graded","mini-SWE-agent · highest reasoning effort · 262 graded","Grok-4.5","mini-SWE-agent · highest reasoning effort · 218 graded","Basic harness · high","Basic harness · max","Basic harness","Prime Agent · high","Claude Code · high","Codex · high","Kimi Code · max","Qwen 3.6 27B","PRO-LONG · high","Samaya Sterling","Legacy Samaya System","high effort","Finance Agent v2 · high","Finance Agent v2 · xhigh","Finance Agent v2","Finance Agent v2 · medium","GPT 5.6 Sol","Finance Agent v2 · max","Web Search · high","Finance Agent v2 · on ( level not supported)","Web Search · medium","max","high","xhigh","Opus-5","Fable-5","Opus-4.8","low","Sonnet-4.6","medium","Haiku-4.5","non-think","think","Gemini-3.1-Flash-Lite","default","anthropic/claude-opus-5-5","anthropic/claude-sonnet-5-5","anthropic/claude-fable-5-1","anthropic/claude-opus-5","openai/gpt-6-astra","effort max","anthropic/claude-fable-5","meta/muse_spark_1_3_max","openai/gpt-5.6-sol","openai/gpt-6-sol","google/gemini-3.8-flash","effort high","anthropic/claude-opus-4-8","meta/muse_spark_1_3","effort xhigh","grok/grok-4.7","openai/gpt-5.6-luna","anthropic/claude-sonnet-5","openai/gpt-5.6-terra","xiaomi/mimo-v2.6-flash","xiaomi/mimo-v2.6-pro","google/gemini-3.7-flash","grok/grok-4.6","openai/gpt-6-luna","deepseek/deepseek-v4.1-flash","kimi/kimi-k3","openai/gpt-5.5","meta/muse_spark_1_2","zai/glm-5.3","anthropic/claude-opus-4-7","tencent/hy4-preview","google/gemini-3.6-flash","meta/muse_spark_1_1","deepseek/deepseek-v4-flash-0731","zai/glm-5.2","google/gemini-3.5-flash","deepseek/deepseek-v4-pro-0813","alibaba/qwen3.8-max","grok/grok-4.5","anthropic/claude-sonnet-4-6","alibaba/qwen3.8-27b","zai/glm-5.3-flash","alibaba/qwen3.7-max","kimi/kimi-k2.6","deepseek/deepseek-v4-pro","minimax/MiniMax-M3","google/gemini-3.1-pro-preview","xiaomi/mimo-v2.5-pro","xiaomi/mimo-v2.5","openai/gpt-5.4-mini-2026-03-17","alibaba/qwen3.7-plus","google/gemini-3.5-flash-lite","thinkingmachines/inkling","effort 0.99","openai/gpt-5.4-nano-2026-03-17","ant/ling-3.0-flash-af-rc3","alibaba/qwen3.6-plus","thinkingmachines/inkling-small","google/gemini-3-flash-preview","nvidia/nemotron-3-ultra-550b-a55b","kimi/kimi-k2.5-thinking","minimax/MiniMax-M2.7","grok/grok-4.3","anthropic/claude-haiku-4-5-20251001-thinking","ant/ling-3.0-flash-2607","mistralai/mistral-medium-3.5","grok/grok-4.20-0309-reasoning","google/gemini-3.1-flash-lite-preview","inception/mercury-2.5","fireworks/nemotron-lightning-3p5-30b-a3b","Claude Code · max","Antigravity · high","Codex · max","SpaceXAI","Grok Build · xhigh","Muse Code · max","Qwen Code · max","Codex · xhigh","GPT 5.4","Claude Opus 4.5 (Nonthinking)","gpt-5.6-terra","glm-5.2","grok-4.5","gemini-3-flash","claude-sonnet-5","claude-opus-4.8","gemini-3.5-flash","minimax-m3","claude-haiku-4.5","GPT5.6-sol","GLM5.2","Hy3","Gemini-3.5-flash","GPT5.5","GLM5.1","Seed-2.1-pro","OpenResearcher","GeoMMAgent","Intern-S1-mini","EarthVerse","Nemotron Nano","MiroThinker 8B","GeoGPT","Qwen3-4B","GeohazardGPT","Doubao Seed 2.0 Mini","Tencent Hy3","Qwen-Planner-Agent","27B planner system","Reference harness","Qwen-3.8-Max","Muse-Spark-1.3","Qwen-3.8-Max-0902","Gemini-3.6-Flash","Qwen-3.8-27B","Qwen-3.7-Max","Qwen-3.7-Plus","v1 · 610 prompts · 3 judges","claude-fable-5-1","claude-opus-5","gemini-3.6-flash","gemini-3.8-flash","gpt-3.5-turbo","mistral-large-latest","mistral-small-latest","qwen3-8b","qwen3.7-plus","gpt-6-luna","kimi-k2p6","kimi-k2p7-code","qwen3p6-plus","glm-5p2","gpt-5.6-luna","kimi-k2p5","qwen3p7-plus","minimax-m2p7","gpt-5-mini","gpt-5.4-mini","Gemini 3.8 Flash Cyber","Antigravity","grok-cli","opencode","NVIDIA Nemotron 3 Ultra (550B-A55B)","Claude Mythos 5.1","google/gemini-4-carbon","alibaba/qwen3.8-max-internal","google/gemini-4-argon","deepseek/deepseek-v4.1-flash-1017","meta/muse-wonder-full","meta/muse-wonder","tencent/hy4","minimax/MiniMax-M3.1-flash","anthropic/claude-haiku-5-5","fireworks/ember-1","mistralai/mistral-large-4-oss","mistralai/mistral-large-4","stepfun/step-5-preview","meta/muse-spark-open","kimi/kimi-k3.1-pro","openai/gpt-6.1-sol","inception/mercury-3","openai/gpt-6.1-terra","baseten/nemotron-3.5-super-vl-120b-a12b","reflection/Beam-1.0","Terminus 2 · live leaderboard","Terminus 2 · highest available at evaluation","Terminus 2 · xhigh","Opus 5.5 Max","CursorBench 4.0","Opus 5.5 Extra High","Opus 5.5 High","Sonnet 5.5 Max","Sonnet 5.5 Extra High","Gemini 4 Argon High","Gemini 4 Argon Medium","Opus 5.5 Medium","Fable 5.1 Max","Fable 5.1 Extra High","Gemini 4 Argon Low","Muse Wonder Max","Fable 5.1 High","Haiku 5.5 Max","Muse Wonder Extra High","Sonnet 5.5 High","DeepSeek V4.1 Flash 1017 Max","Fable 5.1 Medium","Opus 5 Max","Grok 4.7 Extra High","DeepSeek V4.1 Flash 1017 Medium","Opus 5 Extra High","Muse Wonder High","DeepSeek V4.1 Flash 1017 High","Muse Wonder Medium","Fable 5.1 Low","DeepSeek V4.1 Flash 1017 Low","Opus 5 High","Haiku 5.5 Extra High","Grok 4.7 High","Opus 5.5 Low","Opus 5 Medium","GLM 5.3 Max","Haiku 5.5 High","GPT-5.6 Sol Max","Muse Wonder Low","Grok 4.7 Medium","Grok 4.6 Extra High","GPT-5.6 Terra Max","Opus 5 Low","Grok 4.6 High","Gemini 3.8 Flash High","Sonnet 5.5 Medium","GLM 5.3 High","GPT-5.6 Sol Extra High","Muse Spark 1.3 Extra High","Gemini 3.8 Flash Medium","Haiku 5.5 Medium","GLM 5.3 Flash Max","Grok 4.6 Medium","GPT-5.6 Luna Max","Sonnet 5.5 Low","GPT-5.6 Sol High","Muse Wonder Minimal","Sonnet 5 Max","GPT-5.6 Terra Extra High","Grok 4.6 Low","Muse Spark 1.3 High","GLM 5.3 Low","Grok 4.7 Low","GPT-5.6 Luna Extra High","Muse Spark 1.3 Medium","Sonnet 5 Extra High","GPT-5.6 Sol Medium","GLM 5.3 Flash High","Haiku 5.5 Low","Sonnet 5 High","GPT-5.6 Terra High","GPT-5.6 Luna High","Muse Spark 1.3 Low","Sonnet 5 Medium","Composer 2.5","GPT-5.6 Terra Medium","GLM 5.3 Flash Low","GPT-5.6 Terra Low","GPT-5.6 Sol Low","Muse Spark 1.3 Minimal","Sonnet 5 Low","GPT-5.6 Luna Medium","GPT-5.6 Luna Low","HIGH","deepseek-v4-flash-0731","Native harness · Python / SageMath · max reasoning","Native harness · Python / SageMath · high reasoning","Claude-Opus-5.5","Native harness · Python / SageMath · low reasoning","Native harness · Python / SageMath","Native harness · Python / SageMath · Max reasoning","Native harness · Python / SageMath · xhigh reasoning","Native harness · Python / SageMath · Think reasoning","Parallel Search Fast + Extract","Pareto 26.9","Particle-1","DeepSeek V4.1 Flash (1017)","Pareto 26.10","Mistral Large 4.0","MiMo v2.6 Pro","Ultramarine 1 Flash","MiMo v2.6 Flash","Hunyuan 3","Nemotron 3 Ultra 550B","MiMo v2.5","Laguna S 2.1","Command A Plus","Grok-4.6","Grok Build","GPT-5.6-luna","GPT-5.6-terra","Antigravity CLI","Qwen3.8-27B","OpenHands; effort max","OpenHands","OpenHands; effort high","OpenHands; effort 0.99","codex; effort max","claude-code; effort max","google-computer-use; effort high","muse-code; effort max","antigravity; effort high","Claude Code 2.1.280 · max","mini-swe-agent 2.4.6 · max","Codex 0.154.0 · max","Claude Code 2.1.270 · max","mini-swe-agent 2.4.6 · high","Grok Build 1.0.30 · high","Antigravity 1.2.7 · high","high · code and web","high reasoning, no tools","adaptive/max","Muse Glimmer 30B","GPT-5.6 Sol (Aug 2026)","GPT-5.6 Luna (Aug 2026)","GPT-5 Thinking","GPT-4o (March 2025)","Claude Fable 5.1 xhigh","GPT–6 Astra xhigh","Claude Opus 5 xhigh","Grok 4.6 xhigh","Gemini 3.8 Flash high","GPT–5.6 Sol xhigh","Claude Opus 5.5 (Max, Default Fallback)","Claude Fable 5.1 (Max, Default Fallback)","Grok 4.7 (Xhigh)","GLM-5.3 (Max)","MiMo-V2.6-Pro","Mistral Large 4 (High)","Qwen3.8 27B (Xhigh)","Nemotron 3 Ultra 550B A55B (Reasoning)","Qwen3.8 2.4T A95B","GPT-6 Sol (Daybreak Blue, max)","Standardized text evidence","Gemini 3 Pro Preview","GPT-OSS-20B","Qwen3-8B","Morpheus-32B","Fleming-R1-32B","HuatuoGPT-3-32B","Baichuan-M2-32B","MedGemma-27B-text-it","HuatuoGPT-3-8B","Morpheus-7B","Fleming-R1-7B","API · code-only","API · hybrid","Codex · hybrid","Claude Code · hybrid","Bash agent · medium","Bash agent · high","MiMo 2.5 Pro","Claude Code · thinking","Doubao Seed 2.1 Pro","Claude Code · xhigh","Claude Opus-5.5","Text control","GPT-6-sol","Gemini 3.8-Flash","reasoning","gpt-5.4","claude-opus-4-6-v1","claude-opus-4-5-20251101-v1.0","grok-4.3","gemma3 27b","claude-sonnet-4-5-20250929-v1.0","pixtral-large-2502-v1.0","gpt-5.4-nano","claude-haiku-4-5-20251001-v1.0","gemma3 4b","Gemini CLI","Terminus-2","DeepSeek Harness","Kimi Code","Muse Code","Drawing agent","Mistral Large 3 675B Instruct","Claude Code · local protocol adapter","GPT-5.6 luna","GPT-5.6 terra","GLM-5.3 flash","DeepSeek AI","Direct inference · max","Direct inference · xhigh","Direct inference · high","OpenCode","DeepSeek Flash","Codex CLI","Codex CLI, guidance","Codex CLI, comp. use","thinking off","thinking on","ReAct agent · high","ReAct agent · xhigh","ReAct agent · max","Clinician agent","GPT-6-astra","GPT-5.3-codex","GPT-4.1-mini","GPT-4.1-nano","Gemini-3.8-flash","Gemini-3.7-flash","Grok-4.7","Grok-4.3","DeepSeek-v4-pro","DeepSeek-v4-flash","mini-SWE-agent","OpenCode · high","Cursor · high","Seed 2.1 Turbo","Codex · medium","Claude Code · medium","Cursor · medium","Codex 0.154.0","Claude Code 2.1.139","DeepSeek Harness 0.1.5-rc.2","Kimi Code 0.42.0","Qwen Code 0.24.0","OSWorld-Science · default effort","Qwen-CUA","Claude Code · Max · n=2 · Fable 5 GPQA Main cells use Opus 4.8 Max fallback scores.","Claude Code · Max · n=2 · Five Fable 5.1 GPQA Main cells fell back to Opus 5.","Claude Code · Max · n=3","Claude Code · Max · n=2","Codex CLI · High · n=3","Codex CLI · xHigh · n=2","GPT 5.6 (Sol)","Codex CLI · Max · n=2","GPT 6 (Astra)","GPT 6.1 (Sol)","OpenCode · n=3","Cursor CLI · High · n=2","Claude Code · n=3","Claude Code · xHigh · n=3","Claude Code · High · n=2","Claude Code · n=2","Locus","Intology · Opus 5 · n=3 · Run by Intology and reviewed by PostTrainBench.","BrowserCode · default effort · 100/100 graded attempts","BrowserCode · high effort · 100/100 graded attempts","BrowserCode · default effort · 98/100 graded attempts","GPT-5.4 mini","Gemini 3.1 Pro preview","Gemini Lite","Codex CLI 0.147.0 · unguided","Codex CLI 0.147.0 · expert design guidance","Claude Code CLI 2.1.197 · unguided","Claude Code CLI 2.1.197 · expert design guidance","claude-opus-5.5","Open setting","claude-haiku","qwen2.5-coder-32b","qwen2.5-coder-7b","Medium reasoning","gpt-5-6-sol","gpt-5-6-terra","gpt-5-6-luna","opus-5-5","opus-5","opus-4-8","gemini-3-8-flash","gemini-3-7-flash","grok-4-6","glm-5-3-flash","glm-5-3","qwen-3-8-27b","qwen-3-8-flash","deepseek-v4-pro-0813","OpenAI API","Anthropic API","Local serving, greedy decoding, temperature 0","Muse-Glimmer-30B","Gemma-4-31B","UI-Mate-27B","Qwen3-VL-30B-A3B-Thinking","GUI-Owl-1.5-32B-Thinking","GUI-Owl-1.5-32B-Instruct","Qwen3-VL-30B-A3B-Instruct","OpenCUA-32B","UI-TARS-1.5-7B","cheap LLM","Muse Spark Contributor","frontier","Jev 1.13","decision","pplx-decider 27B","Mercury Decide","Solar Decide","Bielik 11B v3","local","Opus 5.0","max, Super-Think","reasoning on","24k thinking budget","max, 96k output","gpt-3.5-turbo-instruct","GPT-5.6-SOL","GPT-5 Mini","GPT 6.1 Sol","Codex, max reasoning","GPT 6 Astra","Antigravity CLI, high reasoning","Claude Code, max reasoning","Muse Code, max reasoning","MiMo Code, default reasoning","MiMo V2.6 Pro UltraSpeed","Inspect Robots, medium thinking","Inspect Robots, high thinking","Codex CLI, medium","Codex CLI, low","Claude Code, high","Gemini-3.8-Flash","OpenRouter, medium","DeepSeek-V4-Flash-Vision-Exp","OpenRouter, high","Goose, provider-default reasoning","Goose, highest reasoning","TBLite data selection","Muse Spark 1.3 (Extra High)","Mistral Large 4 Preview","Muse Spark 1.3 (Minimal)","Direct","OpenRouter","Doubao Seed 2.1 Turbo","Hunyuan HY3","Doubao Seed 2.0 Lite","InternVL3.5-30B-A3B","Gemma-3-12B-IT","GLM-4.1V-9B-Thinking","Nova 2 Lite","OpenRouter, high reasoning","Claude Haiku 5.5 (Xhigh)","Claude Opus 5.5 (Low, Default Fallback)","Elephant 9B","Baseten","Decision model","Celeris-1 Decision","Celeris","Clef","Cloudflare","Clef-Flash","Clef-Omni","Gemma 4 31B (Reasoning)","GLiDE v1","Fastino","GLiDE v1.2","d1","Liquid AI","d1-ultra","Drex Max","Nace.AI","Drex 1.5","Drex 2.0","GPT-6 Luna (Decisions)","GPT-6 Luna (Non-reasoning)","Microsoft-Decision-1","Decider Flash v1.2","Perplexity","Decider v1.1 27B","Decider v1.2","SubQ Snap 1.1","Subquadratic","TypeSafe","Jev 1.14","Medium reasoning, 30 briefs","Medium reasoning, 50 briefs","Zero-shot","NepBERTa","Fine-tuned, three seeds, fixed tok.","IndicBERTv2","Fine-tuned, three seeds","DeBERTa-base Nepali","RoBERTa-Nepali 125M","XLM-R base","MuRIL base","Shushant nepaliBERT","Fine-tuned, three seeds, default tok.","Rajan NepaliBERT","mBERT","Fine-tuned, three seeds, cased","grok-build; effort xhigh","GLM-5.2-FP8","Qwen3.5-397B-A17B-FP8","gemma-4-12b-it","gemma-4-E4B-it","gemma-4-E2B-it","llm-jp-4.1-33b-thinking","llm-jp-4.1-8b-thinking","Gemma 3 12B IT","MUSE Glimmer 30B","Qwen 3.8 Flash","Mistral Small 4","thinking","gemma-4-31b","Nemotron 3 Super 120B","nemotron-3-super","claude-haiku-5-5","muse-glimmer-30b","gemini-flash","inkling","dots.3 note preview","dots-3-note-preview","step-5-preview","nemotron-3-ultra","mimo-v2.6-flash","laguna-s-2.1","ChatGLM-5.1","LLaMA 4 Maverick","GPT-5.6-Luna","GPT-6-Luna","GPT-6.1-Sol","Qwen3.5-35B","Qwen3.5-122B","verbalized probabilities, copilot/claude-opus-5.5@effort-low","verbalized probabilities, copilot/gpt-6-astra@effort-low","verbalized probabilities, copilot/grok-4.7@effort-low","verbalized probabilities, copilot/gpt-6-sol@effort-none","verbalized probabilities, copilot/gpt-5.6-sol@effort-none","label probabilities at the answer token, google/gemma-4-31B-it@label-think0","Decision-2.0-Vega-27B","native probabilities, vllm-sr/Decision-2.0-Vega-27B@7aec49a","native probabilities, typesafe-ai/jev","label probabilities at the answer token, openai/gpt-oss-120b@label-low-think4096","verbalized probabilities, copilot/gpt-6-luna@effort-none","native probabilities, openrouter-decisions/microsoft/microsoft-decision-1","label probabilities at the answer token, Qwen/Qwen3.8-27B@label-think0","verbalized probabilities, copilot/gpt-5.6-luna@effort-none","Gemma-4-26B-A4B","label probabilities at the answer token, google/gemma-4-26B-A4B-it@label-think0","DiffusionGemma-26B-A4B","label probabilities at the answer token, google/diffusiongemma-26B-A4B-it@steps1-reads1-think0-prompt-1","label probabilities at the answer token, openai/gpt-oss-20b@label-low-think4096","OpenAI Decisions API","native probabilities, openai-decisions/gpt-6-luna","Decision-2.0-Lux-9B","native probabilities, vllm-sr/Decision-2.0-Lux-9B@78bf3c0","label probabilities at the answer token, Qwen/Qwen3.5-9B@label-think0","Decision-2.0-Nox-4B","native probabilities, vllm-sr/Decision-2.0-Nox-4B@25e8f67","label probabilities at the answer token, Qwen/Qwen3-32B@label-think0","Llama-3.3-70B","label probabilities at the answer token, RedHatAI/Llama-3.3-70B-Instruct@label-think0","Gemma-4-E4B","label probabilities at the answer token, google/gemma-4-E4B-it@label-think0","label probabilities at the answer token, Qwen/Qwen3-8B@label-think0","label probabilities at the answer token, meta-llama/Llama-3.1-8B-Instruct@label-think0","Decision-2.0-Kai-0.6B","native probabilities, vllm-sr/Decision-2.0-Kai-0.6B@cd49ea3","Laya (421M)","native probabilities, convaiinnovations/laya","CLM-v0.1-8B","native probabilities, Contrastive-LM/CLM-v0.1-8B","Qwen3-0.6B","label probabilities at the answer token, Qwen/Qwen3-0.6B@label-think0","Eight visible schemas, low working set","Qwen3.7 Flash","Qwen3.8 Max (0902)","HY4 Preview","Expert persona","Preliminary R2","Preliminary R2, thinking low","Final","Cursor Agent","Thin runner, Gemini API","Qwen 3.5 9B","Thin runner, Ollama, 32k","Qwen 3.5 4B","Granite 4.1 8B","GPT-6","Gemini-4-Argon","decision/d1-ultra","decision/decider-v1.2","decision/drex-max","decision/mercury-decide","decision/decider-v1.1-27b","decision/d1","decision/microsoft-decision-1","decision/jev-1.14","decision/drex-v1.5","decision/jev-1.13.0","decision/drex-v2.0","decision/gpt-6-luna","decision/decider-flash-v1.2","decision/clef","decision/solar-decide","decision/tev1-4b-experimental","decision/clef-omni","decision/classify-1","decision/clef-flash","decision/solar-decide-flash","GPT-5.6-terra-pro","GPT-5.6-sol-pro","Opus 5 Fast","openrouter/openrouter/auto","fireworks/firerouter","fireworks/firerouter-sol","ramp/switchyard","openrouter/typesafe/jev-router","sakana/fugu-max-v1.0","AgentVFS","Claude Code 2.1.296, max","Codex CLI 0.162.1, xhigh","Pi 1.2.0, high","ZCode 0.16.9, max","Kimi Code 0.31.0, high","Muse Code 1.4.2, max","DeepSeek Harness 0.2.0-rc.2, max","Muse Code 1.4.2, high","Qwen Code 0.25.0, high","Step Code 0.1.0, high","Frozen c1 · API","GPT-5.6 (terra)","Frozen c1 · Weights","GPT-5.6 (luna)","GLM-4.6V","Inkling-Small","Mistral Medium 3.5 128B","Molmo2-8B","Claude Code prompt · Kimi-K3 continuation","GPT-5.6-Terra","Hy4-Preview","DeepSeek-Flash","General-purpose model","GPT-5.6 Sol Pro","GPT-5.6 Terra Pro","GPT-5.6 Luna Pro","Qwen3.7 Plus","Qwen3.6 Plus","Qwen3-VL-235B-A22B Thinking","Qwen3.6 Flash","GPT-5 Nano","Gemini 3.1 Pro Preview (Custom Tools)","Step-3.7-Flash","Stepfun","Grok Build 0.1","MVP Qwen3-VL-32B","Specialist GUI grounding · native prompt and parser","GTA1 32B","UI-Venus 2-9B","MAI-UI 8B","MVP Qwen3-VL-8B","UI-Venus 1.5-8B","OpenCUA 32B","Qwen3-VL 32B-Instruct","ScaleCUA 32B","GTA1 7B","OpenCUA 7B","MVP UI-TARS-1.5-7B","UI-Venus Ground-7B","ScaleCUA 7B","GUI-G2 7B","UI-Ins 32B","Qwen3-VL 8B-Instruct","UI-Ins 7B","Jedi 7B-1080p","UI-TARS 1.5-7B","InfiGUI G1-7B","SE-GUI 7B","GUI-RC UI-TARS-1.5-7B","ScaleCUA 3B","Jedi 3B-1080p","Rex-Omni 3B","MAI-UI 2B","GUI-G2 3B","FocusUI 7B","UGround V1-7B","GUI-Actor 7B","GUI-Spotlight 7B","GUI-AIMA 3B","GUI-Actor 3B","InfiGUI G1-3B","FocusUI 3B","FocusUI Qwen3-VL-2B","UGround V1-2B","SE-GUI 3B","Pi · max effort","Qwen-3.8 Max","max effort","xhigh effort","Harbor 0.22.0 · Terminal-Bench loop · Plain","Kimi-k3","Claude-Sonnet-5","Qwen3.8-max","Qwen3.8-flash","Codex app xhigh","Claude Code app xhigh","vLLM 0.27.1 T=0, p=1","DeepSeek V4-Pro","DeepSeek V4-Flash","vLLM 0.28.1rc1.dev452+g144e79c81 T=0, p=1","SGLang 0.5.17 T=0, p=1","Devstral 2-123B","Agents-A1","vLLM 0.28.0 T=0, p=1","MiMo V2.5-Pro","Qwen3-Coder-Next","GPT-6 sol","Codex CLI · xhigh","GPT-6 luna","Codex CLI · Codex Security","Single-frame VQA","Gemini 3","Fable5.1","Gemini 2.5","Molmo2-4B","PaliGemma","PaliGemma 2","Gemma 3-4B","Qwen2.5-VL","DeepSeek-VL2-Tiny","OpenHelix","Single-frame VQA · extracted VLA backbone","π0.5","Standalone General-Purpose VLMs","Gemma 4 31B-IT","Standalone · high","Angelina Braille Reader","Optical Braille recognition","Llama-3.2-11B","Existing Braille Systems · WMT-Train","Qwen3.5-2B","Synthetic-Adapted Models","Chandra-OCR-2","DeepSeek Harness · high","Kimi Code · high","ZCode · high","DeepSeek v4.1 Flash","micro-cc","Qwen2.5-Coder-32B","Single-shot · model released 2024-11-12","Single-shot · model released 2025-03-06","Single-shot · model released 2025-04-29","Single-shot · model released 2025-05-22 · 40k thinking budget","Single-shot · model released 2025-06-17 · 8,192 thinking budget","Single-shot · model released 2025-08-05","Qwen3-VL-235B-A22B","Single-shot · model released 2025-09-23","Single-shot · model released 2025-09-29 · 40k thinking budget","Single-shot · model released 2025-10-15 · 40k thinking budget","Single-shot · model released 2025-11-24 · 40k thinking budget","Single-shot · model released 2025-12-17","Single-shot · model released approximately 2026-02-01","Single-shot · model released 2026-02-05","Single-shot · model released 2026-02-17 · medium effort","Single-shot · model released 2026-02-19","Single-shot · model released approximately 2026-02-24","Single-shot · model released approximately 2026-03-02","Single-shot · model released 2026-03-05","Single-shot · model released approximately 2026-04-01","Single-shot · model released 2026-04-16","Single-shot · model released 2026-04-22","Single-shot · model released 2026-04-23","Single-shot · model released 2026-05-19","Single-shot · model released 2026-05-28","Single-shot · model released 2026-06-30 · medium effort","Single-shot · model released 2026-07-09 · medium effort","Single-shot · model released 2026-07-09","Single-shot · model released 2026-07-16 · medium effort","Single-shot · model released 2026-07-21","Single-shot · model released 2026-07-24","Single-shot · model released 2026-08-13","Single-shot · model released 2026-08-14","Single-shot · model released 2026-09-01","Single-shot · model released 2026-09-02 · medium effort","Single-shot · model released 2026-09-03","Single-shot · model released 2026-09-22 · medium effort","GPT-6-luna","Single-shot · model released 2026-09-28 · medium effort","GPT-6.1-sol","Single-shot · model released 2026-09-29 · medium effort","Single-shot · model released 2026-10-07","Single-shot · model released 2026-10-13","Messages","Responses","Responses (Meta Model API)","Qwen3-VL 2B","Chat Completions (Ollama)","Claude Code 2.1.284 · Amazon Bedrock · high","Gemini CLI 0.61.0 · Gemini API · high","Codex 0.153.3 · Amazon Bedrock · high","Pi 0.99.2 · Baseten · high","Pi 0.99.2 · Fireworks · high","Grok CLI 1.0.46 · Amazon Bedrock · high","thinking default · effort medium · 20,000 output tokens","thinking default · effort default","thinking adaptive · effort medium · 64,000 output tokens","thinking adaptive · effort high · 20,000 output tokens","thinking on · effort high · 64,000 output tokens","thinking on · effort high · 60,000 output tokens","thinking adaptive · 100,000 output tokens","BrowserGym · AXTree · temperature 0 · 100 steps","Qwen3.8-Flash","Claude-Opus-5-5","Hosted · all pairs · false alarms 95%","Hosted · all pairs · false alarms 12%","medium effort · 36/36 designs finished","medium effort · 35/36 designs finished","Scale-normalized navigation","Gemini-3.7-Flash","Default (lightest API reasoning) · 230/240 tasks solved alone","Default (lightest API reasoning) · 238/240 tasks solved alone","Default (lightest API reasoning) · 236/240 tasks solved alone","Default (lightest API reasoning) · 227/240 tasks solved alone","Default (lightest API reasoning) · 186/240 tasks solved alone","Default (lightest API reasoning) · 231/240 tasks solved alone","Default (lightest API reasoning) · 220/240 tasks solved alone","Default (lightest API reasoning) · 212/240 tasks solved alone","Default (lightest API reasoning) · 207/240 tasks solved alone","Default (lightest API reasoning) · 189/240 tasks solved alone","Default (lightest API reasoning) · 180/240 tasks solved alone","Default (lightest API reasoning) · 201/240 tasks solved alone","Default (lightest API reasoning) · 210/240 tasks solved alone","Default (lightest API reasoning) · 216/240 tasks solved alone","Default (lightest API reasoning) · 202/240 tasks solved alone","Default (lightest API reasoning) · 166/240 tasks solved alone","Default (lightest API reasoning) · 129/240 tasks solved alone","Mini-SWE-Agent","MiMo v2.5 Pro","DS V4 Flash","GPT-5.2-Codex","Gemini 3.5","o4-mini","D=0 writer · high effort","D=0 writer","RAG In-force · 500 held-out items","GLM 5.3 FlashX Legal","Gemma 4 26B-A4B","Nemotron 3.5 Lightning 30B-A3B","OpenThai 2.0 27B","Nemotron 3 Nano 30B-A3B","SEA-LION v4.8 30B-A3B","OpenThai 2.0 Legal 30B-A3B","Nemotron 3 Super 120B-A12B","SEA-LION v4.8 120B-A12B","Typhoon-S 8B","Pathumma 8B Think","THaLLE 0.2 8B","OpenThaiGPT 8B","APC · I2 · high effort · 60 min","DeepSeek 4.1 Flash","Pi","Kimi hosted","DSH","Claude Code CLI","Trae Work","WorkBuddy","DuMate","DuMate CLI","One attempt per task","Agent: Claude Code; Coding Agents; pipeline claude_code_extract_opus_4_8_evidence; reasoning high; 370 documents; 30-minute per-document limit","https://github.com/run-llama/ExtractBench/blob/4064262a67eb19ee6d4213754a2efc108733e46a/leaderboard.csv","Agent: Claude Code; Coding Agents; pipeline claude_code_extract_opus_5_5_evidence; reasoning medium; 370 documents; 30-minute per-document limit","Agent: Claude Code; Coding Agents; pipeline claude_code_extract_sonnet_5_5_evidence; reasoning medium; 370 documents; 30-minute per-document limit","Agent: Codex; Coding Agents; pipeline codex_code_extract_gpt_5_5_low_evidence; reasoning low; 370 documents; 30-minute per-document limit","Agent: Codex; Coding Agents; pipeline codex_code_extract_gpt_5_6_luna_medium_evidence; reasoning medium; 370 documents; 30-minute per-document limit","Agent: Codex; Coding Agents; pipeline codex_code_extract_gpt_5_6_sol_low_evidence; reasoning low; 370 documents; 30-minute per-document limit","Agent: Codex; Coding Agents; pipeline codex_code_extract_gpt_5_6_terra_low_evidence; reasoning low; 370 documents; 30-minute per-document limit","Agent: Codex; Coding Agents; pipeline codex_code_extract_gpt_6_luna_medium_evidence; reasoning medium; 370 documents; 30-minute per-document limit","Agent: Codex; Coding Agents; pipeline codex_code_extract_gpt_6_sol_low_evidence; reasoning low; 370 documents; 30-minute per-document limit","Agent: Codex; Coding Agents; pipeline codex_code_extract_gpt_6_1_sol_low_evidence; reasoning low; 370 documents; 30-minute per-document limit","Agent: Codex; Coding Agents; pipeline codex_code_extract_gpt_6_1_terra_low_evidence; reasoning low; 370 documents; 30-minute per-document limit","Commercial VLM; pipeline deepseek_v4_1_flash_extract_oneshot_structured_output_file; reasoning off; 370 documents; 30-minute per-document limit","Gemma4 12B","OSS; pipeline gemma4_12b_vllm_extract_oneshot_structured_output_file; reasoning default; 370 documents; 30-minute per-document limit","Gemma4 26B","OSS; pipeline gemma4_26b_vllm_extract_oneshot_structured_output_file; reasoning default; 370 documents; 30-minute per-document limit","Gemma4 E2B","OSS; pipeline gemma4_e2b_vllm_extract_oneshot_structured_output_file; reasoning default; 370 documents; 30-minute per-document limit","Gemma4 E4B","OSS; pipeline gemma4_e4b_vllm_extract_oneshot_structured_output_file; reasoning default; 370 documents; 30-minute per-document limit","GLM-4.6V Flash","OSS; pipeline glm_4_6v_flash_vllm_extract_oneshot_structured_output_file; reasoning default; 370 documents; 30-minute per-document limit","Commercial VLM; pipeline glm_5_3_flash_extract_oneshot_structured_output_file; reasoning default; 370 documents; 30-minute per-document limit","Google Gemini 3.5 Flash","Commercial VLM; pipeline gemini_3_5_flash_extract_oneshot_structured_output_file; reasoning low; 370 documents; 30-minute per-document limit","Google Gemini 3.8 Flash","Commercial VLM; pipeline gemini_3_8_flash_extract_oneshot_structured_output_file; reasoning low; 370 documents; 30-minute per-document limit","InternVL3.5 14B","OSS; pipeline internvl3_5_14b_vllm_extract_oneshot_structured_output_file; reasoning default; 370 documents; 30-minute per-document limit","InternVL3.5 8B","OSS; pipeline internvl3_5_8b_vllm_extract_oneshot_structured_output_file; reasoning default; 370 documents; 30-minute per-document limit","Commercial VLM; pipeline kimi_k3_extract_oneshot_structured_output_file; reasoning default; 370 documents; 30-minute per-document limit","Kimi-VL A3B","OSS; pipeline kimi_vl_a3b_vllm_extract_oneshot_structured_output_file; reasoning default; 370 documents; 30-minute per-document limit","Agent: KYNEX; Coding Agents; pipeline kynex_gpt_5_4; 370 documents; 30-minute per-document limit","Lift Datalab 9B","OSS; pipeline lift_extract; 370 documents; 30-minute per-document limit","MiniCPM-V 4.5","OSS; pipeline minicpm_v_4_5_vllm_extract_oneshot_structured_output_file; reasoning default; 370 documents; 30-minute per-document limit","NuExtract 2.0 8B","OSS; pipeline nuextract_2_8b_vllm_extract_oneshot_structured_output_file; reasoning default; 370 documents; 30-minute per-document limit","NuExtract3","OSS; pipeline nuextract3_extract; reasoning off; 370 documents; 30-minute per-document limit","OpenAI GPT-5.4 Nano","Commercial VLM; pipeline openai_gpt_5_4_nano_extract_oneshot_structured_output_file; reasoning off; 370 documents; 30-minute per-document limit","OpenAI GPT-6 Astra","Commercial VLM; pipeline openai_gpt_6_astra_reasoning_low_extract_oneshot_structured_output_file; reasoning low; 370 documents; 30-minute per-document limit","Qwen3.5 0.8B","OSS; pipeline qwen3_5_0_8b_vllm_extract_oneshot_structured_output_file; reasoning default; 370 documents; 30-minute per-document limit","Qwen3.5 2B","OSS; pipeline qwen3_5_2b_vllm_extract_oneshot_structured_output_file; reasoning default; 370 documents; 30-minute per-document limit","Qwen3.5 35B","OSS; pipeline qwen3_5_35b_a3b_fp8_vllm_extract_oneshot_structured_output_file; reasoning default; 370 documents; 30-minute per-document limit","Qwen3.5 4B","OSS; pipeline qwen3_5_4b_vllm_extract_oneshot_structured_output_file; reasoning default; 370 documents; 30-minute per-document limit","Qwen3.5 9B","OSS; pipeline qwen3_5_9b_vllm_extract_oneshot_structured_output_file; reasoning default; 370 documents; 30-minute per-document limit","Qwen3.6 35B","OSS; pipeline qwen3_6_35b_a3b_fp8_vllm_extract_oneshot_structured_output_file; reasoning default; 370 documents; 30-minute per-document limit","OSS; pipeline qwen3_8_27b_fp8_vllm_extract_oneshot_structured_output_file; reasoning default; 370 documents; 30-minute per-document limit","Qwen3.8 Flash Next","OSS; pipeline qwen3_8_flash_next_fp8_vllm_extract_oneshot_structured_output_file; reasoning default; 370 documents; 30-minute per-document limit","Agent: Claude Code; default reasoning effort; 4-hour solve budget; web search disabled; Claude Sonnet 4.6 validity judge","https://arxiv.org/html/2606.24530v2#S4.T4","Agent: Gemini CLI; default reasoning effort; 4-hour solve budget; web search disabled; Claude Sonnet 4.6 validity judge","Agent: Codex CLI; default reasoning effort; 4-hour solve budget; web search disabled; Claude Sonnet 4.6 validity judge","Agent: mini-swe-agent; reasoning effort: xhigh; 4 runs; 335/452 scored attempts across all 113 tasks","Agent: mini-swe-agent; reasoning effort: high; 4 runs; 330/447 scored attempts across all 113 tasks","Agent: mini-swe-agent; reasoning effort: max; 4 runs; 327/444 scored attempts across all 113 tasks","Agent: mini-swe-agent; reasoning effort: max; 4 runs; 327/450 scored attempts across all 113 tasks","Agent: mini-swe-agent; reasoning effort: xhigh; 4 runs; 316/452 scored attempts across all 113 tasks","Agent: mini-swe-agent; reasoning effort: max; 4 runs; 311/451 scored attempts across all 113 tasks","Agent: mini-swe-agent; reasoning effort: max; 4 runs; 309/451 scored attempts across all 113 tasks","grok-4.6","Agent: mini-swe-agent; reasoning effort: medium; 4 runs; 305/452 scored attempts across all 113 tasks","Agent: mini-swe-agent; reasoning effort: max; 4 runs; 301/448 scored attempts across all 113 tasks","Agent: mini-swe-agent; reasoning effort: xhigh; 4 runs; 303/452 scored attempts across all 113 tasks","gemini-3.7-flash","Agent: mini-swe-agent; reasoning effort: medium; 4 runs; 296/452 scored attempts across all 113 tasks","glm-5.3-flash","Agent: mini-swe-agent; reasoning effort: max; 4 runs; 284/448 scored attempts across all 113 tasks","Agent: mini-swe-agent; reasoning effort: max; 4 runs; 284/452 scored attempts across all 113 tasks","Agent: mini-swe-agent; reasoning effort: xhigh; 4 runs; 258/449 scored attempts across all 113 tasks","muse-spark-1.2","Agent: mini-swe-agent; reasoning effort: xhigh; 4 runs; 248/452 scored attempts across all 113 tasks","Agent: mini-swe-agent; reasoning effort: max; 4 runs; 238/442 scored attempts across all 113 tasks","Agent: mini-swe-agent; reasoning effort: max; 4 runs; 241/452 scored attempts across all 113 tasks","Agent: mini-swe-agent; reasoning effort: high; 4 runs; 211/452 scored attempts across all 113 tasks","Agent: mini-swe-agent; reasoning effort: max; 4 runs; 197/450 scored attempts across all 113 tasks","Agent: mini-swe-agent; reasoning effort: high; 4 runs; 163/452 scored attempts across all 113 tasks","Agent: CRM-Agent-Bench graph-routed Burr ReAct; formal round 1; one trial per task; 464/555 strict successes","https://arxiv.org/html/2610.32004v1#A2.T6","Agent: CRM-Agent-Bench graph-routed Burr ReAct; formal round 1; one trial per task; 459/555 strict successes","Agent: CRM-Agent-Bench graph-routed Burr ReAct; formal round 1; one trial per task; 455/555 strict successes","Agent: CRM-Agent-Bench graph-routed Burr ReAct; formal round 1; one trial per task; 453/555 strict successes","Agent: CRM-Agent-Bench graph-routed Burr ReAct; formal round 1; one trial per task; 451/555 strict successes","Agent: CRM-Agent-Bench graph-routed Burr ReAct; formal round 1; one trial per task; 448/555 strict successes","Inkling Small NVFP4","Agent: CRM-Agent-Bench graph-routed Burr ReAct; formal round 1; one trial per task; 442/555 strict successes","Agent: CRM-Agent-Bench graph-routed Burr ReAct; formal round 1; one trial per task; 439/555 strict successes","GPT-5.4 Nano","Agent: CRM-Agent-Bench graph-routed Burr ReAct; formal round 1; one trial per task; 437/555 strict successes","Agent: CRM-Agent-Bench graph-routed Burr ReAct; formal round 1; one trial per task; 422/555 strict successes","Nemotron Super 3 120B","Agent: CRM-Agent-Bench graph-routed Burr ReAct; formal round 1; one trial per task; 364/555 strict successes","Nemotron 3.5 Lightning 30B","Agent: CRM-Agent-Bench graph-routed Burr ReAct; formal round 1; one trial per task; 357/555 strict successes","GPT-4.1 Nano","Agent: Terminus-2 (LHTB Harbor); one 3-hour trial per task; binary-only feedback; 37/77 solved (R ≥ 0.95); one provider refusal contributes zero","https://github.com/zli12321/LHTB#all-77-tasks","Agent: Terminus-2 (LHTB Harbor); one 3-hour trial per task; binary-only feedback; 32/77 solved (R ≥ 0.95); one provider refusal contributes zero","Agent: Terminus-2 (LHTB Harbor); one 3-hour trial per task; binary-only feedback; 17/77 solved (R ≥ 0.95)","Hunyuan hy4","Agent: Terminus-2 (LHTB Harbor); one 3-hour trial per task; binary-only feedback; 15/77 solved (R ≥ 0.95)","Agent: Terminus-2 (LHTB Harbor); one 3-hour trial per task; binary-only feedback; 15/77 solved (R ≥ 0.95); 5 unscored cells contribute zero","Agent: Terminus-2 (LHTB Harbor); one 3-hour trial per task; binary-only feedback; 12/77 solved (R ≥ 0.95)","Agent: Terminus-2 (LHTB Harbor); one 3-hour trial per task; binary-only feedback; 10/77 solved (R ≥ 0.95)","Agent: Terminus-2 (LHTB Harbor); one 3-hour trial per task; binary-only feedback; 9/77 solved (R ≥ 0.95)","Agent: Terminus-2 (LHTB Harbor); one 3-hour trial per task; binary-only feedback; 8/77 solved (R ≥ 0.95)","Agent: Terminus-2 (LHTB Harbor); one 3-hour trial per task; binary-only feedback; 7/77 solved (R ≥ 0.95)","Agent: Terminus-2 (LHTB Harbor); one 3-hour trial per task; binary-only feedback; 5/77 solved (R ≥ 0.95)","Agent: Terminus-2 (LHTB Harbor); one 3-hour trial per task; binary-only feedback; 5/77 solved (R ≥ 0.95); 5 unscored cells contribute zero","Agent: Terminus-2 (LHTB Harbor); one 3-hour trial per task; binary-only feedback; 2/77 solved (R ≥ 0.95)","Agent: ThinkingBox MCP; Azure endpoint, nominal medium reasoning; 6,743/10,140 successes","https://arxiv.org/html/2608.19741v5#A4.T15","Agent: ThinkingBox MCP; Azure endpoint, nominal medium reasoning; 6,628/10,140 successes","Agent: ThinkingBox MCP; Azure endpoint, nominal medium reasoning; 6,278/10,140 successes","Agent: ThinkingBox MCP; Azure endpoint, nominal medium reasoning; 6,002/10,140 successes","Agent: ThinkingBox MCP; Azure endpoint, nominal medium reasoning; 5,913/10,140 successes","Agent: ThinkingBox MCP; Azure endpoint, nominal medium reasoning; 5,817/10,140 successes","Agent: ThinkingBox MCP; local vLLM, qwen3_coder tool parser; 5,242/10,140 successes","Agent: ThinkingBox MCP; Azure endpoint, nominal medium reasoning; 4,693/10,140 successes","Agent: ThinkingBox MCP; Azure endpoint, nominal medium reasoning; 4,387/10,140 successes","Agent: ThinkingBox MCP; Azure endpoint, nominal medium reasoning; 3,819/10,140 successes","Agent: ThinkingBox MCP; Azure endpoint, nominal medium reasoning; 3,365/10,140 successes","Agent: ThinkingBox MCP; local vLLM, qwen3_coder tool parser; 3,340/10,140 successes","Agent: ThinkingBox MCP; Azure endpoint, nominal medium reasoning; 3,254/10,140 successes","o3-pro","Agent: ThinkingBox MCP; Azure Responses API, medium reasoning (temperature unsupported); 1,958/10,140 successes","Agent: ThinkingBox MCP; Azure endpoint, nominal medium reasoning; 1,458/10,140 successes","Agent: ThinkingBox MCP; local vLLM, qwen3_coder tool parser; 573/10,140 successes","Mistral-Large-3","Agent: ThinkingBox MCP; Azure endpoint, reasoning off (unsupported parameters omitted); 473/10,140 successes","Agent: ThinkingBox MCP; Azure endpoint, nominal medium reasoning; 19/10,140 successes","closed book; request reasoning: unspecified","GPT-5.5 [none]","closed book; request reasoning: none","Claude Opus 4.7 [high]","closed book; request reasoning: high","DeepSeek V4 Pro [high]","Claude Opus 4.7 [default]","DeepSeek V4 Pro [default]","Gemma 4 31B IT","closed book; no reasoning mode","Qwen-SEA-LION 4.5 27B Instruct","AI Singapore","Qwen3.5 27B","Apertus v1.5 70B","Swiss AI Initiative","Apertus 70B Instruct","EuroLLM 22B Instruct","UTTER","Olmo 3 32B Think","Allen Institute for AI","Qwen2.5 14B Instruct AWQ","Apertus v1.5 8B","Apertus 8B Instruct","Tiny Aya Global","Cohere Labs","Tiny Aya Earth","Tiny Aya Water","Tiny Aya Fire","Gemini-3-Pro","Proprietary; 1 frame/s","https://arxiv.org/html/2608.25529v1#S4.T1","Gemini-3-Flash","Doubao-Seed-2.0-Pro-260215","Proprietary; 50 sampled frames (API limit)","Qwen2.5-Omni-3B","Instruct; Original video with audio; model-recommended inference settings","Qwen2.5-Omni-7B","Qwen3-Omni-30B-A3B-Instruct","Gemma-4-E4B-it","Instruct; Model-recommended settings; authors use frames with timestamped subtitles for vision-only models and original audio/video for omni models","Gemma-4-26B-A4B-it","Gemma-4-31B-it","InternVL3.5-8B-Instruct","InternVL3.5-14B-Instruct","InternVL3.5-30B-A3B-Instruct","InternVL3.5-38B-Instruct","InternVL3.5-241B-A28B-Instruct","Qwen3-VL-4B-Instruct","Qwen3-VL-8B-Instruct","Qwen3.5-4B-Instruct","Qwen3.5-9B-Instruct","Qwen3.5-27B-Instruct","Qwen3.5-35B-A3B-Instruct","Qwen3.5-122B-A10B-Instruct","Qwen3.5-397B-A17B-Instruct","Qwen3-Omni-30B-A3B-Think","Thinking; Original video with audio; model-recommended inference settings","Qwen3-VL-30B-A3B-Think","Thinking; Model-recommended settings; authors use frames with timestamped subtitles for vision-only models and original audio/video for omni models","Qwen3-VL-235B-A22B-Think","InternVL3.5-8B-Think","InternVL3.5-14B-Think","InternVL3.5-30B-A3B-Think","InternVL3.5-38B-Think","InternVL3.5-241B-A28B-Think","Qwen3.5-9B-Think","Qwen3.5-27B-Think","Qwen3.5-35B-A3B-Think","Qwen3.5-122B-A10B-Think","Qwen3.5-397B-A17B-Think","Claude Code; high reasoning effort","Codex; medium reasoning effort","KnowLine-31B-Gen1","Paper default","https://arxiv.org/html/2611.05827v1#S5.T1","Basal-Omni-1.0-Ultra","Quyet-1.0-Large","DOPEST-1.0-31B","PPLX-Decider-v1.2","PPLX-Decider-v1.1-27B","Torchcast-4B-v1.2","H2O-Lightning-31B","D3","GLiDE-1.2-Open","StartLux-v1.1-27B-Instant","René-31B","PPLX-Decider-Flash-v1.2","D3-Turbo","Torchcast-4B-v1.3","deck31b","Wald-26B-A4B","Jeff-1.1-Medium","Vega-27B","Tacit-9B","D3-Ultra","MeTask-Rain-31B","MeTask-Rain-12B","Liquid AI d1-3B","Torchcast-12B","Tacit-4B","Mapika-Decider-12B","Mammoth-1","Torchcast-4B-v1.1","Joey-1","Tacit-2B","H2O-Lightning-4B","Nox-4B","Lux-9B","Tacit-1.7B","Laya-421M","Kai-0.6B","CLM-8B","Claude-Haiku-4.5","https://arxiv.org/html/2610.27611v1#S2.T2","Doubao-Seed-1.6","Qwen3-Max-Preview","Hunyuan-2.0-Thinking","Spark-X","ERNIE-X1-Turbo-32k","Hunyuan-T1-Latest","ERNIE-X1.1-Preview","Harness: Antigravity","https://arxiv.org/html/2611.05512v1#S7.T1","Harness: Muse Code","Harness: Claude Code","Harness: Grok Build","GPT 6.1 Terra","Harness: Codex CLI","Harness: OpenHands","Harness: Terminus-2","Harness: Gemini CLI","Qwen3.7 Max","GPT 6 Luna","GPT-5.4-Mini","https://arxiv.org/html/2610.18836v1#S3.T1","Gemini-3.1-Pro-Preview","Qwen3-VL-30B-A3B","Step3-VL-10B","Gemma-4-12B-it","GLM-4.6V-Flash","NEO-9B-SFT","SenseNova-U1","NEO-ov","Qwen3-VL-2B-Instruct","InternVL3.5-2B","NEO-2B-SFT","https://arxiv.org/html/2610.18834v1#S3.T1","Claude-Opus-4.8-thinking","Gemma4-31B-it","Gemma4-12B-it","NEO-ov-9B","SenseNova-U1-8B-MoT","MiMo-VL-7B-RL","Harness: Codex","https://arxiv.org/html/2610.19005v1#S5.T2","Harness: OpenCode","Harness: Grok CLI","Harness: Qwen Code","Harness: DeepSeek Harness","Harness: Kimi CLI","Gemma 4 26B-A4B R","https://arxiv.org/html/2610.28340v1#A1.T2","Gemma 4 31B V","Command A 111B","OLMo 3 32B Think","Qwen3 4B","Qwen3 8B","Qwen3 14B","Qwen3 30B-A3B","Qwen3 235B-A22B","Qwen3-Next 80B-A3B","Qwen3.5 35B-A3B","Qwen3.5 122B-A10B G","GPT-4.1 mini","GPT-4.1 nano","GPT-5 nano","GPT-OSS 120B","Gemini 2.5 Flash-Lite","https://arxiv.org/html/2610.33966v1#S3.T2","https://arxiv.org/html/2610.19698v1#S4.T2","https://arxiv.org/html/2610.35488v1#S4.T1","https://arxiv.org/html/2610.28679v1#S3.T2","Doubao Seed 2.0 Code","highest available effort","https://arxiv.org/html/2611.06910v1#A6.T8","effort: max","effort: xhigh","https://arxiv.org/html/2611.07689v1#S5.T5","GPT-5-chat","Doubao-Seed-1.8","DeepSeek-V3.1","DeepSeek-V4","https://arxiv.org/html/2610.26059v1#S3.T1","Qwen3-8B (base)","GLM-4.1V-9B","Qwen3-VL-235B","Qwen3-VL-4B","Qwen3-VL-30B","MiMo-VL-7B","Qwen3-14B","Refusal Hint; binary scores; CR=2","https://arxiv.org/html/2610.25924v1#S2.T1","Gemini-3 Flash","Gemini-3.5 Flash","Gemini-3.6 Flash","DeepSeek V4 Flash (Preview)","DeepSeek V4 Pro (Preview)","Qwen-3.5 Plus","Qwen-3.7 Plus","Qwen3.5-9B Base","https://arxiv.org/html/2611.02536v1#S5.T2","GLM-4-9B-0414-Chat","InternLM3-8B-Instruct","Ministral-3-8B-Instruct","AceCoder-7B","OpenCodeReasoning-Nemotron-7B","Seed-Coder-8B-Reasoning","FORGE","OpenCode; Uninterrupted","https://arxiv.org/html/2611.02017v1#S4.T1","OpenCode; Switch","KimiCode; Uninterrupted","KimiCode; Switch","GPT-5.6 Terra (medium)","Codex; Uninterrupted","Codex; Switch","GPT-5.6 Sol (medium)","Claude Opus 5 (medium)","Claude Code; Uninterrupted","Claude Code; Switch","Gemini CLI; Uninterrupted","Gemini CLI; Switch","ReAct","https://arxiv.org/html/2611.01145v1#S4.T3","TSM","GPT-5.4-nano","Gemini-3.5-flash-lite","https://arxiv.org/html/2610.36384v1#S5.T2","Qwen3.5-0.8B","Qwen3-VL-2B","Qwen3-VL-8B","Qwen3-VL-32B","InternVL3.5-1B","InternVL3.5-14B","Kimi-VL-A3B","MiniCPM-o-4.5-9B","LLaVA-OneVision-1.5-8B","VideoLLaMA3-7B","EgoGPT-7B","EgoVLM-3B","EgoThinker-7B","Batched","https://arxiv.org/html/2610.32429v1#S4.T4","Async","GLM5.3 Flash","Gemini 3 Flash-H","BAGEL","https://arxiv.org/html/2610.28622v1#S4.T1","FLUX.2","Qwen-Image-Edit-2511","Emu3.5","DreamOmni2","SenseNova-U1.5","Nano Banana","Nano Banana Pro","GPT-Image-1.5","GPT-Image-2","BAGEL-SFT","https://arxiv.org/html/2610.26394v1#S4.T2","QwenCode","Avg. / Acc","https://arxiv.org/html/2610.25096v1#S5.T4","Gemma-4-31B-IT","GPT-oss-120B","Gemini-3.1-Flash","Qwen3.6-Flash","https://arxiv.org/html/2610.20731v1#S4.T1","Qwen-Image","GPT-Image-2.5","Seedream 5.0 Pro","Seedream 4.5","Seedance 2.0","Gemini Omni","Wan 3.0","Positive","https://arxiv.org/html/2610.20690v1#S4.T1","https://arxiv.org/html/2610.19564v1#S6.T4","LLaVA-OneVision-7B","Mistral Small 3.2 24B","Qwen2.5-VL-32B","Phi-4-Multimodal","LoRA","0-shot","1-shot","2-shot","3-shot","https://arxiv.org/html/2610.12823v1#S4.T1","Gemini 3.1 Flash","GPT-6 Astra Pro","API layer","https://arxiv.org/pdf/2610.31993v1","October route: Claude Code subscription","Mistral Small","harness: Codex","https://arxiv.org/pdf/2610.34238v1","https://arxiv.org/pdf/2610.38405v1","Llama-3.2-90B-Vision-Instruct","https://arxiv.org/html/2610.27626v1#S4.T2","GLM-4.5V-108B","InternVL3-78B-Instruct","Qwen3.5-Plus","Counterfactual sandbox","https://arxiv.org/html/2610.39234v1#S4.T2","Claude-Haiku-5.5","MiniMax-3.1-Flash","Qwen-3.8-Flash-Next","Direct execution","https://arxiv.org/html/2610.28675v1#S6.T5","Sonnet 4","No spec","https://arxiv.org/html/2610.28595v1#S5.T2","https://arxiv.org/html/2610.41168v1#S6.T1","GLM-5 Turbo","judge: gpt-5.6-sol","https://arxiv.org/html/2610.25954v1#A8.T12","harness: codex","https://arxiv.org/html/2611.08237v1#S4.T2","harness: claude-code","harness: opencode","effort: xhigh; harness: codex","ME","https://arxiv.org/html/2610.35138v1#S5.T2","qwen3.8-max-0902","qwen3.8-27b","direct","https://arxiv.org/html/2611.04894v1#S3.T3","Qwen3.8-Flash-Next","agent: ReAct","agent: DeerFlow","https://arxiv.org/html/2610.34735v1#S4.T1","Agentic; harness: Codex","https://arxiv.org/html/2610.37341v1#S4.T1","Agentic; harness: Pi Agent","Agentic; harness: Claude Code","DeepSeek-v4.1-flash","Non-agentic; harness: non-agentic","Trial 1; harness: Claude Code","https://arxiv.org/html/2610.33956v1#S4.T1","Trial 1; harness: Codex","Trial 1; harness: Hermes-Agent","Doubao-Seed-Evolving","Trial 1; harness: OpenClaw","https://arxiv.org/html/2610.34167v1#S6.T4","harness: Codex CLI","https://arxiv.org/html/2610.35864v1#S4.T2","harness: Claude Code","harness: Antigravity CLI","harness: DSH","harness: Kimi Code","harness: ZCode","All verified applicable","https://arxiv.org/html/2610.23685v1#S4.T2","DeepSeek-Coder-33B (FP16)","Qwen2.5-Coder-14B","CodeLlama-34B (FP16)","Qwen2.5-Coder-7B","CodeLlama-7B (FP16)","GLM-4-9B (FP16)","GLM-4-9B (4-bit)","MCQ","https://arxiv.org/html/2610.20083v1#S4.T2","Doubao-Seed 2.0 Pro","Mimo-V2-Omni","Qwen3-VL-2B †","Qwen3-VL-4B †","Qwen3-VL-8B †","InternVL3.5-8B †","SL-27B AutoThink","https://arxiv.org/html/2610.28522v1#S4.T4","SL-27B Instant","Decision 3.0 d3 ‡","Perplexity Decider v1.1","Decision 2.0 Vega","Kev 27B","JEV-27B","Jebadiah 27B","Decision 3.0 d3-flash ‡","Decision 3.0 d3-mini ‡","Jev 1.13 (hosted)","Blink v0.3","Xor 1.0","GEV-26B-Decide","Hopper 12B","Decision 2.0 Lux","Nimble v3","Decision 2.0 Nox","JEV-9B","FlyMy Decision 4B v1.2 †","Eval Engine Decision-4B †","Clef flash","CoCo-Decision-4B-Ko †","NeoHorse-Jev-4B †","Decision 3.0 d3-nano ‡","Hopper (G) 1.2","Intern-Decision-4B †","Decision 3.0 d3-lite ‡","Intern-Decision-2B †","d1-3B","Decision 2.0 Sol","this-that 1.2 †","Strands decider 2B v21","Decision 2.0 Eos","Bosun 3.1 1.7B","d1-omni-600M","Decision 2.0 Kai †","Decision 3.0 d3-edge ‡","IntentDelta; environment: TravelPlanner","https://arxiv.org/html/2610.33806v1#S5.T3","minimax-m2.5","nova-pro","qwen3-235b","qwen3-32b","IntentDelta; environment: WebShop","input: frames; thinking_level low","https://arxiv.org/html/2610.31986v1#A1.T4","input: frames; reasoning_effort low","input: frames; low resolution, low reasoning","input: frames; thinking_level minimal","input: frames; thinking_budget 128","input: frames; reasoning_strength low","input: frames; reasoning_effort none","input: frames; reasoning_effort minimal, detail low","Gemma 4 31B-it","input: frames; no thinking, temperature 0","input: frames; temperature 0","input: video; SGLang","input: frames; reasoning_effort minimal","input: frames; thinking_budget 0","MiMo-V2.6-Pro-MOPD","input: video; vLLM","Qwen3-VL-8B Instruct","MiMo-V2.6-Pro-RL","MiMo-V2.6-Flash-MOPD","MiMo-V2.6-Flash-RL","MiMo-V2.6-Distill-Qwen-9B","https://arxiv.org/html/2610.26826v1#S5.T1","https://arxiv.org/html/2611.05620v1#S4.T2","Agent: ReAct; source label: ReAct","https://arxiv.org/html/2610.17665v1#A11.T11","Agent: Magentic; source label: Magentic","SpatioLM-InternVL3.5","GPT 5.6-sol","effort: high","https://arxiv.org/html/2610.31494v1#S4.T1","effort: none","GPT 5.6-luna","Gemini 3.1-pro","Gemini 3.7-flash","GLM 4.6v","GLM 4.5v","GLM 4.1V 9B-Thinking","GLM 4.1V 9B-Base","Qwen3.5 397B-A17B","Qwen3.5 122B-a10b","Qwen3.5 35B-a3b","https://arxiv.org/html/2611.07166v1#S3.T5","MiMo 2.6 Pro","MOSS Preview","https://arxiv.org/html/2611.02835v1#S4.T4","AURA","JoyAI","DuplexOmni (S1)","MiniCPM-o 4.5","Venus-Omni","Gander","MOSS-VL","Qwen3.5 Plus","Qwen3.5 Flash","Gemini 3.8 Live","Qwen 3 Omni 30B-A3B","effort: thinking","https://arxiv.org/html/2611.03839v1#S5.T1","effort: non-thinking","Gemma 4 12B","Nemotron 3 Nano Omni 30B-A3B","Nemotron Audex 30B-A3B","Inkling small {}_{\\text{small}}","effort: medium","Qwen 3.5 Omni Plus","GPT 5.6 Terra","protocol: Tool-only","https://arxiv.org/html/2611.01446v1#S5.T2","protocol: PTC","protocol: I-PTC","Qwen 3.5-397B-A17B","GPT-Live","https://arxiv.org/html/2611.02430v1#S5.T4","GPT-Realtime","Gemini Live","Moshi","PersonaPlex","Freeze-Omni","Cascade","CLIP B/32","https://arxiv.org/html/2610.41118v1#S5.T3","CLIP B/16 LAION-2B","CLIP H/14 LAION-2B","SigLIP B/16","SigLIP SO400M/14","SigLIP-2 B/16","SigLIP-2 L/16-384","SigLIP-2 SO400M/14","NegCLIP B/32","CLoVe B/32","DreamLIP B/16 (30M)","DFN H/14","EVA-02 L/14","MetaCLIP-2 H/14","Qwen3.8","Easy","https://arxiv.org/html/2611.01181v1#A4.T1","Medium","Hard","Qwen3-VL-32B-Instruct","Counting, complete","https://arxiv.org/html/2610.38926v1#S4.T1","InternVL3.5-38B-HF","LLaVA-OneVision-2-8B-Instruct","MiniCPM-V-4.5","Phi-4-multimodal-instruct","Kimi-VL-A3B-Instruct","ConvNeXt","Clean, 5-fold mean","https://arxiv.org/html/2610.36850v1#S4.T5","EfficientNet","RegNet","Swin","ShuffleNet","MnasNet","VGG16-BN","GoogLeNet","SqueezeNet","w/o Think","https://arxiv.org/html/2610.39438v1#S3.T3","w/ Think","Hidden Think","Claude-5-Sonnet","Adaptive","XHigh","Sep. 2026","https://arxiv.org/html/2610.39377v1#S5.T3","Feb. 2026","GLM 5.3 FlashX","Jul. 2026","Sep. 2025","Pointwise","https://arxiv.org/html/2610.34662v1#S5.T2","GPT-Realtime-2.1","JoyAI-VL-Interaction","StreamVibe-VL","SFT","RL","DeepSeek-V4 Pro","https://arxiv.org/html/2610.34290v1#S5.T5","MiMo-V2.5 Pro","Kimi K2.7-Code","Gemma-3-12B","No rule, direct answer","https://arxiv.org/html/2610.33448v1#S3.T2","Gemma-3-1B","https://arxiv.org/html/2610.32441v1#S2.T3","All levels, criterion 1","https://arxiv.org/html/2610.32221v1#S3.T3","GLM-5V Turbo","Intern-S1","Gemma-3-4B","4B","https://arxiv.org/html/2610.32485v1#S4.T2","7B","8B","Ministral-8B","Molmo-2-8B","Llama-3.2-11B-V","11B","12B","Ministral-14B","14B","Gemma-3-27B","27B","30B","32B","72B","Llama-3.2-90B-V","90B","109B","Mistral-Large","123B","235B","9B","122B","400B","—","Claude-Sonnet-4.5","Claude-Opus-4.5","Claude-Sonnet-4.5-T","Claude-Opus-4.5-T","Gemini-2.5-Flash-Lite","Grok-4.1","https://arxiv.org/html/2610.31000v1#S6.T4","Solver","https://arxiv.org/html/2610.30116v1#S3.F2","Overall","https://arxiv.org/html/2610.28444v1#S4.T2","Qwen3-Omni-Instruct-30B-A3B","Qwen3-Omni-Thinking-30B-A3B","Nemotron3-Nano-Omni-30B-A3B","MiMo-V2.5-310B-A15B","MiMo-V2.6-Flash-310B-A15B","MiMo-V2.6-Pro-1T-A42B","Doubao-Seed-2.0-Lite","Doubao-Seed-2.1-Lite","Qwen3.5-Omni-Flash","Qwen3.5-Omni-Plus","Qwen3.8-Omni-Flash","Gemini2.5-Flash","Gemini2.5-Flash-Lite","Gemini2.5-Pro","Gemini3.1-Flash-Lite","Gemini3.1-Flash-Lite-Preview","Gemini3.1-Pro-Preview","Gemini3.5-Flash","Gemini3.5-Flash-Lite","Gemini3.6-Flash","Gemini3.7-Flash","Gemini3.8-Flash","Gemini4-Argon","k-mer + RF","https://arxiv.org/html/2610.28310v1#S5.T1","k-mer + XGBoost","GROWREC","k-NN (Sequence)","CNN Encoder","Transformer","Flat","Gating","RNABERT","Linear","https://arxiv.org/html/2610.27793v1#S4.T2","Simple CNN","AR U-Net","non-AR U-Net","AR Swin-ViT","non-AR Swin-ViT","DRCT","Raw","https://arxiv.org/html/2610.28087v1#S4.T1","DDA","AIDE","B-Free","Effort","PGC","OmniAID","CO-SPY","PE","336","518","FakeVLM","BusterX++","Veritas++","Qwen3.6-35B","CROMA","AI4Farms","https://arxiv.org/html/2610.28313v1#S4.T1","DOFA","Prithvi","Scale-MAE","TerraMind-L","GeoMixture","Naive","Sum","MoE","Average of structured and flat","https://arxiv.org/html/2610.28819v1#S4.T2","Qwen3.8-A95B","gemini 3.1 pro","Spanish parse","https://arxiv.org/html/2610.29455v1#S4.T1","gemini 3.7 flash","gemini 3.8 flash","qwen3.5-omni-plus","qwen3.8-omni-flash","muse spark 1.2","doubao-seed-2.0-lite","qwen3-omni-30b-thinking","moss-transcribe-diarize","Azure, Top1","https://arxiv.org/html/2610.29185v1#S4.T2","https://arxiv.org/html/2610.30113v1#S3.T2","none","T2V","https://arxiv.org/html/2610.28416v1#S4.T2","https://arxiv.org/html/2610.26582v1#S3.T3","t0-gamma","TimesFM-3","TS-ICL","Timestral","TabPFN-3.5","Chronos-2","TiRex-2","XGBoost","Linear Regression","Ridge Regression","https://arxiv.org/html/2610.26679v1#S5.T2","T=0","T=1","Command A Reasoning","GLM-4 9B","Instruct 2507","MiniCheck","Bespoke-7B","https://arxiv.org/html/2610.27370v1#S4.T2","AlignScore","roberta-large","PsiloQA","ModernBERT-base","LettuceDetect","lettuce-large","RAGTruth","ft-llama2-13B","FAVA","ft-llama2-7B","Agent: Prompt; reasoning enabled; no prompt optimization; source label: Prompt","Coding Models","https://arxiv.org/html/2610.26850v1#S4.T1","Prompt Engineering on GPT-Image-2.5","PromptEnhancer","Closed-source T2I Models","Nano Banana 2","FLUX.2 pro","GPT-Image-1","Seedream-4.5","FLUX.2 dev","Open-source T2I Models (1024 × \\times 1024, default settings, see Table 7 )","FLUX.2 klein","FLUX.1 dev","FLUX.1 schnell","Qwen-Image 20B","SD3.5 large","SD3.5 medium","Sana","4.8B","0.6B","1.6B","English","https://arxiv.org/html/2610.26155v1#S4.T3","Qwen2.5-7B-Inst","Llama-3.1-8B-Inst","SEA-LION-v4-27B","Sailor-2-20B","Base, 1-hop","https://arxiv.org/html/2610.25087v1#S2.T2","Claude-Sonnet-4","Gemma3-27B","Gemma3-12B","Gemma3-4B","https://arxiv.org/html/2610.22822v1#S5.T2","Gemma 3-12B","Gemma 3-27B","GPT-Realtime Mini","Burden counting, overall","https://arxiv.org/html/2610.25247v1#S5.T1","GRIT","ViGoRL","DeepEyes","Pixel Reasoner","Thyme","PyVision-RL","Mini-o3","Qwen3-VL-4B-T","No-Ref-4B","VScaffold-4B","GPT-6-Sol","No memory","https://arxiv.org/html/2610.25127v1#S3.T1","GLM-5.3-flash","Llama3.3-70B","Naive VGG16","Frontal to gallery","https://arxiv.org/html/2610.24832v1#S5.T2","RST","DPIT-VGG16","Swin-S + Linear","DPIT-ResNet50","ViT-B + Linear","DINOv2-B + Linear","https://arxiv.org/html/2610.25309v1#S3.T2","Dolphins","nuScenes MCQ","https://arxiv.org/html/2610.22795v1#S4.T2","GLM-4.5V","MobAgent","https://arxiv.org/html/2610.23307v1#S4.T4","GPT-5.4-Nano","No claim","https://arxiv.org/html/2610.21159v1#S5.T1","T=64, prediction","https://arxiv.org/html/2610.22312v1#S4.T3","multimodal","https://arxiv.org/html/2610.20012v1#S5.T2","text only","text + reader","Black-box","https://arxiv.org/html/2610.20785v1#S4.T2","DS-V4-Pro","Black-box, thinking","Black-box, no thinking","DS-V4-Flash","White-box","Ministral-3B","No profile","https://arxiv.org/html/2610.19427v1#S4.T3","Qwen3.7-Flash","Pass@1","https://arxiv.org/html/2610.20027v1#S3.T3","DeepSeek-v3.2","Qwen2.5-Coder-3B","Qwen2.5-Coder-0.5B","Focused D1","https://arxiv.org/html/2610.20272v1#S4.T2","https://arxiv.org/html/2610.19704v1#S4.T2","TigerLLM 12B","Gemma 2 9B","Patch-level","https://arxiv.org/html/2610.18004v1#S3.T2","Action, all","https://arxiv.org/html/2610.17728v1#S5.T3","VITA-1.5","Baichuan 1.5","AgiBot","https://arxiv.org/html/2610.17856v1#S5.T1","Qwen3-Omni-30B-A3B-Inst.","Gemini-3-flash","TimeOmni-v (Ours)","https://arxiv.org/html/2610.17827v1#S5.T4","Qwen3-32B †","Qwen3-4B SFT","Qwen3-4B GRPO","Qwen3-4B SFT+GRPO","Qwen3-8B SFT","Qwen3-8B GRPO","Qwen3-8B SFT+GRPO","HealthAdmin, native Pass@3","https://arxiv.org/html/2610.15714v1#S5.T3","GitHub Repo, semantic","https://arxiv.org/html/2610.19021v1#S4.T2","Gemini-2.5","scanner control","https://arxiv.org/html/2610.22244v1#S4.T2","original wording","TS-Guard (7.6B, specialised)","specialised","open","small","Nemotron 3.5 30B-A3B","Nemotron 3 Nano 4B","Nemotron Nano 9B","https://arxiv.org/html/2610.22360v1#S4.T1","GLM-4.6V-106B","OmniVerifier-7B","https://arxiv.org/html/2611.08281v1#S4.T2","SP","Task execution","https://arxiv.org/html/2611.04879v1#S4.T2","Recorded tasks","https://arxiv.org/html/2611.02472v1#S5.T1","DeepSeek-VL2-Small","Claude-Sonnet-4.6","Qwen3-VL-Flash","ScaleCUA-7B","MAGAR","DrivoR","PlanContext","https://arxiv.org/html/2611.03067v1#S4.T2","GoalFlow","TransFuser","SparseDriveV2","DiffusionDrive","StateMLP","CLOVER","DiffusionDriveV2","OneVL","ChainFlow-VLA","ReCogDrive-IL","ReCogDrive-RL","NoRD","AutoVLA","DriveVLA-W0","DP-VLA","Curious-VLA","WAM-Flow","SUV","WoTE","SimWAM","Drive-JEPA","DynVLA Stage 2","Without experience","https://arxiv.org/html/2611.05450v1#S3.T2","Single-A","https://arxiv.org/html/2611.08177v1#A7.T9","AR","Cont-Mono","Disc-Mono","Cont-Mono-Opt","NEO","GLM-4-9B-0414","https://arxiv.org/html/2611.02545v1#S5.T1","Granite-3.3-8B","Hunyuan-7B","Ministral-3-8B-Instruct-2512","Attribute First","OpenScholar","PaperQA2","VeriCite","Default SFT","w/o A","w/o B","FindLink","MAFT+","Large","https://arxiv.org/pdf/2611.05801","FC-CLIP","ConvNeXt-Large","EOV-Seg","OpenWorldSAM","Standard","ODISE","Caption","U3HS","M2A","RbA","PoDS","PSF-Linear","Euclid to Roman","https://arxiv.org/html/2611.04139v1#S4.T2","Sérsic-FM","CodeBrain","diffm4ri","mmgan","ummi","mmsyn","m2dn","RePaint","Palette","asyn-diff","Gemma E4B, 1×L40","https://arxiv.org/pdf/2611.06841","GPT 6 Sol","C2D","ERMA full set, τ=0.5","https://arxiv.org/pdf/2611.06007","I3D","SlowFast","X3D","InternVL2.5-4B","Cosmos-Reason2-8B","UniDriveVLA","DriveLMM-o1","500-way Top-1, 16²","https://arxiv.org/pdf/2611.02540","Seed-2.1 Pro","GeoCLIP","Original background, street 1 km","https://arxiv.org/pdf/2611.06085","Img2Loc","G3","DeepSeek-VL-7B","resemblyzer","Task 1","https://arxiv.org/pdf/2611.05638","ECAPA-TDNN","TitaNet","WeSpeaker","RawNet3","x-vector","WavLM","wav2vec 2.0","XLS-R","HuBERT","Whisper","Machine learning systems, FREE","https://arxiv.org/pdf/2611.03424","Machine learning systems, TOPIC","Machine learning systems, LANDSCAPE","Machine learning systems, FRONTIER","Code generation, FREE","Code generation, TOPIC","Code generation, LANDSCAPE","Code generation, FRONTIER","World models, FREE","World models, TOPIC","World models, LANDSCAPE","World models, FRONTIER","Automated research, FREE","Automated research, TOPIC","Automated research, LANDSCAPE","Automated research, FRONTIER","K2 Horizon 375B A23B","Zero-shot dense captioning","https://arxiv.org/pdf/2611.03646","MiniCPM-V-4.5-8B","LLaVA-OV2-8B","video-SALMONN-2+-3B","CapRL-Video-4B","OmniCaptioner-IF-3B","OmniCaptioner-IF-7B","RynnBrain-4B","Flora","EgoAction (LLaVA-OV2-8B)","EgoAction (Qwen3-VL-4B)","Seedance-2.5","Single-shot","https://arxiv.org/pdf/2611.05422","Seedance-2.0","MiniMax-H3","HappyHorse-1.1","Veo3.1","LTX-2.3","LTX-2.5","Wan2.7","Kling3.0-Omni","Kling3.0","daVinci-MagiHuman","Ovi","Zero-shot, overall","https://arxiv.org/pdf/2611.01934","InteractiveOmni-4B","WasItAI V3 (video model)","Hidden split, original clips","https://www.wasitaigenerated.com/research/ai-image-video-detector-benchmark-2026","WasItAI V2","DDA (NeurIPS 2025)","OmniAID (CLIP)","haywoodsloan/ai-image-detector-deploy","Smogy/SMOGY-Ai-images-detector","All questions","https://arxiv.org/pdf/2610.40530","Gemma 4-31B","Gemma 4-26B-A4B","Gemma 4-12B","Gemma 4-E4B","Gemma 4-E2B","InternVL3.5-241B-A28B","DeepSeek-V4-Flash-Vision","MiniMax-M3-428B-A23B","MiniCPM-V 4.5-8B","MiniCPM-V 4.6-1.3B","Cosmos3-Super-64B","Cosmos3-Nano-16B","Cosmos3-Edge-4B","Ovis2.6-80B-A3B","Ovis2.6-30B-A3B","ERNIE 4.5 VL-28B-A3B","Phi-4-Reasoning-Vision-15B","LLaVA-OneVision 2-8B","Granite Vision 4.1-4B","Mistral Small 4-119B","Gemma-3","FinanceHarness","https://arxiv.org/pdf/2611.01621","Open Deep Research","GPT-Researcher","Original account","https://arxiv.org/pdf/2611.00515","GLM 4.7 Flash","DeepSeek V3.1","GLM 4.7","MiniMax M2.1","Qwen3 235B","GPT Sol 6.1","https://github.com/yeyuxin777/baoyan-agent-benchmark","GPT Astra 6","Volcengine Managed Agent","Qwen Code","API","https://github.com/speak2jerome/llmdyno","UD-Q3_K_XL","UD-IQ3_XXS","UD-Q2_K_XL","Qwen3.6 35B-A3B","MXFP4, experts on CPU","Q8_0","QAT Q4_0","MXFP4","Seed2.0 Pro","256 frames, thinking on","https://arxiv.org/pdf/2611.00769","256 frames, thinking off","GLM-5.3-Flash-FP8","512 frames, thinking off","512 frames, thinking on","Conan","LongVideo-R1","varying frames, thinking on","post-tuned, varying frames, thinking on","18 of 36 runs finished","https://github.com/Funovate-Technology/ai-design-benchmark/blob/main/benchmarks/2026-10-llm-designer/summary.csv","36 of 36 runs finished","35 of 36 runs finished","laguna-s-2.1-ollama","ollama, solo 18 tasks","https://github.com/Adithya-Rajendran/dgx-spark-benchmarks/blob/main/RESULTS.md","nemotron-3.5-bolt","vllm, solo 18 tasks","ornith-1.5-35b","qwen3.6-35b","qwen3.8-flash-next-iq4","llama.cpp, solo 18 tasks","qwen3.8-flash-next-lcpp","Multiple choice","https://arxiv.org/pdf/2610.38678","Gemini-3.8 Flash","MiMo-V2.5","Qwen-3.8 Flash","MOSS-VL-Realtime","MMDuet2","VideoLLM-online","TimeChat-Online","DeepSeek-V4 Flash","Fixed, Historical identification, original","https://arxiv.org/pdf/2610.37919","K2-Horizon-375B","Matrix-Game 2.0","Table 5","https://arxiv.org/pdf/2610.39721","WorldPlay (HY)","WorldPlay (Wan)","LingBot-World","Matrix-Game 3.0","SANA-WM","DreamX-World","AlayaWorld","LingBot-World v2","ABot-World","Low","https://arxiv.org/pdf/2610.38957","minimal","text-guided","https://arxiv.org/pdf/2610.40453","OmniPaint","text-guided, PhysW fine-tune","Insert-Anything","JoyAI-Image-Edit-Plus","PhysInsert","Recognition","https://arxiv.org/pdf/2610.39602","BERT-base","fine-tuned","https://arxiv.org/pdf/2610.40518","LegalBERT-US","InCaseLawBERT","InLegalBERT","Mistral 7B","zero-shot","Qwen2.5 7B","All tasks","https://arxiv.org/pdf/2610.36589","HY4 preview","Native, closed-book","https://arxiv.org/pdf/2610.37739","NIM rerank-qa-mistral-4b","https://arxiv.org/pdf/2610.38090","bge-reranker-v2-m3","Qwen3-Reranker-0.6B","Qwen3-Reranker-4B","Qwen3-Reranker-8B","hosted, five passages","zero-shot, five passages","pplx-decider-v1.1-27b","NeoHorse-Jev-4B","0.6B student","seed 42, one passage","4B student","seed 42, released as GEO Citation Ranker 4B","seed 1337, one passage","seed 7, one passage","8B student","GPT-4o judge","https://arxiv.org/pdf/2610.37599","GPT Image 2","LongCat-Image","Lumina-DiMOO-8B","InternVL-U-4B","VIBE-Image-Edit","OmniGen2","BAGEL-7B-MoT","LongCat-Image-Edit","Lance-3B","DeepGen 1.0","FLUX.2 [klein] 4B","Janus-Pro-1B","Mobile-O-0.5B","SANA-Sprint-0.6B","SANA-600M","DreamLite","Ovis-U1-3B","TJ Codely","https://arxiv.org/pdf/2610.35980","Anomaly OpenCode","OpenAI Codex","GPT 5.6 Luna","Anthropic Claude Code","SpaceXAI Grok Build","Google Antigravity","Alibaba Qwen Code","Moonshot Kimi Code","Simular, CUA","Agent S, CUA","ByteDance UI-TARS, CUA","UI-TARS, CUA","Closed weights","https://arxiv.org/pdf/2610.35215","Open weights","CLIP","Reference CRI","https://raw.githubusercontent.com/alicemkyn/prism-benchmark/main/prism_bench/reference/cri_components.csv","PLIP","CONCH","VIRCHOW2","UNI","GigaPath","H-Optimus-0","MIDNIGHT","reasoning effort high","https://arxiv.org/pdf/2610.35987","reasoning effort medium","reasoning effort low","reasoning effort xhigh","C0 none","https://arxiv.org/pdf/2610.36513","C1 AMA","C2 WHO","C3 mitigation","DeepSeek-V2-Lite","Ministral-3-14B","MiMo-7B-RL","OLMo-2-32B","A2World","Table 3 A2World","https://arxiv.org/pdf/2610.33565","BWM","Table 3 BWM","Cosmos-Predict-2.5","Table 3 Cosmos","Ctrl-World","Table 3 Ctrl","DW05","Table 3 DW05","DreamDojo","Table 3 DreamDojo","GE-Sim 2.0","Table 3 GE-Sim","OpenAI Decisions","https://arxiv.org/pdf/2610.33389","Azure Decision-1","microsoft-decision-1","OpenJev","SemIf","Kev-0.8B","v1.0","Kev-4B","Kev-9B","Kev-27B","v2","audio-visual","https://arxiv.org/pdf/2610.32268","Seed2.0-Lite","Seed2.0-Pro","visual","ChronusOmni","Qwen3-Omni","Qwen2.5-Omni","HumanOmniV2","Ola-7B","MiniCPM-O-2.6","TimeLens-8B","TimeLens-7B","Time-R1-7B","TRACE","VideoChat-R1","TimeSuite","Video-R1-7B","standalone policy","https://arxiv.org/pdf/2610.34649","Xiaomi Robotics-1","InternVLA-A1.5","MolmoAct2","LingBot-VLA2","HY-VLA","Galaxea G0.5","X-VLA","τ0 (low-level)","OpenWAM-Alpha","LingBot-VA","Mini-SWE-agent","https://arxiv.org/pdf/2610.35188","deepseek-r1:14b","2026-10-22; Q4_K_M; 16 GB tier","https://localagentbench.com","deepseek-r1:8b","2026-10-21; Q4_K_M; 16 GB tier","2026-10-28; Q4_K_M","gemma4:12b","gemma4:26b","2026-10-22; Q4_K_M; 24 GB tier","gemma4:e4b","2026-10-21; Q4_K_M; 8 GB tier","2026-10-26; Q4_K_M","gpt-oss:20b","2026-10-22; MXFP4; 16 GB tier","hermes3:8b","2026-10-27; Q4_K_M","llama3.1:8b","llama3.2:3b","2026-10-20; Q4_K_M; 8 GB tier","mistral-nemo:12b","2026-10-21; Q4_K_M; 16 GB tier; RTX 3090","2026-10-21; Q4_K_M; 16 GB tier; Apple M2","mistral-small3.2:24b","2026-10-23; Q4_K_M; 24 GB tier","mistral-small:24b","2026-10-21; Q4_K_M; 24 GB tier","mistral:7b","qwen2.5-coder:14b","qwen2.5-coder:7b","qwen2.5:14b","qwen2.5:7b","qwen3-coder:30b","qwen3-vl:8b","qwen3.5:27b","qwen3.5:9b","qwen3.6:27b","qwen3.8:27b","qwen3:14b","qwen3:30b-a3b","qwen3:4b","qwen3:8b","CHF","FP32","https://arxiv.org/pdf/2610.36340","INT8","DM-Count","GCHF","CSRNet","P2PNet","APGCC","STEERER-VGG19","FIDTM","ZIP-Tiny","STEERER-HRNet","MobileCount","ZIP-Nano","MPCount","ZIP-Pico","ZIP-Small","FFNet-ConvNeXt-T","FFNet-MobileNetV3","MCNN","CCST","Transcrowd-GAP","Transcrowd-Token","CCTrans-Full","CCTrans-Weak","Gramformer","PET","CLIP-EBC-RN50","active","https://arxiv.org/pdf/2610.34149","Gemini-3.1-flash-lite","Gemini-2.5-flash","Gemini-2.5-flash-lite","GPT-5.2-2025-12-11","GPT-5-mini-2025-08-07","GPT-5-nano-2025-08-07","LLaVA-OneVision-2-8B","Gemma-3-4B-it","passive","256 output tokens","https://arxiv.org/pdf/2610.33777","Gemma4-12B","16k output tokens","T3A workflow","https://arxiv.org/pdf/2610.30573","M3A workflow","end-to-end","Step-GUI-4B","GUI-Owl-1.5-8B-Instruct","UI-Venus-1.5-8B","DeepSeek-VL2","https://arxiv.org/pdf/2610.33097","Gemini-3.6","GLM-4.1V","Qwen3-VL-I","Qwen3-VL-T","Project Genie","SunRisesAndSets, 5 rollouts","https://arxiv.org/html/2610.33894","HY-WorldPlay","Waypoint-1.5","video only","https://arxiv.org/html/2610.33377","LLaVA-OneVision 7B","Qwen3-VL 8B","InternVL2.5 8B","MiniCPM 8B","Molmo2 8B","InternVL2.5 26B","Aria 25.3B","Qwen3-VL 32B","Qwen2.5-VL 32B","InternVL3.5 38B","LLaVA-OneVision 72B","Qwen2.5-Omni 7B","MiniCPM-o 2.6 8B","MP-holistic","default detector","https://arxiv.org/pdf/2610.32517","MP-hands","WiLoR","RTMW-wb","RTMP-hand","Sapiens2-1B","https://arxiv.org/pdf/2610.32285","OpenClaw","https://arxiv.org/html/2610.29537","https://arxiv.org/html/2610.30043","Claude Sonnet-5","RoboBrain2.0-32B","Hy-Embodied-VLM-1.0","https://arxiv.org/html/2610.30537","Hermes","Qwen3 Coder 480B","correct condition","https://arxiv.org/html/2610.27777","implicit","explicit","k=10","https://arxiv.org/html/2610.27911","Kimi-VL-16B-A3B-Instruct","k=50","https://arxiv.org/html/2610.27782","Seed-2.1-Lite","Cosmos3-Nano-Reasoner","Cosmos-Reason2-32B","Cosmos-Reason2-2B","MiMo-Embodied-7B","RynnBrain-1.1-9B","RoboBrain2.5-8B","Orca-4B","VideoScore2-8B","VideoPhy2-7B","VideoCon-7B","PhyJudge-9B","RoboJudge-9B","https://arxiv.org/html/2610.31205","GLM-4.6V-Flash-9B","Gemma-4-E2B","RetGrounder-8B","Rifky_IndoBERT_QA","https://arxiv.org/html/2610.31256","IndoLEM_IndoBERT","Cahya_IndoBERT","XLM-R-large","mDeBERTa-v3-base","mT5 (Generative)","5-shot","Sahabat-AI-Llama3-8B","Merak-7B-v4","BioMistral-7B","MedGemma-4B-IT","https://arxiv.org/html/2610.21304","Mistral-Medium","Qwen3-Next-80B-A3B-Inst","Spark X1","Mistral-Large-Latest","Mistral-Small","Qwen3-Next-80B-A3B-Think","ERNIE-5.0-Thinking-Preview","Doubao-Seed-1.6-251015","Qwen3-1.7B","Mistral-Tiny","OpenVLA","https://arxiv.org/html/2610.24587","OpenVLA-OFT","OpenVLA-OFT w","OpenVLA-OFT m","OpenVLA-OFT+","UniVLA","π0","π0-FAST","GR00T N1.7","Cosmos Policy","FastWAM","Anchor-Align","RIPT-VLA","https://arxiv.org/html/2610.20776","Gemma-3-27B-IT","Qwen3-4B-Instruct","GPT-3.5-Turbo","GPT-5-Nano","Gemini-3-Flash-Preview","Gemini-3.1-Flash-Lite-Preview","https://arxiv.org/html/2610.18569","Detailed","Stepwise","Two-shot","Text-only","https://arxiv.org/html/2610.19158","OpenAI Agents SDK","Google ADK","Claude Code; single run","OpenClaw; single run","OpenAI Agents SDK; single run","Google ADK; single run","https://arxiv.org/html/2610.19165","GPT-5.4 nano","Nova Premier","Nova Pro","Nova Micro","Qwen 3.5 397B-A17B (FP8)","Qwen 3.5 122B-A10B","Qwen 3 Next 80B-A3B","Qwen 3.5 35B-A3B","Mistral Large 3 675B-A41B","Magistral Small 24B","Ministral 3 14B","Ministral 3 8B","Ministral 3 3B","https://arxiv.org/html/2610.21767","https://arxiv.org/html/2610.19589","Qwen3.8-2.4T-A95B","Hunyuan-4-Preview","MiMo-V2.5-Pro","MiMo-V2.6-Flash","Nemotron-3.5-Lightning","Nemotron-3-Ultra-550B","Granite-4.2-8B","https://arxiv.org/html/2610.13623","https://arxiv.org/html/2610.23751","Seedream 5.0 Lite","VINCIE","FLUX.1-kontext-dev","FLUX.2-klein-9B","Lance","Qwen-Image-Edit-Causal","OpenClaw 2026.5.19; benign cases","https://arxiv.org/html/2608.09476v1","Qwen3.7-max","Qwen3.7-plus","Deepseek-v4-Pro","Deepseek-v4-Flash","Hunyuan-3.0","https://arxiv.org/html/2608.29397v1","https://arxiv.org/html/2608.29128v1","claude-haiku-4-5","https://arxiv.org/html/2608.09638v1","Qwen3-4B-Think","gpt-oss-120B","Phi-3.5-mini","OLMo-2-7B","Reasoning: Think","https://arxiv.org/html/2608.09282v1","Reasoning: No-think","Doubao-Seed-2.0-Pro","Gemini-3.1 Pro","https://arxiv.org/html/2608.12262v1","Gemini-3.0 Pro","Gemini-3.0 Flash","Claude-4.6 Opus","Seed-2.0 Pro","Screenshot + A11y Tree; 15 steps","https://arxiv.org/html/2608.10366v1","O4-mini","OpenAI CUA","Claude-4.5-Sonnet","Claude-4.6-Sonnet","https://arxiv.org/html/2608.30248v1","Qwen3.5-397B/A17B","Devstral-2-123B","Codestral-22B","DeepSeek-Coder-33B","DeepSeek-Coder-V2-Lite","https://arxiv.org/html/2608.29387v2","Qwen3-Coder","Claude-4.5-Haiku","https://arxiv.org/html/2608.28155v1","ODA-Fin-RL-8B","Fin-o1-14B","DianJin-R1-32B","Hawkish-8B","Fin-R1-7B","https://arxiv.org/html/2608.21832v2","HF","Ollama Q4","OS-Atlas-Base-7B","MiniCPM-V","Ollama","PaliGemma 2-3B-mix-448","Qwen2-VL-7B-Instruct","PaliGemma 2-10B-mix-448","Llama-3.2-11B-Vision","https://arxiv.org/html/2608.25071v1","mistral-large","mistral-small","https://arxiv.org/html/2608.20220v1","Qwen-3.5-397B","https://arxiv.org/html/2608.09106v2","w/ Thinking","Llama3.1-8B","w/ CoT","LegalOne-8B","DISC-LawLLM-7B","https://arxiv.org/html/2608.27442v1","SALMONN","https://arxiv.org/html/2608.22236v2","Qwen2-Audio","Step-Audio-2-mini","Audio Flamingo 3","Audio Flamingo Next","MiMo-Audio-Instruct","Voxtral-Mini","Step-Audio-R1","Audio-Reasoner","Qwen3-Omni-Thinking","AudioMCQ-Weak-To-Strong","Omni-R1","Qwen3-Omni-Instruct","GPT-Audio-1.5","https://arxiv.org/html/2608.15931v1","Codex Agent","Full context; low reasoning where supported; 20,000 output tokens; provider-default sampling","https://arxiv.org/html/2609.01198v1#S6.T3","Qwen 3.5 122B A10B","Qwen 3.6 35B A3B","ReAct agent; Python sandbox; up to 30 turns; medium effort where available","https://arxiv.org/html/2609.01552v1#S3.T2","Temperature 1; 6,000 output tokens; final answer; three-judge protocol","https://arxiv.org/html/2609.00918v1#S3.T2","Claude-Sonnet-4-6","Closed-ended multiple choice; greedy decoding; identical prompts and option order","https://arxiv.org/html/2609.01691v1#S4.T2","GPT-5.2-reasoning","CogVLM-19B","LLaVA-1.6-7B","Llama-3.2-Vision-11B","Definitions supplied; one completion; temperature 0 where supported","https://arxiv.org/html/2609.01548v1#A7.T7","Nemotron-3-Super-120B-A12B","Joint two-agent emotion prompt; full release","https://arxiv.org/html/2609.03394v1#S4.T4","Zero-shot; full benchmark","https://arxiv.org/html/2609.01982v1#S5.T1","Direct; eight paired subscenarios; PostgreSQL","https://arxiv.org/html/2609.06527v1#S5.T6","Interactive; eight paired subscenarios; PostgreSQL","Direct; eight paired subscenarios; Oracle","Interactive; eight paired subscenarios; Oracle","Agent: Codex; medium reasoning; paper budget","https://arxiv.org/html/2609.04075v1#S4.T2","Agent: OpenHands; medium reasoning; paper budget","Agent: Claude Code; medium reasoning; paper budget","GPT-5.6 Sol + Claude Opus 4.8","Agent: Atlantis [ 52 ]; medium reasoning; paper budget","Agent: Buttercup [ 54 ]; medium reasoning; paper budget","Agent: RoboDuck [ 53 ]; medium reasoning; paper budget","Agent: single-model M3A; paper action budget","https://arxiv.org/html/2609.07712v1#S3.T2","Agent: official GUI prompt and tool schema; paper action budget","Doubao-Seed-2.0","Qwen-3.6-Flash","MobileAgent-v3.5","Agent: MobileAgent-v3.5; paper action budget","Agent: UI-TARS-1.5-7B; paper action budget","AgentCPM-GUI","Agent: AgentCPM-GUI; paper action budget","V-Droid","Agent: V-Droid; paper action budget","Agent: Cursor; kimi-k3-high; three runs per task","https://arxiv.org/html/2609.09113v1#S4.T1","Agent: Cursor; claude-opus-5-thinking-high; three runs per task","Agent: Cursor; claude-sonnet-5-thinking-high; three runs per task","Agent: Cursor; cursor-grok-4.6-high; three runs per task","Agent: Cursor; claude-opus-4-8-thinking-high; three runs per task","Agent: Cursor; gemini-3.8-flash-high; three runs per task","Agent: Codex; gpt-5.6-sol; three runs per task","Agent: Cursor; gpt-5.5-high; three runs per task","Agent: Cursor; glm-5.2-high; three runs per task","Agent: Codex; gpt-5.6-luna; three runs per task","Main caption prompt; full question set","https://arxiv.org/html/2609.09973v1#S4.T3","Gemini-2.5-pro","AuroraCap-7B","LLaVA-Video-7B","Tarsier2-7b","VQA; thinking enabled","https://arxiv.org/html/2609.12606v1#S6.T7","Gemini-3.5-Flash + NanoBanana2","Pipeline; thinking enabled","VQA; default thinking setting","GPT-5.4 + GPT-image2","Pipeline; default thinking setting","Qwen3-VL-8B + Qwen-image-edit","CodePlot-CoT-32B","Code; default thinking setting","Bagel-Zebra-7B","Interleave; default thinking setting","SenseNova-U1-8B","MathCanvas-7B","GeoWeave-8B","DeepSeek v4 Flash","De-leaked no-scaffold prompt; full154-task held-out set","https://arxiv.org/html/2609.19180v1#S7.T7","DeepSeek v4 Pro","Agent: Codex CLI; maximum reasoning; full47 tasks","https://arxiv.org/html/2609.21293v1#S4.T3","Agent: Claude Code; maximum reasoning; full47 tasks","Hunyuan-3","Reasoning effort XH; provider-default sampling; paper output cap","https://arxiv.org/html/2609.19883v1#S4.T1","Reasoning effort H; provider-default sampling; paper output cap","Reasoning effort M; provider-default sampling; paper output cap","Reasoning effort default; provider-default sampling; paper output cap","Nemotron 3 Super","Published baseline protocol","https://arxiv.org/html/2610.02770v1#S6.T2","Qwen-3.5-35B","Claude-Haiku-4-5","Claude-Sonnet-4-5","Claude-Opus-4-5","https://arxiv.org/html/2610.04506v1#S4.T1","SpatialVLM","SpatialMLLM","RoboBrain","https://arxiv.org/html/2610.05532v1#S5.T1","Medium; full conversation; three runs","https://arxiv.org/html/2610.08102v1#S3.T2","Thinking; full conversation; three runs","GLM 4.6V","High; full conversation; three runs","Default; full conversation; three runs","Gemma 4 26B-A4B IT","https://arxiv.org/html/2610.12417v1#A5.T24","Intern-S1-Pro","Seed-2.0-Mini","ERNIE-4.5-VL-424B-A47B","Pixtral-Large-2411","Seed-1.6-Flash","ERNIE-4.5-VL-28B-A3B","Mistral-Small-3.2","Mistral-Large-2512","MiniMax-01","Nemotron-Nano-12B-V2-VL","InternVL3.5-Latest","Intern-S1-Mini","Flex-RSI executor+improver; task-specific fixed budget; final held-out test; latest run; 10 rounds per task","https://arxiv.org/html/2611.08946v1#A3.T13","DeepSeek-V4.1 Flash","Flex-RSI executor+improver; task-specific fixed budget; final held-out test; latest run; 10 rounds per task; 1017 model version","https://arxiv.org/html/2611.08946v1#A3.T14","Flex-RSI executor+improver; task-specific fixed budget; final held-out test; latest run","https://arxiv.org/html/2611.08946v1#A3.T15","Flex-RSI executor+improver; task-specific fixed budget; final held-out test; latest run; 1017 model version","https://arxiv.org/html/2611.08946v1#A3.T16","https://arxiv.org/html/2611.08946v1#A3.T17","https://arxiv.org/html/2611.08946v1#A3.T18","https://arxiv.org/html/2611.08946v1#A3.T19","https://arxiv.org/html/2611.08946v1#A3.T20","https://arxiv.org/html/2611.08946v1#A3.T21","https://arxiv.org/html/2611.08946v1#A3.T22","https://arxiv.org/html/2611.08946v1#A3.T23","https://arxiv.org/html/2611.08946v1#A3.T24","https://arxiv.org/html/2611.08946v1#A3.T25","https://arxiv.org/html/2611.08946v1#A3.T28","https://arxiv.org/html/2611.08946v1#A3.T31","https://arxiv.org/html/2610.12881v1#A1.T2","KIMI K3","natural prompt; all 2400 images","https://arxiv.org/html/2610.22009v1#A5.T2","de-biased prompt; all 2400 images","Gemma 4 12B-IT","https://arxiv.org/html/2610.30828v1#S5.T1","Qwen 3.5-397B","Reasoning","https://arxiv.org/html/2608.18726v1","DeepSeek-V4-flash","Qwen-2.5-72B","EuroLLM-9B","https://arxiv.org/html/2608.12894v1","Granite-3.0-8B","Occiglot-7B","Aya-8B","Salamandra-7B","Leo-7B","Teuken-7B","Pharia-7B","GENBA-10B-it","LLaMmlein-7B","https://arxiv.org/html/2608.22753v1","deepseek-chat","gpt-5.1-2025-11-13","gpt-5.5-2026-04-23","CUDA-full","https://arxiv.org/html/2608.25061v3","Triton-full","Triton-core","Symptom-only; Adaptive High; API","https://arxiv.org/html/2608.14863v1","Symptom-only; High; API","GPT 5.4 mini","GLM 5.1 (FP8)","Symptom-only; Enabled; Open, Self-Host","Symptom-only; Enabled; Open, API","Gemma 4 (31B)","Symptom-only; N/A; Open, API","GPT OSS (120B BF16)","Qwen2.5-14b-Ins","https://arxiv.org/html/2608.08459v1","LLaMA-3.1-70b-Ins","Qwen2.5-72b-Ins","Claude-opus-4-6","Qwen3-max","Deepseek-V4-flash","https://arxiv.org/html/2608.29711v1","Kimi-k2.5","Gemma-4-31B-It","https://arxiv.org/html/2608.25417v1","GLM-5V-Turbo","Qwen3-VL-32B-Thinking","Qwen3-VL-30B-A3B-Inst.","Qwen3-VL-30B-A3B-Think.","EASEL-9B","without C2 curriculum","https://arxiv.org/html/2608.09096v2","Minimax-M3","Qwen3.6-27b","https://arxiv.org/html/2608.29632v1","Qwen3-30B-A3B-Thinking","Non-reasoning","https://arxiv.org/html/2608.15535v1","Gemma4 31B","Sarvam 30B","Gemma2 9B","https://arxiv.org/html/2608.17895v1","Qwen3-VL-4B-Thinking","gemma-4-31B-it","https://arxiv.org/html/2608.14546v2","Seedream5 Pro","Seedream4.5","Qwen Image 2.0 Pro","FLUX.2-klein-4B","FireRed-Image-Edit","0.25 FPS","https://arxiv.org/html/2608.13210v1","128 Frames","Qwen3VL-8B","Qwen2.5VL-7B","InternVL3.5","64 Frames","https://arxiv.org/html/2608.15425v1","CLIP-ViT-B/32","LLaVA-1.5-7B","LLaVA-OneVision-Qwen2-7B","Qwen3VL-4B-Instruct","Qwen3VL-8B-Instruct","https://arxiv.org/html/2608.18595v1","https://arxiv.org/html/2608.16765v1","FireRed Image Edit 1.1","Qwen-Image-Edit-2509","https://arxiv.org/html/2608.24145v1","https://arxiv.org/html/2608.10042v1","Hammer2.1-7B","ToolACE-2.5-8B","Watt-Tool-8B","https://arxiv.org/html/2608.10875v1","Agent Loop","https://arxiv.org/html/2608.14718v1","https://arxiv.org/html/2608.10408v1","GPT-5-Mini","Gemini-3.0-Flash","InternVL-3.5-1B","InternVL-3.5-2B","InternVL-3.5-4B","InternVL-3.5-8B","InternVL-3.5-14B","https://arxiv.org/html/2608.27953v1","Common prompt/action schema; one run per task","https://arxiv.org/html/2609.01056v1#S3.T3","Qwen-3-32B","Qwen-3-4B","Self-Eval; avg@3; high reasoning where stated; developer agent: Claude Code2.1.177","https://arxiv.org/html/2609.01437v1#S4.T3","Self-Eval; avg@3; high reasoning where stated; developer agent: Codex0.144.3","Maximum 25 actions; same500-claim subset","https://arxiv.org/html/2609.01383v1#S4.T1","Maximum 10 actions; same500-claim subset","Maximum 1 actions; same500-claim subset","Qwen 3.6 35B","Qwen 3.5 2B","Qwen 3.5 0.8B","Conceptual prompts; default generation configuration","https://arxiv.org/html/2609.02502v1#S3.T1","GPT Image 1.5","FLUX.1-dev","SD 3.5 Large","Z-Image","FLUX.2-dev","Bagel","Janus-Pro","Image+text; paper main comparison under5% invalid-choice ceiling","https://arxiv.org/html/2609.08657v1#S6.T1","Default inference","https://arxiv.org/html/2609.04720v1#S4.T2","Chain-of-thought","Behavior guidance prompt","InternVL3-2B-KoNA","KoNA fine-tuned checkpoint","Qwen2.5-VL-3B-KoNA","llama3-8b-instruct","Zero-shot; authorship attribution; full neural model panel","https://arxiv.org/html/2609.06771v1#A2.T8","llama3.1-8b-instruct","qwen2.5-3b-instruct","qwen2.5-7b-instruct","deepseek-llm-7b-chat","qwen3-4b-instruct","llama3-8b","deepseek-llm-7b-base","qwen2.5-3b","qwen3-4b","e5-mistral-7b-instruct","gte-qwen2-7b-instruct","multilingual-e5-large","multilingual-e5-base","qwen3-embedding-8b","sfr-embedding-mistral","qwen3-embedding-4b","snowflake-arctic-embed-l-v2","qwen3-embedding-0.6b","e5-large-v2","e5-base-v2","facebook-contriever","gte-large-en-v1.5","bge-m3","bge-large-en-v1.5","e5-small-v2","gte-large","mxbai-embed-large-v1","bge-base-en-v1.5","gte-base","bge-base-zh-v1.5","facebook-contriever-msmarco","bge-large-zh-v1.5","distiluse-base-multilingual-cased-v2","all-roberta-large-v1","all-mpnet-base-v2","bge-small-en-v1.5","all-minilm-l12-v2","paraphrase-mpnet-base-v2","bert-base-uncased","all-minilm-l6-v2","paraphrase-multilingual-mpnet-base-v2","msmarco-distilbert-base-v4","allenai-specter","jina-embeddings-v2-small-en","jina-embeddings-v2-base-en","Zero-shot; all eight tasks","https://arxiv.org/html/2609.09396v1#S4.T1","Image-only zero-shot; empty working directory; PNG and PDF output","https://arxiv.org/html/2609.08155v1#S6.T3","CharTide-8B","Intern-S2-Preview","Phi-4 Vision 15B","Molmo-72B","ChartCoder","Task1; S1+S2; author scale judge","https://arxiv.org/html/2609.00525v1#S4.T4","Z-Image-Turbo","Grok Imagine","Qwen-Image 2512","SD3.5-Large","meme-only; human evaluation; three raters; temperature0","https://arxiv.org/html/2609.01772v1#A1.T9","context-aware; human evaluation; three raters; temperature0","Base; six-turn generation;90/5/5 train/validation/test split","https://arxiv.org/html/2609.01810v1#S4.T4","Mistral-Nemo-12B","Mistral-Small-24B","+LoRA; six-turn generation;90/5/5 train/validation/test split","https://arxiv.org/html/2609.02954v1#S6.T1","Legal prompting","Retrieval-augmented generation","Qwen-3-8B","LegalOne-R1","DISC-LawLLM","Non-thinking","https://arxiv.org/html/2609.02573v1#Sx3.T2","Thinking enabled","Paper default inference","https://arxiv.org/html/2609.02059v1#S3.T1","LLaVA-Next-LLaMA3-8B","IDEFICS2-8B","IDEFICS3-LLaMA3-8B","Molmo-7B-O-0924","Molmo-7B-D-0924","InternVL-3.5-30B-A3B","Qwen-2.5-VL-32B","InternVL-3.5-38B","Qwen-2.5-VL-72B","https://arxiv.org/html/2609.05141v1#S5.T2","type: General LLM; thinking: ✓","https://arxiv.org/html/2610.18299v1#S3.T3","type: General LLM; thinking: ✗","Qwen3-Nemotron-8B-BRRM","type: Ft. GenRM; thinking: –","Skywork-Reward-V2-Qwen3-8B","type: Ft. DisRM; thinking: –","Skywork-Reward-V2-Llama-3.2-3B","Skywork-Reward-V2-Llama-3.1-8B","RRM-7B","Llama-3.1-8B-Instruct-RM-RB2","RM-R1-Qwen2.5-Instruct-7B","Skywork-Reward-Llama-3.1-8B-v0.2","InternLM2-1.8B-Reward","self-rollout; 128 frames","https://arxiv.org/html/2610.40131v1#S4.T2","Keye-VL-1.5-8B","mode: NT; default settings","https://arxiv.org/html/2610.39639v1#A2.T2","mode: TH; default settings","Gemma4-E2B","Gemma4-E4B","Gemma4-26B-A4B","Gemma4-31B","mode: TH; default settings; high effort","Grok-4.20","https://arxiv.org/html/2610.31314v1#S5.T3","Gemini 2.5 Flash Image","https://arxiv.org/html/2610.28316v1#S5.T2","Seedream 4.0","Qwen-Image-Edit","FLUX.1 Kontext","Step1X-Edit","Direct MLLM prediction","https://arxiv.org/html/2611.02826v1#A4.T7","RxnMiner","ChemEAGLE","Existing extraction system","https://arxiv.org/html/2610.35031v1#S4.T3","harness: Hermes","harness: OpenClaw","https://arxiv.org/html/2611.08369v1#A3.T29","Gemini-3.1","Claude-4.6","Qwen3-30B","HealthGPT-8B","HealthGPT-4B","Hulu-Med-7B","HuatuoGPT-o1-72B","Lingshu-32B","Hulu-Med-4B","Hulu-Med-32B","HuatuoGPT-o1-7B","Lingshu-7B","ChemVLM-8B","https://arxiv.org/html/2611.06911v1#A7.T12","ChemVLM-26B","ACE-Brain-0-8B","Pelican-VL-1.0-7B","RynnBrain-8B","RynnBrain-CoP-8B","RynnBrain-Plan-8B","Grok-4.20-Non-Reasoning","Grok-4.20-Reasoning","MiMo-V2-Omni","https://arxiv.org/html/2610.32459v1#S4.T2","Seedream 5.0","https://arxiv.org/html/2610.36554v1#S5.T2","OpenAI o3","Qwen3-VL-8B-Inst.","Gemma-3-12B-it","Qwen2.5-VL-7B-Inst.","InternVL3-8B-Inst.","Qwen2.5-VL-3B-Inst.","InternVL3-2B-Inst.","Qwen3-VL-8B-Think.","Kimi-VL-A3B-Think.","Visionary-R1","GLM-4.1V-9B-Think.","https://arxiv.org/html/2610.30103v1#A6.T9","Mistral-Medium-3.5","https://arxiv.org/html/2610.20344v1#S4.T4","https://arxiv.org/html/2610.19399v1#A6.T19","Solar Pro 2","Solar Pro 4","HCX-003","HCX-007","Qwen2 7B","EXAONE 3.0 7.8B","EXAONE 3.5 7.8B","EXAONE 4.0 32B","EXAONE 4.5 33B","search-only","https://arxiv.org/html/2610.37166v1#S4.T1","search-only; effort: Max","search-only; effort: High","Aero-Realtime-4B","Native Audio-Visual Models","https://arxiv.org/html/2610.28405v1#S4.T3","DuplexOmni","Gemini-3.1-Flash-Live","Qwen3.5-Omni-Plus-Realtime","StreamingVLM","Streaming Visual Models with Timestamp-Aligned Text","Mage-VL","Coding agents","https://arxiv.org/html/2610.18507v1#A6.T17","Mistral OCR","OCR and document-understanding baselines","LightOnOCR-2-1B","LandingAI","olmOCR-2-7B","PaddleOCR","PaddleOCR-VL","No-Thinking","https://arxiv.org/html/2610.12961v1#A3.T14","Phi-4-reasoning-vision-15B","Llama-3.1-Nemotron-Nano-VL-8B","Non-thinking Models","https://arxiv.org/html/2610.19560v1#S4.T3","Thinking Models","Qwen3-235B-A22B-T","Tracer-High-1.7B","Tracer-High-4B","Tracer-High-8B","https://arxiv.org/html/2610.29014v1#S4.T3","research agent; harness: Codex","https://arxiv.org/html/2610.26342v1#S4.T3","research agent; harness: Claude Code","https://arxiv.org/html/2611.07541v1#S4.T1","Phi-3.5-Vision-4B","SpatialThinker-3B","RoboBrain2.0-7B","SpaceR-SFT-7B","BLIP-2","https://arxiv.org/html/2611.02519v1#S5.T5","MedSigLIP","SigLIP","RzenEmbed","VLM2Vec-V2.0","MedImageInsight","Qwen3-VL-Embed-2B","Qwen3-VL-Embed-8B","MedEmb-2B","MedEmb-8B","Direct answer","https://arxiv.org/html/2611.04687v1#A4.T5","Claude-sonnet-4.5","Apollo-7B","Direct answer; specialist checkpoint","ShizhenGPT-7B","Llama3-8B-Instruct","Meditron3-8B","UltraMedical-8B","II-Medical-8B","Apollo2-9B","Agent workflow","Ministral3-14B","GPT-oss-20B","ShizhenGPT-32B","Llama3-70B-Instruct","OpenBioLLM-70B","Apollo-72B","Mimo-v2-flash","Llama4-maverick","Deepseek-v3.2","https://arxiv.org/html/2611.04228v1#A2.T10","https://arxiv.org/html/2610.31650v1#A3.T8","MiMo-V2.5-Flash","Direct-VLM; Tier-B; 335 orders","https://arxiv.org/html/2611.00791v1#S4.T3","Doubao-Seed-1.6-Vision","ERNIE-4.5-Turbo-VL","Hunyuan-Vision","Nova-Pro","Pixtral-Large","SWE-agent; Linux; medium effort","https://arxiv.org/html/2610.19368v1#S5.T2","Claude Code; Linux; medium effort","OpenHands; Linux; medium effort","Win-agent; Windows; medium effort","CogVideoX-Fun-V1.5","https://arxiv.org/html/2610.15562v1#S5.F3","Cosmos3-Nano","HunyuanVideo-I2V","LTX-Video-13B","MAGI-1-4.5B","SkyReels-V2-I2V","Wan2.1-I2V-14B","Wan2.2-TI2V-5B","Wan2.2-I2V-A14B","Standard Asynchronous Models; high","https://arxiv.org/html/2610.22155v1#A2.T3","Standard Asynchronous Models; low","GPT Realtime 2","Real-time Streaming Models; high","Real-time Streaming Models; low","Gemini 3.1 Flash Live","Real-time Streaming Models; fixed","Decision Models","OpenWAM","100 episodes per task; control steps/plan: 16","https://arxiv.org/html/2610.26765v1#S5.T5","pi0.5","100 episodes per task; control steps/plan: 30","DM0.5","LingBot-VLA","HY-Embodied","Motus","Spatial Forcing","move_eef; 10 episodes per task","Screenshot-only; five runs per task","https://arxiv.org/html/2609.17885v1#S4.T2","Holo3-35B-A3B","Screenshot-only; Qwen3-VL uses OmniParser; five runs per task","OpenCUA-7B","Chinese; zero-shot; temperature 0; reasoning disabled; 2,048 output tokens","https://arxiv.org/html/2609.17128v1#A5.T6","Gemma4-26B","Llama3.2-3B","Deepseek-7B","Qwen3-2B","Kimi-K2.7","Qwen3.6-Max","Chinese; zero-shot; temperature 0; reasoning disabled","https://arxiv.org/html/2609.17128v1#S5.T2","FG-CLIP","Size: 0.2B; published regional prompting; published checkpoint","https://arxiv.org/html/2609.16847v1#S4.T1","SigLIP2","Size: 0.4B; published regional prompting; published checkpoint","DreamLIP","FineCLIP","EVA-CLIP","Size: 8B; published regional prompting; published checkpoint","MM-EMBED","VLM2VEC","LamRA","mmE5","Size: 11B; published regional prompting; published checkpoint","RegRet-3B","Size: 3B; published regional prompting; REGMB fine-tune","RegRet-8B-zs","RegRet-8B","Size: 8B; published regional prompting; REGMB fine-tune","DSANet","in-domain; published training protocol","https://arxiv.org/html/2609.17843v1#S5.T2","FIDeL","FrameShield","Jigsaw-VAD","LaGOVAD","VadCLIP","UR-DMU","FlowJigsaw-SSL","FlowJigsaw-Sup.","cross-anomaly; published training protocol","cross-task; published training protocol","cross-task-anomaly; published training protocol","Published end-to-end speech configuration","https://arxiv.org/html/2609.16614v1#S4.T2","MiniCPM-o-4.5","Covo-Audio-Chat","ASR: Parakeet-TDT-0.6B-v3; TTS: Qwen3-TTS","GPT-5.6 SOL","Correct association; one investigation per model–website pair; frozen report","https://arxiv.org/html/2609.16900v1#S4.T2","Doubao Seed 2.0","Agent: Codex; xhigh or higher effort; 3-hour task limit; three independent trials","https://arxiv.org/html/2609.17439v1#S2.F4","Agent: Claude Code; xhigh or higher effort; 3-hour task limit; three independent trials","Hunyuan 4 Preview","Agent: OpenCode; xhigh or higher effort; 3-hour task limit; three independent trials","Hunyuan T3","Mobile; benign; ten runs","https://arxiv.org/html/2608.26204v2","Claude 4.7 Opus","Gemini 2.5 CU","Desktop; benign; ten runs","https://arxiv.org/html/2608.26623v1","https://arxiv.org/html/2608.22510v1","GLM-5-Turbo","https://arxiv.org/html/2608.17566v2","Kling O3","VACE","Ditto","VINO","OmniWeaving","KiwiEdit","SAMA","CoinVE-Edit","https://arxiv.org/html/2608.24979v2","Codex; reasoning=max","DeepSeek V4 Pro-0813","DeepSeek V4 Flash-0731","Apodex 1.1","Frontier Agent (Agent Team)","https://arxiv.org/html/2608.21928v1","Mistral-Large-2512 (675B)","Llama-4-Maverick (400B)","Instruct","Ministral-8B-2512","https://arxiv.org/html/2608.28281v2","DeepSeek-V4-Flash-0731","https://arxiv.org/html/2608.14441v1","https://arxiv.org/html/2608.24509v1","Genie 3","https://arxiv.org/html/2608.13552v2","HappyOyster","LingBot-World2","HY-World2","Hunyuan-GameCraft","Matrix-Game-3.0","https://arxiv.org/html/2608.26832v1","Doubao-Seed-2.0-pro","Default chain-of-thought; published mean","https://arxiv.org/html/2609.05405v1#S4.T1","Mistral-Small-3.1","Task1 readiness assessment; balanced panel","https://arxiv.org/html/2609.10539v1#S4.T1","Tool-augmented VLM","https://arxiv.org/html/2609.11318v1#S4.T4","GPT-4o (2024-11)","GPT-4o Mini (2024-07)","Tool-free VLM","o3 Deep Research","Deep Research system","o4-mini Deep Research","Perplexity Sonar Deep Research","Tongyi DeepResearch 30B","minimax-m2.7","Agent: DeerFlow","qwen3-vl-235b","gpt-5.5-xhigh","Full-corpus stage; published automated judge; paper model settings; original model label gpt-5.5-xhigh","https://arxiv.org/html/2609.00646v1#S4.T2","claude-opus-4.8-max","Full-corpus stage; published automated judge; paper model settings; original model label claude-opus-4.8-max","hy3","Full-corpus stage; published automated judge; paper model settings; original model label hy3","doubao-seed-2.1-pro","Full-corpus stage; published automated judge; paper model settings; original model label doubao-seed-2.1-pro","Full-corpus stage; published automated judge; paper model settings; original model label glm-5.2","gemini-3.1-pro","Full-corpus stage; published automated judge; paper model settings; original model label gemini-3.1-pro","Full-corpus stage; published automated judge; paper model settings; original model label kimi-k2.6","Full-corpus stage; published automated judge; paper model settings; original model label qwen3.7-max","mimo-v2.5-pro","Full-corpus stage; published automated judge; paper model settings; original model label mimo-v2.5-pro","gpt-5.6-sol-max","Full-corpus stage; published automated judge; paper model settings; original model label [0.4pt/1.6pt] gpt-5.6-sol-max †","claude-fable-5-max","Full-corpus stage; published automated judge; paper model settings; original model label claude-fable-5-max †","claude-opus-5-max","Full-corpus stage; published automated judge; paper model settings; original model label claude-opus-5-max †","Full-corpus stage; published automated judge; paper model settings; original model label kimi-k3 †","Full-corpus stage; published automated judge; paper model settings; original model label qwen3.8-max †","Full-corpus stage; published automated judge; paper model settings; original model label grok-4.5 †","gemini-3.6-flash-high","Full-corpus stage; published automated judge; paper model settings; original model label gemini-3.6-flash-high †","gpt-image-2","Full-corpus stage; published automated judge; paper model settings; original model label gpt-image-2","gpt-image-1.5","Full-corpus stage; published automated judge; paper model settings; original model label gpt-image-1.5","nano-banana-2","Full-corpus stage; published automated judge; paper model settings; original model label nano-banana-2","seedream-5.0-pro","Full-corpus stage; published automated judge; paper model settings; original model label seedream-5.0-pro","nano-banana-pro","Full-corpus stage; published automated judge; paper model settings; original model label nano-banana-pro","wan2.7-image-pro","Full-corpus stage; published automated judge; paper model settings; original model label wan2.7-image-pro","Task3; fixed zero-shot protocol","https://arxiv.org/html/2609.03597v1#S5.T2","Gemma 4 26B","One decision per scene; completed-response denominator","https://arxiv.org/html/2609.10895v1#S5.T1","History-level EHR; thinking disabled for open-source and medical models","https://arxiv.org/html/2609.07601v1#S3.T2","Phi-4-mini","HealthGPT-Pro-4B","HealthGPT-Pro-8B","Hulu-Med-30A3","Full end-to-end chain; published symbolic rules","https://arxiv.org/html/2609.11185v1#S4.T4","Baichuan M2","Baichuan M3","Pairwise task novelty; full-paper input; author judge assignment","https://arxiv.org/html/2609.11234v1#S4.T3","Gemma4-31B-IT","CycleReviewer-8B","DeepReviewer-14B","Zero-shot with captioning; type+action","https://arxiv.org/html/2609.10356v1#S6.T3","https://arxiv.org/html/2610.08720v1#S3.T3","https://arxiv.org/html/2610.09671v1#S5.T3","https://arxiv.org/html/2610.05715v1#S3.T1","Cosmos-Reason1-7B","Phi-3.5-Vision","deterministic direct answer","https://arxiv.org/html/2610.11312v1#S5.T3","Qwen3-VL-8B-I","Qwen3-VL-8B-T","thinking variant","MedGemma-4B","QoQ-Med-7B","BreastGPT-8B","https://arxiv.org/html/2610.06571v1#S4.T2","Janus-Pro-7B","HuatuoGPT-Vision-7B","MedGemma-4B-it","LLaVA-Med-v1.5","RadFM","https://arxiv.org/html/2610.11559v1#S4.T1","Codex; max","https://arxiv.org/html/2611.09017v1#S3.T3","Claude Code; max","DeepSeek-v4.1-Flash","DSH; max","Kimi-K3.1-Pro","Kimi CLI; max","ZCode; max","Claude Code; xhigh","https://arxiv.org/html/2610.10455v1#S5.T1","Mistral-7B-Instruct","Grok-1.5","https://arxiv.org/html/2610.01092v1#S5.T2","Kling-v3-Pro","Wan-2.7","Cosmos-3","https://arxiv.org/html/2610.10624v1#S3.T2","Nemotron-3-Nano-Omni","DOTS-3-Note-Preview","Spatial-Omni SO-7B","video-SALMONN2+ 7B","OmniVideo-7B (VITA1.5)","OpenCode; 300 tasks; three repeats","https://arxiv.org/html/2610.07557v1#S3.T2","HY4-Preview","Claude Code; 300 tasks; three repeats","Hermes Agent; 300 tasks; three repeats","https://arxiv.org/html/2610.12183v1#S5.T3","https://arxiv.org/html/2610.09374v1#S4.T2","Granite 4.0 350M","https://arxiv.org/html/2610.09152v1#A8.T14","Granite 4.0-H 350M","Qwen3 0.6B","Gemma 3 1B","Granite 4.0 1B","Granite 4.0-H 1B","EuroLLM 1.7B","Qwen3 1.7B","Granite 3.2 2B","Llama 3.2 3B","Tiny-Aya Global","Tiny-Aya Water","Mistral-SK 7B","Mistral-SK 7B (Alpaca)","Teuken 7B","Aya Expanse 8B","Ministral 3 8B (R)","EuroLLM 9B","Bielik 11B","Ministral 3 14B (R)","Qwen3-SK 14B","Qwen3-SK 14B (Alpaca)","gpt-oss 20B","EuroLLM 22B","TildeOpen 30B","TildeOpen 30B-64k","TildeOpen-SK 30B (Alpaca)","Aya Expanse 32B","Granite 4.0-H Small","Apertus 70B","gpt-oss 120B","Mistral Large 3 675B","Full trajectory; exact annotated first error","https://arxiv.org/html/2609.17930v1#S3.T1","Gemini 3.0 Flash","Published default video input","https://arxiv.org/html/2609.17248v1#S6.T2","Audio-text guided token compression; source label Ours","PK-CB; deterministic pass@1","https://arxiv.org/html/2609.18270v1#S5.T3","Qwen3.5 122B-A10B","CGR-CB; deterministic pass@1","CGR-OB; deterministic pass@1","TIA-OB; deterministic pass@1","BGE-M3","sparse","https://arxiv.org/html/2609.18908v1#S5.T3","dense","ModernBERT","raw text","late fusion","Legal-Longformer","raw text; zero-shot","raw text; zero-shot CoT","FireRed2-AED","No context; greedy decoding; 1,024 output tokens","https://arxiv.org/html/2609.18680v1#S4.T2","SenseVoice","Whisper-v3","Qwen3-ASR-0.6B","Qwen3-ASR-1.7B","VibeVoice","SeedASR","Published cultural identification task; multiple choice","https://arxiv.org/html/2609.19088v1#S4.T2","Qwen3-VL-32b","InternVL3-38b","Qwen2.5-VL-72b","Qwen3-VL-8b","InternVL3-14b","Qwen2.5-VL-32b","Qwen2.5-VL-7b","Gemma3-12b-it","Gemma3-27b-it","InternVL3-9b","InternVL3-8b","MiniCPM-Llama3-V-2.5","GLM-4V-9b","Qwen2.5-VL-3b","LLaVA-Next-8b","Gemma3-4b-it","InternVL3-2b","Yi-VL-6b","InternVL3-1b","Yi-VL-34b","CogVLM2-19b","LLaVA-Next-34b","LLaVA-Next-72b","Published two system-prompt modes; three replications; unclear excluded","https://arxiv.org/html/2609.18605v1#S4.T2","Nemotron-3-Ultra","Gemma-4-26B","Seed-OSS-36B","Nemotron-3-Super","CosPlace","Open-set test; CosPlace [ 8 ]","https://arxiv.org/html/2609.02012v1#S4.T1","BoQ","Open-set test; BoQ [ 16 ]","SelaVPR","Open-set test; SelaVPR [ 24 ]","EDTformer","Open-set test; EDTformer [ 25 ]","ImAge","Open-set test; ImAge [ 17 ]","SALAD","Open-set test; SALAD (224) [ 15 ]","Open-set test; SALAD (518) [ 15 ]","FoL","Open-set test; FoL (stage-1) [ 22 ]","Open-set test; FoL (+re-ranking) [ 22 ]","GLAM","Open-set test; GLAM (fused, stage-1)","Open-set test; GLAM (+re-ranking)","Common nine-subtask set; author rubric","https://arxiv.org/html/2609.02864v1#S4.T3","Gemini 3 Pro Image Preview","Gemini 3.1 Flash Image Preview","GPT Image 1","Wan2.2-14B","VBVR-Wan2.2-14B","GLiNER-base","OCR: Textract","https://arxiv.org/html/2609.02207v1#S6.T3","GLiNER-multi-PII","NVIDIA GLiNER PII","GLiNER2","OpenAI Privacy Filter","OCR: —","OCR: —; Code Interpreter","Base-LLM; Task1 personalized strategy","https://arxiv.org/html/2609.00921v1#A11.T10","Meta-Llama-3.1-70B-Instruct","Meta-Llama-3.1-8B-Instruct","Mistral-Small-3.2-24B-Instruct-2506","FHP-LLM; Task1 personalized strategy","PAP-LLM; Task1 personalized strategy","RAP-BM25; Task1 personalized strategy","RAP-BERT; Task1 personalized strategy","P-SFT; Task1 personalized strategy","qwen3-14b","llama3.2-3b","ministral-8b","https://arxiv.org/html/2611.08971v1#S4.T1","Qwen-2.5-32B","Qwen-2.5-7B","6 prompt conditions × 30 generations; 3-judge panel","https://arxiv.org/pdf/2611.09094#page=6","Qwen3.8 27B fp8","Ministral 3 14B 2512","https://arxiv.org/html/2609.20574v1#S5.T5","VISA-7B","Source-attribution fine-tune","MAPPET-7B","MAPPET-8B","MAPPET-I-8B","word-only","https://arxiv.org/html/2609.19916v1#S4.T1","Solar Pro3-12B","Solar-10.7B","EXAONE-3.5-7.8B","EXAONE-4.0-32B","LLaMA-3.1-8B","word with particle","FS","https://arxiv.org/html/2609.20584v1#S4.T3","Ansari","Published MCQ protocol","https://arxiv.org/html/2609.22038v1#S4.T1","gemma-4-31B","gemma-4-26B-A4B","Fanar-Sadiq","gemma-3-12B","ALLaM-7B","Keye-VL 1.5 8B","Video only","https://arxiv.org/html/2609.21521v1#S4.T3","InternVL 3.5 2B","InternVL 3.5 8B","InternVL 3.5 14B","InternVL 3.5 38B","Qwen3-VL 4B","video-SALMONN 2+ 7B","Video and audio","video-SALMONN 2+ 72B","Native audio; fixed 5,038-item denominator; persistent inference failures incorrect","https://arxiv.org/html/2609.23416v1#S6.T2","Doubao seed 2.0 Lite","MOSS-Audio-8B-Thinking","Audio Flamingo Next 8B","Published API setting","https://arxiv.org/html/2609.24210v1#S3.T1","Seed1.6-VL","MiMo-VL-SFT","LLaVA-Critic-R1","DeepSeek-VL","GLM-4V","MiMo-VL-SFT (NoThink)","MiMo-VL-RL (NoThink)","Kimi-VL","Molmo","Molmo2","ReAct; answer + search + interaction; 173 cases","https://arxiv.org/html/2609.24002v1#S4.T6","ReAct; answer + search + interaction; 50 cases","Ollama; four granularities; 3 repeats; quantization 6-bit","https://arxiv.org/html/2609.24161v1#S5.T1","Ollama; four granularities; 3 repeats; quantization 4-bit","Granite4 (3.4B)","Hermes-3-8B","Ollama; four granularities; 3 repeats; quantization MXFP4","xLAM-2-8B","Ollama; four granularities; 3 repeats; quantization 3-bit","FunctionGemma-270M","Ollama; four granularities; 3 repeats; quantization 8-bit","PIPs","Published off-the-shelf/shared-VGGT reconstruction","https://arxiv.org/html/2609.01899v1#A1.T1","PIPs++","TAPTRv3","TAPIR","CoTracker2","CoTracker1","BootsTAPIR","DOT","LocoTrack","CoTracker3","MegaFlow","TAPNext++","Track-On-R","AllTracker","CoWTracker","SpatialTracker","SceneTracker","DELTA","SpatialTrackerV2","TAPIP3D","MV-TAP","LAPA","MVTracker","MVTracker (finetuned)","Published off-the-shelf/shared-VGGT reconstruction; source fine-tuned checkpoint","OmniX","Self-estimated multi-view geometry","Pengi","Original wording; full evaluated illusion panel; source marks over20% unmappable free-text replies","https://arxiv.org/html/2609.02277v1#S5.T2","Original wording; full evaluated illusion panel","MuLLaMa","Kimi-Audio-Instruct","Fun-Audio-Chat","Qwen-Audio-Chat","Qwen2-Audio-Instruct","GLM-4-Voice","Refined wording; full evaluated illusion panel; source marks over20% unmappable free-text replies","Refined wording; full evaluated illusion panel","One bounded episode per task; four-tool scaffold","https://arxiv.org/html/2609.27571v1#S4.T1","Gemini-3.5-Flash-Lite","credulous-stable; explicit semantics; main evaluation set","https://arxiv.org/html/2609.27517v1#Sx4.T1","Qwen2.5-coder 32B","DeepSeek-R1 32B","skeptical-stable; explicit semantics; main evaluation set","well-founded; explicit semantics; main evaluation set","sldnf; explicit semantics; main evaluation set","w/o tool","https://arxiv.org/html/2609.27349v1#S4.T3","w/ tool","Gpt-Oss-120B","Ether0-24B","InternLM-S1-241B","Qwen3.8 max","Active camera; RGB demonstrations; 840 episodes; source table transposed","https://arxiv.org/html/2609.19554v1#S4.T3","Doubao 2.1","Gemini 3.6","Qwen3.7 max","Qwen3.7 plus","Mimo v2.5","oracle-telugu; published Gemini judge; table transposed","https://arxiv.org/html/2609.19879v1#S5.T5","Hex-1","Sarvam-m","seamless-translation; published Gemini judge; table transposed","indic-translation; published Gemini judge; table transposed","single-turn; default","https://arxiv.org/html/2609.21386v1#S4.T1","single-turn; thinking","STAR; default","STAR; thinking","DVD; default","DVD; thinking","AVP; default","AVP; thinking","Video-ReAct; default","Video-ReAct; thinking","Qwen3-VL-4B-Ins","Full context","https://arxiv.org/html/2609.28236v1#S6.T2","Qwen3-VL-8B-Ins","Qwen3-VL-32B-Ins","Mistral-Small-3.1-24B","Cambrian-S-7B","RoboBrain2.5-8B-NV","EMem","EMem-8B","MIRIX","MemVerse","TeleMem","MMA","Generic tool library","https://arxiv.org/html/2610.27618v1#S5.T2","Task-oriented tool library","DeepSeek-V4.1","Standalone model; part track","https://arxiv.org/html/2609.03773v1#S4.T4","Agent: Codex; assembly track","https://arxiv.org/html/2609.03773v1#S4.T5","Agent: Claude Code; assembly track","Standalone model; assembly track","Veo-3.1","Video generation; continuous physical-invariant score","https://arxiv.org/html/2609.04200v1#S4.T3","Cosmos-2.5-2B","Cosmos-2.5-14B","Wan2.2-5B","Omni","Ten-turn dialogue; cumulative false-premise turns4–10; two-detector mean","https://arxiv.org/html/2609.03331v1#A9.T5","LLaVA-NeXT-13B","LLaVA-OneVision-0.5B","Qwen3.5-Plus-2026-04-20","Qwen3.6-Plus-2026-04-02","UniAD","Pseudo-simulation; integrated stages1+2","https://arxiv.org/html/2609.02462v1#S5.T3","SparseDrive","HIP-AD","MomAD","Uni-V2X","CoS-V2X","Agent: Claude Code; high reasoning;250tasks;four rollouts per task","https://cwe-bench.com/#leaderboard","Agent: opencode; high reasoning;250tasks;four rollouts per task","Agent: Antigravity; high reasoning;250tasks;four rollouts per task","Agent: Codex; high reasoning;250tasks;four rollouts per task","DeepSeek-V4.1-Flash (1017)","Qwen3.8-Max (0902)","Published common 380-item protocol","https://arxiv.org/html/2609.21344v1#S4.T6","Ling-flash-2.0","GLM-4-32B-0414","Hunyuan-A13B","English prompts; published repeated-run mean","https://arxiv.org/html/2609.21722v1#Sx4.T3","Chinese prompts; published repeated-run mean","Agent: Claude Code; egress sealed; pass@1","https://arxiv.org/html/2609.21562v1#S4.T3","Agent: Codex; egress sealed; pass@1","Agent: OpenCode; egress sealed; pass@1","UniVideo","Published reference-to-video protocol","https://arxiv.org/html/2609.22069v1#S5.T3","LoomVideo","Bernini","MiniMax H3","Vidu-Q2-Pro","Kling 3.0 Omni","Happy Horse 1.0","Seedance 2.5","embeddinggemma","Cross-language; model size 300M","https://arxiv.org/html/2609.23231v1#S3.T2","Cross-language; model size 568M","qwen3","Cross-language; model size 600M","nomic-v2-moe","Cross-language; model size 475M","granite","Cross-language; model size 278M","LaBSE","Cross-language; model size 471M","e5-large-instruct","Cross-language; model size 560M","SapBERT","Cross-language; model size 270M","GPT-5.3 Chat","3-shot ( T = 0.2 T=0.2 ); three held-out runs","https://arxiv.org/html/2609.23695v1#S5.T4","GPT-5.2 Chat","5-shot ( T = 0.0 T=0.0 ); three held-out runs","xAI Grok 4.5","3-shot ( T = 0.5 T=0.5 ); three held-out runs","3-shot ( T = 0.8 T=0.8 ); three held-out runs","3-shot ( T = 0.0 T=0.0 ); three held-out runs","Nous Hermes 4 70B","Ministral 14B 2512","5-shot ( T = 0.8 T=0.8 ); three held-out runs","5-shot ( T = 0.5 T=0.5 ); three held-out runs","Gemma 4 26B A4B IT","Nous Hermes 3 Llama 3.1 405B","5-shot ( T = 0.2 T=0.2 ); three held-out runs","Nous Hermes 4 405B","Ministral 8B 2512","Mistral Small 2603","IBM Granite 4.0 H-Micro","Microsoft Phi-4","0-shot ( T = 0.5 T=0.5 ); three held-out runs","Llama 3.2 3B Instruct","Amazon Nova Micro V1","Gemma 3 4B IT","Qwen 2.5 7B Instruct","NVIDIA Nemotron 3 Nano Omni 30B","Llama 3.2 1B Instruct","Ministral 3B 2512","0-shot ( T = 0.0 T=0.0 ); three held-out runs","Common basic-tool setting; all 123 valid outputs","https://arxiv.org/html/2609.25192v1#S4.T3","Hunyuan3-Thinking","Three repeats; 2,397 trajectories per model","https://arxiv.org/html/2609.29167v1#S5.T5","Gemma 4 E4B IT","Constrained setting; published DCR cascade","https://arxiv.org/html/2609.30979v1#S4.T2","Doubao-Seed-2-Pro","Mistral-Small-3.2-24B","Agent: Claude Code; reasoning xhigh; 13/13 questions; 39 measured attempts","https://www.eebench.org/","Agent: Codex CLI; reasoning source default; 13/13 questions; 39 measured attempts","Agent: Claude Code; reasoning high; 13/13 questions; 39 measured attempts","Agent: Gemini CLI; reasoning high; 13/13 questions; 39 measured attempts","Agent: Grok CLI; reasoning source default; 13/13 questions; 39 measured attempts","Agent: Codex CLI; reasoning high; 13/13 questions; 39 measured attempts","Agent: Muse Code; reasoning high; 13/13 questions; 39 measured attempts","Agent: Claude Code; reasoning source default; 13/13 questions; 39 measured attempts","Agent: Codex CLI; reasoning xhigh; 13/13 questions; 39 measured attempts","Agent: Claude Code; reasoning max; 13/13 questions; 37 measured attempts","Agent: Gemini CLI; reasoning source default; 13/13 questions; 39 measured attempts","Agent: Claude Code; reasoning source default; 13/13 questions; 43 measured attempts","Agent: Gemini CLI; reasoning source default; 13/13 questions; 36 measured attempts","Agent: Claude Code; reasoning high; 13/13 questions; 46 measured attempts","Agent: Gemini CLI; reasoning source default; 13/13 questions; 16 measured attempts","Agent: Claude Code; reasoning source default; 13/13 questions; 31 measured attempts","Agent: Gemini CLI; reasoning source default; 13/13 questions; 24 measured attempts","Agent: Grok CLI; reasoning source default; 13/13 questions; 28 measured attempts","Agent: Claude Code; reasoning source default; 13/13 questions; 40 measured attempts","Claude Code; effort: max; Harbor 0.21.0; 8h limit","https://arxiv.org/html/2610.18449v1#S9.T6","Codex; effort: max; Harbor 0.21.0; 8h limit","Claude Code; effort: high; Harbor 0.21.0; 8h limit","mini-SWE-agent; effort: provider default; Harbor 0.21.0; 8h limit","Claude Code; effort: provider default; Harbor 0.21.0; 8h limit","Published LLM setting","https://arxiv.org/html/2610.17694v1#A3.T17","2,000 tokens","https://arxiv.org/html/2610.18053v1#S5.T4","Gemini Robotics-ER 2","Both presentation orders; equal engineering/research weight","https://arxiv.org/html/2609.25804v1#A4.T4","Grok 4.20 Reasoning","Agent: Codex; default instruction; evaluation-valid runs","https://arxiv.org/html/2609.19892v1#S4.T1","Agent: Claude Code; default instruction; evaluation-valid runs","Agent: OpenCode; default instruction; evaluation-valid runs","PanDerm","https://arxiv.org/html/2608.22059v1","MedGemma","BiomedCLIP","ViT-B/16","DINOv2","RAD-DINO","LLaVA-Med","RETFound","MAE","ResNet-50","https://arxiv.org/html/2608.09281v1","Llama-3.2-11B-Vision-Instruct","Llama-4-Scout-17B-16E-Instruct","https://arxiv.org/html/2608.29286v1","Doubao-Seed-1.6-vision","https://arxiv.org/html/2608.10396v1","Kimi2.6","SFT on FormStruct-Bench","DeepSeek-VL-2","MinerU 2.5 Pro","PaddleOCR-VL-1.6","https://arxiv.org/html/2608.30903v1","Llama-4-Maverick-17B","Ministral3-8B-2512","https://arxiv.org/html/2608.18423v2","Full source video","https://arxiv.org/html/2608.13049v1","Kling-V3","Mitty-EPIC14B","Ordered source frames, native interface limit","Veo 3.1","First and last source frames","Grok Imagine Video","Up to seven uniformly spaced source frames","SkyReels-V3-R2V","First source frame; 121 output frames at 24 fps","LongCat","HunyuanVideo 1.5-I2V","https://arxiv.org/html/2608.16349v1","Nex-N2-Pro","Claude Opus 4.6","https://arxiv.org/html/2608.08964v1","Gemini 3.1 Pro","DeepSeek V3","Published base model; four independent rollouts per task; single-attempt mean","https://arxiv.org/html/2609.05395v1#S3.T1","gemma-4-26B-A4B-it","EXAONE-4.5-33B","Qwen3.5-4B EDGE-GRPO","EDGE-GRPO fine-tune; four independent rollouts per task; single-attempt mean","Qwen3.5-9B EDGE-GRPO","Published base model; four independent rollouts per task; any-of-four success","EDGE-GRPO fine-tune; four independent rollouts per task; any-of-four success","Solo; common scaffold; six rounds;100 problems","https://arxiv.org/html/2609.04667v1#S4.T1","Arena; common scaffold; six rounds;100 problems","overall; original elderly instructions; native GUI scaffolds;25 steps/120 seconds","https://arxiv.org/html/2609.04850v1#S4.T2","Doubao-1.5-UI-TARS (7B)","GUI-Owl-1.5 (8B-Think)","UI-Venus-1.5 (8B)","AutoGLM (Phone-9B)","online; original elderly instructions; native GUI scaffolds;25 steps/120 seconds","offline; original elderly instructions; native GUI scaffolds;25 steps/120 seconds","Published detector prompt/decoding; video or 32 frames;13 two-class subsets","https://arxiv.org/html/2609.03611v1#S4.T1","Guardian (thinking)","RoboReward-8B","ViFailback-8B","HY-Embodied-0.5","Qwen3-VL-2B-Thinking","RoboFAC-7B","FailSense-Calvin-3B","Region; human verified; greedy;30 steps;one trial;seed 300","https://arxiv.org/html/2609.03588v1#Sx3.T2","Claude-haiku-4-5-20251001","Glm-4.5-Air","Agent: Claude Code;high;three runs;10-minute task limit","https://arxiv.org/html/2609.08040v1#S4.T2","Agent: Codex;xhigh;three runs;10-minute task limit","Agent: OpenCode;default effort;three runs;10-minute task limit","Qwen-Plus","Zero-shot; common candidate labels and task prompts; full published panel","https://arxiv.org/html/2609.09657v1#S5.T5","16-shot; text; greedy;32 output tokens; matched published 606/912 run coverage","https://arxiv.org/html/2609.15028v1#S4.T2","Gemma 4-31B IT","Gemma 4-26B-A4B IT","Gemma 4-E4B IT","16-shot; image; greedy;32 output tokens; matched published 606/912 run coverage","thinking disabled/minimized; cap 2048; natural-language format","https://arxiv.org/html/2610.01605v1#S5.T4","GLM-4.6V-FlashX","thinking enabled; cap 4096; natural-language format","https://arxiv.org/html/2610.02715v1#A3.T6","Direct ST generation; Motion Control; Easy","https://arxiv.org/html/2610.02982v1#S5.T5","Direct ST generation; Motion Control; Medium","Direct ST generation; Motion Control; Hard","Direct ST generation; Material Handling; Easy","Direct ST generation; Material Handling; Medium","Direct ST generation; Material Handling; Hard","DeepSeek-R1-32B","English; 4-shot; non-thinking","https://arxiv.org/html/2610.03329v1#A2.T5","Random; 4-shot; non-thinking","Qwen3.5-VL-9B","rubric-conditioned scoring","https://arxiv.org/html/2610.05010v1#S5.T1","PortraitCraft-4B","holistic-only scorer","ArtiMuse","UniPercept","HumanAesExpert-8B","Q-Align","PortraitAes","Hosted API; baseline","https://arxiv.org/html/2610.02617v1#S4.T2","Qwen 3 Coder 480B","Llama 3 405B","Qwen 2.5 14B","Self-Trained; SFT","Self-Trained; + PPO w/ Vision agent","Self-Trained; + VisRL (Ours)","MIMO 7B","https://arxiv.org/html/2610.02617v1#S4.T3","MOMENT-S","full fine-tuning","https://arxiv.org/html/2610.06450v1#S4.T3","MOMENT-B","BRANT","PatchTST","TimesFM","Chronos-2-S","Chronos-2-B","Lag-Llama","UniTS","new-user generalization","Nemotron Streaming 0.6B","Nemotron (Base-LA13)","https://arxiv.org/html/2610.06587v1#S5.T1","Nemotron (FT-LA13)","Nemotron (Base-LA6)","Nemotron (FT-LA6)","Qwen3-ASR-1.7B Offline","Qwen3-ASR-1.7B Chunking","Parakeet-TDT-0.6B","Parakeet FT Offline","Parakeet Base Offline","Whisper Turbo","Distil-Whisper","Cohere Offline","https://arxiv.org/html/2610.06406v1#S5.T1","RepoGraph","EBG","Non-thinking; three seeds; temperature 1; top-p 1","https://arxiv.org/html/2609.24259v1#S2.T1","Source-marked † configuration; English audio","https://arxiv.org/html/2609.24812v1#S4.T1","GPT Realtime 2.1 (xhigh)","GPT Realtime 2.1 (medium)","GPT Audio 1.5","Default configuration; English audio","GPT Audio","GPT Audio Mini","Qwen3-Omni-30B","MiMo-Audio-7B","Qwen2-Audio-7B","Source-marked † configuration; Mandarin audio","https://arxiv.org/html/2609.24812v1#A2.T4","Default configuration; Mandarin audio","Single-session trained","https://arxiv.org/html/2609.22064v1#S4.T1","MLP","GRU","CNN","CEBRA","NDT","POYO","POYO+","Task-supervised pretraining + finetuning","POSSM","NDT-Stitch","Task-unsupervised pretraining + finetuning","MtM","Published direct-model protocol","https://arxiv.org/html/2609.24921v1#S4.T2","RAG","Tongyi-DR-30B-A3B","Published deep-research agent","DR-Tulu-8B","video track; published Arena protocol","https://arxiv.org/html/2609.24308v1#S3.T2","JoyAI-Echo","Alaya-EVOKE","Lingbot-World-v2","Cosmos3","Yume-1.5","Lyra 2.0","MatrixGame-3.0","MatrixGame-2.0","Open-Oasis","spatial track; published Arena protocol","https://arxiv.org/html/2609.24308v1#S4.T7","Marble","HYWorld-2.0","HYWorld-1.0","Matrix-3D","WorldGen","FlashWorld","Lyra 1.0","embodied track; published Arena protocol","https://arxiv.org/html/2609.24308v1#S5.T14","Grok Imagine Video 1.5","HappyHorse 1.1","Kling 3","Cosmos 3","Sora 2","Internal research gateway; closed book; shell/Python; 91 attempts; original ID internal/GLM-5.3-Flash","https://github.com/vmalguy/rootme-gym/blob/dfdfa4f1598e6a4c95d7d06669cedefaf0e1344f/results/leaderboard.md","Published provider route; closed book; shell/Python; 74 attempts; original ID Qwen3.6-27B","Internal research gateway; variant b; closed book; shell/Python; 92 attempts; original ID internal/Qwen3.8-2.4T-A95B-variant-b","Internal research gateway; variant a; closed book; shell/Python; 93 attempts; original ID internal/Qwen3.8-2.4T-A95B-variant-a","Published provider route; closed book; shell/Python; 84 attempts; original ID Qwen3.8-27B","Internal research gateway; closed book; shell/Python; 82 attempts; original ID internal/DeepSeek-V4-Flash","Internal research gateway; closed book; shell/Python; 88 attempts; original ID internal/GLM-5.3","Internal research gateway; closed book; shell/Python; 86 attempts; original ID internal/Qwen3.8-Flash-Next","Internal research gateway; closed book; shell/Python; 84 attempts; original ID internal/DeepSeek-V4.1-Flash","Published provider route; closed book; shell/Python; 77 attempts; original ID Qwen3.5-397B-A17B","Published provider route; closed book; shell/Python; 71 attempts; original ID gpt-oss-20b","Qwen3-Coder-30B-A3B-Instruct","Published provider route; closed book; shell/Python; 87 attempts; original ID Qwen3-Coder-30B-A3B-Instruct","Published provider route; closed book; shell/Python; 69 attempts; original ID Qwen3.5-9B","Published provider route; closed book; shell/Python; 79 attempts; original ID Qwen3-32B","Meta-Llama-3_3-70B-Instruct","Published provider route; closed book; shell/Python; 82 attempts; original ID Meta-Llama-3_3-70B-Instruct","Published provider route; closed book; shell/Python; 90 attempts; original ID Qwen2.5-VL-72B-Instruct","Published provider route; closed book; shell/Python; 90 attempts; original ID gpt-oss-120b","Published provider route; closed book; shell/Python; 84 attempts; original ID Mistral-Small-3.2-24B-Instruct-2506","Published provider route; closed book; shell/Python; 69 attempts; original ID Mistral-Nemo-Instruct-2407","Published provider route; closed book; shell/Python; 88 attempts; original ID Mistral-7B-Instruct-v0.3","Published common task/tool protocol","https://arxiv.org/html/2608.08814v1#S5.T2","Centralized; image observation; full 897-instance set","https://arxiv.org/html/2608.28266v1#Sx4.T1","Cascade R-CNN","Shared CTDD split; source official baseline implementation","https://arxiv.org/html/2608.28070v1#S4.T1","Faster R-CNN","CenterNet","ATSS","RetinaNet","Dynamic R-CNN","FoveaNet","ETDNet","Conditional DETR","RF-DETR","RT-DETRv2","YOLOv11n","CF-YOLO","Lucy Edit (arXiv 2025)","Input conditioning: T","https://arxiv.org/html/2608.20107v1#S5.T2","VACE (ICCV 2025)","Input conditioning: T+M","CoCoCo (AAAI 2025)","Gen. Omnimatte (CVPR 2025)","ProPainter (ICCV 2023)","Input conditioning: M","DiffuEraser (arXiv 2025)","MiniMax (NeurIPS 2025)","ROSE (NeurIPS 2025)","OmnimatteZero (SIGGRAPH 2025)","EffectErase (CVPR 2026)","Qwen/Qwen2.5-1.5B-Instruct","Anchor-free control; external suite","https://arxiv.org/html/2608.14320v1#A1.T4","Qwen/Qwen2.5-3B-Instruct","Qwen/Qwen2.5-7B-Instruct","meta-llama/Llama-3.2-1B-Instruct","meta-llama/Llama-3.2-3B-Instruct","meta-llama/Llama-3.1-8B-Instruct","google/gemma-3-1b-it","google/gemma-3-4b-it","allenai/OLMo-2-1124-13B-Instruct","allenai/OLMo-2-0325-32B-Instruct","openai/gpt-5.4-mini","anthropic/claude-haiku-4.5","google/gemini-2.5-flash","x-ai/grok-3-mini-beta","Anchor-free control; icl suite","https://arxiv.org/html/2608.14320v1#A1.T6","Anchor-free control; rag suite","https://arxiv.org/html/2608.14320v1#A1.T7","All runs; common 180-task grid; two prompt conditions; two judges","https://arxiv.org/html/2608.30724v1#S4.T1","Common 300-task set; one attempt","https://arxiv.org/html/2608.09296v1#Sx8.T2","Common 120-task set; one attempt","Common document image and visual reconstruction protocol","https://arxiv.org/html/2608.17550v1#S4.T4","Qwen3.5-Flash","Zero-shot Python; finance-normal","https://arxiv.org/html/2608.26757v1#S5.T2","Zero-shot Python; academic-normal","Zero-shot Python; finance-long","Zero-shot Python; academic-long","Trained graph/tabular baseline; five balanced mini-test subsets","https://arxiv.org/html/2608.15177v1#A1.T5","GCN","GraphSAGE","R-GCN","HAN","SeHGNN","HGT","ConsisGAD","PMP","GAAP","Unspecified LLM backbone","Target-only LLM-ZS","Target-only LLM-ICL-4","https://arxiv.org/html/2608.15177v1#A1.T6","VNet","https://arxiv.org/html/2608.30768v1","nnU-Net","UNETR","SwinUNETR","MedNeXt","SegMamba","IVNet","clDice","Topology-aware segmentation variant; not counted as an extra architecture","DSCNet","ADTL-Net","GPT-5 nano","Reasoning enabled; five prompt-template mean","https://arxiv.org/html/2609.03370v1#S3.T3","Gemini 3.1 Flash-Lite","Gemma 4 E2B","Gemma 4 E4B","Gemma 4 31B","Reasoning disabled; five prompt-template mean","LLM-jp 4 8B","Qwen3 Swallow 8B","Qwen3 Swallow 32B","SimpleCNN-N2N","Modalities: Remote Sensing (RS); published common split; SimpleCNN-N2N","https://arxiv.org/html/2609.03804v1#S3.T1","SimpleCNN-Frozen","Modalities: RS; published common split; SimpleCNN-Frozen","CLIP ViT-B/16","Modalities: RS; published common split; CLIP ViT-B/16","DINO v2-Small","Modalities: RS; published common split; DINO v2-Small","DINO v3-Base-LVD","Modalities: RS; published common split; DINO v3-Base-LVD","DINOv3-SAT-Fixed","Modalities: RS; published common split; DINOv3-SAT-Fixed","Modalities: Numerical; published common split; MLP","GBDT","Modalities: Numerical; published common split; GBDT","BERT Chinese","Modalities: Text; published common split; BERT Chinese","CLIP Text OpenAI","Modalities: Text; published common split; CLIP Text OpenAI","DINOv3-SAT+ BERT Chinese","Modalities: RS + Text; published common split; DINOv3-SAT+ BERT Chinese","DINOv3-SAT + MLP","Modalities: RS + Numerical; published common split; DINOv3-SAT + MLP","BERT Chinese + MLP","Modalities: Text + Numerical; published common split; BERT Chinese + MLP","Full CLIP (Three-modal)","Modalities: RS + Text + Numerical; published common split; Full CLIP (Three-modal)","CLIP (BERT Enhanced)","Modalities: RS + Text + Numerical; published common split; CLIP (BERT Enhanced)","MCGC (Ours)","Modalities: RS + Text + Numerical; published common split; MCGC (Ours)","Zero-shot; original Korean; published factuality evaluator","https://arxiv.org/html/2609.04598v1#S4.T3","MedGemma-27B","A.X-4.0-VL-Light","HCX-SEED-Vision-3B","Gemma-2-27B","MedGemma-27B-Text","A.X-4.0-Light","HCX-SEED-Text-1.5B","Published single-call reward prediction; lexical paraphrases","https://arxiv.org/html/2609.05401v1#S3.T2","Gemini2.5-flash-lite","Gemini2.5-flash","Gemini3-flash","Claude-haiku-4.5","Claude-sonnet-4.6","Qwen3-vl-8B","Qwen3-vl-32B","Qwen3-vl-235B-22A","Llama4-scout","RR-4B","RR-8B","Published single-call reward prediction; syntactic paraphrases","Published single-call reward prediction; action-goal paraphrases","DynamicStereo","Released weights; zero-shot","https://arxiv.org/html/2609.06809v1#S5.T3","BiDAStereo","StereoAnyVideo","RAFT-Stereo","IGEV-Stereo","FoundationStereo","Signed adaptation; frozen matching features","Signed adaptation; full training","Qwen15_MoE","Default chat; greedy decoding; temperature 0; judge Qwen2.5-7B-Instruct","https://arxiv.org/html/2609.05117v1#S4.T3","Gemma2B","Mistral7B","GPT-J","Chat-glm3","Default chat; greedy decoding; temperature 0; judge Llama-3.1-8B-Instruct","https://arxiv.org/html/2609.05117v1#S4.T4","Chronos2","TSFM; official shared target/availability protocol","https://arxiv.org/html/2609.15087v1#A4.T15","TimesFM-2.5-200m","Timer-S1","Moirai-2.0-R-small","Sundial-base-128m","Toto-2.0-2.5B","Aurora","MMFM; official shared target/availability protocol","VoT","Ctx-DS; official shared target/availability protocol","Toto-2.0-4m","TaTS","DLinear","Num-DS; official shared target/availability protocol","TimesNet","TESS","GPT4MTS","SeasonalNaive","Stat; official shared target/availability protocol","TimeMixer","CALF","Moirai-1.0-R-large","Timer-base-84m","TimeCMA","TimeLLM","AutoTimes","MovingAverageSeasonal","LastValueNaive","ChatTime-1-7B-Base","General LLM; official shared target/availability protocol","MeanForecast","TimeMoE-200M","ZeroForecast","Qwen3.6-27B-Prompt","RelateAnything","GT regions; graph constrained; A3 matcher tau 0.60","https://arxiv.org/html/2609.12552v1#S6.T5","ROBIN-3B","Captioned demonstration; hidden scalar rewards; fixed initial state; single rollout per task","https://arxiv.org/html/2609.15683v1#S4.T1","Seed-2.0-Lite","DeepSeek-V4F-V","MiMo-v2.5","Read-only; no network/writes/code execution; 15 calls; 3 runs; reasoning xhigh","https://arxiv.org/html/2609.15939v1#S4.T2","Read-only; no network/writes/code execution; 15 calls; 3 runs; reasoning default","Read-only; no network/writes/code execution; 15 calls; 3 runs; published serving setting","CodeScout-14B","Granite-4.0-350M","Granite-4.0-Micro","Granite-4.0-1B","Antares-3B","Antares-1B","Antares-350M","https://arxiv.org/html/2609.15939v1#S4.T3","Shared CoT; published level-specific steps; standard mode","https://arxiv.org/html/2609.12345v1#S1.T2","Gemini 3.1 flash","Gemini 3.1 pro","Qwen 3 32B","GLM 4.5 Air","GLM 4.6","Shared CoT; published level-specific steps; thinking mode","DeepSeek-V4-pro","CLaMP3","Frozen official checkpoint; text-aligned audio encoder","https://arxiv.org/html/2609.16090v1#S4.T3","MERT-95M","Frozen official checkpoint; audio-only encoder","MuQ","MuQ-MuLan","MuQ-MuLan Fusion","Supervised Fusion; 3-seed mean","MuQ-MuLan + Text","Frozen MuQ-MuLan; audio/text blend alpha 0.2","WAVE-7B","Frozen official checkpoint; instruction-aware retrieval","OmniRet","caption-30s;1000questions;self-caption memory or model-specific direct-frame cap","https://arxiv.org/html/2609.17688v1#S5.T3","caption-60s;1000questions;self-caption memory or model-specific direct-frame cap","direct-video;1000questions;self-caption memory or model-specific direct-frame cap","direct;fixed235-question accuracy denominator;paper-defined missing-answer policy","https://arxiv.org/html/2609.11243v1#S3.T2","Intern-S2","tool-use;fixed235-question accuracy denominator;paper-defined missing-answer policy","Seed-2.1-Evolving","Agent: Nanobot;GLM-5.1 judge;weighted criteria;equal task mean","https://arxiv.org/html/2609.07784v1#S4.T3","Agent: OpenCode;GLM-5.1 judge;weighted criteria;equal task mean","Wan2.1 T2V-1.3B","t2v; 3 fps; at most 30 frames; Qwen3.8-27B correspondence judge","https://arxiv.org/html/2610.13412v1#S7.T1","Wan2.2 T2V-A14B","Wan2.2 TI2V-5B","LongCat-Video","HunyuanVideo-1.5","Kandinsky-5 Pro","ref2v; 3 fps; at most 30 frames; Qwen3.8-27B correspondence judge","Edge-only","https://arxiv.org/html/2610.17816v1#S5.T2","Cloud-only","No-Skill; 12-turn limit; common task and submission interface","https://arxiv.org/html/2610.03564v1#S6.T5","gemini-3.1-flash-lite","claude-sonnet-4.6","grok-4","Curated; 12-turn limit; common task and submission interface","Self-Gen; 12-turn limit; common task and submission interface","Qwen2.5-Math-7B-Instruct","Published single-model multimodal setting","https://arxiv.org/html/2610.26943v1#S4.T3","Qwen2.5-Math-72B-Instruct","Qwen3-Omni-30B-A3B-Thinking","DeepSeek-v3.2-thinking","Agent: Claude Code; high effort; three trials","https://arxiv.org/html/2610.25834v1#S4.T3","Agent: Codex; high effort; three trials","ESCounts","real; published k=1–20 test protocol; no linear-probe readout","https://arxiv.org/html/2610.18844v1#S4.T1","RepNet","VideoChat-R1.5-7B","LongVILA-R1-7B","synthetic; published k=1–20 test protocol; no linear-probe readout","DeepDDI","S2; 800-drug subset; three seeds; Published method-specific training","https://arxiv.org/html/2610.05590v1#A2.T22","SSI-DDI","DSN-DDI","HDN-DDI","EmerGNN","TIGER","MKG-FENN","TextDDI","S2; 800-drug subset; three seeds; Llama-3.2-1B fine-tune; P4 prompt","Published common multimodal evaluation setting","https://arxiv.org/html/2610.12402v1#S4.T1","Doubao 2.0 Pro","VST-7B-RL","SenseNova-SI-1.2","SpaceR","SpatialLadder","Spatial-MLLM-v1.1","Zero-shot; All-Pref; LIBERO-SinglePref; published task success condition","https://arxiv.org/html/2610.22372v1#S4.T2","pi0-fast","GR00T-N1.6","SmolVLA","Radar-only; 30-slot xyz and velocity; program-disjoint; three seeds","https://arxiv.org/html/2610.25869v1#S5.T2","LSTM","Attn-TCN-BiGRU","BLAC-like","B2L-lite","Agent: mini-SWE-agent; effort max; best observed effort; three runs","https://arxiv.org/html/2609.26777v1#S5.T3","Agent: mini-SWE-agent; effort xhigh; best observed effort; three runs","DeepSeek V4 Flash (0731)","Agent: mini-SWE-agent; effort high; best observed effort; three runs","Inkling S","AVI-Edit","Blinded human listening; published four-dimension rubric","https://arxiv.org/html/2609.25864v1#S4.T2","InstructAV2AV","ZETA","Audio-Omni","UNISON","SAM Audio","TV-AudioRemover","Signature MLP","Signature vector; Supervised; three seeds","https://arxiv.org/html/2609.25569v1#S5.T4","Graph–signature fusion","Graph + signature; Supervised; three seeds","Dynamic graph-GRU","Graph tensor; Supervised; three seeds","Graph-summary text; LLM zero-shot; three seeds","Non-thinking; same tool/cost contract","https://arxiv.org/html/2609.27385v1#S3.T2","LLaVA-OneVision-Qwen2-0.5B","Full benchmark; source default visual inputs","https://arxiv.org/html/2609.25841v1#S5.T1","VST-7B-SFT","SpaceR-7B","MetricReasoner","DP (0.3B)","Published real-time closed-loop protocol; native model inference pipeline","https://arxiv.org/html/2609.25689v1#S4.T2","ACT (51.7M)","SmolVLA (0.45B)","X-VLA (0.9B)","π 0.5 \\pi_{0.5} (4B)","GR00T N1.7 (3B)","FastWAM ( ∼ \\sim 6B)","Published task-specific semantic rubric; common scenario suite","https://arxiv.org/html/2609.26175v1#S4.T3","Qwen 3.7-Max","Qwen 3-30B-A3B-Instruct-2507","Ministral 3-14B-Instruct-2512","Intern-S2-Preview-35B","Internvl3.5-241b-a28b","Gemma 4-31b-it","Gemma 4-26b-A4B-it","Maximum supported output budget; 11 tasks × 5 levels × 5 seeds","https://arxiv.org/pdf/2609.35581v1#page=3","Gemini Flash Lite","Non-thinking; Same ridehailing tools; three trials per task; final judge-panel success gate","https://arxiv.org/pdf/2609.17985v1#page=19","Same ridehailing tools; three trials per task; final judge-panel success gate","MiMo-v2.5-Pro","Granite-4.1-8B","Nemotron-3-Nano-30B-A3B","https://arxiv.org/pdf/2609.36488v1#page=5","Qwen-3.5-27B","Qwen-3-VL-32B","Agent: AdaptiveAgent; two retrieved trajectories; GPT-5-nano preference extractor","Common multi-view video baseline; full 1,323-question set","https://arxiv.org/pdf/2609.30952v1#page=5","LLaVA-onevision-7B","LLaVA-onevision-72B","Video-LLaVA-7B","MiniCPM-o-2.6-8B","Gemini-3.0-flash","Gemini-3.0-pro","Gemini-3.1-pro","256-step input; H=1; validation-MSE checkpoint","https://arxiv.org/pdf/2609.38719v1#page=7","NLinear","TCN","iTransformer","Autoformer","256-step input; H=16; validation-MSE checkpoint","256-step input; H=32; validation-MSE checkpoint","Paper full digital track; released subset has different coverage","https://arxiv.org/html/2608.22842v1#S4.T6","FinixDoc-VL","DeepSeek-OCR-2","FireRed-OCR","PaddleOCR-VL-1.5","GLM-OCR","Youtu-Parsing","Dots.OCR","MinerU 2.5","Paper full photo track; released subset has different coverage","https://arxiv.org/html/2608.22842v1#S4.T7","Dianjin-R1-7B","Fixed Domain Knowledge task contracts; full 33-configuration paper panel","https://arxiv.org/html/2608.25325v1#A5.T8","FinR1-7B","Ling-2.6-Flash","Ling-2.6-1T","Nemotron-3-Nano","Qwen3-Next-80B","Gemini-3-Flash-preview","Qwen3-VL","Model size: 4B; patient-disjoint readout split","https://arxiv.org/html/2608.12904v1#S4.T3","Model size: –; patient-disjoint readout split","MiniCPM","Model size: 9B; patient-disjoint readout split","Model size: 14B; patient-disjoint readout split","M3D-LaMed","CT-CHAT","Model size: 8B; patient-disjoint readout split","OmniCT","Model size: 3B; patient-disjoint readout split","HounsWorld","EDTalk ( Tan et al., 2024 )","Audio-driven; common identity image/audio/reference; zero-shot","https://arxiv.org/html/2608.26517v1#S5.T5","V-Express ( Wang et al., 2024 )","EchoMimicV3 ( Meng et al., 2026 )","AniTalker ( Liu et al., 2024 )","Hallo2 ( Cui et al., 2025 )","Ditto ( Li et al., 2025 )","Sonic ( Ji et al., 2025 )","LatentSync ( Li et al., 2024a )","Part I authorized; image-based four-stage visual cue understanding","https://arxiv.org/html/2608.17600v1#S3.T1","π 0.5","Action-Sketcher","InstructVLA","Xiaomi-Robotics-0","Common PDF page corpus; average over EN/FR/DE/LB queries","https://arxiv.org/html/2608.21714v1#S4.T2","Qwen3-Embedding-0.6B","jina-embeddings-v4","colSmol-256M","colSmol-500M","colpali-v1.3","colqwen2.5-v0.2","colnomic-embed-multimodal-3b","Paired clinical finding task; image only","https://arxiv.org/html/2608.24118v1#S3.T2","Phi-4-multimodal","NV-Reason-CXR-3B","Claude-Opus-4-7","Paired clinical finding task; reliable note","Paired clinical finding task; misleading note","Common 625 cases; Top-1 primary diagnosis; ICD-11 alignment","https://arxiv.org/html/2608.22323v1#S3.T2","LLaMA-4-Scout","LLaMA-4-Maverick","Qwen3-VL-30B-Thinking","https://arxiv.org/html/2608.20153v3#S4.T4","Anthropic Claude Opus 4.8","Common 180 tasks; no context; strict end-to-end verifier","https://arxiv.org/html/2608.23200v4#A1.T7","Anthropic Claude Sonnet 4.6","Google Gemini 3.1 Flash-Lite Preview","Google Gemini 3.1 Pro Preview","Qwen/Qwen3-Coder-30B-A3B-Instruct","Qwen/Qwen3.5-397B-A17B","GPT-5.6-Sol (xhigh); common TextIn parsed context","https://arxiv.org/html/2608.15064v1#S3.T5","MiniMax-M2.5 (think); common TextIn parsed context","GLM-5.2 (think); common TextIn parsed context","Kimi K2.6 (w/o think); common TextIn parsed context","Qwen3.5-397B (w/o think); common TextIn parsed context","Qwen3.5-397B (think); common TextIn parsed context","Qwen3.5-35B (w/o think); common TextIn parsed context","Qwen3.5-9B (w/o think); common TextIn parsed context","Nano-Banana-2-Lite","Track A: instruction; fixed-seed bench-v1.0","https://arxiv.org/html/2608.24107v1#S5.T3","BAGEL-7B","FLUX-Kontext","InstructPix2Pix","Track B: mask + instruction; fixed-seed bench-v1.0","Qwen-Image-Edit-inpaint","FLUX-Kontext-inpaint","SD3.5-inpaint","OmniGen2-inpaint","FLUX.1-Fill","SDXL-inpaint","Track C: mask + reference; fixed-seed bench-v1.0","FLUX-Kontext-ref","Paint-by-Example","Qwen-Image-Edit-ref","OmniGen2-ref","Full benchmark; objective mastery; exact perfect-score gate","https://arxiv.org/html/2608.25398v1#S3.T3","GLM-4.6v","Qwen3-VL-235B-a22b","Full benchmark; open-ended derivation mastery; exact perfect-score gate","DevEco Code; common 153-task snapshot; source model configuration","https://arxiv.org/html/2608.16022v1#S5.T4","Qwen3.8-Max-Preview","Params: 7B; common data; asynchronous chunk 8/horizon 2","https://arxiv.org/html/2608.14379v1#S5.T1","Params: 4B; common data; asynchronous chunk 8/horizon 2","PUMA","DynamicVLA","Params: 0.5B; common data; asynchronous chunk 8/horizon 2","VLA-Adapter","Params: 1B; common data; asynchronous chunk 8/horizon 2","ReflexVLA","RAD","RAD (rules, no BCE); ten seeds; common relational task","https://arxiv.org/html/2608.23468v1#S7.T4","RAD (rules, BCE); ten seeds; common relational task","RDL","RDL weighted ( Fey et al., 2024 ); ten seeds; common relational task","RDL unweighted ( Fey et al., 2024 ); ten seeds; common relational task","AE","AE+Rank ( Sakurada and Yairi, 2014 ); ten seeds; common relational task","DRL","DRL ( Ye et al., 2025 ); ten seeds; common relational task","AE ( Sakurada and Yairi, 2014 ); ten seeds; common relational task","Deep SVDD","Deep SVDD ( Ruff et al., 2018 ); ten seeds; common relational task","Isolation Forest","Isolation Forest ( Liu et al., 2008 ); ten seeds; common relational task","Supervised MLP","Supervised MLP; ten seeds; common relational task","VGGT- Ω","Published common clip/annotation protocol; frame motion recognition","https://arxiv.org/html/2608.10932v1#S5.T2","Cam-Motion-7B","CamDistill -4B","CamInject -4B","CamDistill -8B","CamInject -8B","Published common clip/annotation protocol; segment motion detection","BEVDet","Camera only; common 215/75 sequence split","https://arxiv.org/html/2608.30657v1#S4.T2","BEVFormer","BEVDepth","TPVFormer","SurroundOcc","CONet","STCOcc","SparseOcc","GaussianFormer","ProSD-Occ","Common scientific instruction set; instruction-level all-required-constraints gate","https://arxiv.org/html/2608.25973v1#S3.T3","Grok-4-Fast","SL-Module","Published trained model; nature perspective; top 15% budget","https://arxiv.org/html/2608.29577v1#S5.T3","PGL-SUM","CSTA","SumDiff","Aha †","Zero-shot; nature perspective; top 15% budget","Qwen3-VL-235B †","TRINITY shared-private experts","Published trained model; emotion perspective; top 15% budget","Zero-shot; emotion perspective; top 15% budget","Published trained model; event perspective; top 15% budget","Zero-shot; event perspective; top 15% budget","L3 common question set; bm25 access","https://arxiv.org/html/2608.10679v2#S4.T1","L3 common question set; dense-rag access","L3 common question set; graphrag access","Eagle2.5-8B","Best published frame-budget/sampling sweep; published recommended prompt","https://arxiv.org/html/2609.09528v1#S5.T2","Motion-o (7B) ‡","Best published frame-budget/sampling sweep; motion-tuned checkpoint","Molmo2-8B [ 2 , 5 ]","Best published frame-budget/sampling sweep; reasoning disabled; provisional n=60","Gemini 3.1 Pro","GPT-5.6 Luna ¶","Best published frame-budget/sampling sweep; 720x1280; detail high","Zero-shot; common CoT prompt","https://arxiv.org/html/2609.07135v1#S5.T2","VeBrain-7B","InternVL3.5-4B-HF","InternVL3.5-8B-HF","GLM-4.1V-9B-Base","GLM-4.1V-9B-Think","Vgent scaffold; Qwen3-VL-4B","Nutri-Vgent scaffold; Qwen3-VL-4B","Vgent scaffold; Qwen3-VL-8B","Nutri-Vgent scaffold; Qwen3-VL-8B","EmbeddingGemma-300M","Fixed Combined+LLM-judged relevance; full corpus","https://arxiv.org/html/2609.08887v1#S4.T3","mDenseOn","Nemotron-3-Embed-1B","Nemotron-3-Embed-8B","pplx-embed-v1-0.6b","pplx-embed-v1-4b","Qwen3-Embedding-4B","Qwen3-Embedding-8B","voyage-4-nano","mLateOn","pplx-embed-v1-late-0.6B","Fixed Combined+LLM-judged relevance; subsampled corpus","Claude Code scaffold; highest reasoning; maximum context; fixed 8 CPUs/32 GiB/H20","https://arxiv.org/html/2609.10226v1#Sx3.T2","Codex scaffold; highest reasoning; maximum context; fixed 8 CPUs/32 GiB/H20","DeepSeek V4Pro","GPT-5.3 Codex Spark / high","Codex runtime; GPT-5.3 Codex Spark / high","https://arxiv.org/html/2609.09849v1#S4.T2","GPT-5.4 mini / medium","Codex runtime; GPT-5.4 mini / medium","GPT-5.4 / medium","Codex runtime; GPT-5.4 / medium","GPT-5.5 / low","Codex runtime; GPT-5.5 / low","GPT-5.5 / medium","Codex runtime; GPT-5.5 / medium","GPT-5.5 / high","Codex runtime; GPT-5.5 / high","GPT-5.5 / xhigh","Codex runtime; GPT-5.5 / xhigh","GPT-5.6 Luna / high","Codex runtime; GPT-5.6 Luna / high","GPT-5.6 Sol / high","Codex runtime; GPT-5.6 Sol / high","Cursor runtime; Claude Opus 4.8","Cursor runtime; Claude Sonnet 5","Cursor runtime; Composer 2.5","Cursor runtime; Gemini 3.1 Pro","Cursor runtime; Gemini 3.5 Flash","Cursor runtime; GPT-5.4","Cursor runtime; GPT-5.4 nano","Cursor runtime; Grok 4.5","DiffusionGemma","OpenCode runtime; DiffusionGemma","OpenCode runtime; DeepSeek V4 Flash","OpenCode runtime; MiniMax M2.5","OpenCode runtime; Qwen3.6-35B","OpenCode runtime; Qwen3 8B","OpenCode runtime; Qwen3 14B","English prompt; 10-shot; normalized output","https://arxiv.org/html/2609.10153v1#S5.T6","Fanar-2-27B-Instruct","Jais-2-70B-Chat","ALLaM-7B-Instruct-preview","Jais-2-8B-Chat","English prompt; zero-shot; normalized output","CNN-IQA","Common source-grouped 80/20 split; 3-seed mean","https://arxiv.org/html/2609.14495v1#S5.T1","DBCNN","WaDIQaM","MUSIQ","CLIP-IQA","CLIP-IQA+","QualiClip","MCIQA-CS","MCIQA-SCM","MCIQA-GN","Claude Code framework; fixed turn-bucket aggregation; common constraint checklist","https://arxiv.org/html/2609.14992v1#S4.T2","OpenCode framework; fixed turn-bucket aggregation; common constraint checklist","https://arxiv.org/html/2609.14992v1#A6.T5","Common image/point panel; subject-centered captioning","https://arxiv.org/html/2609.08300v1#S4.T2","Gemini-3-Flash ( Comanici et al., 2025 )","Gemini-3.1-Pro ( Comanici et al., 2025 )","MiMo-VL-7B-SFT ( Yue et al., 2025 )","MiniCPM-V-4.5 ( Yu et al., 2025 )","Ovis2.5-9B ( Lu et al., 2025 )","InternVL3.5-8B ( Wang et al., 2025 )","InternVL3.5-14B ( Wang et al., 2025 )","Qwen3-VL-8B-Instruct ( Bai et al., 2025 )","Qwen3-VL-8B-Thinking ( Bai et al., 2025 )","Qwen3-VL-32B-Instruct ( Bai et al., 2025 )","Qwen3-VL-235B-A22B-Instruct ( Bai et al., 2025 )","H-SPACE-GRPO","Published checkpoint 2025-10; –; common inverse-prompt and replay protocol","https://arxiv.org/html/2609.08079v1#S4.T2","Published checkpoint 2024-08; –; common inverse-prompt and replay protocol","OmniVinci","Published checkpoint 2025-10; 7B; common inverse-prompt and replay protocol","Published checkpoint 2025-01; 72B; common inverse-prompt and replay protocol","Published checkpoint 2025-09; 8B; common inverse-prompt and replay protocol","Published checkpoint 2025-09; 30B; common inverse-prompt and replay protocol","Published checkpoint 2024-10; 7B; common inverse-prompt and replay protocol","Published checkpoint 2025-09; 4B; common inverse-prompt and replay protocol","Published checkpoint 2025-03; 32B; common inverse-prompt and replay protocol","Published checkpoint 2025-01; 7B; common inverse-prompt and replay protocol","Published checkpoint 2025-11; 9B; common inverse-prompt and replay protocol","VideoLLaMA3","Keye-VL","Published checkpoint 2025-10; 8B; common inverse-prompt and replay protocol","Published checkpoint 2025-01; 3B; common inverse-prompt and replay protocol","InternVL3","Published checkpoint 2025-04; 8B; common inverse-prompt and replay protocol","Published checkpoint 2024-12; 8B; common inverse-prompt and replay protocol","PyVision-Video","Published checkpoint 2025-08; 7B; common inverse-prompt and replay protocol","LongVideoAgent","Published checkpoint 2025-05; 7B; common inverse-prompt and replay protocol","zero-native; fixed 600-item test fold; macro over six language varieties","https://arxiv.org/html/2609.09964v1#S5.T2","Gemma-4-4B","zero-romanized; fixed 600-item test fold; macro over six language varieties","few-native; fixed 600-item test fold; macro over six language varieties","few-romanized; fixed 600-item test fold; macro over six language varieties","Muse Glimmer 30B 3; source-specific serving config Appendix B.4; single inference run","https://arxiv.org/html/2609.10016v1#S3.T2","Qwen3.8-27B 3; source-specific serving config Appendix B.4; single inference run","Qwen3.6-27B 3; source-specific serving config Appendix B.4; single inference run","Qwen3.5-27B base; source-specific serving config Appendix B.4; single inference run","GPT-5.6 luna (medium) 5; source-specific serving config Appendix B.4; single inference run","GPT-5.4 full","GPT-5.4 full (xhigh); source-specific serving config Appendix B.4; single inference run","Qwen3.5-35B-A3B base; source-specific serving config Appendix B.4; single inference run","GPT-5.6 sol (xhigh) 5; source-specific serving config Appendix B.4; single inference run","Qwen3.5-27B + PEFT (n=2); source-specific serving config Appendix B.4; PEFT seed mean","Qwen3.5-4B + PEFT (n=2); source-specific serving config Appendix B.4; PEFT seed mean","Qwen3.5-9B + PEFT (n=2); source-specific serving config Appendix B.4; PEFT seed mean","GPT-5.4 full (high); source-specific serving config Appendix B.4; single inference run","Qwen3.5-9B base; source-specific serving config Appendix B.4; single inference run","Mistral Small 2603 1; source-specific serving config Appendix B.4; single inference run","GPT-5.4 full (medium); source-specific serving config Appendix B.4; single inference run","Qwen3.5-4B base; source-specific serving config Appendix B.4; single inference run","GLM-4.7-Flash 3 4; source-specific serving config Appendix B.4; single inference run","GPT-5.4-nano; source-specific serving config Appendix B.4; single inference run","GPT-5.4-mini; source-specific serving config Appendix B.4; single inference run","Ministral 8B 2512; source-specific serving config Appendix B.4; single inference run","Gemma 4 26B-A4B; source-specific serving config Appendix B.4; single inference run","Qwen3.5-2B + PEFT (n=3); source-specific serving config Appendix B.4; PEFT seed mean","Qwen3.5-2B base; source-specific serving config Appendix B.4; single inference run","Qwen3.5-0.8B base; source-specific serving config Appendix B.4; single inference run","Mistral Nemo 12B","Mistral Nemo 12B; source-specific serving config Appendix B.4; single inference run","GRU seq2seq","Published exterior-context configuration; GRU seq2seq","https://arxiv.org/html/2609.08117v1#A9.T7","siMLPe p","Pose-only; siMLPe p [ 18 ]","Transformer ED","Published exterior-context configuration; Transformer ED","Transformer ED (enriched)","Published exterior-context configuration; Transformer ED (enriched)","Transformer-L p","Pose-only; Transformer-L p","Transformer-XL p","Pose-only; Transformer-XL p","CVAE","Published exterior-context configuration; CVAE","DDPM p","Pose-only; DDPM p","DDPM p (enriched)","Pose-only; DDPM p (enriched)","DDPM","Published exterior-context configuration; DDPM","DDPM (enriched)","Published exterior-context configuration; DDPM (enriched)","AR-LM p","Pose-only; AR-LM p [ 24 ]","AR-LM p (enriched)","Pose-only; AR-LM p (enriched)","AR-LM (enriched)","Published exterior-context configuration; AR-LM (enriched)","LLM (Llama-1B) p","Pose-only; LLM (Llama-1B) p","LLM (Llama-3B) p","Pose-only; LLM (Llama-3B) p","LLM (Qwen-VL-3B) p","Pose-only; LLM (Qwen-VL-3B) p","Full test; Base VLM; published decoding","https://arxiv.org/html/2610.27087v1#S4.T1","LLaVA-OV-1.5-4B","LLaVA-OV-1.5-8B","Qwen3.5-VL-2B","Qwen3.5-VL-4B","Qwen3.5-VL-27B","SciFigCue (Qwen3.5-VL-9B)","Full test; SciFigCue connection-cue pipeline","SciFigCue (Qwen3.5-VL-27B)","Claude-4.5-Sonnet-thinking","ReAct; public search; 42 iterations; thinking","https://arxiv.org/html/2610.26036v1#S4.T2","Claude-4.5-Sonnet-no-thinking","ReAct; public search; 42 iterations; non-thinking","Gemini-3-Pro-thinking","Gemini-3-Pro-no-thinking","GPT-5.2-thinking","Doubao-seed-1-8-251215-thinking","Doubao-seed-1-8-251215","DeepSeek-V3.2-thinking","DeepSeek-V3.2-no-thinking","Kimi-k2-thinking","Kimi-k2","HunYuan-2.0-thinking","HunYuan-2.0-instruct","Grok-4.1-thinking","Qwen-max-20250923-thinking","Qwen-max-20250923-no-thinking","GLM-4.7-thinking","GLM-4.7-no-thinking","SAMTok-CO","Official released checkpoint; native image preprocessing; core expression pairs","https://arxiv.org/html/2610.09478v1#S4.T2","X2SAM","UniPixel-7B","Sa2VA-2B","X-SAM","InstructSeg","HyperSeg-3B","PSALM","ConverSeg-3B","EVF-SAM2","SaFiRe-mixed","SSP-SAM-RefCOCO","SSP-SAM-RefCOCOg","SSP-SAM-RefCOCO+","DETRIS","PaDT-REC-3B","VATEX-Swin-B","LAVT-RefCOCO","LAVT-RefCOCOg","LAVT-RefCOCO+","CoHD","RefChess-hybrid","PP-OCRv6 (thr = 0.90 =0.90 )","Published full character test; fixed threshold; target-conditioned character verdict","https://arxiv.org/html/2610.19551v1#S4.T1","TextPecker (Qwen3-VL-8B)","TextPecker (InternVL3-8B)","TextVerifier","Published full character test; fixed threshold; target-conditioned character verdict; Stage 3","SANA-Video 2.0","t2v; 600 prompt groups; 720p; 10 seconds; fixed seed","https://arxiv.org/html/2610.26711v1#S4.T2","Wan2.1","Wan2.2","LingBot-Video","i2v; 600 prompt groups; 720p; 10 seconds; fixed seed","VL; 32 frames; paired Text and Text+Image; QA only","https://arxiv.org/html/2610.18829v1#S4.T2","Keye-VL-2.0-30B-A3B","TimeLens2-8B","VL; 64 frames; paired Text and Text+Image; QA only","Seed-2.1-Pro-260915","Seed-2.1-Turbo-260628","Seed-2.0-Pro-260215","Qwen3-Omni-30B-A3B","AV; 32 frames; paired Text and Text+Image; QA only","https://arxiv.org/html/2610.18829v1#S4.T3","MiniCPM-o-4.5-8B","VideoLLaMA2.1-7B-AV","AV; 64 frames; paired Text and Text+Image; QA only","Seed-2.0-Lite-260428","Seed-2.0-Mini-260428","LingBot-VLA-v2","ID; 106 tasks; 2 seeds x 25 episodes per task","https://arxiv.org/html/2610.25459v1#S5.T3","GR00T-N1.7","OpenDM-DM05","Xiaomi XR1","OOD; 106 tasks; 2 seeds x 25 episodes per task","https://arxiv.org/html/2610.25459v1#S5.T4","XR-1","SONIC; 234 tasks; 20 trials per seen task and 10 per unseen task or scene; equal task weights","https://arxiv.org/html/2610.30876v1#S4.T2","Fast-WAM","PRTS","Psi0","StarVLA-alpha","Diffusion Policy","SONIC; 100 seen tasks; 20 trials per task at Low/Mid/High perturbations; equal task and severity weights","https://arxiv.org/html/2610.30876v1#S4.T3","Published full-suite raw-success panel; 11 axes and five severity levels","https://arxiv.org/html/2610.18843v1#S3.T2","OFT-Plus","pi0","MolmoAct","NORA","Direct frontier control; reasoning=high; 100 episodes; shared tool/action protocol","https://arxiv.org/html/2610.30051v1#S3.T1","Gemini-4 Argon","Direct frontier control; reasoning=medium; 100 episodes; shared tool/action protocol","Direct frontier control; reasoning=low; 100 episodes; shared tool/action protocol","Shared full-video caption prompt; fixed private probes; text-only evidence judge","https://arxiv.org/html/2610.29552v1#S4.T5","TRACE-Uni","Sa2VA-8B","VideoChat2-HD","OMTG","Time-R1","TimeLens","Video-LLaMA3","Open-o3-Video","Shared full-video caption prompt; fixed private probes; text-only evidence judge; CycleGRPO","Hosted inference; common four-domain panel; level 1; 8 responses per prompt","https://arxiv.org/html/2610.11064v1#A15.T20","Hosted inference; common four-domain panel; level 2; 8 responses per prompt","Hosted inference; common four-domain panel; level 3; 8 responses per prompt","Hosted inference; 13 common reportable domain-level cells; native conditional-diversity estimator","https://arxiv.org/html/2610.11064v1#A15.T19","Gemma-4 31B","Single attempt; task plus ISA reference; native executor","https://arxiv.org/html/2609.22878v1#A3.T4","Phi-4-Reasoning","Devstral-Small-2","Qwen3-Coder 30B","Granite-4.1 8B","Phi-4 14B","Codestral 22B","DeepSeek-Coder-V2-16B","Matched choice/open reports; controlled simulation; structured track","https://arxiv.org/html/2609.23064v1#S5.T2","GLM-4.5-Air (106B-A12B)","Kimi-K2-Instruct (1T-A32B)","Qwen3.5-Plus-02-15","Matched choice/open reports; controlled simulation; visual track","Zero-shot; same pairwise A/B/T rubric prompt","https://arxiv.org/pdf/2609.32533v1#page=7","Five-epoch DPO; source-instance held-out split","Agent: published game-development framework; brief-to-game","https://arxiv.org/pdf/2609.33678v1#page=8","Agent: published game-development framework; gdd-to-game","Agent: published game-development framework; skeleton-completion","Agent: published game-development framework; bug-repair","Agent: published game-development framework; godot-to-unity","Same task-specific source protocol; fixed denominator retains invalid outputs","https://arxiv.org/pdf/2609.34854v1#page=7","LLaVA-OneVision2-8B-Instruct","MiniCPM-V-4.6","AesExpert-7B","Reasoning x-high; n=158 sessions; HC3 95% CI [13.7, 18.4] pp","https://arxiv.org/pdf/2609.28470v1#page=30","GPT-5.5 Pro","Reasoning med; n=162 sessions; HC3 95% CI [13.4, 17.9] pp","Reasoning high; n=165 sessions; HC3 95% CI [13.1, 17.8] pp","Reasoning low; n=177 sessions; HC3 95% CI [11.8, 16.7] pp","Reasoning med; n=179 sessions; HC3 95% CI [12.0, 16.2] pp","Reasoning off; n=157 sessions; HC3 95% CI [11.3, 16.0] pp","Reasoning high; n=161 sessions; HC3 95% CI [11.1, 15.9] pp","Reasoning high; n=159 sessions; HC3 95% CI [10.7, 15.5] pp","Reasoning off; n=161 sessions; HC3 95% CI [10.9, 15.0] pp","Reasoning high; n=153 sessions; HC3 95% CI [9.8, 15.4] pp","Reasoning med; n=161 sessions; HC3 95% CI [9.8, 14.5] pp","Reasoning low; n=168 sessions; HC3 95% CI [9.7, 14.5] pp","One zero-shot prediction per polymer","https://arxiv.org/pdf/2609.27036v1#page=9","Polymer Chemprop","Five-fold cross-validation; published mean test RMSE","Polymer Periodic Graph","polyBERT","Random Forest (RDKit)","Structured scores + RF; five-fold validation","https://arxiv.org/html/2609.23345v1#S4.T2","Doubao-1.5-Vision-Pro","Generated description + fixed GLM-4.5V judge","https://arxiv.org/html/2609.23345v1#S4.T3","GPT 5.4","Zero-shot; source default image-pair interface","https://arxiv.org/html/2609.23336v1#S5.T1","Gemini 2.5 Pro","Doubao-seed-1.8","VLM-R1 (Zero-shot)","UniVG-R1 (Zero-shot)","Migician (Zero-shot)","SFT baseline","Agent: SG-ISA; SG-ISA (2B)","Agent: SG-ISA; SG-ISA (4B)","Agent: SG-ISA; SG-ISA (8B)","Nemotron 3 Nano","Omni native input interface; full applicable 918-instance cohort","https://arxiv.org/html/2609.23417v1#S5.T1","Seed2.0 Lite","VL native input interface; full applicable 918-instance cohort","FAST-WAM","Memory No; common training and seed budget","https://arxiv.org/html/2609.24124v1#S3.T2","SaPaVe","HiF-VLA","Memory Yes; common training and seed budget","MemER","MemoryVLA","ActiveArena-VLA","Memory Yes; ActiveArena -Fixed","Memory Yes; ActiveArena -OFT","Memory Yes; ActiveArena -Plan","Coord. IDW","Coord. IDW; five-fold × five seeds; random","https://arxiv.org/html/2609.25454v1#S3.T2","Cartesian 3D","Cartesian 3D; five-fold × five seeds; random","Wrap","Wrap (Sin/Cos); five-fold × five seeds; random","SINR","SINR (ICML ’23); five-fold × five seeds; random","SatCLIP","SatCLIP (AAAI ’25); five-fold × five seeds; random","Climplicit","Climplicit (ICLRW ’25); five-fold × five seeds; random","GeoCLIP (NeurIPS ’23); five-fold × five seeds; random","CSP-iNat","CSP-iNat (ICML ’23); five-fold × five seeds; random","CSP-fMoW","CSP-fMoW (ICML ’23); five-fold × five seeds; random","GAIR","GAIR (ISPRS ’26); five-fold × five seeds; random","TTE","TTE (ECCV ’26); five-fold × five seeds; random","TaxaBind","TaxaBind (WACV ’25); five-fold × five seeds; random","SLED","SLED (arXiv ’26); five-fold × five seeds; random","MIND","MIND 64 \\methodname{MIND}_{64}; five-fold × five seeds; random","MIND 128 \\methodname{MIND}_{128}; five-fold × five seeds; random","MIND 256 \\methodname{MIND}_{256}; five-fold × five seeds; random","MIND Full \\methodname{MIND}_{\\mathrm{Full}}; five-fold × five seeds; random","MIND + CP \\methodname{MIND{+}CP}; five-fold × five seeds; random","Coord. IDW; five-fold × five seeds; 2deg","Cartesian 3D; five-fold × five seeds; 2deg","Wrap (Sin/Cos); five-fold × five seeds; 2deg","SINR (ICML ’23); five-fold × five seeds; 2deg","SatCLIP (AAAI ’25); five-fold × five seeds; 2deg","Climplicit (ICLRW ’25); five-fold × five seeds; 2deg","GeoCLIP (NeurIPS ’23); five-fold × five seeds; 2deg","CSP-iNat (ICML ’23); five-fold × five seeds; 2deg","CSP-fMoW (ICML ’23); five-fold × five seeds; 2deg","GAIR (ISPRS ’26); five-fold × five seeds; 2deg","TTE (ECCV ’26); five-fold × five seeds; 2deg","TaxaBind (WACV ’25); five-fold × five seeds; 2deg","SLED (arXiv ’26); five-fold × five seeds; 2deg","MIND 64 \\methodname{MIND}_{64}; five-fold × five seeds; 2deg","MIND 128 \\methodname{MIND}_{128}; five-fold × five seeds; 2deg","MIND 256 \\methodname{MIND}_{256}; five-fold × five seeds; 2deg","MIND Full \\methodname{MIND}_{\\mathrm{Full}}; five-fold × five seeds; 2deg","MIND + CP \\methodname{MIND{+}CP}; five-fold × five seeds; 2deg","Coord. IDW; five-fold × five seeds; 10deg","Cartesian 3D; five-fold × five seeds; 10deg","Wrap (Sin/Cos); five-fold × five seeds; 10deg","SINR (ICML ’23); five-fold × five seeds; 10deg","SatCLIP (AAAI ’25); five-fold × five seeds; 10deg","Climplicit (ICLRW ’25); five-fold × five seeds; 10deg","GeoCLIP (NeurIPS ’23); five-fold × five seeds; 10deg","CSP-iNat (ICML ’23); five-fold × five seeds; 10deg","CSP-fMoW (ICML ’23); five-fold × five seeds; 10deg","GAIR (ISPRS ’26); five-fold × five seeds; 10deg","TTE (ECCV ’26); five-fold × five seeds; 10deg","TaxaBind (WACV ’25); five-fold × five seeds; 10deg","SLED (arXiv ’26); five-fold × five seeds; 10deg","MIND 64 \\methodname{MIND}_{64}; five-fold × five seeds; 10deg","MIND 128 \\methodname{MIND}_{128}; five-fold × five seeds; 10deg","MIND 256 \\methodname{MIND}_{256}; five-fold × five seeds; 10deg","MIND Full \\methodname{MIND}_{\\mathrm{Full}}; five-fold × five seeds; 10deg","MIND + CP \\methodname{MIND{+}CP}; five-fold × five seeds; 10deg","Coord. IDW; five-fold × five seeds; 20deg","Cartesian 3D; five-fold × five seeds; 20deg","Wrap (Sin/Cos); five-fold × five seeds; 20deg","SINR (ICML ’23); five-fold × five seeds; 20deg","SatCLIP (AAAI ’25); five-fold × five seeds; 20deg","Climplicit (ICLRW ’25); five-fold × five seeds; 20deg","GeoCLIP (NeurIPS ’23); five-fold × five seeds; 20deg","CSP-iNat (ICML ’23); five-fold × five seeds; 20deg","CSP-fMoW (ICML ’23); five-fold × five seeds; 20deg","GAIR (ISPRS ’26); five-fold × five seeds; 20deg","TTE (ECCV ’26); five-fold × five seeds; 20deg","TaxaBind (WACV ’25); five-fold × five seeds; 20deg","SLED (arXiv ’26); five-fold × five seeds; 20deg","MIND 64 \\methodname{MIND}_{64}; five-fold × five seeds; 20deg","MIND 128 \\methodname{MIND}_{128}; five-fold × five seeds; 20deg","MIND 256 \\methodname{MIND}_{256}; five-fold × five seeds; 20deg","MIND Full \\methodname{MIND}_{\\mathrm{Full}}; five-fold × five seeds; 20deg","MIND + CP \\methodname{MIND{+}CP}; five-fold × five seeds; 20deg","Coord. IDW; five-fold × five seeds; 40deg","Cartesian 3D; five-fold × five seeds; 40deg","Wrap (Sin/Cos); five-fold × five seeds; 40deg","SINR (ICML ’23); five-fold × five seeds; 40deg","SatCLIP (AAAI ’25); five-fold × five seeds; 40deg","Climplicit (ICLRW ’25); five-fold × five seeds; 40deg","GeoCLIP (NeurIPS ’23); five-fold × five seeds; 40deg","CSP-iNat (ICML ’23); five-fold × five seeds; 40deg","CSP-fMoW (ICML ’23); five-fold × five seeds; 40deg","GAIR (ISPRS ’26); five-fold × five seeds; 40deg","TTE (ECCV ’26); five-fold × five seeds; 40deg","TaxaBind (WACV ’25); five-fold × five seeds; 40deg","SLED (arXiv ’26); five-fold × five seeds; 40deg","MIND 64 \\methodname{MIND}_{64}; five-fold × five seeds; 40deg","MIND 128 \\methodname{MIND}_{128}; five-fold × five seeds; 40deg","MIND 256 \\methodname{MIND}_{256}; five-fold × five seeds; 40deg","MIND Full \\methodname{MIND}_{\\mathrm{Full}}; five-fold × five seeds; 40deg","MIND + CP \\methodname{MIND{+}CP}; five-fold × five seeds; 40deg","solo; same eight-domain agent security suite","https://arxiv.org/html/2609.24662v1#S5.T4","dual-control; same eight-domain agent security suite","FFDNet-S","YOLO11; 9M parameters; original CommonForms checkpoint","https://arxiv.org/html/2609.23679v1#S4.T2","FFDNet-L","YOLO11; 25M parameters; original CommonForms checkpoint","TOOD","ResNet-101; 51.02M parameters; MMDetection; 24 epochs","DINO","Swin-L; 218M parameters; MMDetection; 24 epochs","Def. DETR","ResNet-50; 41.075M parameters; MMDetection; 24 epochs","ResNeXt-101; 127M parameters; MMDetection; 24 epochs","Sparse R-CNN","ResNet-101; 125.16M parameters; MMDetection; 24 epochs","ResNeXt-101; 99.651M parameters; MMDetection; 24 epochs","GFL","ResNeXt-101; 50.88M parameters; MMDetection; 24 epochs","Seedance 2.0","Published on-screen generation subset; common GT-calibrated judge rubric","https://arxiv.org/html/2609.04867v1#S4.T1","Kling-v3-omni","Veo 3.1","Veo 3.0","Sora 2","LTX-2","MOVA","Published off-screen generation subset; common GT-calibrated judge rubric","Published mixed generation subset; common GT-calibrated judge rubric","Policy: N/A; image backend: GPT-Image-2; harness: N/A","https://arxiv.org/html/2609.05171v1#S6.T2","Gemini3.1-Flash","Policy: N/A; image backend: Gemini3.1-Flash; harness: N/A","Policy: N/A; image backend: Gemini3.1-Flash-Lite; harness: N/A","Seedream-5.0-pro","Policy: N/A; image backend: Seedream-5.0-pro; harness: N/A","FLUX2-dev","Policy: N/A; image backend: FLUX2-dev; harness: N/A","Qwen-Image(Gen/Edit)","Policy: N/A; image backend: Qwen-Image(Gen/Edit); harness: N/A","Reason+GPT-Image-2","Policy: Kimi-K2.6; image backend: GPT-Image-2; harness: N/A","Reason+Gemini3.1-Flash-Lite","Policy: Kimi-K2.6; image backend: Gemini3.1-Flash-Lite; harness: N/A","Mind Brush","Policy: GPT-5.1; image backend: Qwen-Image(Gen/Edit); harness: Mind Brush","GenSearcher","Policy: GenSearcher-8B; image backend: Qwen-Image(Gen/Edit); harness: GenSearcher","GenEvolve","Policy: GenEvolve-8B; image backend: Qwen-Image(Gen/Edit); harness: GenEvolve","Opus-5 + GPT-Image-2","Policy: Claude-Opus-5; image backend: GPT-Image-2; harness: WeHarness","GPT-5.6-Sol + GPT-Image-2","Policy: GPT-5.6-Sol; image backend: GPT-Image-2; harness: WeHarness","Kimi-K2.6 + GPT-Image-2","Policy: Kimi-K2.6 (1T-A32B); image backend: GPT-Image-2; harness: WeHarness","Kimi-K2.6 + Gemini3.1-Flash-Lite","Policy: Kimi-K2.6 (1T-A32B); image backend: Gemini3.1-Flash-Lite; harness: WeHarness","Qwen-3.5 + GPT-Image-2","Policy: Qwen-3.5 (35B-A3B); image backend: GPT-Image-2; harness: WeHarness","Qwen-3.6 + GPT-Image-2","Policy: Qwen-3.6 (35B-A3B); image backend: GPT-Image-2; harness: WeHarness","Qwen-3.8 + GPT-Image-2","Policy: Qwen-3.8 (27B); image backend: GPT-Image-2; harness: WeHarness","Qwen3-VL + GPT-Image-2 (Baseline 1)","Policy: Qwen3-VL (30B-A3B); image backend: GPT-Image-2; harness: WeHarness","Qwen3-VL + Gemini3.1-Flash-Lite (Baseline 2)","Policy: Qwen3-VL (30B-A3B); image backend: Gemini3.1-Flash-Lite; harness: WeHarness","Qwen3-VL + Qwen-Image(Gen/Edit) (Baseline 3)","Policy: Qwen3-VL (30B-A3B); image backend: Qwen-Image(Gen/Edit); harness: WeHarness","WeAgent-SFT + GPT-Image-2","Policy: WeAgent-SFT (30B-A3B); image backend: GPT-Image-2; harness: WeHarness","WeAgent-RL + GPT-Image-2","Policy: WeAgent-RL (30B-A3B); image backend: GPT-Image-2; harness: WeHarness","WeAgent-SFT + Gemini3.1-Flash-Lite","Policy: WeAgent-SFT (30B-A3B); image backend: Gemini3.1-Flash-Lite; harness: WeHarness","WeAgent-RL + Gemini3.1-Flash-Lite","Policy: WeAgent-RL (30B-A3B); image backend: Gemini3.1-Flash-Lite; harness: WeHarness","WeAgent-RL + Qwen-Image(Gen/Edit)","Policy: WeAgent-RL (30B-A3B); image backend: Qwen-Image(Gen/Edit); harness: WeHarness","WeAgent-RL + WeEdit-M-SFT","Policy: WeAgent-RL (30B-A3B); image backend: WeEdit-M-SFT; harness: WeHarness","WeAgent-RL + WeEdit-M-RL","Policy: WeAgent-RL (30B-A3B); image backend: WeEdit-M-RL; harness: WeHarness","Baseline","https://arxiv.org/html/2609.08402v1#Sx6.T3","AGOS-Agent search-handoff-verify scaffold","smolagents scaffold; Corpus Search; 8 trials","https://arxiv.org/html/2609.09410v1#S6.T2","GLM-5.2-FP4","MiroFlow scaffold; Corpus Search; 8 trials","RealTabPFN v2.5","Reference cell: 10000 features; 100 train samples; same27classificationtargets","https://arxiv.org/html/2609.07441v1#S4.T1","Logistic Regression","TabDPT","TabPFN v3","RealTabPFN v2.0","TabPFN Wide 5k (ne3)","TabPFN Wide (8k)","TabFM","RealMLP","MITRA","TabM","Random Forest","Extra Trees","CatBoost","TabICL","LightGBM","KNN","MiniLM-L6-v2","Published pretrained embedding","https://arxiv.org/html/2609.08174v1#S4.T2","MedEmbed-0.1B","SapBERT-0.1B","BioLORD-0.1B","Tier1-only fine-tune","BioLORD-0.1B‡","Joint all-tier fine-tune","Qwen3-Embed-0.6B","Qwen3-Embed-0.6B‡","Qwen3-Embed-4B","OpenAI (text-embedding-3-large)","DyGIE++","Supervised IE","https://arxiv.org/html/2609.10722v1#S4.T5","OneIE","UIE","Zero-shot LLM","Moonshot-v1-8k","DeepSeek-V3.1-Chat","GoLLIE + CodeLlama-7B","Fine-tuned structured IE framework","GoLLIE + Llama-3.1-8B-Instruct","GoLLIE + Qwen2.5-Coder-7B","GoLLIE + Qwen2.5-7B-Instruct","ADELIE + Llama-3.1-8B-Instruct","ADELIE + Qwen2.5-7B-Instruct","KnowCoder + Llama-3.1-8B-Instruct","Base LLM; observed hit labels; fixed assay-budget simulation","https://arxiv.org/html/2609.11877v1#A3.T4","Claude Opus-4.8","Claude Sonnet-4.6","Claude Haiku-4.5","2D-Maze reasoning; common task-defined target","https://arxiv.org/html/2609.11900v1#S3.T1","Nemotron-Nano-12B-VL-v2","Llama-4-Maverick-17B-128E","Ministral-3-14B-Instruct-2512","ThinkMorph-7B","Offline EQA; six10000-item suites; published capability-balanced mean","https://arxiv.org/html/2609.13082v1#S5.T4","YOLOv8m-seg","Common fixed test split; official instance ownership/ignore masks","https://arxiv.org/html/2609.12350v1#S4.T4","YOLO11m-seg","Mask R-CNN","PointRend","BCNet","Mask2Former R50","Mask2Former Swin-B","CropFormer","DI-MaskDINO-R50","GPT5.4 (CUA)","Common FreeCAD VM and allowlisted screenshot/GUI action runner","https://arxiv.org/html/2609.16251v1#S4.T3","Opus4.8 (CUA)","OpenCUA","Holo 3.1","Qwen3.6","39 short-horizon scenarios; three episodes each; 117 episodes per model","https://arxiv.org/pdf/2610.05414v2","Qwen-3.5-Plus","Doubao-SEED-2.0-Pro","21 long-horizon scenarios; three episodes each; 63 episodes per model","30 composite navigation tasks; three episodes each; BEV mapping; 90 episodes per model","30 composite navigation tasks; three episodes each; no BEV mapping; 90 episodes per model","LIV","Zero-shot; ID SA; common endpoint-window adapter","https://arxiv.org/pdf/2610.26663v1","ProcVLM-2B","RoboMeter-4B","RoboReward-4B","RynnValue-4B","RynnValue-8B","TOPReward-8B","TOPReward-32B","VLAC-2B","VLAC-8B","SegEarth-OV (ViT-B/16)","Zero-shot; RGB only; optical","https://arxiv.org/pdf/2610.29642v1","SegEarth-OV3 (SAM3)","CAT-Seg (ViT-B/16)","Official method fine-tuned on benchmark train split; RGB only; optical","EBSeg (ViT-B/16)","GSNet (ViT-B/16)","RSKT-Seg (ViT-B/16)","CAT-Seg (ViT-L/14)","RSKT-Seg (ViT-L/14)","Zero-shot; RGB only; cloudy","Official method fine-tuned on benchmark train split; RGB only; cloudy","Zero-shot; RGB only; cross","Official method fine-tuned on benchmark train split; RGB only; cross","Standard LIBERO training; 40 tasks; ten trials per task-condition; temporal-corruption","https://arxiv.org/pdf/2609.24350v1#page=5","LaWAM","Standard LIBERO training; 40 tasks; ten trials per task-condition; view-unavailability","Standard LIBERO training; 40 tasks; ten trials per task-condition; task-distractors","Standard LIBERO training; 40 tasks; ten trials per task-condition; task-preconditions","Standard LIBERO training; 40 tasks; ten trials per task-condition; target-geometry","Standard LIBERO training; 40 tasks; ten trials per task-condition; camera-staleness","Agent: OpenClaw; 208 frozen community scenarios; native isolated harness","https://arxiv.org/pdf/2609.34790v1#page=7","Agent: Hermes; 208 frozen community scenarios; native isolated harness","Agent: Codex; 208 frozen community scenarios; native isolated harness","Weak-start optimization; fixed edit/evaluator budget; held-out selected candidate","https://arxiv.org/pdf/2609.32227v1#page=7","GPT-5.1-Codex","https://arxiv.org/html/2608.21827v1#S4.T2","Visual context + dialogue; english split","https://arxiv.org/html/2608.30270v1#S5.T3","HyperCLOVA","LLaVA","KoLLaVA","Qwen3","InternVL 3.5","Visual context + dialogue; korean split","DSRNet (ICCV’23)","S2R-Real; common 50 videos; no-reference removal scorer","https://arxiv.org/html/2608.11562v1#S5.T1","DSIT (NeurIPS’24)","RDNet (CVPR’25)","DAI (AAAI’26)","GenSIRR (CVPR’26)","S2R-Removal","BPR","Common SCOPE-Bench split; sampled candidates; top 20","https://arxiv.org/html/2608.13990v1#S6.T8","NCF","LightGCN","VBPR","GRCN","LATTICE","BM3","FREEDOM","MGCN","LGMRec","DiffMM","REARM","FITMM","Common SCOPE-Bench split; full candidates; top 20","Claude Code; effort low; common 10 tasks and compute contract","https://arxiv.org/html/2608.20318v1#S3.T3","Claude Code; effort medium; common 10 tasks and compute contract","Claude Code; effort high; common 10 tasks and compute contract","Claude Code; effort xhigh; common 10 tasks and compute contract","Claude Code; effort max; common 10 tasks and compute contract","Codex; effort none; common 10 tasks and compute contract","Codex; effort low; common 10 tasks and compute contract","Codex; effort medium; common 10 tasks and compute contract","Codex; effort high; common 10 tasks and compute contract","Codex; effort xhigh; common 10 tasks and compute contract","Codex; effort max; common 10 tasks and compute contract","Codex; GPT-5.5 (xhigh); source repeated-run mean","https://arxiv.org/html/2608.17271v1#S3.T2","Codex; GPT-5.6 Sol (xhigh); source repeated-run mean","Codex; GPT-5.6 Sol (ultra); source repeated-run mean","Claude Code; Claude Opus 5 †; single run","Claude Code; Kimi K3; source repeated-run mean","Claude Code; Claude Opus 4.8; source repeated-run mean","Claude Code; GLM-5.3; source repeated-run mean","Claude Code; GLM-5.2; source repeated-run mean","Claude Code; DeepSeek V4 Flash; source repeated-run mean","Claude Code; Kimi K2.7; source repeated-run mean","Claude Code; MiniMax M3; source repeated-run mean","Claude Code; DeepSeek V4 Pro; source repeated-run mean","Claude Code; MiMo V2.5 Pro; source repeated-run mean","Kimi Code; Kimi K3; source repeated-run mean","Kimi Code; Kimi K2.7; source repeated-run mean","MiMo Code; MiMo V2.5 Pro; source repeated-run mean","OpenHands; DeepSeek V4 Flash; source repeated-run mean","OpenHands; DeepSeek V4 Pro; source repeated-run mean","Learned policy; unseen objects","https://arxiv.org/html/2608.14049v1#S5.T1","3D Diffusion Policy","π 0","FlatLab grasping framework","Learned policy; unseen categories","Chat- UniVi V1.5","Frames: 256; parameters: 7B; common 1,443-question set","https://arxiv.org/html/2608.13113v1#S4.T3","LLaVA- NeXT Video","Frames: 64; parameters: 7B; common 1,443-question set","MiniCPM V4.5","Frames: 256; parameters: 8B; common 1,443-question set","Qwen2 VL","Qwen2.5 VL","Frames: 256; parameters: 32B; common 1,443-question set","Qwen3 VL","Qwen3 VL-30B A3B","Frames: 256; parameters: 30B; common 1,443-question set","Share GPT","Frames: 64; parameters: 8B; common 1,443-question set","ST LLM","Frames: 256; parameters: –; common 1,443-question set","Video LLaMA3","Frames: 512; parameters: 7B; common 1,443-question set","Frames: 16; parameters: 7B; common 1,443-question set","Frames: 1fps; parameters: –; common 1,443-question set","n=879; Claude Opus 4.7 calibrated judge; no system prompt; temperature 0","https://github.com/BanglaLLM/banglasafe/blob/main/src/banglasafe/data/reference_cohort.json","deepseek-v3","gemma-3-12b","gemma-3-27b","gemma-4-26b","llama-3.1-8b","llama-3.2-3b","llama-3.3-70b","mistral-medium-3","qwen3-30b","tigerllm_1b","tigerllm_9b","Common four-module ELBench protocol; paper v2","https://arxiv.org/html/2608.09548v2#S5.T3","Safe-InnoSpark","InnoSpark-235B","Closed book; complete multi-account recall; common 1,094 questions","https://arxiv.org/html/2608.28478v1#S4.T1","OLMo-3.1-32B","OLMo-3-7B","Common ten-subtask spoken EI scoring protocol","https://arxiv.org/html/2608.09189v1#S3.T4","LLaMa-Omni 2","Step-Audio 2","Kimi-Audio","GPT-4o-Audio","EmoS (Qwen2.5-Omni)","Common 250-video/750-rating human gold; 1–5 rating MAE","https://arxiv.org/html/2608.21839v1#S3.T1","FIRM-Video Data Pipeline †","FIRM-Video-8B (InternVL3-8B)","FIRM-Video-8B (Qwen3-VL-8B)","w/ history; common 500 offline reference trajectories","https://arxiv.org/html/2609.00048v1#S3.T2","w/o history; common 500 offline reference trajectories","Code2World","MobileWorld-html-8B","gWorld-32B","gWorld-8B","Gemini 3.1 Flash Image","Vimo","Flux.2-dev","MobileWorld-Diffusion","Source published video-model/agent protocol","https://arxiv.org/html/2608.14015v1#S5.T2","VideoChat-R1-7B","Video-RFT","LongVT-7B-RFT","ReWatch-R1-7B","SurgVidLM","One-shot","MedClaw (Gemini 3.5 Flash vision)","MedClaw with Gemini 3.5 Flash vision","codex; clean evaluation cue; hybrid outcome judge; full 1,661 cases","https://arxiv.org/html/2608.10669v1#S4.T2","Qwen3.5-plus","Qwen-plus-2025-12-01","hermes; clean evaluation cue; hybrid outcome judge; full 1,661 cases","Claude Code; effort low; same 20 migrations","https://arxiv.org/html/2608.23564v1#S3.T3","Claude Code; effort medium; same 20 migrations","Claude Code; effort high; same 20 migrations","Claude Code; effort xhigh; same 20 migrations","Claude Code; effort max; same 20 migrations","Codex; effort none; same 20 migrations","Codex; effort low; same 20 migrations","Codex; effort medium; same 20 migrations","Codex; effort high; same 20 migrations","Codex; effort xhigh; same 20 migrations","Codex; effort max; same 20 migrations","mini-swe-agent; fixed executable multilingual evaluation split; paper v2","https://arxiv.org/html/2608.09802v2#S5.T3","openhands; fixed executable multilingual evaluation split; paper v2","Same professional task set; three independent runs","https://arxiv.org/html/2608.17800v1#S4.T3","Qwen-3.6-Max","CNN-MLP","From scratch; word split","https://arxiv.org/html/2608.28092v1#S4.T1","CNN-Trans","ViT-MLP","ViT-Trans","ViT-Combined","RadDINO-MLP","Frozen backbone; word split","RadDINO-Trans","BiomedCLIP-MLP","BiomedCLIP-Trans","SAM-Med2D-MLP","SAM-Med2D-Trans","Fine-tuned backbone; word split","From scratch; totalsegmentator split","Frozen backbone; totalsegmentator split","Fine-tuned backbone; totalsegmentator split","Zero-shot; common deduplicated 1,600-item test","https://arxiv.org/html/2608.15600v1#Sx4.T1","+Taxonomy; common deduplicated 1,600-item test","CoT-SFT; common deduplicated 1,600-item test","InternLM3-8B","PP-OCRv5 + o3-mini","proactive verification paradigm; all 1,600 invalid tasks","https://arxiv.org/html/2609.00232v1#S3.T2","PP-OCRv5 + DeepSeek-R1-Distill-Qwen-32B","Gemini-3-Flash-Thinking","Gemini-3-Flash-NoThinking","Llama-4-Scout-17B","Yi-VL-Plus","assisted verification paradigm; all 1,600 invalid tasks","SVD","Source unified VGA dimension-aligned prompts; held-out generated videos","https://arxiv.org/html/2608.25452v1#S4.T5","AnimateDiff","LaVie","Show-1 Zhang et al. (2025)","ModelScope","CogVideoX","Latte-1","Mochi","LTXVideo","HunyuanVideo","Sora2","Gemini-3.1 Pro; common human-verified diagnostic questions","https://arxiv.org/html/2608.09573v1#Sx4.T1","Gemini-3.1-Flash Lite","Gemini-3.1-Flash Lite; common human-verified diagnostic questions","Gemini-3 Flash; common human-verified diagnostic questions","Gemini-2.5 Flash","Gemini-2.5 Flash; common human-verified diagnostic questions","VideoLLaMA-3-7B","VideoLLaMA-3-7B; common human-verified diagnostic questions","MiniCPM-V-4.5-8B; common human-verified diagnostic questions","Qwen3-VL-8B Instruct; common human-verified diagnostic questions","Qwen3-VL-8B Thinking; common human-verified diagnostic questions","Qwen3-VL-30B-A3B-Instruct; common human-verified diagnostic questions","Qwen3-VL-30B-A3B-Thinking; common human-verified diagnostic questions","Qwen3.5-35B-A3B; common human-verified diagnostic questions","GLM-4.6V-106B-A12B","GLM-4.6V-106B-A12B; common human-verified diagnostic questions","Mimo-v2.5-310B-A15B","Mimo-v2.5-310B-A15B; common human-verified diagnostic questions","V2Lens","V2Lens (Ours); common human-verified diagnostic questions","ToxScribe Qwen","Veterinary pathology; common 419-image/1,251-question cohort","https://arxiv.org/html/2608.26382v1#S5.T1","ToxScribe Gemma","General-purpose; common 419-image/1,251-question cohort","Qwen 3.5-27B","PathChat+","Human pathology; common 419-image/1,251-question cohort","Patho-R1-7B","Patho-R1-3B","PathGen-LLaVA","Quilt-LLaVA","Uniform-64; same 600 questions","https://arxiv.org/html/2608.22516v1#S5.T2","oob; same 9 scenarios × 15 seeds = 135 episodes per model","https://github.com/rudycelekli/the-value-engine-benchmark/blob/main/datasets/veb-canonical-135/paper-stats.json","moonshotai/kimi-k3","pack; same 9 scenarios × 15 seeds = 135 episodes per model","GPT-5-2025-08-07","Same nine-category artifact test; source overall NED component","https://arxiv.org/html/2608.27169v1#S4.T3","Gemini-3.1-pro-preview","Qwen3.6-Flash-2026-04-16","Doubao-seed-2-0-pro-260215","Doubao-seed-2-0-lite-260428","Gemma-4-E2B-it","MiniCPM-V 4.6","Deepseek-OCR2","HunyuanOCR","OCRVerse","Qianfan-OCR","dots.ocr","dots.mocr","Nanonets-OCR2-3B","PaddleOCR-VL1.5","MonkeyOCR-3B","MonkeyOCR-pro-3B","YouTu-Parsinig","Direct model; common Q1 questions","https://arxiv.org/html/2608.23435v1#S4.T2","BasketballSkills","BasketballSkills agent","https://arxiv.org/html/2608.26807v1#S4.T3","B2T-Agent","Info w/o; full six-relation classification","https://arxiv.org/html/2608.23390v2#A3.T15","Llama-3.2-3B-IT","Llama-3.1-8B-IT","Ministral-3-3B-IT","Ministral-3-8B-IT","Ministral-3-14B-IT","Gemma-3-4B-IT","Info w/; full six-relation classification","Uniform sampling; vehicle counting","https://arxiv.org/html/2608.15539v1#S4.T2","Uniform sampling; robot temporal reasoning","Common 58-query multi-source multihop subset","https://arxiv.org/html/2608.12282v1#A2.T6","LLaMA-405B","Mistral-Large-3-675B","Granite-4h-small","LLaMA-3.3-70B-Instruct","Qwen-2.5-72B-Instruct","Airfoil scaled sound pressure; XL; native quality index","https://github.com/Narnialabs/leaderboard/blob/777f4f1df1f31b83db3cdf8775b503b82cb900d8/output/data/leaderboard/dimension_1d/prediction/scalar/quality/airfoil_noise/scaled_sound_pressure_db/Quality_Leaderboard_XL_airfoil_noise_scaled_sound_pressure_db.csv","NODE","Gaussian Process","FT-Transformer","TabNet","TabPFN","Ridge","DeepJEB image geometry; XL; native quality index","https://github.com/Narnialabs/leaderboard/blob/777f4f1df1f31b83db3cdf8775b503b82cb900d8/output/data/leaderboard/dimension_2d/generation/quality/deepjeb/Quality_Leaderboard_XL_deepjeb.csv","R1GAN","VQVAE","WGAN-GP","LSGAN","DCGAN","VAE (Basic)","GAN (Basic)","WGAN-CP","Attention U-Net","DeepJEB structural displacement field; XL; native quality index","https://github.com/Narnialabs/leaderboard/blob/777f4f1df1f31b83db3cdf8775b503b82cb900d8/output/data/leaderboard/dimension_2d/prediction/field/quality/deepjeb_structural/displacement_stress/displacement/Quality_Leaderboard_XL_deepjeb_structural_displacement_stress_displacement.csv","FPN (ResNet-18)","U-Net","U-Net++","SegFormer-B0","ResNet-UNet","GLPN","DPT-Hybrid","ConvNeXt-Tiny","DeepJEB structural displacement scalar; XL; native quality index","https://github.com/Narnialabs/leaderboard/blob/777f4f1df1f31b83db3cdf8775b503b82cb900d8/output/data/leaderboard/dimension_2d/prediction/scalar/quality/deepjeb_structural/displacement_stress/displacement/Quality_Leaderboard_XL_deepjeb_structural_displacement_stress_displacement.csv","ResNet-34","DenseNet-121","ResNet-18","SimpleCNN","EfficientNet-B0","ViT-Tiny","DeepSDF","DeepJEB geometry; XL; native quality index","https://github.com/Narnialabs/leaderboard/blob/777f4f1df1f31b83db3cdf8775b503b82cb900d8/output/data/leaderboard/dimension_3d/generation/quality/deepjeb/Quality_Leaderboard_XL_deepjeb.csv","PointFlow","3D-GAN","AtlasNet","Diffusion3D","ShapeGF","Transolver++","DeepJEB diagonal x-displacement field; XL; native quality index","https://github.com/Narnialabs/leaderboard/blob/777f4f1df1f31b83db3cdf8775b503b82cb900d8/output/data/leaderboard/dimension_3d/prediction/field/quality/deepjeb_diagonal/displacement_stress/displacement/x/Quality_Leaderboard_XL_deepjeb_diagonal_displacement_stress_displacement_x.csv","hno","transolver3","Transolver","RegDGCNN","GeoFNO","PointNet","Point Transformer-Small","https://github.com/Narnialabs/leaderboard/blob/777f4f1df1f31b83db3cdf8775b503b82cb900d8/output/data/leaderboard/dimension_3d/prediction/scalar/quality/deepjeb_structural/displacement_stress/displacement/Quality_Leaderboard_XL_deepjeb_structural_displacement_stress_displacement.csv","DGCNN","PCT-Small","Point Transformer","PCT","PointNet++ Lite","PointMLP","PointMLP-Elite","PointNet++","RQ1 harmful execution; full40role-matchedassignments; four-way outcome labels","https://arxiv.org/html/2609.06966v1#A8.T11","Qwen3-235B-A22B (abliterated)","Mistral Large 3 2512","Char-Seq2Seq","Char-Seq2Seq; published baseline run","https://arxiv.org/html/2609.09889v1#S5.T4","Llama-3.1 + Uttr2Uttr","Llama-3.1 + Uttr2Uttr; published baseline run","pkoT5 + Uttr2Uttr","pkoT5 + Uttr2Uttr; published baseline run","Llama-3.1 + DCSC (ours.)","Llama-3.1 + DCSC (ours.) †; 3-seed mean","Qwen2.5 + DCSC (ours.)","Qwen2.5 + DCSC (ours.); published baseline run","mT5 + DCSC (ours.)","mT5 + DCSC (ours.); published baseline run","KoBart + DCSC (ours.)","KoBart + DCSC (ours.); published baseline run","pkoT5 + DCSC (ours.)","pkoT5 + DCSC (ours.) †; 3-seed mean","DCL (a)","Inertial; DCL (a); LOSO13subjects; horizon0s","https://arxiv.org/html/2609.08090v1#A5.T11","DCL (a/g)","Inertial; DCL (a/g); LOSO13subjects; horizon0s","MNV3-S","Image; MNV3-S; LOSO13subjects; horizon0s","R18","Image; R18; LOSO13subjects; horizon0s","R50","Image; R50; LOSO13subjects; horizon0s","X3D-XS","Video; X3D-XS; LOSO13subjects; horizon0s","MViT","Video; MViT; LOSO13subjects; horizon0s","KIFNet-Style","Inertial+image; KIFNet-Style; LOSO13subjects; horizon0s","KIFNet (A)","Inertial+image; KIFNet (A); LOSO13subjects; horizon0s","KIFNet (C)","Inertial+image; KIFNet (C); LOSO13subjects; horizon0s","MNV3-DCL (A)","Inertial+image; MNV3-DCL (A); LOSO13subjects; horizon0s","MNV3-DCL (C)","Inertial+image; MNV3-DCL (C); LOSO13subjects; horizon0s","R18-DCL (A)","Inertial+image; R18-DCL (A); LOSO13subjects; horizon0s","R18-DCL (C)","Inertial+image; R18-DCL (C); LOSO13subjects; horizon0s","SFTIK","Inertial+video; SFTIK; LOSO13subjects; horizon0s","IMU-Video-MAE","Inertial+video; IMU-Video-MAE; LOSO13subjects; horizon0s","X3D-XS-DCL (A)","Inertial+video; X3D-XS-DCL (A); LOSO13subjects; horizon0s","X3D-XS-DCL (C)","Inertial+video; X3D-XS-DCL (C); LOSO13subjects; horizon0s","MViT-DCL (A)","Inertial+video; MViT-DCL (A); LOSO13subjects; horizon0s","MViT-DCL (C)","Inertial+video; MViT-DCL (C); LOSO13subjects; horizon0s","Inertial; DCL (a); LOSO13subjects; horizon0.1s","Inertial; DCL (a/g); LOSO13subjects; horizon0.1s","Image; MNV3-S; LOSO13subjects; horizon0.1s","Image; R18; LOSO13subjects; horizon0.1s","Image; R50; LOSO13subjects; horizon0.1s","Video; X3D-XS; LOSO13subjects; horizon0.1s","Video; MViT; LOSO13subjects; horizon0.1s","Inertial+image; KIFNet-Style; LOSO13subjects; horizon0.1s","Inertial+image; KIFNet (A); LOSO13subjects; horizon0.1s","Inertial+image; KIFNet (C); LOSO13subjects; horizon0.1s","Inertial+image; MNV3-DCL (A); LOSO13subjects; horizon0.1s","Inertial+image; MNV3-DCL (C); LOSO13subjects; horizon0.1s","Inertial+image; R18-DCL (A); LOSO13subjects; horizon0.1s","Inertial+image; R18-DCL (C); LOSO13subjects; horizon0.1s","Inertial+video; SFTIK; LOSO13subjects; horizon0.1s","Inertial+video; IMU-Video-MAE; LOSO13subjects; horizon0.1s","Inertial+video; X3D-XS-DCL (A); LOSO13subjects; horizon0.1s","Inertial+video; X3D-XS-DCL (C); LOSO13subjects; horizon0.1s","Inertial+video; MViT-DCL (A); LOSO13subjects; horizon0.1s","Inertial+video; MViT-DCL (C); LOSO13subjects; horizon0.1s","Inertial; DCL (a); LOSO13subjects; horizon0.2s","Inertial; DCL (a/g); LOSO13subjects; horizon0.2s","Image; MNV3-S; LOSO13subjects; horizon0.2s","Image; R18; LOSO13subjects; horizon0.2s","Image; R50; LOSO13subjects; horizon0.2s","Video; X3D-XS; LOSO13subjects; horizon0.2s","Video; MViT; LOSO13subjects; horizon0.2s","Inertial+image; KIFNet-Style; LOSO13subjects; horizon0.2s","Inertial+image; KIFNet (A); LOSO13subjects; horizon0.2s","Inertial+image; KIFNet (C); LOSO13subjects; horizon0.2s","Inertial+image; MNV3-DCL (A); LOSO13subjects; horizon0.2s","Inertial+image; MNV3-DCL (C); LOSO13subjects; horizon0.2s","Inertial+image; R18-DCL (A); LOSO13subjects; horizon0.2s","Inertial+image; R18-DCL (C); LOSO13subjects; horizon0.2s","Inertial+video; SFTIK; LOSO13subjects; horizon0.2s","Inertial+video; IMU-Video-MAE; LOSO13subjects; horizon0.2s","Inertial+video; X3D-XS-DCL (A); LOSO13subjects; horizon0.2s","Inertial+video; X3D-XS-DCL (C); LOSO13subjects; horizon0.2s","Inertial+video; MViT-DCL (A); LOSO13subjects; horizon0.2s","Inertial+video; MViT-DCL (C); LOSO13subjects; horizon0.2s","Inertial; DCL (a); LOSO13subjects; horizon0.3s","Inertial; DCL (a/g); LOSO13subjects; horizon0.3s","Image; MNV3-S; LOSO13subjects; horizon0.3s","Image; R18; LOSO13subjects; horizon0.3s","Image; R50; LOSO13subjects; horizon0.3s","Video; X3D-XS; LOSO13subjects; horizon0.3s","Video; MViT; LOSO13subjects; horizon0.3s","Inertial+image; KIFNet-Style; LOSO13subjects; horizon0.3s","Inertial+image; KIFNet (A); LOSO13subjects; horizon0.3s","Inertial+image; KIFNet (C); LOSO13subjects; horizon0.3s","Inertial+image; MNV3-DCL (A); LOSO13subjects; horizon0.3s","Inertial+image; MNV3-DCL (C); LOSO13subjects; horizon0.3s","Inertial+image; R18-DCL (A); LOSO13subjects; horizon0.3s","Inertial+image; R18-DCL (C); LOSO13subjects; horizon0.3s","Inertial+video; SFTIK; LOSO13subjects; horizon0.3s","Inertial+video; IMU-Video-MAE; LOSO13subjects; horizon0.3s","Inertial+video; X3D-XS-DCL (A); LOSO13subjects; horizon0.3s","Inertial+video; X3D-XS-DCL (C); LOSO13subjects; horizon0.3s","Inertial+video; MViT-DCL (A); LOSO13subjects; horizon0.3s","Inertial+video; MViT-DCL (C); LOSO13subjects; horizon0.3s","Inertial; DCL (a); LOSO13subjects; horizon0.5s","Inertial; DCL (a/g); LOSO13subjects; horizon0.5s","Image; MNV3-S; LOSO13subjects; horizon0.5s","Image; R18; LOSO13subjects; horizon0.5s","Image; R50; LOSO13subjects; horizon0.5s","Video; X3D-XS; LOSO13subjects; horizon0.5s","Video; MViT; LOSO13subjects; horizon0.5s","Inertial+image; KIFNet-Style; LOSO13subjects; horizon0.5s","Inertial+image; KIFNet (A); LOSO13subjects; horizon0.5s","Inertial+image; KIFNet (C); LOSO13subjects; horizon0.5s","Inertial+image; MNV3-DCL (A); LOSO13subjects; horizon0.5s","Inertial+image; MNV3-DCL (C); LOSO13subjects; horizon0.5s","Inertial+image; R18-DCL (A); LOSO13subjects; horizon0.5s","Inertial+image; R18-DCL (C); LOSO13subjects; horizon0.5s","Inertial+video; SFTIK; LOSO13subjects; horizon0.5s","Inertial+video; IMU-Video-MAE; LOSO13subjects; horizon0.5s","Inertial+video; X3D-XS-DCL (A); LOSO13subjects; horizon0.5s","Inertial+video; X3D-XS-DCL (C); LOSO13subjects; horizon0.5s","Inertial+video; MViT-DCL (A); LOSO13subjects; horizon0.5s","Inertial+video; MViT-DCL (C); LOSO13subjects; horizon0.5s","Inertial; DCL (a); LOSO13subjects; horizon1s","Inertial; DCL (a/g); LOSO13subjects; horizon1s","Image; MNV3-S; LOSO13subjects; horizon1s","Image; R18; LOSO13subjects; horizon1s","Image; R50; LOSO13subjects; horizon1s","Video; X3D-XS; LOSO13subjects; horizon1s","Video; MViT; LOSO13subjects; horizon1s","Inertial+image; KIFNet-Style; LOSO13subjects; horizon1s","Inertial+image; KIFNet (A); LOSO13subjects; horizon1s","Inertial+image; KIFNet (C); LOSO13subjects; horizon1s","Inertial+image; MNV3-DCL (A); LOSO13subjects; horizon1s","Inertial+image; MNV3-DCL (C); LOSO13subjects; horizon1s","Inertial+image; R18-DCL (A); LOSO13subjects; horizon1s","Inertial+image; R18-DCL (C); LOSO13subjects; horizon1s","Inertial+video; SFTIK; LOSO13subjects; horizon1s","Inertial+video; IMU-Video-MAE; LOSO13subjects; horizon1s","Inertial+video; X3D-XS-DCL (A); LOSO13subjects; horizon1s","Inertial+video; X3D-XS-DCL (C); LOSO13subjects; horizon1s","Inertial+video; MViT-DCL (A); LOSO13subjects; horizon1s","Inertial+video; MViT-DCL (C); LOSO13subjects; horizon1s","No intervention; forget set n200; max output2048; returned reasoning scored","https://arxiv.org/html/2609.12808v1#S5.T6","Non-realtime audio agent","https://arxiv.org/html/2609.13076v1#S4.T3","Realtime voice agent","Ultravox","Covo-Audio","DuplexCascade","Gemini-2.5 Live","Gemini-3.1 Live","D-3DGS","Reduced-memory background; same sequence/test cameras; causal streaming reconstruction","https://arxiv.org/html/2609.16310v1#S5.T3","HiCoM","QUEEN","ReConGs","Trackersplat","FastFlowGS","GUI; memory notes; sequential100bills; model release Feb 5, 2026","https://apprenticebench.com/assets/index-D78RXqBf.js","GUI; memory notes; sequential100bills; model release Mar 5, 2026","GUI; memory notes; sequential100bills; model release Apr 16, 2026","GUI; memory notes; sequential100bills; model release Apr 23, 2026","GUI; memory notes; sequential100bills; model release Jun 9, 2026","GUI; memory notes; sequential100bills; model release Jul 9, 2026","GUI; memory notes; sequential100bills; model release Jul 16, 2026","GUI; memory notes; sequential100bills; model release Aug 13, 2026","GUI; memory notes; sequential100bills; model release Sep 1, 2026","GUI; memory notes; sequential100bills; model release Sep 2, 2026","GUI; memory notes; sequential100bills; model release Sep 3, 2026","GUI; memory notes; sequential100bills; model release Jul 24, 2026","GUI; memory notes; sequential100bills; model release Aug 12, 2026","API; memory notes; sequential100bills; model release Jan 27, 2026","API; memory notes; sequential100bills; model release Feb 5, 2026","API; memory notes; sequential100bills; model release Mar 5, 2026","API; memory notes; sequential100bills; model release Apr 16, 2026","API; memory notes; sequential100bills; model release Apr 20, 2026","API; memory notes; sequential100bills; model release Apr 23, 2026","API; memory notes; sequential100bills; model release Jun 9, 2026","API; memory notes; sequential100bills; model release Jun 26, 2026","API; memory notes; sequential100bills; model release Jul 8, 2026","API; memory notes; sequential100bills; model release Jul 9, 2026","API; memory notes; sequential100bills; model release Jul 16, 2026","API; memory notes; sequential100bills; model release Jul 21, 2026","API; memory notes; sequential100bills; model release Jul 24, 2026","V4 Flash","API; memory notes; sequential100bills; model release Jul 31, 2026","API; memory notes; sequential100bills; model release Aug 3, 2026","API; memory notes; sequential100bills; model release Aug 5, 2026","API; memory notes; sequential100bills; model release Aug 12, 2026","API; memory notes; sequential100bills; model release Aug 13, 2026","V4 Pro","API; memory notes; sequential100bills; model release Aug 26, 2026","API; memory notes; sequential100bills; model release Sep 1, 2026","API; memory notes; sequential100bills; model release Sep 2, 2026","API; memory notes; sequential100bills; model release Sep 3, 2026","64x64 rgb; seed42; Simple CNN; {\"architecture\": \"conv32-pool-conv64-pool-dense128-dropout0.30-softmax\", \"batch_size\": 32, \"device\": \"cpu\", \"early_stopping_patience\": 3, \"learning_rate\": 0.001, \"loss\": \"cross_entropy\", \"max_epochs\": 20, \"min_improvement\": 0.0, \"optimizer\": \"Adam\", \"pretrained\": false, \"random_state\": 42, \"validation_fraction\": 0.15}","https://raw.githubusercontent.com/srcollins785/Samuel_Collins_CV_Benchmarking/d7e28af8d2a607c33ebcfb8493fce8e73259cb52/benchmark_results/animals10_n100_rgb/benchmark_summary.csv","64x64 rgb; seed42; Random Forest; {\"n_estimators\": 200, \"n_jobs\": -1, \"random_state\": 42}","SVM","64x64 rgb; seed42; SVM; {\"kernel\": \"rbf\", \"random_state\": 42}","64x64 rgb; seed42; Logistic Regression; {\"max_iter\": 1000, \"random_state\": 42}","Neural Network","64x64 rgb; seed42; Neural Network; {\"early_stopping\": true, \"hidden_layer_sizes\": [128, 64], \"max_iter\": 100, \"n_iter_no_change\": 3, \"random_state\": 42, \"validation_fraction\": 0.15}","Decision Tree","64x64 rgb; seed42; Decision Tree; {\"random_state\": 42}","64x64 grayscale; seed42; Simple CNN; {\"architecture\": \"conv32-pool-conv64-pool-dense128-dropout0.30-softmax\", \"batch_size\": 32, \"device\": \"cpu\", \"early_stopping_patience\": 3, \"learning_rate\": 0.001, \"loss\": \"cross_entropy\", \"max_epochs\": 20, \"min_improvement\": 0.0, \"optimizer\": \"Adam\", \"pretrained\": false, \"random_state\": 42, \"validation_fraction\": 0.15}","https://raw.githubusercontent.com/srcollins785/Samuel_Collins_CV_Benchmarking/d7e28af8d2a607c33ebcfb8493fce8e73259cb52/benchmark_results/animals10_n100_grayscale/benchmark_summary.csv","64x64 grayscale; seed42; Random Forest; {\"n_estimators\": 200, \"n_jobs\": -1, \"random_state\": 42}","64x64 grayscale; seed42; SVM; {\"kernel\": \"rbf\", \"random_state\": 42}","64x64 grayscale; seed42; Neural Network; {\"early_stopping\": true, \"hidden_layer_sizes\": [128, 64], \"max_iter\": 100, \"n_iter_no_change\": 3, \"random_state\": 42, \"validation_fraction\": 0.15}","64x64 grayscale; seed42; Logistic Regression; {\"max_iter\": 1000, \"random_state\": 42}","64x64 grayscale; seed42; Decision Tree; {\"random_state\": 42}","https://raw.githubusercontent.com/srcollins785/Samuel_Collins_CV_Benchmarking/d7e28af8d2a607c33ebcfb8493fce8e73259cb52/benchmark_results/animals10_n500_rgb/combined_ml_cnn_benchmark_results.csv","AlexNet","{\"architecture\": \"alexnet\", \"batch_size\": 64, \"device\": \"mps\", \"early_stopping\": false, \"epochs\": 20, \"freeze_backbone\": false, \"input_size\": [224, 224], \"learning_rate\": 0.0001, \"loss\": \"CrossEntropyLoss\", \"num_classes\": 10, \"optimizer\": \"AdamW\", \"pretrained\": true, \"pretrained_weights\": \"AlexNet_Weights.IMAGENET1K_V1\", \"random_seed\": 42, \"weight_decay\": 0.01}","VGG16","{\"architecture\": \"vgg16\", \"batch_size\": 64, \"device\": \"mps\", \"early_stopping\": false, \"epochs\": 20, \"freeze_backbone\": false, \"input_size\": [224, 224], \"learning_rate\": 0.0001, \"loss\": \"CrossEntropyLoss\", \"num_classes\": 10, \"optimizer\": \"AdamW\", \"pretrained\": true, \"pretrained_weights\": \"VGG16_Weights.IMAGENET1K_V1\", \"random_seed\": 42, \"weight_decay\": 0.01}","{\"architecture\": \"googlenet\", \"batch_size\": 64, \"device\": \"mps\", \"early_stopping\": false, \"epochs\": 20, \"freeze_backbone\": false, \"input_size\": [224, 224], \"learning_rate\": 0.001, \"loss\": \"CrossEntropyLoss\", \"num_classes\": 10, \"optimizer\": \"AdamW\", \"pretrained\": true, \"pretrained_weights\": \"GoogLeNet_Weights.IMAGENET1K_V1\", \"random_seed\": 42, \"weight_decay\": 0.01}","ResNet18","{\"architecture\": \"resnet18\", \"batch_size\": 64, \"device\": \"mps\", \"early_stopping\": false, \"epochs\": 20, \"freeze_backbone\": false, \"input_size\": [224, 224], \"learning_rate\": 0.001, \"loss\": \"CrossEntropyLoss\", \"num_classes\": 10, \"optimizer\": \"AdamW\", \"pretrained\": true, \"pretrained_weights\": \"ResNet18_Weights.IMAGENET1K_V1\", \"random_seed\": 42, \"weight_decay\": 0.01}","ResNet50","{\"architecture\": \"resnet50\", \"batch_size\": 64, \"device\": \"mps\", \"early_stopping\": false, \"epochs\": 20, \"freeze_backbone\": false, \"input_size\": [224, 224], \"learning_rate\": 0.001, \"loss\": \"CrossEntropyLoss\", \"num_classes\": 10, \"optimizer\": \"AdamW\", \"pretrained\": true, \"pretrained_weights\": \"ResNet50_Weights.IMAGENET1K_V2\", \"random_seed\": 42, \"weight_decay\": 0.01}","DenseNet121","{\"architecture\": \"densenet121\", \"batch_size\": 64, \"device\": \"mps\", \"early_stopping\": false, \"epochs\": 20, \"freeze_backbone\": false, \"input_size\": [224, 224], \"learning_rate\": 0.001, \"loss\": \"CrossEntropyLoss\", \"num_classes\": 10, \"optimizer\": \"AdamW\", \"pretrained\": true, \"pretrained_weights\": \"DenseNet121_Weights.IMAGENET1K_V1\", \"random_seed\": 42, \"weight_decay\": 0.01}","MobileNetV3-Large","{\"architecture\": \"mobilenet_v3_large\", \"batch_size\": 64, \"device\": \"mps\", \"early_stopping\": false, \"epochs\": 20, \"freeze_backbone\": false, \"input_size\": [224, 224], \"learning_rate\": 0.001, \"loss\": \"CrossEntropyLoss\", \"num_classes\": 10, \"optimizer\": \"AdamW\", \"pretrained\": true, \"pretrained_weights\": \"MobileNet_V3_Large_Weights.IMAGENET1K_V2\", \"random_seed\": 42, \"weight_decay\": 0.01}","{\"architecture\": \"efficientnet_b0\", \"batch_size\": 64, \"device\": \"mps\", \"early_stopping\": false, \"epochs\": 20, \"freeze_backbone\": false, \"input_size\": [224, 224], \"learning_rate\": 0.001, \"loss\": \"CrossEntropyLoss\", \"num_classes\": 10, \"optimizer\": \"AdamW\", \"pretrained\": true, \"pretrained_weights\": \"EfficientNet_B0_Weights.IMAGENET1K_V1\", \"random_seed\": 42, \"weight_decay\": 0.01}","{\"architecture\": \"convnext_tiny\", \"batch_size\": 64, \"device\": \"mps\", \"early_stopping\": false, \"epochs\": 20, \"freeze_backbone\": false, \"input_size\": [224, 224], \"learning_rate\": 0.001, \"loss\": \"CrossEntropyLoss\", \"num_classes\": 10, \"optimizer\": \"AdamW\", \"pretrained\": true, \"pretrained_weights\": \"ConvNeXt_Tiny_Weights.IMAGENET1K_V1\", \"random_seed\": 42, \"weight_decay\": 0.01}","YOLO Classification","{\"architecture\": \"yolo_cls\", \"batch\": 64, \"environment\": {\"numpy_version\": \"2.5.3\", \"python_version\": \"3.14.3\", \"torch_version\": \"2.14.0\", \"ultralytics_version\": \"8.4.154\"}, \"epochs\": 20, \"imgsz\": 224, \"input_size\": [224, 224], \"lr0\": 0.001, \"model\": \"yolo11n-cls.pt\", \"num_classes\": 10, \"optimizer\": \"AdamW\", \"pretrained\": true, \"pretrained_weights\": \"yolo11n-cls.pt\", \"seed\": 42}","https://raw.githubusercontent.com/srcollins785/Samuel_Collins_CV_Benchmarking/d7e28af8d2a607c33ebcfb8493fce8e73259cb52/benchmark_results/animals10_n500_grayscale/benchmark_summary.csv","https://raw.githubusercontent.com/srcollins785/Samuel_Collins_CV_Benchmarking/d7e28af8d2a607c33ebcfb8493fce8e73259cb52/benchmark_results/intel_n500_rgb/benchmark_summary.csv","https://raw.githubusercontent.com/srcollins785/Samuel_Collins_CV_Benchmarking/d7e28af8d2a607c33ebcfb8493fce8e73259cb52/benchmark_results/intel_n500_grayscale/benchmark_summary.csv","HunyuanOCR-1.5","v0.1.0; 177/177 valid pairs; batch1 deterministic; seed20260825; hunyuanocr; NVIDIA L4","https://raw.githubusercontent.com/sahrasayed3-crypto/clouda-ocr-benchmark/27eebc0403c23f15e91575207de51eaa5b6776da/summaries/FINAL_RESULTS.csv","MBZUAI/AIN-7B","v0.1.0; 177/177 valid pairs; batch1 deterministic; seed20260825; ain_native_fa2_96gb_20260826; NVIDIA RTX PRO 6000 Blackwell Server Edition","Qari OCR 0.4.0","v0.1.0; 177/177 valid pairs; batch1 deterministic; seed20260825; qari_native_full; NVIDIA L4","v0.1.0; 177/177 valid pairs; batch1 deterministic; seed20260825; qwen3_vl_4b; NVIDIA L4","v0.1.0; 177/177 valid pairs; batch1 deterministic; seed20260825; deepseek_ocr_2; NVIDIA L4","Arabic Nougat Large","v0.1.0; 177/177 valid pairs; batch1 deterministic; seed20260825; arabic_nougat; NVIDIA L4","2026-10-22; 96 headline items; judge deepseek/deepseek-v3.1; rubric m1.3.0; max_tokens32000; temperature0","https://raw.githubusercontent.com/kcelestinomaria/overlooked-bench/065af988ef5059380564f32f11b31fe221323a05/data/runs/2026-10-22/summary.json","deepseek/deepseek-v3.2","google/gemini-2.5-pro","openai/gpt-5.4","2026-10-22; 24 headline items; judge deepseek/deepseek-v3.1; rubric m1.3.0; max_tokens32000; temperature0","2026-10-22; 15 headline items; judge deepseek/deepseek-v3.1; rubric m1.3.0; max_tokens32000; temperature0","2026-10-22; 12 headline items; judge deepseek/deepseek-v3.1; rubric m1.3.0; max_tokens32000; temperature0","2026-10-22; 18 headline items; judge deepseek/deepseek-v3.1; rubric m1.3.0; max_tokens32000; temperature0","codex-cli; effort=max; seed42; 2026-07-14T23:26:49.312968+00:00; benchmark0.4.0","https://github.com/VSBDev/little-hermione/tree/2698f9d1ea90a53d7b23142a61c7ba662a7f7720/results","gemma4:12b-it-qat","ollama; effort=None; seed42; 2026-07-14T23:28:39.959549+00:00; benchmark0.4.0","gemma4:26b-a4b-it-qat","ollama; effort=None; seed42; 2026-07-14T23:29:15.732228+00:00; benchmark0.4.0","gemma4:31b","ollama; effort=None; seed42; 2026-07-14T23:29:59.256712+00:00; benchmark0.4.0","qwen3-coder-next:q4_K_M","ollama; effort=None; seed42; 2026-07-14T23:34:49.125643+00:00; benchmark0.4.0","gemma4:e4b-it-qat","ollama; effort=None; seed42; 2026-07-14T23:38:10.270735+00:00; benchmark0.4.0","codex-cli; effort=xhigh; seed42; 2026-07-15T00:33:53.915842+00:00; benchmark0.4.0","qwen3.5:0.8b","ollama; effort=None; seed42; 2026-07-15T01:43:04.289826+00:00; benchmark0.4.0","agy-cli; effort=None; seed42; 2026-07-15T15:59:17.983116+00:00; benchmark0.4.0","Gemini 3.1 Pro (High)","agy-cli; effort=None; seed42; 2026-07-15T16:03:37.848207+00:00; benchmark0.4.0","baseline; 100 tasks x2 repeats=200 responses; extension-v1; judge GPT-5.6-Luna","https://raw.githubusercontent.com/junru-zhu/failure-transparent-agents/0cb798d4aa78f123b5c7b1b2b26684e776d90301/data/model_extension_summary.json","mistral.ministral-3-8b-instruct","nvidia.nemotron-super-3-120b","us.amazon.nova-micro-v1:0","us.meta.llama3-1-8b-instruct-v1:0","us.openai.gpt-5.6-terra","transparency; 100 tasks x2 repeats=200 responses; extension-v1; judge GPT-5.6-Luna","evidence_contract; 100 tasks x2 repeats=200 responses; extension-v1; judge GPT-5.6-Luna","DistilBERT","Shared1377held-out reviews; 613negative/764positive; train4129/validation1377; seed42; distilbert-base-uncased; maxlen256; 3epochs; lr2e-5","https://raw.githubusercontent.com/qbalhkim/steam-review-sentiment/b0463b9bb889f75df0ed66d43bc2ae6b2ff58b7b/notebooks/02_deep_learning_benchmark.ipynb","Shared1377held-out reviews; 613negative/764positive; train4129/validation1377; seed42; TF-IDF train-only fit;5000features; unigram+bigram","Naïve Bayes","Support Vector Machine (LinearSVC)","BiLSTM","Shared1377held-out reviews; 613negative/764positive; train4129/validation1377; seed42; Embedding128; bidirectionalLSTM64; 200tokens; max8epochs; earlystop2","BiGRU","Shared1377held-out reviews; 613negative/764positive; train4129/validation1377; seed42; Embedding128; bidirectionalGRU64; 200tokens; max8epochs; earlystop2","Qwen/Qwen3.5-9B","v3-multilingual; fixed native decision rule; 85x300; Qwen/Qwen3.5-9B","https://huggingface.co/datasets/NoeFlandre/benchmark-llms-landuse-relevance/resolve/main/aggregates.csv","LiquidAI/LFM2.5-2.6B","v3-multilingual; fixed native decision rule; 85x300; LiquidAI/LFM2.5-2.6B@sglang-throughput-b16","v3-multilingual; fixed native decision rule; 85x300; LiquidAI/LFM2.5-2.6B+DSpark-throughput-b16","v3-multilingual; fixed native decision rule; 85x300; LiquidAI/LFM2.5-2.6B","Qwen/Qwen3-4B-Instruct-2507","v3-multilingual; fixed native decision rule; 85x300; Qwen/Qwen3-4B-Instruct-2507","Qwen/Qwen3-4B","v3-multilingual; fixed native decision rule; 85x300; Qwen/Qwen3-4B","google/gemma-4-E4B-it","v3-multilingual; fixed native decision rule; 85x300; google/gemma-4-E4B-it","Qwen/Qwen3-8B","v3-multilingual; fixed native decision rule; 85x300; Qwen/Qwen3-8B","Qwen/Qwen3.5-0.8B","v3-multilingual; fixed native decision rule; 85x300; Qwen/Qwen3.5-0.8B","LiquidAI/LFM2.5-8B-A1B","v3-multilingual; fixed native decision rule; 85x300; LiquidAI/LFM2.5-8B-A1B","v3-multilingual; fixed native decision rule; 85x300; LiquidAI/LFM2.5-8B-A1B+DSpark-throughput-b16","v3-multilingual; fixed native decision rule; 85x300; LiquidAI/LFM2.5-2.6B@logprob","LiquidAI/LFM2.5-350M","v3-multilingual; fixed native decision rule; 85x300; LiquidAI/LFM2.5-350M","Qwen/Qwen3-0.6B","v3-multilingual; fixed native decision rule; 85x300; Qwen/Qwen3-0.6B","MoritzLaurer/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7","v3-multilingual; fixed native decision rule; 85x300; MoritzLaurer/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7","BalaRajesh1/mmbert-small-nli","v3-multilingual; fixed native decision rule; 85x300; BalaRajesh1/mmbert-small-nli","fastino/gliner2.5-multi-v1","v3-multilingual; fixed native decision rule; 85x300; fastino/gliner2.5-multi-v1","v3-multilingual; fixed native decision rule; 85x300; LiquidAI/LFM2.5-8B-A1B@sglang-throughput-b16","convaiinnovations/laya-multilingual","v3-multilingual; fixed native decision rule; 85x300; convaiinnovations/laya-multilingual","Alibaba-NLP/gte-multilingual-reranker-base","v3-multilingual; fixed native decision rule; 85x300; Alibaba-NLP/gte-multilingual-reranker-base","Qwen/Qwen3.5-4B","v3-multilingual; fixed native decision rule; 85x300; Qwen/Qwen3.5-4B","mistralai/Ministral-3-8B-Instruct-2512-BF16","v3-multilingual; fixed native decision rule; 85x300; mistralai/Ministral-3-8B-Instruct-2512-BF16","google/gemma-4-E2B-it","v3-multilingual; fixed native decision rule; 85x300; google/gemma-4-E2B-it","LiquidAI/LFM2.5-1.2B-Instruct","v3-multilingual; fixed native decision rule; 85x300; LiquidAI/LFM2.5-1.2B-Instruct","v3-multilingual; fixed native decision rule; 85x300; LiquidAI/LFM2.5-1.2B-Instruct+DSpark-throughput-b16","v3-multilingual; fixed native decision rule; 85x300; LiquidAI/LFM2.5-1.2B-Instruct@sglang-throughput-b16","LiquidAI/LFM2.5-VL-3B","v3-multilingual; fixed native decision rule; 85x300; LiquidAI/LFM2.5-VL-3B","v3-multilingual; fixed native decision rule; 85x300; LiquidAI/LFM2.5-VL-3B+DSpark-throughput-b16","v3-multilingual; fixed native decision rule; 85x300; LiquidAI/LFM2.5-VL-3B@sglang-throughput-b16","microsoft/Phi-4-mini-instruct","v3-multilingual; fixed native decision rule; 85x300; microsoft/Phi-4-mini-instruct","HuggingFaceTB/SmolLM3-3B","v3-multilingual; fixed native decision rule; 85x300; HuggingFaceTB/SmolLM3-3B","tiiuae/Falcon3-3B-Instruct","v3-multilingual; fixed native decision rule; 85x300; tiiuae/Falcon3-3B-Instruct","ibm-granite/granite-3.3-2b-instruct","v3-multilingual; fixed native decision rule; 85x300; ibm-granite/granite-3.3-2b-instruct","unsloth/Qwen3.8-27B-GGUF","v3-multilingual; fixed native decision rule; 85x300; unsloth/Qwen3.8-27B-GGUF@UD-IQ2_XXS","allenai/Olmo-3-7B-Instruct","v3-multilingual; fixed native decision rule; 85x300; allenai/Olmo-3-7B-Instruct","Qwen/Qwen3.5-2B","v3-multilingual; fixed native decision rule; 85x300; Qwen/Qwen3.5-2B","mistralai/Ministral-3-3B-Instruct-2512-BF16","v3-multilingual; fixed native decision rule; 85x300; mistralai/Ministral-3-3B-Instruct-2512-BF16","utter-project/EuroLLM-9B-Instruct-2512","v3-multilingual; fixed native decision rule; 85x300; utter-project/EuroLLM-9B-Instruct-2512","tiiuae/Falcon3-7B-Instruct","v3-multilingual; fixed native decision rule; 85x300; tiiuae/Falcon3-7B-Instruct","MoritzLaurer/bge-m3-zeroshot-v2.0","v3-multilingual; fixed native decision rule; 85x300; MoritzLaurer/bge-m3-zeroshot-v2.0","knowledgator/gliclass-multilang-mini","v3-multilingual; fixed native decision rule; 85x300; knowledgator/gliclass-multilang-mini","ibm-granite/granite-4.1-3b","v3-multilingual; fixed native decision rule; 85x300; ibm-granite/granite-4.1-3b","allenai/OLMo-2-1124-7B-Instruct","v3-multilingual; fixed native decision rule; 85x300; allenai/OLMo-2-1124-7B-Instruct","Qwen/Qwen3-1.7B","v3-multilingual; fixed native decision rule; 85x300; Qwen/Qwen3-1.7B","tiiuae/Falcon-H1-3B-Instruct","v3-multilingual; fixed native decision rule; 85x300; tiiuae/Falcon-H1-3B-Instruct","swiss-ai/Apertus-8B-Instruct-2509","v3-multilingual; fixed native decision rule; 85x300; swiss-ai/Apertus-8B-Instruct-2509","Qwen/Qwen3-Reranker-4B","v3-multilingual; fixed native decision rule; 85x300; Qwen/Qwen3-Reranker-4B","tiiuae/Falcon3-1B-Instruct","v3-multilingual; fixed native decision rule; 85x300; tiiuae/Falcon3-1B-Instruct","Qwen/Qwen3-Reranker-0.6B","v3-multilingual; fixed native decision rule; 85x300; Qwen/Qwen3-Reranker-0.6B","mixedbread-ai/mxbai-rerank-base-v2","v3-multilingual; fixed native decision rule; 85x300; mixedbread-ai/mxbai-rerank-base-v2","LiquidAI/d1-3B","v3-multilingual; fixed native decision rule; 85x300; LiquidAI/d1-3B","knowledgator/gliclass-multilang-ultra","v3-multilingual; fixed native decision rule; 85x300; knowledgator/gliclass-multilang-ultra","native video;1fps; same800evaluation pairs","https://arxiv.org/pdf/2609.09895v1","frames1fps; same800evaluation pairs","LLaVA-NeXT-Video7B","CLIP ViT-L/14;32frames; same800evaluation pairs","Qwen3-VL-Instruct8B","Qwen3-VL-ViT;32frames; same800evaluation pairs","Gemma-4-it12B","Unified encoder;32frames; same800evaluation pairs","InternVL3.5-14.8B","InternViT300M;32frames; same800evaluation pairs","Qwen3.5VisionEncoder;32frames; same800evaluation pairs","Qwen3-Omni-Instruct30B-A3B","Qwen3OmniViT;32frames; same800evaluation pairs","VideoAgent","original video-agent configuration; same800evaluation pairs","VideoHV-Agent","original agent;1fps; same800evaluation pairs","Deep Video Discovery","original agent;2fps; same800evaluation pairs","PAC-S","CLIP ViT-B/32;validation threshold0.7559156418; same800evaluation pairs","EMScore","CLIP ViT-B/32;validation threshold0.2649626136; same800evaluation pairs","Owl-Con7B","mPLUG-Owl7BVideo;CLIPViTL/14;32frames;threshold0.53125; same800evaluation pairs","FIFA","original detector;1fps;threshold0.8; same800evaluation pairs","camera; L1; radius2; 4suites x10tasks x10configurations; 400rollouts; directory=pi05_eval","https://raw.githubusercontent.com/hiroki-oist/LIBERO-ctrl/2c866ac70b4efda3bfb641bdfd9c67c97b383628/analysis/manuscript_numbers.json","camera; L1; radius2; 4suites x10tasks x10configurations; 400rollouts; directory=oft_eval","camera; L1; radius2; 4suites x10tasks x10configurations; 400rollouts; directory=univla_eval","camera; L1; radius2; 4suites x10tasks x10configurations; 400rollouts; directory=smolvla_eval","VLA-JEPA","camera; L1; radius2; 4suites x10tasks x10configurations; 400rollouts; directory=vlajepa_eval","PredVLA","camera; L1; radius2; 4suites x10tasks x10configurations; 400rollouts; directory=predvla_s13_eval","camera; L2; radius4; 4suites x10tasks x10configurations; 400rollouts; directory=pi05_eval","camera; L2; radius4; 4suites x10tasks x10configurations; 400rollouts; directory=oft_eval","camera; L2; radius4; 4suites x10tasks x10configurations; 400rollouts; directory=univla_eval","camera; L2; radius4; 4suites x10tasks x10configurations; 400rollouts; directory=smolvla_eval","camera; L2; radius4; 4suites x10tasks x10configurations; 400rollouts; directory=vlajepa_eval","camera; L2; radius4; 4suites x10tasks x10configurations; 400rollouts; directory=predvla_s13_eval","camera; L3; radius8; 4suites x10tasks x10configurations; 400rollouts; directory=pi05_eval","camera; L3; radius8; 4suites x10tasks x10configurations; 400rollouts; directory=oft_eval","camera; L3; radius8; 4suites x10tasks x10configurations; 400rollouts; directory=univla_eval","camera; L3; radius8; 4suites x10tasks x10configurations; 400rollouts; directory=smolvla_eval","camera; L3; radius8; 4suites x10tasks x10configurations; 400rollouts; directory=vlajepa_eval","camera; L3; radius8; 4suites x10tasks x10configurations; 400rollouts; directory=predvla_s13_eval","lighting; L1; radius2; 4suites x10tasks x10configurations; 400rollouts; directory=pi05_eval","lighting; L1; radius2; 4suites x10tasks x10configurations; 400rollouts; directory=oft_eval","lighting; L1; radius2; 4suites x10tasks x10configurations; 400rollouts; directory=univla_eval","lighting; L1; radius2; 4suites x10tasks x10configurations; 400rollouts; directory=smolvla_eval","lighting; L1; radius2; 4suites x10tasks x10configurations; 400rollouts; directory=vlajepa_eval","lighting; L1; radius2; 4suites x10tasks x10configurations; 400rollouts; directory=predvla_s13_eval","lighting; L2; radius4; 4suites x10tasks x10configurations; 400rollouts; directory=pi05_eval","lighting; L2; radius4; 4suites x10tasks x10configurations; 400rollouts; directory=oft_eval","lighting; L2; radius4; 4suites x10tasks x10configurations; 400rollouts; directory=univla_eval","lighting; L2; radius4; 4suites x10tasks x10configurations; 400rollouts; directory=smolvla_eval","lighting; L2; radius4; 4suites x10tasks x10configurations; 400rollouts; directory=vlajepa_eval","lighting; L2; radius4; 4suites x10tasks x10configurations; 400rollouts; directory=predvla_s13_eval","lighting; L3; radius8; 4suites x10tasks x10configurations; 400rollouts; directory=pi05_eval","lighting; L3; radius8; 4suites x10tasks x10configurations; 400rollouts; directory=oft_eval","lighting; L3; radius8; 4suites x10tasks x10configurations; 400rollouts; directory=univla_eval","lighting; L3; radius8; 4suites x10tasks x10configurations; 400rollouts; directory=smolvla_eval","lighting; L3; radius8; 4suites x10tasks x10configurations; 400rollouts; directory=vlajepa_eval","lighting; L3; radius8; 4suites x10tasks x10configurations; 400rollouts; directory=predvla_s13_eval","robot; L1; radius2; 4suites x10tasks x10configurations; 400rollouts; directory=pi05_eval","robot; L1; radius2; 4suites x10tasks x10configurations; 400rollouts; directory=oft_eval","robot; L1; radius2; 4suites x10tasks x10configurations; 400rollouts; directory=univla_eval","robot; L1; radius2; 4suites x10tasks x10configurations; 400rollouts; directory=smolvla_eval","robot; L1; radius2; 4suites x10tasks x10configurations; 400rollouts; directory=vlajepa_eval","robot; L1; radius2; 4suites x10tasks x10configurations; 400rollouts; directory=predvla_s13_eval","robot; L2; radius4; 4suites x10tasks x10configurations; 400rollouts; directory=pi05_eval","robot; L2; radius4; 4suites x10tasks x10configurations; 400rollouts; directory=oft_eval","robot; L2; radius4; 4suites x10tasks x10configurations; 400rollouts; directory=univla_eval","robot; L2; radius4; 4suites x10tasks x10configurations; 400rollouts; directory=smolvla_eval","robot; L2; radius4; 4suites x10tasks x10configurations; 400rollouts; directory=vlajepa_eval","robot; L2; radius4; 4suites x10tasks x10configurations; 400rollouts; directory=predvla_s13_eval","robot; L3; radius8; 4suites x10tasks x10configurations; 400rollouts; directory=pi05_eval","robot; L3; radius8; 4suites x10tasks x10configurations; 400rollouts; directory=oft_eval","robot; L3; radius8; 4suites x10tasks x10configurations; 400rollouts; directory=univla_eval","robot; L3; radius8; 4suites x10tasks x10configurations; 400rollouts; directory=smolvla_eval","robot; L3; radius8; 4suites x10tasks x10configurations; 400rollouts; directory=vlajepa_eval","robot; L3; radius8; 4suites x10tasks x10configurations; 400rollouts; directory=predvla_s13_eval","sensor; L1; radius2; 4suites x10tasks x10configurations; 400rollouts; directory=pi05_eval","sensor; L1; radius2; 4suites x10tasks x10configurations; 400rollouts; directory=oft_eval","sensor; L1; radius2; 4suites x10tasks x10configurations; 400rollouts; directory=univla_eval","sensor; L1; radius2; 4suites x10tasks x10configurations; 400rollouts; directory=smolvla_eval","sensor; L1; radius2; 4suites x10tasks x10configurations; 400rollouts; directory=vlajepa_eval","sensor; L1; radius2; 4suites x10tasks x10configurations; 400rollouts; directory=predvla_s13_eval","sensor; L2; radius4; 4suites x10tasks x10configurations; 400rollouts; directory=pi05_eval","sensor; L2; radius4; 4suites x10tasks x10configurations; 400rollouts; directory=oft_eval","sensor; L2; radius4; 4suites x10tasks x10configurations; 400rollouts; directory=univla_eval","sensor; L2; radius4; 4suites x10tasks x10configurations; 400rollouts; directory=smolvla_eval","sensor; L2; radius4; 4suites x10tasks x10configurations; 400rollouts; directory=vlajepa_eval","sensor; L2; radius4; 4suites x10tasks x10configurations; 400rollouts; directory=predvla_s13_eval","sensor; L3; radius8; 4suites x10tasks x10configurations; 400rollouts; directory=pi05_eval","sensor; L3; radius8; 4suites x10tasks x10configurations; 400rollouts; directory=oft_eval","sensor; L3; radius8; 4suites x10tasks x10configurations; 400rollouts; directory=univla_eval","sensor; L3; radius8; 4suites x10tasks x10configurations; 400rollouts; directory=smolvla_eval","sensor; L3; radius8; 4suites x10tasks x10configurations; 400rollouts; directory=vlajepa_eval","sensor; L3; radius8; 4suites x10tasks x10configurations; 400rollouts; directory=predvla_s13_eval","actuation; L1; radius2; 4suites x10tasks x10configurations; 400rollouts; directory=pi05_eval","actuation; L1; radius2; 4suites x10tasks x10configurations; 400rollouts; directory=oft_eval","actuation; L1; radius2; 4suites x10tasks x10configurations; 400rollouts; directory=univla_eval","actuation; L1; radius2; 4suites x10tasks x10configurations; 400rollouts; directory=smolvla_eval","actuation; L1; radius2; 4suites x10tasks x10configurations; 400rollouts; directory=vlajepa_eval","actuation; L1; radius2; 4suites x10tasks x10configurations; 400rollouts; directory=predvla_s13_eval","actuation; L2; radius4; 4suites x10tasks x10configurations; 400rollouts; directory=pi05_eval","actuation; L2; radius4; 4suites x10tasks x10configurations; 400rollouts; directory=oft_eval","actuation; L2; radius4; 4suites x10tasks x10configurations; 400rollouts; directory=univla_eval","actuation; L2; radius4; 4suites x10tasks x10configurations; 400rollouts; directory=smolvla_eval","actuation; L2; radius4; 4suites x10tasks x10configurations; 400rollouts; directory=vlajepa_eval","actuation; L2; radius4; 4suites x10tasks x10configurations; 400rollouts; directory=predvla_s13_eval","actuation; L3; radius8; 4suites x10tasks x10configurations; 400rollouts; directory=pi05_eval","actuation; L3; radius8; 4suites x10tasks x10configurations; 400rollouts; directory=oft_eval","actuation; L3; radius8; 4suites x10tasks x10configurations; 400rollouts; directory=univla_eval","actuation; L3; radius8; 4suites x10tasks x10configurations; 400rollouts; directory=smolvla_eval","actuation; L3; radius8; 4suites x10tasks x10configurations; 400rollouts; directory=vlajepa_eval","actuation; L3; radius8; 4suites x10tasks x10configurations; 400rollouts; directory=predvla_s13_eval","language; L1; radius2; 4suites x10tasks x10configurations; 400rollouts; directory=pi05_eval","language; L1; radius2; 4suites x10tasks x10configurations; 400rollouts; directory=oft_eval","language; L1; radius2; 4suites x10tasks x10configurations; 400rollouts; directory=univla_eval","language; L1; radius2; 4suites x10tasks x10configurations; 400rollouts; directory=smolvla_eval","language; L1; radius2; 4suites x10tasks x10configurations; 400rollouts; directory=vlajepa_eval","language; L1; radius2; 4suites x10tasks x10configurations; 400rollouts; directory=predvla_s13_eval","language; L2; radius4; 4suites x10tasks x10configurations; 400rollouts; directory=pi05_eval","language; L2; radius4; 4suites x10tasks x10configurations; 400rollouts; directory=oft_eval","language; L2; radius4; 4suites x10tasks x10configurations; 400rollouts; directory=univla_eval","language; L2; radius4; 4suites x10tasks x10configurations; 400rollouts; directory=smolvla_eval","language; L2; radius4; 4suites x10tasks x10configurations; 400rollouts; directory=vlajepa_eval","language; L2; radius4; 4suites x10tasks x10configurations; 400rollouts; directory=predvla_s13_eval","language; L3; radius8; 4suites x10tasks x10configurations; 400rollouts; directory=pi05_eval","language; L3; radius8; 4suites x10tasks x10configurations; 400rollouts; directory=oft_eval","language; L3; radius8; 4suites x10tasks x10configurations; 400rollouts; directory=univla_eval","language; L3; radius8; 4suites x10tasks x10configurations; 400rollouts; directory=smolvla_eval","language; L3; radius8; 4suites x10tasks x10configurations; 400rollouts; directory=vlajepa_eval","language; L3; radius8; 4suites x10tasks x10configurations; 400rollouts; directory=predvla_s13_eval","combination; L1; radius2; 4suites x10tasks x10configurations; 400rollouts; directory=pi05_eval","combination; L1; radius2; 4suites x10tasks x10configurations; 400rollouts; directory=oft_eval","combination; L1; radius2; 4suites x10tasks x10configurations; 400rollouts; directory=univla_eval","combination; L1; radius2; 4suites x10tasks x10configurations; 400rollouts; directory=smolvla_eval","combination; L1; radius2; 4suites x10tasks x10configurations; 400rollouts; directory=vlajepa_eval","combination; L1; radius2; 4suites x10tasks x10configurations; 400rollouts; directory=predvla_s13_eval","combination; L2; radius4; 4suites x10tasks x10configurations; 400rollouts; directory=pi05_eval","combination; L2; radius4; 4suites x10tasks x10configurations; 400rollouts; directory=oft_eval","combination; L2; radius4; 4suites x10tasks x10configurations; 400rollouts; directory=univla_eval","combination; L2; radius4; 4suites x10tasks x10configurations; 400rollouts; directory=smolvla_eval","combination; L2; radius4; 4suites x10tasks x10configurations; 400rollouts; directory=vlajepa_eval","combination; L2; radius4; 4suites x10tasks x10configurations; 400rollouts; directory=predvla_s13_eval","combination; L3; radius8; 4suites x10tasks x10configurations; 400rollouts; directory=pi05_eval","combination; L3; radius8; 4suites x10tasks x10configurations; 400rollouts; directory=oft_eval","combination; L3; radius8; 4suites x10tasks x10configurations; 400rollouts; directory=univla_eval","combination; L3; radius8; 4suites x10tasks x10configurations; 400rollouts; directory=smolvla_eval","combination; L3; radius8; 4suites x10tasks x10configurations; 400rollouts; directory=vlajepa_eval","combination; L3; radius8; 4suites x10tasks x10configurations; 400rollouts; directory=predvla_s13_eval","schema only; common 47 questions; 1 recorded trials per question; v1 pre-promotion; unversioned original Claude alias","https://github.com/gcorneliusvh/looker-vs-databricks-genie-benchmark/blob/84bd858e1fdd0178cce13ce007554e1153c6bd9b/results/scored.csv","claude-opus","claude-sonnet","schema only; common 47 questions; 5 recorded trials per question; v1 pre-promotion; original model identifier","gemini-2.5-flash-lite","schema only; common 47 questions; 3 recorded trials per question; v1 pre-promotion; original model identifier","document grounding; common 47 questions; 1 recorded trials per question; v1 pre-promotion; unversioned original Claude alias","document grounding; common 47 questions; 5 recorded trials per question; v1 pre-promotion; original model identifier","document grounding; common 47 questions; 3 recorded trials per question; v1 pre-promotion; original model identifier","model in prompt; common 47 questions; 1 recorded trials per question; v1 pre-promotion; unversioned original Claude alias","model in prompt; common 47 questions; 5 recorded trials per question; v1 pre-promotion; original model identifier","model in prompt; common 47 questions; 3 recorded trials per question; v1 pre-promotion; original model identifier","semantic compiler; common 47 questions; 1 recorded trials per question; v1 pre-promotion; unversioned original Claude alias","semantic compiler; common 47 questions; 5 recorded trials per question; v1 pre-promotion; original model identifier","semantic compiler; common 47 questions; 3 recorded trials per question; v1 pre-promotion; original model identifier","History; direct forecast","https://arxiv.org/html/2610.17794v1#S4.T1","Toto-2","Moirai-2","History + effects; direct forecast","History + effects; Agent: MoiraiAgent; GPT-5.6 Sol + Moirai-2 + code; max10 steps","Target frequency","198 queries; 20,431 proteins; non-neural reference","https://arxiv.org/html/2610.24458v1#S5.T1","Weighted k-NN","SEA (Morgan)","DrugCLIP","198 queries; 20,431 proteins; five-seed mean","HypSeek","LigUnity","ConPLex","GenFish","Metric3D","Image-only metric geometry estimation; no depth prior; normal","https://arxiv.org/html/2610.12423v1#S5.T3","MoGe-3","Depth Pro","Unidepth","MoGe-2","MetricAnything","Unidepthv2","Metric3D v2","Image-only metric geometry estimation; no depth prior; challenging","C0; neutral organization; 210 tasks; published shared runtime","https://arxiv.org/html/2610.11561v1#S5.T2","Agent: shared ReAct; Serper + Jina + code; max supported reasoning; three independent runs","https://arxiv.org/html/2610.18092v1#S4.T4","Jev 1.13.0","Identical Jev-targeted benign/adversarial context replay; original application decision boundary; ten-application macro mean","https://arxiv.org/html/2610.18612v1#A6.T11","Qwen2.5 32B","Gemma3 27B","Gemma3 4B","Llama3.3 70B","Llama3.1 8B","Llama3.2 3B","UI component implementation; 421 shared designs; native count-pooled micro-F1","https://arxiv.org/html/2610.10374v1#S5.T2","UI requirement inference; 421 shared designs; native count-pooled micro-F1","STCN","OLiVES training; common OLiVES test split; original VOS evaluation","https://arxiv.org/html/2610.20501v1#S3.T5","AOT","De-AOT","XMem","Cutie","OASIS","InrtlWrist (Ours)","frozen-probe; six matched methods; four-HAR-dataset equal mean","https://arxiv.org/html/2610.19481v1#S5.T2","LIMU-BERT","SimMTM","FOCAL","RelCon","SSL-Wearables","fine-tuning; six matched methods; four-HAR-dataset equal mean","Frozen encoder; canonical class prompt; fixed query-independent mean pooling","https://arxiv.org/html/2610.11329v1#A1.T1","KEEP","MI-Zero","MUSK","PathGen-CLIP","Patho-CLIP","QuiltNet","W2V2P-LV60","Published off-the-shelf model","https://arxiv.org/html/2610.20830v1#S6.T2","W2V2P-XLSR53","MultiIPA","ZIPA-CTC","ZIPA-CTC-NS","HuPER","PhoneticXeus","BabAR","Azure Speech","TIMIT-only fine-tuning; no SC/SW target-pattern training","Unigram","Pilbara cross-province test; cell-level 1m tokens; published method Unigram; runs 1","https://arxiv.org/html/2610.01204v1#A4.T9","Per-region unigram","Pilbara cross-province test; cell-level 1m tokens; published method Per-region unigram; runs 1","Persistence","Pilbara cross-province test; cell-level 1m tokens; published method Persistence; runs 1","Bigram","Pilbara cross-province test; cell-level 1m tokens; published method Bigram; runs 1","3-gram","Pilbara cross-province test; cell-level 1m tokens; published method 3 3 -gram; runs 1","Depth-Markov","Pilbara cross-province test; cell-level 1m tokens; published method Depth-Markov; runs 1","HMM ( K = 24 K{=}24 )","Pilbara cross-province test; cell-level 1m tokens; published method HMM ( K = 24 K{=}24 ); runs 1","Spatial k k -NN, train pool","Pilbara cross-province test; cell-level 1m tokens; published method Spatial k k -NN, train pool; runs 1","Spatial k k -NN, archive pool","Pilbara cross-province test; cell-level 1m tokens; published method Spatial k k -NN, archive pool; runs 1","Indicator kriging, train pool","Pilbara cross-province test; cell-level 1m tokens; published method Indicator kriging, train pool; runs 1","Indicator kriging, archive pool","Pilbara cross-province test; cell-level 1m tokens; published method Indicator kriging, archive pool; runs 1","SIS, train pool","Pilbara cross-province test; cell-level 1m tokens; published method SIS, train pool; runs 1","SIS, archive pool","Pilbara cross-province test; cell-level 1m tokens; published method SIS, archive pool; runs 1","MPS-lite","Pilbara cross-province test; cell-level 1m tokens; published method MPS-lite; runs 1","TPROGS","Pilbara cross-province test; cell-level 1m tokens; published method TPROGS; runs 1","Pilbara cross-province test; cell-level 1m tokens; published method XGBoost; runs 1","Pilbara cross-province test; cell-level 1m tokens; published method LSTM; runs 1","Pilbara cross-province test; cell-level 1m tokens; published method GRU; runs 1","Mamba","Pilbara cross-province test; cell-level 1m tokens; published method Mamba; runs 1","Pilbara cross-province test; cell-level 1m tokens; published method TCN; runs 1","Pilbara cross-province test; cell-level 1m tokens; published method Transformer; runs 1","LSTM + location","Pilbara cross-province test; cell-level 1m tokens; published method LSTM + location; runs 1","LSTM + geochem","Pilbara cross-province test; cell-level 1m tokens; published method LSTM + geochem; runs 1","LSTM + geochem + location","Pilbara cross-province test; cell-level 1m tokens; published method LSTM + geochem + location; runs 1","GRU + location","Pilbara cross-province test; cell-level 1m tokens; published method GRU + location; runs 1","GRU + geochem","Pilbara cross-province test; cell-level 1m tokens; published method GRU + geochem; runs 1","GRU + geochem + location","Pilbara cross-province test; cell-level 1m tokens; published method GRU + geochem + location; runs 1","Transformer + geochem + location","Pilbara cross-province test; cell-level 1m tokens; published method Transformer + geochem + location; runs 1","Mamba + geochem + location","Pilbara cross-province test; cell-level 1m tokens; published method Mamba + geochem + location; runs 1","TCN + geochem + location","Pilbara cross-province test; cell-level 1m tokens; published method TCN + geochem + location; runs 1","Layer marginal","Pilbara cross-province test; layer-level stratum tokens; published method Layer marginal; runs 1","Layer Markov","Pilbara cross-province test; layer-level stratum tokens; published method Layer Markov; runs 1","Pilbara cross-province test; layer-level stratum tokens; published method GRU; runs 10","Pilbara cross-province test; layer-level stratum tokens; published method Transformer; runs 10","GRU + R","Pilbara cross-province test; layer-level stratum tokens; published method GRU + R; runs 10","GRU + PT","Pilbara cross-province test; layer-level stratum tokens; published method GRU + PT; runs 10","GRU + PT + R (ours)","Pilbara cross-province test; layer-level stratum tokens; published method GRU + PT + R (ours); runs 10","Agent: function calling; query-tools; 150 entries × five runs","https://arxiv.org/html/2610.03356v1#S4.T4","Qwen3-80B-A3B","Agent: function calling; full-tools; 150 entries × five runs","Agent: Codex; standard screenshot stream","https://arxiv.org/html/2610.11050v1#S4.T2","Agent: Gemini CLI; standard screenshot stream","Agent: Claude Code; standard screenshot stream","Agent: OpenCode; standard screenshot stream","Qwen 3.6 35B-A3B","Direct: Direct; 2×2 frame mosaic","Direct: Direct; standard screenshot stream","VARS+XVARS","Published query-side instructions and pretrained encoders; common referee-approved precedent pool","https://arxiv.org/html/2610.09742v1#S4.T3","InternVideo2-1B","InternVideo2-6B","Qwen3-VL-Emb-2B-FT","Qwen3-VL-Emb-2B","VLM2Vec-V2","Qwen3-VL-Emb-8B","Published deterministic inference; temperature 0; top-k 1","https://arxiv.org/html/2610.20548v1#S4.T5","Thinking disabled; temperature 0; top-k 1","Thinking enabled; temperature 0; top-k 1","Jev","Fixed curated 374-candidate injection set; complete coverage only; original-task correctness","https://arxiv.org/html/2610.19118v1#A3.T9","KEV","OpenJEV","Bosun","Liquid D1","GEV-26B","Mica-4B","OneJev-4B","OneJev-9B","OneJev-27B","Jev-Omni","StandardOne-3B","StandardOne-8B","KEV-27B","Liquid d1-3B","PPLX-Decider-27B","InnerJev-27B","GLiNER2.5-340M","Imajev-4B","Fixed 223-pair in-image injection; Choice; original-task correctness","https://arxiv.org/html/2610.19118v1#A3.T10","Visual-Jev-4B","NeoHorse-4B","Strands-2B","Imajev-9B","OpenJEV-27B","Train/validation-only selection; frozen protected test","https://arxiv.org/html/2610.08571v1#S4.T3","TF-IDF SVM","TF-IDF LR","TF-IDF RF","MiniLM Ref.","DeBERTa-v3","Keyword","ACT","Sim-only training; L1; standard tools; BallMove; 100 matched-seed episodes","https://arxiv.org/pdf/2610.02089v1","DP","Cosmos 3 Edge Policy","Sim-only training; L1; standard tools; BallRetrieve; 100 matched-seed episodes","Sim-only training; L1; standard tools; IceBreak; 100 matched-seed episodes","Four images/task; no prompt enhancement; 240 shared tasks; native generation resolution; 1024px OCR","https://arxiv.org/pdf/2610.23374v1","Qwen Image 3.0 Pro","Krea 2 Raw","Z-Image Base","SenseNova U1","ERNIE Image","Ideogram 4.0","Qwen Image 2512","MOSS-Image","Agent: claude-agent-sdk; maximum reasoning/context; up to five relaunch rounds","https://arxiv.org/pdf/2610.19159v1","Agent: codex-sdk; maximum reasoning/context; up to five relaunch rounds","Agent: deepseek-harness; maximum reasoning/context; up to five relaunch rounds","Agent: kimi-agent-sdk; maximum reasoning/context; up to five relaunch rounds","Agent: software-agent-sdk; maximum reasoning/context; up to five relaunch rounds","Original pretrained model; source training protocol","https://arxiv.org/pdf/2610.17661v1","Intern3-VL-8B","Intern3.5-VL-8B","Retrained on benchmark train splits","DeQA-Score","VisualQuality-R1","RAPO","SCPA","SolarWM-H3","Camera/action native adapter","https://arxiv.org/pdf/2610.20693v1","Lyra2","EVOKE","Echo-WM","Gen3C","Matrix game 3.5","Genie3","LingBot-World v1","Voyager","FantasyWorld","Astra","Seedance2.5","Text + initial-image adapter","Cosmos3-Super","HunyuanVideo1.5","LTX2.5","BRIEF-Pro","Default sentence budget; dedicated compressor","https://arxiv.org/html/2610.26827v1#S4.T3","RECOMP (extractive)","EXIT","Default sentence budget; ICL; thinking disabled; temperature 0","Neutral P0; active deception; fresh context; temperature 0 where supported","https://arxiv.org/html/2610.13626v1#S4.T2","Neutral P0; passive deception; fresh context; temperature 0 where supported","Claude Opus-5","Agent: Claude Code; native configuration; three runs per evidence condition","https://arxiv.org/html/2610.08662v1#S6.T2","Agent: Codex; native configuration; three runs per evidence condition","Standalone clarification; fixed GPT-5.6-Sol simulator (temperature 0.7); target temperature 0; max twenty rounds; three personas","https://arxiv.org/html/2610.13022v1#S5.T4","1T1V; 576-step history; H=12/24 mean","https://arxiv.org/html/2609.24444v1#S5.T3","XLinear","DUET","xPatch","Timer","TTM","TOTO","SEMPO","STID","TGGC","BigST","FUTP","1T1V; 576-step history; H=72/144 mean","1TMV; 576-step history; H=12/24 mean","VAR","1TMV; 576-step history; H=72/144 mean","MT1V; 576-step history; H=12/24 mean","https://arxiv.org/html/2609.24444v1#S5.T4","STDN","OpenC","FSTA","MT1V; 576-step history; H=72/144 mean","MTMV; 576-step history; H=12/24 mean","MTMV; 576-step history; H=72/144 mean","Paper default inference; expert-derived task-specific evidence rubrics","https://arxiv.org/html/2609.23435v1#S2.T1","ChatMultiOmics","ChatNT","NatureLM","SciReasoner","Zero-shot; single-paragraph","https://arxiv.org/html/2609.24613v1#S5.T5","nomic-embed-text-v1.5","snowflake-arctic-embed-l-v2.0","dinghy-law-4b-v1","jasper-token-compression-600M","LGAI-Embedding-Preview","BERT","Fine-tuned; three epochs; single-paragraph","Legal-BERT","Zero-shot; contextual","Fine-tuned; three epochs; contextual","Common longitudinal multi-view MRI test; published final answer protocol","https://arxiv.org/html/2608.13309v1#S3.T1","InternVL3.5-Inst","Qwen3-VL-Plus-Thinking","Qwen3-VL-235B-Thinking","Qwen3-VL-8B-Inst","Llama-4-Scout-17B-Inst","Llama-4-Maverick-17B-Inst","MedGemma-27B-IT","MedGemma-1.5-4B-IT","Generative VLM MCQ answering","https://arxiv.org/html/2608.08727v1#S4.T1","SmolVLM2-2.2B","LFM2.5-VL-1.6B","Contrastive encoder MCQ scoring","CLIP-ViT-L-14","SCOLD","BioCLIP-ViT-B-32","LlamaGuard-1 7B","Params 7B; combined TRACE English/Chinese reasoning-trace region","https://arxiv.org/html/2608.24232v1#S3.T2","LlamaGuard-2 8B","Params 8B; combined TRACE English/Chinese reasoning-trace region","LlamaGuard-3 1B","Params 1B; combined TRACE English/Chinese reasoning-trace region","LlamaGuard-3 8B","LlamaGuard-4 12B","Params 12B; combined TRACE English/Chinese reasoning-trace region","ShieldGemma 2B","Params 2B; combined TRACE English/Chinese reasoning-trace region","ShieldGemma 9B","Params 9B; combined TRACE English/Chinese reasoning-trace region","WildGuard 7B","NemotronGuard 8B","Octopus 14B","Params 14B; combined TRACE English/Chinese reasoning-trace region","GPTSafeGuard 20B","Params 20B; combined TRACE English/Chinese reasoning-trace region","Qwen3Guard 0.6B","Params 0.6B; combined TRACE English/Chinese reasoning-trace region","Qwen3Guard 4B","Params 4B; combined TRACE English/Chinese reasoning-trace region","Qwen3Guard 8B","PolyGuard 0.5B","Params 0.5B; combined TRACE English/Chinese reasoning-trace region","PolyGuard 8B","YuFeng-XGuard 0.6B","YuFeng-XGuard 8B","Common 400 programs; weak-security functional editing","https://arxiv.org/html/2608.15092v1#S4.T3","Deepseek-v4-flash","Kimi k2.5","Doubao-1.5-pro 32k","OpenAI gpt-realtime Semantic VAD","Test FPR .018; median EOT latency 793 ms; frozen source operating point","https://arxiv.org/html/2608.25218v2#S6.T4","Kyutai SVAD","Test FPR .059; median EOT latency 1007 ms; frozen source operating point","SmartTurn v3","Test FPR .047; median EOT latency 1017 ms; frozen source operating point","VAP","Test FPR .055; median EOT latency 368 ms; frozen source operating point","Mimi-EP","Test FPR .078; median EOT latency 645 ms; frozen source operating point","ESPnet TT-pred.","Test FPR .078; median EOT latency 862 ms; frozen source operating point","ESPnet TT-pred. (per-ch.)","Test FPR .081; median EOT latency 730 ms; frozen source operating point","WavLM-Base (causal)","Test FPR .061; median EOT latency 701 ms; frozen source operating point","WavLM-Large (causal)","Test FPR .054; median EOT latency 683 ms; frozen source operating point","WavLM-Large (anchor)","Test FPR .054; median EOT latency 1076 ms; frozen source operating point","Gemini 3.1 Live","Test FPR .022; median EOT latency 1234 ms; frozen source operating point","Test FPR .044; median EOT latency 702 ms; frozen source operating point","Route deepseek-v3-2; 3 commissioning runs; 10 fixed plants × 20 evaluation seeds; source range -0.3758143362683567 to -0.11617840763625602","https://github.com/aiast1/rtcbench/blob/main/leaderboard/aggregate_2026-09-24.json","Route global-moonshotai-kimi-k3; 3 commissioning runs; 10 fixed plants × 20 evaluation seeds; source range 0.0872683435355276 to 0.5041548492815613","Gemma 3 27B IT","Route google-gemma-3-27b-it; 3 commissioning runs; 10 fixed plants × 20 evaluation seeds; source range -0.37523809975041644 to -0.2","Route minimax-minimax-m2-5; 3 commissioning runs; 10 fixed plants × 20 evaluation seeds; source range -0.24096532506778418 to -0.051146536073840565","Route mistral-mistral-large-3-675b-instruct; 3 commissioning runs; 10 fixed plants × 20 evaluation seeds; source range -0.20957040237843505 to 0.10664765901542801","Route moonshotai-kimi-k2-5; 3 commissioning runs; 10 fixed plants × 20 evaluation seeds; source range -0.2202458187334324 to -0.052792239243714664","Route nvidia-nemotron-super-3-120b; 3 commissioning runs; 10 fixed plants × 20 evaluation seeds; source range -0.32418874223035166 to 0.04402444734958803","Qwen3 Coder Next","Route qwen-qwen3-coder-next; 3 commissioning runs; 10 fixed plants × 20 evaluation seeds; source range -0.25315179626074824 to -0.018428455143665257","Qwen3 Next 80B A3B","Route qwen-qwen3-next-80b-a3b; 3 commissioning runs; 10 fixed plants × 20 evaluation seeds; source range -0.5854222610471821 to -0.09188964946362096","Amazon Nova 2 Lite","Route us-amazon-nova-2-lite-v1-0; 3 commissioning runs; 10 fixed plants × 20 evaluation seeds; source range -0.4487657978758687 to -0.10569819603530187","Route us-anthropic-claude-fable-5; 3 commissioning runs; 10 fixed plants × 20 evaluation seeds; source range 0.5316309940537158 to 0.6141062286285756","Route us-anthropic-claude-haiku-4-5-20251001-v1-0; 3 commissioning runs; 10 fixed plants × 20 evaluation seeds; source range -0.3927329951924162 to -0.17522922221628418","Route us-anthropic-claude-opus-4-8; 3 commissioning runs; 10 fixed plants × 20 evaluation seeds; source range 0.053888638144228375 to 0.31287876429434147","Route us-anthropic-claude-opus-5; 3 commissioning runs; 10 fixed plants × 20 evaluation seeds; source range 0.2216502832128758 to 0.5148516670074044","Route us-anthropic-claude-opus-5-5; 3 commissioning runs; 10 fixed plants × 20 evaluation seeds; source range 0.5458167007846007 to 0.7812359922223414","Route us-anthropic-claude-sonnet-5; 3 commissioning runs; 10 fixed plants × 20 evaluation seeds; source range 0.12432759361682363 to 0.2528718805643384","Route us-deepseek-r1-v1-0; 3 commissioning runs; 10 fixed plants × 20 evaluation seeds; source range -0.12606710567974852 to 0.21844847365334202","Llama 4 Maverick 17B Instruct","Route us-meta-llama4-maverick-17b-instruct-v1-0; 3 commissioning runs; 10 fixed plants × 20 evaluation seeds; source range -0.6 to -0.17490012540080177","Llama 4 Scout 17B Instruct","Route us-meta-llama4-scout-17b-instruct-v1-0; 3 commissioning runs; 10 fixed plants × 20 evaluation seeds; source range -0.3701470180397973 to 0.018870168464218517","Route us-openai-gpt-5-6-luna; 3 commissioning runs; 10 fixed plants × 20 evaluation seeds; source range 0.17737429059104265 to 0.4010848451504848","Route us-openai-gpt-5-6-sol; 3 commissioning runs; 10 fixed plants × 20 evaluation seeds; source range 0.31617485142911683 to 0.5524813861628002","Route us-openai-gpt-5-6-terra; 3 commissioning runs; 10 fixed plants × 20 evaluation seeds; source range 0.2646168795468845 to 0.4323545719122762","Route us-openai-gpt-6-astra; 3 commissioning runs; 10 fixed plants × 20 evaluation seeds; source range 0.4224721654529585 to 0.5793946702100905","Route us-openai-gpt-6-luna; 3 commissioning runs; 10 fixed plants × 20 evaluation seeds; source range 0.26627693589785617 to 0.4847946646423777","Route us-openai-gpt-6-sol; 3 commissioning runs; 10 fixed plants × 20 evaluation seeds; source range 0.09936732299972462 to 0.33469129308049417","Route us-xai-grok-4-6; 3 commissioning runs; 10 fixed plants × 20 evaluation seeds; source range 0.18593386199297965 to 0.5213828986171027","Route zai-glm-5; 3 commissioning runs; 10 fixed plants × 20 evaluation seeds; source range -0.48190167638143083 to 0.36431883956911815","Default single generation; five hypotheses per paper; six leave-one-out judges; source Avg","https://arxiv.org/html/2608.16645v3#S4.T2","GPT-5.6-Sol-Pro","Gemini 3.1-Pro-Preview","Zero-shot non-reasoning; TAR-unexposed source group","https://arxiv.org/html/2608.10317v3#S4.T3","VAD-R1","Zero-shot non-reasoning; TAR-exposed during base-model training","Qwen2.5 0.5B","Direct prompting; Standard; four-task mean","https://arxiv.org/html/2608.10414v1#S3.T3","Qwen2.5 3B","Vistral 7B","SeaLLM 7B","Direct prompting; PNB; four-task mean","Direct prompting; PNN; four-task mean","Direct prompting; PNT1; four-task mean","Direct prompting; PNT2; four-task mean","Direct prompting; PNT3; four-task mean","Direct prompting; PNT4; four-task mean","Full 500-image handwritten document cohort; source overall metric","https://arxiv.org/html/2608.22959v1#Sx5.T2","DeepSeek-OCR2","MinerU-2.5","Unlimited-OCR","Dots.ocr","Dots.mocr","MinerU2.5-Pro","One attempt; weighted normalized protocol-modification rubric","https://arxiv.org/html/2608.23898v1#A5.T2","Full 1,000-item CircuitReason-1k; typed scoring plus blinded semantic consensus","https://arxiv.org/html/2608.09374v1#Sx5.T3","All Skills; manual annotation; all published difficulty levels","https://arxiv.org/html/2608.29210v1#S4.T1","FLUX 1.0-dev","Dall-E 3","Gemini 3.1-Flash-preview","Stable Diffusion XL","Stable Cascade","Full 1,500-sample suite; Source default inference","https://arxiv.org/html/2608.30678v1#S3.T2","GPT-5.4-Medium","Full 1,500-sample suite; Medium reasoning effort","GPT-5.4-Mini-Medium","PCBnet","Design-level held-out PCBnet test; end-to-end netlist reconstruction","https://arxiv.org/html/2608.27923v1#S4.T1","Grok-4.2","Multimodal diagram input; Extended thinking","https://arxiv.org/html/2608.25097v1#S4.T3","Multimodal diagram input; Default reasoning mode","Text-only, without diagrams; Extended thinking","Text-only, without diagrams; Default reasoning mode","Dolphin-Mistral-24B","LLaMA-3.1-70B","GPT-5.6 Sol (max)","Five annual episodes; common simulator/brief; final assets CNY thousands","https://arxiv.org/html/2608.30730v1#S4.T2","Fable5 (max)","Claude Opus 4.8 (max)","Claude Opus 4.7 (max)","Claude Opus 4.6 (max)","GLM 5.2 (high)","DeepSeek-V4-Pro-Preview (max)","GLM 5.2 (max)","Human-aligned pairwise overall rating; source Table 2","https://arxiv.org/html/2608.13136v1#S4.T2","Gemini-3-pro","Qwen2.5-7B(trained)","Qwen2.5-14B(trained)","SD3.5","Ten-language content generation; published TRIGScore Avg","https://arxiv.org/html/2608.11002v1#S3.T2","SDXL","FLUX.1-Krea","Sana 1.5","PixArt- Σ","Lumina-Next","Omni-Diffusion","PEA <FLUX>","X2I < FLUX>","MuLan <PixArt>","GPT4o","100 held-out images; Market Size 0–3 judge rubric","https://arxiv.org/html/2608.11616v2#S4.T3","GPT5-mini","GPT5","LLaVA-NeXT-Qwen-32B","MBA-7B-SFT","MBA-b-7B","MBA-k-7B","glm4.1v-9b","Full 16,320-case factorized setting; attack success is a model failure","https://arxiv.org/html/2608.25490v1#S4.T1","step3-vl-10b","ministral3-8b","llava-onevision-1.5","internvl3-8b","qwen2.5-vl-7b","qwen3-vl-8b","gemma3-12b","glm-4.6v","qwen3-vl-plus","grok-4-fast","claude-sonnet-4.5","Shared true/false tasks; published vanilla/adversarial mean","https://arxiv.org/html/2608.22232v2#S3.T1","Grok-4.1-Fast (Think)","Gemini-3.5-Flash (Think)","Mimo-V2.5","InternVL3.5-2B-Instruct","GLM-4.6V-Flash (Think)","Long-VITA 14B","Params 14B; all seven context ranges; published Avg","https://arxiv.org/html/2608.08630v1#S5.T1","LongLLaVA 9B","Params 9B; all seven context ranges; published Avg","Mantis 8B","Params 8B; all seven context ranges; published Avg","Params 7B; all seven context ranges; published Avg","InternVL2.5 4B","Params 4B; all seven context ranges; published Avg","Ovis2 4B","GlimpsePrune 3B","Params 3B; all seven context ranges; published Avg","VisionZip 3B","VLZip 3B","AVH-Align","Common three protocols; Direct-transfer detector","https://arxiv.org/html/2608.09593v1#S5.T2","AV Anomaly","BA-TFD+","ImageBind A–V","Common three protocols; Frozen encoder with task head","PE-AV Base","CAV-MAE Sync","Common three protocols; Zero-shot omni prompt","Baichuan-Omni-1.5","Native multi-turn conversation history; source L1/L2 mean","https://arxiv.org/html/2608.31022v1#S3.T1","Claude-5-Opus","Claude-5-Fable","Fixed reader-budget benchmark; params –","https://arxiv.org/html/2608.17607v1#S5.T5","Fixed reader-budget benchmark; params 8B","Qwen3-VL-A3B 30B-A3B","Fixed reader-budget benchmark; params 30B-A3B","GLM-4.6V-Flash 10B","Fixed reader-budget benchmark; params 10B","Vision-DeepResearch-8B 8B","LLaVA-Med v1.5 7B","Fixed reader-budget benchmark; params 7B","MedVLM-R1-2B 2B","Fixed reader-budget benchmark; params 2B","HuatuoGPT-V 7B","HuatuoGPT-V 34B","Fixed reader-budget benchmark; params 34B","Lingshu 7B","Lingshu 32B","Fixed reader-budget benchmark; params 32B","MedGemma 1.5 4B","Fixed reader-budget benchmark; params 4B","MedGemma 27B","Fixed reader-budget benchmark; params 27B","Quilt-LLaVA 7B","Patho-R1-7B 7B","SlideChat ∼ 7B","Fixed reader-budget benchmark; params ∼ 7B","WSI-LLaVA ∼ 7B","PathNavigate","PathAgent","SFT 7B","PathoArgus 7B","Published all-at-once LLM judge; 126 human-labeled failed runs","https://arxiv.org/html/2608.22808v4#A4.T5","Nova-Micro","claude-code-cli; source addendum; 126 all-at-once traces","https://arxiv.org/html/2608.22808v4#A4.T17","nairr-gateway; source addendum; 126 all-at-once traces","codex-cli; source published; 126 all-at-once traces","codex-cli; source addendum; 126 all-at-once traces","not-recorded; source published; 126 all-at-once traces","llama-3-70b","aws-bedrock; source addendum; 126 all-at-once traces","llama-3.1-70b","Claude 4.8","AutoDesign; design harness DesignHarness; coding agent Claude Code","https://arxiv.org/html/2608.13560v1#S5.T1","AutoDesign; design harness DesignHarness; coding agent Codex","Claude Design; design harness Claude Design; coding agent Claude Code","OpenDesign; design harness OpenDesign; coding agent Claude Code","OpenDesign; design harness OpenDesign; coding agent Codex","Codex; design harness —; coding agent Codex","Claude Code; design harness —; coding agent Claude Code","Seed 2.1","Doubao; design harness —; coding agent Claude Code","GLM; design harness —; coding agent Claude Code","Kimi; design harness —; coding agent Claude Code","DeepSeekV4-Pro","DeepSeek; design harness —; coding agent Claude Code","PosterGen; design harness —; coding agent —","Any2Poster; design harness —; coding agent —","Paper2Poster; design harness —; coding agent —","BM25","Full silver-test candidate pool; Unigram","https://arxiv.org/html/2608.27394v4#S5.T1","Full silver-test candidate pool; Baseline","Full silver-test candidate pool; Tuned","Full silver-test candidate pool; Tuned + rerank","ModernBERT-embed-large","Stella-en-1.5B-v5","EmbeddingGemma-300m","ReasonIR-8B","answerai-colbert-small-v1","GTE-ModernColBERT-v1","Reason-ModernColBERT","tool-free; ρ=0.2; fifty six-problem math contests; source-calibrated shared budget","https://arxiv.org/html/2608.16033v1#S4.T2","Hy-3","tool-free; ρ=0.2; fifty six-problem code contests; source-calibrated shared budget","tool-free; ρ=0.2; fifty six-problem abstract-reasoning contests; source-calibrated shared budget","tool-free; ρ=0.8; fifty six-problem math contests; source-calibrated shared budget","tool-free; ρ=0.8; fifty six-problem code contests; source-calibrated shared budget","tool-free; ρ=0.8; fifty six-problem abstract-reasoning contests; source-calibrated shared budget","agentic; ρ=0.2; fifty six-problem math contests; source-calibrated shared budget","agentic; ρ=0.2; fifty six-problem code contests; source-calibrated shared budget","agentic; ρ=0.2; fifty six-problem abstract-reasoning contests; source-calibrated shared budget","agentic; ρ=0.8; fifty six-problem math contests; source-calibrated shared budget","agentic; ρ=0.8; fifty six-problem code contests; source-calibrated shared budget","agentic; ρ=0.8; fifty six-problem abstract-reasoning contests; source-calibrated shared budget","Fixed200scenario test split; RoboTwin; initial","https://arxiv.org/html/2609.28952v1#A3.T4","Fixed200scenario test split; RoboTwin; recovery","Fixed200scenario test; equal nine-group macro","https://arxiv.org/html/2609.28952v1#A3.T5","Fixed200scenario test split; LIBERO; initial","Being-H0.5","UniFOLM","Cosmos-Policy","Fixed200scenario test split; LIBERO; recovery","Gemini 3.1","Locked single-turn strategy; public split; 200 instances","https://arxiv.org/html/2609.30027v1#S5.T2","GPT 5.3","Kimi 2.5-thinking (1T-A32B)","DeepSeek V3.2 (671B-A37B)","GLM 5 (744B-A40B)","Qwen 3.5 (397B-A17B)","Mistral Large (123B)","Llama 4 Scout (109B-A17B)","Gemma 3 (27B)","Locked single-turn strategy; public split; 983 instances","Locked single-turn strategy; public split; 276 instances","Morgan-RF","Protocol full; 3-seed mean; fixed seed2026split","https://arxiv.org/html/2609.29740v1#S6.T3","GIN","Protocol full; fingerprint fusion; 3-seed mean; fixed seed2026split","GAT","GPS","D-MPNN","MolFormer","Protocol low; 3-seed mean; fixed seed2026split","https://arxiv.org/html/2609.29740v1#S6.T4","Protocol low; fingerprint fusion; 3-seed mean; fixed seed2026split","Protocol mini-hard; 3-seed mean; fixed seed2026split","https://arxiv.org/html/2609.29740v1#S6.T5","Protocol mini-hard; fingerprint fusion; 3-seed mean; fixed seed2026split","Protocol mini-random; 3-seed mean; fixed seed2026split","Protocol mini-random; fingerprint fusion; 3-seed mean; fixed seed2026split","Direct baseline; temperature0; top_p1; mean of3runs; 600 tasks","https://arxiv.org/html/2609.30749v1#S5.T2","MiniMax-M2-1","Qwen3-Coder-480B","Direct baseline; temperature0; top_p1; mean of3runs; 400 tasks","Direct baseline; temperature0; top_p1; mean of3runs; 1600 tasks","Direct baseline;200projects; mean3runs; temperature0","https://arxiv.org/html/2609.30749v1#A6.T9","Seq2Seq","SatNav seen; declared baseline; same4574 episodes","https://arxiv.org/html/2609.31507v1#S5.T1","CMA","OpenFly","SatNav seen; starred adaptation; same4574 episodes","NaVILA","UniNaVid","StreamVLN","SwiftVLN","SatNav unseen; declared baseline; same8756 episodes","SatNav unseen; starred adaptation; same8756 episodes","Online1FPS; recent-history native cap 384frames; temporal unit1; no future frames","https://arxiv.org/html/2609.30981v1#S4.T1","Online1FPS; recent-history native cap 768frames; temporal unit2; no future frames","Online1FPS; recent-history native cap 180frames; temporal unit1 (pack 3); no future frames","Online1FPS; recent-history native cap 256frames; temporal unit2; no future frames","Online1FPS; recent-history native cap 240frames; temporal unit2; no future frames","InternVideo2.5-8B","Online1FPS; recent-history native cap 512frames; temporal unit4; no future frames","Online1FPS; recent-history native cap 128frames; temporal unit2; no future frames","Online1FPS; recent-history native cap 240frames; temporal unit1; no future frames","LLaVA-NeXT-Video-7B-DPO","Online1FPS; recent-history native cap 32frames; temporal unit1; no future frames","Behavior reasoning;1500shared audit samples;1run","https://arxiv.org/html/2609.30722v1#Sx4.T3","TRACE1FPS;833QA records; Native persistent session and streaming generation; local weights; model+Adapter","https://arxiv.org/html/2609.30670v1#S5.F3","MOSS-Preview","TRACE1FPS;833QA records; Native real-time session/asynchronous frame queue; local weights; model+Adapter","ThinkStream","TRACE1FPS;833QA records; Native two-frame blocks/persistent state; local weights; model+Adapter","VideoLLM-Online","TRACE1FPS;833QA records; Native per-frame representations/persistent KV; local weights; model+Adapter","TRACE1FPS;833QA records; Non-native QA prefix processing; proactive persistent incremental state; local service; model+Adapter","TRACE1FPS;407records/1270windows; Native persistent session and streaming generation; local weights; model+Adapter","https://arxiv.org/html/2609.30670v1#S5.T5","TRACE1FPS;407records/1270windows; Non-native QA prefix processing; proactive persistent incremental state; local service; model+Adapter","TRACE1FPS;407records/1270windows; Native real-time session/asynchronous frame queue; local weights; model+Adapter","TRACE1FPS;407records/1270windows; Native two-frame blocks/persistent state; local weights; model+Adapter","TRACE1FPS;407records/1270windows; Native per-frame representations/persistent KV; local weights; model+Adapter","ProSST-2048","No missing-query fallback; fixed123assays; T1","https://arxiv.org/html/2609.28921v1#S4.T6","S3F","ESM-2","ProGen2-base","No missing-query fallback; fixed74assays; T2","https://arxiv.org/html/2609.28921v1#S4.T8","No missing-query fallback; fixed67assays; T3","No missing-query fallback; fixed29assays; T4","https://arxiv.org/html/2609.28921v1#S4.T9","GLM","Five-seed fixed split; GLM","https://arxiv.org/html/2609.31082v1#S3.T2","BRT","Five-seed fixed split; BRT","GAM","Five-seed fixed split; GAM","MaxEnt","Five-seed fixed split; MaxEnt","RF","Five-seed fixed split; RF","DeepSDM MLP","Five-seed fixed split; Baseline","Five-seed fixed split; + Aggregation","Five-seed fixed split; + Subsampling","DeepSDM ResNet","Five-seed fixed split; + Architecture","Five-seed fixed split; + Loss","https://arxiv.org/html/2609.31082v1#A4.T10","Five-seed fixed split; Random Forest","DeepSDM baseline","Five-seed fixed split; DeepSDM baseline","+ Data aggregation","Five-seed fixed split; + Data aggregation","Browser Use; temperature0; 90loaded+15cold sessions; scenario SC01","https://arxiv.org/html/2609.32915v1#A6.T9","Qwen3-VL 235B","Browser Use; temperature0; 90loaded+15cold sessions; scenario SC02","Browser Use; temperature0; 60loaded+15cold sessions; scenario SC04","Browser Use; temperature0; 60loaded+15cold sessions; scenario SC06","Browser Use; temperature0; 45loaded+15cold sessions; scenario SC07","Browser Use; temperature0; 60loaded+15cold sessions; scenario SC08","Browser Use; temperature0; 60loaded+15cold sessions; scenario SC09","Browser Use; temperature0; 30loaded+15cold sessions; scenario SC10","Browser Use; temperature0; 60loaded+15cold sessions; scenario SC11","Browser Use; temperature0; 45loaded+15cold sessions; scenario SC12","Browser Use; temperature0; 60loaded+15cold sessions; scenario SC13","Browser Use; temperature0; 45loaded+15cold sessions; scenario SC14","Browser Use; temperature0; 90loaded+15cold sessions; scenario SC15","Browser Use; temperature0; 60loaded+15cold sessions; scenario SC16","Browser Use; temperature0; 90loaded+15cold sessions; scenario SC18","Browser Use; temperature0; 15loaded+15cold sessions; scenario SC19","Browser Use; temperature0; 60loaded+15cold sessions; scenario SC20","entity-mask; original search/reasoning setting: DeepSeek V4 Flash; 143 shared transcripts","https://arxiv.org/html/2610.05586v1#S4.T2","entity-mask; original search/reasoning setting: Gemini 3.1 Pro Preview; 143 shared transcripts","entity-mask; original search/reasoning setting: Gemini 3.7 Flash; 143 shared transcripts","entity-mask; original search/reasoning setting: GLM 5.3 Flash; 143 shared transcripts","GPT-4.1 Mini","entity-mask; original search/reasoning setting: GPT-4.1 Mini (+search); 143 shared transcripts","entity-mask; original search/reasoning setting: GPT-4.1 Mini (no search); 143 shared transcripts","entity-mask; original search/reasoning setting: GPT-5.6 Terra (none); 143 shared transcripts","entity-mask; original search/reasoning setting: GPT-5.6 Terra (xhigh); 143 shared transcripts","entity-mask; original search/reasoning setting: GPT-5 Mini (high); 143 shared transcripts","entity-mask; original search/reasoning setting: GPT-5 Mini (minimal); 143 shared transcripts","Kimi K3 Max","entity-mask; original search/reasoning setting: Kimi K3 Max; 143 shared transcripts","entity-mask; original search/reasoning setting: Llama 4 Maverick; 143 shared transcripts","privacy-instruction; original search/reasoning setting: DeepSeek V4 Flash; 143 shared transcripts","privacy-instruction; original search/reasoning setting: Gemini 3.1 Pro Preview; 143 shared transcripts","privacy-instruction; original search/reasoning setting: Gemini 3.7 Flash; 143 shared transcripts","privacy-instruction; original search/reasoning setting: GLM 5.3 Flash; 143 shared transcripts","privacy-instruction; original search/reasoning setting: GPT-4.1 Mini (+search); 143 shared transcripts","privacy-instruction; original search/reasoning setting: GPT-4.1 Mini (no search); 143 shared transcripts","privacy-instruction; original search/reasoning setting: GPT-5.6 Terra (none); 143 shared transcripts","privacy-instruction; original search/reasoning setting: GPT-5.6 Terra (xhigh); 143 shared transcripts","privacy-instruction; original search/reasoning setting: GPT-5 Mini (high); 143 shared transcripts","privacy-instruction; original search/reasoning setting: GPT-5 Mini (minimal); 143 shared transcripts","privacy-instruction; original search/reasoning setting: Kimi K3 Max; 143 shared transcripts","privacy-instruction; original search/reasoning setting: Llama 4 Maverick; 143 shared transcripts","Stable Diffusion 1.5","Published method checkpoints; common paired test; 1008×1296 external transform","https://arxiv.org/html/2610.10703v1#S5.T2","DSRNet","ER 2 Net","Stable Diffusion 3.5","DSIT","RDNet","DL2G","ReflexSplit","OcuFlow","Native typed decision interface; original checkpoint temperature","https://arxiv.org/html/2610.11392v1#A4.T8","Laya english","Laya typed-dec.","Kev -0.8B","Kev -4B","Kev -9B","Gemma 4 31B, constrained","Constrained LLM baseline","Qwen3.5-9B, constrained","Agent: copilot-cli 1.0.74; default /review prompt; max available reasoning unless source model label specifies none/high","https://arxiv.org/html/2610.13668v1#S4.T1","claude-opus-4.6","Graph2SMILES","Benchmark-trained; no supplied EC; Product Split; canonical top-1 matching","https://arxiv.org/html/2610.11694v1#S4.T3","Retroformer","MEGAN","R-SMILES","BioNavi","G2Retro","RSGPT","VenusRX","Benchmark-trained; no supplied EC; Product Split; canonical top-1 matching; MLCD","FlowScore; published per-family selected checkpoint; original encoder frozen","https://arxiv.org/html/2610.21561v1#S5.T3","DSE-BERT-base","FlowScore; published per-family selected checkpoint; original encoder frozen; Dialog2Flow joint DSE fine-tune","CNN-BC","PointGoal; unseen scenes; common fixed 800 episodes","https://arxiv.org/html/2610.29699v1#S4.T2","ViNT","OmniVLA","NWM","NavMorph","Published common image-disjoint perception test; exact match","https://arxiv.org/html/2610.18455v1#S5.T4","LLaVA-OneVision-1.5-8B-Instruct","InternVL3-8B-Instruct","Emotion-Qwen","EmoCaliber","EEmo-Logic","EmoVIT","AffectGPT","Published common image-disjoint perception test; exact match; EmoProbe emotion fine-tune","Trace Anything","Primary moving-region track; native output; matched windows; pooled 2086 frames","https://arxiv.org/html/2610.31209v1#S4.T1","Any4D","4RC","MoVieS","UniQuery4R","FlowSAM","SegAnyMo","RoMo","Easi3R","VGGT4D","MoRe","GeoMotion","GMOS","GMOS-S","MoRSe","ViduQ2","Common SKU-disjoint 220-case test; global full-prompt/frame similarity","https://arxiv.org/html/2610.10047v1#S5.T3","ViduQ3","Pixverse V5","Pixverse V6","Kling3.0 Omni","SkyReels-V3","Phantom","Refalign","Kaleido","HunyuanCustom","MV-S2V","MAGREF","Common SKU-disjoint 220-case test; global full-prompt/frame similarity; AdSpark rank-64 LoRA","K2-V2-70B-Instruct","Mean across six common audio-captioner configurations; ten-minute independent text windows","https://arxiv.org/html/2610.07338v1#S3.T3","OLMo-3.1-32B-Instruct","Direct Replay; all action instances; first slash component","https://arxiv.org/html/2610.17738v1#S5.T4","Mini-Omni2","Voila-Autonomous","FLM-Audio","Raon-SpeechChat","BayLing-Duplex","Lychee-FD","Grok Voice","Qwen3.5-Omni-Flash-Realtime","NL+Data; same seed-42 fifteen-percent subset; one attempt; AMPL/CBC","https://arxiv.org/html/2610.22889v1#A8.T8","Ministral-3-8B-2512","Mistral-Large-2411","NL+Data; same seed-42 fifteen-percent subset; one attempt; AMPL/CBC; Five-element","GPT-3.5-turbo","NL+Data; same seed-42 fifteen-percent subset; one attempt; AMPL/CBC; CoE","NL+Data; same seed-42 fifteen-percent subset; one attempt; AMPL/CBC; SFT","Per-comment zero-shot; abstention allowed","https://arxiv.org/html/2610.15732v1#S4.T4","XLM-RoBERTa base","Fine-tuned on 1661 batched-label training comments; 184 validation; mean five seeds","https://arxiv.org/html/2610.15732v1#S5.T5","mBERT (cased)","IRIIS RoBERTa 125M","IRIIS BERT 110M","Core-15; fixed staged generation; family-balanced mean over three replicas","https://arxiv.org/html/2610.04253v1#A6.T18","Qwen3.8 Max-0902","Original main Table2; unidirectional timing; causal streaming; native PRS composite","https://arxiv.org/html/2610.21568v1#S4.T2","Em-Garde","JoyVL","VideoChat3","Original main Table2; bidirectional timing; causal streaming; native PRS composite","OpenCode; single-pass; three runs/task; printed bar-label precision","https://arxiv.org/pdf/2610.21074v1#page=7","Muse-Spark-1.1","OpenCode; gated; three runs/task; printed bar-label precision","Three-shot; full 5000-document corpus; thinking on; size –","https://arxiv.org/pdf/2610.21333v1#page=7","Three-shot; full 5000-document corpus; thinking off; size –","Three-shot; full 5000-document corpus; thinking on; size 14B","Three-shot; full 5000-document corpus; thinking off; size 14B","Three-shot; full 5000-document corpus; thinking on; size 12B","Three-shot; full 5000-document corpus; thinking off; size 12B","Three-shot; full 5000-document corpus; thinking off; size 9B","Three-shot; full 5000-document corpus; thinking on; size 9B","Three-shot; full 5000-document corpus; thinking off; size 8B","Ministral-3-8B","Three-shot; full 5000-document corpus; thinking on; size 4B","Three-shot; full 5000-document corpus; thinking off; size 4B","Phi-4-Mini","Three-shot; full 5000-document corpus; thinking off; size 3.8B","Three-shot; full 5000-document corpus; thinking off; size 3B","Three-shot; full 5000-document corpus; thinking off; size 32B","Three-shot; full 5000-document corpus; thinking on; size 31B","Three-shot; full 5000-document corpus; thinking off; size 31B","Three-shot; full 5000-document corpus; thinking on; size 27B","Three-shot; full 5000-document corpus; thinking off; size 27B","Three-shot; full 5000-document corpus; thinking off; size 24B","Opus 4.6","Agent: ClaudeCode; linux; without-poc; 200 tool calls; single trajectory","https://arxiv.org/html/2609.25591v1#S3.T2","Sonnet 4.6","Haiku 4.5","Agent: Codex; linux; without-poc; 200 tool calls; single trajectory","GPT-5.4 Mini","Agent: OpenCode; linux; without-poc; 200 tool calls; single trajectory","Kimi K2.5","Agent: ClaudeCode; linux; with-poc; 200 tool calls; single trajectory","Agent: Codex; linux; with-poc; 200 tool calls; single trajectory","Agent: OpenCode; linux; with-poc; 200 tool calls; single trajectory","Agent: ClaudeCode; windows; without-poc; 300 tool calls; single trajectory","Agent: Codex; windows; without-poc; 300 tool calls; single trajectory","Agent: OpenCode; windows; without-poc; 300 tool calls; single trajectory","Agent: ClaudeCode; windows; with-poc; 300 tool calls; single trajectory","Agent: Codex; windows; with-poc; 300 tool calls; single trajectory","Agent: OpenCode; windows; with-poc; 300 tool calls; single trajectory","Agent: Browser Use; live browser; 600 instances; full-motion","https://arxiv.org/html/2609.27461v1#S4.T2","Claude-Opus-4-6","Agent: Native computer use; live browser; 600 instances; full-motion","Agent: Browser Use; live browser; 600 instances; foreground-only","Agent: Native computer use; live browser; 600 instances; foreground-only","Common source prompt and hyperparameters; full diagnostic QA set","https://arxiv.org/html/2609.24172v1#S5.T4","LLaVA-HR","ChartInstruct","ChartMLLM","Grounding DINO","Source pretrained checkpoint; validated ground truth; COCO IoU .50:.95","https://arxiv.org/html/2609.21822v1#S5.T7.st1","YOLOv8-World v2","OWL-ViT","RT-DETR-L","YOLOv8-L","Faster R-CNN (ResNet 50)","Source pretrained checkpoint; probabilistic labels; IoU .50:.95","https://arxiv.org/html/2609.21822v1#S5.T8","Common fine-tuning; all scenes/levels; instruction-conditioned policy","https://arxiv.org/html/2609.25636v1#S3.T1","Common fine-tuning; all scenes/levels; empty-language ablation","GR00T N1.6","XVLA","ACoT-VLA","Lingbot-VLA","MOTUS","Original statement condition; deterministic registered-answer classification","https://arxiv.org/html/2609.27603v1#S3.T1","Offline QA; common task set; source model-native inference","https://arxiv.org/html/2609.25001v1#S4.T4","Doubao-Seed-2.1-Turbo","GELab-Zero-4B-Preview","Four exploration rounds; reasoning max; mean of three trajectories; three answers per question","https://arxiv.org/html/2609.30199v1#S4.T1","Four exploration rounds; reasoning high; mean of three trajectories; three answers per question","Four exploration rounds; reasoning xhigh; mean of three trajectories; three answers per question","Agent: Codex; one RTX 4090; 4h per task; native standard harness; single run","https://arxiv.org/pdf/2609.27308v1#page=4","Agent: Claude Code; one RTX 4090; 4h per task; native standard harness; single run","Flux-2.0","Complete eleven-task source cohort; multi-image support; visual","https://arxiv.org/html/2609.26375v1#S5.T8","Joy-Edit-Plus","FireRed-Edit","Qwen-Edit-2511","KwaiMind","Seedream","NanoBanana-2","Complete eleven-task source cohort; multi-image support; ctr-ranking","31.3B source parameters; shared 3584-token checked documents; July 2026 batch","https://arxiv.org/html/2609.27510v1#A1.T5","Qwen3.5-35B-A3B-Base","34.7B source parameters; shared 3584-token checked documents; July 2026 batch","25.8B source parameters; shared 3584-token checked documents; July 2026 batch","24.0B source parameters; shared 3584-token checked documents; July 2026 batch","27.4B source parameters; shared 3584-token checked documents; July 2026 batch","Seed-OSS-36B-Base","36.2B source parameters; shared 3584-token checked documents; July 2026 batch","31.6B source parameters; shared 3584-token checked documents; July 2026 batch","32.8B source parameters; shared 3584-token checked documents; July 2026 batch","Qwen3-30B-A3B-Base","30.5B source parameters; shared 3584-token checked documents; July 2026 batch","OLMo-3-32B","32.2B source parameters; shared 3584-token checked documents; July 2026 batch","marin-32b-base","32.5B source parameters; shared 3584-token checked documents; July 2026 batch","RWKV7-G1J-13.3B","13.3B source parameters; shared 3584-token checked documents; July 2026 batch","13.9B source parameters; shared 3584-token checked documents; July 2026 batch","Qwen3-14B-Base","14.8B source parameters; shared 3584-token checked documents; July 2026 batch","12.2B source parameters; shared 3584-token checked documents; July 2026 batch","gemma-3-12b-pt","Gravity-16B-A3B-Base","16.2B source parameters; shared 3584-token checked documents; July 2026 batch","Llama-2-13b-hf","13.0B source parameters; shared 3584-token checked documents; July 2026 batch","Qwen3.5-9B-Base","9.0B source parameters; shared 3584-token checked documents; July 2026 batch","8.9B source parameters; shared 3584-token checked documents; July 2026 batch","RWKV7-G1J-7.2B","7.2B source parameters; shared 3584-token checked documents; July 2026 batch","Meta-Llama-3.1-8B","8.0B source parameters; shared 3584-token checked documents; July 2026 batch","Qwen3-8B-Base","8.2B source parameters; shared 3584-token checked documents; July 2026 batch","gemma-4-E4B","7.9B source parameters; shared 3584-token checked documents; July 2026 batch","Falcon-H1-7B-Base","7.6B source parameters; shared 3584-token checked documents; July 2026 batch","marin-8b-base","Hunyuan-7B-Pretrain","7.5B source parameters; shared 3584-token checked documents; July 2026 batch","Apertus-8B-2509","8.1B source parameters; shared 3584-token checked documents; July 2026 batch","Minitron-8B","8.3B source parameters; shared 3584-token checked documents; July 2026 batch","Olmo-Hybrid-7B","7.4B source parameters; shared 3584-token checked documents; July 2026 batch","ZAYA1-base","8.8B source parameters; shared 3584-token checked documents; July 2026 batch","7.3B source parameters; shared 3584-token checked documents; July 2026 batch","Llama-2-7b-hf","6.7B source parameters; shared 3584-token checked documents; July 2026 batch","Falcon3-7B-Base","Qwen3.5-4B-Base","4.2B source parameters; shared 3584-token checked documents; July 2026 batch","Spark-X2.5-4B-Base","4.1B source parameters; shared 3584-token checked documents; July 2026 batch","RWKV7-G1J-2.9B","2.9B source parameters; shared 3584-token checked documents; July 2026 batch","Nanbeige4.2-3B-Base","Qwen3-4B-Base","4.0B source parameters; shared 3584-token checked documents; July 2026 batch","Ministral-3-3B","3.8B source parameters; shared 3584-token checked documents; July 2026 batch","gemma-3-4b-pt","4.3B source parameters; shared 3584-token checked documents; July 2026 batch","3.2B source parameters; shared 3584-token checked documents; July 2026 batch","gemma-4-E2B","5.1B source parameters; shared 3584-token checked documents; July 2026 batch","Nanbeige4-3B-Base","3.9B source parameters; shared 3584-token checked documents; July 2026 batch","3.1B source parameters; shared 3584-token checked documents; July 2026 batch","Falcon-H1-3B-Base","MiniCPM5-2B-Base","2.5B source parameters; shared 3584-token checked documents; July 2026 batch","SmolLM3-3B-Base","Llama-3.1-Minitron-4B-Width","4.5B source parameters; shared 3584-token checked documents; July 2026 batch","gemma-2-2b","2.6B source parameters; shared 3584-token checked documents; July 2026 batch","Minitron-4B-Base","stablelm-3b-4e1t","2.8B source parameters; shared 3584-token checked documents; July 2026 batch","Index-1.9B","2.2B source parameters; shared 3584-token checked documents; July 2026 batch","Llama-3.1-Minitron-4B-Depth","RWKV7-G1J-1.5B","1.5B source parameters; shared 3584-token checked documents; July 2026 batch","Spark-X2.5-1.7B-Base","1.7B source parameters; shared 3584-token checked documents; July 2026 batch","Qwen3.5-2B-Base","1.9B source parameters; shared 3584-token checked documents; July 2026 batch","Qwen3-1.7B-Base","Youtu-LLM-2B-Base","2.0B source parameters; shared 3584-token checked documents; July 2026 batch","Falcon-H1-1.5B-Deep-Base","1.6B source parameters; shared 3584-token checked documents; July 2026 batch","1.2B source parameters; shared 3584-token checked documents; July 2026 batch","Falcon-H1-1.5B-Base","stablelm-2-1_6b","Qwen3.5-0.8B-Base","0.8B source parameters; shared 3584-token checked documents; July 2026 batch","MiniCPM5-1B-Base","1.1B source parameters; shared 3584-token checked documents; July 2026 batch","Qwen3-0.6B-Base","0.6B source parameters; shared 3584-token checked documents; July 2026 batch","gemma-3-1b-pt","1.0B source parameters; shared 3584-token checked documents; July 2026 batch","RWKV7-G1D-0.4B","0.5B source parameters; shared 3584-token checked documents; July 2026 batch","ERNIE-4.5-0.3B-Base-PT","0.4B source parameters; shared 3584-token checked documents; July 2026 batch","Falcon-H1-0.5B-Base","RWKV7-G1D-0.1B","0.2B source parameters; shared 3584-token checked documents; July 2026 batch","gemma-3-270m","0.3B source parameters; shared 3584-token checked documents; July 2026 batch","0.1B source parameters; shared 3584-token checked documents; July 2026 batch","Agent: Codex; GPT-OSS-120B monitor; three selected valid runs/task","https://arxiv.org/html/2609.30217v1#S0.F1","Agent: Codex; GPT-OSS-120B monitor; three selected valid runs/task; author marks cybersecurity refusals or model downgrades","Agent: Claude Code; GPT-OSS-120B monitor; three selected valid runs/task; author marks cybersecurity refusals or model downgrades","Deblurring ConvMAE","full-finetuning; common source optimization; AUL pretraining overlap: not listed in pretraining","https://arxiv.org/pdf/2609.28610v1#page=24","Deblurring MAE","DINOv3 ConvNeXt-B","full-finetuning; common source optimization; AUL pretraining overlap: not auditable; no known ultrasound images","DINOv3 ConvNeXt-L","DINOv3 ViT-B","DINOv3 ViT-L","EchoCare","full-finetuning; common source optimization; AUL pretraining overlap: unknown; constituent pretraining datasets undisclosed","OpenUS","full-finetuning; common source optimization; AUL pretraining overlap: included in pretraining","SAMUS","TinyUSFM","UltraFedFM","UltraSAM","URFM-B","URFM-L","USFM","USF-MAE","full-finetuning; common source optimization; BUS-BRA pretraining overlap: not listed in pretraining","full-finetuning; common source optimization; BUS-BRA pretraining overlap: not auditable; no known ultrasound images","full-finetuning; common source optimization; BUS-BRA pretraining overlap: unknown; constituent pretraining datasets undisclosed","full-finetuning; common source optimization; COVID-BLUES pretraining overlap: not listed in pretraining","full-finetuning; common source optimization; COVID-BLUES pretraining overlap: not auditable; no known ultrasound images","full-finetuning; common source optimization; COVID-BLUES pretraining overlap: unknown; constituent pretraining datasets undisclosed","full-finetuning; common source optimization; COVID-BLUES pretraining overlap: included in pretraining","full-finetuning; common source optimization; GBCU pretraining overlap: not listed in pretraining","full-finetuning; common source optimization; GBCU pretraining overlap: not auditable; no known ultrasound images","full-finetuning; common source optimization; GBCU pretraining overlap: unknown; constituent pretraining datasets undisclosed","full-finetuning; common source optimization; MASLD pretraining overlap: not listed in pretraining","full-finetuning; common source optimization; MASLD pretraining overlap: not auditable; no known ultrasound images","full-finetuning; common source optimization; MASLD pretraining overlap: unknown; constituent pretraining datasets undisclosed","full-finetuning; common source optimization; MMOTU pretraining overlap: not listed in pretraining","full-finetuning; common source optimization; MMOTU pretraining overlap: not auditable; no known ultrasound images","full-finetuning; common source optimization; MMOTU pretraining overlap: unknown; constituent pretraining datasets undisclosed","full-finetuning; common source optimization; MMOTU pretraining overlap: included in pretraining","full-finetuning; common source optimization; TN3K pretraining overlap: not listed in pretraining","full-finetuning; common source optimization; TN3K pretraining overlap: not auditable; no known ultrasound images","full-finetuning; common source optimization; TN3K pretraining overlap: unknown; constituent pretraining datasets undisclosed","full-finetuning; common source optimization; TN3K pretraining overlap: included in pretraining","full-finetuning; common source optimization; CAMUS pretraining overlap: not listed in pretraining","full-finetuning; common source optimization; CAMUS pretraining overlap: not auditable; no known ultrasound images","full-finetuning; common source optimization; CAMUS pretraining overlap: unknown; constituent pretraining datasets undisclosed","full-finetuning; common source optimization; CAMUS pretraining overlap: included in pretraining","full-finetuning; common source optimization; Fetal Planes pretraining overlap: not listed in pretraining","full-finetuning; common source optimization; Fetal Planes pretraining overlap: not auditable; no known ultrasound images","full-finetuning; common source optimization; Fetal Planes pretraining overlap: unknown; constituent pretraining datasets undisclosed","full-finetuning; common source optimization; Fetal Planes pretraining overlap: included in pretraining","head-only; common source optimization; AUL pretraining overlap: not listed in pretraining","head-only; common source optimization; AUL pretraining overlap: not auditable; no known ultrasound images","head-only; common source optimization; AUL pretraining overlap: unknown; constituent pretraining datasets undisclosed","head-only; common source optimization; AUL pretraining overlap: included in pretraining","head-only; common source optimization; BUS-BRA pretraining overlap: not listed in pretraining","head-only; common source optimization; BUS-BRA pretraining overlap: not auditable; no known ultrasound images","head-only; common source optimization; BUS-BRA pretraining overlap: unknown; constituent pretraining datasets undisclosed","head-only; common source optimization; COVID-BLUES pretraining overlap: not listed in pretraining","head-only; common source optimization; COVID-BLUES pretraining overlap: not auditable; no known ultrasound images","head-only; common source optimization; COVID-BLUES pretraining overlap: unknown; constituent pretraining datasets undisclosed","head-only; common source optimization; COVID-BLUES pretraining overlap: included in pretraining","head-only; common source optimization; GBCU pretraining overlap: not listed in pretraining","head-only; common source optimization; GBCU pretraining overlap: not auditable; no known ultrasound images","head-only; common source optimization; GBCU pretraining overlap: unknown; constituent pretraining datasets undisclosed","head-only; common source optimization; MASLD pretraining overlap: not listed in pretraining","head-only; common source optimization; MASLD pretraining overlap: not auditable; no known ultrasound images","head-only; common source optimization; MASLD pretraining overlap: unknown; constituent pretraining datasets undisclosed","head-only; common source optimization; MMOTU pretraining overlap: not listed in pretraining","head-only; common source optimization; MMOTU pretraining overlap: not auditable; no known ultrasound images","head-only; common source optimization; MMOTU pretraining overlap: unknown; constituent pretraining datasets undisclosed","head-only; common source optimization; MMOTU pretraining overlap: included in pretraining","head-only; common source optimization; TN3K pretraining overlap: not listed in pretraining","head-only; common source optimization; TN3K pretraining overlap: not auditable; no known ultrasound images","head-only; common source optimization; TN3K pretraining overlap: unknown; constituent pretraining datasets undisclosed","head-only; common source optimization; TN3K pretraining overlap: included in pretraining","head-only; common source optimization; CAMUS pretraining overlap: not listed in pretraining","head-only; common source optimization; CAMUS pretraining overlap: not auditable; no known ultrasound images","head-only; common source optimization; CAMUS pretraining overlap: unknown; constituent pretraining datasets undisclosed","head-only; common source optimization; CAMUS pretraining overlap: included in pretraining","head-only; common source optimization; Fetal Planes pretraining overlap: not listed in pretraining","head-only; common source optimization; Fetal Planes pretraining overlap: not auditable; no known ultrasound images","head-only; common source optimization; Fetal Planes pretraining overlap: unknown; constituent pretraining datasets undisclosed","head-only; common source optimization; Fetal Planes pretraining overlap: included in pretraining","full-finetuning; common source optimization; Open Kidney pretraining overlap: not listed in pretraining","full-finetuning; common source optimization; Open Kidney pretraining overlap: not auditable; no known ultrasound images","full-finetuning; common source optimization; Open Kidney pretraining overlap: unknown; constituent pretraining datasets undisclosed","full-finetuning; common source optimization; Open Kidney pretraining overlap: included in pretraining","full-finetuning; common source optimization; PSFHS pretraining overlap: not listed in pretraining","full-finetuning; common source optimization; PSFHS pretraining overlap: not auditable; no known ultrasound images","full-finetuning; common source optimization; PSFHS pretraining overlap: unknown; constituent pretraining datasets undisclosed","full-finetuning; common source optimization; PSFHS pretraining overlap: included in pretraining","full-finetuning; common source optimization; STMUS pretraining overlap: not listed in pretraining","full-finetuning; common source optimization; STMUS pretraining overlap: not auditable; no known ultrasound images","full-finetuning; common source optimization; STMUS pretraining overlap: unknown; constituent pretraining datasets undisclosed","full-finetuning; common source optimization; STMUS pretraining overlap: included in pretraining","https://arxiv.org/pdf/2609.28610v1#page=29","head-only; common source optimization; Open Kidney pretraining overlap: not listed in pretraining","head-only; common source optimization; Open Kidney pretraining overlap: not auditable; no known ultrasound images","head-only; common source optimization; Open Kidney pretraining overlap: unknown; constituent pretraining datasets undisclosed","head-only; common source optimization; Open Kidney pretraining overlap: included in pretraining","head-only; common source optimization; PSFHS pretraining overlap: not listed in pretraining","head-only; common source optimization; PSFHS pretraining overlap: not auditable; no known ultrasound images","head-only; common source optimization; PSFHS pretraining overlap: unknown; constituent pretraining datasets undisclosed","head-only; common source optimization; PSFHS pretraining overlap: included in pretraining","head-only; common source optimization; STMUS pretraining overlap: not listed in pretraining","head-only; common source optimization; STMUS pretraining overlap: not auditable; no known ultrasound images","head-only; common source optimization; STMUS pretraining overlap: unknown; constituent pretraining datasets undisclosed","head-only; common source optimization; STMUS pretraining overlap: included in pretraining","Doubao-Seed-2.1-pro","Reasoning; source native setting; find-and-replace correction template","https://arxiv.org/html/2609.24983v1#A1.T3","Reasoning; xhigh; find-and-replace correction template","Step-5-Preview","Instruct; find-and-replace correction template","Direct term prediction; OpenRouter; temperature 0; 16384 output-token limit; awo","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Term/03_evaluation_results/deepseek-v4-flash/awo/06_cq_terms/eval_cq_terms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Term/03_evaluation_results/deepseek-v4-pro/awo/06_cq_terms/eval_cq_terms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Term/03_evaluation_results/deepseek_deepseek-v3.2/awo/06_cq_terms/eval_cq_terms_result.json","Gemma 4 26B-A4B-IT","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Term/03_evaluation_results/google_gemma-4-26b-a4b-it/awo/06_cq_terms/eval_cq_terms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Term/03_evaluation_results/google_gemma-4-31b-it/awo/06_cq_terms/eval_cq_terms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Term/03_evaluation_results/qwen_qwen3.6-27b/awo/06_cq_terms/eval_cq_terms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Term/03_evaluation_results/qwen_qwen3.6-35b-a3b/awo/06_cq_terms/eval_cq_terms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Term/03_evaluation_results/qwen_qwen3.6-flash/awo/06_cq_terms/eval_cq_terms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Term/03_evaluation_results/qwen_qwen3.6-plus/awo/06_cq_terms/eval_cq_terms_result.json","Direct term prediction; OpenRouter; temperature 0; 16384 output-token limit; odrl","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Term/03_evaluation_results/deepseek-v4-flash/odrl/06_cq_terms/eval_cq_terms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Term/03_evaluation_results/deepseek-v4-pro/odrl/06_cq_terms/eval_cq_terms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Term/03_evaluation_results/deepseek_deepseek-v3.2/odrl/06_cq_terms/eval_cq_terms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Term/03_evaluation_results/google_gemma-4-26b-a4b-it/odrl/06_cq_terms/eval_cq_terms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Term/03_evaluation_results/google_gemma-4-31b-it/odrl/06_cq_terms/eval_cq_terms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Term/03_evaluation_results/qwen_qwen3.6-27b/odrl/06_cq_terms/eval_cq_terms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Term/03_evaluation_results/qwen_qwen3.6-35b-a3b/odrl/06_cq_terms/eval_cq_terms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Term/03_evaluation_results/qwen_qwen3.6-flash/odrl/06_cq_terms/eval_cq_terms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Term/03_evaluation_results/qwen_qwen3.6-plus/odrl/06_cq_terms/eval_cq_terms_result.json","Direct term prediction; OpenRouter; temperature 0; 16384 output-token limit; swo","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Term/03_evaluation_results/deepseek-v4-flash/swo/06_cq_terms/eval_cq_terms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Term/03_evaluation_results/deepseek-v4-pro/swo/06_cq_terms/eval_cq_terms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Term/03_evaluation_results/deepseek_deepseek-v3.2/swo/06_cq_terms/eval_cq_terms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Term/03_evaluation_results/google_gemma-4-26b-a4b-it/swo/06_cq_terms/eval_cq_terms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Term/03_evaluation_results/google_gemma-4-31b-it/swo/06_cq_terms/eval_cq_terms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Term/03_evaluation_results/qwen_qwen3.6-27b/swo/06_cq_terms/eval_cq_terms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Term/03_evaluation_results/qwen_qwen3.6-35b-a3b/swo/06_cq_terms/eval_cq_terms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Term/03_evaluation_results/qwen_qwen3.6-flash/swo/06_cq_terms/eval_cq_terms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Term/03_evaluation_results/qwen_qwen3.6-plus/swo/06_cq_terms/eval_cq_terms_result.json","Direct term prediction; OpenRouter; temperature 0; 16384 output-token limit; vgo","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Term/03_evaluation_results/deepseek-v4-flash/vgo/06_cq_terms/eval_cq_terms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Term/03_evaluation_results/deepseek-v4-pro/vgo/06_cq_terms/eval_cq_terms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Term/03_evaluation_results/deepseek_deepseek-v3.2/vgo/06_cq_terms/eval_cq_terms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Term/03_evaluation_results/google_gemma-4-26b-a4b-it/vgo/06_cq_terms/eval_cq_terms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Term/03_evaluation_results/google_gemma-4-31b-it/vgo/06_cq_terms/eval_cq_terms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Term/03_evaluation_results/qwen_qwen3.6-27b/vgo/06_cq_terms/eval_cq_terms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Term/03_evaluation_results/qwen_qwen3.6-35b-a3b/vgo/06_cq_terms/eval_cq_terms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Term/03_evaluation_results/qwen_qwen3.6-flash/vgo/06_cq_terms/eval_cq_terms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Term/03_evaluation_results/qwen_qwen3.6-plus/vgo/06_cq_terms/eval_cq_terms_result.json","Direct term prediction; OpenRouter; temperature 0; 16384 output-token limit; water","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Term/03_evaluation_results/deepseek-v4-flash/water/06_cq_terms/eval_cq_terms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Term/03_evaluation_results/deepseek-v4-pro/water/06_cq_terms/eval_cq_terms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Term/03_evaluation_results/deepseek_deepseek-v3.2/water/06_cq_terms/eval_cq_terms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Term/03_evaluation_results/google_gemma-4-26b-a4b-it/water/06_cq_terms/eval_cq_terms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Term/03_evaluation_results/google_gemma-4-31b-it/water/06_cq_terms/eval_cq_terms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Term/03_evaluation_results/qwen_qwen3.6-27b/water/06_cq_terms/eval_cq_terms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Term/03_evaluation_results/qwen_qwen3.6-35b-a3b/water/06_cq_terms/eval_cq_terms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Term/03_evaluation_results/qwen_qwen3.6-flash/water/06_cq_terms/eval_cq_terms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Term/03_evaluation_results/qwen_qwen3.6-plus/water/06_cq_terms/eval_cq_terms_result.json","Direct term prediction; OpenRouter; temperature 0; 16384 output-token limit; wine","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Term/03_evaluation_results/deepseek-v4-flash/wine/06_cq_terms/eval_cq_terms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Term/03_evaluation_results/deepseek-v4-pro/wine/06_cq_terms/eval_cq_terms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Term/03_evaluation_results/deepseek_deepseek-v3.2/wine/06_cq_terms/eval_cq_terms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Term/03_evaluation_results/google_gemma-4-26b-a4b-it/wine/06_cq_terms/eval_cq_terms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Term/03_evaluation_results/google_gemma-4-31b-it/wine/06_cq_terms/eval_cq_terms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Term/03_evaluation_results/qwen_qwen3.6-27b/wine/06_cq_terms/eval_cq_terms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Term/03_evaluation_results/qwen_qwen3.6-35b-a3b/wine/06_cq_terms/eval_cq_terms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Term/03_evaluation_results/qwen_qwen3.6-flash/wine/06_cq_terms/eval_cq_terms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Term/03_evaluation_results/qwen_qwen3.6-plus/wine/06_cq_terms/eval_cq_terms_result.json","All CQs in one prompt; OpenRouter; temperature 0; 16384 output-token limit; awo","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/normal/deepseek-v4-flash/awo/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/normal/deepseek-v4-pro/awo/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/normal/deepseek_deepseek-v3.2/awo/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/normal/google_gemma-4-26b-a4b-it/awo/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/normal/google_gemma-4-31b-it/awo/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/normal/qwen_qwen3.6-27b/awo/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/normal/qwen_qwen3.6-35b-a3b/awo/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/normal/qwen_qwen3.6-flash/awo/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/normal/qwen_qwen3.6-plus/awo/04_axiom/eval_axioms_result.json","All CQs in one prompt; OpenRouter; temperature 0; 16384 output-token limit; odrl","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/normal/deepseek-v4-flash/odrl/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/normal/deepseek-v4-pro/odrl/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/normal/deepseek_deepseek-v3.2/odrl/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/normal/google_gemma-4-26b-a4b-it/odrl/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/normal/google_gemma-4-31b-it/odrl/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/normal/qwen_qwen3.6-27b/odrl/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/normal/qwen_qwen3.6-35b-a3b/odrl/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/normal/qwen_qwen3.6-flash/odrl/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/normal/qwen_qwen3.6-plus/odrl/04_axiom/eval_axioms_result.json","All CQs in one prompt; OpenRouter; temperature 0; 16384 output-token limit; swo","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/normal/deepseek-v4-flash/swo/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/normal/deepseek-v4-pro/swo/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/normal/deepseek_deepseek-v3.2/swo/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/normal/google_gemma-4-26b-a4b-it/swo/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/normal/google_gemma-4-31b-it/swo/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/normal/qwen_qwen3.6-27b/swo/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/normal/qwen_qwen3.6-35b-a3b/swo/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/normal/qwen_qwen3.6-flash/swo/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/normal/qwen_qwen3.6-plus/swo/04_axiom/eval_axioms_result.json","All CQs in one prompt; OpenRouter; temperature 0; 16384 output-token limit; vgo","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/normal/deepseek-v4-flash/vgo/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/normal/deepseek-v4-pro/vgo/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/normal/deepseek_deepseek-v3.2/vgo/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/normal/google_gemma-4-26b-a4b-it/vgo/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/normal/google_gemma-4-31b-it/vgo/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/normal/qwen_qwen3.6-27b/vgo/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/normal/qwen_qwen3.6-35b-a3b/vgo/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/normal/qwen_qwen3.6-flash/vgo/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/normal/qwen_qwen3.6-plus/vgo/04_axiom/eval_axioms_result.json","All CQs in one prompt; OpenRouter; temperature 0; 16384 output-token limit; water","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/normal/deepseek-v4-flash/water/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/normal/deepseek-v4-pro/water/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/normal/deepseek_deepseek-v3.2/water/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/normal/google_gemma-4-26b-a4b-it/water/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/normal/google_gemma-4-31b-it/water/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/normal/qwen_qwen3.6-27b/water/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/normal/qwen_qwen3.6-35b-a3b/water/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/normal/qwen_qwen3.6-flash/water/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/normal/qwen_qwen3.6-plus/water/04_axiom/eval_axioms_result.json","All CQs in one prompt; OpenRouter; temperature 0; 16384 output-token limit; wine","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/normal/deepseek-v4-flash/wine/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/normal/deepseek-v4-pro/wine/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/normal/deepseek_deepseek-v3.2/wine/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/normal/google_gemma-4-26b-a4b-it/wine/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/normal/google_gemma-4-31b-it/wine/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/normal/qwen_qwen3.6-27b/wine/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/normal/qwen_qwen3.6-35b-a3b/wine/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/normal/qwen_qwen3.6-flash/wine/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/normal/qwen_qwen3.6-plus/wine/04_axiom/eval_axioms_result.json","One CQ at a time; OpenRouter; temperature 0; 16384 output-token limit; awo","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/cqbycq/deepseek-v4-flash/awo/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/cqbycq/deepseek-v4-pro/awo/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/cqbycq/deepseek_deepseek-v3.2/awo/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/cqbycq/google_gemma-4-26b-a4b-it/awo/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/cqbycq/google_gemma-4-31b-it/awo/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/cqbycq/qwen_qwen3.6-27b/awo/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/cqbycq/qwen_qwen3.6-35b-a3b/awo/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/cqbycq/qwen_qwen3.6-flash/awo/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/cqbycq/qwen_qwen3.6-plus/awo/04_axiom/eval_axioms_result.json","One CQ at a time; OpenRouter; temperature 0; 16384 output-token limit; odrl","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/cqbycq/deepseek-v4-flash/odrl/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/cqbycq/deepseek-v4-pro/odrl/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/cqbycq/deepseek_deepseek-v3.2/odrl/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/cqbycq/google_gemma-4-26b-a4b-it/odrl/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/cqbycq/google_gemma-4-31b-it/odrl/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/cqbycq/qwen_qwen3.6-27b/odrl/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/cqbycq/qwen_qwen3.6-35b-a3b/odrl/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/cqbycq/qwen_qwen3.6-flash/odrl/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/cqbycq/qwen_qwen3.6-plus/odrl/04_axiom/eval_axioms_result.json","One CQ at a time; OpenRouter; temperature 0; 16384 output-token limit; swo","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/cqbycq/deepseek-v4-flash/swo/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/cqbycq/deepseek-v4-pro/swo/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/cqbycq/deepseek_deepseek-v3.2/swo/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/cqbycq/google_gemma-4-26b-a4b-it/swo/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/cqbycq/google_gemma-4-31b-it/swo/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/cqbycq/qwen_qwen3.6-27b/swo/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/cqbycq/qwen_qwen3.6-35b-a3b/swo/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/cqbycq/qwen_qwen3.6-flash/swo/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/cqbycq/qwen_qwen3.6-plus/swo/04_axiom/eval_axioms_result.json","One CQ at a time; OpenRouter; temperature 0; 16384 output-token limit; vgo","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/cqbycq/deepseek-v4-flash/vgo/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/cqbycq/deepseek-v4-pro/vgo/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/cqbycq/deepseek_deepseek-v3.2/vgo/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/cqbycq/google_gemma-4-26b-a4b-it/vgo/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/cqbycq/google_gemma-4-31b-it/vgo/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/cqbycq/qwen_qwen3.6-27b/vgo/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/cqbycq/qwen_qwen3.6-35b-a3b/vgo/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/cqbycq/qwen_qwen3.6-flash/vgo/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/cqbycq/qwen_qwen3.6-plus/vgo/04_axiom/eval_axioms_result.json","One CQ at a time; OpenRouter; temperature 0; 16384 output-token limit; water","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/cqbycq/deepseek-v4-flash/water/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/cqbycq/deepseek-v4-pro/water/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/cqbycq/deepseek_deepseek-v3.2/water/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/cqbycq/google_gemma-4-26b-a4b-it/water/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/cqbycq/google_gemma-4-31b-it/water/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/cqbycq/qwen_qwen3.6-27b/water/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/cqbycq/qwen_qwen3.6-35b-a3b/water/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/cqbycq/qwen_qwen3.6-flash/water/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/cqbycq/qwen_qwen3.6-plus/water/04_axiom/eval_axioms_result.json","One CQ at a time; OpenRouter; temperature 0; 16384 output-token limit; wine","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/cqbycq/deepseek-v4-flash/wine/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/cqbycq/deepseek-v4-pro/wine/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/cqbycq/deepseek_deepseek-v3.2/wine/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/cqbycq/google_gemma-4-26b-a4b-it/wine/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/cqbycq/google_gemma-4-31b-it/wine/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/cqbycq/qwen_qwen3.6-27b/wine/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/cqbycq/qwen_qwen3.6-35b-a3b/wine/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/cqbycq/qwen_qwen3.6-flash/wine/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/cqbycq/qwen_qwen3.6-plus/wine/04_axiom/eval_axioms_result.json","Agent: MASEO; up to three repair/backtracking iterations; OpenRouter; temperature 0; 16384 output-token limit; awo","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/agent/deepseek-v4-flash/awo/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/agent/deepseek-v4-pro/awo/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/agent/deepseek_deepseek-v3.2/awo/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/agent/google_gemma-4-26b-a4b-it/awo/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/agent/google_gemma-4-31b-it/awo/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/agent/qwen_qwen3.6-27b/awo/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/agent/qwen_qwen3.6-35b-a3b/awo/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/agent/qwen_qwen3.6-flash/awo/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/agent/qwen_qwen3.6-plus/awo/04_axiom/eval_axioms_result.json","Agent: MASEO; up to three repair/backtracking iterations; OpenRouter; temperature 0; 16384 output-token limit; odrl","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/agent/deepseek-v4-flash/odrl/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/agent/deepseek-v4-pro/odrl/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/agent/deepseek_deepseek-v3.2/odrl/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/agent/google_gemma-4-26b-a4b-it/odrl/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/agent/google_gemma-4-31b-it/odrl/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/agent/qwen_qwen3.6-27b/odrl/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/agent/qwen_qwen3.6-35b-a3b/odrl/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/agent/qwen_qwen3.6-flash/odrl/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/agent/qwen_qwen3.6-plus/odrl/04_axiom/eval_axioms_result.json","Agent: MASEO; up to three repair/backtracking iterations; OpenRouter; temperature 0; 16384 output-token limit; swo","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/agent/deepseek-v4-flash/swo/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/agent/deepseek-v4-pro/swo/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/agent/deepseek_deepseek-v3.2/swo/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/agent/google_gemma-4-26b-a4b-it/swo/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/agent/google_gemma-4-31b-it/swo/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/agent/qwen_qwen3.6-27b/swo/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/agent/qwen_qwen3.6-35b-a3b/swo/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/agent/qwen_qwen3.6-flash/swo/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/agent/qwen_qwen3.6-plus/swo/04_axiom/eval_axioms_result.json","Agent: MASEO; up to three repair/backtracking iterations; OpenRouter; temperature 0; 16384 output-token limit; vgo","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/agent/deepseek-v4-flash/vgo/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/agent/deepseek-v4-pro/vgo/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/agent/deepseek_deepseek-v3.2/vgo/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/agent/google_gemma-4-26b-a4b-it/vgo/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/agent/google_gemma-4-31b-it/vgo/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/agent/qwen_qwen3.6-27b/vgo/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/agent/qwen_qwen3.6-35b-a3b/vgo/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/agent/qwen_qwen3.6-flash/vgo/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/agent/qwen_qwen3.6-plus/vgo/04_axiom/eval_axioms_result.json","Agent: MASEO; up to three repair/backtracking iterations; OpenRouter; temperature 0; 16384 output-token limit; water","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/agent/deepseek-v4-flash/water/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/agent/deepseek-v4-pro/water/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/agent/deepseek_deepseek-v3.2/water/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/agent/google_gemma-4-26b-a4b-it/water/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/agent/google_gemma-4-31b-it/water/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/agent/qwen_qwen3.6-27b/water/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/agent/qwen_qwen3.6-35b-a3b/water/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/agent/qwen_qwen3.6-flash/water/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/agent/qwen_qwen3.6-plus/water/04_axiom/eval_axioms_result.json","Agent: MASEO; up to three repair/backtracking iterations; OpenRouter; temperature 0; 16384 output-token limit; wine","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/agent/deepseek-v4-flash/wine/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/agent/deepseek-v4-pro/wine/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/agent/deepseek_deepseek-v3.2/wine/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/agent/google_gemma-4-26b-a4b-it/wine/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/agent/google_gemma-4-31b-it/wine/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/agent/qwen_qwen3.6-27b/wine/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/agent/qwen_qwen3.6-35b-a3b/wine/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/agent/qwen_qwen3.6-flash/wine/04_axiom/eval_axioms_result.json","https://github.com/oeg-upm/cq4oe-benchmark/blob/ded082853bf4e8d0a84070063c67c9f156654594/CQ2Onto/03_evaluation_results/agent/qwen_qwen3.6-plus/wine/04_axiom/eval_axioms_result.json","EasyHuman video L=256; published answer accuracy","https://arxiv.org/html/2609.00103v2#A1.T1","InternVL3.5 (8B)","InternVL3.5 Thinking (30B, A3B)","Phi-4-MM","Qwen3-VL Thinking","MiMo-VL","GPT-5.4 Pro","Audio-Flamingo-3 8B","DailyTalk subset; source size 8B; Audio-language embedding","https://arxiv.org/html/2608.16203v1#S5.T4","Qwen2.5-Omni 3B","DailyTalk subset; source size 3B; Audio-language embedding","DailyTalk subset; source size 7B; Audio-language embedding","Qwen3-Omni 30B","DailyTalk subset; source size 30B; Audio-language embedding","Voxtral-Mini 3B","Voxtral-Small 24B","DailyTalk subset; source size 24B; Audio-language embedding","DailyTalk subset; source size -B; Cascaded textual pipeline","SentenceBERT 0.1B","DailyTalk subset; source size 0.1B; Cascaded textual pipeline","E5-Mistral 7B","DailyTalk subset; source size 7B; Cascaded textual pipeline","Qwen3-Embedding 8B","DailyTalk subset; source size 8B; Cascaded textual pipeline","HuBERT-Large 0.3B","DailyTalk subset; source size 0.3B; Speech embedding","WavLM-Large 0.3B","LAION-CLAP 0.2B","DailyTalk subset; source size 0.2B; Contrastive audio-language embedding","VCTK subset; source size 8B; Audio-language embedding","VCTK subset; source size 3B; Audio-language embedding","VCTK subset; source size 7B; Audio-language embedding","VCTK subset; source size 30B; Audio-language embedding","VCTK subset; source size 24B; Audio-language embedding","VCTK subset; source size -B; Cascaded textual pipeline","VCTK subset; source size 0.1B; Cascaded textual pipeline","VCTK subset; source size 7B; Cascaded textual pipeline","VCTK subset; source size 8B; Cascaded textual pipeline","VCTK subset; source size 0.3B; Speech embedding","VCTK subset; source size 0.2B; Contrastive audio-language embedding","Expresso subset; source size 8B; Audio-language embedding","Expresso subset; source size 3B; Audio-language embedding","Expresso subset; source size 7B; Audio-language embedding","Expresso subset; source size 30B; Audio-language embedding","Expresso subset; source size 24B; Audio-language embedding","Expresso subset; source size -B; Cascaded textual pipeline","Expresso subset; source size 0.1B; Cascaded textual pipeline","Expresso subset; source size 7B; Cascaded textual pipeline","Expresso subset; source size 8B; Cascaded textual pipeline","Expresso subset; source size 0.3B; Speech embedding","Expresso subset; source size 0.2B; Contrastive audio-language embedding","Synthetic subset; source size 8B; Audio-language embedding","Synthetic subset; source size 3B; Audio-language embedding","Synthetic subset; source size 7B; Audio-language embedding","Synthetic subset; source size 30B; Audio-language embedding","Synthetic subset; source size 24B; Audio-language embedding","Synthetic subset; source size -B; Cascaded textual pipeline","Synthetic subset; source size 0.1B; Cascaded textual pipeline","Synthetic subset; source size 7B; Cascaded textual pipeline","Synthetic subset; source size 8B; Cascaded textual pipeline","Synthetic subset; source size 0.3B; Speech embedding","Synthetic subset; source size 0.2B; Contrastive audio-language embedding","Stockfish (d15)","maia split; Task-specific chess engine/policy","https://arxiv.org/html/2609.00474v2#A4.T14","Maia","Allie-Policy","Allie-Adaptive-Search","GPT-5 (text only)","maia split; Five-shot frontier baseline","GPT-5 + Lc0","Qwen3-14B + Lc0","LLAMIA-Verb-4B","maia split; Verbalized SFT","LLAMIA-Verb-8B","LLAMIA-Verb-14B","maia split; Verbalized DAPO","LLAMIA-4B","maia split; Latent SFT","LLAMIA-8B","LLAMIA-14B","maia split; Latent DAPO","wild split; Task-specific chess engine/policy","wild split; Five-shot frontier baseline","wild split; Verbalized SFT","wild split; Verbalized DAPO","wild split; Latent SFT","wild split; Latent DAPO","claude-opus-4.7","Closed-book regions questions; source default inference","https://arxiv.org/html/2608.20106v1#A3.T12","qwen-2.5-72b","qwen-2.5-7b","mistral-large-2411","gemini-2.5-pro-thinking","Closed-book regions questions; thinking enabled","claude-opus-4.7-thinking","Closed-book varieties questions; source default inference","Closed-book varieties questions; thinking enabled","Closed-book producers questions; source default inference","Closed-book producers questions; thinking enabled","Closed-book viticulture questions; source default inference","Closed-book viticulture questions; thinking enabled","Closed-book winemaking questions; source default inference","Closed-book winemaking questions; thinking enabled","Closed-book business questions; source default inference","Closed-book business questions; thinking enabled","Qwen2.5VL 3B","Image + vision-language model; token correction F1; no oracle OCR text","https://arxiv.org/html/2608.27154v1#S3.T1","Qwen2.5VL 7B","Qwen2.5VL 32B","Qwen2.5VL 72B","Qwen3VL 2B","Qwen3VL 4B","Qwen3VL 8B","Qwen3VL 32B","Qwen3VL 235B","InternVL 2B","InternVL 4B","InternVL 8B","InternVL 14B","InternVL 38B","GPT–4o","GPT–5.4","Seed–1.8","Seed–2.0","Gemini2.5–Pro","Gemini3.1–Pro","XiYanSQL-32B","Fine-tuned SQL checkpoint; original pipeline","https://arxiv.org/html/2608.29345v1#A2.T11","CscSQL-7B","SLM-SQL-1.5B","SQL-R1-14B","Qwen3 qwen-plus-2025-07-28","Prompt framework MAC-SQL; temperature 0","Prompt framework Alpha-SQL; temperature 0","Prompt framework TA-SQL; temperature 0","Prompt framework DIN-SQL; temperature 0","Prompt framework Feather-SQL; temperature 0","Foundation model original prompting; temperature 0","https://arxiv.org/html/2608.29345v1#A2.T12","Llama3.3-70B-Instruct","OpenClaw; 128 lifecycle cases × 3 independent sandbox repetitions; ASR lower is better","https://arxiv.org/pdf/2608.17597#page=7","Nanobot; 128 lifecycle cases × 3 independent sandbox repetitions; ASR lower is better","Hermes; 128 lifecycle cases × 3 independent sandbox repetitions; ASR lower is better","OpenCode; 163 repository tasks × 2 runs; executable patch and regression tests","https://arxiv.org/pdf/2608.09072#page=5","Claude Code; 163 repository tasks × 2 runs; executable patch and regression tests","Codex; 163 repository tasks × 2 runs; executable patch and regression tests","NVIDIA A100 MIG 40 GB; 256px; batch 1; 100 warmups/1,000 timed passes × 5 repeats; model-only","https://arxiv.org/html/2608.29677v1#S4.T3","MISSFormer","HiFormer-B*","SegFormer","SegNeXt","VW-MiT","CENet","VW-Conv","TransNeXt","InternImage","SU-Mamba","NVIDIA A100 MIG 40 GB; 512px; batch 1; 100 warmups/1,000 timed passes × 5 repeats; model-only","Published Figure 3A; same 20 tasks; source bar 1","https://arxiv.org/pdf/2608.25286#page=5","Published Figure 3A; same 20 tasks; source bar 2","Published Figure 3A; same 20 tasks; source bar 3","Published Figure 3A; same 20 tasks; source bar 4","Published Figure 3A; same 20 tasks; source bar 5","Published Figure 3A; same 20 tasks; source bar 6","Published Figure 3A; same 20 tasks; source bar 7","Published Figure 3A; same 20 tasks; source bar 8","Published Figure 3A; same 20 tasks; source bar 9","Published Figure 3A; same 20 tasks; source bar 10","Published Figure 3A; same 20 tasks; source bar 11","Published Figure 3A; same 20 tasks; source bar 12","Published Figure 3A; same 20 tasks; source bar 13","Chess fixed exploration/evaluation setup; ICL direct history context","https://arxiv.org/html/2608.31100v1#S5.T4","Chess fixed exploration/evaluation setup; Cross-episode summary memory","Minesweeper fixed exploration/evaluation setup; ICL direct history context","Minesweeper fixed exploration/evaluation setup; Cross-episode summary memory","Nullify fixed exploration/evaluation setup; ICL direct history context","Nullify fixed exploration/evaluation setup; Cross-episode summary memory","Tetris fixed exploration/evaluation setup; ICL direct history context","Tetris fixed exploration/evaluation setup; Cross-episode summary memory","Snake fixed exploration/evaluation setup; ICL direct history context","Snake fixed exploration/evaluation setup; Cross-episode summary memory","PvZ fixed exploration/evaluation setup; ICL direct history context","PvZ fixed exploration/evaluation setup; Cross-episode summary memory","TrustEvo fixed exploration/evaluation setup; ICL direct history context","TrustEvo fixed exploration/evaluation setup; Cross-episode summary memory","Gemini-Omni-Flash","Common 150-example testmini; human rubric 1–5","https://arxiv.org/html/2608.09873v1#S5.T4","Veo-3.1-Fast","Sora-2","Kling-2.6","Wan-2.6","CogVideoX1.5-5B","Native official page-image input (V); fixed Gemini 3.5 Flash judge; whole-sequence scoring","https://arxiv.org/html/2608.30517v1#A5.T7","Seed 2.0 Pro","Frozen question-only text input (T); fixed Gemini 3.5 Flash judge; whole-sequence scoring","DeepSeekV4-Flash","Claude Code (Opus 4.6)","Agentic system without Numina; full 178 problems; strict SA-Pass","https://arxiv.org/html/2608.29270v3#S5.T3","Agentic system; Numina tool access; full 178 problems; strict SA-Pass","Claude Code (Opus 4.8)","Claude Code (Qwen3 235B)","Codex (GPT-5.4)","Non-agentic LLM; full 178 problems; strict SA-Pass","DeepSeek R1 0528","Qwen3-coder","Llama3.1 70b","Llama3.1 8b","Kimina 7B → 8B","Specialized autoformalizer → prover pipeline; full 178 problems; strict SA-Pass","Goedel 8B → 32B","Goedel 8B → 8B","HeadQA expert-reviewed MCQA; EN/DE/ES/PT/JA/ZH/TH/SW/ZU equal-language mean","https://arxiv.org/html/2608.19981v1#A8.T10","LLaMA3.3-70B-Instruct","Qwen3-32B-thinking","MediPhi","MedQA expert-reviewed MCQA; EN/DE/ES/PT/JA/ZH/TH/SW/ZU equal-language mean","MedExpQA expert-reviewed MCQA; EN/DE/ES/PT/JA/ZH/TH/SW/ZU equal-language mean","MMLU-Pro expert-reviewed MCQA; EN/DE/ES/PT/JA/ZH/TH/SW/ZU equal-language mean","Baseline original figure + original caption; native weighted MQM","https://arxiv.org/html/2608.13267v1#S3.T3","Llama 4 Maverick 17B 128E","Qwen3-VL 235B A22B Instruct","Qwen3-VL 8B Instruct","Qwen3-VL 30B A3B Instruct","Phi-4 Multimodal","Same 1,000 capability questions on 250 figures; source Overall","https://arxiv.org/html/2608.13267v1#A4.T13","SigLIP2-SO400M","Joint image+text retrieval; source corruption Overall Recall@1","https://arxiv.org/html/2608.29607v1#S4.T2","CLIP-ViT-L/14","BLIP-ITM-L","SigLIP-SO400M","VLM2Vec-Full","E5-V","UME-R1-7B","GME-2B","Jina-V4","GME-7B","RzenEmbed-7B","Ops-MM-2B","Ops-MM-7B","Shared single-family deterministic tool perturbations; source Overall E2E success","https://arxiv.org/html/2608.23635v1#S5.T5","qwen-plus","All provided modalities; simple checklist-question mean","https://arxiv.org/html/2608.30068v2#S5.T3","Gemma 4 E4B","Gemma 4 12B","InternVL3.5-30B","Gemma 4 26B","Gemini 3.5 Flash","All provided modalities; specific checklist-question mean","Three independent runs; maximum 50 turns; GPT-5.5 (xhigh)","https://arxiv.org/pdf/2608.10692#page=7","Three independent runs; maximum 50 turns; Gemini 3.1 Pro (high)","Three independent runs; maximum 50 turns; Claude Opus 4.8 (max)","DeepSeek-V4-Pro (max)","Three independent runs; maximum 50 turns; DeepSeek-V4-Pro (max)","Gemini 3.1 Pro (low)","Three independent runs; maximum 50 turns; Gemini 3.1 Pro (low)","Claude Opus 4.8 (none)","Three independent runs; maximum 50 turns; Claude Opus 4.8 (none)","Kimi K2.6 (high)","Three independent runs; maximum 50 turns; Kimi K2.6 (high)","GLM-5.2 (max)","Three independent runs; maximum 50 turns; GLM-5.2 (max)","Hy3 (high)","Three independent runs; maximum 50 turns; Hy3 (high)","DeepSeek-V4-Pro (none)","Three independent runs; maximum 50 turns; DeepSeek-V4-Pro (none)","Qwen3.7-Plus (high)","Three independent runs; maximum 50 turns; Qwen3.7-Plus (high)","Seed-2.1-Pro (high)","Three independent runs; maximum 50 turns; Seed-2.1-Pro (high)","GLM-5.2 (none)","Three independent runs; maximum 50 turns; GLM-5.2 (none)","GPT-5.5 (none)","Three independent runs; maximum 50 turns; GPT-5.5 (none)","Seed-2.1-Pro (minimal)","Three independent runs; maximum 50 turns; Seed-2.1-Pro (minimal)","Qwen3.7-Plus (none)","Three independent runs; maximum 50 turns; Qwen3.7-Plus (none)","Hy3 (no_think)","Three independent runs; maximum 50 turns; Hy3 (no_think)","Kimi K2.6 (none)","Three independent runs; maximum 50 turns; Kimi K2.6 (none)","Gold n1843; structured; original fixed reranker recipe; source checkpoint: ZooWork-ShopRanker-8B","https://arxiv.org/html/2609.31002v1#A2.T12","Gold n1843; structured; original fixed reranker recipe; source checkpoint: ZooWork-ShopRanker-4B","Gold n1843; structured; original fixed reranker recipe; source checkpoint: Qwen3-Rnk-8B","Jina-m0 (2.4B)","Gold n1843; structured; original fixed reranker recipe; source checkpoint: Jina-m0 (2.4B)","Gold n1843; structured; original fixed reranker recipe; source checkpoint: Qwen3-Rnk-4B","Gold n1843; structured; original fixed reranker recipe; source checkpoint: ZooWork-ShopRanker-0.6B","Gold n1843; structured; original fixed reranker recipe; source checkpoint: Qwen3-Rnk-0.6B","BGE-v2-m3 (0.6B)","Gold n1843; structured; original fixed reranker recipe; source checkpoint: BGE-v2-m3 (0.6B)","BGE-large (0.6B)","Gold n1843; structured; original fixed reranker recipe; source checkpoint: BGE-large (0.6B)","Gold n1843; natural-language; original fixed reranker recipe; source checkpoint: ZooWork-ShopRanker-8B","https://arxiv.org/html/2609.31002v1#A2.T13","Gold n1843; natural-language; original fixed reranker recipe; source checkpoint: ZooWork-ShopRanker-4B","Gold n1843; natural-language; original fixed reranker recipe; source checkpoint: Qwen3-Rnk-8B","Gold n1843; natural-language; original fixed reranker recipe; source checkpoint: Jina-m0 (2.4B)","Gold n1843; natural-language; original fixed reranker recipe; source checkpoint: Qwen3-Rnk-4B","Gold n1843; natural-language; original fixed reranker recipe; source checkpoint: ZooWork-ShopRanker-0.6B","Gold n1843; natural-language; original fixed reranker recipe; source checkpoint: BGE-v2-m3 (0.6B)","Gold n1843; natural-language; original fixed reranker recipe; source checkpoint: BGE-large (0.6B)","Gold n1843; natural-language; original fixed reranker recipe; source checkpoint: Qwen3-Rnk-0.6B","GLM-4V-Flash","Zero-shot; full20000QA; native normalized composite","https://arxiv.org/html/2609.32352v1#S3.T1","GLM-4.1V-Flash","Intern-S2-35B","Intern-S2-397B","GPT-6 Astra (Codex CLI)","GPT-6 Astra (Codex CLI);3attempts per fixed task; task-specific shared budgets","https://arxiv.org/html/2609.33807v1#A4.T10","Claude Opus 5 (Reference);3attempts per fixed task; task-specific shared budgets","Claude Opus 5 (Claude Code);3attempts per fixed task; task-specific shared budgets","Gemini 3.6 Flash (Reference)","Gemini 3.6 Flash (Reference);3attempts per fixed task; task-specific shared budgets","GPT-5.6 Sol (Reference)","GPT-5.6 Sol (Reference);3attempts per fixed task; task-specific shared budgets","Qwen 3.8 Max (Reference)","Qwen 3.8 Max (Reference);3attempts per fixed task; task-specific shared budgets","Grok 4.6 (Reference)","Grok 4.6 (Reference);3attempts per fixed task; task-specific shared budgets","Kimi K3 (Reference)","Kimi K3 (Reference);3attempts per fixed task; task-specific shared budgets","Claude Sonnet 5 (Reference)","Claude Sonnet 5 (Reference);3attempts per fixed task; task-specific shared budgets","Fixed19tasks; Single","https://arxiv.org/html/2609.32662v1#A9.T12","Nemotron-3.5","Fixed19tasks; Serial","Fixed19tasks; Async-Private","Fixed19tasks; Async-Manager-RO","Fixed19tasks; Async-Manager","REALIS training; cleanTest100000; original published detector recipe","https://arxiv.org/html/2609.32734v1#S4.T1","DIRE","EFFORT","FreqNet","MaskSim","RINE","SAFE","SPAI","UFD","Video-LLaMA2 7B","Base ReAct; synthetic all; provider-native audio input; Video-LLaMA2 7B","https://arxiv.org/html/2609.32569v1#S4.T2","Base ReAct; synthetic all; provider-native audio input; Qwen2.5-Omni 7B","Spatial-Omni 7B","Base ReAct; synthetic all; provider-native audio input; Spatial-Omni 7B","Base ReAct; synthetic all; provider-native audio input; video-SALMONN2+ 7B","video-SALMONN-o1 7B","Base ReAct; synthetic all; provider-native audio input; video-SALMONN-o1 7B","OmniVinci 9B","Base ReAct; synthetic all; provider-native audio input; OmniVinci 9B","MiniCPM-o-4.5 9B","Base ReAct; synthetic all; provider-native audio input; MiniCPM-o-4.5 9B","Base ReAct; synthetic all; provider-native audio input; Gemma4 12B","Nemotron3-Omni 30B-A3B","Base ReAct; synthetic all; provider-native audio input; Nemotron3-Omni 30B-A3B","Qwen3-Omni 30B-A3B","Base ReAct; synthetic all; provider-native audio input; Qwen3-Omni-Instruct 30B-A3B","Base ReAct; synthetic all; provider-native audio input; Qwen3-Omni-Think 30B-A3B","MiMo-V2.5 310B-A15B","Base ReAct; synthetic all; provider-native audio input; MiMo-V2.5 310B-A15B","Base ReAct; synthetic all; provider-native audio input; Gemini-2.5-Flash","Base ReAct; synthetic all; provider-native audio input; Gemini-2.5-Pro","Gemini-3.1-Pro (Preview)","Base ReAct; synthetic all; provider-native audio input; Gemini-3.1-Pro (Preview)","Base ReAct; synthetic all; provider-native audio input; Qwen3.8-Omni-Flash","Base ReAct; synthetic speech; provider-native audio input; Video-LLaMA2 7B","Base ReAct; synthetic speech; provider-native audio input; Qwen2.5-Omni 7B","Base ReAct; synthetic speech; provider-native audio input; Spatial-Omni 7B","Base ReAct; synthetic speech; provider-native audio input; video-SALMONN2+ 7B","Base ReAct; synthetic speech; provider-native audio input; video-SALMONN-o1 7B","Base ReAct; synthetic speech; provider-native audio input; OmniVinci 9B","Base ReAct; synthetic speech; provider-native audio input; MiniCPM-o-4.5 9B","Base ReAct; synthetic speech; provider-native audio input; Gemma4 12B","Base ReAct; synthetic speech; provider-native audio input; Nemotron3-Omni 30B-A3B","Base ReAct; synthetic speech; provider-native audio input; Qwen3-Omni-Instruct 30B-A3B","Base ReAct; synthetic speech; provider-native audio input; Qwen3-Omni-Think 30B-A3B","Base ReAct; synthetic speech; provider-native audio input; MiMo-V2.5 310B-A15B","Base ReAct; synthetic speech; provider-native audio input; Gemini-2.5-Flash","Base ReAct; synthetic speech; provider-native audio input; Gemini-2.5-Pro","Base ReAct; synthetic speech; provider-native audio input; Gemini-3.1-Pro (Preview)","Base ReAct; synthetic speech; provider-native audio input; Qwen3.8-Omni-Flash","Base ReAct; synthetic gesture; provider-native audio input; Video-LLaMA2 7B","Base ReAct; synthetic gesture; provider-native audio input; Qwen2.5-Omni 7B","Base ReAct; synthetic gesture; provider-native audio input; Spatial-Omni 7B","Base ReAct; synthetic gesture; provider-native audio input; video-SALMONN2+ 7B","Base ReAct; synthetic gesture; provider-native audio input; video-SALMONN-o1 7B","Base ReAct; synthetic gesture; provider-native audio input; OmniVinci 9B","Base ReAct; synthetic gesture; provider-native audio input; MiniCPM-o-4.5 9B","Base ReAct; synthetic gesture; provider-native audio input; Gemma4 12B","Base ReAct; synthetic gesture; provider-native audio input; Nemotron3-Omni 30B-A3B","Base ReAct; synthetic gesture; provider-native audio input; Qwen3-Omni-Instruct 30B-A3B","Base ReAct; synthetic gesture; provider-native audio input; Qwen3-Omni-Think 30B-A3B","Base ReAct; synthetic gesture; provider-native audio input; MiMo-V2.5 310B-A15B","Base ReAct; synthetic gesture; provider-native audio input; Gemini-2.5-Flash","Base ReAct; synthetic gesture; provider-native audio input; Gemini-2.5-Pro","Base ReAct; synthetic gesture; provider-native audio input; Gemini-3.1-Pro (Preview)","Base ReAct; synthetic gesture; provider-native audio input; Qwen3.8-Omni-Flash","Base ReAct; synthetic sound; provider-native audio input; Video-LLaMA2 7B","Base ReAct; synthetic sound; provider-native audio input; Qwen2.5-Omni 7B","Base ReAct; synthetic sound; provider-native audio input; Spatial-Omni 7B","Base ReAct; synthetic sound; provider-native audio input; video-SALMONN2+ 7B","Base ReAct; synthetic sound; provider-native audio input; video-SALMONN-o1 7B","Base ReAct; synthetic sound; provider-native audio input; OmniVinci 9B","Base ReAct; synthetic sound; provider-native audio input; MiniCPM-o-4.5 9B","Base ReAct; synthetic sound; provider-native audio input; Gemma4 12B","Base ReAct; synthetic sound; provider-native audio input; Nemotron3-Omni 30B-A3B","Base ReAct; synthetic sound; provider-native audio input; Qwen3-Omni-Instruct 30B-A3B","Base ReAct; synthetic sound; provider-native audio input; Qwen3-Omni-Think 30B-A3B","Base ReAct; synthetic sound; provider-native audio input; MiMo-V2.5 310B-A15B","Base ReAct; synthetic sound; provider-native audio input; Gemini-2.5-Flash","Base ReAct; synthetic sound; provider-native audio input; Gemini-2.5-Pro","Base ReAct; synthetic sound; provider-native audio input; Gemini-3.1-Pro (Preview)","Base ReAct; synthetic sound; provider-native audio input; Qwen3.8-Omni-Flash","Base ReAct; synthetic space; provider-native audio input; Video-LLaMA2 7B","Base ReAct; synthetic space; provider-native audio input; Qwen2.5-Omni 7B","Base ReAct; synthetic space; provider-native audio input; Spatial-Omni 7B","Base ReAct; synthetic space; provider-native audio input; video-SALMONN2+ 7B","Base ReAct; synthetic space; provider-native audio input; video-SALMONN-o1 7B","Base ReAct; synthetic space; provider-native audio input; OmniVinci 9B","Base ReAct; synthetic space; provider-native audio input; MiniCPM-o-4.5 9B","Base ReAct; synthetic space; provider-native audio input; Gemma4 12B","Base ReAct; synthetic space; provider-native audio input; Nemotron3-Omni 30B-A3B","Base ReAct; synthetic space; provider-native audio input; Qwen3-Omni-Instruct 30B-A3B","Base ReAct; synthetic space; provider-native audio input; Qwen3-Omni-Think 30B-A3B","Base ReAct; synthetic space; provider-native audio input; MiMo-V2.5 310B-A15B","Base ReAct; synthetic space; provider-native audio input; Gemini-2.5-Flash","Base ReAct; synthetic space; provider-native audio input; Gemini-2.5-Pro","Base ReAct; synthetic space; provider-native audio input; Gemini-3.1-Pro (Preview)","Base ReAct; synthetic space; provider-native audio input; Qwen3.8-Omni-Flash","Base ReAct; synthetic identity; provider-native audio input; Video-LLaMA2 7B","Base ReAct; synthetic identity; provider-native audio input; Qwen2.5-Omni 7B","Base ReAct; synthetic identity; provider-native audio input; Spatial-Omni 7B","Base ReAct; synthetic identity; provider-native audio input; video-SALMONN2+ 7B","Base ReAct; synthetic identity; provider-native audio input; video-SALMONN-o1 7B","Base ReAct; synthetic identity; provider-native audio input; OmniVinci 9B","Base ReAct; synthetic identity; provider-native audio input; MiniCPM-o-4.5 9B","Base ReAct; synthetic identity; provider-native audio input; Gemma4 12B","Base ReAct; synthetic identity; provider-native audio input; Nemotron3-Omni 30B-A3B","Base ReAct; synthetic identity; provider-native audio input; Qwen3-Omni-Instruct 30B-A3B","Base ReAct; synthetic identity; provider-native audio input; Qwen3-Omni-Think 30B-A3B","Base ReAct; synthetic identity; provider-native audio input; MiMo-V2.5 310B-A15B","Base ReAct; synthetic identity; provider-native audio input; Gemini-2.5-Flash","Base ReAct; synthetic identity; provider-native audio input; Gemini-2.5-Pro","Base ReAct; synthetic identity; provider-native audio input; Gemini-3.1-Pro (Preview)","Base ReAct; synthetic identity; provider-native audio input; Qwen3.8-Omni-Flash","MIC-Bench id; original baseline CLIP","https://arxiv.org/html/2609.33441v1#S6.T2","MiRAGe","MIC-Bench id; original baseline MiRAGe","SNIFFER","MIC-Bench id; original baseline SNIFFER","MIC-Bench id; original baseline GPT-5.4-mini","MIC-Bench id; original baseline Qwen2.5-VL-3B-Instruct","MIC-Bench id; original baseline Qwen3-VL-4B-Instruct","MIC-Bench id; original baseline Qwen3-VL-8B-Instruct","MIC-Bench id; original baseline Qwen3-VL-30B-A3B","MIC-Bench id; original baseline LLaVA-OneVision-7B","MIC-Bench id; original baseline InternVL3-8B","MIC-Bench id; original baseline Qwen3-VL-8B-Thinking","MIC-Bench id; SFT+GRPO LoRA rank64 onQwen3VL4B","MIC-Bench ood; original baseline CLIP","MIC-Bench ood; original baseline MiRAGe","MIC-Bench ood; original baseline SNIFFER","MIC-Bench ood; original baseline GPT-5.4-mini","MIC-Bench ood; original baseline Qwen2.5-VL-3B-Instruct","MIC-Bench ood; original baseline Qwen3-VL-4B-Instruct","MIC-Bench ood; original baseline Qwen3-VL-8B-Instruct","MIC-Bench ood; original baseline Qwen3-VL-30B-A3B","MIC-Bench ood; original baseline LLaVA-OneVision-7B","MIC-Bench ood; original baseline InternVL3-8B","MIC-Bench ood; original baseline Qwen3-VL-8B-Thinking","MIC-Bench ood; SFT+GRPO LoRA rank64 onQwen3VL4B","Qwen2-Audio Instruct","Original unmitigated model; same2000questions; provider-native decoding","https://arxiv.org/html/2609.33893v1#S4.T1","Kimi-Audio Instruct","Step-Audio 2 mini","MiMo-Audio Instruct","Qwen3-Omni Instruct","Original unmitigated model; same4000questions; provider-native decoding","Original unmitigated model; same12000questions; provider-native decoding","Full benchmark base model; GPT-4o-mini; separate frozen Qwen3.5-4B judge forT2–T4","https://arxiv.org/html/2609.33261v1#S3.T4","Full benchmark base model; GPT-5.4; separate frozen Qwen3.5-4B judge forT2–T4","Full benchmark base model; GPT-5.6 Luna; separate frozen Qwen3.5-4B judge forT2–T4","Full benchmark base model; Claude Opus 4.7; separate frozen Qwen3.5-4B judge forT2–T4","Grok 4.3 (non-reasoning)","Full benchmark base model; Grok 4.3 (non-reasoning); separate frozen Qwen3.5-4B judge forT2–T4","Full benchmark base model; Qwen3.5-4B; separate frozen Qwen3.5-4B judge forT2–T4","Full benchmark base model; Qwen3.5-9B; separate frozen Qwen3.5-4B judge forT2–T4","Full benchmark base model; Qwen3.5-27B; separate frozen Qwen3.5-4B judge forT2–T4","Qwen3.6-27B (non-thinking)","Full benchmark base model; Qwen3.6-27B (non-thinking); separate frozen Qwen3.5-4B judge forT2–T4","Qwen3.8-27B (non-thinking)","Full benchmark base model; Qwen3.8-27B (non-thinking); separate frozen Qwen3.5-4B judge forT2–T4","Full benchmark base model; Qwen3-VL-8B-Instruct; separate frozen Qwen3.5-4B judge forT2–T4","Full benchmark base model; Qwen3-VL-2B-Thinking; separate frozen Qwen3.5-4B judge forT2–T4","Full benchmark base model; Qwen3-VL-8B-Thinking; separate frozen Qwen3.5-4B judge forT2–T4","Full benchmark base model; InternVL3.5-8B; separate frozen Qwen3.5-4B judge forT2–T4","Full benchmark base model; InternVL3.5-14B; separate frozen Qwen3.5-4B judge forT2–T4","Gemma 4-31B-IT","Full benchmark base model; Gemma 4-31B-IT; separate frozen Qwen3.5-4B judge forT2–T4","Full benchmark base model; MiMo-VL-7B-RL; separate frozen Qwen3.5-4B judge forT2–T4","LLava-Video-32B","Tracked visual boxes;32frames; direct answer; deterministic; full","https://arxiv.org/html/2609.34547v1#S4.T2","Tracked visual boxes;32frames; direct answer; deterministic; review","Gemini-3 Pro","Full seven-tool sandbox;20turns/200Ktokens/30points","https://arxiv.org/html/2609.34428v1#S4.T2","GPT-5.3 codex","Full seven-tool sandbox;20turns/200Ktokens/30points; mode: DeepSeek-reasoner","Full seven-tool sandbox;20turns/200Ktokens/30points; mode: DeepSeek-chat","Instruction-onlyC1; Pro Expense; fiveaw-inheritedtemplates ×3runs","https://arxiv.org/html/2609.34139v1#A5.T3","UI Voyager-4B","Instruction-onlyC1; Pro Expense; fivenewtemplates ×3runs","Instruction-onlyC1; Oinkoin; fiveaw-inheritedtemplates ×3runs","Instruction-onlyC1; Oinkoin; fivenewtemplates ×3runs","Instruction-onlyC1; OpenMoney Box; fiveaw-inheritedtemplates ×3runs","Instruction-onlyC1; OpenMoney Box; fivenewtemplates ×3runs","Instruction-onlyC1; My Expenses; fiveaw-inheritedtemplates ×3runs","Instruction-onlyC1; My Expenses; fivenewtemplates ×3runs","Instruction-onlyC1; Finance Manager; fiveaw-inheritedtemplates ×3runs","Instruction-onlyC1; Finance Manager; fivenewtemplates ×3runs","Instruction-onlyC1; Sushi; fiveaw-inheritedtemplates ×3runs","Instruction-onlyC1; Sushi; fivenewtemplates ×3runs","Guardian [ 11 ]","3-shot; five seeds;1000test pairs; GNN","https://arxiv.org/html/2609.33521v1#S5.T4","GATrust [ 39 ]","TrustGNN [ 40 ]","3-shot; five seeds;1000test pairs; LLM few-shot+knowledge+role","Medley [ 29 ]","https://arxiv.org/html/2609.33521v1#S5.T5","DTrust [ 42 ]","TrustGuard [ 10 ]","Original main table; shared 5000 instructions; stratified calibration; TCPR only","https://arxiv.org/html/2610.19693v1#S4.T2","Qwen3-235B-A22B-Thinking","Qwen-Image-2.1","Without baseline PE; same product cases; official native model/system settings","https://arxiv.org/html/2610.20603v1#S5.T1","FireRed-Image-Edit-1.0","FLUX.2 [klein] 9B","Qwen-Image-2.0-Pro","SKU-DiT-6B","Without baseline PE; same product cases; official native model/system settings; CommerceGen-Studio; PERL-Agent and LTO","Daily; Intervention Level; native streaming framework and start/stop tool lifecycle","https://arxiv.org/html/2610.30089v1#S5.T4","MMDuet","JoyAI-VL","Native Boolean predictive track; published solver-specific supervised/prompted configuration; only","https://arxiv.org/html/2610.21382v1#S5.T6","Native Boolean predictive track; published solver-specific supervised/prompted configuration; GraphCtx","Native Boolean predictive track; published solver-specific supervised/prompted configuration","RelGNN","RelGT","StructOpt-RDL","AnyPPG","Frozen released representation; nested subject-disjoint linear probe; five outer folds and five seeds","https://arxiv.org/html/2610.13435v1#S5.T1","PaPaGei-S","PaPaGei-P","PulsePPG","SigmaPPG","MOMENT-1-small","Common250 contracts ×10 independent generations; raw; source fraction definition","https://arxiv.org/html/2610.18745v1#S5.T2","Qwen3-Coder-30B","Ornith-35B","Devstral-24B","Common250 contracts ×10 independent generations; repaired; source fraction definition","Common recorded states; task-weighted; published default option scoring; Jev","https://arxiv.org/html/2610.21412v1#A2.T8","Intern-Decision-4B","Common recorded states; task-weighted; published default option scoring; Intern-Decision-4B","StartLux-27B","Common recorded states; task-weighted; published default option scoring; StartLux-27B","StartLux-9B","Common recorded states; task-weighted; published default option scoring; StartLux-9B ‡","StartLux-4B","Common recorded states; task-weighted; published default option scoring; StartLux-4B","StartLux-2B","Common recorded states; task-weighted; published default option scoring; StartLux-2B ‡","StartLux-0.8B","Common recorded states; task-weighted; published default option scoring; StartLux-0.8B ‡","Common recorded states; task-weighted; published default option scoring; Qwen3-4B (untrained)","Common recorded states; task-weighted; published default option scoring; SFT v1 ∗","Common recorded states; task-weighted; published default option scoring; AgentDecision-4B v1 ∗","Common recorded states; task-weighted; published default option scoring; StartLux-4B SFT (track 2) ∗‡","Audio-Flamingo-3","Speech implicit; no explicit demographic label; temperature0; original released speech model","https://arxiv.org/html/2610.13550v1#A1.T7","Audio-Flamingo-Next","DeSTA2.5-Audio-Llama-3.1-8B","MOSS-Audio-8B-Instruct","Voxtral-Small-24B-2507","Zero-shot native moderation prompt","https://arxiv.org/html/2610.31713v1#Sx4.T2","Authors’ fine-tuned moderation variant","MultiGuard","MultiGuard internal-representation detector","GuardReasoner-Omni fine-tune","OmniGuard fine-tune","Authors’ additional fine-tuned moderation variant","https://arxiv.org/html/2610.31713v1#A2.T8","Source native inference; semantics QA","https://arxiv.org/html/2609.37576v1#S4.T2","xhigh reasoning; semantics QA","CogAgent-18B","DeepSeek-VL-7B-Chat","DeepSeek-VL2-small","Llama3.2-11B-Vision","Llama3-LLaVA-Next-8B","MiniCPM-V-4-5","mPLUG-Owl3-7B","LoRA-adapted source checkpoint; semantics QA","Source native inference; quality QA","https://arxiv.org/html/2609.37576v1#S4.T3","xhigh reasoning; quality QA","LoRA-adapted source checkpoint; quality QA","Source native inference; authenticity QA","https://arxiv.org/html/2609.37576v1#S4.T4","xhigh reasoning; authenticity QA","LoRA-adapted source checkpoint; authenticity QA","Source native inference; responsibility QA","https://arxiv.org/html/2609.37576v1#S4.T5","xhigh reasoning; responsibility QA","LoRA-adapted source checkpoint; responsibility QA","Role: user proxy; hosted; fixed GPT-5.5 agent; one rollout per task","https://arxiv.org/html/2609.38043v1#A1.T1","Role: user proxy; open; fixed GPT-5.5 agent; one rollout per task","Gemma-31B","Gemma-26B","Single-pass generator; common references and structured visual prompt","https://arxiv.org/html/2609.37709v1#S4.T2","Agent: multi-step image editing","Qwen-Image-2511","Qwen-Image-2509","Image-input native inference; common selected benchmark and rubric judge","https://arxiv.org/html/2609.37287v1#S4.T2","Gemma-4 E4B","GLM-4.1V 9B","Common data-research harness; three runs per task; best-ID checkpoint per run; task-specific fixed training resources","https://arxiv.org/html/2609.40097v1#S4.T1","Claude-4.7","Kimi-3","Agent: OpenCode; one patch/abstention per held-out item","https://arxiv.org/html/2609.39909v1#S6.T1","Agent: Codex; one patch/abstention per held-out item","Agent: Claude Code; one patch/abstention per held-out item","unified processing; common verified video stream; event coverage only","https://arxiv.org/html/2609.36691v1#S2.T1","factorized processing; common verified video stream; event coverage only","Native omni audio-visual inference; no reference evidence chain; over-video MCQ","https://arxiv.org/pdf/2609.39490v1#page=8","Doubao-2.0-Lite","Muse-Spark-1.2","SFT+MFSD adaptation of Qwen3-Omni; frozen audio/visual encoders; no reference evidence chain; over-video MCQ","Native omni audio-visual inference; no reference evidence chain; beyond-video MCQ","SFT+MFSD adaptation of Qwen3-Omni; frozen audio/visual encoders; no reference evidence chain; beyond-video MCQ","Non-thinking mode","https://arxiv.org/html/2608.12781v1#S4.T3","Thinking mode","Seed-1.6-Vision","Mimo-v2-Omni","Mimo-v2.5","easy subset; source mean video IoU","https://arxiv.org/html/2608.30584v1#S5.T1","LLaVA-OneVision-1.5","TubeDETR","CG-STVG","TA-STVG","LLaVA-ST","CurrSTVG","medium subset; source mean video IoU","hard subset; source mean video IoU","seen simulation regime; native success fraction; source scale averaging","https://arxiv.org/html/2608.22301v1#S4.T3","RDT-1B","UniSkill","XSkill","DP/DINOv2","DP/SigLIP2","DP/VideoMAE","VQ-BeT/DINOv2","VQ-BeT/SigLIP2","VQ-BeT/VideoMAE","ACT/DINOv2","ACT/SigLIP2","ACT/VideoMAE","zero-shot simulation regime; native success fraction; source scale averaging","scratch simulation regime; native success fraction; source scale averaging","pretrain-finetune simulation regime; native success fraction; source scale averaging","parametric; source strict macro accuracy; excludes +partial increment","https://arxiv.org/html/2608.08512v1#S4.T3","rag; source strict macro accuracy; excludes +partial increment","icl; source strict macro accuracy; excludes +partial increment","standard prompts; mean of three runs; source model/effort Gem. 2.5 Pro","https://arxiv.org/html/2608.13167v1#S4.T2","standard prompts; mean of three runs; source model/effort Gem. 2.5 Flash","Gemini 2.5 Flash (no thinking)","standard prompts; mean of three runs; source model/effort Gem. 2.5 Fl. NT","standard prompts; mean of three runs; source model/effort Gem. 3.1 Pro","Gemini 3.1 Pro (reasoning low)","standard prompts; mean of three runs; source model/effort Gem. 3.1 Pro RL","standard prompts; mean of three runs; source model/effort Gem. 3.1 Fl. Lt","Qwen3-VL 235B A22B Thinking","standard prompts; mean of three runs; source model/effort Qw3-VL Think","standard prompts; mean of three runs; source model/effort Qw3-VL Inst.","standard prompts; single greedy pass; source model/effort Qw3-VL 8B","standard prompts; mean of three runs; source model/effort GPT-5 Pro","standard prompts; mean of three runs; source model/effort GPT-5 High","GPT-5 (medium)","standard prompts; mean of three runs; source model/effort GPT-5 Med","standard prompts; mean of three runs; source model/effort GPT-5 Mini","GPT-5 (no reasoning)","standard prompts; mean of three runs; source model/effort GPT-5 NR","standard prompts; mean of three runs; source model/effort Gemma 4 E4B","standard prompts; mean of three runs; source model/effort LLaVA-Video-7B","json prompts; mean of three runs; source model/effort Gem. 2.5 Pro","json prompts; mean of three runs; source model/effort Gem. 2.5 Flash","json prompts; mean of three runs; source model/effort Gem. 2.5 Fl. NT","json prompts; mean of three runs; source model/effort Gem. 3.1 Pro","json prompts; mean of three runs; source model/effort Gem. 3.1 Pro RL","json prompts; mean of three runs; source model/effort Gem. 3.1 Fl. Lt","json prompts; mean of three runs; source model/effort Qw3-VL Think","json prompts; mean of three runs; source model/effort Qw3-VL Inst.","json prompts; single greedy pass; source model/effort Qw3-VL 8B","json prompts; mean of three runs; source model/effort GPT-5 Pro","json prompts; mean of three runs; source model/effort GPT-5 High","json prompts; mean of three runs; source model/effort GPT-5 Med","json prompts; mean of three runs; source model/effort GPT-5 Mini","json prompts; mean of three runs; source model/effort GPT-5 NR","json prompts; mean of three runs; source model/effort Gemma 4 E4B","json prompts; mean of three runs; source model/effort LLaVA-Video-7B","guided prompts; mean of three runs; source model/effort Gem. 2.5 Pro","guided prompts; mean of three runs; source model/effort Gem. 2.5 Flash","guided prompts; mean of three runs; source model/effort Gem. 2.5 Fl. NT","guided prompts; mean of three runs; source model/effort Gem. 3.1 Pro","guided prompts; mean of three runs; source model/effort Gem. 3.1 Pro RL","guided prompts; mean of three runs; source model/effort Gem. 3.1 Fl. Lt","guided prompts; mean of three runs; source model/effort Qw3-VL Think","guided prompts; mean of three runs; source model/effort Qw3-VL Inst.","guided prompts; single greedy pass; source model/effort Qw3-VL 8B","guided prompts; mean of three runs; source model/effort GPT-5 Pro","guided prompts; mean of three runs; source model/effort GPT-5 High","guided prompts; mean of three runs; source model/effort GPT-5 Med","guided prompts; mean of three runs; source model/effort GPT-5 Mini","guided prompts; mean of three runs; source model/effort GPT-5 NR","guided prompts; mean of three runs; source model/effort Gemma 4 E4B","guided prompts; mean of three runs; source model/effort LLaVA-Video-7B","CSP","NGSIM-native 3 s history, 0.2 s sampling; common 5 s target; seven-template macro RMSE (m)","https://arxiv.org/html/2608.24094v1#A1.T11","STDAN","BAT","EMP-M","Argoverse2-native 5 s history, 0.1 s sampling; resampled future; common 5 s target; seven-template macro RMSE (m)","EMP-D","DeMo","DeMo+RealMotion","GuardReasoner-VL-3B","SafeAtlas-Bench image target; unsafe-class F1; source operating threshold","https://arxiv.org/html/2608.29098v1#S4.T3","GuardReasoner-VL-7B","ProGuard-3B","ProGuard-7B","Llama Guard 3 Vision-11B","Llama Guard 4-12B","LLaVAGuard-v1.2-7B","LLaVAShield-7B","Nemotron 3.5 CS-4B","SafeGuard-VL-7B","SafeAtlas Guard -2B","SafeAtlas Guard -4B","SafeAtlas Guard -8B","SafeAtlas-Bench request target; unsafe-class F1; source operating threshold","SafeAtlas-Bench response target; unsafe-class F1; source operating threshold","short-easy VideoRover-Bench subset; Direct answer","https://arxiv.org/html/2608.23329v2#S6.T1","short-easy VideoRover-Bench subset; Agentic tool use; base checkpoint","VideoRover -8B-SFT","short-easy VideoRover-Bench subset; Agentic tool use; SFT checkpoint","VideoRover -8B-RL","short-easy VideoRover-Bench subset; Agentic tool use; RL checkpoint","short-medium VideoRover-Bench subset; Direct answer","short-medium VideoRover-Bench subset; Agentic tool use; base checkpoint","short-medium VideoRover-Bench subset; Agentic tool use; SFT checkpoint","short-medium VideoRover-Bench subset; Agentic tool use; RL checkpoint","short-hard VideoRover-Bench subset; Direct answer","short-hard VideoRover-Bench subset; Agentic tool use; base checkpoint","short-hard VideoRover-Bench subset; Agentic tool use; SFT checkpoint","short-hard VideoRover-Bench subset; Agentic tool use; RL checkpoint","long-easy VideoRover-Bench subset; Direct answer","long-easy VideoRover-Bench subset; Agentic tool use; base checkpoint","long-easy VideoRover-Bench subset; Agentic tool use; SFT checkpoint","long-easy VideoRover-Bench subset; Agentic tool use; RL checkpoint","long-medium VideoRover-Bench subset; Direct answer","long-medium VideoRover-Bench subset; Agentic tool use; base checkpoint","long-medium VideoRover-Bench subset; Agentic tool use; SFT checkpoint","long-medium VideoRover-Bench subset; Agentic tool use; RL checkpoint","long-hard VideoRover-Bench subset; Direct answer","long-hard VideoRover-Bench subset; Agentic tool use; base checkpoint","long-hard VideoRover-Bench subset; Agentic tool use; SFT checkpoint","long-hard VideoRover-Bench subset; Agentic tool use; RL checkpoint","Direct answer; params 30B (3B); original score 0.00 ± 0.00","https://arxiv.org/html/2608.28062v2#S7.T1","Direct answer; params 1T (32B); original score 1.99 ± 0.68","WeAgent-Harness; params –; original score 42.00","GPT-5.6-Sol-High","WeAgent-Harness; params –; original score 49.33","Opus-5-High","WeAgent-Harness; params –; original score 40.67","WeAgent-Harness; params 1T (32B); original score 33.78 ± 3.15","WeAgent-Harness; params 397B (17B); original score 30.44 ± 0.77","WeAgent-Harness; params 35B (3B); original score 36.22 ± 0.38","WeAgent-Harness; params 27B; original score 31.45 ± 1.07","OpenSearch-VL-30B","Lightweight search; params 30B (3B); original score 16.89 ± 1.02 *","WeAgent-MMSearch-Base","WeAgent-MMSearch Base; params 30B (3B); original score 8.00 ± 3.53","WeAgent-MMSearch-SFT","WeAgent-MMSearch SFT; params 30B (3B); original score 24.00 ± 1.76","WeAgent-MMSearch-RL","WeAgent-MMSearch RL; params 30B (3B); original score 30.22 ± 0.32","Qwen3 (8B)","rust standard single-turn test generation; source TPR","https://arxiv.org/html/2608.25939v2#S3.T2","Yi-Coder (9B)","Codestral (22B)","Qwen3-Coder-Next (80B)","GPT-OSS (120B)","Llama-3.3 (70B)","DeepSeek V4-pro","Kimi-k2.5 (Instruct)","julia standard single-turn test generation; source TPR","go standard single-turn test generation; source TPR","ruby standard single-turn test generation; source TPR","php standard single-turn test generation; source TPR","Berlin airt detection; ML; source feature configuration RF","https://arxiv.org/html/2608.23857v1#A5.T15","RF+m","Berlin airt detection; ML; source feature configuration RF+m","RF+s","Berlin airt detection; ML; source feature configuration RF+s","RF+s+m","Berlin airt detection; ML; source feature configuration RF+s+m","Berlin airt detection; ML; source feature configuration XGBoost","XGBoost+m","Berlin airt detection; ML; source feature configuration XGBoost+m","XGBoost+s","Berlin airt detection; ML; source feature configuration XGBoost+s","XGBoost+s+m","Berlin airt detection; ML; source feature configuration XGBoost+s+m","IsolationForest","Berlin airt detection; Unsupervised ML; source feature configuration IsolationForest","IsolationForest+m","Berlin airt detection; Unsupervised ML; source feature configuration IsolationForest+m","IsolationForest+s","Berlin airt detection; Unsupervised ML; source feature configuration IsolationForest+s","IsolationForest+s+m","Berlin airt detection; Unsupervised ML; source feature configuration IsolationForest+s+m","OneClassSVM","Berlin airt detection; Unsupervised ML; source feature configuration OneClassSVM","OneClassSVM+m","Berlin airt detection; Unsupervised ML; source feature configuration OneClassSVM+m","OneClassSVM+s","Berlin airt detection; Unsupervised ML; source feature configuration OneClassSVM+s","OneClassSVM+s+m","Berlin airt detection; Unsupervised ML; source feature configuration OneClassSVM+s+m","Berlin airt detection; DL; source feature configuration LSTM","LSTM+m","Berlin airt detection; DL; source feature configuration LSTM+m","LSTM+s","Berlin airt detection; DL; source feature configuration LSTM+s","LSTM+s+m","Berlin airt detection; DL; source feature configuration LSTM+s+m","Berlin airt detection; FM; source feature configuration TimesFM","Berlin airt detection; FM; source feature configuration Chronos-2","Chronos-2 (+m)","Berlin airt detection; FM; source feature configuration Chronos-2 (+m)","Chronos-2 (+s)","Berlin airt detection; FM; source feature configuration Chronos-2 (+s)","Chronos-2 (+s+m)","Berlin airt detection; FM; source feature configuration Chronos-2 (+s+m)","MOIRAI-2","Berlin airt detection; FM; source feature configuration MOIRAI-2","MOIRAI-2 (+m)","Berlin airt detection; FM; source feature configuration MOIRAI-2 (+m)","MOIRAI-2 (+s)","Berlin airt detection; FM; source feature configuration MOIRAI-2 (+s)","MOIRAI-2 (+s+m)","Berlin airt detection; FM; source feature configuration MOIRAI-2 (+s+m)","Berlin lst detection; ML; source feature configuration RF","Berlin lst detection; ML; source feature configuration RF+m","Berlin lst detection; ML; source feature configuration RF+s","Berlin lst detection; ML; source feature configuration RF+s+m","Berlin lst detection; ML; source feature configuration XGBoost","Berlin lst detection; ML; source feature configuration XGBoost+m","Berlin lst detection; ML; source feature configuration XGBoost+s","Berlin lst detection; ML; source feature configuration XGBoost+s+m","Berlin lst detection; Unsupervised ML; source feature configuration IsolationForest","Berlin lst detection; Unsupervised ML; source feature configuration IsolationForest+m","Berlin lst detection; Unsupervised ML; source feature configuration IsolationForest+s","Berlin lst detection; Unsupervised ML; source feature configuration IsolationForest+s+m","Berlin lst detection; Unsupervised ML; source feature configuration OneClassSVM","Berlin lst detection; Unsupervised ML; source feature configuration OneClassSVM+m","Berlin lst detection; Unsupervised ML; source feature configuration OneClassSVM+s","Berlin lst detection; Unsupervised ML; source feature configuration OneClassSVM+s+m","Berlin lst detection; DL; source feature configuration LSTM","Berlin lst detection; DL; source feature configuration LSTM+m","Berlin lst detection; DL; source feature configuration LSTM+s","Berlin lst detection; DL; source feature configuration LSTM+s+m","Berlin lst detection; FM; source feature configuration TimesFM","Berlin lst detection; FM; source feature configuration Chronos-2","Berlin lst detection; FM; source feature configuration Chronos-2 (+m)","Berlin lst detection; FM; source feature configuration Chronos-2 (+s)","Berlin lst detection; FM; source feature configuration Chronos-2 (+s+m)","Berlin lst detection; FM; source feature configuration MOIRAI-2","Berlin lst detection; FM; source feature configuration MOIRAI-2 (+m)","Berlin lst detection; FM; source feature configuration MOIRAI-2 (+s)","Berlin lst detection; FM; source feature configuration MOIRAI-2 (+s+m)","Munich airt detection; ML; source feature configuration RF","Munich airt detection; ML; source feature configuration RF+m","Munich airt detection; ML; source feature configuration RF+s","Munich airt detection; ML; source feature configuration RF+s+m","Munich airt detection; ML; source feature configuration XGBoost","Munich airt detection; ML; source feature configuration XGBoost+m","Munich airt detection; ML; source feature configuration XGBoost+s","Munich airt detection; ML; source feature configuration XGBoost+s+m","Munich airt detection; Unsupervised ML; source feature configuration IsolationForest","Munich airt detection; Unsupervised ML; source feature configuration IsolationForest+m","Munich airt detection; Unsupervised ML; source feature configuration IsolationForest+s","Munich airt detection; Unsupervised ML; source feature configuration IsolationForest+s+m","Munich airt detection; Unsupervised ML; source feature configuration OneClassSVM","Munich airt detection; Unsupervised ML; source feature configuration OneClassSVM+m","Munich airt detection; Unsupervised ML; source feature configuration OneClassSVM+s","Munich airt detection; Unsupervised ML; source feature configuration OneClassSVM+s+m","Munich airt detection; DL; source feature configuration LSTM","Munich airt detection; DL; source feature configuration LSTM+m","Munich airt detection; DL; source feature configuration LSTM+s","Munich airt detection; DL; source feature configuration LSTM+s+m","Munich airt detection; FM; source feature configuration TimesFM","Munich airt detection; FM; source feature configuration Chronos-2","Munich airt detection; FM; source feature configuration Chronos-2 (+m)","Munich airt detection; FM; source feature configuration Chronos-2 (+s)","Munich airt detection; FM; source feature configuration Chronos-2 (+s+m)","Munich airt detection; FM; source feature configuration MOIRAI-2","Munich airt detection; FM; source feature configuration MOIRAI-2 (+m)","Munich airt detection; FM; source feature configuration MOIRAI-2 (+s)","Munich airt detection; FM; source feature configuration MOIRAI-2 (+s+m)","Munich lst detection; ML; source feature configuration RF","Munich lst detection; ML; source feature configuration RF+m","Munich lst detection; ML; source feature configuration RF+s","Munich lst detection; ML; source feature configuration RF+s+m","Munich lst detection; ML; source feature configuration XGBoost","Munich lst detection; ML; source feature configuration XGBoost+m","Munich lst detection; ML; source feature configuration XGBoost+s","Munich lst detection; ML; source feature configuration XGBoost+s+m","Munich lst detection; Unsupervised ML; source feature configuration IsolationForest","Munich lst detection; Unsupervised ML; source feature configuration IsolationForest+m","Munich lst detection; Unsupervised ML; source feature configuration IsolationForest+s","Munich lst detection; Unsupervised ML; source feature configuration IsolationForest+s+m","Munich lst detection; Unsupervised ML; source feature configuration OneClassSVM","Munich lst detection; Unsupervised ML; source feature configuration OneClassSVM+m","Munich lst detection; Unsupervised ML; source feature configuration OneClassSVM+s","Munich lst detection; Unsupervised ML; source feature configuration OneClassSVM+s+m","Munich lst detection; DL; source feature configuration LSTM","Munich lst detection; DL; source feature configuration LSTM+m","Munich lst detection; DL; source feature configuration LSTM+s","Munich lst detection; DL; source feature configuration LSTM+s+m","Munich lst detection; FM; source feature configuration TimesFM","Munich lst detection; FM; source feature configuration Chronos-2","Munich lst detection; FM; source feature configuration Chronos-2 (+m)","Munich lst detection; FM; source feature configuration Chronos-2 (+s)","Munich lst detection; FM; source feature configuration Chronos-2 (+s+m)","Munich lst detection; FM; source feature configuration MOIRAI-2","Munich lst detection; FM; source feature configuration MOIRAI-2 (+m)","Munich lst detection; FM; source feature configuration MOIRAI-2 (+s)","Munich lst detection; FM; source feature configuration MOIRAI-2 (+s+m)","Hamburg airt detection; ML; source feature configuration RF","Hamburg airt detection; ML; source feature configuration RF+m","Hamburg airt detection; ML; source feature configuration RF+s","Hamburg airt detection; ML; source feature configuration RF+s+m","Hamburg airt detection; ML; source feature configuration XGBoost","Hamburg airt detection; ML; source feature configuration XGBoost+m","Hamburg airt detection; ML; source feature configuration XGBoost+s","Hamburg airt detection; ML; source feature configuration XGBoost+s+m","Hamburg airt detection; Unsupervised ML; source feature configuration IsolationForest","Hamburg airt detection; Unsupervised ML; source feature configuration IsolationForest+m","Hamburg airt detection; Unsupervised ML; source feature configuration IsolationForest+s","Hamburg airt detection; Unsupervised ML; source feature configuration IsolationForest+s+m","Hamburg airt detection; Unsupervised ML; source feature configuration OneClassSVM","Hamburg airt detection; Unsupervised ML; source feature configuration OneClassSVM+m","Hamburg airt detection; Unsupervised ML; source feature configuration OneClassSVM+s","Hamburg airt detection; Unsupervised ML; source feature configuration OneClassSVM+s+m","Hamburg airt detection; DL; source feature configuration LSTM","Hamburg airt detection; DL; source feature configuration LSTM+m","Hamburg airt detection; DL; source feature configuration LSTM+s","Hamburg airt detection; DL; source feature configuration LSTM+s+m","Hamburg airt detection; FM; source feature configuration TimesFM","Hamburg airt detection; FM; source feature configuration Chronos-2","Hamburg airt detection; FM; source feature configuration Chronos-2 (+m)","Hamburg airt detection; FM; source feature configuration Chronos-2 (+s)","Hamburg airt detection; FM; source feature configuration Chronos-2 (+s+m)","Hamburg airt detection; FM; source feature configuration MOIRAI-2","Hamburg airt detection; FM; source feature configuration MOIRAI-2 (+m)","Hamburg airt detection; FM; source feature configuration MOIRAI-2 (+s)","Hamburg airt detection; FM; source feature configuration MOIRAI-2 (+s+m)","Hamburg lst detection; ML; source feature configuration RF","Hamburg lst detection; ML; source feature configuration RF+m","Hamburg lst detection; ML; source feature configuration RF+s","Hamburg lst detection; ML; source feature configuration RF+s+m","Hamburg lst detection; ML; source feature configuration XGBoost","Hamburg lst detection; ML; source feature configuration XGBoost+m","Hamburg lst detection; ML; source feature configuration XGBoost+s","Hamburg lst detection; ML; source feature configuration XGBoost+s+m","Hamburg lst detection; Unsupervised ML; source feature configuration IsolationForest","Hamburg lst detection; Unsupervised ML; source feature configuration IsolationForest+m","Hamburg lst detection; Unsupervised ML; source feature configuration IsolationForest+s","Hamburg lst detection; Unsupervised ML; source feature configuration IsolationForest+s+m","Hamburg lst detection; Unsupervised ML; source feature configuration OneClassSVM","Hamburg lst detection; Unsupervised ML; source feature configuration OneClassSVM+m","Hamburg lst detection; Unsupervised ML; source feature configuration OneClassSVM+s","Hamburg lst detection; Unsupervised ML; source feature configuration OneClassSVM+s+m","Hamburg lst detection; DL; source feature configuration LSTM","Hamburg lst detection; DL; source feature configuration LSTM+m","Hamburg lst detection; DL; source feature configuration LSTM+s","Hamburg lst detection; DL; source feature configuration LSTM+s+m","Hamburg lst detection; FM; source feature configuration TimesFM","Hamburg lst detection; FM; source feature configuration Chronos-2","Hamburg lst detection; FM; source feature configuration Chronos-2 (+m)","Hamburg lst detection; FM; source feature configuration Chronos-2 (+s)","Hamburg lst detection; FM; source feature configuration Chronos-2 (+s+m)","Hamburg lst detection; FM; source feature configuration MOIRAI-2","Hamburg lst detection; FM; source feature configuration MOIRAI-2 (+m)","Hamburg lst detection; FM; source feature configuration MOIRAI-2 (+s)","Hamburg lst detection; FM; source feature configuration MOIRAI-2 (+s+m)","Cairo airt detection; ML; source feature configuration RF","Cairo airt detection; ML; source feature configuration RF+m","Cairo airt detection; ML; source feature configuration RF+s","Cairo airt detection; ML; source feature configuration RF+s+m","Cairo airt detection; ML; source feature configuration XGBoost","Cairo airt detection; ML; source feature configuration XGBoost+m","Cairo airt detection; ML; source feature configuration XGBoost+s","Cairo airt detection; ML; source feature configuration XGBoost+s+m","Cairo airt detection; Unsupervised ML; source feature configuration IsolationForest","Cairo airt detection; Unsupervised ML; source feature configuration IsolationForest+m","Cairo airt detection; Unsupervised ML; source feature configuration IsolationForest+s","Cairo airt detection; Unsupervised ML; source feature configuration IsolationForest+s+m","Cairo airt detection; Unsupervised ML; source feature configuration OneClassSVM","Cairo airt detection; Unsupervised ML; source feature configuration OneClassSVM+m","Cairo airt detection; Unsupervised ML; source feature configuration OneClassSVM+s","Cairo airt detection; Unsupervised ML; source feature configuration OneClassSVM+s+m","Cairo airt detection; DL; source feature configuration LSTM","Cairo airt detection; DL; source feature configuration LSTM+m","Cairo airt detection; DL; source feature configuration LSTM+s","Cairo airt detection; DL; source feature configuration LSTM+s+m","Cairo airt detection; FM; source feature configuration TimesFM","Cairo airt detection; FM; source feature configuration Chronos-2","Cairo airt detection; FM; source feature configuration Chronos-2 (+m)","Cairo airt detection; FM; source feature configuration Chronos-2 (+s)","Cairo airt detection; FM; source feature configuration Chronos-2 (+s+m)","Cairo airt detection; FM; source feature configuration MOIRAI-2","Cairo airt detection; FM; source feature configuration MOIRAI-2 (+m)","Cairo airt detection; FM; source feature configuration MOIRAI-2 (+s)","Cairo airt detection; FM; source feature configuration MOIRAI-2 (+s+m)","Cairo lst detection; ML; source feature configuration RF","Cairo lst detection; ML; source feature configuration RF+m","Cairo lst detection; ML; source feature configuration RF+s","Cairo lst detection; ML; source feature configuration RF+s+m","Cairo lst detection; ML; source feature configuration XGBoost","Cairo lst detection; ML; source feature configuration XGBoost+m","Cairo lst detection; ML; source feature configuration XGBoost+s","Cairo lst detection; ML; source feature configuration XGBoost+s+m","Cairo lst detection; Unsupervised ML; source feature configuration IsolationForest","Cairo lst detection; Unsupervised ML; source feature configuration IsolationForest+m","Cairo lst detection; Unsupervised ML; source feature configuration IsolationForest+s","Cairo lst detection; Unsupervised ML; source feature configuration IsolationForest+s+m","Cairo lst detection; Unsupervised ML; source feature configuration OneClassSVM","Cairo lst detection; Unsupervised ML; source feature configuration OneClassSVM+m","Cairo lst detection; Unsupervised ML; source feature configuration OneClassSVM+s","Cairo lst detection; Unsupervised ML; source feature configuration OneClassSVM+s+m","Cairo lst detection; DL; source feature configuration LSTM","Cairo lst detection; DL; source feature configuration LSTM+m","Cairo lst detection; DL; source feature configuration LSTM+s","Cairo lst detection; DL; source feature configuration LSTM+s+m","Cairo lst detection; FM; source feature configuration TimesFM","Cairo lst detection; FM; source feature configuration Chronos-2","Cairo lst detection; FM; source feature configuration Chronos-2 (+m)","Cairo lst detection; FM; source feature configuration Chronos-2 (+s)","Cairo lst detection; FM; source feature configuration Chronos-2 (+s+m)","Cairo lst detection; FM; source feature configuration MOIRAI-2","Cairo lst detection; FM; source feature configuration MOIRAI-2 (+m)","Cairo lst detection; FM; source feature configuration MOIRAI-2 (+s)","Cairo lst detection; FM; source feature configuration MOIRAI-2 (+s+m)","Lagos airt detection; ML; source feature configuration RF","Lagos airt detection; ML; source feature configuration RF+m","Lagos airt detection; ML; source feature configuration RF+s","Lagos airt detection; ML; source feature configuration RF+s+m","Lagos airt detection; ML; source feature configuration XGBoost","Lagos airt detection; ML; source feature configuration XGBoost+m","Lagos airt detection; ML; source feature configuration XGBoost+s","Lagos airt detection; ML; source feature configuration XGBoost+s+m","Lagos airt detection; Unsupervised ML; source feature configuration IsolationForest","Lagos airt detection; Unsupervised ML; source feature configuration IsolationForest+m","Lagos airt detection; Unsupervised ML; source feature configuration IsolationForest+s","Lagos airt detection; Unsupervised ML; source feature configuration IsolationForest+s+m","Lagos airt detection; Unsupervised ML; source feature configuration OneClassSVM","Lagos airt detection; Unsupervised ML; source feature configuration OneClassSVM+m","Lagos airt detection; Unsupervised ML; source feature configuration OneClassSVM+s","Lagos airt detection; Unsupervised ML; source feature configuration OneClassSVM+s+m","Lagos airt detection; DL; source feature configuration LSTM","Lagos airt detection; DL; source feature configuration LSTM+m","Lagos airt detection; DL; source feature configuration LSTM+s","Lagos airt detection; DL; source feature configuration LSTM+s+m","Lagos airt detection; FM; source feature configuration TimesFM","Lagos airt detection; FM; source feature configuration Chronos-2","Lagos airt detection; FM; source feature configuration Chronos-2 (+m)","Lagos airt detection; FM; source feature configuration Chronos-2 (+s)","Lagos airt detection; FM; source feature configuration Chronos-2 (+s+m)","Lagos airt detection; FM; source feature configuration MOIRAI-2","Lagos airt detection; FM; source feature configuration MOIRAI-2 (+m)","Lagos airt detection; FM; source feature configuration MOIRAI-2 (+s)","Lagos airt detection; FM; source feature configuration MOIRAI-2 (+s+m)","Lagos lst detection; ML; source feature configuration RF","Lagos lst detection; ML; source feature configuration RF+m","Lagos lst detection; ML; source feature configuration RF+s","Lagos lst detection; ML; source feature configuration RF+s+m","Lagos lst detection; ML; source feature configuration XGBoost","Lagos lst detection; ML; source feature configuration XGBoost+m","Lagos lst detection; ML; source feature configuration XGBoost+s","Lagos lst detection; ML; source feature configuration XGBoost+s+m","Lagos lst detection; Unsupervised ML; source feature configuration IsolationForest","Lagos lst detection; Unsupervised ML; source feature configuration IsolationForest+m","Lagos lst detection; Unsupervised ML; source feature configuration IsolationForest+s","Lagos lst detection; Unsupervised ML; source feature configuration IsolationForest+s+m","Lagos lst detection; Unsupervised ML; source feature configuration OneClassSVM","Lagos lst detection; Unsupervised ML; source feature configuration OneClassSVM+m","Lagos lst detection; Unsupervised ML; source feature configuration OneClassSVM+s","Lagos lst detection; Unsupervised ML; source feature configuration OneClassSVM+s+m","Lagos lst detection; DL; source feature configuration LSTM","Lagos lst detection; DL; source feature configuration LSTM+m","Lagos lst detection; DL; source feature configuration LSTM+s","Lagos lst detection; DL; source feature configuration LSTM+s+m","Lagos lst detection; FM; source feature configuration TimesFM","Lagos lst detection; FM; source feature configuration Chronos-2","Lagos lst detection; FM; source feature configuration Chronos-2 (+m)","Lagos lst detection; FM; source feature configuration Chronos-2 (+s)","Lagos lst detection; FM; source feature configuration Chronos-2 (+s+m)","Lagos lst detection; FM; source feature configuration MOIRAI-2","Lagos lst detection; FM; source feature configuration MOIRAI-2 (+m)","Lagos lst detection; FM; source feature configuration MOIRAI-2 (+s)","Lagos lst detection; FM; source feature configuration MOIRAI-2 (+s+m)","Johannesburg airt detection; ML; source feature configuration RF","Johannesburg airt detection; ML; source feature configuration RF+m","Johannesburg airt detection; ML; source feature configuration RF+s","Johannesburg airt detection; ML; source feature configuration RF+s+m","Johannesburg airt detection; ML; source feature configuration XGBoost","Johannesburg airt detection; ML; source feature configuration XGBoost+m","Johannesburg airt detection; ML; source feature configuration XGBoost+s","Johannesburg airt detection; ML; source feature configuration XGBoost+s+m","Johannesburg airt detection; Unsupervised ML; source feature configuration IsolationForest","Johannesburg airt detection; Unsupervised ML; source feature configuration IsolationForest+m","Johannesburg airt detection; Unsupervised ML; source feature configuration IsolationForest+s","Johannesburg airt detection; Unsupervised ML; source feature configuration IsolationForest+s+m","Johannesburg airt detection; Unsupervised ML; source feature configuration OneClassSVM","Johannesburg airt detection; Unsupervised ML; source feature configuration OneClassSVM+m","Johannesburg airt detection; Unsupervised ML; source feature configuration OneClassSVM+s","Johannesburg airt detection; Unsupervised ML; source feature configuration OneClassSVM+s+m","Johannesburg airt detection; DL; source feature configuration LSTM","Johannesburg airt detection; DL; source feature configuration LSTM+m","Johannesburg airt detection; DL; source feature configuration LSTM+s","Johannesburg airt detection; DL; source feature configuration LSTM+s+m","Johannesburg airt detection; FM; source feature configuration TimesFM","Johannesburg airt detection; FM; source feature configuration Chronos-2","Johannesburg airt detection; FM; source feature configuration Chronos-2 (+m)","Johannesburg airt detection; FM; source feature configuration Chronos-2 (+s)","Johannesburg airt detection; FM; source feature configuration Chronos-2 (+s+m)","Johannesburg airt detection; FM; source feature configuration MOIRAI-2","Johannesburg airt detection; FM; source feature configuration MOIRAI-2 (+m)","Johannesburg airt detection; FM; source feature configuration MOIRAI-2 (+s)","Johannesburg airt detection; FM; source feature configuration MOIRAI-2 (+s+m)","Johannesburg lst detection; ML; source feature configuration RF","Johannesburg lst detection; ML; source feature configuration RF+m","Johannesburg lst detection; ML; source feature configuration RF+s","Johannesburg lst detection; ML; source feature configuration RF+s+m","Johannesburg lst detection; ML; source feature configuration XGBoost","Johannesburg lst detection; ML; source feature configuration XGBoost+m","Johannesburg lst detection; ML; source feature configuration XGBoost+s","Johannesburg lst detection; ML; source feature configuration XGBoost+s+m","Johannesburg lst detection; Unsupervised ML; source feature configuration IsolationForest","Johannesburg lst detection; Unsupervised ML; source feature configuration IsolationForest+m","Johannesburg lst detection; Unsupervised ML; source feature configuration IsolationForest+s","Johannesburg lst detection; Unsupervised ML; source feature configuration IsolationForest+s+m","Johannesburg lst detection; Unsupervised ML; source feature configuration OneClassSVM","Johannesburg lst detection; Unsupervised ML; source feature configuration OneClassSVM+m","Johannesburg lst detection; Unsupervised ML; source feature configuration OneClassSVM+s","Johannesburg lst detection; Unsupervised ML; source feature configuration OneClassSVM+s+m","Johannesburg lst detection; DL; source feature configuration LSTM","Johannesburg lst detection; DL; source feature configuration LSTM+m","Johannesburg lst detection; DL; source feature configuration LSTM+s","Johannesburg lst detection; DL; source feature configuration LSTM+s+m","Johannesburg lst detection; FM; source feature configuration TimesFM","Johannesburg lst detection; FM; source feature configuration Chronos-2","Johannesburg lst detection; FM; source feature configuration Chronos-2 (+m)","Johannesburg lst detection; FM; source feature configuration Chronos-2 (+s)","Johannesburg lst detection; FM; source feature configuration Chronos-2 (+s+m)","Johannesburg lst detection; FM; source feature configuration MOIRAI-2","Johannesburg lst detection; FM; source feature configuration MOIRAI-2 (+m)","Johannesburg lst detection; FM; source feature configuration MOIRAI-2 (+s)","Johannesburg lst detection; FM; source feature configuration MOIRAI-2 (+s+m)","Riyadh airt detection; ML; source feature configuration RF","Riyadh airt detection; ML; source feature configuration RF+m","Riyadh airt detection; ML; source feature configuration RF+s","Riyadh airt detection; ML; source feature configuration RF+s+m","Riyadh airt detection; ML; source feature configuration XGBoost","Riyadh airt detection; ML; source feature configuration XGBoost+m","Riyadh airt detection; ML; source feature configuration XGBoost+s","Riyadh airt detection; ML; source feature configuration XGBoost+s+m","Riyadh airt detection; Unsupervised ML; source feature configuration IsolationForest","Riyadh airt detection; Unsupervised ML; source feature configuration IsolationForest+m","Riyadh airt detection; Unsupervised ML; source feature configuration IsolationForest+s","Riyadh airt detection; Unsupervised ML; source feature configuration IsolationForest+s+m","Riyadh airt detection; Unsupervised ML; source feature configuration OneClassSVM","Riyadh airt detection; Unsupervised ML; source feature configuration OneClassSVM+m","Riyadh airt detection; Unsupervised ML; source feature configuration OneClassSVM+s","Riyadh airt detection; Unsupervised ML; source feature configuration OneClassSVM+s+m","Riyadh airt detection; DL; source feature configuration LSTM","Riyadh airt detection; DL; source feature configuration LSTM+m","Riyadh airt detection; DL; source feature configuration LSTM+s","Riyadh airt detection; DL; source feature configuration LSTM+s+m","Riyadh airt detection; FM; source feature configuration TimesFM","Riyadh airt detection; FM; source feature configuration Chronos-2","Riyadh airt detection; FM; source feature configuration Chronos-2 (+m)","Riyadh airt detection; FM; source feature configuration Chronos-2 (+s)","Riyadh airt detection; FM; source feature configuration Chronos-2 (+s+m)","Riyadh airt detection; FM; source feature configuration MOIRAI-2","Riyadh airt detection; FM; source feature configuration MOIRAI-2 (+m)","Riyadh airt detection; FM; source feature configuration MOIRAI-2 (+s)","Riyadh airt detection; FM; source feature configuration MOIRAI-2 (+s+m)","Riyadh lst detection; ML; source feature configuration RF","Riyadh lst detection; ML; source feature configuration RF+m","Riyadh lst detection; ML; source feature configuration RF+s","Riyadh lst detection; ML; source feature configuration RF+s+m","Riyadh lst detection; ML; source feature configuration XGBoost","Riyadh lst detection; ML; source feature configuration XGBoost+m","Riyadh lst detection; ML; source feature configuration XGBoost+s","Riyadh lst detection; ML; source feature configuration XGBoost+s+m","Riyadh lst detection; Unsupervised ML; source feature configuration IsolationForest","Riyadh lst detection; Unsupervised ML; source feature configuration IsolationForest+m","Riyadh lst detection; Unsupervised ML; source feature configuration IsolationForest+s","Riyadh lst detection; Unsupervised ML; source feature configuration IsolationForest+s+m","Riyadh lst detection; Unsupervised ML; source feature configuration OneClassSVM","Riyadh lst detection; Unsupervised ML; source feature configuration OneClassSVM+m","Riyadh lst detection; Unsupervised ML; source feature configuration OneClassSVM+s","Riyadh lst detection; Unsupervised ML; source feature configuration OneClassSVM+s+m","Riyadh lst detection; DL; source feature configuration LSTM","Riyadh lst detection; DL; source feature configuration LSTM+m","Riyadh lst detection; DL; source feature configuration LSTM+s","Riyadh lst detection; DL; source feature configuration LSTM+s+m","Riyadh lst detection; FM; source feature configuration TimesFM","Riyadh lst detection; FM; source feature configuration Chronos-2","Riyadh lst detection; FM; source feature configuration Chronos-2 (+m)","Riyadh lst detection; FM; source feature configuration Chronos-2 (+s)","Riyadh lst detection; FM; source feature configuration Chronos-2 (+s+m)","Riyadh lst detection; FM; source feature configuration MOIRAI-2","Riyadh lst detection; FM; source feature configuration MOIRAI-2 (+m)","Riyadh lst detection; FM; source feature configuration MOIRAI-2 (+s)","Riyadh lst detection; FM; source feature configuration MOIRAI-2 (+s+m)","Native video-output regime; Original released generator","https://arxiv.org/html/2608.26105v2#S4.T4","Kling VIDEO 3.0","CogVideoX1.5-5B-I2V","Wan2.1-I2V-14B-720P","LTX-2.3-I2AV","VBVR-Wan2.2","Native video-output regime; VBVR trained checkpoint","VBVR-Pro-LTX2.3","Native video-output regime; VBVR-Pro trained checkpoint","VBVR-Pro-Wan2.1-I2V-14B","VBVR-Pro-Wan2.2-TI2V-5B","VBVR-Pro-Wan2.2-I2V-A14B","Symbolic input; four-candidate; easy difficulty; first-move hit rate","https://arxiv.org/html/2608.13221v1#A4.T13","Symbolic input; four-candidate; medium difficulty; first-move hit rate","Symbolic input; four-candidate; hard difficulty; first-move hit rate","Symbolic input; open; easy difficulty; first-move hit rate","Symbolic input; open; medium difficulty; first-move hit rate","Symbolic input; open; hard difficulty; first-move hit rate","binary prompt; nine-generator source-equal mean; top value only","https://arxiv.org/pdf/2608.14391#page=47","Qwen3.7-Plus (thinking)","diagnostic prompt; nine-generator source-equal mean; top value only","rating prompt; nine-generator source-equal mean; top value only","Qwen2.5-3B-Instruct","story-plain; same 42 seeds; original 0–100 Generation coverage","https://arxiv.org/pdf/2608.15654#page=5","story-memory; same 42 seeds; original 0–100 Generation coverage","story-planning; same 42 seeds; original 0–100 Generation coverage","story-memory-planning; same 42 seeds; original 0–100 Generation coverage","centralized-plain; same 42 seeds; original 0–100 Generation coverage","centralized-memory; same 42 seeds; original 0–100 Generation coverage","centralized-planning; same 42 seeds; original 0–100 Generation coverage","centralized-memory-planning; same 42 seeds; original 0–100 Generation coverage","distributed-plain; same 42 seeds; original 0–100 Generation coverage","distributed-memory; same 42 seeds; original 0–100 Generation coverage","distributed-planning; same 42 seeds; original 0–100 Generation coverage","distributed-memory-planning; same 42 seeds; original 0–100 Generation coverage","story-plain; same 42 seeds; original 0–100 Observed consistency C_obs","story-memory; same 42 seeds; original 0–100 Observed consistency C_obs","story-planning; same 42 seeds; original 0–100 Observed consistency C_obs","story-memory-planning; same 42 seeds; original 0–100 Observed consistency C_obs","centralized-plain; same 42 seeds; original 0–100 Observed consistency C_obs","centralized-memory; same 42 seeds; original 0–100 Observed consistency C_obs","centralized-planning; same 42 seeds; original 0–100 Observed consistency C_obs","centralized-memory-planning; same 42 seeds; original 0–100 Observed consistency C_obs","distributed-plain; same 42 seeds; original 0–100 Observed consistency C_obs","distributed-memory; same 42 seeds; original 0–100 Observed consistency C_obs","distributed-planning; same 42 seeds; original 0–100 Observed consistency C_obs","distributed-memory-planning; same 42 seeds; original 0–100 Observed consistency C_obs","story-plain; same 42 seeds; original 0–100 Narrative richness","story-memory; same 42 seeds; original 0–100 Narrative richness","story-planning; same 42 seeds; original 0–100 Narrative richness","story-memory-planning; same 42 seeds; original 0–100 Narrative richness","centralized-plain; same 42 seeds; original 0–100 Narrative richness","centralized-memory; same 42 seeds; original 0–100 Narrative richness","centralized-planning; same 42 seeds; original 0–100 Narrative richness","centralized-memory-planning; same 42 seeds; original 0–100 Narrative richness","distributed-plain; same 42 seeds; original 0–100 Narrative richness","distributed-memory; same 42 seeds; original 0–100 Narrative richness","distributed-planning; same 42 seeds; original 0–100 Narrative richness","distributed-memory-planning; same 42 seeds; original 0–100 Narrative richness","Official EigenformAI self-reported baseline; fifty fixed questions; judge OpenAI GPT-5.5; coe rubric; arithmetic mean of score_b","https://huggingface.co/datasets/EigenformAI/groundtruth-dynamic-benchmarking-submissions/resolve/main/submissions/model-benchmark/deepseek-coe/scores.json","Gemini 3.1 Pro (preview)","https://huggingface.co/datasets/EigenformAI/groundtruth-dynamic-benchmarking-submissions/resolve/main/submissions/model-benchmark/gemini-coe/scores.json","https://huggingface.co/datasets/EigenformAI/groundtruth-dynamic-benchmarking-submissions/resolve/main/submissions/model-benchmark/glm-coe/scores.json","https://huggingface.co/datasets/EigenformAI/groundtruth-dynamic-benchmarking-submissions/resolve/main/submissions/model-benchmark/gpt-coe/scores.json","https://huggingface.co/datasets/EigenformAI/groundtruth-dynamic-benchmarking-submissions/resolve/main/submissions/model-benchmark/grok-coe/scores.json","https://huggingface.co/datasets/EigenformAI/groundtruth-dynamic-benchmarking-submissions/resolve/main/submissions/model-benchmark/kimi-coe/scores.json","https://huggingface.co/datasets/EigenformAI/groundtruth-dynamic-benchmarking-submissions/resolve/main/submissions/model-benchmark/sonnet5-coe/scores.json","OxAlpha","https://huggingface.co/datasets/EigenformAI/groundtruth-dynamic-benchmarking-submissions/resolve/main/submissions/model-benchmark/oxalpha-coe/scores.json","Official EigenformAI self-reported baseline; fifty fixed questions; judge OpenAI GPT-5.5; supergene rubric; arithmetic mean of score_b","https://huggingface.co/datasets/EigenformAI/groundtruth-dynamic-benchmarking-submissions/resolve/main/submissions/model-benchmark/deepseek-supergene/scores.json","https://huggingface.co/datasets/EigenformAI/groundtruth-dynamic-benchmarking-submissions/resolve/main/submissions/model-benchmark/gemini-supergene/scores.json","https://huggingface.co/datasets/EigenformAI/groundtruth-dynamic-benchmarking-submissions/resolve/main/submissions/model-benchmark/glm-supergene/scores.json","https://huggingface.co/datasets/EigenformAI/groundtruth-dynamic-benchmarking-submissions/resolve/main/submissions/model-benchmark/gpt-supergene/scores.json","https://huggingface.co/datasets/EigenformAI/groundtruth-dynamic-benchmarking-submissions/resolve/main/submissions/model-benchmark/grok-supergene/scores.json","https://huggingface.co/datasets/EigenformAI/groundtruth-dynamic-benchmarking-submissions/resolve/main/submissions/model-benchmark/kimi-supergene/scores.json","https://huggingface.co/datasets/EigenformAI/groundtruth-dynamic-benchmarking-submissions/resolve/main/submissions/model-benchmark/sonnet5-supergene/scores.json","https://huggingface.co/datasets/EigenformAI/groundtruth-dynamic-benchmarking-submissions/resolve/main/submissions/model-benchmark/oxalpha-supergene/scores.json","Official EigenformAI self-reported baseline; fifty fixed questions; judge OpenAI GPT-5.5; technical rubric; arithmetic mean of score_b","https://huggingface.co/datasets/EigenformAI/groundtruth-dynamic-benchmarking-submissions/resolve/main/submissions/model-benchmark/deepseek-technical/scores.json","https://huggingface.co/datasets/EigenformAI/groundtruth-dynamic-benchmarking-submissions/resolve/main/submissions/model-benchmark/gemini-technical/scores.json","https://huggingface.co/datasets/EigenformAI/groundtruth-dynamic-benchmarking-submissions/resolve/main/submissions/model-benchmark/glm-technical/scores.json","https://huggingface.co/datasets/EigenformAI/groundtruth-dynamic-benchmarking-submissions/resolve/main/submissions/model-benchmark/gpt-technical/scores.json","https://huggingface.co/datasets/EigenformAI/groundtruth-dynamic-benchmarking-submissions/resolve/main/submissions/model-benchmark/grok-technical/scores.json","https://huggingface.co/datasets/EigenformAI/groundtruth-dynamic-benchmarking-submissions/resolve/main/submissions/model-benchmark/kimi-technical/scores.json","https://huggingface.co/datasets/EigenformAI/groundtruth-dynamic-benchmarking-submissions/resolve/main/submissions/model-benchmark/sonnet5-technical/scores.json","https://huggingface.co/datasets/EigenformAI/groundtruth-dynamic-benchmarking-submissions/resolve/main/submissions/model-benchmark/oxalpha-technical/scores.json","Presidio (Redact)","Presidio (Redact); fixed seven-dataset protocol; μ1.1 native composite","https://arxiv.org/html/2608.29624v1#S5.T2","Presidio (Replace)","Presidio (Replace); fixed seven-dataset protocol; μ1.1 native composite","Philter","Philter; fixed seven-dataset protocol; μ1.1 native composite","Textwash","Textwash; fixed seven-dataset protocol; μ1.1 native composite","GLiNER (Redact)","GLiNER (Redact); fixed seven-dataset protocol; μ1.1 native composite","GLiNER (Replace)","GLiNER (Replace); fixed seven-dataset protocol; μ1.1 native composite","1-Diffractor ( ε )","1-Diffractor ( ε ); fixed seven-dataset protocol; μ1.1 native composite","1-Diffractor ( 2 ⋅ ε )","1-Diffractor ( 2 ⋅ ε ); fixed seven-dataset protocol; μ1.1 native composite","SanText ( ε )","SanText ( ε ); fixed seven-dataset protocol; μ1.1 native composite","SanText ( 2 ⋅ ε )","SanText ( 2 ⋅ ε ); fixed seven-dataset protocol; μ1.1 native composite","SanText+ ( ε )","SanText+ ( ε ); fixed seven-dataset protocol; μ1.1 native composite","SanText+ ( 2 ⋅ ε )","SanText+ ( 2 ⋅ ε ); fixed seven-dataset protocol; μ1.1 native composite","CusText ( ε )","CusText ( ε ); fixed seven-dataset protocol; μ1.1 native composite","CusText ( 2 ⋅ ε )","CusText ( 2 ⋅ ε ); fixed seven-dataset protocol; μ1.1 native composite","CusText+ ( ε )","CusText+ ( ε ); fixed seven-dataset protocol; μ1.1 native composite","CusText+ ( 2 ⋅ ε )","CusText+ ( 2 ⋅ ε ); fixed seven-dataset protocol; μ1.1 native composite","DP-Paraphrase ( ε )","DP-Paraphrase ( ε ); fixed seven-dataset protocol; μ1.1 native composite","DP-Paraphrase ( 2 ⋅ ε )","DP-Paraphrase ( 2 ⋅ ε ); fixed seven-dataset protocol; μ1.1 native composite","DP-Prompt ( ε )","DP-Prompt ( ε ); fixed seven-dataset protocol; μ1.1 native composite","DP-Prompt ( 2 ⋅ ε )","DP-Prompt ( 2 ⋅ ε ); fixed seven-dataset protocol; μ1.1 native composite","DP-MLM ( ε )","DP-MLM ( ε ); fixed seven-dataset protocol; μ1.1 native composite","DP-MLM ( 2 ⋅ ε )","DP-MLM ( 2 ⋅ ε ); fixed seven-dataset protocol; μ1.1 native composite","ADePT ( ε )","ADePT ( ε ); fixed seven-dataset protocol; μ1.1 native composite","ADePT ( 2 ⋅ ε )","ADePT ( 2 ⋅ ε ); fixed seven-dataset protocol; μ1.1 native composite","DP-BART ( ε )","DP-BART ( ε ); fixed seven-dataset protocol; μ1.1 native composite","DP-BART ( 2 ⋅ ε )","DP-BART ( 2 ⋅ ε ); fixed seven-dataset protocol; μ1.1 native composite","RUPTA","RUPTA; fixed seven-dataset protocol; μ1.1 native composite","Privacy Filter (Redact)","Privacy Filter (Redact); fixed seven-dataset protocol; μ1.1 native composite","Privacy Filter (Replace)","Privacy Filter (Replace); fixed seven-dataset protocol; μ1.1 native composite","Wan 2.2","Same 100 simulation examples; full-frame SSIM; Wan 2.2","https://arxiv.org/html/2608.15555v1#S4.T3","Cosmos 2.5","Same 100 simulation examples; full-frame SSIM; Cosmos 2.5","Same 100 simulation examples; full-frame SSIM; Veo 3.1","Veo 3.1 Fast","Same 100 simulation examples; full-frame SSIM; Veo 3.1 Fast","Kling 3.0","Same 100 simulation examples; full-frame SSIM; Kling 3.0","Same 100 simulation examples; full-frame SSIM; Seedance 2.0","Seedance 2.0 Fast","Same 100 simulation examples; full-frame SSIM; Seedance 2.0 Fast","Same 100 simulation examples; full-frame SSIM; Grok Imagine","Mistral-Large-675B","Same cleaned 22,021-question deterministic stratified sample; explicit boxed A–D parser; source parse rate 99.9%","https://arxiv.org/html/2610.19848v1#S6.T3","Same cleaned 22,021-question deterministic stratified sample; explicit boxed A–D parser; source parse rate 100.0%","Nemotron-Super-120B","Same cleaned 22,021-question deterministic stratified sample; explicit boxed A–D parser; source parse rate 89.2%","Same cleaned 22,021-question deterministic stratified sample; explicit boxed A–D parser; source parse rate 94.0%","gemma-3-4B-T1-it","Same cleaned 22,021-question deterministic stratified sample; explicit boxed A–D parser; source parse rate 95.4%","Llama-3.2-3B-F1 (Instruct)","Same cleaned 22,021-question deterministic stratified sample; Instruct mode; explicit boxed A–D parser","Llama-3.2-3B-F1 (Reasoning)","Same cleaned 22,021-question deterministic stratified sample; Reasoning mode; explicit boxed A–D parser","Llama-3.1-TAIDE-LX-8B","Same cleaned 22,021-question deterministic stratified sample; explicit boxed A–D parser; source parse rate 32.2%","Vals Index v2.2; eight fixed components; GDP weights8.0:5.6:1.2:0.5; official November5 result snapshot","https://www.vals.ai/benchmarks/vals_index","Hy4","Inkling 2 Small","Robo-Dopamine (Forward)","Specialized PRM; Robo-Dopamine (Forward); same2,244 annotated intervals","https://arxiv.org/html/2608.14284v1#A3.T6","Robo-Dopamine (Backward)","Specialized PRM; Robo-Dopamine (Backward); same2,244 annotated intervals","Robo-Dopamine (Incremental)","Specialized PRM; Robo-Dopamine (Incremental); same2,244 annotated intervals","RoboMeter","Specialized PRM; RoboMeter; same2,244 annotated intervals","LRM (Temporal)","Specialized PRM; LRM (Temporal); same2,244 annotated intervals","LRM (Absolute)","Specialized PRM; LRM (Absolute); same2,244 annotated intervals","TOPReward","Specialized PRM; TOPReward; same2,244 annotated intervals","VLAC","Specialized PRM; VLAC; same2,244 annotated intervals","GVL","Specialized PRM; GVL; same2,244 annotated intervals","PRIMO","Specialized PRM; PRIMO; same2,244 annotated intervals","General VLM; Pair Style; same2,244 annotated intervals","General VLM; Sequence Style; same2,244 annotated intervals","Image-editing native interface;1000fixed expand-text edits","https://arxiv.org/html/2609.33286v1#S3.T2","Nanobanana-Pro","Nanobanana-2","Nanobanana","Seedream-5.0-Lite","HunyuanImage-3.0","FLUX.2-klein-base-9B","Image-editing native interface;1000fixed insert-element edits","Image-editing native interface;1000fixed swap-block edits","Image-editing native interface;1000fixed reshape-canvas edits","Four equally weighted tasks;4000edits; fixed native generation recipe","Brief instruction;16tasks×3runs; harness=DeepSeek H.; zero provider refusals;47/1/0/0","https://arxiv.org/html/2609.34274v1#S5.T1","Brief instruction;16tasks×3runs; harness=Claude Code; zero provider refusals;48/0/0/0","Brief instruction;16tasks×3runs; harness=DeepSeek H.; zero provider refusals;45/3/0/0","Brief instruction;16tasks×3runs; harness=DeepSeek H.; zero provider refusals;44/4/0/0","Brief instruction;16tasks×3runs; harness=DeepSeek H.; zero provider refusals;48/0/0/0","ML Alpha52 long-only; 2025-backtest; validation-selected frozen trading settings;fixed non-LLM seed","https://arxiv.org/html/2609.34510v1#S4.T1","SFM","GeneralPtNN","ALSTM","TFT","KRNN","Localformer","TRA","AdaRNN","TCTS","ADD","DoubleEnsemble","GATs","Sandwich","IGMTF","ML Alpha52 long-only; 2026-paper-trading; validation-selected frozen trading settings;fixed non-LLM seed","https://arxiv.org/html/2609.34510v1#S4.T2","Full-task100rollouts/task; one training seed; native local observation/policy sharing","https://arxiv.org/html/2609.34782v1#S4.T2","GauDP","Claude Code; high reasoning effort where exposed;RTX4090;one attempt/task","https://arxiv.org/html/2609.35639v1#S4.T1","Codex CLI; high reasoning effort where exposed;RTX4090;one attempt/task","Gemini CLI; high reasoning effort where exposed;RTX4090;one attempt/task","DeepSeek Harness; high reasoning effort where exposed;RTX4090;one attempt/task","Qwen Code; high reasoning effort where exposed;RTX4090;one attempt/task","OpenCode; high reasoning effort where exposed;RTX4090;one attempt/task","facts condition;same15development dossiers;30turns/60calls;native provider settings","https://arxiv.org/html/2609.34913v1#S7.T7","GPT-6 Luna Pro","CLAUDE SONNET 4.6","Main paper setting;independent questions;T0;GPT-OSS low effort;exact/JM protocolv1; original checkpoint=CLAUDE SONNET 4.6","https://arxiv.org/html/2609.35312v1#S4.T5","Main paper setting;independent questions;T0;GPT-OSS low effort;exact/JM protocolv1; original checkpoint=GPT-OSS 120B","QWEN3.5 27B","Main paper setting;independent questions;T0;GPT-OSS low effort;exact/JM protocolv1; original checkpoint=QWEN3.5 27B","GPT-OSS 20B","Main paper setting;independent questions;T0;GPT-OSS low effort;exact/JM protocolv1; original checkpoint=GPT-OSS 20B","QWEN3.5 35B-A3B","Main paper setting;independent questions;T0;GPT-OSS low effort;exact/JM protocolv1; original checkpoint=QWEN3.5 35B-A3B","GEMMA-4 26B-A4B-IT","Main paper setting;independent questions;T0;GPT-OSS low effort;exact/JM protocolv1; original checkpoint=GEMMA-4 26B-A4B-IT","LLAMA-3.1 8B-INSTRUCT","Main paper setting;independent questions;T0;GPT-OSS low effort;exact/JM protocolv1; original checkpoint=LLAMA-3.1 8B-INSTRUCT","Main paper setting;independent questions;T0;GPT-OSS low effort;exact/JM protocolv1; original checkpoint=OLMO-3 7B-THINK","Main paper setting;independent questions;T0;GPT-OSS low effort;exact/JM protocolv1; original checkpoint=OLMO-3 7B-INSTRUCT","Zero-shot;24tasks/1160examples;original checkpoint=Qwen3-VL-8B-Instruct ( 2025 );greedy/nonthinking/64tokens","https://arxiv.org/html/2609.34072v1#S4.T1.fig1","Zero-shot;24tasks/1160examples;original checkpoint=LLaVA-OneVision-2-8B ( 2026 );greedy/nonthinking/64tokens","Zero-shot;24tasks/1160examples;original checkpoint=Intern-S1-Pro ( 2026 );API T0/16384tokens/high effort where supported","Zero-shot;24tasks/1160examples;original checkpoint=Kimi K2.6 ( 2026 );API T0/16384tokens/high effort where supported","Zero-shot;24tasks/1160examples;original checkpoint=GLM-5V-Turbo ( 2026 );API T0/16384tokens/high effort where supported","Zero-shot;24tasks/1160examples;original checkpoint=Qwen3.6-Plus ( 2026 );API T0/16384tokens/high effort where supported","Zero-shot;24tasks/1160examples;original checkpoint=Doubao-Seed-2.0-Pro ( 2026 );API T0/16384tokens/high effort where supported","Zero-shot;24tasks/1160examples;original checkpoint=Claude Opus 4.7 ( 2026 );API T0/16384tokens/high effort where supported","Zero-shot;24tasks/1160examples;original checkpoint=Gemini 3 Flash ( 2025 );API T0/16384tokens/high effort where supported","Zero-shot;24tasks/1160examples;original checkpoint=Gemini 3.1 Pro ( 2026a );API T0/16384tokens/high effort where supported","Zero-shot;24tasks/1160examples;original checkpoint=Gemini 3.5 Flash ( 2026b );API T0/16384tokens/high effort where supported","Zero-shot;24tasks/1160examples;original checkpoint=GPT-5.5 ( 2026 );API T0/16384tokens/high effort where supported","Multi-SWT-Bench-Lite340; harness=MSWE-agent;original checkpoint=GPT-5.6 Luna;native API/subscription inference","https://arxiv.org/html/2609.34752v1#S4.T4","Multi-SWT-Bench-Lite340; harness=MSWE-agent;original checkpoint=Claude Sonnet 5;native API/subscription inference","Multi-SWT-Bench-Lite340; harness=MSWE-agent;original checkpoint=DeepSeek-V4-Flash;native API/subscription inference","Multi-SWT-Bench-Lite340; harness=MSWE-agent;original checkpoint=DeepSeek-V4-Pro;native API/subscription inference","Multi-SWT-Bench-Lite340; harness=MOpenHands;original checkpoint=GPT-5.6 Luna;native API/subscription inference","Multi-SWT-Bench-Lite340; harness=MOpenHands;original checkpoint=Claude Sonnet 5;native API/subscription inference","Multi-SWT-Bench-Lite340; harness=MOpenHands;original checkpoint=DeepSeek-V4-Flash;native API/subscription inference","Multi-SWT-Bench-Lite340; harness=MOpenHands;original checkpoint=DeepSeek-V4-Pro;native API/subscription inference","Multi-SWT-Bench-Lite340; harness=Codex;original checkpoint=GPT-5.6 Luna;native API/subscription inference","Multi-SWT-Bench-Lite340; harness=Codex;original checkpoint=GPT-6 Astra;native API/subscription inference","Multi-SWT-Bench-Lite340; harness=ClaudeCode;original checkpoint=Claude Sonnet 5;native API/subscription inference","Multi-SWT-Bench-Lite340; harness=ClaudeCode;original checkpoint=Claude Opus 5;native API/subscription inference","Gemma-4 E2B (5B)","Scaffold=Monolithic VLM;original backbone=Gemma-4 E2B (5B);zero-shot;RTX4090","https://arxiv.org/html/2609.35032v1#S5.T3","Gemma-4 E4B (8B)","Scaffold=Monolithic VLM;original backbone=Gemma-4 E4B (8B);zero-shot;RTX4090","Scaffold=Monolithic VLM;original backbone=Qwen3-VL 4B;zero-shot;RTX4090","Scaffold=Monolithic VLM;original backbone=Qwen3-VL 8B;zero-shot;RTX4090","Scaffold=Monolithic VLM;original backbone=Qwen3.5 4B;zero-shot;RTX4090","Scaffold=Monolithic VLM;original backbone=Qwen3.5 9B;zero-shot;RTX4090","Scaffold=Chain-of-Thought;original backbone=Gemma-4 E2B (5B);zero-shot;RTX4090","Scaffold=Chain-of-Thought;original backbone=Gemma-4 E4B (8B);zero-shot;RTX4090","Scaffold=Chain-of-Thought;original backbone=Qwen3-VL 4B;zero-shot;RTX4090","Scaffold=Chain-of-Thought;original backbone=Qwen3-VL 8B;zero-shot;RTX4090","Scaffold=Chain-of-Thought;original backbone=Qwen3.5 4B;zero-shot;RTX4090","Scaffold=Chain-of-Thought;original backbone=Qwen3.5 9B;zero-shot;RTX4090","Scaffold=Search-Recognize-Pipeline;original backbone=Gemma-4 E2B (5B);zero-shot;RTX4090","Scaffold=Search-Recognize-Pipeline;original backbone=Gemma-4 E4B (8B);zero-shot;RTX4090","Scaffold=Search-Recognize-Pipeline;original backbone=Qwen3-VL 4B;zero-shot;RTX4090","Scaffold=Search-Recognize-Pipeline;original backbone=Qwen3-VL 8B;zero-shot;RTX4090","Scaffold=Search-Recognize-Pipeline;original backbone=Qwen3.5 4B;zero-shot;RTX4090","Scaffold=Search-Recognize-Pipeline;original backbone=Qwen3.5 9B;zero-shot;RTX4090","Scaffold=ReAct;original backbone=Gemma-4 E2B (5B);zero-shot;RTX4090","Scaffold=ReAct;original backbone=Gemma-4 E4B (8B);zero-shot;RTX4090","Scaffold=ReAct;original backbone=Qwen3-VL 4B;zero-shot;RTX4090","Scaffold=ReAct;original backbone=Qwen3-VL 8B;zero-shot;RTX4090","Scaffold=ReAct;original backbone=Qwen3.5 4B;zero-shot;RTX4090","Scaffold=ReAct;original backbone=Qwen3.5 9B;zero-shot;RTX4090","Claude-3-Haiku","Claude-3-Haiku; original native reasoning mode; shared task prompt","https://arxiv.org/pdf/2609.35409v1#page=6","Claude-Sonnet-4; original native reasoning mode; shared task prompt","Claude-Opus-4.1","Claude-Opus-4.1; original native reasoning mode; shared task prompt","DeepSeek-V3; original native reasoning mode; shared task prompt","DeepSeek-R1; original native reasoning mode; shared task prompt","Gemini-2.0-Flash; original native reasoning mode; shared task prompt","Gemini-2.5-Flash-Nothinking; original native reasoning mode; shared task prompt","Gemini-2.5-Flash-Thinking; original native reasoning mode; shared task prompt","Gemini-2.5-Pro; original native reasoning mode; shared task prompt","GPT-4; original native reasoning mode; shared task prompt","GPT-5-Chat; original native reasoning mode; shared task prompt","O4-mini; original native reasoning mode; shared task prompt","GPT-5-Thinking; original native reasoning mode; shared task prompt","GPT-OSS-20B; original native reasoning mode; shared task prompt","GPT-OSS-120B; original native reasoning mode; shared task prompt","Qwen2.5-7B; original native reasoning mode; shared task prompt","Qwen3-8B; original native reasoning mode; shared task prompt","Qwen3-235B-A22B; original native reasoning mode; shared task prompt","Qwen-3.8-Max;128frames640x480+ASR;retrieved;native serving/token configuration","https://arxiv.org/pdf/2609.34314v1#page=7","Qwen-3.7-Flash","Qwen-3.7-Flash;128frames640x480+ASR;retrieved;native serving/token configuration","GPT-5.6-Terra;128frames640x480+ASR;retrieved;native serving/token configuration","Kimi-K2.6;128frames640x480+ASR;retrieved;native serving/token configuration","Gemma-4-26B-A4B;128frames640x480+ASR;retrieved;native serving/token configuration","Gemma-4-E2B;128frames640x480+ASR;retrieved;native serving/token configuration","Gemma-4-E4B;128frames640x480+ASR;retrieved;native serving/token configuration","Qwen-3.5-9B","Qwen-3.5-9B;128frames640x480+ASR;retrieved;native serving/token configuration","Qwen-3.5-4B","Qwen-3.5-4B;128frames640x480+ASR;retrieved;native serving/token configuration","Qwen-3.5-2B","Qwen-3.5-2B;128frames640x480+ASR;retrieved;native serving/token configuration","InternLM-2.5-Reward","InternLM-2.5-Reward;128frames640x480+ASR;retrieved;native serving/token configuration","VideoJudge-3B","VideoJudge-3B;128frames640x480+ASR;retrieved;native serving/token configuration","VideoJudge-7B","VideoJudge-7B;128frames640x480+ASR;retrieved;native serving/token configuration","Qwen-3.8-Max;128frames640x480+ASR;uniform;native serving/token configuration","Qwen-3.7-Flash;128frames640x480+ASR;uniform;native serving/token configuration","GPT-5.6-Terra;128frames640x480+ASR;uniform;native serving/token configuration","Kimi-K2.6;128frames640x480+ASR;uniform;native serving/token configuration","Gemma-4-26B-A4B;128frames640x480+ASR;uniform;native serving/token configuration","Gemma-4-E2B;128frames640x480+ASR;uniform;native serving/token configuration","Gemma-4-E4B;128frames640x480+ASR;uniform;native serving/token configuration","Qwen-3.5-9B;128frames640x480+ASR;uniform;native serving/token configuration","Qwen-3.5-4B;128frames640x480+ASR;uniform;native serving/token configuration","Qwen-3.5-2B;128frames640x480+ASR;uniform;native serving/token configuration","InternLM-2.5-Reward;128frames640x480+ASR;uniform;native serving/token configuration","VideoJudge-3B;128frames640x480+ASR;uniform;native serving/token configuration","VideoJudge-7B;128frames640x480+ASR;uniform;native serving/token configuration","Frozen761devices;build seed20260906/scope seed11;50calls/300s;medium effort where exposed;checkpoint=GPT-5.6-Sol","https://arxiv.org/html/2609.34492v1#S4.T4","Frozen761devices;build seed20260906/scope seed11;50calls/300s;medium effort where exposed;checkpoint=GLM-5.3","Frozen761devices;build seed20260906/scope seed11;50calls/300s;medium effort where exposed;checkpoint=Doubao-Seed-Evolving","Frozen761devices;build seed20260906/scope seed11;50calls/300s;medium effort where exposed;checkpoint=DeepSeek-V4-Pro","Frozen761devices;build seed20260906/scope seed11;50calls/300s;medium effort where exposed;checkpoint=Gemini-3.6-Flash","Frozen761devices;build seed20260906/scope seed11;50calls/300s;medium effort where exposed;checkpoint=Kimi-K3","Frozen761devices;build seed20260906/scope seed11;50calls/300s;medium effort where exposed;checkpoint=MiniMax-M3","Frozen761devices;build seed20260906/scope seed11;50calls/300s;medium effort where exposed;checkpoint=Qwen3.8-Max","Frozen761devices;build seed20260906/scope seed11;50calls/300s;medium effort where exposed;checkpoint=Llama-4-Maverick","iMCTS","FrozenCore50;seeds520/521/522;3hours/run;algorithm=iMCTS;native search/operator recipe","https://arxiv.org/html/2609.35113v1#S4.T2","QLattice","FrozenCore50;seeds520/521/522;3hours/run;algorithm=QLattice;native search/operator recipe","JAXSR","FrozenCore50;seeds520/521/522;3hours/run;algorithm=JAXSR;native search/operator recipe","PySR","FrozenCore50;seeds520/521/522;3hours/run;algorithm=PySR;native search/operator recipe","PyOperon","FrozenCore50;seeds520/521/522;3hours/run;algorithm=PyOperon;native search/operator recipe","gplearn","FrozenCore50;seeds520/521/522;3hours/run;algorithm=gplearn;native search/operator recipe","SymbolFit","FrozenCore50;seeds520/521/522;3hours/run;algorithm=SymbolFit;native search/operator recipe","uDSR","FrozenCore50;seeds520/521/522;3hours/run;algorithm=uDSR;native search/operator recipe","DSO","FrozenCore50;seeds520/521/522;3hours/run;algorithm=DSO;native search/operator recipe","FePySR","FrozenCore50;seeds520/521/522;3hours/run;algorithm=FePySR;native search/operator recipe","RAG-SR","FrozenCore50;seeds520/521/522;3hours/run;algorithm=RAG-SR;native search/operator recipe","E2ESR","FrozenCore50;seeds520/521/522;3hours/run;algorithm=E2ESR;native search/operator recipe","TPSR","FrozenCore50;seeds520/521/522;3hours/run;algorithm=TPSR;native search/operator recipe","DrSR","FrozenCore50;seeds520/521/522;3hours/run;algorithm=DrSR;native search/operator recipe","LLM-SR","FrozenCore50;seeds520/521/522;3hours/run;algorithm=LLM-SR;native search/operator recipe","Natural904prompts×16;family=DeepSeek-R1-Distill-Llama;size=8B;native generation recipe","https://arxiv.org/html/2609.34455v1#S5.T1","Natural904prompts×16;family=DeepSeek-R1-Distill-Llama;size=70B;native generation recipe","Natural904prompts×16;family=gpt-oss;size=20B;native generation recipe","Natural904prompts×16;family=gpt-oss;size=120B;native generation recipe","Natural904prompts×16;family=Qwen3.5;size=9B;native generation recipe","Natural904prompts×16;family=Qwen3.5;size=27B;native generation recipe","Full balanced held-out3564responses;AppendixD.2 score extraction;checkpoint=GPT-5.6 Sol","https://arxiv.org/html/2609.34455v1#S5.T2","Full balanced held-out3564responses;AppendixD.2 score extraction;checkpoint=Gemini-2.5-Pro","Full balanced held-out3564responses;AppendixD.2 score extraction;checkpoint=GLM-5.2","Full balanced held-out3564responses;AppendixD.2 score extraction;checkpoint=Qwen3.5-0.8B","Full balanced held-out3564responses;AppendixD.2 score extraction;checkpoint=Qwen3.5-4B","Full balanced held-out3564responses;AppendixD.2 score extraction;checkpoint=Qwen3.5-9B","Full balanced held-out3564responses;AppendixD.2 score extraction;checkpoint=Qwen3.5-27B","Full balanced held-out3564responses;AppendixD.2 score extraction;checkpoint=Qwen3.5-35B-A3B","Full balanced held-out3564responses;AppendixD.2 score extraction;checkpoint=gpt-oss-20B","Full balanced held-out3564responses;AppendixD.2 score extraction;checkpoint=gpt-oss-120B","Full balanced held-out3564responses;AppendixD.2 score extraction;checkpoint=DeepSeek-R1-Distill-Llama-8B","Full balanced held-out3564responses;AppendixD.2 score extraction;checkpoint=DeepSeek-R1-Distill-Llama-70B","Full balanced held-out3564responses;AppendixD.2 score extraction;checkpoint=Llama-3.1-8B-Instruct","Full balanced held-out3564responses;AppendixD.2 score extraction;checkpoint=Llama-3.1-70B-Instruct","Full balanced held-out3564responses;AppendixD.2 score extraction;checkpoint=Skywork-Reward-V2-Llama-3.1-8B","dORM-8B","Full balanced held-out3564responses;AppendixD.2 score extraction;checkpoint=dORM-8B","TRM-8B","Full balanced held-out3564responses;AppendixD.2 score extraction;checkpoint=TRM-8B","Split=full;one image/prompt;frozen pixel verifier;native recipe=GPT-Image-2","https://arxiv.org/html/2609.35641v1#S3.T3","GPT-Image-1-mini","Split=full;one image/prompt;frozen pixel verifier;native recipe=GPT-Image-1-mini","Split=full;one image/prompt;frozen pixel verifier;native recipe=FLUX.2-dev","HunyuanImage-2.1","Split=full;one image/prompt;frozen pixel verifier;native recipe=HunyuanImage-2.1","Qwen-Image-2512","Split=full;one image/prompt;frozen pixel verifier;native recipe=Qwen-Image-2512","HiDream-I1-Full","Split=full;one image/prompt;frozen pixel verifier;native recipe=HiDream-I1-Full","FLUX.1","Split=full;one image/prompt;frozen pixel verifier;native recipe=FLUX.1-dev","Split=full;one image/prompt;frozen pixel verifier;native recipe=FLUX.1-schnell","SD3.5 Medium","Split=full;one image/prompt;frozen pixel verifier;native recipe=SD3.5 Medium","SD3.5 Large","Split=full;one image/prompt;frozen pixel verifier;native recipe=SD3.5 Large","SDXL 1.0","Split=full;one image/prompt;frozen pixel verifier;native recipe=SDXL 1.0","Sana 1.6B","Split=full;one image/prompt;frozen pixel verifier;native recipe=Sana 1.6B","GPT-Image-2.5-Sunburst","Split=fast;one image/prompt;frozen pixel verifier;native recipe=GPT-Image-2.5-Sunburst","https://arxiv.org/html/2609.35641v1#A4.T10","Split=fast;one image/prompt;frozen pixel verifier;native recipe=GPT-Image-2","Gemini-3.1-Flash-Image","Split=fast;one image/prompt;frozen pixel verifier;native recipe=Gemini-3.1-Flash-Image","Gemini-2.5-Flash-Image","Split=fast;one image/prompt;frozen pixel verifier;native recipe=Gemini-2.5-Flash-Image","Gemini-3.1-Flash-Lite-Image","Split=fast;one image/prompt;frozen pixel verifier;native recipe=Gemini-3.1-Flash-Lite-Image","Gemini-3-Pro-Image","Split=fast;one image/prompt;frozen pixel verifier;native recipe=Gemini-3-Pro-Image","Split=fast;one image/prompt;frozen pixel verifier;native recipe=GPT-Image-1-mini","Split=challenge;one image/prompt;frozen pixel verifier;native recipe=GPT-Image-2.5-Sunburst","https://arxiv.org/html/2609.35641v1#S3.F2.fig2","Split=challenge;one image/prompt;frozen pixel verifier;native recipe=GPT-Image-2","Split=challenge;one image/prompt;frozen pixel verifier;native recipe=Gemini-3.1-Flash-Lite-Image","Split=challenge;one image/prompt;frozen pixel verifier;native recipe=Gemini-3-Pro-Image","Split=challenge;one image/prompt;frozen pixel verifier;native recipe=Gemini-3.1-Flash-Image","Split=challenge;one image/prompt;frozen pixel verifier;native recipe=Gemini-2.5-Flash-Image","Split=challenge;one image/prompt;frozen pixel verifier;native recipe=GPT-Image-1-mini","screenshot-only;harness=qwen3-vl;original checkpoint=qwen3.8-27b;native single-run defaults","https://arxiv.org/html/2609.35026v1#S4.T3","fara-1.5-9b","screenshot-only;harness=fara;original checkpoint=fara-1.5-9b;native single-run defaults","fara-1.5-27b","screenshot-only;harness=fara;original checkpoint=fara-1.5-27b;native single-run defaults","opencua-72b","screenshot-only;harness=opencua;original checkpoint=opencua-72b;native single-run defaults","opencua-32b","screenshot-only;harness=opencua;original checkpoint=opencua-32b;native single-run defaults","uitars-1.5-7b","screenshot-only;harness=uitars;original checkpoint=uitars-1.5-7b;native single-run defaults","EvoCUA-32B","screenshot-only;harness=evocua;original checkpoint=evocua-32b-s1;native single-run defaults","screenshot-only;harness=evocua;original checkpoint=evocua-32b-s2;native single-run defaults","Sonnet-4.5","Task=fabrication;method=disinfocap;60topics;GPT5.2judge;target checkpoint=Sonnet-4.5","https://arxiv.org/html/2609.35028v1#A2.T4","Task=fabrication;method=disinfocap;60topics;GPT5.2judge;target checkpoint=GPT-5.4","Task=fabrication;method=disinfocap;60topics;GPT5.2judge;target checkpoint=Gemini-3.1-Pro","Task=fabrication;method=disinfocap;60topics;GPT5.2judge;target checkpoint=Qwen3.5-Flash","Task=fabrication;method=disinfocap;60topics;GPT5.2judge;target checkpoint=Grok-4.1-Fast","Task=fabrication;method=disinfocap;60topics;GPT5.2judge;target checkpoint=DeepSeek-V4-Pro","Task=fabrication;method=disinfocap;60topics;GPT5.2judge;target checkpoint=Kimi-K2.5","Task=fabrication;method=isc;60topics;GPT5.2judge;target checkpoint=Sonnet-4.5","Task=fabrication;method=isc;60topics;GPT5.2judge;target checkpoint=GPT-5.4","Task=fabrication;method=isc;60topics;GPT5.2judge;target checkpoint=Gemini-3.1-Pro","Task=fabrication;method=isc;60topics;GPT5.2judge;target checkpoint=Qwen3.5-Flash","Task=fabrication;method=isc;60topics;GPT5.2judge;target checkpoint=Grok-4.1-Fast","Task=fabrication;method=isc;60topics;GPT5.2judge;target checkpoint=DeepSeek-V4-Pro","Task=fabrication;method=isc;60topics;GPT5.2judge;target checkpoint=Kimi-K2.5","Task=fabrication;method=jnb;60topics;GPT5.2judge;target checkpoint=Sonnet-4.5","Task=fabrication;method=jnb;60topics;GPT5.2judge;target checkpoint=GPT-5.4","Task=fabrication;method=jnb;60topics;GPT5.2judge;target checkpoint=Gemini-3.1-Pro","Task=fabrication;method=jnb;60topics;GPT5.2judge;target checkpoint=Qwen3.5-Flash","Task=fabrication;method=jnb;60topics;GPT5.2judge;target checkpoint=Grok-4.1-Fast","Task=fabrication;method=jnb;60topics;GPT5.2judge;target checkpoint=DeepSeek-V4-Pro","Task=fabrication;method=jnb;60topics;GPT5.2judge;target checkpoint=Kimi-K2.5","Task=fabrication;method=misinfoqa;60topics;GPT5.2judge;target checkpoint=Sonnet-4.5","Task=fabrication;method=misinfoqa;60topics;GPT5.2judge;target checkpoint=GPT-5.4","Task=fabrication;method=misinfoqa;60topics;GPT5.2judge;target checkpoint=Gemini-3.1-Pro","Task=fabrication;method=misinfoqa;60topics;GPT5.2judge;target checkpoint=Qwen3.5-Flash","Task=fabrication;method=misinfoqa;60topics;GPT5.2judge;target checkpoint=Grok-4.1-Fast","Task=fabrication;method=misinfoqa;60topics;GPT5.2judge;target checkpoint=DeepSeek-V4-Pro","Task=fabrication;method=misinfoqa;60topics;GPT5.2judge;target checkpoint=Kimi-K2.5","Task=fabrication;method=poisoned-rag;60topics;GPT5.2judge;target checkpoint=Sonnet-4.5","Task=fabrication;method=poisoned-rag;60topics;GPT5.2judge;target checkpoint=GPT-5.4","Task=fabrication;method=poisoned-rag;60topics;GPT5.2judge;target checkpoint=Gemini-3.1-Pro","Task=fabrication;method=poisoned-rag;60topics;GPT5.2judge;target checkpoint=Qwen3.5-Flash","Task=fabrication;method=poisoned-rag;60topics;GPT5.2judge;target checkpoint=Grok-4.1-Fast","Task=fabrication;method=poisoned-rag;60topics;GPT5.2judge;target checkpoint=DeepSeek-V4-Pro","Task=fabrication;method=poisoned-rag;60topics;GPT5.2judge;target checkpoint=Kimi-K2.5","Task=fabrication;method=pap;60topics;GPT5.2judge;target checkpoint=Sonnet-4.5","Task=fabrication;method=pap;60topics;GPT5.2judge;target checkpoint=GPT-5.4","Task=fabrication;method=pap;60topics;GPT5.2judge;target checkpoint=Gemini-3.1-Pro","Task=fabrication;method=pap;60topics;GPT5.2judge;target checkpoint=Qwen3.5-Flash","Task=fabrication;method=pap;60topics;GPT5.2judge;target checkpoint=Grok-4.1-Fast","Task=fabrication;method=pap;60topics;GPT5.2judge;target checkpoint=DeepSeek-V4-Pro","Task=fabrication;method=pap;60topics;GPT5.2judge;target checkpoint=Kimi-K2.5","Task=rewrite;method=direct;60topics;GPT5.2judge;target checkpoint=GPT-5.4","https://arxiv.org/html/2609.35028v1#A2.T5","Task=rewrite;method=direct;60topics;GPT5.2judge;target checkpoint=Gemini-3.1-Pro","Task=rewrite;method=direct;60topics;GPT5.2judge;target checkpoint=Qwen3.5-Flash","Task=rewrite;method=direct;60topics;GPT5.2judge;target checkpoint=Grok-4.1-Fast","Task=rewrite;method=direct;60topics;GPT5.2judge;target checkpoint=DeepSeek-V4-Pro","Task=rewrite;method=direct;60topics;GPT5.2judge;target checkpoint=Kimi-K2.5","Task=rewrite;method=disinfocap;60topics;GPT5.2judge;target checkpoint=Sonnet-4.5","Task=rewrite;method=disinfocap;60topics;GPT5.2judge;target checkpoint=GPT-5.4","Task=rewrite;method=disinfocap;60topics;GPT5.2judge;target checkpoint=Gemini-3.1-Pro","Task=rewrite;method=disinfocap;60topics;GPT5.2judge;target checkpoint=Qwen3.5-Flash","Task=rewrite;method=disinfocap;60topics;GPT5.2judge;target checkpoint=Grok-4.1-Fast","Task=rewrite;method=disinfocap;60topics;GPT5.2judge;target checkpoint=DeepSeek-V4-Pro","Task=rewrite;method=disinfocap;60topics;GPT5.2judge;target checkpoint=Kimi-K2.5","Task=rewrite;method=isc;60topics;GPT5.2judge;target checkpoint=Sonnet-4.5","Task=rewrite;method=isc;60topics;GPT5.2judge;target checkpoint=GPT-5.4","Task=rewrite;method=isc;60topics;GPT5.2judge;target checkpoint=Gemini-3.1-Pro","Task=rewrite;method=isc;60topics;GPT5.2judge;target checkpoint=Qwen3.5-Flash","Task=rewrite;method=isc;60topics;GPT5.2judge;target checkpoint=Grok-4.1-Fast","Task=rewrite;method=isc;60topics;GPT5.2judge;target checkpoint=DeepSeek-V4-Pro","Task=rewrite;method=isc;60topics;GPT5.2judge;target checkpoint=Kimi-K2.5","Task=rewrite;method=jnb;60topics;GPT5.2judge;target checkpoint=Sonnet-4.5","Task=rewrite;method=jnb;60topics;GPT5.2judge;target checkpoint=GPT-5.4","Task=rewrite;method=jnb;60topics;GPT5.2judge;target checkpoint=Gemini-3.1-Pro","Task=rewrite;method=jnb;60topics;GPT5.2judge;target checkpoint=Qwen3.5-Flash","Task=rewrite;method=jnb;60topics;GPT5.2judge;target checkpoint=Grok-4.1-Fast","Task=rewrite;method=jnb;60topics;GPT5.2judge;target checkpoint=DeepSeek-V4-Pro","Task=rewrite;method=jnb;60topics;GPT5.2judge;target checkpoint=Kimi-K2.5","Task=rewrite;method=misinfoqa;60topics;GPT5.2judge;target checkpoint=Sonnet-4.5","Task=rewrite;method=misinfoqa;60topics;GPT5.2judge;target checkpoint=GPT-5.4","Task=rewrite;method=misinfoqa;60topics;GPT5.2judge;target checkpoint=Gemini-3.1-Pro","Task=rewrite;method=misinfoqa;60topics;GPT5.2judge;target checkpoint=Qwen3.5-Flash","Task=rewrite;method=misinfoqa;60topics;GPT5.2judge;target checkpoint=Grok-4.1-Fast","Task=rewrite;method=misinfoqa;60topics;GPT5.2judge;target checkpoint=DeepSeek-V4-Pro","Task=rewrite;method=misinfoqa;60topics;GPT5.2judge;target checkpoint=Kimi-K2.5","Task=rewrite;method=poisoned-rag;60topics;GPT5.2judge;target checkpoint=Sonnet-4.5","Task=rewrite;method=poisoned-rag;60topics;GPT5.2judge;target checkpoint=GPT-5.4","Task=rewrite;method=poisoned-rag;60topics;GPT5.2judge;target checkpoint=Gemini-3.1-Pro","Task=rewrite;method=poisoned-rag;60topics;GPT5.2judge;target checkpoint=Qwen3.5-Flash","Task=rewrite;method=poisoned-rag;60topics;GPT5.2judge;target checkpoint=Grok-4.1-Fast","Task=rewrite;method=poisoned-rag;60topics;GPT5.2judge;target checkpoint=DeepSeek-V4-Pro","Task=rewrite;method=poisoned-rag;60topics;GPT5.2judge;target checkpoint=Kimi-K2.5","Task=rewrite;method=pap;60topics;GPT5.2judge;target checkpoint=Sonnet-4.5","Task=rewrite;method=pap;60topics;GPT5.2judge;target checkpoint=GPT-5.4","Task=rewrite;method=pap;60topics;GPT5.2judge;target checkpoint=Gemini-3.1-Pro","Task=rewrite;method=pap;60topics;GPT5.2judge;target checkpoint=Qwen3.5-Flash","Task=rewrite;method=pap;60topics;GPT5.2judge;target checkpoint=Grok-4.1-Fast","Task=rewrite;method=pap;60topics;GPT5.2judge;target checkpoint=DeepSeek-V4-Pro","Task=rewrite;method=pap;60topics;GPT5.2judge;target checkpoint=Kimi-K2.5","Full five-game panel; native TACE completeness","https://arxiv.org/html/2610.11129v1#S5.F4","One video/prompt; native audio yes; frozen Gemini3.1Pro judge","https://arxiv.org/html/2610.15452v1#S4.F3","Wan-3.0","MiniMax-H3-Max (fal)","Kling-3.0","Wan-2.2","One video/prompt; native audio no; frozen Gemini3.1Pro judge","Kandinsky-5.0-T2V-Pro","LongCat-Video-13.6B","Hermes Agent Hermes Agent v0.22.2+2363.g9cb34e9 (2026.9.24) · stable · local 9cb34e9e285d; Ollama 0.40.6; Q4_K_M; ollama-default (harness sets no temperature/seed); actual context32768; hardware NVIDIA GeForce RTX 3090; host 37becd33558a; run 20261021-gemma4-e4b-239167; measured 2026-10-21T23:05:19+00:00; scoring3,3 trials","https://github.com/localagentbench/Localagentbench/blob/b52a97f45fe522430a78d62489dba71e1c0400a3/data/leaderboard.json","Hermes Agent Hermes Agent v0.22.2+2363.g9cb34e9 (2026.9.24) · stable · local 9cb34e9e285d; Ollama 0.40.6; Q4_K_M; ollama-default (harness sets no temperature/seed); actual context32768; hardware NVIDIA GeForce RTX 3090; host 37becd33558a; run 20261021-qwen3-4b-e68e65; measured 2026-10-21T16:51:01+00:00; scoring3,3 trials","Hermes Agent Hermes Agent v0.22.2+2363.g9cb34e9 (2026.9.24) · stable · local 9cb34e9e285d; Ollama 0.40.6; Q4_K_M; ollama-default (harness sets no temperature/seed); actual context32768; hardware NVIDIA GeForce RTX 3090; host dc7718586fbe; run 20261020-qwen2.5-7b-5ea37d; measured 2026-10-20T22:28:12+00:00; scoring3,3 trials","Hermes Agent Hermes Agent v0.22.2+2363.g9cb34e9 (2026.9.24) · stable · local 9cb34e9e285d; Ollama 0.40.6; Q4_K_M; ollama-default (harness sets no temperature/seed); actual context32768; hardware NVIDIA GeForce RTX 3090; host dc7718586fbe; run 20261020-llama3.2-3b-4559ab; measured 2026-10-20T19:47:27+00:00; scoring3,3 trials","Hermes Agent Hermes Agent v0.22.2+2363.g9cb34e9 (2026.9.24) · stable · local 9cb34e9e285d; Ollama 0.40.6; Q4_K_M; ollama-default (harness sets no temperature/seed); actual context32768; hardware NVIDIA GeForce RTX 3090; host dc7718586fbe; run 20261020-qwen2.5-coder-7b-7d2eee; measured 2026-10-20T23:29:27+00:00; scoring3,3 trials","Hermes Agent Hermes Agent v0.22.2+2363.g9cb34e9 (2026.9.24) · stable · local 9cb34e9e285d; Ollama 0.40.6; Q4_K_M; ollama-default (harness sets no temperature/seed); actual context32768; hardware NVIDIA GeForce RTX 3090; host 37becd33558a; run 20261022-gemma4-12b-b67c82; measured 2026-10-22T06:41:17+00:00; scoring3,3 trials","Hermes Agent Hermes Agent v0.22.2+2363.g9cb34e9 (2026.9.24) · stable · local 9cb34e9e285d; Ollama 0.40.6; MXFP4; ollama-default (harness sets no temperature/seed); actual context32768; hardware NVIDIA GeForce RTX 3090; host 37becd33558a; run 20261022-gpt-oss-20b-6f19a1; measured 2026-10-22T08:18:15+00:00; scoring3,3 trials","Hermes Agent Hermes Agent v0.22.2+2363.g9cb34e9 (2026.9.24) · stable · local 9cb34e9e285d; Ollama 0.40.6; Q4_K_M; ollama-default (harness sets no temperature/seed); actual context32768; hardware NVIDIA GeForce RTX 3090; host b7e2c79acebd; run 20261022-qwen3-vl-8b-de2a43; measured 2026-10-22T22:53:39+00:00; scoring3,3 trials","Hermes Agent Hermes Agent v0.22.2+2363.g9cb34e9 (2026.9.24) · stable · local 9cb34e9e285d; Ollama 0.40.6; Q4_K_M; ollama-default (harness sets no temperature/seed); actual context32768; hardware NVIDIA GeForce RTX 3090; host 37becd33558a; run 20261022-qwen3-14b-c137a5; measured 2026-10-22T02:46:30+00:00; scoring3,3 trials","Hermes Agent Hermes Agent v0.22.2+2363.g9cb34e9 (2026.9.24) · stable · local 9cb34e9e285d; Ollama 0.40.6; Q4_K_M; ollama-default (harness sets no temperature/seed); actual context32768; hardware NVIDIA GeForce RTX 3090; host 37becd33558a; run 20261021-qwen3.5-9b-cfa86e; measured 2026-10-22T00:41:43+00:00; scoring3,3 trials","Hermes Agent Hermes Agent v0.22.2+2363.g9cb34e9 (2026.9.24) · stable · local 9cb34e9e285d; Ollama 0.40.6; Q4_K_M; ollama-default (harness sets no temperature/seed); actual context32768; hardware NVIDIA GeForce RTX 3090; host dc7718586fbe; run 20261021-qwen3-8b-5020e7; measured 2026-10-21T03:34:58+00:00; scoring3,3 trials","Hermes Agent Hermes Agent v0.22.2+2363.g9cb34e9 (2026.9.24) · stable · local 9cb34e9e285d; Ollama 0.40.6; Q4_K_M; ollama-default (harness sets no temperature/seed); actual context32768; hardware NVIDIA GeForce RTX 3090; host dc7718586fbe; run 20261021-mistral-nemo-12b-c42ba9; measured 2026-10-21T06:42:21+00:00; scoring3,3 trials","Hermes Agent Hermes Agent v0.22.2+2363.g9cb34e9 (2026.9.24) · stable · local 9cb34e9e285d; Ollama 0.40.6; Q4_K_M; ollama-default (harness sets no temperature/seed); actual context32768; hardware NVIDIA GeForce RTX 3090; host dc7718586fbe; run 20261021-qwen2.5-14b-bfd007; measured 2026-10-21T08:17:32+00:00; scoring3,3 trials","Hermes Agent Hermes Agent v0.22.2+2363.g9cb34e9 (2026.9.24) · stable · local 9cb34e9e285d; Ollama 0.40.6; Q4_K_M; ollama-default (harness sets no temperature/seed); actual context32768; hardware NVIDIA GeForce RTX 3090; host dc7718586fbe; run 20261020-llama3.1-8b-8c90b4; measured 2026-10-21T00:34:18+00:00; scoring3,3 trials","Hermes Agent Hermes Agent v0.22.2+2363.g9cb34e9 (2026.9.24) · stable · local 9cb34e9e285d; Ollama 0.40.6; Q4_K_M; ollama-default (harness sets no temperature/seed); actual context32768; hardware NVIDIA GeForce RTX 3090; host dc7718586fbe; run 20261021-hermes3-8b-551378; measured 2026-10-21T05:34:32+00:00; scoring3,3 trials","Hermes Agent Hermes Agent v0.22.2+2363.g9cb34e9 (2026.9.24) · stable · local 9cb34e9e285d; Ollama 0.40.6; Q4_K_M; ollama-default (harness sets no temperature/seed); actual context32768; hardware NVIDIA GeForce RTX 3090; host 37becd33558a; run 20261021-deepseek-r1-8b-f9a788; measured 2026-10-21T21:40:21+00:00; scoring3,3 trials","Hermes Agent Hermes Agent v0.22.2+2363.g9cb34e9 (2026.9.24) · stable · local 9cb34e9e285d; Ollama 0.40.6; Q4_K_M; ollama-default (harness sets no temperature/seed); actual context32768; hardware NVIDIA GeForce RTX 3090; host 37becd33558a; run 20261022-deepseek-r1-14b-938625; measured 2026-10-22T04:48:30+00:00; scoring3,3 trials","Hermes Agent Hermes Agent v0.22.2+2363.g9cb34e9 (2026.9.24) · stable · local 9cb34e9e285d; Ollama 0.40.6; Q4_K_M; ollama-default (harness sets no temperature/seed); actual context32768; hardware NVIDIA GeForce RTX 3090; host b7e2c79acebd; run 20261022-qwen2.5-coder-14b-c99868; measured 2026-10-22T23:45:15+00:00; scoring3,3 trials","Hermes Agent Hermes Agent v0.22.2+2363.g9cb34e9 (2026.9.24) · stable · local 9cb34e9e285d; Ollama 0.40.6; Q4_K_M; ollama-default (harness sets no temperature/seed); actual context32768; hardware NVIDIA GeForce RTX 3090; host 37becd33558a; run 20261021-mistral-7b-83c59b; measured 2026-10-21T18:07:31+00:00; scoring3,3 trials","Hermes Agent Hermes Agent v0.22.2+2363.g9cb34e9 (2026.9.24) · stable · local 9cb34e9e285d; Ollama 0.40.6; Q4_K_M; ollama-default (harness sets no temperature/seed); actual context32768; hardware NVIDIA GeForce RTX 3090; host 37becd33558a; run 20261022-gemma4-26b-143678; measured 2026-10-22T09:51:04+00:00; scoring3,3 trials","Hermes Agent Hermes Agent v0.22.2+2363.g9cb34e9 (2026.9.24) · stable · local 9cb34e9e285d; Ollama 0.40.6; Q4_K_M; ollama-default (harness sets no temperature/seed); actual context32768; hardware NVIDIA GeForce RTX 3090; host dc7718586fbe; run 20261021-qwen3-30b-a3b-677da7; measured 2026-10-21T12:04:15+00:00; scoring3,3 trials","Hermes Agent Hermes Agent v0.22.2+2363.g9cb34e9 (2026.9.24) · stable · local 9cb34e9e285d; Ollama 0.40.6; Q4_K_M; ollama-default (harness sets no temperature/seed); actual context32768; hardware NVIDIA GeForce RTX 3090; host b7e2c79acebd; run 20261023-qwen3.6-27b-79610a; measured 2026-10-23T02:03:55+00:00; scoring3,3 trials","Hermes Agent Hermes Agent v0.22.2+2363.g9cb34e9 (2026.9.24) · stable · local 9cb34e9e285d; Ollama 0.40.6; Q4_K_M; ollama-default (harness sets no temperature/seed); actual context32768; hardware NVIDIA GeForce RTX 3090; host 37becd33558a; run 20261022-qwen3.5-27b-d87444; measured 2026-10-22T11:56:01+00:00; scoring3,3 trials","Hermes Agent Hermes Agent v0.22.2+2363.g9cb34e9 (2026.9.24) · stable · local 9cb34e9e285d; Ollama 0.40.6; Q4_K_M; ollama-default (harness sets no temperature/seed); actual context32768; hardware NVIDIA GeForce RTX 3090; host b7e2c79acebd; run 20261023-qwen3.8-27b-25675d; measured 2026-10-23T03:28:09+00:00; scoring3,3 trials","Hermes Agent Hermes Agent v0.22.2+2363.g9cb34e9 (2026.9.24) · stable · local 9cb34e9e285d; Ollama 0.40.6; Q4_K_M; ollama-default (harness sets no temperature/seed); actual context32768; hardware NVIDIA GeForce RTX 3090; host 37becd33558a; run 20261022-qwen3-coder-30b-3a277d; measured 2026-10-22T13:16:30+00:00; scoring3,3 trials","Hermes Agent Hermes Agent v0.22.2+2363.g9cb34e9 (2026.9.24) · stable · local 9cb34e9e285d; Ollama 0.40.6; Q4_K_M; ollama-default (harness sets no temperature/seed); actual context32768; hardware NVIDIA GeForce RTX 3090; host b7e2c79acebd; run 20261022-mistral-small3.2-24b-a1b6e6; measured 2026-10-23T00:47:17+00:00; scoring3,3 trials","Hermes Agent Hermes Agent v0.22.2+2363.g9cb34e9 (2026.9.24) · stable · local 9cb34e9e285d; Ollama 0.40.6; Q4_K_M; ollama-default (harness sets no temperature/seed); actual context32768; hardware NVIDIA GeForce RTX 3090; host dc7718586fbe; run 20261021-mistral-small-24b-1461db; measured 2026-10-21T10:25:37+00:00; scoring3,3 trials","Hermes Agent Hermes Agent v0.22.2+2363.g9cb34e9.dirty (2026.9.24) · stable · local 9cb34e9e285d; Ollama 0.40.6; Q4_K_M; ollama-default (harness sets no temperature/seed); actual context32768; hardware Apple M2 GPU (unified memory); host 8b306c4ac94a; run 20261021-qwen2.5-14b-84e803; measured 2026-10-22T02:01:39+00:00; scoring3,3 trials","Hermes Agent Hermes Agent v0.22.2+2363.g9cb34e9.dirty (2026.9.24) · stable · local 9cb34e9e285d; Ollama 0.40.6; Q4_K_M; ollama-default (harness sets no temperature/seed); actual context32768; hardware Apple M2 GPU (unified memory); host 8b306c4ac94a; run 20261021-qwen3-8b-eae4c3; measured 2026-10-21T16:47:31+00:00; scoring3,3 trials","Hermes Agent Hermes Agent v0.22.2+2363.g9cb34e9.dirty (2026.9.24) · stable · local 9cb34e9e285d; Ollama 0.40.6; Q4_K_M; ollama-default (harness sets no temperature/seed); actual context32768; hardware Apple M2 GPU (unified memory); host 8b306c4ac94a; run 20261021-mistral-nemo-12b-adc3a1; measured 2026-10-21T21:09:13+00:00; scoring3,3 trials","Hermes Agent Hermes Agent v0.22.2+2363.g9cb34e9.dirty (2026.9.24) · stable · local 9cb34e9e285d; Ollama 0.40.6; Q4_K_M; ollama-default (harness sets no temperature/seed); actual context32768; hardware Apple M2 GPU (unified memory); host 8b306c4ac94a; run 20261021-qwen2.5-7b-79d3e8; measured 2026-10-21T05:08:45+00:00; scoring3,3 trials","Hermes Agent Hermes Agent v0.22.2+2363.g9cb34e9.dirty (2026.9.24) · stable · local 9cb34e9e285d; Ollama 0.40.6; Q4_K_M; ollama-default (harness sets no temperature/seed); actual context32768; hardware Apple M2 GPU (unified memory); host 8b306c4ac94a; run 20261021-llama3.2-3b-3941c7; measured 2026-10-21T01:55:23+00:00; scoring3,3 trials","Hermes Agent Hermes Agent v0.22.2+2363.g9cb34e9.dirty (2026.9.24) · stable · local 9cb34e9e285d; Ollama 0.40.6; Q4_K_M; ollama-default (harness sets no temperature/seed); actual context32768; hardware Apple M2 GPU (unified memory); host 8b306c4ac94a; run 20261021-llama3.1-8b-0c49e9; measured 2026-10-21T08:32:28+00:00; scoring3,3 trials","Hermes Agent Hermes Agent v0.22.2+2363.g9cb34e9.dirty (2026.9.24) · stable · local 9cb34e9e285d; Ollama 0.40.6; Q4_K_M; ollama-default (harness sets no temperature/seed); actual context32768; hardware Apple M2 GPU (unified memory); host 8b306c4ac94a; run 20261021-hermes3-8b-4894d4; measured 2026-10-21T19:02:35+00:00; scoring3,3 trials","Hermes Agent Hermes Agent v0.22.2+2363.g9cb34e9.dirty (2026.9.24) · stable · local 9cb34e9e285d; Ollama 0.40.6; Q4_K_M; ollama-default (harness sets no temperature/seed); actual context32768; hardware Apple M2 GPU (unified memory); host 8b306c4ac94a; run 20261021-qwen2.5-coder-7b-f4f1f3; measured 2026-10-21T06:35:23+00:00; scoring3,3 trials","English; direct; GPT5.6Terra calibrated judge; published original150-case batch","https://arxiv.org/html/2610.21789v1#A3.T11","English; correction seeking; GPT5.6Terra calibrated judge; published original150-case batch","Spec; official API default reasoning; 89 fixed annotated tasks","https://arxiv.org/html/2610.22869v1#S5.T1","Spec; official API default reasoning; 77 fixed annotated tasks","Original Fable 5.1-Code; highest exposed reasoning; full101 tasks","https://arxiv.org/html/2610.31711v1#A4.F9","Original Fable 5.1-Cowork; highest exposed reasoning; full101 tasks","Original Astra-Code; highest exposed reasoning; full101 tasks","Original Astra-Work; highest exposed reasoning; full101 tasks","Original GPT-6-Pro; highest exposed reasoning; full101 tasks","Original Opus 5-Cowork; highest exposed reasoning; full101 tasks","Original Opus 5-Code; highest exposed reasoning; full101 tasks","Original Fable 5.1-Excel; highest exposed reasoning; full101 tasks","Original Sol-Excel; highest exposed reasoning; full101 tasks","Original Opus 5-Excel; highest exposed reasoning; full101 tasks","Original Grok 4.6-Code; highest exposed reasoning; full101 tasks","Original Qwen 3.8-Code; highest exposed reasoning; full101 tasks","Original Kimi K3-Code; highest exposed reasoning; full101 tasks","Original Fable 5.1; highest exposed reasoning; full101 tasks","Original Astra; highest exposed reasoning; full101 tasks","Original Gemini 3.8-Code; highest exposed reasoning; full101 tasks","Original Kimi K3; highest exposed reasoning; full101 tasks","Original Qwen 3.8; highest exposed reasoning; full101 tasks","Original Grok 4.6; highest exposed reasoning; full101 tasks","Original Gemini 3.8; highest exposed reasoning; full101 tasks","https://arxiv.org/html/2610.31711v1#S3.F6","Original checkpoint google/gemma-4-E2B-it; reasoning off; source sampling T = 1.0 , p = .95 , k = 64 T=1.0,\\ p=.95,\\ k=64; seed 0; max output 65536; context 131072; frozen1,440 instances","https://arxiv.org/html/2609.38409v1#S5.T2","Original checkpoint google/gemma-4-E2B-it; reasoning on; source sampling T = 1.0 , p = .95 , k = 64 T=1.0,\\ p=.95,\\ k=64; seed 0; max output 65536; context 131072; frozen1,440 instances","Original checkpoint google/gemma-4-E4B-it; reasoning off; source sampling T = 1.0 , p = .95 , k = 64 T=1.0,\\ p=.95,\\ k=64; seed 0; max output 65536; context 131072; frozen1,440 instances","Original checkpoint google/gemma-4-E4B-it; reasoning on; source sampling T = 1.0 , p = .95 , k = 64 T=1.0,\\ p=.95,\\ k=64; seed 0; max output 65536; context 131072; frozen1,440 instances","Original checkpoint google/gemma-4-26B-A4B-it; reasoning off; source sampling T = 1.0 , p = .95 , k = 64 T=1.0,\\ p=.95,\\ k=64; seed 0; max output 65536; context 262144; frozen1,440 instances","Original checkpoint google/gemma-4-26B-A4B-it; reasoning on; source sampling T = 1.0 , p = .95 , k = 64 T=1.0,\\ p=.95,\\ k=64; seed 0; max output 65536; context 262144; frozen1,440 instances","Original checkpoint google/gemma-4-31B-it; reasoning off; source sampling T = 1.0 , p = .95 , k = 64 T=1.0,\\ p=.95,\\ k=64; seed 0; max output 65536; context 262144; frozen1,440 instances","Original checkpoint google/gemma-4-31B-it; reasoning on; source sampling T = 1.0 , p = .95 , k = 64 T=1.0,\\ p=.95,\\ k=64; seed 0; max output 65536; context 262144; frozen1,440 instances","Original checkpoint openai/gpt-oss-20b; reasoning low; source sampling T = 1.0 , p = 1.0 T=1.0,\\ p=1.0; seed 0; max output 65536; context 131072; frozen1,440 instances","Original checkpoint openai/gpt-oss-20b; reasoning medium; source sampling T = 1.0 , p = 1.0 T=1.0,\\ p=1.0; seed 0; max output 65536; context 131072; frozen1,440 instances","Original checkpoint openai/gpt-oss-20b; reasoning high; source sampling T = 1.0 , p = 1.0 T=1.0,\\ p=1.0; seed 0; max output 65536; context 131072; frozen1,440 instances","Original checkpoint openai/gpt-oss-120b; reasoning low; source sampling T = 1.0 , p = 1.0 T=1.0,\\ p=1.0; seed 0; max output 65536; context 131072; frozen1,440 instances","Original checkpoint openai/gpt-oss-120b; reasoning medium; source sampling T = 1.0 , p = 1.0 T=1.0,\\ p=1.0; seed 0; max output 65536; context 131072; frozen1,440 instances","Original checkpoint openai/gpt-oss-120b; reasoning high; source sampling T = 1.0 , p = 1.0 T=1.0,\\ p=1.0; seed 0; max output 65536; context 131072; frozen1,440 instances","Original checkpoint Qwen/Qwen3.5-0.8B; reasoning off; source sampling T = 1.0 , p = 1.0 , k = 20 , PP = 2.0 T=1.0,\\ p=1.0,\\ k=20,\\ \\mathrm{PP}=2.0; seed 0; max output 65536; context 262144; frozen1,440 instances","Original checkpoint Qwen/Qwen3.5-0.8B; reasoning on; source sampling T = 1.0 , p = .95 , k = 20 , PP = 1.5 T=1.0,\\ p=.95,\\ k=20,\\ \\mathrm{PP}=1.5; seed 0; max output 65536; context 262144; frozen1,440 instances","Original checkpoint Qwen/Qwen3.5-2B; reasoning off; source sampling T = 1.0 , p = 1.0 , k = 20 , PP = 2.0 T=1.0,\\ p=1.0,\\ k=20,\\ \\mathrm{PP}=2.0; seed 0; max output 65536; context 262144; frozen1,440 instances","Original checkpoint Qwen/Qwen3.5-2B; reasoning on; source sampling T = 1.0 , p = .95 , k = 20 , PP = 1.5 T=1.0,\\ p=.95,\\ k=20,\\ \\mathrm{PP}=1.5; seed 0; max output 65536; context 262144; frozen1,440 instances","Original checkpoint Qwen/Qwen3.5-4B; reasoning off; source sampling T = 1.0 , p = .95 , k = 20 , PP = 1.5 T=1.0,\\ p=.95,\\ k=20,\\ \\mathrm{PP}=1.5; seed 0; max output 65536; context 262144; frozen1,440 instances","Original checkpoint Qwen/Qwen3.5-4B; reasoning on; source sampling T = 1.0 , p = .95 , k = 20 , PP = 1.5 T=1.0,\\ p=.95,\\ k=20,\\ \\mathrm{PP}=1.5; seed 0; max output 65536; context 262144; frozen1,440 instances","Original checkpoint Qwen/Qwen3.5-9B; reasoning off; source sampling T = 1.0 , p = .95 , k = 20 , PP = 1.5 T=1.0,\\ p=.95,\\ k=20,\\ \\mathrm{PP}=1.5; seed 0; max output 65536; context 262144; frozen1,440 instances","Original checkpoint Qwen/Qwen3.5-9B; reasoning on; source sampling T = 1.0 , p = .95 , k = 20 , PP = 1.5 T=1.0,\\ p=.95,\\ k=20,\\ \\mathrm{PP}=1.5; seed 0; max output 65536; context 262144; frozen1,440 instances","Original checkpoint Qwen/Qwen3.5-35B-A3B; reasoning off; source sampling T = 1.0 , p = .95 , k = 20 , PP = 1.5 T=1.0,\\ p=.95,\\ k=20,\\ \\mathrm{PP}=1.5; seed 0; max output 65536; context 262144; frozen1,440 instances","Original checkpoint Qwen/Qwen3.5-35B-A3B; reasoning on; source sampling T = 1.0 , p = .95 , k = 20 , PP = 1.5 T=1.0,\\ p=.95,\\ k=20,\\ \\mathrm{PP}=1.5; seed 0; max output 65536; context 262144; frozen1,440 instances","Original checkpoint Qwen/Qwen3.5-27B; reasoning off; source sampling T = 1.0 , p = .95 , k = 20 , PP = 1.5 T=1.0,\\ p=.95,\\ k=20,\\ \\mathrm{PP}=1.5; seed 0; max output 65536; context 262144; frozen1,440 instances","Original checkpoint Qwen/Qwen3.5-27B; reasoning on; source sampling T = 1.0 , p = .95 , k = 20 , PP = 1.5 T=1.0,\\ p=.95,\\ k=20,\\ \\mathrm{PP}=1.5; seed 0; max output 65536; context 262144; frozen1,440 instances","Original checkpoint Qwen/Qwen3.8-27B; reasoning low; source sampling T = 1.0 , p = .95 , k = 20 , PP = 0 T=1.0,\\ p=.95,\\ k=20,\\ \\mathrm{PP}=0; seed 0; max output 65536; context 131072; frozen1,440 instances","Original checkpoint Qwen/Qwen3.8-27B; reasoning medium; source sampling T = 1.0 , p = .95 , k = 20 , PP = 0 T=1.0,\\ p=.95,\\ k=20,\\ \\mathrm{PP}=0; seed 0; max output 65536; context 131072; frozen1,440 instances","Original checkpoint Qwen/Qwen3.8-27B; reasoning xhigh; source sampling T = 1.0 , p = .95 , k = 20 , PP = 0 T=1.0,\\ p=.95,\\ k=20,\\ \\mathrm{PP}=0; seed 0; max output 65536; context 131072; frozen1,440 instances","Original checkpoint deepseek-v4.1-flash; reasoning high; source sampling T = 1.0 , p = .95 T=1.0,\\ p=.95; seed –; max output 65536; context –; frozen1,440 instances","Original checkpoint gemini-2.5-pro; reasoning low; source sampling provider defaults; seed –; max output 65536; context –; frozen1,440 instances","Original checkpoint gemini-2.5-pro; reasoning medium; source sampling provider defaults; seed –; max output 65536; context –; frozen1,440 instances","Original checkpoint gemini-2.5-pro; reasoning high; source sampling provider defaults; seed –; max output 65536; context –; frozen1,440 instances","Original checkpoint gemini-3.5-flash-lite; reasoning low; source sampling provider defaults; seed –; max output 65536; context –; frozen1,440 instances","Original checkpoint gemini-3.5-flash-lite; reasoning medium; source sampling provider defaults; seed –; max output 65536; context –; frozen1,440 instances","Original checkpoint gemini-3.5-flash-lite; reasoning high; source sampling provider defaults; seed –; max output 65536; context –; frozen1,440 instances","Original checkpoint gemini-3.1-pro-preview; reasoning low; source sampling provider defaults; seed –; max output 65536; context –; frozen1,440 instances","Original checkpoint gemini-3.1-pro-preview; reasoning medium; source sampling provider defaults; seed –; max output 65536; context –; frozen1,440 instances","Original checkpoint gemini-3.1-pro-preview; reasoning high; source sampling provider defaults; seed –; max output 65536; context –; frozen1,440 instances","Original checkpoint gemini-3.8-flash; reasoning low; source sampling provider defaults; seed –; max output 65536; context –; frozen1,440 instances","Original checkpoint gemini-3.8-flash; reasoning medium; source sampling provider defaults; seed –; max output 65536; context –; frozen1,440 instances","Original checkpoint gemini-3.8-flash; reasoning high; source sampling provider defaults; seed –; max output 65536; context –; frozen1,440 instances","Original Sentinel; checkpoint Qwen/Qwen3-Coder-30B-A3B-Instruct with Sentinel LoRA r16/alpha32; mini-swe-agent; same359 PR-disjoint test instances; source judgment-or-correct-repair acceptance rule","https://arxiv.org/html/2609.37216v1#S4.T2","Original GLM-5.3; checkpoint GLM-5.3; mini-swe-agent; same359 PR-disjoint test instances; source judgment-or-correct-repair acceptance rule","Original kimi-k3; checkpoint kimi-k3; mini-swe-agent; same359 PR-disjoint test instances; source judgment-or-correct-repair acceptance rule","Original Claude-Opus-5; checkpoint Claude-Opus-5; mini-swe-agent; same359 PR-disjoint test instances; source judgment-or-correct-repair acceptance rule","Original DeepSeek-V4.1-Flash; checkpoint DeepSeek-V4.1-Flash; mini-swe-agent; same359 PR-disjoint test instances; source judgment-or-correct-repair acceptance rule","Original GPT-5.5; checkpoint GPT-5.5; mini-swe-agent; same359 PR-disjoint test instances; source judgment-or-correct-repair acceptance rule","Original Qwen3-Coder-30B-A3B-Instruct; checkpoint Qwen3-Coder-30B-A3B-Instruct; mini-swe-agent; same359 PR-disjoint test instances; source judgment-or-correct-repair acceptance rule","Original checkpoint Claude Opus 4.7; Guided oracle schema+gold history;3 actions per turn; 721 sessions/5,927 turns; original source inference protocol","https://arxiv.org/html/2609.36987v1#S4.T2","Original checkpoint GPT-5.5; Guided oracle schema+gold history;3 actions per turn; 721 sessions/5,927 turns; original source inference protocol","Original checkpoint Kimi-K2.5; Guided oracle schema+gold history;3 actions per turn; 721 sessions/5,927 turns; original source inference protocol","Original checkpoint Gemini-3.1-Flash-Lite; Guided oracle schema+gold history;3 actions per turn; 721 sessions/5,927 turns; original source inference protocol","Original checkpoint Qwen3-235B; Guided oracle schema+gold history;3 actions per turn; 721 sessions/5,927 turns; original source inference protocol","Original checkpoint GLM-5; Guided oracle schema+gold history;3 actions per turn; 721 sessions/5,927 turns; original source inference protocol","Original checkpoint DeepSeek-V3.2; Guided oracle schema+gold history;3 actions per turn; 721 sessions/5,927 turns; original source inference protocol","ERNIE-5.0","Original checkpoint ERNIE-5.0; Guided oracle schema+gold history;3 actions per turn; 721 sessions/5,927 turns; original source inference protocol","Original checkpoint MiMo-V2.5-Pro; Guided oracle schema+gold history;3 actions per turn; 721 sessions/5,927 turns; original source inference protocol","Original checkpoint MiniMax-M2.7; Guided oracle schema+gold history;3 actions per turn; 721 sessions/5,927 turns; original source inference protocol","Original checkpoint Llama-3.1-8B; Guided oracle schema+gold history;3 actions per turn; 721 sessions/5,927 turns; original source inference protocol","Original checkpoint Gemma-2-9B; Guided oracle schema+gold history;3 actions per turn; 721 sessions/5,927 turns; original source inference protocol","Original checkpoint STRuCT-LLM-Novo; Guided oracle schema+gold history;3 actions per turn; 721 sessions/5,927 turns; original source inference protocol","Original checkpoint text-to-cypher-gemma; Guided oracle schema+gold history;3 actions per turn; 721 sessions/5,927 turns; original source inference protocol","CypherRI-7B","Original checkpoint CypherRI-7B †; Guided oracle schema+gold history;3 actions per turn; 721 sessions/5,927 turns; CypherRI FIM-token contamination reported by source; not included in source aggregate statistics","Original checkpoint Claude Opus 4.7; Agentic discovery+self-history; shared session budget3×turn count; 721 sessions/5,927 turns; original source inference protocol","Original checkpoint GPT-5.5; Agentic discovery+self-history; shared session budget3×turn count; 721 sessions/5,927 turns; original source inference protocol","Original checkpoint Kimi-K2.5; Agentic discovery+self-history; shared session budget3×turn count; 721 sessions/5,927 turns; original source inference protocol","Original checkpoint Gemini-3.1-Flash-Lite; Agentic discovery+self-history; shared session budget3×turn count; 721 sessions/5,927 turns; original source inference protocol","Original checkpoint Qwen3-235B; Agentic discovery+self-history; shared session budget3×turn count; 721 sessions/5,927 turns; original source inference protocol","Original checkpoint GLM-5; Agentic discovery+self-history; shared session budget3×turn count; 721 sessions/5,927 turns; original source inference protocol","Original checkpoint DeepSeek-V3.2; Agentic discovery+self-history; shared session budget3×turn count; 721 sessions/5,927 turns; original source inference protocol","Original checkpoint ERNIE-5.0; Agentic discovery+self-history; shared session budget3×turn count; 721 sessions/5,927 turns; original source inference protocol","Original checkpoint MiMo-V2.5-Pro; Agentic discovery+self-history; shared session budget3×turn count; 721 sessions/5,927 turns; original source inference protocol","Original checkpoint MiniMax-M2.7; Agentic discovery+self-history; shared session budget3×turn count; 721 sessions/5,927 turns; original source inference protocol","Original checkpoint Llama-3.1-8B; Agentic discovery+self-history; shared session budget3×turn count; 721 sessions/5,927 turns; original source inference protocol","Original checkpoint Gemma-2-9B; Agentic discovery+self-history; shared session budget3×turn count; 721 sessions/5,927 turns; original source inference protocol","Original checkpoint STRuCT-LLM-Novo; Agentic discovery+self-history; shared session budget3×turn count; 721 sessions/5,927 turns; original source inference protocol","Original checkpoint text-to-cypher-gemma; Agentic discovery+self-history; shared session budget3×turn count; 721 sessions/5,927 turns; original source inference protocol","Original checkpoint CypherRI-7B †; Agentic discovery+self-history; shared session budget3×turn count; 721 sessions/5,927 turns; CypherRI FIM-token contamination reported by source; not included in source aggregate statistics","Original Claude Opus 5; reasoning Max; zero-shot native engineering harness; common300 tasks/152CLI+148GUI; most recent15 rounds; GUI200/CLI100 decision limits,300/150 for multi-software and open-ended","https://arxiv.org/html/2609.37686v1#S4.T2","Original GPT-5.6 Sol; reasoning XHigh; zero-shot native engineering harness; common300 tasks/152CLI+148GUI; most recent15 rounds; GUI200/CLI100 decision limits,300/150 for multi-software and open-ended","Original Qwen3.8 Max; reasoning XHigh; zero-shot native engineering harness; common300 tasks/152CLI+148GUI; most recent15 rounds; GUI200/CLI100 decision limits,300/150 for multi-software and open-ended","Original DeepSeek V4.1 Flash; reasoning Max; zero-shot native engineering harness; common300 tasks/152CLI+148GUI; most recent15 rounds; GUI200/CLI100 decision limits,300/150 for multi-software and open-ended","Original Gemini 3.7 Flash; reasoning High; zero-shot native engineering harness; common300 tasks/152CLI+148GUI; most recent15 rounds; GUI200/CLI100 decision limits,300/150 for multi-software and open-ended","Original Kimi K3; reasoning Max; zero-shot native engineering harness; common300 tasks/152CLI+148GUI; most recent15 rounds; GUI200/CLI100 decision limits,300/150 for multi-software and open-ended","Original Qwen3.8 Flash; reasoning XHigh; zero-shot native engineering harness; common300 tasks/152CLI+148GUI; most recent15 rounds; GUI200/CLI100 decision limits,300/150 for multi-software and open-ended","Original checkpoint π 0.5 \\pi_{0.5}; PaliGemma with a flow-matching action expert; heterogeneous robot, semantic, and web pretraining followed by LIBERO adaptation ( Black et al., 2025a ) .; native H=50, Q=5; max base; fixed8000 assigned cases","https://arxiv.org/html/2609.36518v1#S4.T2","Original checkpoint OpenVLA-OFT; Prismatic OpenVLA with continuous regression, parallel decoding, proprioception, and wrist images; combined checkpoint LoRA-optimized on the four standard LIBERO suites ( Kim et al., 2024 ; Kim et al., 2025 ) .; native H=8, Q=8; max base; fixed8000 assigned cases","Original checkpoint X-VLA; Cross-embodiment VLA with embodiment-specific soft prompts and a flow-matching decoder; released LIBERO checkpoint without MAX-specific adaptation ( Zheng et al., 2026 ) .; native H=30, Q=30; max base; fixed8000 assigned cases","Original checkpoint Xiaomi-Robotics-0; Cross-embodiment and vision-language pretraining with asynchronous action-chunk deployment; released LIBERO checkpoint with its native query cadence ( Cai et al., 2026 ) .; native H=30, Q=10; max base; fixed8000 assigned cases","Original checkpoint MolmoAct2; Embodied-reasoning VLM with a flow-matching action expert conditioned through per-layer key-value caches; released LIBERO policy with continuous action decoding ( Fang et al., 2026a ) .; native H=10, Q=10; max base; fixed8000 assigned cases","Original checkpoint SmolVLA; SmolVLM2 with flow-matching action generation and asynchronous robot-data training; released HuggingFaceVLA/smolvla_libero without MAX-specific adaptation ( Shukor et al., 2025 ) .; native H=50, Q=10; max base; fixed8000 assigned cases","Original checkpoint GR00T N1.7; Cosmos-Reason2-2B with proprioception and a flow-matching action transformer; NVIDIA’s official LIBERO post-trained checkpoint without MAX-specific adaptation ( Bjorck et al., 2025 ) .; native H=40, Q=8; max base; fixed8000 assigned cases","Original checkpoint DM0.5; Dexmal’s multimodal flow-matching VLA for open-world control; official DM05-libero checkpoint without MAX-specific adaptation.; native H=10, Q=10; max base; fixed8000 assigned cases","Original checkpoint VLA-JEPA; Qwen3-VL-2B with a V-JEPA2 teacher and latent future-state alignment; human video and single-embodiment robot pretraining followed by end-to-end LIBERO adaptation ( Sun et al., 2026a ; Assran et al., 2025 ) .; native H=7, Q=7; max base; fixed8000 assigned cases","Original checkpoint Cosmos-Policy; Cosmos-Predict2-2B adapted without a separate policy backbone; single-stage post-training on target LIBERO demonstrations ( Agarwal et al., 2025 ; Kim et al., 2026 ) .; native H=16, Q=16; max base; fixed8000 assigned cases","Original checkpoint Fast-WAM; Wan2.2-TI2V-5B plus ActionDiT, jointly trained on video and actions from clean LIBERO demonstrations; inference omits future video generation ( Yuan et al., 2026 ) .; native H=32, Q=16; max base; fixed8000 assigned cases","HiMem-WAM","Original checkpoint HiMem-WAM; Low-level motion and high-level skill latents with boundary-triggered task memory; released LIBERO checkpoint with native memory-gated inference ( Sun et al., 2026b ) .; native H=32, Q=10; max base; fixed8000 assigned cases","Light-WAM","Original checkpoint Light-WAM; Compact video backbone with latent future-video supervision and state-fusion action decoding; released LIBERO checkpoint with its native query cadence ( Li et al., 2026b ) .; native H=32, Q=10; max base; fixed8000 assigned cases","DiT4DiT","Original checkpoint DiT4DiT; Coupled video-dynamics and action diffusion transformers with dual flow-matching objectives; released LIBERO checkpoint without MAX-specific adaptation ( Ma et al., 2026 ) .; native H=8, Q=8; max base; fixed8000 assigned cases","Original checkpoint π 0.5 \\pi_{0.5}; PaliGemma with a flow-matching action expert; heterogeneous robot, semantic, and web pretraining followed by LIBERO adaptation ( Black et al., 2025a ) .; native H=50, Q=5; max dynamic; fixed8000 assigned cases","Original checkpoint OpenVLA-OFT; Prismatic OpenVLA with continuous regression, parallel decoding, proprioception, and wrist images; combined checkpoint LoRA-optimized on the four standard LIBERO suites ( Kim et al., 2024 ; Kim et al., 2025 ) .; native H=8, Q=8; max dynamic; fixed8000 assigned cases","Original checkpoint X-VLA; Cross-embodiment VLA with embodiment-specific soft prompts and a flow-matching decoder; released LIBERO checkpoint without MAX-specific adaptation ( Zheng et al., 2026 ) .; native H=30, Q=30; max dynamic; fixed8000 assigned cases","Original checkpoint Xiaomi-Robotics-0; Cross-embodiment and vision-language pretraining with asynchronous action-chunk deployment; released LIBERO checkpoint with its native query cadence ( Cai et al., 2026 ) .; native H=30, Q=10; max dynamic; fixed8000 assigned cases","Original checkpoint MolmoAct2; Embodied-reasoning VLM with a flow-matching action expert conditioned through per-layer key-value caches; released LIBERO policy with continuous action decoding ( Fang et al., 2026a ) .; native H=10, Q=10; max dynamic; fixed8000 assigned cases","Original checkpoint SmolVLA; SmolVLM2 with flow-matching action generation and asynchronous robot-data training; released HuggingFaceVLA/smolvla_libero without MAX-specific adaptation ( Shukor et al., 2025 ) .; native H=50, Q=10; max dynamic; fixed8000 assigned cases","Original checkpoint GR00T N1.7; Cosmos-Reason2-2B with proprioception and a flow-matching action transformer; NVIDIA’s official LIBERO post-trained checkpoint without MAX-specific adaptation ( Bjorck et al., 2025 ) .; native H=40, Q=8; max dynamic; fixed8000 assigned cases","Original checkpoint DM0.5; Dexmal’s multimodal flow-matching VLA for open-world control; official DM05-libero checkpoint without MAX-specific adaptation.; native H=10, Q=10; max dynamic; fixed8000 assigned cases","Original checkpoint VLA-JEPA; Qwen3-VL-2B with a V-JEPA2 teacher and latent future-state alignment; human video and single-embodiment robot pretraining followed by end-to-end LIBERO adaptation ( Sun et al., 2026a ; Assran et al., 2025 ) .; native H=7, Q=7; max dynamic; fixed8000 assigned cases","Original checkpoint Cosmos-Policy; Cosmos-Predict2-2B adapted without a separate policy backbone; single-stage post-training on target LIBERO demonstrations ( Agarwal et al., 2025 ; Kim et al., 2026 ) .; native H=16, Q=16; max dynamic; fixed8000 assigned cases","Original checkpoint Fast-WAM; Wan2.2-TI2V-5B plus ActionDiT, jointly trained on video and actions from clean LIBERO demonstrations; inference omits future video generation ( Yuan et al., 2026 ) .; native H=32, Q=16; max dynamic; fixed8000 assigned cases","Original checkpoint HiMem-WAM; Low-level motion and high-level skill latents with boundary-triggered task memory; released LIBERO checkpoint with native memory-gated inference ( Sun et al., 2026b ) .; native H=32, Q=10; max dynamic; fixed8000 assigned cases","Original checkpoint Light-WAM; Compact video backbone with latent future-video supervision and state-fusion action decoding; released LIBERO checkpoint with its native query cadence ( Li et al., 2026b ) .; native H=32, Q=10; max dynamic; fixed8000 assigned cases","Original checkpoint DiT4DiT; Coupled video-dynamics and action diffusion transformers with dual flow-matching objectives; released LIBERO checkpoint without MAX-specific adaptation ( Ma et al., 2026 ) .; native H=8, Q=8; max dynamic; fixed8000 assigned cases","Original checkpoint π 0.5 \\pi_{0.5}; PaliGemma with a flow-matching action expert; heterogeneous robot, semantic, and web pretraining followed by LIBERO adaptation ( Black et al., 2025a ) .; native H=50, Q=5; lite base; fixed800 assigned cases","https://arxiv.org/html/2609.36518v1#A3.T7","Original checkpoint OpenVLA-OFT; Prismatic OpenVLA with continuous regression, parallel decoding, proprioception, and wrist images; combined checkpoint LoRA-optimized on the four standard LIBERO suites ( Kim et al., 2024 ; Kim et al., 2025 ) .; native H=8, Q=8; lite base; fixed800 assigned cases","Original checkpoint X-VLA; Cross-embodiment VLA with embodiment-specific soft prompts and a flow-matching decoder; released LIBERO checkpoint without MAX-specific adaptation ( Zheng et al., 2026 ) .; native H=30, Q=30; lite base; fixed800 assigned cases","Original checkpoint Xiaomi-Robotics-0; Cross-embodiment and vision-language pretraining with asynchronous action-chunk deployment; released LIBERO checkpoint with its native query cadence ( Cai et al., 2026 ) .; native H=30, Q=10; lite base; fixed800 assigned cases","Original checkpoint MolmoAct2; Embodied-reasoning VLM with a flow-matching action expert conditioned through per-layer key-value caches; released LIBERO policy with continuous action decoding ( Fang et al., 2026a ) .; native H=10, Q=10; lite base; fixed800 assigned cases","Original checkpoint SmolVLA; SmolVLM2 with flow-matching action generation and asynchronous robot-data training; released HuggingFaceVLA/smolvla_libero without MAX-specific adaptation ( Shukor et al., 2025 ) .; native H=50, Q=10; lite base; fixed800 assigned cases","Original checkpoint GR00T N1.7; Cosmos-Reason2-2B with proprioception and a flow-matching action transformer; NVIDIA’s official LIBERO post-trained checkpoint without MAX-specific adaptation ( Bjorck et al., 2025 ) .; native H=40, Q=8; lite base; fixed800 assigned cases","Original checkpoint DM0.5; Dexmal’s multimodal flow-matching VLA for open-world control; official DM05-libero checkpoint without MAX-specific adaptation.; native H=10, Q=10; lite base; fixed800 assigned cases","Original checkpoint VLA-JEPA; Qwen3-VL-2B with a V-JEPA2 teacher and latent future-state alignment; human video and single-embodiment robot pretraining followed by end-to-end LIBERO adaptation ( Sun et al., 2026a ; Assran et al., 2025 ) .; native H=7, Q=7; lite base; fixed800 assigned cases","Original checkpoint Cosmos-Policy; Cosmos-Predict2-2B adapted without a separate policy backbone; single-stage post-training on target LIBERO demonstrations ( Agarwal et al., 2025 ; Kim et al., 2026 ) .; native H=16, Q=16; lite base; fixed800 assigned cases","Original checkpoint Fast-WAM; Wan2.2-TI2V-5B plus ActionDiT, jointly trained on video and actions from clean LIBERO demonstrations; inference omits future video generation ( Yuan et al., 2026 ) .; native H=32, Q=16; lite base; fixed800 assigned cases","Original checkpoint HiMem-WAM; Low-level motion and high-level skill latents with boundary-triggered task memory; released LIBERO checkpoint with native memory-gated inference ( Sun et al., 2026b ) .; native H=32, Q=10; lite base; fixed800 assigned cases","Original checkpoint Light-WAM; Compact video backbone with latent future-video supervision and state-fusion action decoding; released LIBERO checkpoint with its native query cadence ( Li et al., 2026b ) .; native H=32, Q=10; lite base; fixed800 assigned cases","Original checkpoint DiT4DiT; Coupled video-dynamics and action diffusion transformers with dual flow-matching objectives; released LIBERO checkpoint without MAX-specific adaptation ( Ma et al., 2026 ) .; native H=8, Q=8; lite base; fixed800 assigned cases","Original checkpoint π 0.5 \\pi_{0.5}; PaliGemma with a flow-matching action expert; heterogeneous robot, semantic, and web pretraining followed by LIBERO adaptation ( Black et al., 2025a ) .; native H=50, Q=5; lite dynamic; fixed800 assigned cases","Original checkpoint OpenVLA-OFT; Prismatic OpenVLA with continuous regression, parallel decoding, proprioception, and wrist images; combined checkpoint LoRA-optimized on the four standard LIBERO suites ( Kim et al., 2024 ; Kim et al., 2025 ) .; native H=8, Q=8; lite dynamic; fixed800 assigned cases","Original checkpoint X-VLA; Cross-embodiment VLA with embodiment-specific soft prompts and a flow-matching decoder; released LIBERO checkpoint without MAX-specific adaptation ( Zheng et al., 2026 ) .; native H=30, Q=30; lite dynamic; fixed800 assigned cases","Original checkpoint Xiaomi-Robotics-0; Cross-embodiment and vision-language pretraining with asynchronous action-chunk deployment; released LIBERO checkpoint with its native query cadence ( Cai et al., 2026 ) .; native H=30, Q=10; lite dynamic; fixed800 assigned cases","Original checkpoint MolmoAct2; Embodied-reasoning VLM with a flow-matching action expert conditioned through per-layer key-value caches; released LIBERO policy with continuous action decoding ( Fang et al., 2026a ) .; native H=10, Q=10; lite dynamic; fixed800 assigned cases","Original checkpoint SmolVLA; SmolVLM2 with flow-matching action generation and asynchronous robot-data training; released HuggingFaceVLA/smolvla_libero without MAX-specific adaptation ( Shukor et al., 2025 ) .; native H=50, Q=10; lite dynamic; fixed800 assigned cases","Original checkpoint GR00T N1.7; Cosmos-Reason2-2B with proprioception and a flow-matching action transformer; NVIDIA’s official LIBERO post-trained checkpoint without MAX-specific adaptation ( Bjorck et al., 2025 ) .; native H=40, Q=8; lite dynamic; fixed800 assigned cases","Original checkpoint DM0.5; Dexmal’s multimodal flow-matching VLA for open-world control; official DM05-libero checkpoint without MAX-specific adaptation.; native H=10, Q=10; lite dynamic; fixed800 assigned cases","Original checkpoint VLA-JEPA; Qwen3-VL-2B with a V-JEPA2 teacher and latent future-state alignment; human video and single-embodiment robot pretraining followed by end-to-end LIBERO adaptation ( Sun et al., 2026a ; Assran et al., 2025 ) .; native H=7, Q=7; lite dynamic; fixed800 assigned cases","Original checkpoint Cosmos-Policy; Cosmos-Predict2-2B adapted without a separate policy backbone; single-stage post-training on target LIBERO demonstrations ( Agarwal et al., 2025 ; Kim et al., 2026 ) .; native H=16, Q=16; lite dynamic; fixed800 assigned cases","Original checkpoint Fast-WAM; Wan2.2-TI2V-5B plus ActionDiT, jointly trained on video and actions from clean LIBERO demonstrations; inference omits future video generation ( Yuan et al., 2026 ) .; native H=32, Q=16; lite dynamic; fixed800 assigned cases","Original checkpoint HiMem-WAM; Low-level motion and high-level skill latents with boundary-triggered task memory; released LIBERO checkpoint with native memory-gated inference ( Sun et al., 2026b ) .; native H=32, Q=10; lite dynamic; fixed800 assigned cases","Original checkpoint Light-WAM; Compact video backbone with latent future-video supervision and state-fusion action decoding; released LIBERO checkpoint with its native query cadence ( Li et al., 2026b ) .; native H=32, Q=10; lite dynamic; fixed800 assigned cases","Original checkpoint DiT4DiT; Coupled video-dynamics and action diffusion transformers with dual flow-matching objectives; released LIBERO checkpoint without MAX-specific adaptation ( Ma et al., 2026 ) .; native H=8, Q=8; lite dynamic; fixed800 assigned cases","Original checkpoint label: Doubao seed-1-8; NaviAgent main condition; screenshot-only; Avantage;50 steps,T1.0,top-p0.9; API","https://arxiv.org/html/2609.37053v1#Sx4.T2","Original checkpoint label: Kimi k2.5; NaviAgent main condition; screenshot-only; Avantage;50 steps,T1.0,top-p0.9; API","Original checkpoint label: Claude sonnet-4.6; NaviAgent main condition; screenshot-only; Avantage;50 steps,T1.0,top-p0.9; API","Original checkpoint label: GPT 5.4; NaviAgent main condition; screenshot-only; Avantage;50 steps,T1.0,top-p0.9; API","Original checkpoint label: Qwen3-VL 235B; NaviAgent main condition; screenshot-only; Avantage;50 steps,T1.0,top-p0.9; API","Original checkpoint label: Qwen3-VL 32B; NaviAgent main condition; screenshot-only; Avantage;50 steps,T1.0,top-p0.9; API","Original checkpoint label: Qwen3-VL 8B; NaviAgent main condition; screenshot-only; Avantage;50 steps,T1.0,top-p0.9; API","Original checkpoint label: Doubao seed-1-8; NaviAgent main condition; screenshot-only; DigitalMicrograph;50 steps,T1.0,top-p0.9; API","Original checkpoint label: Kimi k2.5; NaviAgent main condition; screenshot-only; DigitalMicrograph;50 steps,T1.0,top-p0.9; API","Original checkpoint label: Claude sonnet-4.6; NaviAgent main condition; screenshot-only; DigitalMicrograph;50 steps,T1.0,top-p0.9; API","Original checkpoint label: GPT 5.4; NaviAgent main condition; screenshot-only; DigitalMicrograph;50 steps,T1.0,top-p0.9; API","Original checkpoint label: Qwen3-VL 235B; NaviAgent main condition; screenshot-only; DigitalMicrograph;50 steps,T1.0,top-p0.9; API","Original checkpoint label: Qwen3-VL 32B; NaviAgent main condition; screenshot-only; DigitalMicrograph;50 steps,T1.0,top-p0.9; API","Original checkpoint label: Qwen3-VL 8B; NaviAgent main condition; screenshot-only; DigitalMicrograph;50 steps,T1.0,top-p0.9; API","Original checkpoint label: Doubao seed-1-8; NaviAgent main condition; screenshot-only; Materials Studio;50 steps,T1.0,top-p0.9; API","Original checkpoint label: Kimi k2.5; NaviAgent main condition; screenshot-only; Materials Studio;50 steps,T1.0,top-p0.9; API","Original checkpoint label: Claude sonnet-4.6; NaviAgent main condition; screenshot-only; Materials Studio;50 steps,T1.0,top-p0.9; API","Original checkpoint label: GPT 5.4; NaviAgent main condition; screenshot-only; Materials Studio;50 steps,T1.0,top-p0.9; API","Original checkpoint label: Qwen3-VL 235B; NaviAgent main condition; screenshot-only; Materials Studio;50 steps,T1.0,top-p0.9; API","Original checkpoint label: Qwen3-VL 32B; NaviAgent main condition; screenshot-only; Materials Studio;50 steps,T1.0,top-p0.9; API","Original checkpoint label: Qwen3-VL 8B; NaviAgent main condition; screenshot-only; Materials Studio;50 steps,T1.0,top-p0.9; API","Original checkpoint label: Doubao seed-1-8; NaviAgent main condition; screenshot-only; VESTA;50 steps,T1.0,top-p0.9; API","Original checkpoint label: Kimi k2.5; NaviAgent main condition; screenshot-only; VESTA;50 steps,T1.0,top-p0.9; API","Original checkpoint label: Claude sonnet-4.6; NaviAgent main condition; screenshot-only; VESTA;50 steps,T1.0,top-p0.9; API","Original checkpoint label: GPT 5.4; NaviAgent main condition; screenshot-only; VESTA;50 steps,T1.0,top-p0.9; API","Original checkpoint label: Qwen3-VL 235B; NaviAgent main condition; screenshot-only; VESTA;50 steps,T1.0,top-p0.9; API","Original checkpoint label: Qwen3-VL 32B; NaviAgent main condition; screenshot-only; VESTA;50 steps,T1.0,top-p0.9; API","Original checkpoint label: Qwen3-VL 8B; NaviAgent main condition; screenshot-only; VESTA;50 steps,T1.0,top-p0.9; API","Original GPT-5.6-sol; controlled six-option MCQ; published full6077 items; published released checkpoint/API","https://arxiv.org/html/2609.37773v1#A1.T34","Original Grok-4.6; controlled six-option MCQ; published full6077 items; published released checkpoint/API","Original Claude-Sonnet-4-5; controlled six-option MCQ; published full6077 items; published released checkpoint/API","Original GPT-5.6-luna; controlled six-option MCQ; published full6077 items; published released checkpoint/API","Original Qwen3-VL-8B-SFT; controlled six-option MCQ; published full6077 items; authors LoRA SFT, same8B base","Original Qwen3-VL-8B; controlled six-option MCQ; published full6077 items; published released checkpoint/API","Original Qwen3-VL-4B; controlled six-option MCQ; published full6077 items; published released checkpoint/API","Original InternVL3.5-8B; controlled six-option MCQ; published full6077 items; published released checkpoint/API","Original InternVL3.5-4B; controlled six-option MCQ; published full6077 items; published released checkpoint/API","Original Qwen2.5-VL-7B; controlled six-option MCQ; published full6077 items; published released checkpoint/API","Original LLaVA-OneVision-7B; controlled six-option MCQ; published full6077 items; published released checkpoint/API","Original Qwen2.5-VL-3B; controlled six-option MCQ; published full6077 items; published released checkpoint/API","Original Qwen3-VL-2B; controlled six-option MCQ; published full6077 items; published released checkpoint/API","Original InternVL3.5-2B; controlled six-option MCQ; published full6077 items; published released checkpoint/API","Original SmolVLM2-2.2B; controlled six-option MCQ; published full6077 items; published released checkpoint/API","LLaVA-Med-Mistral-7B","Original LLaVA-Med-Mistral-7B; controlled six-option MCQ; published full6077 items; published released checkpoint/API","Original GPT-6-Astra; size –; published default mode; zero-shot current action; full5407 questions","https://arxiv.org/html/2609.36605v1#S4.T3","RynnBrain1.1-122B-A10B","Original RynnBrain1.1-122B-A10B; size 122B; published default mode; zero-shot current action; full5407 questions","Original Doubao-Seed-2.0-Lite; size –; published default mode; zero-shot current action; full5407 questions","RynnBrain1.1-9B","Original RynnBrain1.1-9B; size 9B; published default mode; zero-shot current action; full5407 questions","Original Qwen3-VL-235B-A22B; size –; published default mode; zero-shot current action; full5407 questions","Original Qwen3-VL-32B; size 32B; published default mode; zero-shot current action; full5407 questions","Original Qwen3-VL-30B-A3B; size 30B-A3B; published default mode; zero-shot current action; full5407 questions","RynnBrain1.1-2B","Original RynnBrain1.1-2B; size 2B; published default mode; zero-shot current action; full5407 questions","Original Qwen3-VL-8B; size 8B; published default mode; zero-shot current action; full5407 questions","Original Qwen3-VL-8B-Thinking; size 8B; Thinking mode; zero-shot current action; full5407 questions","Original MiMo-VL-7B-RL; size 7B; published default mode; zero-shot current action; full5407 questions","RynnBrain-2B","Original RynnBrain-2B; size 2B; published default mode; zero-shot current action; full5407 questions","Original Cosmos-Reason2-2B; size 2B; published default mode; zero-shot current action; full5407 questions","Original Gemini-3.6-Flash; size –; published default mode; zero-shot current action; full5407 questions","Original Qwen3-VL-2B; size 2B; published default mode; zero-shot current action; full5407 questions","SenseNova-SI-1.1-2B","Original SenseNova-SI-1.1-2B; size 2B; published default mode; zero-shot current action; full5407 questions","Cosmos3-Edge-2B","Original Cosmos3-Edge-2B; size 2B; published default mode; zero-shot current action; full5407 questions","Original InternVL3-2B; size 2B; published default mode; zero-shot current action; full5407 questions","Original GPT-6-Astra; size –; published default mode; zero-shot next action; full4880 questions","Original RynnBrain1.1-122B-A10B; size 122B; published default mode; zero-shot next action; full4880 questions","Original Doubao-Seed-2.0-Lite; size –; published default mode; zero-shot next action; full4880 questions","Original RynnBrain1.1-9B; size 9B; published default mode; zero-shot next action; full4880 questions","Original Qwen3-VL-235B-A22B; size –; published default mode; zero-shot next action; full4880 questions","Original Qwen3-VL-32B; size 32B; published default mode; zero-shot next action; full4880 questions","Original Qwen3-VL-30B-A3B; size 30B-A3B; published default mode; zero-shot next action; full4880 questions","Original RynnBrain1.1-2B; size 2B; published default mode; zero-shot next action; full4880 questions","Original Qwen3-VL-8B; size 8B; published default mode; zero-shot next action; full4880 questions","Original Qwen3-VL-8B-Thinking; size 8B; Thinking mode; zero-shot next action; full4880 questions","Original MiMo-VL-7B-RL; size 7B; published default mode; zero-shot next action; full4880 questions","Original RynnBrain-2B; size 2B; published default mode; zero-shot next action; full4880 questions","Original Cosmos-Reason2-2B; size 2B; published default mode; zero-shot next action; full4880 questions","Original Gemini-3.6-Flash; size –; published default mode; zero-shot next action; full4880 questions","Original Qwen3-VL-2B; size 2B; published default mode; zero-shot next action; full4880 questions","Original SenseNova-SI-1.1-2B; size 2B; published default mode; zero-shot next action; full4880 questions","Original Cosmos3-Edge-2B; size 2B; published default mode; zero-shot next action; full4880 questions","Original InternVL3-2B; size 2B; published default mode; zero-shot next action; full4880 questions","Original GPT-6-Astra; size –; published default mode; zero-shot goal-conditioned next action; full4880 questions","Original RynnBrain1.1-122B-A10B; size 122B; published default mode; zero-shot goal-conditioned next action; full4880 questions","Original Doubao-Seed-2.0-Lite; size –; published default mode; zero-shot goal-conditioned next action; full4880 questions","Original RynnBrain1.1-9B; size 9B; published default mode; zero-shot goal-conditioned next action; full4880 questions","Original Qwen3-VL-235B-A22B; size –; published default mode; zero-shot goal-conditioned next action; full4880 questions","Original Qwen3-VL-32B; size 32B; published default mode; zero-shot goal-conditioned next action; full4880 questions","Original Qwen3-VL-30B-A3B; size 30B-A3B; published default mode; zero-shot goal-conditioned next action; full4880 questions","Original RynnBrain1.1-2B; size 2B; published default mode; zero-shot goal-conditioned next action; full4880 questions","Original Qwen3-VL-8B; size 8B; published default mode; zero-shot goal-conditioned next action; full4880 questions","Original Qwen3-VL-8B-Thinking; size 8B; Thinking mode; zero-shot goal-conditioned next action; full4880 questions","Original MiMo-VL-7B-RL; size 7B; published default mode; zero-shot goal-conditioned next action; full4880 questions","Original RynnBrain-2B; size 2B; published default mode; zero-shot goal-conditioned next action; full4880 questions","Original Cosmos-Reason2-2B; size 2B; published default mode; zero-shot goal-conditioned next action; full4880 questions","Original Gemini-3.6-Flash; size –; published default mode; zero-shot goal-conditioned next action; full4880 questions","Original Qwen3-VL-2B; size 2B; published default mode; zero-shot goal-conditioned next action; full4880 questions","Original SenseNova-SI-1.1-2B; size 2B; published default mode; zero-shot goal-conditioned next action; full4880 questions","Original Cosmos3-Edge-2B; size 2B; published default mode; zero-shot goal-conditioned next action; full4880 questions","Original InternVL3-2B; size 2B; published default mode; zero-shot goal-conditioned next action; full4880 questions","Original GPT-6-Astra; size –; published default mode; zero-shot frame matching; full5325 questions","Original RynnBrain1.1-122B-A10B; size 122B; published default mode; zero-shot frame matching; full5325 questions","Original Doubao-Seed-2.0-Lite; size –; published default mode; zero-shot frame matching; full5325 questions","Original RynnBrain1.1-9B; size 9B; published default mode; zero-shot frame matching; full5325 questions","Original Qwen3-VL-235B-A22B; size –; published default mode; zero-shot frame matching; full5325 questions","Original Qwen3-VL-32B; size 32B; published default mode; zero-shot frame matching; full5325 questions","Original Qwen3-VL-30B-A3B; size 30B-A3B; published default mode; zero-shot frame matching; full5325 questions","Original RynnBrain1.1-2B; size 2B; published default mode; zero-shot frame matching; full5325 questions","Original Qwen3-VL-8B; size 8B; published default mode; zero-shot frame matching; full5325 questions","Original Qwen3-VL-8B-Thinking; size 8B; Thinking mode; zero-shot frame matching; full5325 questions","Original MiMo-VL-7B-RL; size 7B; published default mode; zero-shot frame matching; full5325 questions","Original RynnBrain-2B; size 2B; published default mode; zero-shot frame matching; full5325 questions","Original Cosmos-Reason2-2B; size 2B; published default mode; zero-shot frame matching; full5325 questions","Original Gemini-3.6-Flash; size –; published default mode; zero-shot frame matching; full5325 questions","Original Qwen3-VL-2B; size 2B; published default mode; zero-shot frame matching; full5325 questions","Original SenseNova-SI-1.1-2B; size 2B; published default mode; zero-shot frame matching; full5325 questions","Original Cosmos3-Edge-2B; size 2B; published default mode; zero-shot frame matching; full5325 questions","Original InternVL3-2B; size 2B; published default mode; zero-shot frame matching; full5325 questions","Original GPT-6-Astra; size –; published default mode; zero-shot view matching; full4835 questions","Original RynnBrain1.1-122B-A10B; size 122B; published default mode; zero-shot view matching; full4835 questions","Original Doubao-Seed-2.0-Lite; size –; published default mode; zero-shot view matching; full4835 questions","Original RynnBrain1.1-9B; size 9B; published default mode; zero-shot view matching; full4835 questions","Original Qwen3-VL-235B-A22B; size –; published default mode; zero-shot view matching; full4835 questions","Original Qwen3-VL-32B; size 32B; published default mode; zero-shot view matching; full4835 questions","Original Qwen3-VL-30B-A3B; size 30B-A3B; published default mode; zero-shot view matching; full4835 questions","Original RynnBrain1.1-2B; size 2B; published default mode; zero-shot view matching; full4835 questions","Original Qwen3-VL-8B; size 8B; published default mode; zero-shot view matching; full4835 questions","Original Qwen3-VL-8B-Thinking; size 8B; Thinking mode; zero-shot view matching; full4835 questions","Original MiMo-VL-7B-RL; size 7B; published default mode; zero-shot view matching; full4835 questions","Original RynnBrain-2B; size 2B; published default mode; zero-shot view matching; full4835 questions","Original Cosmos-Reason2-2B; size 2B; published default mode; zero-shot view matching; full4835 questions","Original Gemini-3.6-Flash; size –; published default mode; zero-shot view matching; full4835 questions","Original Qwen3-VL-2B; size 2B; published default mode; zero-shot view matching; full4835 questions","Original SenseNova-SI-1.1-2B; size 2B; published default mode; zero-shot view matching; full4835 questions","Original Cosmos3-Edge-2B; size 2B; published default mode; zero-shot view matching; full4835 questions","Original InternVL3-2B; size 2B; published default mode; zero-shot view matching; full4835 questions","Original GPT-6-Astra; size –; published default mode; zero-shot frame ordering; full3979 questions","Original RynnBrain1.1-122B-A10B; size 122B; published default mode; zero-shot frame ordering; full3979 questions","Original Doubao-Seed-2.0-Lite; size –; published default mode; zero-shot frame ordering; full3979 questions","Original RynnBrain1.1-9B; size 9B; published default mode; zero-shot frame ordering; full3979 questions","Original Qwen3-VL-235B-A22B; size –; published default mode; zero-shot frame ordering; full3979 questions","Original Qwen3-VL-32B; size 32B; published default mode; zero-shot frame ordering; full3979 questions","Original Qwen3-VL-30B-A3B; size 30B-A3B; published default mode; zero-shot frame ordering; full3979 questions","Original RynnBrain1.1-2B; size 2B; published default mode; zero-shot frame ordering; full3979 questions","Original Qwen3-VL-8B; size 8B; published default mode; zero-shot frame ordering; full3979 questions","Original Qwen3-VL-8B-Thinking; size 8B; Thinking mode; zero-shot frame ordering; full3979 questions","Original MiMo-VL-7B-RL; size 7B; published default mode; zero-shot frame ordering; full3979 questions","Original RynnBrain-2B; size 2B; published default mode; zero-shot frame ordering; full3979 questions","Original Cosmos-Reason2-2B; size 2B; published default mode; zero-shot frame ordering; full3979 questions","Original Gemini-3.6-Flash; size –; published default mode; zero-shot frame ordering; full3979 questions","Original Qwen3-VL-2B; size 2B; published default mode; zero-shot frame ordering; full3979 questions","Original SenseNova-SI-1.1-2B; size 2B; published default mode; zero-shot frame ordering; full3979 questions","Original Cosmos3-Edge-2B; size 2B; published default mode; zero-shot frame ordering; full3979 questions","Original InternVL3-2B; size 2B; published default mode; zero-shot frame ordering; full3979 questions","Original backbone GPT-5.4; image expert FLUX.1-dev; speech expert Parler-large; source orchestration/separate-target-modality calls; narration900/900","https://arxiv.org/html/2609.37317v1#A7.T29","Original backbone GPT-5.4; image expert FLUX.1-Kontext; speech expert Parler-large; source orchestration/separate-target-modality calls; narration900/900","Original backbone GPT-5.4; image expert FLUX.1-dev; speech expert VoxCPM2; source orchestration/separate-target-modality calls; narration900/900","Original backbone GPT-5.4; image expert FLUX.1-Kontext; speech expert VoxCPM2; source orchestration/separate-target-modality calls; narration900/900","Original backbone GLM-4.6V; image expert FLUX.1-dev; speech expert Parler-large; source orchestration/separate-target-modality calls; narration900/900","Original backbone GLM-4.6V; image expert FLUX.1-Kontext; speech expert Parler-large; source orchestration/separate-target-modality calls; narration900/900","Original backbone GLM-4.6V; image expert FLUX.1-dev; speech expert VoxCPM2; source orchestration/separate-target-modality calls; narration900/900","Original backbone GLM-4.6V; image expert FLUX.1-Kontext; speech expert VoxCPM2; source orchestration/separate-target-modality calls; narration900/900","Original backbone Claude-Opus-4.7; image expert FLUX.1-dev; speech expert Parler-large; source orchestration/separate-target-modality calls; narration900/900","Original backbone Claude-Opus-4.7; image expert FLUX.1-Kontext; speech expert Parler-large; source orchestration/separate-target-modality calls; narration900/900","Original backbone Claude-Opus-4.7; image expert FLUX.1-dev; speech expert VoxCPM2; source orchestration/separate-target-modality calls; narration900/900","Original backbone Claude-Opus-4.7; image expert FLUX.1-Kontext; speech expert VoxCPM2; source orchestration/separate-target-modality calls; narration900/900","Original backbone InternVL3.5-4B; image expert FLUX.1-dev; speech expert Parler-large; source orchestration/separate-target-modality calls; narration900/900","Original backbone InternVL3.5-4B; image expert FLUX.1-Kontext; speech expert Parler-large; source orchestration/separate-target-modality calls; narration900/900","Original backbone InternVL3.5-4B; image expert Nitro-T-1.2B; speech expert Parler-large; source orchestration/separate-target-modality calls; narration900/900","Original backbone Qwen3.5-4B; image expert FLUX.1-dev; speech expert VoxCPM2; source orchestration/separate-target-modality calls; narration900/900","Original backbone Qwen3.5-4B; image expert FLUX.1-Kontext; speech expert VoxCPM2; source orchestration/separate-target-modality calls; narration900/900","Original backbone Qwen3.5-4B; image expert Nitro-T-1.2B; speech expert VoxCPM2; source orchestration/separate-target-modality calls; narration900/900","AnyGPT","Original backbone AnyGPT; image expert native; speech expert native; source orchestration/separate-target-modality calls; narration900/900","Original backbone Omni-Diffusion; image expert native; speech expert native; source orchestration/separate-target-modality calls; narration900/900","Dynin-Omni","Original backbone Dynin-Omni; image expert native; speech expert native; simultaneous single-pass three-modality generation; narration900/900","Original Claude Opus 5; S6.T2 Mean F1","https://arxiv.org/html/2611.14419v1#S6.T2","Original DeepSeek V4 Flash; S6.T2 Mean F1","Original GPT-5.6 Sol; S6.T2 Mean F1","Original Llama 3.3 70B; S6.T2 Mean F1","Original Qwen 3.8 27B; S6.T2 Mean F1","Original GPT-6.1-Sol; S4.T5 Overall ISR","https://arxiv.org/html/2611.14364v1#S4.T5","Original Claude Opus 5.5; S4.T5 Overall ISR","Original DeepSeek-V4.1-Flash; S4.T5 Overall ISR","Original Qwen3.8-Max; S4.T5 Overall ISR","Original GLM-5.3-Flash; S4.T5 Overall ISR","Original Kimi-K3; S4.T5 Overall ISR","Original Qwen3.8-27B; S4.T5 Overall ISR","Jev 1.14.0","Original Jev 1.14.0; Closed-source decision models; S4.T3 Overall ALL","https://arxiv.org/html/2611.14309v1#S4.T3","Microsoft Decision-1","Original Microsoft Decision-1; Closed-source decision models; S4.T3 Overall ALL","Liquid-d1-Ultra","Original Liquid-d1-Ultra; Closed-source decision models; S4.T3 Overall ALL","Liquid-d1","Original Liquid-d1; Closed-source decision models; S4.T3 Overall ALL","GPT-6 Luna Decisions","Original GPT-6 Luna Decisions; Closed-source decision models; S4.T3 Overall ALL","WeMM-JEV 27B","Original WeMM-JEV 27B; Open-source decision models; S4.T3 Overall ALL","KnowLine 31B Gen1","Original KnowLine 31B Gen1; Open-source decision models; S4.T3 Overall ALL","Perplexity Decider-V1.2","Original Perplexity Decider-V1.2; Open-source decision models; S4.T3 Overall ALL","Autotrust JEV 27B-VL","Original Autotrust JEV 27B-VL; Open-source decision models; S4.T3 Overall ALL","Decisio-Gemma-4-31B","Original Decisio-Gemma-4-31B; Open-source decision models; S4.T3 Overall ALL","Original Clef; Open-source decision models; S4.T3 Overall ALL","Perplexity Decider-V1","Original Perplexity Decider-V1; Open-source decision models; S4.T3 Overall ALL","Perplexity Decider-Flash-V1.2","Original Perplexity Decider-Flash-V1.2; Open-source decision models; S4.T3 Overall ALL","H2O-Lightning 4B","Original H2O-Lightning 4B; Open-source decision models; S4.T3 Overall ALL","Decisio-Gemma-4-12B","Original Decisio-Gemma-4-12B; Open-source decision models; S4.T3 Overall ALL","Original Quyet-1.0-Large; Open-source decision models; S4.T3 Overall ALL","OpenJev 4B","Original OpenJev 4B; Open-source decision models; S4.T3 Overall ALL","Original Clef-Omni; Open-source decision models; S4.T3 Overall ALL","Mapika Decider 4B","Original Mapika Decider 4B; Open-source decision models; S4.T3 Overall ALL","Original Clef-Flash; Open-source decision models; S4.T3 Overall ALL","Decisio-Qwen3.6-35B-A3B","Original Decisio-Qwen3.6-35B-A3B; Open-source decision models; S4.T3 Overall ALL","Intern-Decision 4B","Original Intern-Decision 4B; Open-source decision models; S4.T3 Overall ALL","JevK5 4B","Original JevK5 4B; Open-source decision models; S4.T3 Overall ALL","Strands-Decider 2B","Original Strands-Decider 2B; Open-source decision models; S4.T3 Overall ALL","SemIf 4B","Original SemIf 4B; Open-source decision models; S4.T3 Overall ALL","Intern-Decision 2B","Original Intern-Decision 2B; Open-source decision models; S4.T3 Overall ALL","Intern-Decision 0.8B","Original Intern-Decision 0.8B; Open-source decision models; S4.T3 Overall ALL","NanoJev 0.6B","Original NanoJev 0.6B; Open-source decision models; S4.T3 Overall ALL","Original GLM 5.3 Flash; S9.T2 Correct","https://arxiv.org/html/2611.14122v1#S9.T2","Original DeepSeek V4.1 Flash; S9.T2 Correct","GPT Luna 6","Original GPT Luna 6; S9.T2 Correct","Original Gemini 3.5 Flash Lite; S9.T2 Correct","Original Gemma 3 27B IT; S9.T2 Correct","Original Ministral 8B; S9.T2 Correct","Original Llama 4 Scout; S9.T2 Correct","LLaVA 1.5 13B","Original LLaVA 1.5 13B; S3.T4 w/ scenario · HMS","https://arxiv.org/html/2611.13997v1#S3.T4","LLaVA NeXT Vicuna 13B","Original LLaVA NeXT Vicuna 13B; S3.T4 w/ scenario · HMS","LLaVA OneVision Qwen2 7B","Original LLaVA OneVision Qwen2 7B; S3.T4 w/ scenario · HMS","Llama 3.2 11B Vision Instruct","Original Llama 3.2 11B Vision Instruct; S3.T4 w/ scenario · HMS","Original Pixtral 12B; S3.T4 w/ scenario · HMS","Original GLM 4V 9B; S3.T4 w/ scenario · HMS","Deepseek VL2","Original Deepseek VL2; S3.T4 w/ scenario · HMS","InternVL 2.5 4B","Original InternVL 2.5 4B; S3.T4 w/ scenario · HMS","InternVL 2.5 8B","Original InternVL 2.5 8B; S3.T4 w/ scenario · HMS","InternVL 2.5 38B","Original InternVL 2.5 38B; S3.T4 w/ scenario · HMS","InternVL 2.5 78B","Original InternVL 2.5 78B; S3.T4 w/ scenario · HMS","Qwen2 VL 7B Instruct","Original Qwen2 VL 7B Instruct; S3.T4 w/ scenario · HMS","Qwen2.5 VL 3B Instruct","Original Qwen2.5 VL 3B Instruct; S3.T4 w/ scenario · HMS","Qwen2.5 VL 7B Instruct","Original Qwen2.5 VL 7B Instruct; S3.T4 w/ scenario · HMS","Qwen2.5 VL 32B Instruct","Original Qwen2.5 VL 32B Instruct; S3.T4 w/ scenario · HMS","Qwen2.5 VL 72B Instruct","Original Qwen2.5 VL 72B Instruct; S3.T4 w/ scenario · HMS","Phi 4 Multimodal Instruct","Original Phi 4 Multimodal Instruct; S3.T4 w/ scenario · HMS","Mistral Small 3.1 24B Instruct 2503","Original Mistral Small 3.1 24B Instruct 2503; S3.T4 w/ scenario · HMS","GPT 4o 2024-11-20","Original GPT 4o 2024-11-20; S3.T4 w/ scenario · HMS","Original Gemini 2.0 Flash; S3.T4 w/ scenario · HMS","o4-mini 2025-04-16","Original o4-mini 2025-04-16; S3.T4 w/ scenario · HMS","Original GPT-6 Astra; Scaffold: Codex CLI; S3.T4 Mean","https://arxiv.org/html/2611.13994v1#S3.T4","Original Claude Opus 5.5; Scaffold: Claude Code; S3.T4 Mean","Original GPT-6 Astra; Scaffold: OpenCode; S3.T4 Mean","Original Claude Opus 5.5; Scaffold: OpenCode; S3.T4 Mean","Original Claude Opus 5; Scaffold: Claude Code; S3.T4 Mean","Original Claude Opus 5.5; Scaffold: MLEvolve; S3.T4 Mean","Original Claude Opus 5.5; Scaffold: Biomni; S3.T4 Mean","Original GPT-6 Astra; Scaffold: Biomni; S3.T4 Mean","Original DeepSeek V4.1 Flash (1017); Scaffold: OpenCode; S3.T4 Mean","Original Gemini 3.8 Flash; Scaffold: OpenCode; S3.T4 Mean","Original Gemini 3.8 Flash; Scaffold: Biomni; S3.T4 Mean","Original GPT-6 Astra; Scaffold: MLEvolve; S3.T4 Mean","Original Gemini 3.8 Flash; Scaffold: Gemini CLI; S3.T4 Mean","Original Claude Opus 5.5; Scaffold: Minimal; S3.T4 Mean","Original GPT-6 Astra; Scaffold: Minimal; S3.T4 Mean","Original GPT-5.6 Sol; Scaffold: Codex CLI; S3.T4 Mean","Original Gemini 3.8 Flash; Scaffold: MLEvolve; S3.T4 Mean","Original Gemini 3.1 Pro (preview); Scaffold: Gemini CLI; S3.T4 Mean","Original DeepSeek V4.1 Flash (1017); Scaffold: Biomni; S3.T4 Mean","Original DeepSeek V4.1 Flash (1017); Scaffold: MLEvolve; S3.T4 Mean","Original Gemini 3.8 Flash; Scaffold: Minimal; S3.T4 Mean","Original DeepSeek V4.1 Flash (1017); Scaffold: Minimal; S3.T4 Mean","Original GPT-4o-mini; Single-Shot LLM; S2.T1 (a) Reference-based: RTL-OPT","https://arxiv.org/html/2611.13713v1#S2.T1","Original Gemini-2.5; Single-Shot LLM; S2.T1 (a) Reference-based: RTL-OPT","Original DeepSeek V3; Single-Shot LLM; S2.T1 (a) Reference-based: RTL-OPT","Original DeepSeek R1; Single-Shot LLM; S2.T1 (a) Reference-based: RTL-OPT","Claude Opus","Original Claude Opus; (b) Open-ended: Original RTL; Single-Shot LLM; S2.T1 (a) Reference-based: RTL-OPT","Original GPT 5.3; (b) Open-ended: Original RTL; Single-Shot LLM; S2.T1 (a) Reference-based: RTL-OPT","RTLRewriter","Original RTLRewriter; (b) Open-ended: Original RTL; SOTA Baselines ⋆; S2.T1 (a) Reference-based: RTL-OPT","SymRTLo","Original SymRTLo; (b) Open-ended: Original RTL; SOTA Baselines ⋆; S2.T1 (a) Reference-based: RTL-OPT","Dr. RTL","Original Dr. RTL; (b) Open-ended: Original RTL; Ours; S2.T1 (a) Reference-based: RTL-OPT","Original Claude Fable 5; S4.T4 P3","https://arxiv.org/html/2611.13667v1#S4.T4","Original Claude Sonnet 5; S4.T4 P3","Original GPT-5.6 Sol; S4.T4 P3","Original GPT-5.6 Terra; S4.T4 P3","Original Gemini 3.1 Pro Preview; S4.T4 P3","Original Gemini 3.7 Flash; S4.T4 P3","Original Qwen3-VL-8B; Zero-shot VLMs; A3.T10 PC · ρ","https://arxiv.org/html/2611.13576v1#A3.T10","Original Qwen3-VL-8B; Zero-shot VLMs; A3.T10 PC · ρ; row 5","Original Qwen3-VL-32B; Zero-shot VLMs; A3.T10 PC · ρ","Original Qwen3-VL-32B; Zero-shot VLMs; A3.T10 PC · ρ; row 7","Original Qwen3-VL-235B; Zero-shot VLMs; A3.T10 PC · ρ","Original Qwen3-VL-235B; Zero-shot VLMs; A3.T10 PC · ρ; row 9","VideoScore","Original VideoScore; General video evaluators; A3.T10 PC · ρ","VideoScore2","Original VideoScore2; General video evaluators; A3.T10 PC · ρ","Original VideoScore2; General video evaluators; A3.T10 PC · ρ; row 13","VideoCon-Physics","Original VideoCon-Physics; Physics-specific evaluators; A3.T10 PC · ρ","VideoPhy-2-AutoEval","Original VideoPhy-2-AutoEval; Physics-specific evaluators; A3.T10 PC · ρ","Original VideoPhy-2-AutoEval; Physics-specific evaluators; A3.T10 PC · ρ; row 17","Original Gemini 4 Argon; Physics-specific evaluators; A3.T10 PC · ρ","Original Gemini 4 Argon; Physics-specific evaluators; A3.T10 PC · ρ; row 20","PhyRubric-Judge (ours)","Original PhyRubric-Judge (ours); Physics-specific evaluators; A3.T10 PC · ρ","Baichuan-M4","Original Baichuan-M4; Sx8.p11.1 Length-adjusted ETS","https://arxiv.org/html/2611.13562v1#Sx8.p11.1","Original DeepSeek-V4-Pro; Sx8.p11.1 Length-adjusted ETS","Original DeepSeek-V4-Pro-0813; Sx8.p11.1 Length-adjusted ETS","Original Doubao-Seed-2.1-Pro; Sx8.p11.1 Length-adjusted ETS","Original GLM-5.3; Sx8.p11.1 Length-adjusted ETS","Original GPT-5.6-sol; Sx8.p11.1 Length-adjusted ETS","Original Kimi-K3; Sx8.p11.1 Length-adjusted ETS","OpenEvidence","Original OpenEvidence; Sx8.p11.1 Length-adjusted ETS","Original Qwen3.8-Max; Sx8.p11.1 Length-adjusted ETS","Spark X2","Original Spark X2; Sx8.p11.1 Length-adjusted ETS","Original Baichuan-M4; Sx8.p11.1 Length-adjusted ETS; row 14","Original DeepSeek-V4-Pro; Sx8.p11.1 Length-adjusted ETS; row 15","Original DeepSeek-V4-Pro-0813; Sx8.p11.1 Length-adjusted ETS; row 16","Original Doubao-Seed-2.1-Pro; Sx8.p11.1 Length-adjusted ETS; row 17","Original GLM-5.3; Sx8.p11.1 Length-adjusted ETS; row 18","Original GPT-5.6-sol; Sx8.p11.1 Length-adjusted ETS; row 19","Original Kimi-K3; Sx8.p11.1 Length-adjusted ETS; row 20","Original OpenEvidence; Sx8.p11.1 Length-adjusted ETS; row 21","Original Qwen3.8-Max; Sx8.p11.1 Length-adjusted ETS; row 22","Original Spark X2; Sx8.p11.1 Length-adjusted ETS; row 23","Original Baichuan-M4; Sx8.p11.1 Length-adjusted ETS; row 25","Original DeepSeek-V4-Pro; Sx8.p11.1 Length-adjusted ETS; row 26","Original DeepSeek-V4-Pro-0813; Sx8.p11.1 Length-adjusted ETS; row 27","Original Doubao-Seed-2.1-Pro; Sx8.p11.1 Length-adjusted ETS; row 28","Original GLM-5.3; Sx8.p11.1 Length-adjusted ETS; row 29","Original GPT-5.6-sol; Sx8.p11.1 Length-adjusted ETS; row 30","Original Kimi-K3; Sx8.p11.1 Length-adjusted ETS; row 31","Original OpenEvidence; Sx8.p11.1 Length-adjusted ETS; row 32","Original Qwen3.8-Max; Sx8.p11.1 Length-adjusted ETS; row 33","Original Spark X2; Sx8.p11.1 Length-adjusted ETS; row 34","Original GPT 6 Astra; S3.T1 PSR","https://arxiv.org/html/2611.13355v1#S3.T1","Original GPT 6.1 Sol; S3.T1 PSR","Gemini 4","Original Gemini 4; S3.T1 PSR","Original Claude Opus 5.5; S3.T1 PSR","Original Claude Sonnet 5.5; S3.T1 PSR","Original Grok 4.7; S3.T1 PSR","Original Gemini 3.8 Flash; S3.T1 PSR","Original GPT 6 Luna; S3.T1 PSR","Original Seed 2.1 Pro; S3.T1 PSR","Original Doubao-Seed-Evolving; S3.T1 PSR","Seed 2.1 Lite","Original Seed 2.1 Lite; S3.T1 PSR","Original Seed 2.1 Turbo; S3.T1 PSR","Original Claude Opus 4.8; Direct generation, zero-shot ( k = 3 ); S3.T1 Pass rate","https://arxiv.org/html/2611.13183v1#S3.T1","Original GPT-5.5; Direct generation, zero-shot ( k = 3 ); S3.T1 Pass rate","Gemma-4 31B IT","Original Gemma-4 31B IT; Direct generation, zero-shot ( k = 3 ); S3.T1 Pass rate","Original GPT-5 mini; Direct generation, zero-shot ( k = 3 ); S3.T1 Pass rate","Original Claude Sonnet 5; Direct generation, zero-shot ( k = 3 ); S3.T1 Pass rate","Original Opus 4.8; Direct generation, zero-shot ( k = 3 ); Configuration: Baseline (no skills); S3.T1 Pass rate","Original Opus 4.8; Direct generation, zero-shot ( k = 3 ); Configuration: Skills v1 (inert docs); S3.T1 Pass rate","Original Opus 4.8; Direct generation, zero-shot ( k = 3 ); Configuration: Skills v2 (live plugin); S3.T1 Pass rate","Original Opus 4.8; Direct generation, zero-shot ( k = 3 ); Configuration: Skills v3 (rewritten); S3.T1 Pass rate","Original Opus 4.8; Direct generation, zero-shot ( k = 3 ); Configuration: afcode (minimal CLI); S3.T1 Pass rate","Original InternVL3-1B; S4.T2 Pointing","https://arxiv.org/html/2611.13150v1#S4.T2","Original InternVL3.5-1B; S4.T2 Pointing","Original InternVL3-2B; S4.T2 Pointing","Original InternVL3.5-2B; S4.T2 Pointing","Original Qwen2-VL-2B; S4.T2 Pointing","Original Qwen3-VL-2B; S4.T2 Pointing","Ministral3-3B","Original Ministral3-3B; S4.T2 Pointing","Original Qwen2.5-VL-3B; S4.T2 Pointing","Original Gemma3-4B; S4.T2 Pointing","Original InternVL3.5-4B; S4.T2 Pointing","Original Qwen3-VL-4B; S4.T2 Pointing","Original Qwen2-VL-7B; S4.T2 Pointing","Original Qwen2.5-VL-7B; S4.T2 Pointing","Original InternVL3-8B; S4.T2 Pointing","Original InternVL3.5-8B; S4.T2 Pointing","Ministral3-8B","Original Ministral3-8B; S4.T2 Pointing","Original Qwen3-VL-8B; S4.T2 Pointing","Original Gemma3-12B; S4.T2 Pointing","Original InternVL3-14B; S4.T2 Pointing","Original InternVL3.5-14B; S4.T2 Pointing","Original Ministral3-14B; S4.T2 Pointing","Original Gemma3-27B; S4.T2 Pointing","Original Qwen2.5-VL-32B; S4.T2 Pointing","Original Qwen3-VL-32B; S4.T2 Pointing","Original InternVL3-38B; S4.T2 Pointing","Original InternVL3.5-38B; S4.T2 Pointing","Original GPT-6-Astra; GTAVBench; S4.T2 Overall","https://arxiv.org/html/2611.13028v1#S4.T2","Original GPT-6.1-Sol; GTAVBench; S4.T2 Overall","Original DeepSeek-V4.1-Flash-1017; GTAVBench; S4.T2 Overall","Original GLM-5.3-Flash; GTAVBench; S4.T2 Overall","Original Kimi-K3; GTAVBench; S4.T2 Overall","Original Qwen3.8-27B; GTAVBench; S4.T2 Overall","Original GPT-6-Astra; GTAVBench-AGI (Codex); S4.T2 Overall","Original GPT-6.1-Sol; GTAVBench-AGI (Codex); S4.T2 Overall","Original DeepSeek-V4.1-Flash-1017; GTAVBench-AGI (Codex); S4.T2 Overall","Original GLM-5.3-Flash; GTAVBench-AGI (Codex); S4.T2 Overall","Original Kimi-K3; GTAVBench-AGI (Codex); S4.T2 Overall","Original Qwen3.8-27B; GTAVBench-AGI (Codex); S4.T2 Overall","Original GPT-5.5; Agent: SWE-agent; S4.T2 Overall","https://arxiv.org/html/2611.12966v1#S4.T2","Original DeepSeek-V4-Flash; Agent: SWE-agent; S4.T2 Overall","Original Claude-Opus-4.7; Agent: SWE-agent; S4.T2 Overall","Original GLM-5.1; Agent: SWE-agent; S4.T2 Overall","Original Qwen3.6-Plus; Agent: SWE-agent; S4.T2 Overall","Original Claude-Opus-4.7; Agent: Claude Code; S4.T2 Overall","Original GPT-5.5; Agent: Codex; S4.T2 Overall","Original GPT-5.5; Agent: Spider-Agent-Lite; S4.T2 Overall","Original DeepSeek-V4-Flash; Agent: Spider-Agent-Lite; S4.T2 Overall","Original Opus 5.5; S2.T2 Overall","https://arxiv.org/html/2611.12934v1#S2.T2","Original GPT-6 Sol; S2.T2 Overall","Original GPT-6.1 Sol; S2.T2 Overall","Original Sonnet 5.5; S2.T2 Overall","Original GPT-6 Astra; S2.T2 Overall","Original GPT-6.1 Terra; S2.T2 Overall","Original Gemini 3 Flash; S2.T2 Overall","Original DeepSeek Flash; S2.T2 Overall","Original Kimi K2.5; S2.T2 Overall","Original Haiku 5.5; S2.T2 Overall","Qwen 3.7 Flash","Original Qwen 3.7 Flash; S2.T2 Overall","Original GPT-6 Luna; S2.T2 Overall","Majority class","Original Majority class; S2.T2 Overall","Always no change","Original Always no change; S2.T2 Overall","Original Claude Opus 5.5; Harness: Claude Code; Effort: max; S4.T1 Overall","https://arxiv.org/html/2611.12856v1#S4.T1","Original Claude Opus 5.5; Harness: Claude Code; Effort: high; S4.T1 Overall","Original Claude Opus 5.5; Harness: Claude Code; Effort: xhigh; S4.T1 Overall","Original Claude Sonnet 5.5; Harness: Claude Code; Effort: max; S4.T1 Overall","Original GPT-6 Astra; Harness: Codex; Effort: xhigh; S4.T1 Overall","Original GPT-6 Astra; Harness: Codex; Effort: high; S4.T1 Overall","Original GPT-6.1 Sol; Harness: Codex; Effort: xhigh; S4.T1 Overall","Original Claude Opus 5.5; Harness: Claude Code; Effort: medium; S4.T1 Overall","Original GPT-6 Astra; Harness: Codex; Effort: medium; S4.T1 Overall","Original GPT-6.1 Sol; Harness: Codex; Effort: max; S4.T1 Overall","Original GPT-6 Astra; Harness: Codex; Effort: low; S4.T1 Overall","Original GPT-6.1 Sol; Harness: Codex; Effort: low; S4.T1 Overall","Original GPT-6 Astra; Harness: Codex; Effort: max; S4.T1 Overall","Original GPT-6.1 Sol; Harness: Codex; Effort: medium; S4.T1 Overall","Original GPT-6.1 Sol; Harness: Codex; Effort: high; S4.T1 Overall","Original Claude Sonnet 5.5; Harness: Claude Code; Effort: high; S4.T1 Overall","Original Claude Sonnet 5.5; Harness: Claude Code; Effort: xhigh; S4.T1 Overall","Original Claude Opus 5.5; Harness: Claude Code; Effort: low; S4.T1 Overall","Original Claude Sonnet 5.5; Harness: Claude Code; Effort: medium; S4.T1 Overall","Original Claude Sonnet 5.5; Harness: Claude Code; Effort: low; S4.T1 Overall","Original Kimi K3.1 Pro; Harness: Claude Code; Effort: max; S4.T1 Overall","Original GPT-6 Luna; Harness: Codex; Effort: max; S4.T1 Overall","Original Doubao-Seed-Evolving; Harness: Claude Code; Effort: high; S4.T1 Overall","Original DeepSeek V4.1 Flash; Harness: Claude Code; Effort: max; S4.T1 Overall","Original GLM-5.3 Flash; Harness: Claude Code; Effort: max; S4.T1 Overall","Original claude-opus-5.5; S5.T1 End to end","https://arxiv.org/html/2611.12393v1#S5.T1","Original claude-opus-5; S5.T1 End to end","Original gpt-5.6-sol; S5.T1 End to end","grok-4.7","Original grok-4.7; S5.T1 End to end","Original gpt-5.5; S5.T1 End to end","Original DeepSeek-V4-Pro; S5.T1 End to end","Original gpt-6.1-sol; S5.T1 End to end","Original claude-opus-4.8; S5.T1 End to end","Original gpt-5.4; S5.T1 End to end","Original DeepSeek-V4.1-Flash; S5.T1 End to end","Original Kimi-K2.6; S5.T1 End to end","Original claude-sonnet-5; S5.T1 End to end","Original gpt-5.6-luna; S5.T1 End to end","Original gpt-6-luna; S5.T1 End to end","Original gpt-5.4-mini; S5.T1 End to end","Original DeepSeek-V4-Flash; S5.T1 End to end","Original claude-haiku-4.5; S5.T1 End to end","CLIP ViT-L/14","Original CLIP ViT-L/14; S4.T2 State · Cars","https://arxiv.org/html/2611.12364v1#S4.T2","SigLIP-2-so400m","Original SigLIP-2-so400m; S4.T2 State · Cars","jina-clip-v2","Original jina-clip-v2; S4.T2 State · Cars","WeMM-Embedding-9B","Original WeMM-Embedding-9B; Discriminative embedders; S4.T2 State · Cars","E5-Omni-7B","Original E5-Omni-7B; Discriminative embedders; S4.T2 State · Cars","RzenEmbed-v2","Original RzenEmbed-v2; Discriminative embedders; S4.T2 State · Cars","Qwen3-VL-Emb.-8B","Original Qwen3-VL-Emb.-8B; Discriminative embedders; S4.T2 State · Cars","Qwen3-VL-Emb.-2B","Original Qwen3-VL-Emb.-2B; Discriminative embedders; S4.T2 State · Cars","OmniEmbed-3B","Original OmniEmbed-3B; Discriminative embedders; S4.T2 State · Cars","VLM2Vec-V2.0-2B","Original VLM2Vec-V2.0-2B; Discriminative embedders; S4.T2 State · Cars","VLM2Vec-2B","Original VLM2Vec-2B; Discriminative embedders; S4.T2 State · Cars","VLM2Vec-7B","Original VLM2Vec-7B; Discriminative embedders; S4.T2 State · Cars","Embed-RL-2B (no CoT)","Original Embed-RL-2B (no CoT); Discriminative embedders; S4.T2 State · Cars","Embed-RL-4B (no CoT)","Original Embed-RL-4B (no CoT); Discriminative embedders; S4.T2 State · Cars","B-Tokens-2B","Original B-Tokens-2B; Discriminative embedders; S4.T2 State · Cars","UME-R1-7B (Disc)","Original UME-R1-7B (Disc); Discriminative embedders; S4.T2 State · Cars","UME-R1-7B (Gen)","Original UME-R1-7B (Gen); Discriminative embedders; S4.T2 State · Cars","TWN-4B","Original TWN-4B; Discriminative embedders; S4.T2 State · Cars","Embed-RL-2B + T-CoT","Original Embed-RL-2B + T-CoT; Discriminative embedders; S4.T2 State · Cars","Embed-RL-4B + T-CoT","Original Embed-RL-4B + T-CoT; Discriminative embedders; S4.T2 State · Cars","Top-30 reranking (ceiling probe)","Original Top-30 reranking (ceiling probe); Discriminative embedders; S4.T2 State · Cars","Original OpenVLA; S4.T2 AVG","https://arxiv.org/html/2611.12175v1#S4.T2","Original π 0; S4.T2 AVG","Original π 0.5; S4.T2 AVG","SpatialVLA","Original SpatialVLA; S4.T2 AVG","InternVLA-M1","Original InternVLA-M1; S4.T2 AVG","Spatial-Forcing","Original Spatial-Forcing; S4.T2 AVG","Original GPT-4o; COPA; S4.T2 AVG","Original Gemini-2.5-Flash; COPA; S4.T2 AVG","Original Claude-Sonnet-4.6; COPA; S4.T2 AVG","Original GPT-4o; MOKA; S4.T2 AVG","Original Gemini-2.5-Flash; MOKA; S4.T2 AVG","Original Claude-Sonnet-4.6; MOKA; S4.T2 AVG","Claude Haiku 4.5 r","Original Claude Haiku 4.5 r; S5.T3 ℛ A F","https://arxiv.org/html/2611.12167v1#S5.T3","Claude Sonnet 4.5 r","Original Claude Sonnet 4.5 r; S5.T3 ℛ A F","Claude Opus 4.5 r","Original Claude Opus 4.5 r; S5.T3 ℛ A F","GPT-5 Nano r","Original GPT-5 Nano r; S5.T3 ℛ A F","GPT-5 Mini r","Original GPT-5 Mini r; S5.T3 ℛ A F","GPT-5 r","Original GPT-5 r; S5.T3 ℛ A F","Original Gemini 2.5 Flash Lite; S5.T3 ℛ A F","Original Gemini 2.5 Flash; S5.T3 ℛ A F","Gemini 2.5 Pro r","Original Gemini 2.5 Pro r; S5.T3 ℛ A F","Grok 4.1 Fast (Non-Reasoning)","Original Grok 4.1 Fast (Non-Reasoning); S5.T3 ℛ A F","Grok 4.1 Fast (Reasoning) r","Original Grok 4.1 Fast (Reasoning) r; S5.T3 ℛ A F","DeepSeek V3.2 r","Original DeepSeek V3.2 r; S5.T3 ℛ A F","DeepSeek R1 r","Original DeepSeek R1 r; S5.T3 ℛ A F","Qwen3 4B (Non-Reasoning)","Original Qwen3 4B (Non-Reasoning); S5.T3 ℛ A F","Qwen3 4B (Reasoning) r","Original Qwen3 4B (Reasoning) r; S5.T3 ℛ A F","Qwen3 235B (Non-Reasoning)","Original Qwen3 235B (Non-Reasoning); S5.T3 ℛ A F","Qwen3 235B (Reasoning) r","Original Qwen3 235B (Reasoning) r; S5.T3 ℛ A F","GLM-4.7 r","Original GLM-4.7 r; S5.T3 ℛ A F","Kimi K2 (Non-Reasoning)","Original Kimi K2 (Non-Reasoning); S5.T3 ℛ A F","Original GPT-OSS 120B; S5.T3 ℛ A F","Original Claude Opus 4.8; S3.T2 SWE-Time Avg.","https://arxiv.org/html/2611.12083v1#S3.T2","Original Claude Sonnet 4.6; S3.T2 SWE-Time Avg.","Original Gemini 3.1 Pro; S3.T2 SWE-Time Avg.","Original Gemini 3.8 Flash; S3.T2 SWE-Time Avg.","Original Gemini 3.6 Flash; S3.T2 SWE-Time Avg.","Original DeepSeek V4 Flash; S3.T2 SWE-Time Avg.","Original GLM 5.3; S3.T2 SWE-Time Avg.","Original GLM 5.2; S3.T2 SWE-Time Avg.","Original Qwen 3.8 Max; S3.T2 SWE-Time Avg.","Original Qwen 3.6 Plus; S3.T2 SWE-Time Avg.","Original Kimi K3; S3.T2 SWE-Time Avg.","Original Kimi K2.6; S3.T2 SWE-Time Avg.","Original Hunyuan 3; S3.T2 SWE-Time Avg.","Original GPT 5.5; Codex; S3.T2 SWE-Time Avg.","Original GPT 5.4; Codex; S3.T2 SWE-Time Avg.","Nova-SI","Original Nova-SI; Harness: —; S4.T1 Q mean","https://arxiv.org/html/2611.12048v1#S4.T1","GPT-6 Astra [xhigh]","Original GPT-6 Astra [xhigh]; General; Harness: Codex; S4.T1 Q mean","GPT-6.1 Sol [xhigh]","Original GPT-6.1 Sol [xhigh]; General; Harness: Codex; S4.T1 Q mean","DeepSeek-V4.1-Flash-1017 [max]","Original DeepSeek-V4.1-Flash-1017 [max]; General; Harness: DeepSeek Harness; S4.T1 Q mean","GLM-5.3 [max]","Original GLM-5.3 [max]; General; Harness: zcode; S4.T1 Q mean","GLM-5.3-Flash [max]","Original GLM-5.3-Flash [max]; General; Harness: zcode; S4.T1 Q mean","Original DeepSeek-V4.1-Flash-1017 [max]; File-only; Harness: InternBootcamp; S4.T1 Q mean","Original GLM-5.3 [max]; File-only; Harness: InternBootcamp; S4.T1 Q mean","Original GLM-5.3-Flash [max]; File-only; Harness: InternBootcamp; S4.T1 Q mean","Original Gemini 3 Pro; S3.T1 Depth Score","https://arxiv.org/html/2611.11794v1#S3.T1","Original GPT-5; S3.T1 Depth Score","Original Gemini 2.5 Pro; S3.T1 Depth Score","Original DeepSeek-V3.2-Speciale; S3.T1 Depth Score","Original DeepSeek-V3.2; S3.T1 Depth Score","Original Qwen3-235B-A22B-Thinking-2507; S3.T1 Depth Score","Ring-1T","Original Ring-1T; S3.T1 Depth Score","Original DeepSeek-R1; S3.T1 Depth Score","Ling-1T","Original Ling-1T; Inference Mode: Non-thinking; S3.T1 Depth Score","Original DeepSeek-V3.1; Inference Mode: Non-thinking; S3.T1 Depth Score","Original DeepSeek-V4.1-Flash-1017; S6.T2 Localization Stability","https://arxiv.org/html/2611.11588v2#S6.T2","Original Qwen3.8-27B; S6.T2 Localization Stability","Original Qwen3.5-122B-A10B; S6.T2 Localization Stability","Original Qwen3.5-397B-A17B; S6.T2 Localization Stability","Original GPT-6-Astra; GPT family; S6.T2 Localization Stability","Original GPT-5.6-Sol; GPT family; S6.T2 Localization Stability","Original Gemini-3.8-Flash; GPT family; S6.T2 Localization Stability","Original Claude-Haiku-5.5; GPT family; S6.T2 Localization Stability","Original Claude-Haiku-4.5; GPT family; S6.T2 Localization Stability","deepseek-v3.2","Original deepseek-v3.2; S6.T10 Full","https://arxiv.org/html/2611.11469v1#S6.T10","qwen3-235b-a22b-instruct","Original qwen3-235b-a22b-instruct; S6.T10 Full","Original glm-4.7; S6.T10 Full","Original gemini-3.1-pro; S6.T10 Full","gpt-5.2","Original gpt-5.2; S6.T10 Full","starcoder","Original starcoder; S6.T10 Full","Original GPT-4o-mini; S2.T2 Phase Rec. Acc.","https://arxiv.org/html/2611.11437v1#S2.T2","Original Claude-Haiku-4.5; S2.T2 Phase Rec. Acc.","Original InternVL2.5-8B; S2.T2 Phase Rec. Acc.","Original Qwen2.5-VL-7B-Instruct; S2.T2 Phase Rec. Acc.","Original Qwen3-VL-8B-Instruct; S2.T2 Phase Rec. Acc.","Original Qwen3-VL-8B-Thinking; S2.T2 Phase Rec. Acc.","Original Pixtral-12B-2409; S2.T2 Phase Rec. Acc.","Original InternVL3-14B; S2.T2 Phase Rec. Acc.","Original InternVL2.5-26B; S2.T2 Phase Rec. Acc.","Original Gemma-3-27B-it; S2.T2 Phase Rec. Acc.","Original Qwen2.5-VL-32B-Instruct; S2.T2 Phase Rec. Acc.","Original Qwen3-VL-32B-Instruct; S2.T2 Phase Rec. Acc.","Original Qwen2.5-VL-72B-Instruct; S2.T2 Phase Rec. Acc.","Original InternVL2.5-78B; S2.T2 Phase Rec. Acc.","HuatuoGPT-Vision-34B-hf","Original HuatuoGPT-Vision-34B-hf; Medical-domain; S2.T2 Phase Rec. Acc.","Original Lingshu-32B; Medical-domain; S2.T2 Phase Rec. Acc.","MedGemma-27B-it","Original MedGemma-27B-it; Medical-domain; S2.T2 Phase Rec. Acc.","Original Qwen3-8B; A6.T12 All","https://arxiv.org/html/2611.11284v2#A6.T12","Original Llama-3.1-8B; A6.T12 All","Original Gemma-3-12B; A6.T12 All","Lapa-12B","Original Lapa-12B; A6.T12 All","MamayLM-12B","Original MamayLM-12B; A6.T12 All","MamayLM-27B","Original MamayLM-27B; A6.T12 All","Original Qwen3-8B; A6.T12 All; row 17","Original Llama-3.1-8B; A6.T12 All; row 19","Original Gemma-3-12B; A6.T12 All; row 21","Original Lapa-12B; A6.T12 All; row 23","Original MamayLM-12B; A6.T12 All; row 25","Original MamayLM-27B; A6.T12 All; row 27","GPT-6 Astra (max)","Original GPT-6 Astra (max); S5.T3 Score","https://arxiv.org/html/2611.10926v1#S5.T3","Claude Opus 5.5 (max)","Original Claude Opus 5.5 (max); S5.T3 Score","Claude Sonnet 5.5 (max)","Original Claude Sonnet 5.5 (max); S5.T3 Score","GPT-6.1 Sol (max)","Original GPT-6.1 Sol (max); S5.T3 Score","GPT-6.1 Terra (max)","Original GPT-6.1 Terra (max); S5.T3 Score","Claude Fable 5.1 (max)","Original Claude Fable 5.1 (max); S5.T3 Score","Muse Wonder 1.0 (max)","Original Muse Wonder 1.0 (max); S5.T3 Score","Gemini 4 Argon (high)","Original Gemini 4 Argon (high); S5.T3 Score","Claude Haiku 5.5 (max)","Original Claude Haiku 5.5 (max); S5.T3 Score","Muse Spark 1.3 (max)","Original Muse Spark 1.3 (max); S5.T3 Score","Kimi K3.1 Pro (max)","Original Kimi K3.1 Pro (max); S5.T3 Score","Grok 4.7 (xhigh)","Original Grok 4.7 (xhigh); S5.T3 Score","Qwen 3.8 Max (xhigh)","Original Qwen 3.8 Max (xhigh); S5.T3 Score","Inkling 2 Small (max)","Original Inkling 2 Small (max); S5.T3 Score","GLM 5.3 (max)","Original GLM 5.3 (max); S5.T3 Score","DeepSeek V4.1 Flash (max)","Original DeepSeek V4.1 Flash (max); S5.T3 Score","GPT-6 Luna (max)","Original GPT-6 Luna (max); S5.T3 Score","Mistral Large 4 (high)","Original Mistral Large 4 (high); S5.T3 Score","DeepSeek V4 Pro (max)","Original DeepSeek V4 Pro (max); S5.T3 Score","Nemotron 3 Ultra (high)","Original Nemotron 3 Ultra (high); S5.T3 Score","Inkling (max)","Original Inkling (max); S5.T3 Score","Inkling Small (max)","Original Inkling Small (max); S5.T3 Score","Original Qwen3.6-27B; S5.T3 Strict Overall","https://arxiv.org/html/2611.10626v1#S5.T3","Original Qwen3.5-9B; S5.T3 Strict Overall","Original Qwen3.5-27B; S5.T3 Strict Overall","Original Qwen3-VL-32B; S5.T3 Strict Overall","Original Hulu-Med-32B; S5.T3 Strict Overall","Hulu-Med-14B","Original Hulu-Med-14B; S5.T3 Strict Overall","Original Qwen3-VL-8B; S5.T3 Strict Overall","Original Hulu-Med-7B; S5.T3 Strict Overall","Original Hulu-Med-4B; S5.T3 Strict Overall","Original MedGemma-27B; S5.T3 Strict Overall","Original Qwen3.5-4B; S5.T3 Strict Overall","Original MedGemma-4B; S5.T3 Strict Overall","Original MedGemma-1.5-4B; S5.T3 Strict Overall","ColonR1","Original ColonR1; S5.T3 Strict Overall","Gemini 3-flash","Original Gemini 3-flash; Set C — proprietary spot-check; S5.T3 Strict Overall","ChatGPT 5.4-mini","Original ChatGPT 5.4-mini; Set C — proprietary spot-check; S5.T3 Strict Overall","Original Claude Sonnet 4.6; Set C — proprietary spot-check; S5.T3 Strict Overall","Clinician panel","Original Clinician panel; Set B — clinician baseline; S5.T3 Strict Overall","Original Claude Haiku 4.5; Harness: ReAct-FC; S4.T3 Avg Score","https://arxiv.org/html/2611.10600v1#S4.T3","Original Claude Haiku 4.5; Harness: Pi; S4.T3 Avg Score","Original Claude Haiku 4.5; Harness: Deep Agents; S4.T3 Avg Score","Original Claude Haiku 4.5; Harness: DSH; S4.T3 Avg Score","Original Claude Haiku 4.5; Harness: Codex; S4.T3 Avg Score","Original Claude Haiku 5.5; Harness: ReAct-FC; S4.T3 Avg Score","Original Claude Haiku 5.5; Harness: Pi; S4.T3 Avg Score","Original Claude Haiku 5.5; Harness: Deep Agents; S4.T3 Avg Score","Original Claude Haiku 5.5; Harness: DSH; S4.T3 Avg Score","Original Claude Haiku 5.5; Harness: Codex; S4.T3 Avg Score","Original Claude Sonnet 5.5; Harness: ReAct-FC; S4.T3 Avg Score","Original Claude Sonnet 5.5; Harness: Pi; S4.T3 Avg Score","Original Claude Sonnet 5.5; Harness: Deep Agents; S4.T3 Avg Score","Original Claude Sonnet 5.5; Harness: DSH; S4.T3 Avg Score","Original Claude Sonnet 5.5; Harness: Codex; S4.T3 Avg Score","MAI-Code-1.1-Flash","Original MAI-Code-1.1-Flash; Harness: ReAct-FC; S4.T3 Avg Score","Original MAI-Code-1.1-Flash; Harness: Pi; S4.T3 Avg Score","Original MAI-Code-1.1-Flash; Harness: Deep Agents; S4.T3 Avg Score","Original MAI-Code-1.1-Flash; Harness: DSH; S4.T3 Avg Score","Original MAI-Code-1.1-Flash; Harness: Codex; S4.T3 Avg Score","Original Grok 4.7; Harness: ReAct-FC; S4.T3 Avg Score","Original Grok 4.7; Harness: Pi; S4.T3 Avg Score","Original Grok 4.7; Harness: Deep Agents; S4.T3 Avg Score","Original Grok 4.7; Harness: DSH; S4.T3 Avg Score","Original Grok 4.7; Harness: Codex; S4.T3 Avg Score","Original GPT-5.6 Luna; Harness: ReAct-FC; S4.T3 Avg Score","Original GPT-5.6 Luna; Harness: Pi; S4.T3 Avg Score","Original GPT-5.6 Luna; Harness: Deep Agents; S4.T3 Avg Score","Original GPT-5.6 Luna; Harness: DSH; S4.T3 Avg Score","Original GPT-5.6 Luna; Harness: Codex; S4.T3 Avg Score","Original GPT-5.6 Terra; Harness: ReAct-FC; S4.T3 Avg Score","Original GPT-5.6 Terra; Harness: Pi; S4.T3 Avg Score","Original GPT-5.6 Terra; Harness: Deep Agents; S4.T3 Avg Score","Original GPT-5.6 Terra; Harness: DSH; S4.T3 Avg Score","Original GPT-5.6 Terra; Harness: Codex; S4.T3 Avg Score","Original GPT-6 Luna; Harness: ReAct-FC; S4.T3 Avg Score","Original GPT-6 Luna; Harness: Pi; S4.T3 Avg Score","Original GPT-6 Luna; Harness: Deep Agents; S4.T3 Avg Score","Original GPT-6 Luna; Harness: DSH; S4.T3 Avg Score","Original GPT-6 Luna; Harness: Codex; S4.T3 Avg Score","Original GPT-6.1 Sol; Harness: ReAct-FC; S4.T3 Avg Score","Original GPT-6.1 Sol; Harness: Pi; S4.T3 Avg Score","Original GPT-6.1 Sol; Harness: Deep Agents; S4.T3 Avg Score","Original GPT-6.1 Sol; Harness: DSH; S4.T3 Avg Score","Original GPT-6.1 Sol; Harness: Codex; S4.T3 Avg Score","Sonnet 5 (xhigh)","Original Sonnet 5 (xhigh); A3.T5 Program","https://arxiv.org/html/2611.10592v1#A3.T5","Opus 4.8 (xhigh)","Original Opus 4.8 (xhigh); A3.T5 Program","Opus 5 (max)","Original Opus 5 (max); A3.T5 Program","Original GPT-5.5 (xhigh); A3.T5 Program","GPT-5.6 Luna (max)","Original GPT-5.6 Luna (max); A3.T5 Program","Original GPT-5.6 Sol (max); A3.T5 Program","GPT-5.6 Terra (max)","Original GPT-5.6 Terra (max); A3.T5 Program","Gemini 3.6 Flash (high)","Original Gemini 3.6 Flash (high); A3.T5 Program","Gemini 3.5 Flash (high)","Original Gemini 3.5 Flash (high); A3.T5 Program","Gemini 3 Flash Preview (high)","Original Gemini 3 Flash Preview (high); A3.T5 Program","Original Gemini 3.1 Pro (high); A3.T5 Program","Original GLM 5.2 (max); A3.T5 Program","Kimi K3 (max)","Original Kimi K3 (max); A3.T5 Program","Gemma 4 31B IT (default)","Original Gemma 4 31B IT (default); A3.T5 Program","Qwen 3.6 27B (thinking)","Original Qwen 3.6 27B (thinking); A3.T5 Program","DeepSeek V4 Flash 0731 (max)","Original DeepSeek V4 Flash 0731 (max); A3.T5 Program","Original Qwen3-8B; A1.T6 COMET","https://arxiv.org/html/2611.10555v1#A1.T6","Original Qwen3.8-27B; A1.T6 COMET","Seed-X-PPO","Original Seed-X-PPO; A1.T6 COMET","Seed-X-Instruct","Original Seed-X-Instruct; A1.T6 COMET","Qwen3.7","Original Qwen3.7; A1.T6 COMET","Deepseek-flash","Original Deepseek-flash; A1.T6 COMET","Original GPT5.5; A1.T6 COMET","Gemini3.1","Original Gemini3.1; A1.T6 COMET","Original Gemma3-27B; A1.T6 COMET","Original Claude Sonnet 5; Sx4.T3 Clean (all h > 0 )","https://arxiv.org/html/2611.10541v1#Sx4.T3","Original DeepSeek V4 Flash; Sx4.T3 Clean (all h > 0 )","Original GLM-5; Sx4.T3 Clean (all h > 0 )","Original Gemini 2.5 Flash; Sx4.T3 Clean (all h > 0 )","Original Kimi K2.6; Sx4.T3 Clean (all h > 0 )","Original Llama 4 Maverick; Sx4.T3 Clean (all h > 0 )","Original MiMo V2.5; Sx4.T3 Clean (all h > 0 )","Original GPT-4.1; Sx4.T3 Clean (all h > 0 )","Original GPT-5; Sx4.T3 Clean (all h > 0 )","Qwen3-Instruct","Original Qwen3-Instruct; Sx4.T3 Clean (all h > 0 )","Qwen3-Thinking","Original Qwen3-Thinking; Sx4.T3 Clean (all h > 0 )","Claude Code (Opus 5)","Original Claude Code (Opus 5); S4.T3 Overall · Case","https://arxiv.org/html/2611.10475v1#S4.T3","Codex (GPT-5.6, high)","Original Codex (GPT-5.6, high); S4.T3 Overall · Case","Kimi Code (K3, high)","Original Kimi Code (K3, high); S4.T3 Overall · Case","OpenCode (Gemini 3.8 Flash, high)","Original OpenCode (Gemini 3.8 Flash, high); S4.T3 Overall · Case","ZCode (GLM-5.2, high)","Original ZCode (GLM-5.2, high); S4.T3 Overall · Case","OpenCode (DeepSeek V4 Pro)","Original OpenCode (DeepSeek V4 Pro); S4.T3 Overall · Case","Qwen Code (Qwen3.8-27B)","Original Qwen Code (Qwen3.8-27B); S4.T3 Overall · Case","Mini Agent (MiniMax M3)","Original Mini Agent (MiniMax M3); S4.T3 Overall · Case","Human (mean)","Original Human (mean); S4.T3 Overall · Case","Original Kimi-K3; S3.T2 Overall","https://arxiv.org/html/2611.10412v1#S3.T2","ANTH-O-4.6","Original ANTH-O-4.6; S3.T2 Overall","Original DeepSeek-V4.1-Flash; S3.T2 Overall","OAI-5.5","Original OAI-5.5; S3.T2 Overall","Original MiniMax-M3; S3.T2 Overall","Original DeepSeek-V4-Pro; S3.T2 Overall","Original MiniMax-M2.7; S3.T2 Overall","Original GLM-5; S3.T2 Overall","Original Qwen3.6-27B; S3.T2 Overall","Original Gemma 4 E4B; Environment: Supermarket; S3.T3 NDCG@10 (%)","https://arxiv.org/html/2611.10349v1#S3.T3","Original Qwen3-VL-8B; Environment: Supermarket; S3.T3 NDCG@10 (%)","Original GPT-4o; Environment: Supermarket; S3.T3 NDCG@10 (%)","Original Qwen3.6-Plus; Environment: Supermarket; S3.T3 NDCG@10 (%)","Original Claude Opus 5; Environment: Supermarket; S3.T3 NDCG@10 (%)","Original GPT-5.6 Sol; Environment: Supermarket; S3.T3 NDCG@10 (%)","Original Gemini 3.1 Pro; Environment: Supermarket; S3.T3 NDCG@10 (%)","Original Qwen3-VL-8B; Environment: Home Furnishings Store; S3.T3 NDCG@10 (%)","Original Gemma 4 E4B; Environment: Home Furnishings Store; S3.T3 NDCG@10 (%)","Original GPT-4o; Environment: Home Furnishings Store; S3.T3 NDCG@10 (%)","Original Claude Opus 5; Environment: Home Furnishings Store; S3.T3 NDCG@10 (%)","Original Qwen3.6-Plus; Environment: Home Furnishings Store; S3.T3 NDCG@10 (%)","Original GPT-5.6 Sol; Environment: Home Furnishings Store; S3.T3 NDCG@10 (%)","Original Gemini 3.1 Pro; Environment: Home Furnishings Store; S3.T3 NDCG@10 (%)","Original GPT 4o; S3.T2 Avg.","https://arxiv.org/html/2611.10344v1#S3.T2","Original GPT 5.1; S3.T2 Avg.","Original Gemini-2.5-Flash; S3.T2 Avg.","Original Gemini-2.5-Pro; S3.T2 Avg.","Spatial-MLLM-4B","Original Spatial-MLLM-4B; Open-Source Spatial Models; S3.T2 Avg.","Original VST-7B-RL; Open-Source Spatial Models; S3.T2 Avg.","VLM-3R-7B","Original VLM-3R-7B; Open-Source Spatial Models; S3.T2 Avg.","Qwen2-VL-2B-Instruct","Original Qwen2-VL-2B-Instruct; Open-Source General Models; S3.T2 Avg.","Original Qwen2-VL-7B-Instruct; Open-Source General Models; S3.T2 Avg.","Original Qwen2.5-VL-3B-Instruct; Open-Source General Models; S3.T2 Avg.","Original Qwen2.5-VL-7B-Instruct; Open-Source General Models; S3.T2 Avg.","LLaVa-onevision-qwen2-7B","Original LLaVa-onevision-qwen2-7B; Open-Source General Models; S3.T2 Avg.","Original LLaVA-Video-7B; Open-Source General Models; S3.T2 Avg.","LLaVA-NeXT-Video-7B","Original LLaVA-NeXT-Video-7B; Open-Source General Models; S3.T2 Avg.","Original InternVL3-2B; Open-Source General Models; S3.T2 Avg.","Original InternVL3-8B; Open-Source General Models; S3.T2 Avg.","Original MiniCPM-V-2.6; Open-Source General Models; S3.T2 Avg.","Molmo-7B-D","Original Molmo-7B-D; Open-Source General Models; S3.T2 Avg.","Original mPLUG-Owl3-7B; Open-Source General Models; S3.T2 Avg.","VILA1.5-7B","Original VILA1.5-7B; Open-Source General Models; S3.T2 Avg.","Original Ovis2.5-9B; Open-Source General Models; S3.T2 Avg.","Oryx-7B","Original Oryx-7B; Open-Source General Models; S3.T2 Avg.","Phi-3.5-Vision-Instruct-4B","Original Phi-3.5-Vision-Instruct-4B; Open-Source General Models; S3.T2 Avg.","Opus 5.5 (high)","Original Opus 5.5 (high); S3.T1 All-3","https://arxiv.org/html/2611.10304v1#S3.T1","GPT 6.1 Sol (high)","Original GPT 6.1 Sol (high); S3.T1 All-3","GPT 5.6 Sol (high)","Original GPT 5.6 Sol (high); S3.T1 All-3","GPT 5.6 Luna (xhigh)","Original GPT 5.6 Luna (xhigh); S3.T1 All-3","Original Kimi K3 (max); S3.T1 All-3","DeepSeek V4 Flash (high)","Original DeepSeek V4 Flash (high); S3.T1 All-3","GLM-5.3-Flash (max)","Original GLM-5.3-Flash (max); S3.T1 All-3","Qwen 3.8 27B (xhigh)","Original Qwen 3.8 27B (xhigh); S3.T1 All-3","Original Qwen 3.6 27B (thinking); S3.T1 All-3","Original google/gemini-2.5-flash; S3.T2 Average","https://arxiv.org/html/2611.10156v1#S3.T2","Qwen/Qwen3-235B-A22B-Instruct-2507","Original Qwen/Qwen3-235B-A22B-Instruct-2507; S3.T2 Average","Qwen/Qwen3-235B-A22B-Thinking-2507","Original Qwen/Qwen3-235B-A22B-Thinking-2507; S3.T2 Average","moonshotai/Kimi-K2-Instruct-0905","Original moonshotai/Kimi-K2-Instruct-0905; S3.T2 Average","openai/gpt-4.1","Original openai/gpt-4.1; S3.T2 Average","deepseek-ai/DeepSeek-R1-0528","Original deepseek-ai/DeepSeek-R1-0528; S3.T2 Average","deepseek-ai/DeepSeek-V3-0324","Original deepseek-ai/DeepSeek-V3-0324; S3.T2 Average","allenai/Llama-3.1-70B-Instruct-RM-RB2","Original allenai/Llama-3.1-70B-Instruct-RM-RB2; S3.T2 Average","Skywork/Skywork-Reward-V2-Llama-3.1-8B","Original Skywork/Skywork-Reward-V2-Llama-3.1-8B; S3.T2 Average","Skywork/Skywork-Reward-V2-Llama-3.2-3B","Original Skywork/Skywork-Reward-V2-Llama-3.2-3B; S3.T2 Average","RLHFlow/ArmoRM-Llama3-8B-v0.1","Original RLHFlow/ArmoRM-Llama3-8B-v0.1; S3.T2 Average","hendrydong/Mistral-RM-for-RAFT-GSHF-v0","Original hendrydong/Mistral-RM-for-RAFT-GSHF-v0; S3.T2 Average","weqweasdas/RM-Mistral-7B","Original weqweasdas/RM-Mistral-7B; S3.T2 Average","Ray2333/reward-model-Mistral-7B-Unified","Original Ray2333/reward-model-Mistral-7B-Unified; S3.T2 Average","sfairXC/FsfairX-LLaMA3-RM-v0.1","Original sfairXC/FsfairX-LLaMA3-RM-v0.1; S3.T2 Average","0-hero/Matter-0.1-7B-boost-DPO-preview","Original 0-hero/Matter-0.1-7B-boost-DPO-preview; S3.T2 Average","prometheus-eval/prometheus-7b-v2.0","Original prometheus-eval/prometheus-7b-v2.0; S3.T2 Average","NousResearch/Nous-Hermes-2-Mistral-7B-DPO","Original NousResearch/Nous-Hermes-2-Mistral-7B-DPO; S3.T2 Average","RLHFlow/pair-preference-model-LLaMA3-8B","Original RLHFlow/pair-preference-model-LLaMA3-8B; S3.T2 Average","Original GPT-6.1 Sol; A7.T9 E","https://arxiv.org/html/2611.10044v2#A7.T9","Original GPT-6 Astra; A7.T9 E","Original GPT-5.5; A7.T9 E","Original Gemini 4 Argon; A7.T9 E","Original Gemini 3.8 Flash; A7.T9 E","Original Gemini 3.7 Flash; A7.T9 E","Original Gemini 3.1 Pro; A7.T9 E","Original Claude Fable 5.1; A7.T9 E","Original Claude Opus 5.5; A7.T9 E","Original Claude Opus 4.7; A7.T9 E","Original Claude Opus 4.8; A7.T9 E","Original Gemma 3 27B; A6.T8 Success","https://arxiv.org/html/2611.09794v2#A6.T8","Original Grok 4.3; A6.T8 Success","Original Grok 4.6; A6.T8 Success","Original Llama 4 Maverick; A6.T8 Success","Original Amazon Nova Pro; A6.T8 Success","Original Qwen3-VL-8B; A6.T8 Success","Sonar Pro","Original Sonar Pro; A6.T8 Success","Original GPT-6 Astra; S4.tab1 Animation Fidelity · CLIP","https://arxiv.org/html/2611.09769v1#S4.tab1","Original GPT-6 Sol; S4.tab1 Animation Fidelity · CLIP","Original GPT-5.6 Sol; S4.tab1 Animation Fidelity · CLIP","Original Claude Fable 5.1; S4.tab1 Animation Fidelity · CLIP","Original Claude Opus 5; S4.tab1 Animation Fidelity · CLIP","Original Grok 4.7; S4.tab1 Animation Fidelity · CLIP","Original Qwen3.8 Max (0902); S4.tab1 Animation Fidelity · CLIP","Original Gemini 3.1 Pro Preview; S4.tab1 Animation Fidelity · CLIP","Original Gemini 3.8 Flash; S4.tab1 Animation Fidelity · CLIP","Original Doubao-Seed-2.1-pro; S4.tab1 Animation Fidelity · CLIP","Original Kimi K3; S4.tab1 Animation Fidelity · CLIP","Original DeepSeek V4.1 Flash; S4.tab1 Animation Fidelity · CLIP","Original GLM 5.3 Flash; S4.tab1 Animation Fidelity · CLIP","Qwen3 VL 235B A22B Thinking","Original Qwen3 VL 235B A22B Thinking; S4.tab1 Animation Fidelity · CLIP","Original Gemma 3 27B; S5.T3 CE","https://arxiv.org/html/2611.09326v1#S5.T3","Mistral Small 3.1 24B","Original Mistral Small 3.1 24B; S5.T3 CE","Original Mistral Small 3.2 24B; S5.T3 CE","Original GLM-4.5V; S5.T3 CE","Original GPT-4o mini; S5.T3 CE","Original GPT-4o; S5.T3 CE","Original GPT-4.1 mini; S5.T3 CE","Original GPT-4.1; S5.T3 CE","Claude Sonnet 3.7","Original Claude Sonnet 3.7; S5.T3 CE","Original o3-mini; S5.T3 CE","Original o3; S5.T3 CE","Claude Sonnet 3.7 Thinking","Original Claude Sonnet 3.7 Thinking; S5.T3 CE","K-EXAONE","Original K-EXAONE; S5.T5 All · A3","https://arxiv.org/html/2611.09223v1#S5.T5","Solar Open","Original Solar Open; S5.T5 All · A3","Original Aya Expanse 32B; S5.T5 All · A3","Original GLM-4.7; S5.T5 All · A3","Original Qwen3.5; S5.T5 All · A3","Original Claude Sonnet 5.5; S5.T5 All · A3","Original GPT-6.1 Sol; S5.T5 All · A3","Gemini-4-argon","Original Gemini-4-argon; A3.T1 Score","https://arxiv.org/html/2611.09222v1#A3.T1","Claude-opus-5.5","Original Claude-opus-5.5; A3.T1 Score","Original Gemini-3.8-flash; A3.T1 Score","Original GPT-6-sol; A3.T1 Score","Original GPT-5.6-sol; A3.T1 Score","Claude-opus-5","Original Claude-opus-5; A3.T1 Score","Original GPT-6-luna; A3.T1 Score","Original Gemini-3.5-flash-lite; A3.T1 Score","Original DeepSeek-V4-Pro; A3.T1 Score","Original Gemma-4-31B; A3.T1 Score","Original Qwen3.5-397B; A3.T1 Score","Claude-haiku-5.5","Original Claude-haiku-5.5; A3.T1 Score","Original GLM-5.3-flash; A3.T1 Score","Original Llama-4-Maverick; A3.T1 Score","Original Grok-4.3; A3.T1 Score","Gemma-SEA-LION-9B","Original Gemma-SEA-LION-9B; A3.T1 Score","Sailor2-8B","Original Sailor2-8B; A3.T1 Score","SeaLLMs-v3-7B","Original SeaLLMs-v3-7B; A3.T1 Score","Llama-SEA-LION-8B","Original Llama-SEA-LION-8B; A3.T1 Score","Gemini 2.5 Flash (off)","Original Gemini 2.5 Flash (off); S4.T2 Overall","https://arxiv.org/html/2611.08755v1#S4.T2","Gemini 2.5 Flash (on)","Original Gemini 2.5 Flash (on); S4.T2 Overall","Gemini 2.5 Pro (default)","Original Gemini 2.5 Pro (default); S4.T2 Overall","Gemini 3 Flash (low)","Original Gemini 3 Flash (low); S4.T2 Overall","Original Gemini 3 Flash (high); S4.T2 Overall","Original Gemini 3.1 Pro (low); S4.T2 Overall","Original Gemini 3.1 Pro (high); S4.T2 Overall","GPT-5.4 (low)","Original GPT-5.4 (low); S4.T2 Overall","Original GPT-5.4 (high); S4.T2 Overall","Claude Opus 4.6 (low)","Original Claude Opus 4.6 (low); S4.T2 Overall","Original Claude Opus 4.6 (high); S4.T2 Overall","Original Claude-Opus-5; S3.T4 +weights","https://arxiv.org/html/2611.08472v1#S3.T4","Original GPT-5.6-sol; S3.T4 +weights","DS-4.1-Flash","Original DS-4.1-Flash; S3.T4 +weights","Original GLM-5.3-Flash; S3.T4 +weights","Original Muse-Spark-1.3; S3.T4 +weights","Original GPT-5.6-terra; S3.T4 +weights","Original GPT-5.6-luna; S3.T4 +weights","Original Gemini 4 Argon; Frontier API models; S4.T2 Interaction quality Overall","https://arxiv.org/html/2611.08286v1#S4.T2","Original Gemini 3.8 Flash; Frontier API models; S4.T2 Interaction quality Overall","Original GPT-6.1 Sol; Frontier API models; S4.T2 Interaction quality Overall","Original GPT-6 Astra; Frontier API models; S4.T2 Interaction quality Overall","GPT-Realtime 2.1","Original GPT-Realtime 2.1; Real-time streaming API models; S4.T2 Interaction quality Overall","Original Gemini 3.8 Live; Real-time streaming API models; S4.T2 Interaction quality Overall","Muse Glimmer","Original Muse Glimmer; Open-weight non-streaming VLMs; S4.T2 Interaction quality Overall","Original Qwen3.8-27B; Open-weight non-streaming VLMs; S4.T2 Interaction quality Overall","InternVideo3-8B","Original InternVideo3-8B; Open-weight non-streaming VLMs; S4.T2 Interaction quality Overall","Kimi-VL-Thinking","Original Kimi-VL-Thinking; Open-weight non-streaming VLMs; S4.T2 Interaction quality Overall","Original Molmo2-8B; Open-weight non-streaming VLMs; S4.T2 Interaction quality Overall","Original Qwen3-Omni-30B; Open-weight streaming VLMs; S4.T2 Interaction quality Overall","Original MiniCPM-o 4.5; Open-weight streaming VLMs; S4.T2 Interaction quality Overall","Original StreamingVLM; Open-weight streaming VLMs; S4.T2 Interaction quality Overall","Proact-VL","Original Proact-VL; Open-weight streaming VLMs; S4.T2 Interaction quality Overall","Original Opus 5.5; S5.T3 Open conjectures Resolved","https://arxiv.org/html/2611.07962v1#S5.T3","Original GPT-6 Astra; S5.T3 Open conjectures Resolved","Original Kimi K3.1 Pro; S5.T3 Open conjectures Resolved","Original Kimi K3; S5.T3 Open conjectures Resolved","Original GLM-5.3; S5.T3 Open conjectures Resolved","Original Qwen3.8 Max (0902); S5.T3 Open conjectures Resolved","Original DeepSeek V4.1 Flash 1017; S5.T3 Open conjectures Resolved","Original Grok 4.6; S5.T3 Open conjectures Resolved","DeepSeek V4.1 Flash 0910","Original DeepSeek V4.1 Flash 0910; S5.T3 Open conjectures Resolved","Original DeepSeek V3.1; Defense: None; A5.T12 ASR All","https://arxiv.org/html/2611.07640v1#A5.T12","Original Kimi K2.5; A5.T12 ASR All","Original Sonnet 4; A5.T12 ASR All","Original Sonnet 4.5; A5.T12 ASR All","Original GLM-5; A5.T12 ASR All","Original DeepSeek V3.1; Defense: Spotlight; A5.T12 ASR All","Original Kimi K2.5; A5.T12 ASR All; row 9","Original Sonnet 4; A5.T12 ASR All; row 10","Original Sonnet 4.5; A5.T12 ASR All; row 11","Original GLM-5; A5.T12 ASR All; row 12","Original DeepSeek V3.1; Defense: Sandwich; A5.T12 ASR All","Original Kimi K2.5; A5.T12 ASR All; row 14","Original Sonnet 4; A5.T12 ASR All; row 15","Original Sonnet 4.5; A5.T12 ASR All; row 16","Original GLM-5; A5.T12 ASR All; row 17","Original DeepSeek V3.1; Defense: TH-generic; A5.T12 ASR All","Original Kimi K2.5; A5.T12 ASR All; row 19","Original Sonnet 4; A5.T12 ASR All; row 20","Original Sonnet 4.5; A5.T12 ASR All; row 21","Original GLM-5; A5.T12 ASR All; row 22","Original DeepSeek V3.1; Defense: TH-env; A5.T12 ASR All","Original Kimi K2.5; A5.T12 ASR All; row 24","Original Sonnet 4; A5.T12 ASR All; row 25","Original Sonnet 4.5; A5.T12 ASR All; row 26","Original GLM-5; A5.T12 ASR All; row 27","Original Llama-3.2-1B-Instruct; A6.T17 Avg.","https://arxiv.org/html/2611.07530v1#A6.T17","Original Llama-3.2-3B-Instruct; A6.T17 Avg.","Original Llama-3.1-8B-Instruct; A6.T17 Avg.","Llama-3.2-11B-Instruct","Original Llama-3.2-11B-Instruct; A6.T17 Avg.","Original Qwen2.5-0.5B-Instruct; A6.T17 Avg.","Original Qwen2.5-1.5B-Instruct; A6.T17 Avg.","Original Qwen2.5-3B-Instruct; A6.T17 Avg.","Original Qwen2.5-7B-Instruct; A6.T17 Avg.","Original Qwen2.5-32B-Instruct; A6.T17 Avg.","Original Qwen3.5-0.8B; A6.T17 Avg.","Original Qwen3.5-4B; A6.T17 Avg.","Original Qwen3.5-9B; A6.T17 Avg.","Original Qwen3.5-27B; A6.T17 Avg.","Original Qwen3.5-35B-A3B; A6.T17 Avg.","Gemma-3-1B-IT","Original Gemma-3-1B-IT; A6.T17 Avg.","Original Gemma-3-4B-IT; A6.T17 Avg.","Gemma-3-9B-IT","Original Gemma-3-9B-IT; A6.T17 Avg.","Original Gemma-3-12B-IT; A6.T17 Avg.","Original Gemma-3-27B-IT; A6.T17 Avg.","Original Mistral-7B-Instruct-v0.3; A6.T17 Avg.","Original Mistral-Nemo-Instruct-2407; A6.T17 Avg.","Original Mistral-Small-24B-Instruct-2501; A6.T17 Avg.","Original Mixtral-8x7B-Instruct-v0.1; A6.T17 Avg.","Original Yi-6B-Chat; A6.T17 Avg.","Yi-34B-Chat","Original Yi-34B-Chat; A6.T17 Avg.","Vicuna-7B-v1.5","Original Vicuna-7B-v1.5; A6.T17 Avg.","Vicuna-13B-v1.5","Original Vicuna-13B-v1.5; A6.T17 Avg.","Vicuna-33B-v1.3","Original Vicuna-33B-v1.3; A6.T17 Avg.","Original Zephyr-7B-Beta; A6.T17 Avg.","Dolphin-2.2.1-Mistral-7B","Original Dolphin-2.2.1-Mistral-7B; A6.T17 Avg.","Dolphin-2.9-Llama3.1-8B","Original Dolphin-2.9-Llama3.1-8B; A6.T17 Avg.","Dolphin-2.6-Mixtral-8x7B","Original Dolphin-2.6-Mixtral-8x7B; A6.T17 Avg.","Original ChatGLM3-6B; A6.T17 Avg.","Original GLM-4-9B-Chat; A6.T17 Avg.","Original OpenChat-3.5-7B-0106; A6.T17 Avg.","OpenChat-3.6-8B-20240618","Original OpenChat-3.6-8B-20240618; A6.T17 Avg.","Original Claude-Haiku-4.5; A6.T17 Avg.","Original Claude-Sonnet-4.5; A6.T17 Avg.","Original Claude-Opus-4.5; A6.T17 Avg.","Original GPT-4.1; A6.T17 Avg.","Original GPT-4o; A6.T17 Avg.","Original GPT-5.1; A6.T17 Avg.","Original Gemini-3-Pro; A6.T17 Avg.","Original Gemini-2.5-Pro; A6.T17 Avg.","Original Gemini-2.5-Flash; A6.T17 Avg.","Original GPT-6 Astra; Harness: Codex; S5.T2 Full History (C1)","https://arxiv.org/html/2611.07307v1#S5.T2","Original GPT-6 Sol; S5.T2 Full History (C1)","Original GPT-5.6 Sol; S5.T2 Full History (C1)","Original GPT-5.6 Terra; S5.T2 Full History (C1)","Original Opus 5.5; Harness: Claude Code; S5.T2 Full History (C1)","Original Sonnet 5; S5.T2 Full History (C1)","Original GPT-5.6 Sol; S5.T3 Task-Only","https://arxiv.org/html/2611.17267v1#S5.T3","Original Claude Opus 5; S5.T3 Task-Only","Original Claude Sonnet 5; S5.T3 Task-Only","Original Gemini 3.1 Pro Preview; S5.T3 Task-Only","Original Gemini 3.7 Flash; S5.T3 Task-Only","Original Grok 4.5; S5.T3 Task-Only","Original Qwen3.8-Max; S5.T3 Task-Only","Original Kimi K3; S5.T3 Task-Only","Original DeepSeek-V4-Flash-Vision-Exp; S5.T3 Task-Only","Structured vs free-form, Claude Opus 4.7","Original Structured vs free-form, Claude Opus 4.7; Astrophysics corpus; A7.T9 Total","https://arxiv.org/html/2611.17052v1#A7.T9","Structured vs free-form, GPT-5.5","Original Structured vs free-form, GPT-5.5; Astrophysics corpus; A7.T9 Total","Structured vs free-form, Gemini 3.1 Pro","Original Structured vs free-form, Gemini 3.1 Pro; Astrophysics corpus; A7.T9 Total","Structured vs free-form, Claude Code","Original Structured vs free-form, Claude Code; Astrophysics corpus; A7.T9 Total","Claude Code vs single call, free-form","Original Claude Code vs single call, free-form; Astrophysics corpus; A7.T9 Total","Claude Code vs single call, structured","Original Claude Code vs single call, structured; Astrophysics corpus; A7.T9 Total","Skepthical vs Claude Code, structured","Original Skepthical vs Claude Code, structured; Astrophysics corpus; A7.T9 Total","Skepthical vs GPT-5.5, structured","Original Skepthical vs GPT-5.5, structured; Astrophysics corpus; A7.T9 Total","Original Structured vs free-form, Claude Opus 4.7; Machine-learning corpus; A7.T9 Total","Original Structured vs free-form, GPT-5.5; Machine-learning corpus; A7.T9 Total","Original Structured vs free-form, Gemini 3.1 Pro; Machine-learning corpus; A7.T9 Total","Original Structured vs free-form, Claude Code; Machine-learning corpus; A7.T9 Total","Original Claude Code vs single call, free-form; Machine-learning corpus; A7.T9 Total","Original Claude Code vs single call, structured; Machine-learning corpus; A7.T9 Total","Original Skepthical vs Claude Code, structured; Machine-learning corpus; A7.T9 Total","Original Skepthical vs GPT-5.5, structured; Machine-learning corpus; A7.T9 Total","Novara Agent System [locally deployed]","Original Novara Agent System [locally deployed]; S4.T1 Avg. Score","https://arxiv.org/html/2611.16991v1#S4.T1","DeepSeek V4 Pro [max]","Original DeepSeek V4 Pro [max]; Harness: Claude Code; S4.T1 Avg. Score","DeepSeek V4.1 Flash [max]","Original DeepSeek V4.1 Flash [max]; Harness: Claude Code; S4.T1 Avg. Score","Original DeepSeek V4.1 Flash [max]; Harness: DSH; S4.T1 Avg. Score","GLM-5.2 FP8 [max]","Original GLM-5.2 FP8 [max]; Harness: OpenCode; S4.T1 Avg. Score","Original GLM-5.3 [max]; Harness: OpenCode; S4.T1 Avg. Score","Original GLM-5.3 [max]; Harness: InternBootcamp; S4.T1 Avg. Score","GLM-5.3 Flash [max]","Original GLM-5.3 Flash [max]; Harness: OpenCode; S4.T1 Avg. Score","Original GLM-5.3 Flash [max]; Harness: InternBootcamp; S4.T1 Avg. Score","Kimi-K2.6 [max]","Original Kimi-K2.6 [max]; Harness: Kimi Code; S4.T1 Avg. Score","Kimi-K3 [max]","Original Kimi-K3 [max]; Harness: Kimi Code; S4.T1 Avg. Score","Qwen3.8 Flash Next [xhigh]","Original Qwen3.8 Flash Next [xhigh]; Harness: Qwen Code; S4.T1 Avg. Score","GPT-6 Astra [max]","Original GPT-6 Astra [max]; Harness: Codex; S4.T1 Avg. Score","GPT-6.1 Sol [max]","Original GPT-6.1 Sol [max]; Harness: Codex; S4.T1 Avg. Score","Original GPT-6 Astra; Effort: max; S3.T1 Acc.","https://arxiv.org/html/2611.16861v1#S3.T1","Original GPT-6 Astra; Effort: xhigh; S3.T1 Acc.","Original GPT-6 Astra; Effort: high; S3.T1 Acc.","Original GPT-6 Astra; Effort: medium; S3.T1 Acc.","Original GPT-6 Astra; Effort: low; S3.T1 Acc.","Original GPT-6.1 Sol; Effort: max; S3.T1 Acc.","Original GPT-6.1 Sol; Effort: xhigh; S3.T1 Acc.","Original GPT-6.1 Sol; Effort: high; S3.T1 Acc.","Original GPT-6.1 Sol; Effort: medium; S3.T1 Acc.","Original GPT-6.1 Sol; Effort: low; S3.T1 Acc.","Original Claude Opus 5.5; Effort: max; S3.T1 Acc.","Original Claude Opus 5.5; Effort: xhigh; S3.T1 Acc.","Original Claude Opus 5.5; Effort: high; S3.T1 Acc.","Original Claude Opus 5.5; Effort: medium; S3.T1 Acc.","Original Claude Opus 5.5; Effort: low; S3.T1 Acc.","Original GPT-6 Luna; Effort: max; S3.T1 Acc.","Original GPT-6 Luna; Effort: xhigh; S3.T1 Acc.","Original GPT-6 Luna; Effort: high; S3.T1 Acc.","Original GPT-6 Luna; Effort: medium; S3.T1 Acc.","Original GPT-6 Luna; Effort: low; S3.T1 Acc.","Original GPT-6 Luna; Effort: none; S3.T1 Acc.","Original Claude Sonnet 5.5; Effort: max; S3.T1 Acc.","Original Claude Sonnet 5.5; Effort: xhigh; S3.T1 Acc.","Original Claude Sonnet 5.5; Effort: high; S3.T1 Acc.","Original Claude Sonnet 5.5; Effort: medium; S3.T1 Acc.","Original Claude Sonnet 5.5; Effort: low; S3.T1 Acc.","Original DeepSeek V4.1 Flash; Effort: max; S3.T1 Acc.","Original Qwen 3.8 Max; Effort: xhigh; S3.T1 Acc.","Original Kimi K3.1 Pro; Effort: max; S3.T1 Acc.","Original GPT-5; S4.T1 Avg Agree","https://arxiv.org/html/2611.16767v1#S4.T1","Original Gemini-3-Flash; S4.T1 Avg Agree","Qwen3-VL (235B)","Original Qwen3-VL (235B); S4.T1 Avg Agree","Qwen3-VL (8B)","Original Qwen3-VL (8B); S4.T1 Avg Agree","Qwen3-VL (32B)","Original Qwen3-VL (32B); S4.T1 Avg Agree","Ovis2.5 (9B)","Original Ovis2.5 (9B); S4.T1 Avg Agree","InternVL3.5 (241B)","Original InternVL3.5 (241B); S4.T1 Avg Agree","InternVL3.5 (38B)","Original InternVL3.5 (38B); S4.T1 Avg Agree","Tarsier2 (7B)","Original Tarsier2 (7B); S4.T1 Avg Agree","MiniCPM4.5 (8B)","Original MiniCPM4.5 (8B); S4.T1 Avg Agree","KeyeVL (8B)","Original KeyeVL (8B); S4.T1 Avg Agree","Original InternVL3.5 (8B); S4.T1 Avg Agree","InternVideo2.5 (8B)","Original InternVideo2.5 (8B); S4.T1 Avg Agree","IXC-2.5 (7B)","Original IXC-2.5 (7B); S4.T1 Avg Agree","GLM-4.1V-Thinking (9B)","Original GLM-4.1V-Thinking (9B); S4.T1 Avg Agree","VITA-1.5 (8B)","Original VITA-1.5 (8B); S4.T1 Avg Agree","LLaVA-OV-1.5 (8B)","Original LLaVA-OV-1.5 (8B); S4.T1 Avg Agree","LLaVA-Next-Video (7B)","Original LLaVA-Next-Video (7B); S4.T1 Avg Agree","LlaVA-OV-RS (7B)","Original LlaVA-OV-RS (7B); Specialists; S4.T1 Avg Agree","CrashChat (7B)","Original CrashChat (7B); Specialists; S4.T1 Avg Agree","DriveLMM-o1 (8B)","Original DriveLMM-o1 (8B); Specialists; S4.T1 Avg Agree","DriveMM (8B)","Original DriveMM (8B); Specialists; S4.T1 Avg Agree","MiniInternVL2-BDD (4B)","Original MiniInternVL2-BDD (4B); Specialists; S4.T1 Avg Agree","MiniInternVL2-DLM (4B)","Original MiniInternVL2-DLM (4B); Specialists; S4.T1 Avg Agree","Dolphins (7B)","Original Dolphins (7B); Specialists; S4.T1 Avg Agree","Original GPT-4o; A7.T24 opentom Acc","https://arxiv.org/html/2611.16662v1#A7.T24","Original GPT-3.5-Turbo; A7.T24 opentom Acc","Original GPT-OSS-120B; A7.T24 opentom Acc","Llama-8B","Original Llama-8B; A7.T24 opentom Acc","Llama-405B","Original Llama-405B; A7.T24 opentom Acc","Original Llama-4-Maverick-17B; A7.T24 opentom Acc","Original Mistral-7B-Instruct; A7.T24 opentom Acc","Original Mixtral-8x22B; A7.T24 opentom Acc","Original Qwen3-235B-A22B; A7.T24 opentom Acc","Original QwQ-32B; A7.T24 opentom Acc","Original DeepSeek-R1; A7.T24 opentom Acc","Phi-3-Medium-4K","Original Phi-3-Medium-4K; A7.T24 opentom Acc","Original DeepSeek-V3.1; A7.T24 opentom Acc","Original DeepSeek-V3.2; A7.T24 opentom Acc","Original Kimi-K2-Thinking; A7.T24 opentom Acc","MiniMax-M2.1","Original MiniMax-M2.1; A7.T24 opentom Acc","Original MiniMax-M2.5; A7.T24 opentom Acc","Original Qwen3.5-397B-A17B; A7.T24 opentom Acc","Opus 5.5 + CC","Original Opus 5.5 + CC; S5.T5 Mid-training Acc","https://arxiv.org/html/2611.16591v1#S5.T5","GPT-6.1 Sol + Codex","Original GPT-6.1 Sol + Codex; S5.T5 Mid-training Acc","Opus 5 + CC","Original Opus 5 + CC; S5.T5 Mid-training Acc","DS V4.1 Flash + Codex","Original DS V4.1 Flash + Codex; S5.T5 Mid-training Acc","GLM 5.3 + CC","Original GLM 5.3 + CC; S5.T5 Mid-training Acc","M3 + CC","Original M3 + CC; S5.T5 Mid-training Acc","Kimi K3 + CC","Original Kimi K3 + CC; S5.T5 Mid-training Acc","DS V4.1 Flash + CC","Original DS V4.1 Flash + CC; S5.T5 Mid-training Acc","Kimi K3 + Codex","Original Kimi K3 + Codex; S5.T5 Mid-training Acc","GLM 5.3 + Codex","Original GLM 5.3 + Codex; S5.T5 Mid-training Acc","M3 + Codex","Original M3 + Codex; S5.T5 Mid-training Acc","GPT-OSS-120b (110)","Original GPT-OSS-120b (110); Agent: OpenHands; S6.T7 MM","https://arxiv.org/html/2611.16588v1#S6.T7","Qwen-3.5 (96)","Original Qwen-3.5 (96); Agent: OpenHands; S6.T7 MM","Kimi-2.5 (94)","Original Kimi-2.5 (94); Agent: OpenHands; S6.T7 MM","GLM-5.2 (71)","Original GLM-5.2 (71); Agent: OpenHands; S6.T7 MM","GPT-5.3 (70)","Original GPT-5.3 (70); Agent: OpenHands; S6.T7 MM","Grok-4.6 (67)","Original Grok-4.6 (67); Agent: OpenHands; S6.T7 MM","Gemini-3.1 (66)","Original Gemini-3.1 (66); Agent: OpenHands; S6.T7 MM","Sonnet-4.6 (63)","Original Sonnet-4.6 (63); Agent: OpenHands; S6.T7 MM","Opus-4.6 (53)","Original Opus-4.6 (53); Agent: OpenHands; S6.T7 MM","GPT-OSS-120b (114)","Original GPT-OSS-120b (114); Agent: SWE-agent; S6.T7 MM","Qwen-3.5 (97)","Original Qwen-3.5 (97); Agent: SWE-agent; S6.T7 MM","Original Kimi-2.5 (94); Agent: SWE-agent; S6.T7 MM","GLM-5.2 (73)","Original GLM-5.2 (73); Agent: SWE-agent; S6.T7 MM","GPT-5.3 (72)","Original GPT-5.3 (72); Agent: SWE-agent; S6.T7 MM","Grok-4.6 (68)","Original Grok-4.6 (68); Agent: SWE-agent; S6.T7 MM","Original Gemini-3.1 (66); Agent: SWE-agent; S6.T7 MM","Sonnet-4.6 (64)","Original Sonnet-4.6 (64); Agent: SWE-agent; S6.T7 MM","Opus-4.6 (56)","Original Opus-4.6 (56); Agent: SWE-agent; S6.T7 MM","DS-Flash","Original DS-Flash; Method: FA; A2.T16a Overall (177)","https://arxiv.org/html/2611.16493v1#A2.T16a","Original DS-Flash; Method: miniCoT; A2.T16a Overall (177)","Original DS-Flash; Method: FHA; A2.T16a Overall (177)","Original DS-Flash; Method: WA; A2.T16a Overall (177)","DS-Pro","Original DS-Pro; Method: FA; A2.T16a Overall (177)","Original DS-Pro; Method: miniCoT; A2.T16a Overall (177)","Original DS-Pro; Method: FHA; A2.T16a Overall (177)","Original DS-Pro; Method: WA; A2.T16a Overall (177)","Qwen-Flash","Original Qwen-Flash; Method: FA; A2.T16a Overall (177)","Original Qwen-Flash; Method: miniCoT; A2.T16a Overall (177)","Original Qwen-Flash; Method: FHA; A2.T16a Overall (177)","Original Qwen-Flash; Method: WA; A2.T16a Overall (177)","Qwen-max","Original Qwen-max; Method: FA; A2.T16a Overall (177)","Original Qwen-max; Method: miniCoT; A2.T16a Overall (177)","Original Qwen-max; Method: FHA; A2.T16a Overall (177)","Original Qwen-max; Method: WA; A2.T16a Overall (177)","Grok-low","Original Grok-low; Method: FA; A2.T16a Overall (177)","Original Grok-low; Method: miniCoT; A2.T16a Overall (177)","Original Grok-low; Method: FHA; A2.T16a Overall (177)","Original Grok-low; Method: WA; A2.T16a Overall (177)","Grok-high","Original Grok-high; Method: FA; A2.T16a Overall (177)","Original Grok-high; Method: miniCoT; A2.T16a Overall (177)","Original Grok-high; Method: FHA; A2.T16a Overall (177)","Original Grok-high; Method: WA; A2.T16a Overall (177)","GPT-luna","Original GPT-luna; Method: FA; A2.T16a Overall (177)","Original GPT-luna; Method: miniCoT; A2.T16a Overall (177)","Original GPT-luna; Method: FHA; A2.T16a Overall (177)","Original GPT-luna; Method: WA; A2.T16a Overall (177)","GPT-sol","Original GPT-sol; Method: FA; A2.T16a Overall (177)","Original GPT-sol; Method: miniCoT; A2.T16a Overall (177)","Original GPT-sol; Method: FHA; A2.T16a Overall (177)","Original GPT-sol; Method: WA; A2.T16a Overall (177)","Qwen3-VL-Instruct","Original Qwen3-VL-Instruct; S5.T2 RelCap- F1 all","https://arxiv.org/html/2611.16413v1#S5.T2","Original InternVL3.5; S5.T2 RelCap- F1 all","Ovis2.5","Original Ovis2.5; S5.T2 RelCap- F1 all","Gemma-3-it","Original Gemma-3-it; S5.T2 RelCap- F1 all","LLaVA-OneVision-1.5 -Instruct","Original LLaVA-OneVision-1.5 -Instruct; S5.T2 RelCap- F1 all","Original Qwen3-VL-Instruct; S5.T2 RelCap- F1 all; row 7","Original InternVL3.5; S5.T2 RelCap- F1 all; row 8","LLaVA-OneVision-2 -Instruct","Original LLaVA-OneVision-2 -Instruct; S5.T2 RelCap- F1 all","LocateAnything","Original LocateAnything; S5.T2 RelCap- F1 all","GeoChat","Original GeoChat; S5.T2 RelCap- F1 all","GeoPixel","Original GeoPixel; S5.T2 RelCap- F1 all","EarthDial","Original EarthDial; S5.T2 RelCap- F1 all","Original Qwen3.8-Max; S5.T2 RelCap- F1 all","Original GPT-6-Luna; S5.T2 RelCap- F1 all","Original Gemini-3.8-Flash; S5.T2 RelCap- F1 all","Original claude-opus-5; S4.T2 for an AI","https://arxiv.org/html/2611.16318v1#S4.T2","Original claude-sonnet-5; S4.T2 for an AI","Original gpt-5.6-sol; S4.T2 for an AI","Original qwen3-8b; S4.T2 for an AI","llama-4-maverick","Original llama-4-maverick; S4.T2 for an AI","Original gemini-3.8-flash; S4.T2 for an AI","Original deepseek-v4-pro-0813; S4.T2 for an AI","Original kimi-k3; S4.T2 for an AI","Original glm-5.2; S4.T2 for an AI","GPT-6 Astra max","Original GPT-6 Astra max; S4.T2 Overall","https://arxiv.org/html/2611.15789v1#S4.T2","GPT-6.1 Sol max","Original GPT-6.1 Sol max; S4.T2 Overall","Gemini 4 Argon high","Original Gemini 4 Argon high; S4.T2 Overall","Opus 5.5 high","Original Opus 5.5 high; S4.T2 Overall","GPT-5.6 Sol max","Original GPT-5.6 Sol max; S4.T2 Overall","DeepSeek V4.1 Flash max","Original DeepSeek V4.1 Flash max; S4.T2 Overall","Kimi K3 xhigh","Original Kimi K3 xhigh; S4.T2 Overall","Original Gemini 3.8 Flash high; S4.T2 Overall","GLM 5.3 Flash high","Original GLM 5.3 Flash high; S4.T2 Overall","Doubao Seed 2.1 Pro max","Original Doubao Seed 2.1 Pro max; S4.T2 Overall","Muse Spark 1.3 xhigh","Original Muse Spark 1.3 xhigh; S4.T2 Overall","Llama 3.3 70B, MC → OE","Original Llama 3.3 70B, MC → OE; Format; A9.T9 h","https://arxiv.org/html/2611.15718v1#A9.T9","Llama 3.3 70B, MC → TS","Original Llama 3.3 70B, MC → TS; Format; A9.T9 h","Llama 3.3 70B, OE → TS","Original Llama 3.3 70B, OE → TS; Format; A9.T9 h","Mistral Small 3.1, MC → OE","Original Mistral Small 3.1, MC → OE; Format; A9.T9 h","Mistral Small 3.1, MC → TS","Original Mistral Small 3.1, MC → TS; Format; A9.T9 h","Mistral Small 3.1, OE → TS","Original Mistral Small 3.1, OE → TS; Format; A9.T9 h","Llama 4 Maverick, MC → OE","Original Llama 4 Maverick, MC → OE; Format; A9.T9 h","Llama 4 Maverick, MC → TS","Original Llama 4 Maverick, MC → TS; Format; A9.T9 h","Llama 4 Maverick, OE → TS","Original Llama 4 Maverick, OE → TS; Format; A9.T9 h","Granite 4.0 H, MC → OE","Original Granite 4.0 H, MC → OE; Format; A9.T9 h","Granite 4.0 H, MC → TS","Original Granite 4.0 H, MC → TS; Format; A9.T9 h","Granite 4.0 H, OE → TS","Original Granite 4.0 H, OE → TS; Format; A9.T9 h","Mistral Medium, MC → OE","Original Mistral Medium, MC → OE; Format; A9.T9 h","Mistral Medium, MC → TS","Original Mistral Medium, MC → TS; Format; A9.T9 h","Mistral Medium, OE → TS","Original Mistral Medium, OE → TS; Format; A9.T9 h","Original Llama 3.3 70B; Standard vs. hard multiple-choice; A9.T9 h","Original Mistral Small 3.1; Standard vs. hard multiple-choice; A9.T9 h","Original Llama 4 Maverick; Standard vs. hard multiple-choice; A9.T9 h","Granite 4.0 H","Original Granite 4.0 H; Standard vs. hard multiple-choice; A9.T9 h","Mistral Medium","Original Mistral Medium; Standard vs. hard multiple-choice; A9.T9 h","Original Llama 3.3 70B; Other vs. diagnostic multiple-choice; A9.T9 h","Original Mistral Small 3.1; Other vs. diagnostic multiple-choice; A9.T9 h","Original Llama 4 Maverick; Other vs. diagnostic multiple-choice; A9.T9 h","Original Granite 4.0 H; Other vs. diagnostic multiple-choice; A9.T9 h","Original Mistral Medium; Other vs. diagnostic multiple-choice; A9.T9 h","Original Llama 3.3 70B; OE multi-hop vs. troubleshooting; A9.T9 h","Original Mistral Small 3.1; OE multi-hop vs. troubleshooting; A9.T9 h","Original Llama 4 Maverick; OE multi-hop vs. troubleshooting; A9.T9 h","Original Granite 4.0 H; OE multi-hop vs. troubleshooting; A9.T9 h","Original Mistral Medium; OE multi-hop vs. troubleshooting; A9.T9 h","Gemma 12B","Original Gemma 12B; Static Agents; Global Registry: Reflex; A4.T6 Out of context","https://arxiv.org/html/2611.15521v1#A4.T6","Original Gemma 4 26B A4B; Static Agents; Global Registry: Reflex; A4.T6 Out of context","Original Muse Glimmer 30B; Static Agents; Global Registry: Reflex; A4.T6 Out of context","Original Qwen3.5-0.8B; Static Agents; Global Registry: Reflex; A4.T6 Out of context","Qwen3.5 35B A3B","Original Qwen3.5 35B A3B; Static Agents; Global Registry: Reflex; A4.T6 Out of context","QWen3.8 27B","Original QWen3.8 27B; Static Agents; Global Registry: Reflex; A4.T6 Out of context","Original Gemma 12B; Static Agents; Global Registry: Reflex + History; A4.T6 Out of context","Original Gemma 4 26B A4B; Static Agents; Global Registry: Reflex + History; A4.T6 Out of context","Original Muse Glimmer 30B; Static Agents; Global Registry: Reflex + History; A4.T6 Out of context","Original Qwen3.5-0.8B; Static Agents; Global Registry: Reflex + History; A4.T6 Out of context","Original Qwen3.5 35B A3B; Static Agents; Global Registry: Reflex + History; A4.T6 Out of context","Original QWen3.8 27B; Static Agents; Global Registry: Reflex + History; A4.T6 Out of context","Original Gemma 12B; Static Agents; Global Registry: Broadcast; A4.T6 Out of context","Original Gemma 4 26B A4B; Static Agents; Global Registry: Broadcast; A4.T6 Out of context","Original Muse Glimmer 30B; Static Agents; Global Registry: Broadcast; A4.T6 Out of context","Original Qwen3.5-0.8B; Static Agents; Global Registry: Broadcast; A4.T6 Out of context","Original Qwen3.5 35B A3B; Static Agents; Global Registry: Broadcast; A4.T6 Out of context","Original QWen3.8 27B; Static Agents; Global Registry: Broadcast; A4.T6 Out of context","Original Gemma 12B; Static Agents; Global Registry: Oracle Route (Entry Only); A4.T6 Out of context","Original Gemma 4 26B A4B; Static Agents; Global Registry: Oracle Route (Entry Only); A4.T6 Out of context","Original Muse Glimmer 30B; Static Agents; Global Registry: Oracle Route (Entry Only); A4.T6 Out of context","Original Gemma 12B; Static Agents; Global Registry: Oracle Route (All Agents); A4.T6 Out of context","Original Gemma 4 26B A4B; Static Agents; Global Registry: Oracle Route (All Agents); A4.T6 Out of context","Original Muse Glimmer 30B; Static Agents; Global Registry: Oracle Route (All Agents); A4.T6 Out of context","Original Gemma 12B; Self-Improved Agents; Global Registry: Numeric Q-Routing, Learn w/ All; A4.T6 Out of context","Original Gemma 4 26B A4B; Self-Improved Agents; Global Registry: Numeric Q-Routing, Learn w/ All; A4.T6 Out of context","Original Muse Glimmer 30B; Self-Improved Agents; Global Registry: Numeric Q-Routing, Learn w/ All; A4.T6 Out of context","Original Gemma 12B; Self-Improved Agents; Global Registry: Success-Trajectory Memory; A4.T6 Out of context","Original Gemma 4 26B A4B; Self-Improved Agents; Global Registry: Success-Trajectory Memory; A4.T6 Out of context","Original Muse Glimmer 30B; Self-Improved Agents; Global Registry: Success-Trajectory Memory; A4.T6 Out of context","Original Gemma 12B; Self-Improved Agents; Global Registry: Location Learning; A4.T6 Out of context","Original Gemma 4 26B A4B; Self-Improved Agents; Global Registry: Location Learning; A4.T6 Out of context","Original Muse Glimmer 30B; Self-Improved Agents; Global Registry: Location Learning; A4.T6 Out of context","Original Gemma 12B; Self-Improved Agents; Global Registry: LLM Q-Learning (Our); A4.T6 Out of context","Original Gemma 4 26B A4B; Self-Improved Agents; Global Registry: LLM Q-Learning (Our); A4.T6 Out of context","Original Muse Glimmer 30B; Self-Improved Agents; Global Registry: LLM Q-Learning (Our); A4.T6 Out of context","Original Gemma 12B; Self-Improved Agents; Global Registry: Warmup 2 Hop + LLM Q-Learning (Our); A4.T6 Out of context","Original Gemma 4 26B A4B; Self-Improved Agents; Global Registry: Warmup 2 Hop + LLM Q-Learning (Our); A4.T6 Out of context","Original Muse Glimmer 30B; Self-Improved Agents; Global Registry: Warmup 2 Hop + LLM Q-Learning (Our); A4.T6 Out of context","Original Qwen3.5-0.8B; Self-Improved Agents; Global Registry: Warmup 2 Hop + LLM Q-Learning (Our); A4.T6 Out of context","Original Qwen3.5 35B A3B; Self-Improved Agents; Global Registry: Warmup 2 Hop + LLM Q-Learning (Our); A4.T6 Out of context","Original QWen3.8 27B; Self-Improved Agents; Global Registry: Warmup 2 Hop + LLM Q-Learning (Our); A4.T6 Out of context","Original Qwen3.5-0.8B; Trained Agents; Global Registry: SFT; A4.T6 Out of context","Original Qwen3.5-2B; Trained Agents; Global Registry: SFT; A4.T6 Out of context","Original Qwen3.5-4B; Trained Agents; Global Registry: SFT; A4.T6 Out of context","Original Qwen3.5-0.8B; Trained Agents; Global Registry: GRPO; A4.T6 Out of context","Original Qwen3.5-2B; Trained Agents; Global Registry: GRPO; A4.T6 Out of context","Original Qwen3.5-4B; Trained Agents; Global Registry: GRPO; A4.T6 Out of context","Original Qwen3.5-0.8B; Trained Agents; Global Registry: OPSD; A4.T6 Out of context","Original Qwen3.5-2B; Trained Agents; Global Registry: OPSD; A4.T6 Out of context","Original Qwen3.5-4B; Trained Agents; Global Registry: OPSD; A4.T6 Out of context","Original Qwen3.5-0.8B; Trained Agents; Global Registry: Reflex; A4.T6 Out of context","Original Qwen3.5 35B A3B; Trained Agents; Global Registry: Reflex; A4.T6 Out of context","Original QWen3.8 27B; Trained Agents; Global Registry: Reflex; A4.T6 Out of context","Gemma3n-E4B","Original Gemma3n-E4B; A6.T7 A all %","https://arxiv.org/html/2611.15433v1#A6.T7","Original Gemma3n-E4B; A6.T7 A all %; row 3","Original Gemma3n-E4B; A6.T7 A all %; row 4","Original Gemma3n-E4B; A6.T7 A all %; row 5","Original Gemma4-E4B; A6.T7 A all %","Original Gemma4-E4B; A6.T7 A all %; row 7","Original Gemma4-E4B; A6.T7 A all %; row 8","Original Gemma4-E4B; A6.T7 A all %; row 9","Original Qwen3.5-27B; A6.T7 A all %","Original Qwen3.5-27B; A6.T7 A all %; row 11","Original Qwen3.5-27B; A6.T7 A all %; row 12","Original Qwen3.5-27B; A6.T7 A all %; row 13","Original Qwen3.5-9B; A6.T7 A all %","Original Qwen3.5-9B; A6.T7 A all %; row 15","Original Qwen3.5-9B; A6.T7 A all %; row 16","Original Qwen3.5-9B; A6.T7 A all %; row 17","Original Qwen3.5-4B; A6.T7 A all %","Original Qwen3.5-4B; A6.T7 A all %; row 19","Original Qwen3.5-4B; A6.T7 A all %; row 20","Original Qwen3.5-4B; A6.T7 A all %; row 21","Original GPT-5.5; S5.T3 Open-ended","https://arxiv.org/html/2611.15342v1#S5.T3","Original GPT-5.6-Sol; S5.T3 Open-ended","Original GPT-5.6-Terra; S5.T3 Open-ended","Original GPT-OSS-20B; S5.T3 Open-ended","Qwen3-8B Chat","Original Qwen3-8B Chat; S5.T3 Open-ended","Qwen3-8B Base","Original Qwen3-8B Base; S5.T3 Open-ended","MechGPT","Original MechGPT; S5.T3 Open-ended","Original Qwen2.5-VL-3B; S11.T8 CaseScope-Atlas All","https://arxiv.org/html/2611.15325v1#S11.T8","Original Qwen2.5-VL-7B; S11.T8 CaseScope-Atlas All","Original Qwen2.5-VL-32B; S11.T8 CaseScope-Atlas All","Original Qwen2-VL-7B; S11.T8 CaseScope-Atlas All","Original InternVL2.5-8B; S11.T8 CaseScope-Atlas All","Original InternVL3-8B; S11.T8 CaseScope-Atlas All","Original InternVL3.5-8B; S11.T8 CaseScope-Atlas All","Original Qwen3-VL-8B; S11.T8 CaseScope-Atlas All","Original Qwen3.5-9B; S11.T8 CaseScope-Atlas All","Original MiMo-VL-7B; S11.T8 CaseScope-Atlas All","Original Qwen3-VL-8B-Thinking; S11.T8 CaseScope-Atlas All","Original GLM-4.1V-9B-Thinking; S11.T8 CaseScope-Atlas All","Original MiniCPM-V-4.5; S11.T8 CaseScope-Atlas All","Original InternVL3.5-14B; S11.T8 CaseScope-Atlas All","Original HuatuoGPT-Vision-7B; S11.T8 CaseScope-Atlas All","HuatuoGPT-Vision-7B (Q2.5-VL)","Original HuatuoGPT-Vision-7B (Q2.5-VL); S11.T8 CaseScope-Atlas All","Original Lingshu-7B; S11.T8 CaseScope-Atlas All","OctoMed-7B","Original OctoMed-7B; S11.T8 CaseScope-Atlas All","OctoMed-8B","Original OctoMed-8B; S11.T8 CaseScope-Atlas All","Original MedGemma-1.5-4B; S11.T8 CaseScope-Atlas All","Original Hulu-Med-7B; S11.T8 CaseScope-Atlas All","Original LLaVA-Med-v1.5; S11.T8 CaseScope-Atlas All","Quilt-LLaVA-7B","Original Quilt-LLaVA-7B; S11.T8 CaseScope-Atlas All","PathGen-LLaVA-13B","Original PathGen-LLaVA-13B; S11.T8 CaseScope-Atlas All","Original CLOVER; S11.T8 CaseScope-Atlas All","Original Patho-R1-3B; S11.T8 CaseScope-Atlas All","Original Patho-R1-7B; S11.T8 CaseScope-Atlas All","ScaleReasoner-R1","Original ScaleReasoner-R1; S11.T8 CaseScope-Atlas All","CLIP B/32 (zero-shot)","Original CLIP B/32 (zero-shot); S5.T3 I2T · R@10","https://arxiv.org/html/2611.15314v1#S5.T3","CLIP B/16 (zero-shot)","Original CLIP B/16 (zero-shot); S5.T3 I2T · R@10","BLIP (zero-shot)","Original BLIP (zero-shot); S5.T3 I2T · R@10","SigLIP2 (zero-shot)","Original SigLIP2 (zero-shot); S5.T3 I2T · R@10","Original SigLIP2 (zero-shot); S5.T3 I2T · R@10; row 7","CLIP B/16 + place loss","Original CLIP B/16 + place loss; S5.T3 I2T · R@10","Original CLIP B/16 + place loss; S5.T3 I2T · R@10; row 9","BLIP + MS adaptation","Original BLIP + MS adaptation; S5.T3 I2T · R@10","Original BLIP + MS adaptation; S5.T3 I2T · R@10; row 11","SigLIP2 + MS","Original SigLIP2 + MS; S5.T3 I2T · R@10","Original SigLIP2 + MS; S5.T3 I2T · R@10; row 13","SigLIP2 + place loss","Original SigLIP2 + place loss; S5.T3 I2T · R@10","Original SigLIP2 + place loss; S5.T3 I2T · R@10; row 15","Original GPT-6 Astra; A1.T2 Versions","https://arxiv.org/html/2611.15262v1#A1.T2","Original GPT-6.1 Sol; A1.T2 Versions","Original DeepSeek-V4-Flash; A1.T2 Versions","Original Kimi K2.6; A1.T2 Versions","Original Gemini 3.8 Flash; A1.T2 Versions","Grok4.7","Original Grok4.7; A1.T2 Versions","Original Claude Opus 5.5; A1.T2 Versions","Original Qwen3.8-flash; A1.T2 Versions","Original GLM-5.3-flash; A1.T2 Versions","Original Claude Sonnet 5.5; A1.T2 Versions","Original Qwen3.5-27B; A1.T2 Versions","Original GPT-6 Luna; A1.T2 Versions","Original Qwen3.5-4B; A1.T2 Versions","Original Qwen3.5-122B-A10B; A1.T2 Versions","Original Qwen3.5-9B; A1.T2 Versions","Original GPT-5.5; S3.T2 Driving Reasoning Avg","https://arxiv.org/html/2611.15225v1#S3.T2","Original GPT-5; S3.T2 Driving Reasoning Avg","Original GPT-4o; S3.T2 Driving Reasoning Avg","Original Gemini-2.5-Pro; S3.T2 Driving Reasoning Avg","Original Gemini-3-Pro; S3.T2 Driving Reasoning Avg","Original Claude Opus 4.6; S3.T2 Driving Reasoning Avg","Original Qwen3.5-397B-A17B; Open Source MoE Models; S3.T2 Driving Reasoning Avg","ERNIE-4.5-VL-28B-A3B-Thinking","Original ERNIE-4.5-VL-28B-A3B-Thinking; Open Source MoE Models; S3.T2 Driving Reasoning Avg","Original Qwen3-VL-30B-A3B-Thinking; Open Source MoE Models; S3.T2 Driving Reasoning Avg","Original InternVL3-78B; Open Source Models > 7B; S3.T2 Driving Reasoning Avg","Original Qwen2.5-VL-72B; Open Source Models > 7B; S3.T2 Driving Reasoning Avg","Original InternVL3-38B; Open Source Models > 7B; S3.T2 Driving Reasoning Avg","Original Qwen2.5-VL-32B; Open Source Models > 7B; S3.T2 Driving Reasoning Avg","Original Qwen3.5-27B; Open Source Models > 7B; S3.T2 Driving Reasoning Avg","Original GLM-4.1V-9B-Thinking; Open Source Models > 7B; S3.T2 Driving Reasoning Avg","Original Video-R1-7B; Open Source Models > 7B; S3.T2 Driving Reasoning Avg","LLaVA-OneVision-Qwen2-7B-OV","Original LLaVA-OneVision-Qwen2-7B-OV; Open Source Models > 7B; S3.T2 Driving Reasoning Avg","VGLM-QA-VGGT-8B","Original VGLM-QA-VGGT-8B; Open Source Models > 7B; S3.T2 Driving Reasoning Avg","ReCogDrive-VLM-8B","Original ReCogDrive-VLM-8B; Specialist Models; S3.T2 Driving Reasoning Avg","Original Cosmos-Reason1-7B; Specialist Models; S3.T2 Driving Reasoning Avg","Original Cosmos-Reason2-8B; Specialist Models; S3.T2 Driving Reasoning Avg","MiMoEmbodied-7B","Original MiMoEmbodied-7B; Specialist Models; S3.T2 Driving Reasoning Avg","Original MiMo-VL-7B-SFT; Specialist Models; S3.T2 Driving Reasoning Avg","GeoLLaVA-8K","Original GeoLLaVA-8K; Remote Sensing MLLMs; A3.T2 Reasoning · Mean","https://arxiv.org/html/2611.14953v1#A3.T2","VHM","Original VHM; Remote Sensing MLLMs; A3.T2 Reasoning · Mean","GeoChat-7B","Original GeoChat-7B; Remote Sensing MLLMs; A3.T2 Reasoning · Mean","SenseNova-SI-1.1-InternVL3-2B","Original SenseNova-SI-1.1-InternVL3-2B; Embodied MLLMs; A3.T2 Reasoning · Mean","Pelican1.0-VL-7B (2K)","Original Pelican1.0-VL-7B (2K); Embodied MLLMs; A3.T2 Reasoning · Mean","Original RynnBrain-2B; Embodied MLLMs; A3.T2 Reasoning · Mean","Original Cosmos-Reason2-2B; Embodied MLLMs; A3.T2 Reasoning · Mean","Original RynnBrain1.1-2B; Embodied MLLMs; A3.T2 Reasoning · Mean","SenseNova-SI-1.1-InternVL3-8B","Original SenseNova-SI-1.1-InternVL3-8B; Embodied MLLMs; A3.T2 Reasoning · Mean","ACE-Brain-0.5-8B","Original ACE-Brain-0.5-8B; Embodied MLLMs; A3.T2 Reasoning · Mean","SenseNova-SI-1.3-InternVL3-8B","Original SenseNova-SI-1.3-InternVL3-8B; Embodied MLLMs; A3.T2 Reasoning · Mean","SenseNova-SI-1.5-InternVL3-8B","Original SenseNova-SI-1.5-InternVL3-8B; Embodied MLLMs; A3.T2 Reasoning · Mean","SenseNova-SI-1.4-InternVL3-8B","Original SenseNova-SI-1.4-InternVL3-8B; Embodied MLLMs; A3.T2 Reasoning · Mean","Original Cosmos-Reason2-8B; Embodied MLLMs; A3.T2 Reasoning · Mean","Original ACE-Brain-0-8B; Embodied MLLMs; A3.T2 Reasoning · Mean","Original MiMo-Embodied-7B; Embodied MLLMs; A3.T2 Reasoning · Mean","PhysBrain1.5-8B","Original PhysBrain1.5-8B; Embodied MLLMs; A3.T2 Reasoning · Mean","Original RynnBrain-8B; Embodied MLLMs; A3.T2 Reasoning · Mean","Original RoboBrain2.5-8B-NV; Embodied MLLMs; A3.T2 Reasoning · Mean","Original RynnBrain1.1-9B; Embodied MLLMs; A3.T2 Reasoning · Mean","ACE-Brain-0.6-9B","Original ACE-Brain-0.6-9B; Embodied MLLMs; A3.T2 Reasoning · Mean","Original Qwen3.5-2B; Embodied MLLMs; A3.T2 Reasoning · Mean","Original InternVL3.5-8B; Embodied MLLMs; A3.T2 Reasoning · Mean","Original InternVL3.5-4B; Embodied MLLMs; A3.T2 Reasoning · Mean","Original Qwen3-VL-8B-Instruct; Embodied MLLMs; A3.T2 Reasoning · Mean","Original Qwen3.5-9B; Embodied MLLMs; A3.T2 Reasoning · Mean","DeepSeek-V4.1-Flash-1017 / low","Original DeepSeek-V4.1-Flash-1017 / low; Embodied MLLMs; A3.T2 Reasoning · Mean","GPT-5.6-Luna-low","Original GPT-5.6-Luna-low; Embodied MLLMs; A3.T2 Reasoning · Mean","GPT-6.1-Sol-low","Original GPT-6.1-Sol-low; Embodied MLLMs; A3.T2 Reasoning · Mean","GPT-6-Astra-low","Original GPT-6-Astra-low; Embodied MLLMs; A3.T2 Reasoning · Mean","Original Qwen3.8 Max; S4.T1 All pass","https://arxiv.org/html/2611.14952v1#S4.T1","Original Kimi K3; S4.T1 All pass","Original GPT-5.6 Sol; S4.T1 All pass","Original DeepSeek-V4-Pro-0813; S4.T1 All pass","Original GLM-5.3; S4.T1 All pass","Original GPT-4o-mini; S4.T4 FU 2","https://arxiv.org/html/2611.14925v1#S4.T4","Original GPT-4.1-mini; S4.T4 FU 2","Original GPT-5.4-mini; S4.T4 FU 2","Original Opus 4.8; S4.T4 FU 2","Original Qwen3.5-397B; S4.T4 FU 2","Original Qwen3.6-27B; S4.T4 FU 2","Original Gemma 4-31B; S4.T4 FU 2","Original GPT-OSS-120B; S4.T4 FU 2","GLM-5.2-744B","Original GLM-5.2-744B; S4.T4 FU 2","Original Gemini 3.1 Pro; S5.T4 L5","https://arxiv.org/html/2611.14881v1#S5.T4","Original Claude Sonnet 4.5; S5.T4 L5","Original GPT-OSS 120B; S5.T4 L5","Original Kimi K2.6; S5.T4 L5","Original Claude Haiku 4.5; S5.T4 L5","Original GLM 5.1; S5.T4 L5","Original GPT-4.1 Mini; S5.T4 L5","Original Qwen3 Coder Next; S5.T4 L5","Llama 4","Original Llama 4; S5.T4 L5","EuroVLM (9B)","Original EuroVLM (9B); Small ( < 10B); S4.T2 AVG","https://arxiv.org/html/2611.14758v1#S4.T2","Original GLM-4.1V (9B); Small ( < 10B); S4.T2 AVG","Granite-Vision-3.3 (2B)","Original Granite-Vision-3.3 (2B); Small ( < 10B); S4.T2 AVG","Idefics3 (8B)","Original Idefics3 (8B); Small ( < 10B); S4.T2 AVG","Keye-VL (8B)","Original Keye-VL (8B); Small ( < 10B); S4.T2 AVG","MiMo-VL (7B)","Original MiMo-VL (7B); Small ( < 10B); S4.T2 AVG","Nanonets-OCR-s (3B)","Original Nanonets-OCR-s (3B); Small ( < 10B); S4.T2 AVG","Original Ovis2.5 (9B); Small ( < 10B); S4.T2 AVG","Qwen2-VL (2B)","Original Qwen2-VL (2B); Small ( < 10B); S4.T2 AVG","dots.ocr (1.7B)","Original dots.ocr (1.7B); Small ( < 10B); S4.T2 AVG","olmOCR-2 (7B)","Original olmOCR-2 (7B); Small ( < 10B); S4.T2 AVG","Gemma3 (27B)","Original Gemma3 (27B); Medium (10–40B); S4.T2 AVG","Gemma4 (31B)","Original Gemma4 (31B); Medium (10–40B); S4.T2 AVG","Original InternVL3.5 (38B); Medium (10–40B); S4.T2 AVG","Kimi-VL (16B-A3B)","Original Kimi-VL (16B-A3B); Medium (10–40B); S4.T2 AVG","Pixtral (12B)","Original Pixtral (12B); Medium (10–40B); S4.T2 AVG","Qwen3.5 (27B)","Original Qwen3.5 (27B); Medium (10–40B); S4.T2 AVG","Qwen3.6 (27B)","Original Qwen3.6 (27B); Medium (10–40B); S4.T2 AVG","Original GLM-4.5V (106B-A12B); Large ( > 40B); S4.T2 AVG","InternVL3 (78B)","Original InternVL3 (78B); Large ( > 40B); S4.T2 AVG","Qwen2.5-VL (72B)","Original Qwen2.5-VL (72B); Large ( > 40B); S4.T2 AVG","Qwen3-VL (235B-A22B)","Original Qwen3-VL (235B-A22B); Large ( > 40B); S4.T2 AVG","Gemini 3.8 Flash (commercial, API)","Original Gemini 3.8 Flash (commercial, API); Large ( > 40B); S4.T2 AVG","gpt-4.1-mini, no policy","Original gpt-4.1-mini, no policy; S10.T3 Overall","https://arxiv.org/html/2611.14581v1#S10.T3","gpt-4.1-mini, policy","Original gpt-4.1-mini, policy; S10.T3 Overall","gpt-4.1, no policy","Original gpt-4.1, no policy; S10.T3 Overall","gpt-4.1, policy","Original gpt-4.1, policy; S10.T3 Overall","gpt-5.6 Sol, no policy","Original gpt-5.6 Sol, no policy; S10.T3 Overall","gpt-5.6 Sol, policy","Original gpt-5.6 Sol, policy; S10.T3 Overall","Boundary evaluator (reference)","Original Boundary evaluator (reference); S10.T3 Overall","Native scoring (reference)","Original Native scoring (reference); S10.T3 Overall","• GPT-6 Astra","Original • GPT-6 Astra; S5.T2 RoboRisk-R · SSR","https://arxiv.org/html/2611.14115v1#S5.T2","Original • GPT-6 Astra; Method: LEASH; S5.T2 RoboRisk-R · SSR","• GPT-6.1 Sol","Original • GPT-6.1 Sol; S5.T2 RoboRisk-R · SSR","Original • GPT-6.1 Sol; Method: LEASH; S5.T2 RoboRisk-R · SSR","• Claude Opus 5.5","Original • Claude Opus 5.5; S5.T2 RoboRisk-R · SSR","Original • Claude Opus 5.5; Method: LEASH; S5.T2 RoboRisk-R · SSR","• Gemini 4 Argon","Original • Gemini 4 Argon; S5.T2 RoboRisk-R · SSR","Original • Gemini 4 Argon; Method: LEASH; S5.T2 RoboRisk-R · SSR","• Grok 4.7","Original • Grok 4.7; S5.T2 RoboRisk-R · SSR","Original • Grok 4.7; Method: LEASH; S5.T2 RoboRisk-R · SSR","Original gpt-5.4-mini-2026-03-17; Family: OpenAI GPT; Reasoning setting: None; A2.T6 Output cap","https://arxiv.org/html/2611.12750v1#A2.T6","Original gpt-6-luna; Family: OpenAI GPT; Reasoning setting: None; A2.T6 Output cap","openai/gpt-oss-20b (low)","Original openai/gpt-oss-20b (low); Family: OpenAI GPT-OSS; A2.T6 Output cap","openai/gpt-oss-120b","Original openai/gpt-oss-120b; Family: OpenAI GPT-OSS; Reasoning setting: Low; A2.T6 Output cap","openai/gpt-oss-20b (medium)","Original openai/gpt-oss-20b (medium); Family: OpenAI GPT-OSS; A2.T6 Output cap","Original gemini-2.5-pro; Family: Google Gemini; Reasoning setting: 128-token thinking budget; A2.T6 Output cap","Original gemini-2.5-flash; Family: Google Gemini; Reasoning setting: Zero thinking budget; A2.T6 Output cap","Original gemini-3.8-flash; Family: Google Gemini; Reasoning setting: Low thinking setting; A2.T6 Output cap","google/gemma-4-26B-A4B-it","Original google/gemma-4-26B-A4B-it; Family: Google Gemma; Reasoning setting: Disabled; A2.T6 Output cap","deepseek-ai/DeepSeek-V4-Flash","Original deepseek-ai/DeepSeek-V4-Flash; Reasoning setting: Disabled; A2.T6 Output cap","Original Qwen/Qwen3.5-0.8B; Reasoning setting: Disabled; A2.T6 Output cap","Original Qwen/Qwen3.5-9B; Reasoning setting: Disabled; A2.T6 Output cap","Qwen/Qwen3.5-9B-Base","Original Qwen/Qwen3.5-9B-Base; Reasoning setting: Base model; matched-family format; A2.T6 Output cap","Original kimi-k2.6; Family: Moonshot; Reasoning setting: Disabled; A2.T6 Output cap","CohereLabs/c4ai-command-a-03-2025","Original CohereLabs/c4ai-command-a-03-2025; Reasoning setting: No separate thinking mode; A2.T6 Output cap","A-FINE","Original A-FINE; A3.T8 Noise","https://arxiv.org/html/2611.18429v1#A3.T8","ARNIQA","Original ARNIQA; A3.T8 Noise","ARNIQA-CLIVE","Original ARNIQA-CLIVE; A3.T8 Noise","ARNIQA-CSIQ","Original ARNIQA-CSIQ; A3.T8 Noise","ARNIQA-FLIVE","Original ARNIQA-FLIVE; A3.T8 Noise","ARNIQA-KADID","Original ARNIQA-KADID; A3.T8 Noise","ARNIQA-LIVE","Original ARNIQA-LIVE; A3.T8 Noise","ARNIQA-SPAQ","Original ARNIQA-SPAQ; A3.T8 Noise","ARNIQA-TID","Original ARNIQA-TID; A3.T8 Noise","BRISQUE","Original BRISQUE; A3.T8 Noise","BRISQUE (MATLAB)","Original BRISQUE (MATLAB); A3.T8 Noise","CCF","Original CCF; A3.T8 Noise","Original CLIP-IQA; A3.T8 Noise","Original CLIP-IQA+; A3.T8 Noise","CLIP-IQA+ RN50","Original CLIP-IQA+ RN50; A3.T8 Noise","CLIP-IQA+ ViT-L","Original CLIP-IQA+ ViT-L; A3.T8 Noise","CNNIQA","Original CNNIQA; A3.T8 Noise","Original DBCNN; A3.T8 Noise","Entropy","Original Entropy; A3.T8 Noise","FDUM","Original FDUM; A3.T8 Noise","HyperIQA","Original HyperIQA; A3.T8 Noise","IL-NIQE","Original IL-NIQE; A3.T8 Noise","LAION-Aes","Original LAION-Aes; A3.T8 Noise","LIQE","Original LIQE; A3.T8 Noise","MA-CLIP","Original MA-CLIP; A3.T8 Noise","MANIQA","Original MANIQA; A3.T8 Noise","MANIQA-KADID","Original MANIQA-KADID; A3.T8 Noise","MANIQA-PIPAL","Original MANIQA-PIPAL; A3.T8 Noise","MetaIQA","Original MetaIQA; A3.T8 Noise","Original MUSIQ; A3.T8 Noise","MUSIQ-AVA","Original MUSIQ-AVA; A3.T8 Noise","MUSIQ-PaQ2PiQ","Original MUSIQ-PaQ2PiQ; A3.T8 Noise","MUSIQ-SPAQ","Original MUSIQ-SPAQ; A3.T8 Noise","NIMA","Original NIMA; A3.T8 Noise","NIMA-KONIQ","Original NIMA-KONIQ; A3.T8 Noise","NIMA-SPAQ","Original NIMA-SPAQ; A3.T8 Noise","NIMA-VGG16-AVA","Original NIMA-VGG16-AVA; A3.T8 Noise","NIQE","Original NIQE; A3.T8 Noise","NIQE (MATLAB)","Original NIQE (MATLAB); A3.T8 Noise","NMFC","Original NMFC; A3.T8 Noise","NR-UIQA","Original NR-UIQA; A3.T8 Noise","NRQM","Original NRQM; A3.T8 Noise","PaQ-2-PiQ","Original PaQ-2-PiQ; A3.T8 Noise","PI","Original PI; A3.T8 Noise","PIQE","Original PIQE; A3.T8 Noise","Original Q-Align; A3.T8 Noise","QualiCLIP","Original QualiCLIP; A3.T8 Noise","QualiCLIP+","Original QualiCLIP+; A3.T8 Noise","QualiCLIP+-CLIVE","Original QualiCLIP+-CLIVE; A3.T8 Noise","QualiCLIP+-FLIVE","Original QualiCLIP+-FLIVE; A3.T8 Noise","QualiCLIP+-SPAQ","Original QualiCLIP+-SPAQ; A3.T8 Noise","QUIQ","Original QUIQ; A3.T8 Noise","RALI","Original RALI; A3.T8 Noise","TOPIQ-IAA","Original TOPIQ-IAA; A3.T8 Noise","TOPIQ-IAA R50","Original TOPIQ-IAA R50; A3.T8 Noise","TOPIQ-NR","Original TOPIQ-NR; A3.T8 Noise","TOPIQ-NR-FLIVE","Original TOPIQ-NR-FLIVE; A3.T8 Noise","TOPIQ-NR-SPAQ","Original TOPIQ-NR-SPAQ; A3.T8 Noise","TReS","Original TReS; A3.T8 Noise","TReS-FLIVE","Original TReS-FLIVE; A3.T8 Noise","Twice-Mixing","Original Twice-Mixing; A3.T8 Noise","UCIQE","Original UCIQE; A3.T8 Noise","UICM","Original UICM; A3.T8 Noise","UIConM","Original UIConM; A3.T8 Noise","UIQAVSI","Original UIQAVSI; A3.T8 Noise","UIQM","Original UIQM; A3.T8 Noise","UISM","Original UISM; A3.T8 Noise","UNIQUE","Original UNIQUE; A3.T8 Noise","URanker","Original URanker; A3.T8 Noise","Original VisualQuality-R1; A3.T8 Noise","WaDIQaM-NR","Original WaDIQaM-NR; A3.T8 Noise","Claude Fable 5 (max)","Original Claude Fable 5 (max); S5.T3 Score","https://arxiv.org/html/2611.18130v1#S5.T3","Gemini 3.8 Flash (high)","Original Gemini 3.8 Flash (high); S5.T3 Score","Original Kimi K3 (max); S5.T3 Score","DS V4.1 Flash 1017 (max)","Original DS V4.1 Flash 1017 (max); S5.T3 Score","Qwen3.8 Flash (thinking)","Original Qwen3.8 Flash (thinking); S5.T3 Score","MiMo-V2.6-Pro (thinking)","Original MiMo-V2.6-Pro (thinking); S5.T3 Score","Kimi K2.6 (thinking)","Original Kimi K2.6 (thinking); S5.T3 Score","Mistral Large 4 (thinking)","Original Mistral Large 4 (thinking); S5.T3 Score","GLM 5.3 Flash (max)","Original GLM 5.3 Flash (max); S5.T3 Score","MiMo-V2.6-Flash (thinking)","Original MiMo-V2.6-Flash (thinking); S5.T3 Score","Original Gemini 3.1 Pro; Scheme: Single-pass; A2.T5 𝐅 𝟏","https://arxiv.org/html/2611.17966v1#A2.T5","Original Qwen3.5-397B-A17B; A2.T5 𝐅 𝟏","Original Qwen3.5-27B; A2.T5 𝐅 𝟏","Original GPT-5.5; A2.T5 𝐅 𝟏","Original MiMo-V2.5; A2.T5 𝐅 𝟏","Original Gemini 3.1 Pro; Scheme: Pipeline; A2.T5 𝐅 𝟏","Original MiMo-V2.5; A2.T5 𝐅 𝟏; row 8","Original Qwen3.6-27B; A2.T5 𝐅 𝟏","Original Qwen3.5-27B; A2.T5 𝐅 𝟏; row 10","Original GPT-5.5; A2.T5 𝐅 𝟏; row 11","Original GPT-5.5; Scheme: Agentic (E); A2.T5 𝐅 𝟏","Original MiMo-V2.5; A2.T5 𝐅 𝟏; row 13","Original Qwen3.6-27B; A2.T5 𝐅 𝟏; row 14","Original Qwen3.5-27B; A2.T5 𝐅 𝟏; row 15","Original GPT-5.5; Scheme: Agentic (O); A2.T5 𝐅 𝟏","Original GPT-5.5; Scheme: Agentic (S); A2.T5 𝐅 𝟏","Original Qwen 3.5 9B; Agent: GenericAgent; S4.T5 Overall SR","https://arxiv.org/html/2611.17955v1#S4.T5","Original Qwen 3.6 27B; Agent: GenericAgent; S4.T5 Overall SR","Original Qwen 3.8 27B; Agent: GenericAgent; S4.T5 Overall SR","Gemma4 26B A4B","Original Gemma4 26B A4B; Agent: GenericAgent; S4.T5 Overall SR","Original Gemma4 31B; Agent: GenericAgent; S4.T5 Overall SR","Original Gemini 3.1 Flash Lite; Agent: GenericAgent; S4.T5 Overall SR","Gemini 3.1 Pro omni","Original Gemini 3.1 Pro omni; Agent: GenericAgent; S4.T5 Overall SR","Original Claude Opus 4.7; S4.T5 Overall SR","Original Gemini 3.1 Pro; Agent: OpenCode; S4.T5 Overall SR","Qwen 3.8 Max Prime","Original Qwen 3.8 Max Prime; Agent: OpenCode; S4.T5 Overall SR","Kimi 3.1 Pro","Original Kimi 3.1 Pro; Agent: OpenCode; S4.T5 Overall SR","Original GPT 6.1 Sol; Agent: Codex; S4.T5 Overall SR","Original Claude Opus 5.5; Agent: Claude Code; S4.T5 Overall SR","Sol","Original Sol; A4.T10 g = 7 Mean gain","https://arxiv.org/html/2611.17933v1#A4.T10","Luna","Original Luna; A4.T10 g = 7 Mean gain","MiMo","Original MiMo; A4.T10 g = 7 Mean gain","Opus","Original Opus; A4.T10 g = 7 Mean gain","Original Kimi; A4.T10 g = 7 Mean gain","Original GLM; A4.T10 g = 7 Mean gain","Original DeepSeek V4.1 Flash 0910; A4.T10 g = 7 Mean gain","Original Sol; A4.T10 g = 7 Mean gain; row 11","Original Luna; A4.T10 g = 7 Mean gain; row 12","Original MiMo; A4.T10 g = 7 Mean gain; row 13","Original Opus; A4.T10 g = 7 Mean gain; row 14","Original Kimi; A4.T10 g = 7 Mean gain; row 15","Original GLM; A4.T10 g = 7 Mean gain; row 16","Original DeepSeek V4.1 Flash 0910; A4.T10 g = 7 Mean gain; row 17","Original GPT-6.1 Sol; Agent: OpenCode; S5.T3 Overall · F1","https://arxiv.org/html/2611.17931v1#S5.T3","Original GPT-6 Luna; S5.T3 Overall · F1","Original Claude Sonnet 5.5; S5.T3 Overall · F1","Original Claude Haiku 5.5; S5.T3 Overall · F1","Original DeepSeek-V4.1-Flash; S5.T3 Overall · F1","Original Qwen3.8-27B; S5.T3 Overall · F1","Sage-A1.5","Original Sage-A1.5; S5.T3 Overall · F1","Original GPT-6.1 Sol; Agent: OpenHands; S5.T3 Overall · F1","Original GPT-6 Luna; S5.T3 Overall · F1; row 11","Original Claude Sonnet 5.5; S5.T3 Overall · F1; row 12","Original Claude Haiku 5.5; S5.T3 Overall · F1; row 13","Original DeepSeek-V4.1-Flash; S5.T3 Overall · F1; row 14","Original Qwen3.8-27B; S5.T3 Overall · F1; row 15","Original Sage-A1.5; S5.T3 Overall · F1; row 16","Original GPT-6.1 Sol; Agent: Biomni; S5.T3 Overall · F1","Original GPT-6 Luna; S5.T3 Overall · F1; row 18","Original Claude Sonnet 5.5; S5.T3 Overall · F1; row 19","Original Claude Haiku 5.5; S5.T3 Overall · F1; row 20","Original DeepSeek-V4.1-Flash; S5.T3 Overall · F1; row 21","Original Qwen3.8-27B; S5.T3 Overall · F1; row 22","Original Sage-A1.5; S5.T3 Overall · F1; row 23","Original GPT-6.1 Sol; Agent: ChemToolAgent; S5.T3 Overall · F1","Original GPT-6 Luna; S5.T3 Overall · F1; row 25","Original Claude Sonnet 5.5; S5.T3 Overall · F1; row 26","Original Claude Haiku 5.5; S5.T3 Overall · F1; row 27","Original DeepSeek-V4.1-Flash; S5.T3 Overall · F1; row 28","Original Qwen3.8-27B; S5.T3 Overall · F1; row 29","Original Sage-A1.5; S5.T3 Overall · F1; row 30","Original DeepSeek V4.1 Flash; Task: T4.1; S4.T9 NED","https://arxiv.org/html/2611.17870v1#S4.T9","Original Qwen 3.7 Plus; S4.T9 NED","Original Qwen 3.8 Max; S4.T9 NED","Original Kimi K3; S4.T9 NED","Original Kimi K3.1 Pro; S4.T9 NED","Original GLM 5.3 Flash; S4.T9 NED","Original Claude Fable 5.1; S4.T9 NED","Original Claude Opus 5.5; S4.T9 NED","Original Gemini 3.8 Flash; S4.T9 NED","Original Gemini 4 Argon; S4.T9 NED","Original GPT-6.1 Sol; S4.T9 NED","Original GPT-6 Astra; S4.T9 NED","Original DeepSeek V4.1 Flash; Task: T4.2; S4.T9 NED","Original Qwen 3.7 Plus; S4.T9 NED; row 15","Original Qwen 3.8 Max; S4.T9 NED; row 16","Original Kimi K3; S4.T9 NED; row 17","Original Kimi K3.1 Pro; S4.T9 NED; row 18","Original GLM 5.3 Flash; S4.T9 NED; row 19","Original Claude Fable 5.1; S4.T9 NED; row 20","Original Claude Opus 5.5; S4.T9 NED; row 21","Original Gemini 3.8 Flash; S4.T9 NED; row 22","Original Gemini 4 Argon; S4.T9 NED; row 23","Original GPT-6.1 Sol; S4.T9 NED; row 24","Original GPT-6 Astra; S4.T9 NED; row 25","Original DeepSeek V4.1 Flash; Task: T4.3; S4.T9 NED","Original Qwen 3.7 Plus; S4.T9 NED; row 27","Original Qwen 3.8 Max; S4.T9 NED; row 28","Original Kimi K3; S4.T9 NED; row 29","Original Kimi K3.1 Pro; S4.T9 NED; row 30","Original GLM 5.3 Flash; S4.T9 NED; row 31","Original Claude Fable 5.1; S4.T9 NED; row 32","Original Claude Opus 5.5; S4.T9 NED; row 33","Original Gemini 3.8 Flash; S4.T9 NED; row 34","Original Gemini 4 Argon; S4.T9 NED; row 35","Original GPT-6.1 Sol; S4.T9 NED; row 36","Original GPT-6 Astra; S4.T9 NED; row 37","Original DeepSeek V4.1 Flash; Task: T4.4; S4.T9 NED","Original Qwen 3.7 Plus; S4.T9 NED; row 39","Original Qwen 3.8 Max; S4.T9 NED; row 40","Original Kimi K3; S4.T9 NED; row 41","Original Kimi K3.1 Pro; S4.T9 NED; row 42","Original GLM 5.3 Flash; S4.T9 NED; row 43","Original Claude Fable 5.1; S4.T9 NED; row 44","Original Claude Opus 5.5; S4.T9 NED; row 45","Original Gemini 3.8 Flash; S4.T9 NED; row 46","Original Gemini 4 Argon; S4.T9 NED; row 47","Original GPT-6.1 Sol; S4.T9 NED; row 48","Original GPT-6 Astra; S4.T9 NED; row 49","Original DeepSeek-V4.1-Flash; Harness: CC; S4.T6 Richness","https://arxiv.org/html/2611.17681v1#S4.T6","Original DeepSeek-V4-Flash-0731; Harness: CC; S4.T6 Richness","Original Fable 5; Harness: CC; S4.T6 Richness","Original Fable 5.1; Harness: CC; S4.T6 Richness","Original GLM-5.2; Harness: CC; S4.T6 Richness","Original GLM-5.3; Harness: CC; S4.T6 Richness","Original Opus 5; Harness: CC; S4.T6 Richness","Original DeepSeek-V4-Pro; Harness: CX; S4.T6 Richness","Original GPT-5.5; Harness: CX; S4.T6 Richness","Original GPT-5.6-sol; Harness: CX; S4.T6 Richness","Original GPT-6-astra; Harness: CX; S4.T6 Richness","Original GPT-6.1 Sol; S1.T1 w/ text","https://arxiv.org/html/2611.17680v1#S1.T1","Original GPT-6 Luna; S1.T1 w/ text","Original Claude Sonnet 5.5; S1.T1 w/ text","Original Claude Haiku 5.5; S1.T1 w/ text","Original Qwen3-8B; S1.T1 w/ text","TableGPT2-7B","Original TableGPT2-7B; S1.T1 w/ text","Original Aurora; S1.T1 w/ text","TimeOmni-1-7B","Original TimeOmni-1-7B; S1.T1 w/ text","DoubleCast","Original DoubleCast; S1.T1 w/ text","TimeBraid-2.5B","Original TimeBraid-2.5B; S1.T1 w/ text","AGIEval SAT-Math","Original AGIEval SAT-Math; S4.T1 SCoRe","https://arxiv.org/html/2611.17563v1#S4.T1","AIME 2025","Original AIME 2025; S4.T1 SCoRe","GPQA (main ∖ diamond)","Original GPQA (main ∖ diamond); S4.T1 SCoRe","LiveCodeBench","Original LiveCodeBench; S4.T1 SCoRe","MMLU-Redux","Original MMLU-Redux; S4.T1 SCoRe","MuSR-murder","Original MuSR-murder; S4.T1 SCoRe","Omni-MATH","Original Omni-MATH; S4.T1 SCoRe","SuperGPQA","Original SuperGPQA; S4.T1 SCoRe","TheoremQA","Original TheoremQA; S4.T1 SCoRe","Gemma-2 9B","Original Gemma-2 9B; S4.T1 SCoRe","Original Gemini-3.8-flash; S4.T1 SCoRe","Original Claude-Haiku-4.5; Qwen3-8B; S4.T1 SCoRe","Original Gemini-3.8-flash; S4.T1 SCoRe; row 16","Original Claude-Haiku-4.5; Gemma-2 9B; S4.T1 SCoRe","Original FunctionGemma-270M; S4.T1 Acc.","https://arxiv.org/html/2611.17549v1#S4.T1","Gemma4:E2B","Original Gemma4:E2B; S4.T1 Acc.","Gemma4:E4B","Original Gemma4:E4B; S4.T1 Acc.","Ministral3:3B","Original Ministral3:3B; S4.T1 Acc.","Ministral3:8B","Original Ministral3:8B; S4.T1 Acc.","Original gpt-oss-120b; S4.T4 Om.","https://arxiv.org/html/2611.17460v1#S4.T4","Original gpt-oss-20b; S4.T4 Om.","Original Gemma 4 31B; S4.T4 Om.","Original Gemma 4 26B; S4.T4 Om.","Original Muse Glimmer 30B; S4.T4 Om.","Original Gemma 4 12B; S4.T4 Om.","Original DeepSeek V4 Pro; S4.T4 Om.","Qwen 3.8 Flash-Next","Original Qwen 3.8 Flash-Next; S4.T4 Om.","Original Qwen 3.8 27B; S4.T4 Om.","Original GLM 5.3 Flash; S4.T4 Om.","Original DeepSeek V4.1 Flash; S4.T4 Om.","Original GLM 5.3; S4.T4 Om.","LlamaParse (Agentic Plus)","Original LlamaParse (Agentic Plus); DOCUMENT PARSERS; A3.T18 Pages limited by the cap (%) All","https://arxiv.org/html/2611.17400v1#A3.T18","LlamaParse (Agentic)","Original LlamaParse (Agentic); DOCUMENT PARSERS; A3.T18 Pages limited by the cap (%) All","LlamaParse (Cost Effective)","Original LlamaParse (Cost Effective); DOCUMENT PARSERS; A3.T18 Pages limited by the cap (%) All","Reducto","Original Reducto; DOCUMENT PARSERS; A3.T18 Pages limited by the cap (%) All","Extend (Parse 2)","Original Extend (Parse 2); DOCUMENT PARSERS; A3.T18 Pages limited by the cap (%) All","Pulse (Ultra 2)","Original Pulse (Ultra 2); DOCUMENT PARSERS; A3.T18 Pages limited by the cap (%) All","Extend (Parse Light)","Original Extend (Parse Light); DOCUMENT PARSERS; A3.T18 Pages limited by the cap (%) All","Datalab (Accurate)","Original Datalab (Accurate); DOCUMENT PARSERS; A3.T18 Pages limited by the cap (%) All","Datalab (Balanced)","Original Datalab (Balanced); DOCUMENT PARSERS; A3.T18 Pages limited by the cap (%) All","Datalab (Fast)","Original Datalab (Fast); DOCUMENT PARSERS; A3.T18 Pages limited by the cap (%) All","Databricks AI Parse","Original Databricks AI Parse; DOCUMENT PARSERS; A3.T18 Pages limited by the cap (%) All","Reducto (r-1)","Original Reducto (r-1); DOCUMENT PARSERS; A3.T18 Pages limited by the cap (%) All","Mistral OCR 4.1 (annotation)","Original Mistral OCR 4.1 (annotation); DOCUMENT PARSERS; A3.T18 Pages limited by the cap (%) All","Reducto (r-1-mini)","Original Reducto (r-1-mini); DOCUMENT PARSERS; A3.T18 Pages limited by the cap (%) All","Original LandingAI; DOCUMENT PARSERS; A3.T18 Pages limited by the cap (%) All","Extend (Parse)","Original Extend (Parse); DOCUMENT PARSERS; A3.T18 Pages limited by the cap (%) All","Mistral OCR 4.1","Original Mistral OCR 4.1; DOCUMENT PARSERS; A3.T18 Pages limited by the cap (%) All","Reducto (non-agentic)","Original Reducto (non-agentic); DOCUMENT PARSERS; A3.T18 Pages limited by the cap (%) All","Azure Document Intelligence (layout)","Original Azure Document Intelligence (layout); DOCUMENT PARSERS; A3.T18 Pages limited by the cap (%) All","Original Claude Opus 5.5; DIRECT VISION MODELS; A3.T18 Pages limited by the cap (%) All","Gemini 4 argon (low)","Original Gemini 4 argon (low); DIRECT VISION MODELS; A3.T18 Pages limited by the cap (%) All","GPT-5.6 terra (low)","Original GPT-5.6 terra (low); DIRECT VISION MODELS; A3.T18 Pages limited by the cap (%) All","Gemini 3 Flash (minimal)","Original Gemini 3 Flash (minimal); DIRECT VISION MODELS; A3.T18 Pages limited by the cap (%) All","Gemini 3.7 Flash (low)","Original Gemini 3.7 Flash (low); DIRECT VISION MODELS; A3.T18 Pages limited by the cap (%) All","Gemini 3.8 Flash (low)","Original Gemini 3.8 Flash (low); DIRECT VISION MODELS; A3.T18 Pages limited by the cap (%) All","Original Gemini 3.5 Flash; DIRECT VISION MODELS; A3.T18 Pages limited by the cap (%) All","Original Gemini 3.6 Flash; DIRECT VISION MODELS; A3.T18 Pages limited by the cap (%) All","Muse Ember 1.0 (low)","Original Muse Ember 1.0 (low); DIRECT VISION MODELS; A3.T18 Pages limited by the cap (%) All","GPT-6.1 terra (low)","Original GPT-6.1 terra (low); DIRECT VISION MODELS; A3.T18 Pages limited by the cap (%) All","Original Claude Sonnet 5.5; DIRECT VISION MODELS; A3.T18 Pages limited by the cap (%) All","GPT-6 luna (low)","Original GPT-6 luna (low); DIRECT VISION MODELS; A3.T18 Pages limited by the cap (%) All","Original Claude Haiku 5.5; DIRECT VISION MODELS; A3.T18 Pages limited by the cap (%) All","GPT-5.6 luna (low)","Original GPT-5.6 luna (low); DIRECT VISION MODELS; A3.T18 Pages limited by the cap (%) All","Original Gemini 3.1 Flash-Lite; DIRECT VISION MODELS; A3.T18 Pages limited by the cap (%) All","Original Gemini 3.5 Flash-Lite; DIRECT VISION MODELS; A3.T18 Pages limited by the cap (%) All","Original Gemini 3.5 Flash; Native Omni-Modal Agents; S3.T1 Avg.","https://arxiv.org/html/2611.19202v1#S3.T1","Original Gemini 3.1 Pro Preview; Native Omni-Modal Agents; S3.T1 Avg.","Original Gemini 3 Flash Preview; Native Omni-Modal Agents; S3.T1 Avg.","Original Gemini 2.5 Pro; Native Omni-Modal Agents; S3.T1 Avg.","Original MiMo-V2.5; Native Omni-Modal Agents; S3.T1 Avg.","Qwen3.5 Omni Plus","Original Qwen3.5 Omni Plus; Native Omni-Modal Agents; S3.T1 Avg.","Qwen3.5 Omni Flash","Original Qwen3.5 Omni Flash; Native Omni-Modal Agents; S3.T1 Avg.","Original MiniCPM-o-4.5; Native Omni-Modal Agents; S3.T1 Avg.","Original Nemotron-3-Nano-Omni; Native Omni-Modal Agents; S3.T1 Avg.","Original Qwen3-Omni-Instruct; Native Omni-Modal Agents; S3.T1 Avg.","Original Qwen3.7-Plus; Native Omni-Modal Agents; S3.T1 Avg.","Original Qwen3.6-Plus; Native Omni-Modal Agents; S3.T1 Avg.","Original Kimi K2.6; Native Omni-Modal Agents; S3.T1 Avg.","Original MiniMax-M3; Native Omni-Modal Agents; S3.T1 Avg.","Original Qwen3.5-122B-A10B; Native Omni-Modal Agents; S3.T1 Avg.","Original Qwen3.5-35B-A3B; Native Omni-Modal Agents; S3.T1 Avg.","Original Qwen3-VL-30B-A3B-Instruct; Native Omni-Modal Agents; S3.T1 Avg.","Original Qwen3-VL-8B-Instruct; Native Omni-Modal Agents; S3.T1 Avg.","Moondream 2B","Original Moondream 2B; S3.T3 Counterfactual SCS (%) Overall","https://arxiv.org/html/2611.19184v1#S3.T3","PaliGemma 3B","Original PaliGemma 3B; S3.T3 Counterfactual SCS (%) Overall","SmolVLM 2.2B","Original SmolVLM 2.2B; S3.T3 Counterfactual SCS (%) Overall","LLaVA-OV 8B","Original LLaVA-OV 8B; S3.T3 Counterfactual SCS (%) Overall","Original InternVL3 8B; S3.T3 Counterfactual SCS (%) Overall","Original Qwen2.5-VL 7B; S3.T3 Counterfactual SCS (%) Overall","Original GPT-5.4; Frontier API models; S5.T2 S coh","https://arxiv.org/html/2611.18880v1#S5.T2","Original Gemini 3.1 Pro; Frontier API models; S5.T2 S coh","Original Gemini 3 Flash; Frontier API models; S5.T2 S coh","SmolVLM2-2.2B-Instruct","Original SmolVLM2-2.2B-Instruct; Open-weight models, model size ≤ 4B; S5.T2 S coh","Original Qwen3-VL-2B-Instruct; Open-weight models, model size ≤ 4B; S5.T2 S coh","VideoLLaMA3-2B","Original VideoLLaMA3-2B; Open-weight models, model size ≤ 4B; S5.T2 S coh","Video-XL-Pro-3B","Original Video-XL-Pro-3B; Open-weight models, model size ≤ 4B; S5.T2 S coh","Original Qwen3.5-9B-Instruct; Open-weight models, 4B < model size ≤ 9B; S5.T2 S coh","Original Qwen3-VL-8B-Instruct; Open-weight models, 4B < model size ≤ 9B; S5.T2 S coh","Original InternVL3.5-8B-Instruct; Open-weight models, 4B < model size ≤ 9B; S5.T2 S coh","Original MiniCPM-V 4.5; Open-weight models, 4B < model size ≤ 9B; S5.T2 S coh","Original Molmo2-8B; Open-weight models, 4B < model size ≤ 9B; S5.T2 S coh","Original Keye-VL-1.5-8B; Open-weight models, 4B < model size ≤ 9B; S5.T2 S coh","MiMo-VL-7B-RL-2508","Original MiMo-VL-7B-RL-2508; Open-weight models, 4B < model size ≤ 9B; S5.T2 S coh","Original VideoLLaMA3-7B; Open-weight models, 4B < model size ≤ 9B; S5.T2 S coh","InternVideo2.5-Chat-8B","Original InternVideo2.5-Chat-8B; Open-weight models, 4B < model size ≤ 9B; S5.T2 S coh","Original mPLUG-Owl3-7B; Open-weight models, 4B < model size ≤ 9B; S5.T2 S coh","LLaVA-Video-7B-Qwen2","Original LLaVA-Video-7B-Qwen2; Open-weight models, 4B < model size ≤ 9B; S5.T2 S coh","Original Kimi-VL-A3B-Instruct; Open-weight models, 4B < model size ≤ 9B; S5.T2 S coh","Original Qwen3-VL-30B-A3B-Instruct; Open-weight models, 4B < model size ≤ 9B; S5.T2 S coh","Original InternVL3.5-30B-A3B; Open-weight models, 4B < model size ≤ 9B; S5.T2 S coh","Original Qwen3-VL-235B-A22B-Instruct; Open-weight models, 4B < model size ≤ 9B; S5.T2 S coh","Original InternVL3.5-241B-A28B; Open-weight models, 4B < model size ≤ 9B; S5.T2 S coh","UniRS","Original UniRS; Remote-sensing VLMs; S5.T2 S coh","SkyEyeGPT","Original SkyEyeGPT; Remote-sensing VLMs; S5.T2 S coh","Original EarthDial; Remote-sensing VLMs; S5.T2 S coh","Original SDXL; Diffusion T2I Models; Sx6.T3 Overall","https://arxiv.org/html/2611.18053v1#Sx6.T3","Original FLUX.1-dev; Diffusion T2I Models; Sx6.T3 Overall","Original Janus-Pro-7B; Autoregressive T2I; Sx6.T3 Overall","LlamaGen","Original LlamaGen; Autoregressive T2I; Sx6.T3 Overall","Infinity-2B","Original Infinity-2B; Finetuned Visual Autoregressive T2I; Sx6.T3 Overall","Infinity-8B-512","Original Infinity-8B-512; Finetuned Visual Autoregressive T2I; Sx6.T3 Overall","Infinity-8B","Original Infinity-8B; Finetuned Visual Autoregressive T2I; Sx6.T3 Overall","Original Qwen3-VL-8B-Instruct; Mode: polled; S6.T2 [ 0 , 1 ]","https://arxiv.org/html/2611.16400v1#S6.T2","Original Qwen2.5-Omni-7B; Mode: polled; S6.T2 [ 0 , 1 ]","Original LLaVA-OneVision-2-8B-Instruct; Mode: polled; S6.T2 [ 0 , 1 ]","LiveCC-7B-Instruct","Original LiveCC-7B-Instruct; Mode: native, adapted; S6.T2 [ 0 , 1 ]","VideoChat3-4B","Original VideoChat3-4B; Mode: native; S6.T2 [ 0 , 1 ]","Gemma-3n-E4B-it","Original Gemma-3n-E4B-it; Mode: polled; S6.T2 [ 0 , 1 ]","Original Claude Code (Opus 4.6); S4.T2 76.9%","https://arxiv.org/html/2611.15189v1#S4.T2","Claude Code (Sonnet 4.6)","Original Claude Code (Sonnet 4.6); S4.T2 76.9%","Codex (GPT-5.5)","Original Codex (GPT-5.5); S4.T2 76.9%","Claude Code (GLM 5.2)","Original Claude Code (GLM 5.2); S4.T2 76.9%","Original Codex (GPT-5.4); S4.T2 76.9%","Kimi CLI (Kimi K2.6)","Original Kimi CLI (Kimi K2.6); S4.T2 76.9%","Original Phi-4-multimodal; Vision-Only; S4.T2 MCQs Accuracy (%) Avg.","https://arxiv.org/html/2611.15161v1#S4.T2","Original InternVL3.5; Vision-Only; S4.T2 MCQs Accuracy (%) Avg.","Ministral-3","Original Ministral-3; Vision-Only; S4.T2 MCQs Accuracy (%) Avg.","Original GLM-4.1V; Vision-Only; S4.T2 MCQs Accuracy (%) Avg.","GLM-4.6V-","Original GLM-4.6V-; Vision-Only; S4.T2 MCQs Accuracy (%) Avg.","Original GLM-4.6V; Vision-Only; S4.T2 MCQs Accuracy (%) Avg.","Original Qwen3-VL (8B); Vision-Only; S4.T2 MCQs Accuracy (%) Avg.","Original Qwen3-VL (32B); Vision-Only; S4.T2 MCQs Accuracy (%) Avg.","Qwen2.5-","Original Qwen2.5-; Vision-Only; S4.T2 MCQs Accuracy (%) Avg.","Original MiniCPM-o-2.6; Audio-Capable; S4.T2 MCQs Accuracy (%) Avg.","Original Qwen2.5-; Audio-Capable; S4.T2 MCQs Accuracy (%) Avg.","Qwen3-","Original Qwen3-; Audio-Capable; S4.T2 MCQs Accuracy (%) Avg.","Original MiniCPM-o-4.5; Audio-Capable; S4.T2 MCQs Accuracy (%) Avg.","Original MiMo-v2.5; Audio-Capable; S4.T2 MCQs Accuracy (%) Avg.","Original GPT-5; Vision-Only; S4.T2 MCQs Accuracy (%) Avg.","Qwen3.5-","Original Qwen3.5-; Audio-Capable; S4.T2 MCQs Accuracy (%) Avg.","Gemini-3","Original Gemini-3; Audio-Capable; S4.T2 MCQs Accuracy (%) Avg.","Claude Fable 5.2 (adaptive)","Original Claude Fable 5.2 (adaptive); S4.T3 Score at default","https://arxiv.org/html/2611.21541v1#S4.T3","GPT-6 Astra (provider default)","Original GPT-6 Astra (provider default); S4.T3 Score at default","Claude Sonnet 5.5 (high)","Original Claude Sonnet 5.5 (high); S4.T3 Score at default","Claude Opus 5.5 (medium)","Original Claude Opus 5.5 (medium); S4.T3 Score at default","Claude Haiku 5.5 (medium)","Original Claude Haiku 5.5 (medium); S4.T3 Score at default","Claude Sonnet 4.5 (no thinking)","Original Claude Sonnet 4.5 (no thinking); S4.T3 Score at default","Qwen3.8-27B (thinking)","Original Qwen3.8-27B (thinking); S4.T3 Score at default","Muse Glimmer 30B (high)","Original Muse Glimmer 30B (high); S4.T3 Score at default","Gemma 4 31B (no thinking)","Original Gemma 4 31B (no thinking); S4.T3 Score at default","Original GPT-5.6 Luna; Reasoning effort: medium; S4.T4 Overall","https://arxiv.org/html/2611.21273v1#S4.T4","Original GPT-6 Luna; Reasoning effort: medium; S4.T4 Overall","Original GPT-5.6 Sol; Reasoning effort: medium; S4.T4 Overall","Original GPT-6 Sol; Reasoning effort: medium; S4.T4 Overall","Original GPT-6.1 Sol; Reasoning effort: medium; S4.T4 Overall","Original GPT-6 Astra; Reasoning effort: medium; S4.T4 Overall","Original Gemini 3.8 Flash; Reasoning effort: medium; S4.T4 Overall","Original Gemini 4 Argon; Reasoning effort: medium; S4.T4 Overall","Original DeepSeek V4.1 Flash; Reasoning effort: high; S4.T4 Overall","Original Qwen3.8-Flash; Reasoning effort: medium; S4.T4 Overall","Original Kimi K2.7 Code; S4.T4 Overall","Original GLM-5.3-Flash; Reasoning effort: high; S4.T4 Overall","Original Gemini 4 Argon; S5.T2 Agency","https://arxiv.org/html/2611.21218v1#S5.T2","Original Claude Opus 5.5; S5.T2 Agency","Original Claude Fable 5.1; S5.T2 Agency","Original GPT-6 Astra; S5.T2 Agency","Original GPT-6.1 Sol; S5.T2 Agency","Original DeepSeek V4.1 Flash; S5.T2 Agency","Original Grok 4.7; S5.T2 Agency","Original Kimi K3.1 Pro; S5.T2 Agency","Original Qwen3.8 Max; S5.T2 Agency","Original Kimi K3; S5.T2 Agency","Original Gemini 3.8 Flash; S5.T2 Agency","Original GPT-5.6 Luna; S5.T2 Agency","Original GLM-5.3; S5.T2 Agency","Original DeepSeek V4 Pro; S5.T2 Agency","Original GPT-6 Luna; S5.T2 Agency","Original MiniMax M3.1 Flash; S5.T2 Agency","Original Step 5 Preview; S5.T2 Agency","Original MiMo V2.6 Pro; S5.T2 Agency","Original GLM-5.3 Flash; S5.T2 Agency","Original Qwen3.8 27B; S5.T2 Agency","Original Grok 4.3; S5.T2 Agency","Step 3.7 Flash","Original Step 3.7 Flash; S5.T2 Agency","Minicpm-V","Original Minicpm-V; S4.T2 Total","https://arxiv.org/html/2611.21105v1#S4.T2","Original Qwen2.5-VL-7B; S4.T2 Total","InternVL","Original InternVL; S4.T2 Total","Original LLaVA-OneVision-7B; S4.T2 Total","Original Phi-3.5-Vision; S4.T2 Total","Original Qwen2.5-VL-3B-Instruct; S4.T2 Total","Original InternVL3-1B; S4.T2 Total","Original Qwen3-VL-8B-Instruct; S4.T2 Total","DeepSeek VL2","Original DeepSeek VL2; S4.T2 Total","InternVL3_5-14B-HF","Original InternVL3_5-14B-HF; S4.T2 Total","Mistral-Small-3.1-24B-Instruct-2503","Original Mistral-Small-3.1-24B-Instruct-2503; S4.T2 Total","Original Qwen-VL-Plus; S4.T2 Total","Original GPT-4.1-mini; S4.T2 Total","Original GPT-4.1; S4.T2 Total","Original GPT-5-mini; S4.T2 Total","Original GPT-5; S4.T2 Total","Original Gemini-2.5-pro; S4.T2 Total","Original Gemini-3-Pro; S5.T4 Depth","https://arxiv.org/html/2611.20956v1#S5.T4","Original Gemini-3-Flash; S5.T4 Depth","Original Claude-Opus-4.6; S5.T4 Depth","Original Claude-Sonnet-4.6; S5.T4 Depth","Original GPT-5; S5.T4 Depth","GPT-5-Chat","Original GPT-5-Chat; S5.T4 Depth","Original o3; S5.T4 Depth","Original GLM-4.6V; S5.T4 Depth","Original Qwen3-VL-235B-A22B-Thinking; S5.T4 Depth","Original Qwen3-VL-235B-A22B-Instruct; S5.T4 Depth","Original Qwen3-VL-32B-Thinking; S5.T4 Depth","Original Qwen3-VL-32B-Instruct; S5.T4 Depth","Original Claude Sonnet 5.5; Harness: Claude Code; S4.T1 Score@1","https://arxiv.org/html/2611.20882v1#S4.T1","Original Claude Opus 5.5; Harness: Claude Code; S4.T1 Score@1","Original GPT-6.1 Sol; Harness: Codex; S4.T1 Score@1","Original GPT-6 Astra; Harness: Codex; S4.T1 Score@1","Original Hy4-Preview; Harness: Claude Code; S4.T1 Score@1","Original Hy4; Harness: Claude Code; S4.T1 Score@1","Original Kimi K3; Harness: Claude Code; S4.T1 Score@1","Original DeepSeek-V4.1-Flash-1017; Harness: Claude Code; S4.T1 Score@1","Original GLM-5.3; Harness: Claude Code; S4.T1 Score@1","Original GLM-5.3-Flash; Harness: Claude Code; S4.T1 Score@1","Original DeepSeek-V4-Pro; Harness: Claude Code; S4.T1 Score@1","Original GPT-6.1 Terra; Harness: Codex; S4.T1 Score@1","Original DeepSeek-V4-Flash; Harness: Claude Code; S4.T1 Score@1","Original GLM-5.2; Harness: Claude Code; S4.T1 Score@1","Original Grok 4.6; Harness: Claude Code; S4.T1 Score@1","Original Hy3; Harness: Claude Code; S4.T1 Score@1","Original Claude; S5.T2 Success rate","https://arxiv.org/html/2611.20844v1#S5.T2","Original Kimi; S5.T2 Success rate","Original DeepSeek; S5.T2 Success rate","Original Qwen; S5.T2 Success rate","Original GLM; S5.T2 Success rate","Original Qwen2.5-7B; open; S6.T7 Cohen’s d","https://arxiv.org/html/2611.20821v1#S6.T7","Original Qwen2.5-32B; open; S6.T7 Cohen’s d","Original Llama-3.1-8B; open; S6.T7 Cohen’s d","Original Llama-4-Scout-17B; open; S6.T7 Cohen’s d","Original GPT-5.4; closed; S6.T7 Cohen’s d","Original GPT-5.5; closed; S6.T7 Cohen’s d","Original Gemini-2.5-flash; closed; S6.T7 Cohen’s d","Original Gemini-3.5-flash; closed; S6.T7 Cohen’s d","Original GPT-5.6 Sol; S4.T3 EEG Cross-modal Alignment · EVM","https://arxiv.org/html/2611.20779v1#S4.T3","Original Gemini-3.8-Flash; S4.T3 EEG Cross-modal Alignment · EVM","Original DeepSeek-V4.1-Flash; S4.T3 EEG Cross-modal Alignment · EVM","MiniMax-M3 †,","Original MiniMax-M3 †,; S4.T3 EEG Cross-modal Alignment · EVM","Original GPT-5.6 Luna; S4.T3 EEG Cross-modal Alignment · EVM","Original GPT-4.1 mini; S4.T3 EEG Cross-modal Alignment · EVM","Original Qwen3.7-Plus; S4.T3 EEG Cross-modal Alignment · EVM","Original Ministral-3-8B; S4.T3 EEG Cross-modal Alignment · EVM","Original Gemma-4-12B; S4.T3 EEG Cross-modal Alignment · EVM","Original Qwen3.5-4B; S4.T3 EEG Cross-modal Alignment · EVM","Original Qwen3-VL-4B-Instruct; S4.T3 EEG Cross-modal Alignment · EVM","Deepseek-V3.1","Original Deepseek-V3.1; S5.T4 Mean","https://arxiv.org/html/2611.20695v1#S5.T4","Deepseek-V4-Pro-0813","Original Deepseek-V4-Pro-0813; S5.T4 Mean","Original Gemini-2.5-Pro; S5.T4 Mean","Original GLM-4.5; S5.T4 Mean","Original GLM-4.5-Air; S5.T4 Mean","GLM-4","Original GLM-4; S5.T4 Mean","Original GLM-4-9B; S5.T4 Mean","Original GPT-4.1; S5.T4 Mean","Original GPT-5; S5.T4 Mean","Original GPT-6-sol; S5.T4 Mean","Original Llama-3.1-8B; S5.T4 Mean","Original Llama-3.1-70B; S5.T4 Mean","Original Qwen3-0.6B; S5.T4 Mean","Original Qwen3-1.7B; S5.T4 Mean","Original Qwen3-4B; S5.T4 Mean","Original Qwen3-8B; S5.T4 Mean","Original Qwen3-14B; S5.T4 Mean","Original Qwen-Plus; S5.T4 Mean","GPT-6 Astra (high)","Original GPT-6 Astra (high); S6.T2 All (100) Score","https://arxiv.org/html/2611.20628v1#S6.T2","Claude Opus 5 (low)","Original Claude Opus 5 (low); S6.T2 All (100) Score","Original Qwen3.8 Max; S6.T2 All (100) Score","Original Gemini 3.8 Flash (high); S6.T2 All (100) Score","Original GPT-5.6 Sol (medium); S6.T2 All (100) Score","DeepSeek V4.1 Flash (high)","Original DeepSeek V4.1 Flash (high); S6.T2 All (100) Score","GLM-5.3 Flash (max)","Original GLM-5.3 Flash (max); S6.T2 All (100) Score","Original Kimi K3 (max); S6.T2 All (100) Score","Gemini 3.1 Pro Preview (high)","Original Gemini 3.1 Pro Preview (high); S6.T2 All (100) Score","DeepSeek V4 Pro (high)","Original DeepSeek V4 Pro (high); S6.T2 All (100) Score","Original DeepSeek V4 Flash (high); S6.T2 All (100) Score","Original Qwen3.5-397B-A17B; S6.T2 All (100) Score","DS-F","Original DS-F; OpenClaw; S3.T2 S ¯","https://arxiv.org/html/2611.20535v1#S3.T2","Original GLM; OpenClaw; S3.T2 S ¯","Original GPT; OpenClaw; S3.T2 S ¯","Original Opus; OpenClaw; S3.T2 S ¯","Agent mean","Original Agent mean; OpenClaw; S3.T2 S ¯","Original DS-F; Hermes; S3.T2 S ¯","Original GLM; Hermes; S3.T2 S ¯","Original GPT; Hermes; S3.T2 S ¯","Original Opus; Hermes; S3.T2 S ¯","Original Agent mean; Hermes; S3.T2 S ¯","Original DS-F; OpenSquilla; S3.T2 S ¯","Original GLM; OpenSquilla; S3.T2 S ¯","Original GPT; OpenSquilla; S3.T2 S ¯","Original Opus; OpenSquilla; S3.T2 S ¯","Original Agent mean; OpenSquilla; S3.T2 S ¯","Original DS-F; OpenSquilla; S3.T2 S ¯; row 22","Original GLM; OpenSquilla; S3.T2 S ¯; row 23","Original GPT; OpenSquilla; S3.T2 S ¯; row 24","Original Opus; OpenSquilla; S3.T2 S ¯; row 25","Original Agent mean; OpenSquilla; S3.T2 S ¯; row 26","Original DS-F; Codex; S3.T2 S ¯","Original GLM; Codex; S3.T2 S ¯","Original GPT; Codex; S3.T2 S ¯","Original Opus; Codex; S3.T2 S ¯","Original Agent mean; Codex; S3.T2 S ¯","Original DS-F; Codex; S3.T2 S ¯; row 34","Original GLM; Codex; S3.T2 S ¯; row 35","Original GPT; Codex; S3.T2 S ¯; row 36","Original Opus; Codex; S3.T2 S ¯; row 37","Original Agent mean; Codex; S3.T2 S ¯; row 38","Track mean","Original Track mean; Codex; S3.T2 S ¯","Cowrie","Original Cowrie; Traditional Honeypot; S4.T2 Authenticity Score Avg.","https://arxiv.org/html/2611.20528v1#S4.T2","Original Sonnet-4.6; Traditional Honeypot; S4.T2 Authenticity Score Avg.","Doubao-1.5","Original Doubao-1.5; Traditional Honeypot; S4.T2 Authenticity Score Avg.","Original Gemini-2.5; Traditional Honeypot; S4.T2 Authenticity Score Avg.","Original GPT-5-nano; Traditional Honeypot; S4.T2 Authenticity Score Avg.","Moonshot-v1","Original Moonshot-v1; Traditional Honeypot; S4.T2 Authenticity Score Avg.","Original GPT-5.1; Traditional Honeypot; S4.T2 Authenticity Score Avg.","Original Gemini-3-Flash; Traditional Honeypot; S4.T2 Authenticity Score Avg.","Original Qwen3-Max; Traditional Honeypot; S4.T2 Authenticity Score Avg.","Original DeepSeek-V3.2; Traditional Honeypot; S4.T2 Authenticity Score Avg.","Original DeepSeek-R1; Traditional Honeypot; S4.T2 Authenticity Score Avg.","Original Hunyuan-7B; Traditional Honeypot; S4.T2 Authenticity Score Avg.","Original Llama3.1-8B; Traditional Honeypot; S4.T2 Authenticity Score Avg.","Original Mistral-7B; Traditional Honeypot; S4.T2 Authenticity Score Avg.","Original Qwen2.5-7B; Traditional Honeypot; S4.T2 Authenticity Score Avg.","Qwen3-0.6B (unthinking)","Original Qwen3-0.6B (unthinking); Traditional Honeypot; S4.T2 Authenticity Score Avg.","Qwen3-0.6B (thinking)","Original Qwen3-0.6B (thinking); Traditional Honeypot; S4.T2 Authenticity Score Avg.","Qwen3-8B (unthinking)","Original Qwen3-8B (unthinking); Traditional Honeypot; S4.T2 Authenticity Score Avg.","Qwen3-8B (thinking)","Original Qwen3-8B (thinking); Traditional Honeypot; S4.T2 Authenticity Score Avg.","Qwen3-14B (unthinking)","Original Qwen3-14B (unthinking); Traditional Honeypot; S4.T2 Authenticity Score Avg.","Qwen3-14B (thinking)","Original Qwen3-14B (thinking); Traditional Honeypot; S4.T2 Authenticity Score Avg.","SecGPT-1.5B","Original SecGPT-1.5B; Traditional Honeypot; S4.T2 Authenticity Score Avg.","SecGPT-7B","Original SecGPT-7B; Traditional Honeypot; S4.T2 Authenticity Score Avg.","SecGPT-14B","Original SecGPT-14B; Traditional Honeypot; S4.T2 Authenticity Score Avg.","Gemini CUA","Original Gemini CUA; Task Type: Overall; S4.T2 Korean · GSR","https://arxiv.org/html/2611.20488v1#S4.T2","Claude CUA","Original Claude CUA; Task Type: Overall; S4.T2 Korean · GSR","GPT CUA","Original GPT CUA; Task Type: Overall; S4.T2 Korean · GSR","MolmoWeb-8B-Native","Original MolmoWeb-8B-Native; Task Type: Overall; S4.T2 Korean · GSR","Fara-7B","Original Fara-7B; Task Type: Overall; S4.T2 Korean · GSR","Original UI-TARS-1.5-7B; Task Type: Overall; S4.T2 Korean · GSR","Original Gemini CUA; Task Type: Contextual Inference; S4.T2 Korean · GSR","Original Claude CUA; Task Type: Contextual Inference; S4.T2 Korean · GSR","Original GPT CUA; Task Type: Contextual Inference; S4.T2 Korean · GSR","Original MolmoWeb-8B-Native; Task Type: Contextual Inference; S4.T2 Korean · GSR","Original Fara-7B; Task Type: Contextual Inference; S4.T2 Korean · GSR","Original UI-TARS-1.5-7B; Task Type: Contextual Inference; S4.T2 Korean · GSR","Original Gemini CUA; Task Type: Semantic Grounding; S4.T2 Korean · GSR","Original Claude CUA; Task Type: Semantic Grounding; S4.T2 Korean · GSR","Original GPT CUA; Task Type: Semantic Grounding; S4.T2 Korean · GSR","Original MolmoWeb-8B-Native; Task Type: Semantic Grounding; S4.T2 Korean · GSR","Original Fara-7B; Task Type: Semantic Grounding; S4.T2 Korean · GSR","Original UI-TARS-1.5-7B; Task Type: Semantic Grounding; S4.T2 Korean · GSR","Original Gemini CUA; Task Type: Visual Grounding; S4.T2 Korean · GSR","Original Claude CUA; Task Type: Visual Grounding; S4.T2 Korean · GSR","Original GPT CUA; Task Type: Visual Grounding; S4.T2 Korean · GSR","Original MolmoWeb-8B-Native; Task Type: Visual Grounding; S4.T2 Korean · GSR","Original Fara-7B; Task Type: Visual Grounding; S4.T2 Korean · GSR","Original UI-TARS-1.5-7B; Task Type: Visual Grounding; S4.T2 Korean · GSR","Original mistral-small3.2:24b; S3.T1 Max out","https://arxiv.org/html/2611.20447v1#S3.T1","gemma3:27b-it-qat","Original gemma3:27b-it-qat; S3.T1 Max out","Original gpt-oss:20b; S3.T1 Max out","Original deepseek-r1:14b; S3.T1 Max out","phi4-reasoning:plus","Original phi4-reasoning:plus; S3.T1 Max out","google.gemma-4-12b-it","Original google.gemma-4-12b-it; S3.T1 Max out","meta-llama-llama-3.1-8b-instruct","Original meta-llama-llama-3.1-8b-instruct; S3.T1 Max out","mistralai_-_mistral-7b-instruct-v0.3","Original mistralai_-_mistral-7b-instruct-v0.3; S3.T1 Max out","microsoft_phi-4-mini-instruct","Original microsoft_phi-4-mini-instruct; S3.T1 Max out","granite-3.3-8b-instruct","Original granite-3.3-8b-instruct; S3.T1 Max out","qwen/qwen3.5-9b","Original qwen/qwen3.5-9b; S3.T1 Max out","google/gemma-4-12b-qat","Original google/gemma-4-12b-qat; S3.T1 Max out","google/gemma-4-e2b","Original google/gemma-4-e2b; S3.T1 Max out","deepseek/deepseek-r1-distill-qwen-7b","Original deepseek/deepseek-r1-distill-qwen-7b; S3.T1 Max out","deepseek/deepseek-r1-distill-llama-8b","Original deepseek/deepseek-r1-distill-llama-8b; S3.T1 Max out","microsoft/phi-4-reasoning-plus","Original microsoft/phi-4-reasoning-plus; S3.T1 Max out","Original Gemini 3.8 Flash; A6.T8 H3 all","https://arxiv.org/html/2611.20385v1#A6.T8","Gemini 4 Hydrogen","Original Gemini 4 Hydrogen; A6.T8 H3 all","Original Gemini 4 Argon; A6.T8 H3 all","Original Claude Opus 5.5; A6.T8 H3 all","Original Gemini 4 Carbon; A6.T8 H3 all","Claude Fable 5.2","Original Claude Fable 5.2; A6.T8 H3 all","Original Claude Opus 4.1; A6.T8 H3 all","Original Claude Sonnet 5.5; A6.T8 H3 all","Original Gemma 4 26B-A4B; A6.T8 H3 all","Original Claude Haiku 5.5; A6.T8 H3 all","Original Claude Sonnet 4.5; A6.T8 H3 all","dots-3-note","Original dots-3-note; A6.T8 H3 all","Original Qwen3.8-27B; A6.T8 H3 all","Original Nemotron 3.5 Super VL; A6.T8 H3 all","Original Muse Glimmer 30B; A6.T8 H3 all","Original Kimi-VL-A3B-Thinking; A6.T8 H3 all","Nemotron 3 Nano Omni","Original Nemotron 3 Nano Omni; A6.T8 H3 all","Original Claude Haiku 4.5; A6.T8 H3 all","Original Mistral-Small-3.2-24B; A6.T8 H3 all","Original Gemma 4 31B; A6.T8 H3 all","Original Qwen3.6-35B-A3B; A6.T8 H3 all","Original Qwen3-VL-32B; A6.T8 H3 all","Original Qwen3-VL-8B-Thinking; A6.T8 H3 all","Original InternVL3.5-38B; A6.T8 H3 all","Original Qwen3-VL-8B-Instruct; A6.T8 H3 all","Original Qwen2.5-VL-7B; A6.T8 H3 all","Original Qwen3-VL-30B-A3B; A6.T8 H3 all","Original SDXL; Open-source models: baseline; S6.T2 SCS","https://arxiv.org/html/2611.20381v1#S6.T2","StoryDiffusion","Original StoryDiffusion; Open-source models: baseline; S6.T2 SCS","Original FLUX.1-dev; Open-source models: baseline; S6.T2 SCS","Original SD3.5-Large; Open-source models: baseline; S6.T2 SCS","Original Qwen-Image-2512; Open-source models: baseline; S6.T2 SCS","SDXL + TBV","Original SDXL + TBV; Open-source models: +TBV; S6.T2 SCS","StoryDiffusion + TBV","Original StoryDiffusion + TBV; Open-source models: +TBV; S6.T2 SCS","FLUX.1-dev + TBV","Original FLUX.1-dev + TBV; Open-source models: +TBV; S6.T2 SCS","SD3.5-Large + TBV","Original SD3.5-Large + TBV; Open-source models: +TBV; S6.T2 SCS","Qwen-Image-2512 + TBV","Original Qwen-Image-2512 + TBV; Open-source models: +TBV; S6.T2 SCS","Original GPT Image 2; Closed-source models: baseline; S6.T2 SCS","Original Nano Banana; Closed-source models: baseline; S6.T2 SCS","GPT Image 2 + TBV","Original GPT Image 2 + TBV; Closed-source models: +TBV; S6.T2 SCS","Nano Banana + TBV","Original Nano Banana + TBV; Closed-source models: +TBV; S6.T2 SCS","Original Qwen2.5-Omni-7B; Zero-shot MLLMs; S3.T5 NS","https://arxiv.org/html/2611.20349v1#S3.T5","Original Qwen2.5-Omni-3B; Zero-shot MLLMs; S3.T5 NS","Original MiniCPM-o 2.6; Zero-shot MLLMs; S3.T5 NS","Original Qwen2-Audio-7B; Zero-shot MLLMs; S3.T5 NS","Original InternVL3-8B; Zero-shot MLLMs; S3.T5 NS","Original LLaVA-OV-7B; Zero-shot MLLMs; S3.T5 NS","Original Qwen2.5-VL-7B; Zero-shot MLLMs; S3.T5 NS","Original Qwen2.5-7B; Zero-shot MLLMs; S3.T5 NS","Text inversion","Original Text inversion; Zero-shot MLLMs; S3.T5 NS","Always “negative”","Original Always “negative”; Zero-shot MLLMs; S3.T5 NS","CoVeR, robust","Original CoVeR, robust; Zero-shot MLLMs; S3.T5 NS","CoVeR, accuracy","Original CoVeR, accuracy; Zero-shot MLLMs; S3.T5 NS","MMIN","Original MMIN; Zero-shot MLLMs; S3.T5 NS","MulT","Original MulT; Zero-shot MLLMs; S3.T5 NS","MoMKE","Original MoMKE; Zero-shot MLLMs; S3.T5 NS","LLM-Fusion","Original LLM-Fusion; Zero-shot MLLMs; S3.T5 NS","+ EmoMM-style state steering","Original + EmoMM-style state steering; Zero-shot MLLMs; S3.T5 NS","TFR-Net","Original TFR-Net; Zero-shot MLLMs; S3.T5 NS","IMDer","Original IMDer; Zero-shot MLLMs; S3.T5 NS","MPLMM","Original MPLMM; Zero-shot MLLMs; S3.T5 NS","LNLN","Original LNLN; Zero-shot MLLMs; S3.T5 NS","MISA","Original MISA; Zero-shot MLLMs; S3.T5 NS","DiCMoR","Original DiCMoR; Zero-shot MLLMs; S3.T5 NS","Llama-3.1-8B BF16","Original Llama-3.1-8B BF16; S4.T2 KV capacity (req.)","https://arxiv.org/html/2611.20289v1#S4.T2","Llama-3.1-8B FP8","Original Llama-3.1-8B FP8; S4.T2 KV capacity (req.)","Llama-3.1-8B NVFP4","Original Llama-3.1-8B NVFP4; S4.T2 KV capacity (req.)","Qwen3.6-27B BF16","Original Qwen3.6-27B BF16; S4.T2 KV capacity (req.)","Qwen3.6-27B FP8","Original Qwen3.6-27B FP8; S4.T2 KV capacity (req.)","Qwen3.6-27B NVFP4","Original Qwen3.6-27B NVFP4; S4.T2 KV capacity (req.)","Gemma-4-31B NVFP4","Original Gemma-4-31B NVFP4; S4.T2 KV capacity (req.)","Mistral-Medium-3.5-128B NVFP4","Original Mistral-Medium-3.5-128B NVFP4; S4.T2 KV capacity (req.)","Qwen3.6-35B-A3B BF16","Original Qwen3.6-35B-A3B BF16; S4.T2 KV capacity (req.)","Qwen3.6-35B-A3B FP8","Original Qwen3.6-35B-A3B FP8; S4.T2 KV capacity (req.)","Qwen3.6-35B-A3B NVFP4","Original Qwen3.6-35B-A3B NVFP4; S4.T2 KV capacity (req.)","Gemma-4-26B-A4B NVFP4","Original Gemma-4-26B-A4B NVFP4; S4.T2 KV capacity (req.)","Nemotron-3.5-Lightning-30B-A3B NVFP4","Original Nemotron-3.5-Lightning-30B-A3B NVFP4; S4.T2 KV capacity (req.)","Qwen3-Next-80B-A3B NVFP4","Original Qwen3-Next-80B-A3B NVFP4; S4.T2 KV capacity (req.)","Qwen3-Next-80B-A3B BF16","Original Qwen3-Next-80B-A3B BF16; S4.T2 KV capacity (req.)","gpt-oss-120b MXFP4","Original gpt-oss-120b MXFP4; S4.T2 KV capacity (req.)","Qwen3-235B-A22B NVFP4","Original Qwen3-235B-A22B NVFP4; S4.T2 KV capacity (req.)","Original GPT-6 Astra; Tool calls: 182; A7.T9 Requirements obtained (%) All","https://arxiv.org/html/2611.20218v1#A7.T9","Original Claude-Opus-5.5; Tool calls: 213; A7.T9 Requirements obtained (%) All","Original GLM 5.3; Tool calls: 785; A7.T9 Requirements obtained (%) All","Original Qwen3.8-Max; Tool calls: 696; A7.T9 Requirements obtained (%) All","Seed Evolving","Original Seed Evolving; Tool calls: 503; A7.T9 Requirements obtained (%) All","Original Qwen3.8-Flash; Tool calls: 902; A7.T9 Requirements obtained (%) All","Original Kimi K3; Tool calls: 301; A7.T9 Requirements obtained (%) All","Original DeepSeek V4.1 Flash; Tool calls: 1603; A7.T9 Requirements obtained (%) All","Original Seed 2.1 Pro; Tool calls: 229; A7.T9 Requirements obtained (%) All","Original GLM 5.2; Tool calls: 361; A7.T9 Requirements obtained (%) All","Original DeepSeek V4 Pro; Tool calls: 150; A7.T9 Requirements obtained (%) All","Original Claude Fable 5; A2.T4 Temp.","https://arxiv.org/html/2611.19643v1#A2.T4","Original Claude Opus 4.8; A2.T4 Temp.","Original Claude Sonnet 5; A2.T4 Temp.","Original GPT-5.6 Sol; A2.T4 Temp.","Original GPT-5.6 Terra; A2.T4 Temp.","Original GPT-5.6 Luna; A2.T4 Temp.","Original Qwen3.7 Plus; A2.T4 Temp.","Original Kimi K2.6; A2.T4 Temp.","Original Kimi K3; A2.T4 Temp.","Original DeepSeek-V4 Pro; A2.T4 Temp.","Original DeepSeek-V4 Flash; A2.T4 Temp.","Mistral 3 Large","Original Mistral 3 Large; A2.T4 Temp.","Original Qwen3.6 27B; A2.T4 Temp.","Original Llama 4 Scout; A2.T4 Temp.","Qwen-SEA-LION-v4.5-27B-IT","Original Qwen-SEA-LION-v4.5-27B-IT; A2.T4 Temp.","Gemma-SEA-LION-v4-27B-IT","Original Gemma-SEA-LION-v4-27B-IT; A2.T4 Temp.","Original Gemma 3 27B IT; A2.T4 Temp.","Original Haiku 4.5; A2.T13 Decisions","https://arxiv.org/html/2611.19530v1#A2.T13","Original Haiku 4.5; A2.T13 Decisions; row 3","Original Haiku 4.5; A2.T13 Decisions; row 4","Original Haiku 5.5; A2.T13 Decisions","Original Haiku 5.5; A2.T13 Decisions; row 6","Original Haiku 5.5; A2.T13 Decisions; row 7","Original Sonnet 4.6; A2.T13 Decisions","Original Sonnet 4.6; A2.T13 Decisions; row 9","Original Sonnet 4.6; A2.T13 Decisions; row 10","Original Gemini 3.5 Flash; A2.T13 Decisions","Original Gemini 3.5 Flash; A2.T13 Decisions; row 12","Original Gemini 3.5 Flash; A2.T13 Decisions; row 13","Original Gemini 3.5 Flash; A2.T13 Decisions; row 14","Original Gemini 3.5 Flash; A2.T13 Decisions; row 15","Original Gemini 3.5 Flash; A2.T13 Decisions; row 16","Original Gemini 3.5 Flash; A2.T13 Decisions; row 17","Original Gemini 3.5 Flash; A2.T13 Decisions; row 18","Original Gemini 3.5 Flash; A2.T13 Decisions; row 19","Original Gemini 3.5 Flash; A2.T13 Decisions; row 20","Original Gemini 3.5 Flash; A2.T13 Decisions; row 21","Original Gemini 3.5 Flash; A2.T13 Decisions; row 22","Original GPT-5.6 Luna; A2.T13 Decisions","Original GPT-5.6 Luna; A2.T13 Decisions; row 24","Original GPT-5.6 Luna; A2.T13 Decisions; row 25","Original GPT-5.6 Luna; A2.T13 Decisions; row 26","Original GPT-5.6 Luna; A2.T13 Decisions; row 27","Original GPT-5.6 Luna; A2.T13 Decisions; row 28","Original GPT-5.6 Luna; A2.T13 Decisions; row 29","Original GPT-5.6 Luna; A2.T13 Decisions; row 30","Original GPT-5.6 Luna; A2.T13 Decisions; row 31","Original GPT-5.6 Luna; A2.T13 Decisions; row 32","Original GPT-5.6 Luna; A2.T13 Decisions; row 33","Original GPT-5.6 Luna; A2.T13 Decisions; row 34","Original GPT-5.6 Luna; A2.T13 Decisions; row 35","Original GPT-5.6 Luna; A2.T13 Decisions; row 36","Original GPT-5.6 Luna; A2.T13 Decisions; row 37","Original GPT-5.6 Luna; A2.T13 Decisions; row 38","Original GPT-5.6 Luna; A2.T13 Decisions; row 39","Original GPT-5.6 Luna; A2.T13 Decisions; row 40","Original Llama 3.2 3B; A2.T13 Decisions","Original Llama 3.2 3B; A2.T13 Decisions; row 42","Original Llama 3.2 3B; A2.T13 Decisions; row 43","Original Llama 3.2 3B; A2.T13 Decisions; row 44","Original Llama 3.2 3B; A2.T13 Decisions; row 45","Original Llama 3.2 3B; A2.T13 Decisions; row 46","Original Llama 3.2 3B; A2.T13 Decisions; row 47","Original Llama 3.2 3B; A2.T13 Decisions; row 48","Original Llama 3.2 3B; A2.T13 Decisions; row 49","Original GLM-5.3-flash; Lang.: EN; S3.T3 Avg.","https://arxiv.org/html/2611.19364v1#S3.T3","Original GPT-6.1-sol; Lang.: EN; S3.T3 Avg.","Original Qwen2.5-VL; S3.T3 Avg.","LLaVA-OV","Original LLaVA-OV; Lang.: EN; S3.T3 Avg.","LLaVA-1.5","Original LLaVA-1.5; Lang.: EN; S3.T3 Avg.","Original InternVL3; Lang.: EN; S3.T3 Avg.","Original GeoChat; Lang.: EN; S3.T3 Avg.","SARChat","Original SARChat; Lang.: EN; S3.T3 Avg.","Original GLM-5.3-flash; Lang.: ZH; S3.T3 Avg.","Original GPT-6.1-sol; Lang.: ZH; S3.T3 Avg.","Original Qwen2.5-VL; Lang.: ZH; S3.T3 Avg.","Original LLaVA-OV; Lang.: ZH; S3.T3 Avg.","Original LLaVA-1.5; Lang.: ZH; S3.T3 Avg.","Original InternVL3; Lang.: ZH; S3.T3 Avg.","Original GeoChat; Lang.: ZH; S3.T3 Avg.","Original SARChat; Lang.: ZH; S3.T3 Avg.","Original GPT-5.6 Terra; S6.T4 Accuracy","https://arxiv.org/html/2611.23045v1#S6.T4","Original Claude Sonnet 5; S6.T4 Accuracy","Original Gemini 3.8 Flash; S6.T4 Accuracy","Original DeepSeek-V4.1-Flash; S6.T4 Accuracy","Original GPT-6.1 Sol; S6.T4 Accuracy","Market","Original Market; S6.T4 Accuracy","Fietje 2 Instruct","Original Fietje 2 Instruct; Dutch models; S4.T3 Acc.","https://arxiv.org/html/2611.22935v1#S4.T3","Fietje 2 Chat","Original Fietje 2 Chat; Dutch models; S4.T3 Acc.","GEITje 7B Ultra","Original GEITje 7B Ultra; Dutch models; S4.T3 Acc.","ChocoLlama 3 8B Instruct","Original ChocoLlama 3 8B Instruct; Dutch models; S4.T3 Acc.","Mistral 7B Instruct v0.1","Original Mistral 7B Instruct v0.1; Open-weight non-reasoning; S4.T3 Acc.","Llama 3 8B Instruct","Original Llama 3 8B Instruct; Open-weight non-reasoning; S4.T3 Acc.","Salamandra 7B Instruct","Original Salamandra 7B Instruct; Open-weight non-reasoning; S4.T3 Acc.","Original Aya Expanse 8B; Open-weight non-reasoning; S4.T3 Acc.","Original Aya Expanse 32B; Open-weight non-reasoning; S4.T3 Acc.","Original Llama 3.1 8B Instruct; Open-weight non-reasoning; S4.T3 Acc.","Original Gemma 3 4B IT; Open-weight non-reasoning; S4.T3 Acc.","Original Gemma 3 12B IT; Open-weight non-reasoning; S4.T3 Acc.","Original Gemma 3 27B IT; Open-weight non-reasoning; S4.T3 Acc.","EuroLLM-9B Instruct","Original EuroLLM-9B Instruct; Open-weight non-reasoning; S4.T3 Acc.","EuroLLM-22B Instruct","Original EuroLLM-22B Instruct; Open-weight non-reasoning; S4.T3 Acc.","Ministral 3 14B Instruct","Original Ministral 3 14B Instruct; Open-weight non-reasoning; S4.T3 Acc.","Original Qwen3-8B; Open-weight reasoning; S4.T3 Acc.","Original Qwen3.5-9B; Open-weight reasoning; S4.T3 Acc.","Phi-4 Mini Reasoning","Original Phi-4 Mini Reasoning; Open-weight reasoning; S4.T3 Acc.","Phi-4 Reasoning","Original Phi-4 Reasoning; Open-weight reasoning; S4.T3 Acc.","Original GPT-OSS-20B; Open-weight reasoning; S4.T3 Acc.","Original GPT-4o (2024-11-20); Open-weight reasoning; S4.T3 Acc.","GPT-4.1 (2025-04-14)","Original GPT-4.1 (2025-04-14); Open-weight reasoning; S4.T3 Acc.","GPT-5.6 Luna (none)","Original GPT-5.6 Luna (none); Open-weight reasoning; S4.T3 Acc.","Original GPT-5.6 Luna (max); Open-weight reasoning; S4.T3 Acc.","GPT-5.6 Sol (none)","Original GPT-5.6 Sol (none); Open-weight reasoning; S4.T3 Acc.","Original GPT-5.6 Sol (max); Open-weight reasoning; S4.T3 Acc.","Original GPT-5.4; S4.T3 Overall","https://arxiv.org/html/2611.22905v1#S4.T3","Original GPT-5; S4.T3 Overall","Original GPT-4.1 Mini; S4.T3 Overall","Original Gemini 3.1 Flash; S4.T3 Overall","Original Gemini 2.5 Flash; S4.T3 Overall","Original Claude Opus 4.6; S4.T3 Overall","Original Claude Sonnet 4.6; S4.T3 Overall","DeepSeek-R1 8B","Original DeepSeek-R1 8B; S4.T3 Overall","Original Qwen2.5 7B; S4.T3 Overall","Original Muse Spark 1.3; Social Signal Bench Video; A7.T10 Eng. MF1","https://arxiv.org/html/2611.22704v1#A7.T10","Original Gemini 3.8 Flash; Social Signal Bench Video; A7.T10 Eng. MF1","Original Qwen3.5-Omni-Plus; Social Signal Bench Video; A7.T10 Eng. MF1","Original Gemini 4 Argon; Social Signal Bench Video; A7.T10 Eng. MF1","Original GPT-6.1 Sol; Social Signal Bench Video; A7.T10 Eng. MF1","Original Gemma-4-31B-it; Social Signal Bench Video; A7.T10 Eng. MF1","Qwen2.5-Omni-7B (untuned)","Original Qwen2.5-Omni-7B (untuned); Social Signal Bench Video; A7.T10 Eng. MF1","Qwen2.5-Omni-7B (post-trained)","Original Qwen2.5-Omni-7B (post-trained); Social Signal Bench Video; A7.T10 Eng. MF1","Original Gemini 3.1 Pro; Social Signal Bench Audio; A7.T10 Eng. MF1","Original Gemini 3.8 Flash; Social Signal Bench Audio; A7.T10 Eng. MF1","Original Qwen3.5-Omni-Plus; Social Signal Bench Audio; A7.T10 Eng. MF1","GPT-Audio","Original GPT-Audio; Social Signal Bench Audio; A7.T10 Eng. MF1","Kimi-Audio-7B-Instruct","Original Kimi-Audio-7B-Instruct; Social Signal Bench Audio; A7.T10 Eng. MF1","Qwen2-Audio-7B-Instruct","Original Qwen2-Audio-7B-Instruct; Social Signal Bench Audio; A7.T10 Eng. MF1","Original Audio-Flamingo-3; Social Signal Bench Audio; A7.T10 Eng. MF1","Ultravox-v0.5-Llama-3.1-8B","Original Ultravox-v0.5-Llama-3.1-8B; Social Signal Bench Audio; A7.T10 Eng. MF1","Qwen2-Audio-7B-Instruct (post-trained)","Original Qwen2-Audio-7B-Instruct (post-trained); Social Signal Bench Audio; A7.T10 Eng. MF1","Original GPT-6.1 Sol; S4.T2 Real (200) Overall","https://arxiv.org/html/2611.22657v1#S4.T2","Original Gemini 4 Argon; S4.T2 Real (200) Overall","Qwen3-VL-8B Base","Original Qwen3-VL-8B Base; S4.T2 Real (200) Overall","SpatialCast-GRPO (8B)","Original SpatialCast-GRPO (8B); S4.T2 Real (200) Overall","Qwen3-VL-4B Base","Original Qwen3-VL-4B Base; S4.T2 Real (200) Overall","Spatial-SSRL-4B","Original Spatial-SSRL-4B; S4.T2 Real (200) Overall","SpatialBlock-4B","Original SpatialBlock-4B; S4.T2 Real (200) Overall","SpatialCast-GRPO (4B)","Original SpatialCast-GRPO (4B); S4.T2 Real (200) Overall","Qwen2.5-VL-3B Base","Original Qwen2.5-VL-3B Base; S4.T2 Real (200) Overall","Spatial-MLLM","Original Spatial-MLLM; S4.T2 Real (200) Overall","SpatialLadder-3B","Original SpatialLadder-3B; S4.T2 Real (200) Overall","Spatial-SSRL-3B","Original Spatial-SSRL-3B; S4.T2 Real (200) Overall","SpatialCast-GRPO (3B)","Original SpatialCast-GRPO (3B); S4.T2 Real (200) Overall","InternVL3-2B Base","Original InternVL3-2B Base; S4.T2 Real (200) Overall","SpatialCast-GRPO (2B)","Original SpatialCast-GRPO (2B); S4.T2 Real (200) Overall","GPT-4o (text)","Original GPT-4o (text); Text-only LLMs; S4.T2 Overall","https://arxiv.org/html/2611.22514v1#S4.T2","Original Claude-3.5-Sonnet; Text-only LLMs; S4.T2 Overall","Original DeepSeek-V3; Text-only LLMs; S4.T2 Overall","Original Qwen3-32B; Text-only LLMs; S4.T2 Overall","Original Llama-3.3-70B; Text-only LLMs; S4.T2 Overall","Original GPT-5.5; Text-only LLMs; S4.T2 Overall","Original GPT-4o; Text-only LLMs; S4.T2 Overall","Original GPT-4.1; Text-only LLMs; S4.T2 Overall","Original Gemini-2.5-Pro; Text-only LLMs; S4.T2 Overall","Original Gemini-3-Flash; Text-only LLMs; S4.T2 Overall","Original Gemini-3.5-Flash; Text-only LLMs; S4.T2 Overall","Original Claude-4.5-Sonnet; Text-only LLMs; S4.T2 Overall","Original Claude Opus 4.7; Text-only LLMs; S4.T2 Overall","Original Qwen3-VL-235B; Text-only LLMs; S4.T2 Overall","Original DeepSeek-R1; Code-augmented Multimodal Systems; S4.T2 Overall","Qwen3-Coder-32B","Original Qwen3-Coder-32B; Code-augmented Multimodal Systems; S4.T2 Overall","Original TableGPT2-7B; Code-augmented Multimodal Systems; S4.T2 Overall","Gemini Deep Research","Original Gemini Deep Research; Deep Research Agents; S4.T2 Overall","ChatGPT Deep Research","Original ChatGPT Deep Research; Deep Research Agents; S4.T2 Overall","Perplexity Sonar DR","Original Perplexity Sonar DR; Deep Research Agents; S4.T2 Overall","Codex / GPT-5.6 Sol","Original Codex / GPT-5.6 Sol; S5.T4 Physics","https://arxiv.org/html/2611.22490v1#S5.T4","Claude Code / Opus 5","Original Claude Code / Opus 5; S5.T4 Physics","Claude Code / Qwen3.8-Max","Original Claude Code / Qwen3.8-Max; S5.T4 Physics","Claude Code / Opus 4.8","Original Claude Code / Opus 4.8; S5.T4 Physics","Claude Code / Kimi K3","Original Claude Code / Kimi K3; S5.T4 Physics","Claude Code / GLM-5.3-Flash","Original Claude Code / GLM-5.3-Flash; S5.T4 Physics","Codex / GPT-5.5","Original Codex / GPT-5.5; S5.T4 Physics","Codex / Kimi K3","Original Codex / Kimi K3; S5.T4 Physics","Codex / Seed2.1 Pro","Original Codex / Seed2.1 Pro; S5.T4 Physics","Codex / Gemini 3.5 Flash","Original Codex / Gemini 3.5 Flash; S5.T4 Physics","Codex / GLM-5.3-Flash","Original Codex / GLM-5.3-Flash; S5.T4 Physics","Claude Code / Seed2.1 Pro","Original Claude Code / Seed2.1 Pro; S5.T4 Physics","Claude Code / Gemini 3.5 Flash","Original Claude Code / Gemini 3.5 Flash; S5.T4 Physics","LLaVA-NeXT-8B","Original LLaVA-NeXT-8B; Language Model: Llama-3-8B-Instruct; S4.T2 Contra-Acc( % )","https://arxiv.org/html/2611.22360v1#S4.T2","Original LLaVA-NeXT-8B; S4.T2 Contra-Acc( % )","Original LLaVA-NeXT-8B; S4.T2 Contra-Acc( % ); row 9","Original DeepSeek-VL-7B-Chat; Language Model: DeepSeek-LLM-7B; S4.T2 Contra-Acc( % )","Original DeepSeek-VL-7B-Chat; S4.T2 Contra-Acc( % )","Original DeepSeek-VL-7B-Chat; S4.T2 Contra-Acc( % ); row 12","Original Llama-3.2-11B-Vision; Language Model: Llama3.1-8B; S4.T2 Contra-Acc( % )","Original Llama-3.2-11B-Vision; S4.T2 Contra-Acc( % )","Original Llama-3.2-11B-Vision; S4.T2 Contra-Acc( % ); row 15","Original Qwen2.5-VL-7B-Instruct; Language Model: Qwen2.5 LLM; S4.T2 Contra-Acc( % )","Original Gemini-1.5-pro; Close-source Models; S4.T2 Contra-Acc( % )","Original Gemini-1.5-pro; Close-source Models; S4.T2 Contra-Acc( % ); row 21","Original Gemini-1.5-pro; Close-source Models; S4.T2 Contra-Acc( % ); row 22","Gemini-2.0-flash","Original Gemini-2.0-flash; Close-source Models; S4.T2 Contra-Acc( % )","Original Gemini-2.0-flash; Close-source Models; S4.T2 Contra-Acc( % ); row 24","Original Gemini-2.0-flash; Close-source Models; S4.T2 Contra-Acc( % ); row 25","Original GPT-4o; Close-source Models; S4.T2 Contra-Acc( % )","Original GPT-4o; Close-source Models; S4.T2 Contra-Acc( % ); row 27","Original GPT-4o; Close-source Models; S4.T2 Contra-Acc( % ); row 28","Original GPT-4.1; Close-source Models; S4.T2 Contra-Acc( % )","Original kimi-k3; S6.T4 Full med.","https://arxiv.org/html/2611.22262v1#S6.T4","Original glm-5.2; S6.T4 Full med.","Original qwen3.7-plus; S6.T4 Full med.","Original deepseek-v4-pro; S6.T4 Full med.","Original glm-5.1; S6.T4 Full med.","Original qwen3.7-max; S6.T4 Full med.","Original deepseek-v4-flash; S6.T4 Full med.","qwen3.6-max","Original qwen3.6-max; S6.T4 Full med.","Original qwen3.6-plus; S6.T4 Full med.","Original kimi-k2.6; S6.T4 Full med.","Original Qwen2.5-7B; S5.T2 V2 − V3","https://arxiv.org/html/2611.22241v1#S5.T2","Original Qwen2.5-14B; S5.T2 V2 − V3","Original Qwen2.5-32B; S5.T2 V2 − V3","Original Phi-4; S5.T2 V2 − V3","Original Llama-3.1-8B; S5.T2 V2 − V3","Mean (four)","Original Mean (four); S5.T2 V2 − V3","masked","Original masked; S5.T2 V2 − V3","14B, alt. prompt","Original 14B, alt. prompt; S5.T2 V2 − V3","E1, validation","Original E1, validation; S5.T2 V2 − V3","Text classifier","Original Text classifier; S5.T2 V2 − V3","Original Adaptive; S2.T3 per task","https://arxiv.org/html/2611.22138v1#S2.T3","Hierarchical","Original Hierarchical; S2.T3 per task","Decentralized","Original Decentralized; S2.T3 per task","Single-agent","Original Single-agent; S2.T3 per task","Robot tool only","Original Robot tool only; S2.T3 per task","Low resolution","Original Low resolution; S2.T3 per task","Original gpt-6.1-sol; S2.T3 per task","opus-5.5","Original opus-5.5; S2.T3 per task","Original opus-5.5; Other models (adaptive system, Codex); S2.T3 per task","kimi-k3.1-pro","Original kimi-k3.1-pro; Other models (adaptive system, Codex); S2.T3 per task","qwen3.8-flash","Original qwen3.8-flash; Other models (adaptive system, Codex); S2.T3 per task","Original deepseek-v4.1-flash; Other models (adaptive system, Codex); S2.T3 per task","Original glm-5.3-flash; Other models (adaptive system, Codex); S2.T3 per task","Original Claude Opus 5; S5.T3 Mean score","https://arxiv.org/html/2611.22085v1#S5.T3","Original Claude Fable 5.1; S5.T3 Mean score","Original GPT-6 Astra; S5.T3 Mean score","Original Kimi K3; S5.T3 Mean score","Original Grok 4.6; S5.T3 Mean score","Original GLM 5.3; S5.T3 Mean score","Original Gemini 3.8 Flash; S5.T3 Mean score","Original GPT-5.6 Sol; S5.T3 Mean score","Original Gemini 3.7 Flash; S5.T3 Mean score","Original DeepSeek V4 Pro; S5.T3 Mean score","Original Gemini 3.1 Pro Preview; S5.T3 Mean score","GPT (225)","Original GPT (225); A1.T5 Event R","https://arxiv.org/html/2611.22009v1#A1.T5","GPT (502)","Original GPT (502); A1.T5 Event R","Claude (225)","Original Claude (225); A1.T5 Event R","Claude (498)","Original Claude (498); A1.T5 Event R","Gemini (225)","Original Gemini (225); A1.T5 Event R","Gemini (502)","Original Gemini (502); A1.T5 Event R","Kimi (225)","Original Kimi (225); A1.T5 Event R","Original GPT-5.6 Luna; S6.T3 zephyr ( N = 107 ) · L3-I","https://arxiv.org/html/2611.21845v1#S6.T3","Original GPT-5.6 Sol; S6.T3 zephyr ( N = 107 ) · L3-I","Original GPT-6 Luna; S6.T3 zephyr ( N = 107 ) · L3-I","Original GPT-6.1 Sol; S6.T3 zephyr ( N = 107 ) · L3-I","Original Gemini 3.8 Flash; S6.T3 zephyr ( N = 107 ) · L3-I","Original Grok 4.6; S6.T3 zephyr ( N = 107 ) · L3-I","Original Nemotron 3.5 Super VL; S6.T3 zephyr ( N = 107 ) · L3-I","Original Muse Glimmer 30B; S6.T3 zephyr ( N = 107 ) · L3-I","Original GPT-6 Astra; S5.T3 Envelope","https://arxiv.org/html/2611.21707v1#S5.T3","Original GPT-6.1 Sol; S5.T3 Envelope","Original Claude Opus 5.5; S5.T3 Envelope","Original GPT-6.1 Terra; S5.T3 Envelope","Original Muse Spark 1.3; S5.T3 Envelope","Original Claude Sonnet 5.5; S5.T3 Envelope","Original Mistral Large 4; S5.T3 Envelope","Original DeepSeek V4.1 Flash; S5.T3 Envelope","Original Claude Haiku 5.5; S5.T3 Envelope","Original GPT-6 Luna; S5.T3 Envelope","Qwen3.5-9B + RL (ours)","Original Qwen3.5-9B + RL (ours); S5.T3 Envelope","Original Qwen3.5-9B; S5.T3 Envelope","Original gemma-4-26B-A4B-it; S4.T2 Average","https://arxiv.org/html/2611.24762v1#S4.T2","Original InternVideo2.5-8B; S4.T2 Average","Original InternVL2.5-8B; S4.T2 Average","Original InternVL3-9B; S4.T2 Average","Original InternVL3.5-8B; S4.T2 Average","Original LLaVA-Video-7B-Qwen2; S4.T2 Average","Original Ovis2.5-9B; S4.T2 Average","Qwen2.5-VL-7B-","Original Qwen2.5-VL-7B-; S4.T2 Average","Qwen3-VL-8B-","Original Qwen3-VL-8B-; S4.T2 Average","Original VideoLLaMA3-7B; S4.T2 Average","Qwen2.5-VL-72B-","Original Qwen2.5-VL-72B-; S4.T2 Average","Original GPT 5.1; S4.T2 Average","Original Qwen3-VL-8B-; S4.T2 Average; row 16","InternVL3.5-8B CoT prompt","Original InternVL3.5-8B CoT prompt; S4.T2 Average","Original InternVL3.5-8B; S4.T2 Average; row 18","Original Opus 4.8; Sx6.T2 V + T","https://arxiv.org/html/2611.24517v1#Sx6.T2","Qwen-A10B","Original Qwen-A10B; Sx6.T2 V + T","Qwen-27B","Original Qwen-27B; Sx6.T2 V + T","Original GPT-5.5; Sx6.T2 V + T","Original Haiku 4.5; Sx6.T2 V + T","Original Gemini 4 Argon; Tool use (%): 88.1; S5.T2 Gain","https://arxiv.org/html/2611.24184v1#S5.T2","Original Gemini 3.8 Flash; Tool use (%): 86.4; S5.T2 Gain","Original Gemini 3.6 Flash; Tool use (%): 79.8; S5.T2 Gain","Original Qwen3.8-Flash; Tool use (%): 76.2; S5.T2 Gain","Original Qwen3.7-Flash; Tool use (%): 59.3; S5.T2 Gain","Original Gemini 2.5 Pro; S3.T1 Avg. IoU","https://arxiv.org/html/2611.24098v1#S3.T1","Gemini 3.0 Pro","Original Gemini 3.0 Pro; S3.T1 Avg. IoU","Original GPT-5.4; S3.T1 Avg. IoU","Original GPT-5.5; S3.T1 Avg. IoU","Original Qwen3.5-Plus; S3.T1 Avg. IoU","Original Qwen3.7-Plus; S3.T1 Avg. IoU","Original GLM-5V-Turbo; S3.T1 Avg. IoU","Original Seed-2.0-Lite; S3.T1 Avg. IoU","Original DeepSeek V4.1 Flash; Only Text as Input; S4.T2 OA","https://arxiv.org/html/2611.24068v1#S4.T2","EarthDial-4B-RGB","Original EarthDial-4B-RGB; Only Text as Input; S4.T2 OA","Original SkyEyeGPT; Only Text as Input; S4.T2 OA","TEOChat-7B","Original TEOChat-7B; Only Text as Input; S4.T2 OA","Original GeoChat-7B; Only Text as Input; S4.T2 OA","LHRS-Bot-Nova","Original LHRS-Bot-Nova; Only Text as Input; S4.T2 OA","Original Molmo2-8B; Only Text as Input; S4.T2 OA","Original InternVL3.5-8B; Only Text as Input; S4.T2 OA","LLaVA-OneVision-2","Original LLaVA-OneVision-2; Only Text as Input; S4.T2 OA","Original Qwen3.5-9B; Only Text as Input; S4.T2 OA","Original Claude Opus 5.5; Only Text as Input; S4.T2 OA","Gemini 3.8","Original Gemini 3.8; Only Text as Input; S4.T2 OA","Original Grok 4.7; Only Text as Input; S4.T2 OA","Original Qwen3.8 Max; Only Text as Input; S4.T2 OA","Original DeepSeek V4.1 Flash; Only Text as Input; S4.T2 OA; row 20","Original Kimi K3.1 Pro; Only Text as Input; S4.T2 OA","Original GLM 5.3 Flash; Only Text as Input; S4.T2 OA","Original Doubao Seed 2.1 Pro; Only Text as Input; S4.T2 OA","Nemotron 3.5","Original Nemotron 3.5; Only Text as Input; S4.T2 OA","Original GPT-6.1 Sol; Only Text as Input; S4.T2 OA","GeoMM-Agent","Original GeoMM-Agent; S4.T2 OA","HyGeoAgent DeepSeek","Original HyGeoAgent DeepSeek; S4.T2 OA","HyGeoAgent GPT","Original HyGeoAgent GPT; S4.T2 OA","HyGeoAgent Claude","Original HyGeoAgent Claude; S4.T2 OA","Original DeepSeek-V3; Pre-cutoff; S5.T2 Verify","https://arxiv.org/html/2611.23601v1#S5.T2","Llama-4-M","Original Llama-4-M; Pre-cutoff; S5.T2 Verify","Original GPT-4.1-Mini; Pre-cutoff; S5.T2 Verify","Original GPT-5.1; Pre-cutoff; S5.T2 Verify","Original o4-mini; Pre-cutoff; S5.T2 Verify","Original DeepSeek-V3; Post-cutoff; S5.T2 Verify","Original Llama-4-M; Post-cutoff; S5.T2 Verify","Original GPT-4.1-Mini; Post-cutoff; S5.T2 Verify","Original GPT-5.1; Post-cutoff; S5.T2 Verify","Original o4-mini; Post-cutoff; S5.T2 Verify","Claude-3.7","Original Claude-3.7; Post-cutoff; S5.T2 Verify","Original GPT-5; S4.T6 Propagation","https://arxiv.org/html/2611.23533v1#S4.T6","Original Gemini-3.6-Flash; S4.T6 Propagation","Original Llama-3.1-8B; S4.T6 Propagation","Original Llama-3.3-70B; S4.T6 Propagation","Original Qwen3-8B; S4.T6 Propagation","Original Qwen3-32B; S4.T6 Propagation","AnalogSeeker","Original AnalogSeeker; A9.T8 Strict all ≤ 3 dB","https://arxiv.org/html/2611.23410v1#A9.T8","Claude 4.5","Original Claude 4.5; A9.T8 Strict all ≤ 3 dB","Original GPT-5.4-mini; A9.T8 Strict all ≤ 3 dB","Llama 3.2-11B","Original Llama 3.2-11B; A9.T8 Strict all ≤ 3 dB","Original Qwen3-VL-8B; A9.T8 Strict all ≤ 3 dB","Qwen3 SFT","Original Qwen3 SFT; A9.T8 Strict all ≤ 3 dB","Qwen3 GRPO","Original Qwen3 GRPO; A9.T8 Strict all ≤ 3 dB"],"benchmarks":[{"id":"swe-bench-verified","name":"SWE-bench Verified","group":"编程","unit":"%","metric":"Resolved","protocol":{"en":"500 Verified tasks; mini-SWE-agent, one attempt. Agent version is shown for each result.","zh":"500 项 Verified 任务；mini-SWE-agent，单次尝试。各条成绩注明 Agent 版本。"},"source":{"name":"SWE-bench","url":"https://www.swebench.com/","status":"current_snapshot","published_at":"2024-08-13"},"results":[[0,0.768,1,2,-1],[3,0.758,4,2,-1],[5,0.758,6,2,-1],[7,0.7559999999999999,1,2,-1],[8,0.7440000000000001,1,9,-1],[10,0.742,4,11,-1],[12,0.728,13,2,-1],[14,0.728,15,2,-1],[16,0.728,15,2,-1],[17,0.718,15,18,-1],[19,0.7140000000000001,1,2,-1],[20,0.708,21,2,-1],[22,0.706,1,23,-1],[24,0.7,25,2,-1],[26,0.696,4,2,-1],[27,0.69,15,18,-1],[28,0.6759999999999999,1,29,-1],[30,0.6659999999999999,1,2,-1],[31,0.66,15,11,-1],[32,0.66,15,9,-1],[33,0.65,15,34,-1],[35,0.6493000000000001,1,29,-1],[36,0.634,21,18,-1],[37,0.61,6,38,-1],[39,0.6,40,41,-1],[42,0.598,15,34,-1],[43,0.584,15,29,-1],[44,0.564,45,18,-1],[46,0.562,15,2,-1],[47,0.5539999999999999,48,41,-1],[49,0.5539999999999999,50,29,-1],[51,0.542,48,52,-1],[53,0.5379999999999999,45,18,-1],[54,0.536,4,29,-1],[55,0.528,1,56,-1],[57,0.45,15,29,-1],[58,0.43799999999999994,21,34,-1],[59,0.3958,15,29,-1],[60,0.348,15,34,-1],[61,0.2873,4,29,-1],[62,0.26,15,34,-1],[63,0.2394,15,56,-1],[64,0.2162,15,56,-1],[65,0.2104,66,56,-1],[67,0.1352,4,56,-1],[68,0.0906,66,56,-1],[69,0.09,50,29,-1]]},{"id":"arc-agi-2","name":"ARC-AGI-2","group":"知识与推理","unit":"%","metric":"Accuracy","protocol":{"en":"Semi-private evaluation; published LLM and reasoning configurations. Human baselines and competition systems are excluded.","zh":"半私有评测集；保留官方公开的大模型及推理配置成绩，不含人类基线和竞赛专用系统。"},"source":{"name":"ARC Prize","url":"https://arcprize.org/leaderboard","status":"current_snapshot","published_at":"2025-05-17"},"results":[[70,0.95,15,71,-1],[72,0.9416666666666667,15,71,-1],[73,0.9333333333333333,1,71,-1],[74,0.9333333333333333,15,71,-1],[75,0.925,1,71,-1],[76,0.925,15,71,-1],[77,0.9208333333333333,15,71,-1],[78,0.9208333333333333,15,71,-1],[79,0.9166666666666666,1,71,-1],[80,0.9166666666666666,15,71,-1],[81,0.9166666666666666,15,71,-1],[82,0.9125,83,71,-1],[84,0.9042,1,71,-1],[85,0.9,1,71,-1],[86,0.9,1,71,-1],[87,0.9,15,71,-1],[88,0.8958333333333334,15,71,-1],[89,0.8916666666666667,1,71,-1],[90,0.8916666666666667,83,71,-1],[91,0.8875,1,71,-1],[92,0.8833333333333333,1,71,-1],[93,0.8833,1,71,-1],[94,0.875,1,71,-1],[95,0.875,1,71,-1],[96,0.8708333333333333,83,71,-1],[97,0.8666666666666667,15,71,-1],[98,0.8625,1,71,-1],[99,0.8542,15,71,-1],[100,0.8541666666666666,15,71,-1],[101,0.85,15,71,-1],[102,0.8458333333333333,83,71,-1],[103,0.8458,15,71,-1],[104,0.8458,83,71,-1],[105,0.8416,15,71,-1],[106,0.839,15,71,-1],[107,0.8388888888888889,1,71,-1],[108,0.8333,15,71,-1],[109,0.8333,15,71,-1],[110,0.8291666666666667,83,71,-1],[111,0.825,1,71,-1],[112,0.7833333333333333,1,71,-1],[113,0.7805555555555556,25,71,-1],[114,0.7805555555555556,15,71,-1],[115,0.775,83,71,-1],[116,0.7708,83,71,-1],[117,0.7680555555555556,118,71,-1],[119,0.7680555555555555,1,71,-1],[120,0.7666666666666667,15,71,-1],[121,0.7583,1,71,-1],[122,0.7417,15,71,-1],[123,0.7395,15,71,-1],[124,0.7361111111111112,25,71,-1],[125,0.7291666666666666,25,71,-1],[126,0.729,15,127,-1],[128,0.7208,1,71,-1],[129,0.7208,83,71,-1],[130,0.7167,1,71,-1],[131,0.7042,15,71,-1],[132,0.701388888888889,1,71,-1],[133,0.6917,1,71,-1],[134,0.6888888888888888,15,71,-1],[135,0.6875,1,71,-1],[136,0.6861111111111112,25,71,-1],[137,0.6833,1,71,-1],[138,0.675,1,71,-1],[139,0.675,25,71,-1],[140,0.675,15,71,-1],[141,0.6736111111111112,1,71,-1],[142,0.6708333333333333,143,71,-1],[144,0.6708,15,71,-1],[145,0.6708,15,71,-1],[146,0.6625,1,71,-1],[147,0.6583333333333333,48,71,-1],[148,0.6514,143,71,-1],[149,0.6513888888888888,143,71,-1],[150,0.6458,1,71,-1],[151,0.6375,83,71,-1],[152,0.6222,1,71,-1],[153,0.6208,1,71,-1],[154,0.6138888888888888,25,71,-1],[155,0.6138888888888888,143,71,-1],[156,0.6125,1,71,-1],[157,0.6125,25,71,-1],[158,0.6125,143,71,-1],[159,0.6055555555555555,25,71,-1],[160,0.6042,1,71,-1],[161,0.6041666666666666,83,71,-1],[162,0.6041666666666666,21,71,-1],[163,0.5972222222222221,25,71,-1],[164,0.5958333333333333,15,71,-1],[165,0.5958333333333333,15,71,-1],[166,0.5954,15,71,-1],[167,0.5930555555555556,15,71,-1],[168,0.5875,143,71,-1],[169,0.5833333333333334,143,71,-1],[170,0.5833,1,71,-1],[171,0.5777777777777778,15,71,-1],[172,0.575,173,71,-1],[174,0.5625,25,71,-1],[175,0.5597222222222221,25,71,-1],[176,0.5542,15,71,-1],[177,0.55,21,71,-1],[178,0.5416,15,71,-1],[179,0.54,83,180,-1],[181,0.5291666666666667,83,71,-1],[182,0.5291,15,71,-1],[183,0.5264,143,71,-1],[184,0.5264,143,71,-1],[185,0.5041666666666667,83,71,-1],[186,0.5013888888888889,48,71,-1],[187,0.4764,15,71,-1],[188,0.475,173,71,-1],[189,0.4597222222222222,25,71,-1],[190,0.4514,83,71,-1],[191,0.4347222222222222,15,71,-1],[192,0.4333,15,71,-1],[193,0.425,15,71,-1],[194,0.4236111111111111,195,71,-1],[196,0.42083333333333334,83,71,-1],[197,0.41944444444444445,15,71,-1],[198,0.40138888888888885,199,71,-1],[200,0.3847,15,71,-1],[201,0.3764,1,71,-1],[202,0.375,15,71,-1],[203,0.36527777777777776,199,71,-1],[204,0.3361,83,71,-1],[205,0.3333333333333333,15,71,-1],[206,0.3333,15,71,-1],[207,0.3306,143,71,-1],[208,0.33055555555555555,199,71,-1],[209,0.3277777777777777,1,71,-1],[210,0.3152777777777777,15,71,-1],[211,0.3138888888888889,15,71,-1],[26,0.3111,83,71,-1],[212,0.3069444444444444,1,71,-1],[213,0.30416666666666664,83,71,-1],[214,0.2944,143,215,-1],[216,0.2931,15,71,-1],[217,0.2917,15,71,-1],[218,0.2791666666666667,48,71,-1],[219,0.2763888888888889,143,71,-1],[220,0.275,173,71,-1],[221,0.2667,15,71,-1],[222,0.24027777777777776,1,71,-1],[223,0.23472222222222222,224,71,-1],[225,0.2278,1,71,-1],[226,0.22777777777777777,195,71,-1],[227,0.22775,48,71,-1],[228,0.2,224,71,-1],[229,0.189,15,71,-1],[230,0.1875,15,71,-1],[231,0.1833,15,71,-1],[232,0.18055555555555555,15,71,-1],[233,0.1764,15,71,-1],[234,0.16666666666666666,143,71,-1],[235,0.15975,143,71,-1],[236,0.1389,1,71,-1],[237,0.13611111111111113,199,71,-1],[238,0.1361,1,71,-1],[239,0.13194444444444445,195,71,-1],[240,0.1319,15,71,-1],[241,0.1278,83,71,-1],[242,0.12361111111111112,21,71,-1],[243,0.12222222222222222,173,71,-1],[244,0.1181,21,71,-1],[245,0.10277777777777779,83,71,-1],[246,0.0986,15,71,-1],[247,0.0972,15,71,-1],[248,0.09444444444444443,1,71,-1],[249,0.0889,83,71,-1],[250,0.0861,1,71,-1],[251,0.0778,1,71,-1],[252,0.0749,15,71,-1],[253,0.0736111111111111,15,71,-1],[254,0.0736,15,71,-1],[255,0.0694,1,71,-1],[256,0.0694,1,71,-1],[257,0.0653,15,71,-1],[258,0.0653,15,71,-1],[259,0.0638888888888889,224,71,-1],[260,0.0611,15,71,-1],[261,0.0593,1,71,-1],[262,0.0583,1,71,-1],[263,0.0569,15,71,-1],[264,0.0528,143,71,-1],[265,0.05277777777777778,83,71,-1],[266,0.0514,15,71,-1],[267,0.04861111111111112,199,71,-1],[268,0.0486,48,71,-1],[269,0.0486,83,71,-1],[270,0.0486,6,71,-1],[271,0.0486,15,272,-1],[273,0.04583333333333333,15,71,-1],[274,0.0452,1,71,-1],[275,0.0444,15,71,-1],[276,0.0444,15,71,-1],[277,0.0403,1,71,-1],[278,0.0403,25,279,-1],[280,0.0403,15,71,-1],[281,0.0403,83,71,-1],[282,0.04027777777777777,199,71,-1],[283,0.0375,1,279,-1],[284,0.03611111111111111,15,71,-1],[285,0.0361,15,71,-1],[286,0.0333,83,71,-1],[287,0.0299,15,71,-1],[288,0.0298,15,71,-1],[289,0.0292,83,71,-1],[290,0.0278,1,71,-1],[291,0.02777777777777778,224,71,-1],[292,0.027777777777777776,1,71,-1],[293,0.02638888888888889,83,71,-1],[294,0.0261,15,71,-1],[295,0.0254,83,71,-1],[296,0.0236,15,71,-1],[297,0.0216,83,71,-1],[298,0.0212,1,71,-1],[299,0.0212,83,71,-1],[300,0.020833333333333332,25,71,-1],[301,0.020833333333333332,25,71,-1],[302,0.0208,15,71,-1],[303,0.0205,15,272,-1],[304,0.0199,15,71,-1],[305,0.0198,83,71,-1],[306,0.019444444444444445,307,71,-1],[308,0.0194,15,71,-1],[309,0.0194,15,71,-1],[310,0.0194,15,71,-1],[311,0.0194,15,272,-1],[312,0.0169,83,71,-1],[313,0.0167,1,71,-1],[314,0.0167,15,71,-1],[315,0.0167,15,71,-1],[316,0.016666666666666666,15,71,-1],[317,0.0153,15,71,-1],[318,0.015277777777777777,83,71,-1],[319,0.015277777777777777,307,71,-1],[320,0.015277777777777777,195,71,-1],[321,0.013,25,71,-1],[67,0.013,83,279,-1],[322,0.0127,1,71,-1],[323,0.0127,1,279,-1],[324,0.0127,15,71,-1],[325,0.0125,1,279,-1],[326,0.0125,1,71,-1],[327,0.0125,25,71,-1],[328,0.0125,83,71,-1],[329,0.0125,195,279,-1],[330,0.0112,25,71,-1],[331,0.0111,15,71,-1],[332,0.009,1,71,-1],[333,0.0088,15,71,-1],[334,0.0085,1,71,-1],[335,0.008333333333333333,25,71,-1],[336,0.008333333333333333,15,71,-1],[337,0.008333333333333333,83,71,-1],[338,0.008333333333333333,307,71,-1],[339,0.0083,15,71,-1],[340,0.0083,15,279,-1],[341,0.0083,15,71,-1],[342,0.008,15,279,-1],[343,0.008,83,279,-1],[344,0.007,1,71,-1],[345,0.0042,15,279,-1],[346,0.0042,15,71,-1],[347,0.0042,143,71,-1],[348,0.004,1,71,-1],[349,0.0,1,279,-1],[350,0.0,1,71,-1],[351,0.0,15,279,-1],[352,0.0,15,279,-1],[353,0.0,15,279,-1],[354,0.0,15,279,-1],[355,0.0,15,279,-1],[356,0.0,15,71,-1],[357,0.0,15,71,-1],[358,0.0,15,71,-1],[359,0.0,83,71,-1],[360,0.0,143,279,-1],[361,0.0,199,71,-1],[362,0.0,66,279,-1],[363,0.0,66,279,-1],[364,0.0,365,71,-1],[366,0.0,365,71,-1],[367,0.0,365,71,-1],[368,0.0,15,71,-1]]},{"id":"frontiermath-tiers-1-3-v2","name":"FrontierMath Tiers 1–3 (v2)","group":"数学与科学","unit":"%","metric":"mean_score","protocol":{"en":"Epoch AI evaluations; mean score for the named tier and version. Model and reasoning configuration names follow the source.","zh":"Epoch AI 测评，使用对应难度层级和版本的平均成绩；模型及推理配置名称沿用原始记录。"},"source":{"name":"Epoch AI","url":"https://epoch.ai/frontiermath","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"status":"current_snapshot","published_at":"2024-11-07"},"results":[[369,0.9368421052631579,15,370,-1],[371,0.9368421052631579,15,370,-1],[372,0.9122807017543859,1,370,-1],[373,0.9087719298245615,15,370,-1],[374,0.9017543859649123,1,370,-1],[375,0.8982456140350877,15,370,-1],[376,0.8912280701754386,15,370,-1],[377,0.887719298245614,1,370,-1],[378,0.8771929824561403,15,370,-1],[379,0.8701754385964913,1,370,-1],[380,0.8596491228070176,4,370,-1],[381,0.8596491228070176,15,370,-1],[382,0.856140350877193,1,370,-1],[383,0.8526315789473684,15,370,-1],[384,0.8245614035087719,15,370,-1],[385,0.8210526315789474,15,370,-1],[386,0.8210526315789474,173,370,-1],[387,0.8,1,370,-1],[388,0.7894736842105263,15,370,-1],[389,0.7859649122807018,15,370,-1],[390,0.7508771929824561,1,370,-1],[391,0.7473684210526316,195,370,-1],[392,0.743859649122807,173,370,-1],[393,0.7403508771929824,173,370,-1],[394,0.74,15,370,-1],[395,0.721830985915493,396,370,-1],[397,0.7157894736842105,4,370,-1],[398,0.712280701754386,25,370,-1],[399,0.7017543859649122,1,370,-1],[400,0.6982456140350877,173,370,-1],[401,0.6877192982456141,402,370,-1],[403,0.6842105263157895,4,370,-1],[404,0.674,15,370,-1],[405,0.6736842105263158,25,370,-1],[406,0.6596491228070176,1,370,-1],[407,0.6596491228070176,143,370,-1],[408,0.656140350877193,1,370,-1],[409,0.656140350877193,195,370,-1],[410,0.6456140350877193,25,370,-1],[411,0.6456140350877193,195,370,-1],[412,0.6280701754385964,4,370,-1],[413,0.5964912280701754,4,370,-1],[414,0.592057761732852,402,370,-1],[415,0.5894736842105263,4,370,-1],[416,0.5859649122807018,417,370,-1],[418,0.5754385964912281,25,370,-1],[419,0.5719298245614035,143,370,-1],[420,0.5719298245614035,396,370,-1],[421,0.5578947368421052,402,370,-1],[422,0.5578947368421052,15,370,-1],[423,0.5543859649122806,15,370,-1],[424,0.5473684210526316,402,370,-1],[425,0.5403508771929825,396,370,-1],[426,0.5298245614035088,143,370,-1],[427,0.512280701754386,4,370,-1],[428,0.512280701754386,15,370,-1],[429,0.4666666666666667,15,370,-1],[430,0.4631578947368421,199,370,-1],[431,0.45263157894736844,25,370,-1],[432,0.44912280701754387,15,370,-1],[433,0.44912280701754387,143,370,-1],[434,0.4280701754385965,143,370,-1],[435,0.4245614035087719,402,370,-1],[436,0.41403508771929826,15,370,-1],[437,0.39649122807017545,15,370,-1],[438,0.3824561403508772,195,370,-1],[439,0.3719298245614035,15,370,-1],[440,0.3684210526315789,402,370,-1],[441,0.36140350877192984,15,370,-1],[442,0.3508771929824561,195,370,-1],[443,0.34385964912280703,1,370,-1],[444,0.34385964912280703,195,370,-1],[445,0.34035087719298246,195,370,-1],[446,0.3333333333333333,199,370,-1],[447,0.3333333333333333,15,370,-1],[448,0.32280701754385965,195,370,-1],[449,0.312280701754386,195,370,-1],[450,0.2982456140350877,15,370,-1],[451,0.29473684210526313,195,370,-1],[452,0.28771929824561404,15,370,-1],[453,0.2771929824561403,83,370,-1],[454,0.2631578947368421,15,370,-1],[455,0.2596491228070175,4,370,-1],[456,0.24912280701754386,402,370,-1],[457,0.24561403508771928,4,370,-1],[458,0.24561403508771928,15,370,-1],[459,0.23859649122807017,1,370,-1],[460,0.22456140350877193,83,370,-1],[461,0.22456140350877193,195,370,-1],[462,0.21403508771929824,83,370,-1],[463,0.20350877192982456,15,370,-1],[464,0.20350877192982456,195,370,-1],[465,0.2,15,370,-1],[466,0.19298245614035087,15,370,-1],[467,0.19298245614035087,195,370,-1],[468,0.19298245614035087,195,370,-1],[469,0.18947368421052632,195,370,-1],[470,0.18596491228070175,15,370,-1],[471,0.1824561403508772,15,370,-1],[472,0.1824561403508772,15,370,-1],[473,0.1824561403508772,195,370,-1],[474,0.17543859649122806,195,370,-1],[475,0.17192982456140352,15,370,-1],[476,0.17192982456140352,195,370,-1],[477,0.16140350877192983,15,370,-1],[478,0.14736842105263157,15,370,-1],[479,0.12631578947368421,1,370,-1],[480,0.10526315789473684,15,370,-1],[481,0.10175438596491228,15,370,-1],[482,0.09473684210526316,195,370,-1],[483,0.08421052631578947,15,370,-1],[484,0.06666666666666667,15,370,-1],[485,0.05964912280701754,15,370,-1],[486,0.05964912280701754,15,370,-1],[487,0.05964912280701754,15,370,-1],[488,0.0456140350877193,15,370,-1],[489,0.03859649122807018,15,370,-1],[490,0.017543859649122806,15,370,-1],[491,0.007017543859649123,15,370,-1],[492,0.007017543859649123,15,370,-1],[493,0.0035087719298245615,15,370,-1],[494,0.0,15,370,-1]]},{"id":"frontiermath-tier-4-v2","name":"FrontierMath Tier 4 (v2)","group":"数学与科学","unit":"%","metric":"mean_score","protocol":{"en":"Epoch AI evaluations; mean score for the named tier and version. Model and reasoning configuration names follow the source.","zh":"Epoch AI 测评，使用对应难度层级和版本的平均成绩；模型及推理配置名称沿用原始记录。"},"source":{"name":"Epoch AI","url":"https://epoch.ai/frontiermath","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"status":"current_snapshot","published_at":"2024-11-07"},"results":[[371,1.0,15,370,-1],[495,0.976,15,370,-1],[369,0.976,15,370,-1],[496,0.976,15,370,-1],[497,0.975609756097561,15,370,-1],[372,0.95,1,370,-1],[373,0.926829268292683,15,370,-1],[379,0.902,1,370,-1],[375,0.9,15,370,-1],[374,0.8780487804878049,1,370,-1],[498,0.8780487804878049,15,370,-1],[376,0.8292682926829268,15,370,-1],[499,0.8292682926829268,15,370,-1],[377,0.8048780487804879,1,370,-1],[500,0.8048780487804879,15,370,-1],[380,0.7804878048780488,4,370,-1],[378,0.7804878048780488,15,370,-1],[501,0.756,4,370,-1],[382,0.7317073170731707,1,370,-1],[383,0.725,15,370,-1],[381,0.7073170731707317,15,370,-1],[386,0.7073170731707317,173,370,-1],[385,0.6097560975609756,15,370,-1],[384,0.5853658536585366,15,370,-1],[387,0.5609756097560976,1,370,-1],[388,0.5609756097560976,15,370,-1],[389,0.49,15,370,-1],[390,0.4634146341463415,1,370,-1],[398,0.4634146341463415,25,370,-1],[393,0.4634146341463415,173,370,-1],[391,0.4634146341463415,195,370,-1],[394,0.46,15,370,-1],[392,0.4146341463414634,173,370,-1],[395,0.3902439024390244,396,370,-1],[397,0.36585365853658536,4,370,-1],[411,0.34146341463414637,195,370,-1],[409,0.34146341463414637,195,370,-1],[400,0.3170731707317073,173,370,-1],[399,0.3170731707317073,1,370,-1],[407,0.3170731707317073,143,370,-1],[404,0.317,15,370,-1],[408,0.2926829268292683,1,370,-1],[414,0.2926829268292683,402,370,-1],[401,0.2926829268292683,402,370,-1],[406,0.2682926829268293,1,370,-1],[410,0.2682926829268293,25,370,-1],[405,0.2682926829268293,25,370,-1],[413,0.2682926829268293,4,370,-1],[412,0.2682926829268293,4,370,-1],[416,0.2682926829268293,417,370,-1],[420,0.2564102564102564,396,370,-1],[418,0.24390243902439024,25,370,-1],[419,0.24390243902439024,143,370,-1],[415,0.21951219512195122,4,370,-1],[403,0.21951219512195122,4,370,-1],[423,0.21951219512195122,15,370,-1],[422,0.1951219512195122,15,370,-1],[427,0.17073170731707318,4,370,-1],[421,0.17073170731707318,402,370,-1],[433,0.17073170731707318,143,370,-1],[426,0.17073170731707318,143,370,-1],[430,0.17073170731707318,199,370,-1],[434,0.14634146341463414,143,370,-1],[429,0.12195121951219512,15,370,-1],[432,0.12195121951219512,15,370,-1],[425,0.12195121951219512,396,370,-1],[428,0.0975609756097561,15,370,-1],[446,0.04878048780487805,199,370,-1],[443,0.04878048780487805,1,370,-1],[441,0.04878048780487805,15,370,-1],[479,0.024390243902439025,1,370,-1],[459,0.024390243902439025,1,370,-1],[431,0.024390243902439025,25,370,-1],[465,0.024390243902439025,15,370,-1],[454,0.024390243902439025,15,370,-1],[457,0.0,4,370,-1],[455,0.0,4,370,-1],[470,0.0,15,370,-1]]},{"id":"mmlu-pro","name":"MMLU-Pro","group":"知识与推理","unit":"%","metric":"Overall accuracy","protocol":{"en":"Overall accuracy from rows attributed to TIGER-Lab in the official leaderboard; self-reported submissions are excluded.","zh":"官方榜单中标注由 TIGER-Lab 测评的 Overall 准确率；不含自行提交的成绩。"},"source":{"name":"TIGER-Lab","url":"https://huggingface.co/spaces/TIGER-Lab/MMLU-Pro","status":"current_snapshot","published_at":"2024-06-03"},"results":[[502,0.9116,83,370,-1],[503,0.8452,83,370,-1],[504,0.7764,1,370,-1],[505,0.7624,83,370,-1],[506,0.7612,1,370,-1],[507,0.7468,15,370,-1],[508,0.7314,15,370,-1],[509,0.7311,370,370,-1],[510,0.7255,15,370,-1],[511,0.7025,83,370,-1],[512,0.7021,370,370,-1],[513,0.6907,370,370,-1],[514,0.6845,1,370,-1],[515,0.6794,417,370,-1],[516,0.6592,66,370,-1],[517,0.6591,417,370,-1],[518,0.6583,25,370,-1],[519,0.6438,195,370,-1],[520,0.6409,83,370,-1],[521,0.6393,370,370,-1],[522,0.6371,15,370,-1],[523,0.6363,25,370,-1],[354,0.6309,15,370,-1],[524,0.6284,66,370,-1],[525,0.6278,66,370,-1],[526,0.6238,527,370,-1],[528,0.6212,1,370,-1],[529,0.6035,370,370,-1],[530,0.5912,83,370,-1],[531,0.5891,370,370,-1],[532,0.5809,527,370,-1],[533,0.568,1,370,-1],[534,0.5654,83,370,-1],[535,0.5633,417,370,-1],[536,0.562,66,370,-1],[537,0.557,370,370,-1],[538,0.5559,195,370,-1],[539,0.5481,25,370,-1],[540,0.5278,66,370,-1],[541,0.5264,195,370,-1],[542,0.5247,66,370,-1],[543,0.5229,527,370,-1],[544,0.5208,83,370,-1],[545,0.5191,370,370,-1],[546,0.504,370,370,-1],[547,0.4993,195,370,-1],[548,0.4946,370,370,-1],[549,0.484,417,370,-1],[550,0.4801,551,370,-1],[552,0.4792,551,370,-1],[553,0.4787,554,370,-1],[555,0.4724,195,370,-1],[556,0.4624,370,370,-1],[557,0.4595,527,370,-1],[558,0.4566,370,370,-1],[559,0.4541,370,370,-1],[560,0.451,83,370,-1],[561,0.4481,417,370,-1],[562,0.4425,66,370,-1],[563,0.4386,370,370,-1],[564,0.4335,370,370,-1],[565,0.4327,417,370,-1],[566,0.4303,527,370,-1],[567,0.4229,1,370,-1],[568,0.42,370,370,-1],[569,0.4157,25,370,-1],[570,0.4103,370,370,-1],[571,0.4103,417,370,-1],[572,0.4098,66,370,-1],[573,0.4085,370,370,-1],[574,0.4073,195,370,-1],[575,0.3977,417,370,-1],[576,0.3924,370,370,-1],[577,0.391,370,370,-1],[578,0.3823,527,370,-1],[579,0.3802,195,370,-1],[580,0.3793,370,370,-1],[581,0.379,370,370,-1],[582,0.379,370,370,-1],[583,0.3753,66,370,-1],[584,0.3724,370,370,-1],[585,0.371,370,370,-1],[586,0.3693,66,370,-1],[587,0.366,66,370,-1],[588,0.3536,66,370,-1],[589,0.353,25,370,-1],[590,0.3437,25,370,-1],[591,0.3374,370,370,-1],[592,0.3373,83,370,-1],[593,0.335,370,370,-1],[594,0.3308,370,370,-1],[595,0.3297,370,370,-1],[596,0.3197,370,370,-1],[597,0.3103,370,370,-1],[598,0.3088,417,370,-1],[599,0.3084,417,370,-1],[600,0.3043,417,370,-1],[601,0.2906,195,370,-1],[602,0.2884,527,370,-1],[603,0.2874,370,370,-1],[604,0.2651,527,370,-1],[605,0.2585,370,370,-1],[606,0.2575,417,370,-1],[607,0.2542,370,370,-1],[608,0.2534,66,370,-1],[609,0.2389,370,370,-1],[610,0.2345,370,370,-1],[611,0.2262,195,370,-1],[612,0.2256,195,370,-1],[613,0.2217,66,370,-1],[614,0.2172,370,370,-1],[615,0.2039,370,370,-1],[616,0.2032,66,370,-1],[617,0.1831,370,370,-1],[618,0.1593,195,370,-1],[619,0.1585,83,370,-1],[620,0.1497,195,370,-1],[621,0.1327,370,370,-1],[622,0.1234,370,370,-1],[623,0.1195,66,370,-1],[624,0.1193,370,370,-1],[625,0.1138,370,370,-1],[626,0.1122,370,370,-1],[627,0.1095,370,370,-1],[628,0.1085,370,370,-1]]},{"id":"bfcl-v4","name":"BFCL v4","group":"Agent 与工具","unit":"%","metric":"Overall accuracy","protocol":{"en":"BFCL v4 overall accuracy across subcategories. FC and Prompt labels retain the reported calling mode.","zh":"BFCL v4 各子类汇总准确率；FC 和 Prompt 标签保留原始工具调用方式。"},"source":{"name":"Berkeley / Gorilla","url":"https://gorilla.cs.berkeley.edu/leaderboard.html","status":"current_snapshot","published_at":"2024-09-19"},"results":[[629,0.7746999999999999,1,370,-1],[630,0.7323999999999999,1,370,-1],[631,0.7251000000000001,83,370,-1],[632,0.7238,633,370,-1],[634,0.6957,143,370,-1],[635,0.687,1,370,-1],[636,0.6814,83,370,-1],[637,0.6305,15,370,-1],[638,0.6297,143,370,-1],[639,0.6138,143,370,-1],[640,0.5906,641,370,-1],[642,0.5829,143,370,-1],[643,0.5706,644,370,-1],[645,0.5672999999999999,25,370,-1],[646,0.5624,83,370,-1],[647,0.5587,15,370,-1],[648,0.5546,15,370,-1],[649,0.5466,650,370,-1],[651,0.5412,25,370,-1],[652,0.5396,15,370,-1],[653,0.5324,15,370,-1],[654,0.5307,650,370,-1],[655,0.5215,50,370,-1],[656,0.5145000000000001,15,370,-1],[657,0.514,658,370,-1],[659,0.509,83,370,-1],[660,0.5045000000000001,15,370,-1],[661,0.5025999999999999,15,370,-1],[662,0.48710000000000003,50,370,-1],[663,0.48560000000000003,15,370,-1],[664,0.4799,50,370,-1],[665,0.4768,658,370,-1],[666,0.4678,50,370,-1],[667,0.4668,650,370,-1],[668,0.46490000000000004,644,370,-1],[669,0.4623,670,370,-1],[671,0.4537,672,370,-1],[673,0.45270000000000005,15,370,-1],[674,0.4257,50,370,-1],[675,0.4244,676,370,-1],[677,0.4139,50,370,-1],[678,0.4122,650,370,-1],[679,0.4103,50,370,-1],[680,0.4043,50,370,-1],[681,0.39380000000000004,15,370,-1],[682,0.3837,417,370,-1],[683,0.37770000000000004,50,370,-1],[684,0.37689999999999996,417,370,-1],[685,0.37560000000000004,417,370,-1],[686,0.3729,66,370,-1],[687,0.3715,417,370,-1],[688,0.36869999999999997,83,370,-1],[689,0.36700000000000005,50,370,-1],[690,0.3568,50,370,-1],[691,0.3552,50,370,-1],[692,0.35359999999999997,672,370,-1],[693,0.3347,1,370,-1],[694,0.33049999999999996,15,370,-1],[695,0.32380000000000003,417,370,-1],[696,0.3214,672,370,-1],[697,0.3207,644,370,-1],[698,0.319,66,370,-1],[699,0.3184,417,370,-1],[700,0.31670000000000004,701,370,-1],[702,0.3044,650,370,-1],[703,0.3043,83,370,-1],[704,0.2973,15,370,-1],[705,0.29710000000000003,701,370,-1],[706,0.29469999999999996,83,370,-1],[707,0.2879,554,370,-1],[708,0.2841,50,370,-1],[709,0.2813,66,370,-1],[710,0.2803,83,370,-1],[711,0.2799,712,370,-1],[713,0.2788,701,370,-1],[714,0.2787,715,370,-1],[716,0.2783,15,370,-1],[717,0.2763,417,370,-1],[718,0.2755,15,370,-1],[719,0.271,720,370,-1],[721,0.271,722,370,-1],[723,0.2701,724,370,-1],[725,0.2687,722,370,-1],[726,0.2681,712,370,-1],[727,0.2583,66,370,-1],[728,0.2555,729,370,-1],[730,0.2526,1,370,-1],[731,0.24969999999999998,720,370,-1],[732,0.249,1,370,-1],[733,0.2488,15,370,-1],[734,0.2403,724,370,-1],[735,0.23929999999999998,50,370,-1],[736,0.2323,722,370,-1],[737,0.2238,50,370,-1],[738,0.2229,720,370,-1],[739,0.2225,740,370,-1],[741,0.2208,729,370,-1],[742,0.2195,66,370,-1],[743,0.21899999999999997,744,370,-1],[745,0.2122,701,370,-1],[746,0.1962,83,370,-1],[747,0.1931,417,370,-1],[748,0.1898,722,370,-1],[749,0.1625,724,370,-1],[750,0.111,417,370,-1],[751,0.1108,724,370,-1],[752,0.1082,66,370,-1],[753,0.1,307,370,-1],[754,0.0717,83,370,-1]]},{"id":"osworld-2-v2-1-full","name":"OSWorld 2.0 (v2.1, full)","group":"Agent 与工具","unit":"%","metric":"Binary task completion","protocol":{"en":"Official end-to-end model runs, 500-step budget. Score is full task completion, not partial reward; reasoning settings are retained.","zh":"官方端到端模型运行，最多 500 步。使用完整任务完成率，不使用部分奖励；保留推理设置。"},"source":{"name":"OSWorld / XLANG Lab","url":"https://osworld-v2.xlang.ai/","status":"current_snapshot","published_at":"2026-09-15"},"results":[[755,0.44329999999999997,1,756,-1],[757,0.3925,66,756,-1],[755,0.3689,1,758,-1],[755,0.3333,1,759,-1],[755,0.3301,1,760,-1],[755,0.1881,1,761,-1]]},{"id":"osworld-2-20260808-offline","name":"OSWorld 2.0 (2026.08.08, offline)","group":"Agent 与工具","unit":"%","metric":"Binary task completion","protocol":{"en":"Official end-to-end model runs, 500-step budget. Score is full task completion, not partial reward; reasoning settings are retained.","zh":"官方端到端模型运行，最多 500 步。使用完整任务完成率，不使用部分奖励；保留推理设置。"},"source":{"name":"OSWorld / XLANG Lab","url":"https://osworld-v2.xlang.ai/","status":"current_snapshot","published_at":"2026-08-08"},"results":[[755,0.3472,1,756,-1],[755,0.33380000000000004,1,759,-1],[755,0.32689999999999997,1,758,-1],[762,0.281,15,756,-1],[755,0.2713,1,760,-1],[762,0.2463,15,759,-1],[755,0.24609999999999999,1,761,-1],[762,0.2,15,758,-1],[762,0.17559999999999998,15,760,-1],[762,0.0732,15,761,-1],[762,0.0341,15,763,-1]]},{"id":"longbench-v2-cot","name":"LongBench v2 (CoT)","group":"指令与长文","unit":"%","metric":"Overall accuracy with CoT","protocol":{"en":"Overall accuracy with chain-of-thought reasoning; missing CoT scores are omitted. Context limits follow each published entry.","zh":"使用思维链设置的总体准确率，缺少该设置成绩的模型不计入；上下文上限沿用各条官方记录。"},"source":{"name":"LongBench / THUDM","url":"https://longbench2.github.io/","status":"current_snapshot","published_at":"2024-12-19"},"results":[[764,0.633,83,765,-1],[766,0.621,83,765,-1],[767,0.606,195,768,-1],[769,0.583,25,770,-1],[771,0.583,195,768,-1],[772,0.5770000000000001,15,770,-1],[773,0.5670000000000001,25,770,-1],[774,0.565,775,776,-1],[777,0.56,83,765,-1],[778,0.525,83,779,-1],[780,0.514,15,770,-1],[507,0.512,15,770,-1],[781,0.511,83,765,-1],[782,0.503,783,770,-1],[784,0.501,195,770,-1],[785,0.501,195,768,-1],[786,0.49200000000000005,195,770,-1],[513,0.489,195,770,-1],[787,0.486,783,770,-1],[788,0.467,1,789,-1],[790,0.461,783,770,-1],[791,0.44299999999999995,21,770,-1],[792,0.435,195,770,-1],[793,0.425,195,770,-1],[794,0.396,417,770,-1],[341,0.389,15,770,-1],[795,0.36200000000000004,66,770,-1],[796,0.36200000000000004,66,770,-1],[797,0.35600000000000004,195,770,-1],[798,0.35200000000000004,799,770,-1],[800,0.336,417,770,-1],[801,0.324,15,770,-1],[802,0.32,803,804,-1],[805,0.316,644,770,-1],[552,0.308,783,770,-1],[806,0.304,66,770,-1]]},{"id":"video-mme-no-subs","name":"Video-MME (no subtitles)","group":"视觉与视频","unit":"%","metric":"Overall accuracy without subtitles","protocol":{"en":"Overall accuracy without subtitles. Frame sampling settings are preserved for each model.","zh":"不使用字幕的总体准确率；保留各模型的采样帧设置。"},"source":{"name":"Video-MME","url":"https://video-mme.github.io/home_page.html#leaderboard","status":"current_snapshot","published_at":"2024-05-31"},"results":[[807,0.797,808,809,-1],[343,0.75,83,810,-1],[811,0.735,812,813,-1],[814,0.73,815,816,-1],[817,0.73,818,819,-1],[820,0.721,821,816,-1],[353,0.7190000000000001,15,822,-1],[823,0.7120000000000001,195,824,-1],[825,0.706,826,827,-1],[828,0.706,829,816,-1],[830,0.703,83,810,-1],[831,0.703,832,833,-1],[834,0.6759999999999999,835,836,-1],[837,0.675,838,839,-1],[840,0.6729999999999999,841,819,-1],[842,0.67,843,839,-1],[844,0.6629999999999999,829,845,-1],[846,0.662,195,847,-1],[848,0.659,849,816,-1],[850,0.6559999999999999,851,852,-1],[853,0.653,854,855,-1],[801,0.648,15,856,-1],[857,0.6459999999999999,858,859,-1],[860,0.642,307,813,-1],[861,0.6409999999999999,862,863,-1],[864,0.639,865,816,-1],[866,0.639,812,813,-1],[867,0.624,195,845,-1],[868,0.612,821,869,-1],[870,0.609,865,816,-1],[788,0.6,1,871,-1],[872,0.599,15,873,-1],[874,0.593,195,819,-1],[875,0.573,876,819,-1],[877,0.561,878,869,-1],[879,0.56,880,816,-1],[881,0.5579999999999999,878,845,-1],[882,0.555,883,819,-1],[884,0.532,885,852,-1],[886,0.529,720,816,-1],[887,0.526,888,819,-1],[889,0.513,195,890,-1],[891,0.507,821,873,-1],[892,0.45299999999999996,893,894,-1],[895,0.43200000000000005,849,816,-1],[896,0.41100000000000003,195,890,-1],[897,0.406,898,816,-1],[899,0.39899999999999997,821,869,-1],[900,0.39899999999999997,898,894,-1],[901,0.395,821,869,-1],[902,0.379,898,816,-1]]},{"id":"gaia-2023-test","name":"GAIA (2023 test)","group":"Agent 与工具","unit":"%","metric":"Task success","protocol":{"en":"301 test tasks. Named submissions from the official top 100 plus identifiable submissions with external project links in the full export. Teams report their agent and base models.","zh":"301 道测试题。收录官方展示的前 100 条中身份完整的提交，以及全量文件中带外部项目链接的具名提交。Agent 和基础模型由提交方填写。"},"source":{"name":"GAIA authors","url":"https://huggingface.co/spaces/gaia-benchmark/leaderboard","status":"current_snapshot","published_at":"2023-11-21"},"results":[[903,0.9401993355481728,904,905,-1],[906,0.9401993355481728,907,908,-1],[909,0.9335548172757475,910,911,912],[913,0.9335548172757475,914,915,-1],[916,0.9302325581395349,917,918,919],[920,0.9302325581395349,904,921,-1],[922,0.9302325581395349,907,908,923],[924,0.9302325581395349,907,908,-1],[925,0.9302325581395349,907,926,-1],[927,0.9302325581395349,907,928,-1],[929,0.9302325581395349,907,928,-1],[930,0.9302325581395349,907,908,-1],[931,0.9269102990033222,907,926,-1],[932,0.9269102990033222,907,908,-1],[933,0.9269102990033222,907,934,-1],[935,0.9235880398671097,936,937,938],[939,0.9235880398671097,940,926,-1],[941,0.9235880398671097,940,928,-1],[942,0.9235880398671097,907,943,923],[944,0.920265780730897,945,946,-1],[947,0.920265780730897,917,948,919],[949,0.920265780730897,950,951,-1],[952,0.920265780730897,910,953,912],[954,0.920265780730897,910,955,912],[956,0.920265780730897,910,946,957],[958,0.920265780730897,914,915,-1],[959,0.920265780730897,940,926,-1],[960,0.920265780730897,961,962,923],[963,0.9169435215946844,950,964,-1],[965,0.9169435215946844,950,964,-1],[966,0.9169435215946844,950,964,-1],[967,0.9169435215946844,950,964,-1],[968,0.9169435215946844,950,964,-1],[969,0.9169435215946844,950,964,-1],[970,0.9169435215946844,910,946,912],[971,0.9169435215946844,907,972,-1],[973,0.9169435215946844,907,926,-1],[974,0.9169435215946844,907,928,-1],[975,0.9169435215946844,907,962,923],[976,0.9169435215946844,907,977,-1],[978,0.9169435215946844,979,980,981],[982,0.9136212624584718,950,964,-1],[983,0.9136212624584718,950,964,-1],[984,0.9136212624584718,950,964,-1],[985,0.9136212624584718,950,964,-1],[986,0.9136212624584718,910,946,912],[987,0.9136212624584718,910,946,912],[988,0.9136212624584718,989,990,991],[992,0.9136212624584718,907,993,-1],[994,0.9102990033222591,910,946,912],[995,0.9102990033222591,996,997,-1],[998,0.9102990033222591,999,962,-1],[1000,0.9102990033222591,907,908,-1],[1001,0.9102990033222591,961,943,1002],[1003,0.9102990033222591,907,962,-1],[1004,0.9102990033222591,1005,1006,-1],[1007,0.9069767441860465,910,1008,912],[1009,0.9069767441860465,910,946,912],[1010,0.9036544850498339,307,1011,1012],[1013,0.9036544850498339,307,1011,1012],[1014,0.9003322259136213,910,946,1015],[1016,0.8970099667774086,910,946,1015],[1017,0.893687707641196,910,946,1015],[1018,0.8903654485049833,910,946,1015],[1019,0.8837209302325582,910,946,1015],[1020,0.8837209302325582,989,1021,991],[1022,0.8803986710963455,910,946,1015],[1023,0.8737541528239202,307,1011,1012],[1024,0.8704318936877077,917,1025,919],[1026,0.8571428571428571,1027,1028,1029],[1030,0.8571428571428571,1031,1032,1033],[1027,0.8504983388704319,1027,1028,1029],[1034,0.8504983388704319,1034,1035,1036],[1037,0.8471760797342193,1031,1032,1033],[1038,0.8471760797342193,1039,1040,1041],[1042,0.8438538205980066,917,1025,919],[1043,0.8438538205980066,1044,946,1015],[1045,0.840531561461794,917,1046,919],[1047,0.8372093023255814,1048,1049,1050],[1051,0.8338870431893688,910,946,1015],[1052,0.8338870431893688,1053,1054,1055],[1056,0.8305647840531561,1044,946,1015],[1057,0.8205980066445183,1031,1032,1033],[1058,0.8172757475083057,1059,1060,1061],[1062,0.7973421926910299,1039,1040,1041],[1063,0.7973421926910299,1048,1064,1065],[1066,0.7906976744186046,1067,946,1015],[1068,0.7906976744186046,1069,1070,1071],[1072,0.7906976744186046,1073,1074,1075],[1076,0.7906976744186046,1048,1077,1065],[1078,0.7873754152823921,1079,1080,1081],[1082,0.7873754152823921,1083,1070,1071],[1084,0.770764119601329,1059,1060,1061],[1085,0.7674418604651163,1031,1032,1033],[1086,0.7574750830564784,1087,1088,1089],[1090,0.7475083056478405,1069,1070,1091],[1092,0.7475083056478405,1048,1093,1050],[1094,0.7441860465116279,1069,1095,1071],[1096,0.7441860465116279,1097,1098,1099],[1100,0.7408637873754153,1101,1102,1103],[1104,0.7408637873754153,910,1105,1106],[1107,0.7375415282392026,910,1105,1106],[1108,0.7342192691029901,1109,1110,1111],[1112,0.7242524916943521,1109,946,1111],[1113,0.7242524916943521,1059,1114,1061],[1115,0.717607973421927,1116,1117,1118],[1119,0.717607973421927,910,1105,1106],[1120,0.7142857142857143,1109,946,1111],[1121,0.7109634551495017,1122,1123,1124],[1125,0.707641196013289,1069,1070,1091],[1126,0.7043189368770764,1109,946,1111],[1096,0.7009966777408638,1096,1088,1127],[1128,0.6976744186046512,1087,1129,1130],[1131,0.6943521594684385,1132,1133,1134],[1135,0.6877076411960132,1109,946,1111],[1136,0.6810631229235881,1137,1138,1139],[1140,0.6810631229235881,1141,1142,1143],[1144,0.6710963455149501,1109,946,1111],[1145,0.6710963455149501,1146,1147,1148],[1149,0.6710963455149501,1150,1151,1152],[1153,0.6710963455149501,1141,1154,1155],[1156,0.6677740863787376,1109,946,1111],[1157,0.6677740863787376,1158,1159,1160],[1161,0.6644518272425249,1109,946,1111],[1162,0.6611295681063123,1163,1164,1165],[1166,0.6578073089700996,1150,1167,1168],[1169,0.6578073089700996,1170,1123,1171],[1172,0.6511627906976745,1173,1174,1175],[1176,0.6511627906976745,1146,1147,1148],[1177,0.6511627906976745,1048,1178,1050],[1179,0.6478405315614618,1180,1181,1182],[1183,0.6478405315614618,1132,1133,1184],[1153,0.6411960132890365,1153,1185,1186],[1187,0.6312292358803987,1188,1114,1061],[1189,0.6212624584717608,1173,1174,1175],[1190,0.6212624584717608,1132,1191,1134],[1192,0.5980066445182725,1132,1191,1134],[1193,0.5913621262458472,1109,1194,1111],[1195,0.5880398671096345,1196,1197,1198],[1199,0.5813953488372093,1132,1200,1134],[1201,0.5614617940199336,1109,1194,1111],[1202,0.5548172757475083,1203,1204,1205],[1206,0.5548172757475083,1207,1208,1209],[1210,0.5548172757475083,1207,1208,1209],[1211,0.5481727574750831,1203,1204,1205],[1212,0.5415282392026578,1109,1213,1111],[1214,0.5415282392026578,1196,1215,1216],[1217,0.5415282392026578,1207,1208,1209],[1218,0.5249169435215947,1203,1204,1205],[1219,0.5182724252491694,1132,1220,1134],[1221,0.5149501661129569,1222,1223,1224],[1225,0.5116279069767442,1226,1227,1228],[1229,0.5116279069767442,1150,1230,1168],[1231,0.5049833887043189,1188,1232,1061],[1233,0.4983388704318937,1234,1235,1236],[1237,0.4883720930232558,1196,1197,1238],[1239,0.4883720930232558,1048,1240,1241],[1242,0.48172757475083056,1243,1244,1245],[1246,0.4717607973421927,1246,1247,1248],[1249,0.4717607973421927,1250,1251,1252],[1253,0.4684385382059801,1207,1213,1209],[1254,0.4485049833887043,1255,1256,1257],[1258,0.42524916943521596,1259,1260,1261],[1262,0.4119601328903654,1263,1264,1265],[1266,0.3953488372093023,1158,1267,1268],[1269,0.38205980066445183,1270,1271,1272],[1273,0.3787375415282392,1259,1274,1275],[1276,0.3754152823920266,1277,1278,1279],[1280,0.3554817275747508,1158,1281,1282],[1283,0.34551495016611294,1284,1285,1286],[1287,0.33554817275747506,1288,946,1289],[1290,0.33222591362126247,1291,1292,1293],[1294,0.33222591362126247,1295,1285,1296],[1297,0.3222591362126246,1259,1285,1275],[1298,0.3222591362126246,1259,1299,1300],[1301,0.2956810631229236,1291,1292,1293],[1302,0.2956810631229236,1291,1292,1228],[1303,0.27906976744186046,1291,1292,1293],[1304,0.26245847176079734,1291,1305,1228],[1306,0.2558139534883721,1307,1308,1309],[1310,0.25249169435215946,1311,1312,1313],[1314,0.24916943521594684,1315,1316,1317],[1318,0.2425249169435216,1319,1299,1320],[1321,0.23588039867109634,1322,1323,1324],[1325,0.23588039867109634,1207,1326,1209],[1327,0.2292358803986711,1328,1329,1330],[1331,0.2159468438538206,1319,1299,1320],[1332,0.19933554817275748,1291,1292,1293],[1333,0.16943521594684385,1334,1335,1336],[1337,0.16943521594684385,1338,1339,1340],[1341,0.15946843853820597,1311,1342,1313],[1343,0.15614617940199335,1344,1345,1346],[1347,0.14285714285714285,1348,1349,1350],[1351,0.13953488372093023,1352,1353,1354],[1355,0.12956810631229235,1291,1356,1228],[40,0.12956810631229235,1357,1358,1359],[1360,0.12292358803986711,1361,1362,1363],[1364,0.11627906976744186,1365,1366,1367],[1368,0.11627906976744186,1207,1369,1370],[1371,0.08305647840531562,1372,1373,1374],[1375,0.07973421926910298,1338,1376,1377],[1378,0.07973421926910298,1338,1376,1379],[1380,0.0664451827242525,1381,1382,1383],[1384,0.0664451827242525,1338,1385,1386],[1387,0.053156146179401995,1255,1256,1257],[1388,0.04983388704318937,1389,1390,1391],[1392,0.046511627906976744,1393,1394,1395],[1396,0.03986710963455149,1381,1382,1397],[1398,0.036544850498338874,1196,1197,1399],[1400,0.036544850498338874,1401,1402,1403],[1404,0.026578073089700997,1381,1405,1383],[1406,0.016611295681063124,1207,1285,1407],[1408,0.006644518272425249,1067,1409,1015],[1410,0.006644518272425249,1031,1411,1412],[1413,0.006644518272425249,1414,946,1015],[1415,0.0033222591362126247,1067,946,1015],[1416,0.0033222591362126247,1067,946,1015],[1417,0.0033222591362126247,1067,946,1015],[1418,0.0033222591362126247,1067,946,1015],[1419,0.0033222591362126247,1196,1215,1216],[1420,0.0033222591362126247,1421,1422,1423]]},{"id":"swe-bench-pro-v2-full","name":"SWE-bench Pro V2 (Full)","group":"编程","unit":"%","metric":"Resolved","protocol":{"en":"Scale V2 Full, 642 tasks. Model and agent settings follow the published entries.","zh":"Scale V2 Full，642 道任务。模型与 Agent 配置沿用发布名称。"},"source":{"name":"Scale AI","url":"https://labs.scale.com/leaderboard/swe_bench_pro_public_v2","status":"current_snapshot","published_at":"2026-09-22"},"results":[[1424,0.9940000000000001,1425,1426,-1],[1427,0.9922,1428,1426,-1],[1429,0.991,1425,1426,-1],[1430,0.977,1431,1426,-1],[1432,0.9690000000000001,1428,1426,-1],[1433,0.9673,1428,1426,-1],[1434,0.9611,1428,1426,-1],[1435,0.9579000000000001,1425,1426,-1],[1436,0.956,1437,1426,-1],[1438,0.955,1428,1426,-1],[1439,0.9501999999999999,1428,1426,-1],[1440,0.9486,1441,1426,-1],[1442,0.9315000000000001,1425,1426,-1],[1443,0.9237000000000001,1428,1426,-1],[1444,0.8987999999999999,1445,1426,-1],[1446,0.8723000000000001,1425,1426,-1]]},{"id":"aider-polyglot","name":"Aider Polyglot","group":"编程","unit":"%","metric":"Correct after repair","protocol":{"en":"225 exercises across six languages. Percent correct includes a second repair attempt. Edit format and reasoning effort are retained.","zh":"6 种语言、225 道任务；答对率包含第二次修复机会。保留编辑格式与推理强度。"},"source":{"name":"Aider","url":"https://aider.chat/docs/leaderboards/","status":"current_snapshot","published_at":"2024-12-21"},"results":[[1447,0.88,15,1448,-1],[1449,0.867,15,1450,-1],[1451,0.8490000000000001,15,1452,-1],[1453,0.831,83,1454,-1],[1455,0.813,15,1456,-1],[1457,0.813,370,1458,-1],[1459,0.7959999999999999,143,1460,-1],[1461,0.7909999999999999,83,1454,-1],[1462,0.769,83,1463,-1],[1464,0.769,15,1465,-1],[1466,0.742,25,1467,-1],[1468,0.7290000000000001,83,1469,-1],[1470,0.72,1,1471,-1],[1472,0.72,15,1473,-1],[1474,0.7070000000000001,1,1471,-1],[1475,0.7020000000000001,25,1467,-1],[1476,0.649,1,1477,-1],[1478,0.64,25,1479,-1],[1480,0.613,1,1471,-1],[1481,0.604,1,1482,-1],[1483,0.596,195,1484,-1],[1485,0.591,21,1486,-1],[1487,0.569,25,1488,-1],[1489,0.564,1,1471,-1],[1490,0.551,25,1491,-1],[1492,0.551,83,1471,-1],[1493,0.547,370,1494,-1],[1495,0.5379999999999999,15,1496,-1],[1497,0.5329999999999999,143,1498,-1],[1499,0.529,370,1498,-1],[1500,0.524,15,1501,-1],[1502,0.516,1,1488,-1],[1503,0.493,143,1504,-1],[1505,0.484,25,1506,-1],[1507,0.47100000000000003,83,1508,-1],[1509,0.45299999999999996,15,1510,-1],[1511,0.44,83,1471,-1],[1512,0.418,15,1486,-1],[1513,0.4,195,1484,-1],[1514,0.382,83,1515,-1],[1516,0.35600000000000004,83,1517,-1],[1518,0.34700000000000003,143,1519,-1],[1520,0.32899999999999996,15,1515,-1],[1521,0.324,15,1501,-1],[1522,0.28,1,1506,-1],[1523,0.262,370,1524,-1],[493,0.231,15,1525,-1],[1526,0.222,83,1515,-1],[1527,0.218,195,1496,-1],[513,0.209,370,1528,-1],[1529,0.182,83,1530,-1],[1531,0.182,15,1525,-1],[1532,0.17800000000000002,25,1506,-1],[1533,0.16399999999999998,195,1515,-1],[362,0.156,66,1519,-1],[1534,0.129,527,1515,-1],[1535,0.12,370,1536,-1],[1537,0.111,370,1538,-1],[1539,0.102,370,1540,-1],[1541,0.08900000000000001,15,1542,-1],[1533,0.08,195,1506,-1],[1543,0.049,83,1536,-1],[492,0.036000000000000004,15,1515,-1]]},{"id":"bigcodebench-hard-instruct","name":"BigCodeBench Hard (Instruct)","group":"编程","unit":"%","metric":"Calibrated pass@1","protocol":{"en":"Hard-Instruct, calibrated pass@1 with greedy decoding. Each run retains its prompt and prefilling settings.","zh":"Hard-Instruct，采用贪心解码的校准 pass@1。各条结果保留提示与预填设置。"},"source":{"name":"BigCodeBench","url":"https://bigcode-bench.github.io/","status":"current_snapshot","published_at":"2024-06-18"},"results":[[1544,0.331,15,1545,1546],[1547,0.324,1,1545,1548],[1549,0.324,15,1545,1550],[1551,0.324,15,1545,1546],[1552,0.318,1,1545,1548],[1553,0.318,15,1545,1350],[1554,0.318,15,1545,1350],[1555,0.318,370,1545,1556],[1557,0.311,143,1545,1558],[1559,0.311,15,1545,1546],[1560,0.304,370,1545,1561],[1562,0.297,370,1563,1564],[769,0.297,25,1545,1565],[503,0.297,83,1545,1566],[1567,0.297,370,1545,1568],[1569,0.297,15,1545,1550],[1570,0.29100000000000004,15,1563,1383],[1571,0.29100000000000004,195,1545,1572],[1573,0.284,25,1545,1574],[1575,0.284,15,1545,1350],[516,0.284,66,1563,1576],[1577,0.284,15,1545,1550],[1578,0.27699999999999997,25,1545,1579],[1580,0.27699999999999997,15,1545,1581],[778,0.27699999999999997,83,1545,1582],[1583,0.27699999999999997,66,1545,1584],[1533,0.27699999999999997,195,1563,1585],[1586,0.27699999999999997,15,1545,1587],[512,0.27,370,1563,1588],[1589,0.27,25,1563,-1],[1590,0.27,83,1563,1591],[1592,0.27,83,1563,1593],[1594,0.27,143,1545,1558],[1595,0.264,83,1545,1582],[1596,0.264,370,1563,1597],[1598,0.257,15,1545,1599],[1600,0.257,1,1545,1601],[1602,0.257,1,1545,1601],[1603,0.257,1,1563,1601],[1604,0.257,25,1563,1359],[1605,0.257,370,1563,1606],[1607,0.25,25,1545,1359],[1608,0.25,15,1563,1609],[1610,0.25,15,1563,1611],[1612,0.25,83,1545,1591],[1613,0.25,83,1563,1614],[1615,0.25,370,1563,1616],[523,0.243,25,1545,1617],[1618,0.243,83,1563,1582],[1619,0.243,83,1563,1582],[1620,0.243,83,1563,1582],[1621,0.243,554,1563,1622],[1623,0.23600000000000002,25,1563,1624],[1625,0.23600000000000002,15,1563,1626],[1627,0.23600000000000002,83,1563,1593],[1628,0.23600000000000002,143,1545,1558],[1629,0.23600000000000002,143,1545,1630],[1631,0.23600000000000002,195,1563,1632],[1633,0.23,25,1563,1359],[1634,0.23,83,1563,1593],[524,0.23,66,1563,1635],[1636,0.23,15,1545,1587],[1637,0.223,1,1545,1638],[1639,0.223,83,1563,1640],[536,0.223,66,1563,1641],[1642,0.223,66,1563,1643],[517,0.223,417,1545,1644],[1645,0.223,195,1563,1646],[1647,0.21600000000000003,25,1563,1648],[1649,0.21600000000000003,417,1563,1650],[1651,0.21600000000000003,195,1563,1652],[1653,0.209,25,1563,1654],[1655,0.209,370,1563,1656],[511,0.209,83,1545,1591],[1657,0.209,66,1563,1658],[1659,0.209,195,1563,1660],[1661,0.203,1,1545,1638],[1662,0.203,25,1563,1663],[1664,0.203,370,1563,1665],[1666,0.203,195,1563,1667],[1668,0.196,83,1563,1640],[1669,0.196,83,1563,1591],[1670,0.196,83,1563,1593],[1671,0.196,370,1563,1672],[1673,0.18899999999999997,370,1563,1674],[1675,0.18899999999999997,370,1563,1676],[1677,0.18899999999999997,417,1563,1678],[1679,0.182,370,1563,1680],[1681,0.182,370,1563,1682],[1683,0.182,15,1563,1684],[1685,0.182,66,1563,1686],[1687,0.182,370,1563,1688],[1689,0.182,370,1563,1690],[1691,0.17600000000000002,25,1563,1692],[1693,0.17600000000000002,83,1563,1694],[1695,0.17600000000000002,554,1563,1696],[1697,0.17600000000000002,370,1563,1698],[567,0.16899999999999998,1,1545,1638],[1699,0.16899999999999998,370,1563,1700],[1701,0.16899999999999998,370,1545,1680],[1702,0.16899999999999998,66,1545,1703],[1704,0.16899999999999998,554,1563,1705],[569,0.162,25,1563,1706],[1707,0.155,370,1563,1708],[1709,0.155,370,1563,1710],[1711,0.155,527,1563,1712],[1713,0.149,15,1563,1383],[1714,0.149,370,1563,1715],[519,0.149,195,1563,1716],[1717,0.149,370,1563,1718],[1719,0.142,370,1563,1720],[1721,0.142,370,1563,1722],[1723,0.142,370,1563,1724],[1725,0.135,370,1563,1726],[805,0.135,370,1563,1727],[562,0.135,66,1563,1728],[1729,0.135,370,1563,1730],[1731,0.135,370,1563,1732],[1733,0.128,370,1563,1734],[1735,0.128,370,1563,1736],[1737,0.128,417,1563,1738],[1739,0.128,370,1563,1740],[1741,0.128,370,1563,1742],[1743,0.122,370,1563,1744],[1745,0.122,195,1563,1746],[541,0.122,195,1563,1747],[1748,0.122,195,1563,1749],[1750,0.122,370,1563,1751],[1752,0.115,370,1563,1753],[1754,0.115,370,1563,1755],[1756,0.115,370,1563,1757],[1758,0.115,417,1563,1759],[1760,0.115,370,1563,1761],[1762,0.115,370,1563,1763],[1764,0.115,527,1563,1765],[1766,0.10800000000000001,370,1563,1767],[549,0.10800000000000001,417,1563,1768],[1769,0.10800000000000001,554,1563,1770],[1771,0.10800000000000001,554,1563,1772],[1773,0.10099999999999999,25,1563,1774],[1775,0.10099999999999999,83,1563,1776],[1777,0.10099999999999999,195,1563,1778],[555,0.10099999999999999,195,1563,1779],[1780,0.10099999999999999,370,1563,1781],[557,0.10099999999999999,527,1563,1782],[1783,0.095,370,1563,1784],[1785,0.095,370,1563,1786],[1787,0.095,195,1563,1788],[1789,0.08800000000000001,370,1563,1790],[1791,0.08800000000000001,370,1563,1792],[1793,0.08800000000000001,370,1563,1794],[1795,0.08800000000000001,370,1563,1796],[543,0.08800000000000001,527,1563,1797],[1798,0.07400000000000001,370,1563,1799],[1800,0.07400000000000001,370,1563,1801],[1802,0.07400000000000001,554,1563,1696],[572,0.068,66,1563,1803],[1804,0.054000000000000006,66,1563,1805],[1806,0.054000000000000006,370,1563,1807],[1808,0.054000000000000006,195,1563,1809],[1810,0.047,417,1563,1811],[1812,0.040999999999999995,417,1563,1738],[578,0.040999999999999995,527,1563,1813],[1814,0.034,370,1563,1815],[1816,0.034,25,1563,1817],[1818,0.034,25,1563,1403],[1819,0.034,1820,1563,1821],[1822,0.02,25,1563,1823],[1824,0.02,370,1563,1825],[1826,0.013999999999999999,195,1563,1827],[1828,0.006999999999999999,66,1563,1829],[1830,0.006999999999999999,195,1563,1831],[1832,0.0,25,1563,1833]]},{"id":"humaneval","name":"HumanEval","group":"编程","unit":"%","metric":"pass@1","protocol":{"en":"EvalPlus published pass@1. HumanEval has 164 tasks; MBPP uses the 399-task EvalPlus subset. Base tests and expanded + tests remain separate.","zh":"EvalPlus 发布的 pass@1。HumanEval 为 164 道任务；MBPP 使用 EvalPlus 的 399 道题子集。原始测试与扩展 + 测试分别展示。"},"source":{"name":"EvalPlus","url":"https://evalplus.github.io/leaderboard.html","status":"current_snapshot","published_at":"2021-07-07"},"results":[[1834,0.963,370,1835,-1],[1836,0.963,370,1835,-1],[1837,0.927,15,1835,-1],[1533,0.9209999999999999,195,1835,-1],[1838,0.915,25,1835,-1],[1839,0.902,25,1835,-1],[1840,0.902,15,1835,-1],[1841,0.89,83,1835,-1],[1842,0.884,15,1835,-1],[1843,0.884,15,1835,-1],[1844,0.884,143,1835,-1],[1845,0.872,1,1835,-1],[523,0.8540000000000001,25,1835,-1],[1846,0.8540000000000001,15,1835,-1],[1675,0.835,370,1835,-1],[1847,0.8290000000000001,1,1835,-1],[1848,0.823,83,1835,-1],[1689,0.8170000000000001,370,1835,-1],[1849,0.8109999999999999,25,1835,-1],[1699,0.799,370,1835,-1],[1850,0.799,370,1835,-1],[1851,0.7929999999999999,370,1835,-1],[1852,0.774,66,1835,-1],[1731,0.774,370,1835,-1],[1853,0.774,370,1835,-1],[1854,0.768,15,1835,-1],[1729,0.768,370,1835,-1],[1855,0.768,1,1835,-1],[535,0.762,417,1835,-1],[1743,0.7559999999999999,370,1835,-1],[1856,0.7559999999999999,25,1835,-1],[1857,0.7559999999999999,370,1835,-1],[1741,0.75,370,1835,-1],[1858,0.75,370,1835,-1],[1859,0.7440000000000001,25,1835,-1],[1860,0.7440000000000001,370,1835,-1],[1861,0.738,370,1835,-1],[1862,0.732,15,1835,-1],[1863,0.732,370,1835,-1],[1864,0.726,370,1835,-1],[1754,0.72,370,1835,-1],[1865,0.72,370,1835,-1],[1739,0.713,370,1866,-1],[1867,0.713,370,1835,-1],[1868,0.7070000000000001,370,1835,-1],[1869,0.7070000000000001,1,1835,-1],[1870,0.695,66,1835,-1],[1871,0.695,417,1835,-1],[1872,0.695,1,1835,-1],[1873,0.6829999999999999,83,1835,-1],[541,0.6829999999999999,195,1835,-1],[1874,0.677,370,1835,-1],[1875,0.6709999999999999,370,1835,-1],[1876,0.659,25,1835,-1],[1877,0.6459999999999999,370,1835,-1],[1878,0.6459999999999999,554,1835,-1],[1879,0.64,370,1835,-1],[1880,0.64,370,1835,-1],[1881,0.634,83,1835,-1],[1882,0.616,66,1835,-1],[1883,0.604,370,1835,-1],[1884,0.573,1,1835,-1],[1885,0.5670000000000001,370,1835,-1],[1886,0.561,370,1835,-1],[1887,0.561,370,1835,-1],[1888,0.555,370,1866,-1],[1889,0.5489999999999999,370,1835,-1],[1890,0.5489999999999999,370,1835,-1],[1891,0.5429999999999999,370,1835,-1],[1892,0.518,370,1866,-1],[1893,0.518,370,1866,-1],[1894,0.512,25,1866,-1],[1895,0.506,370,1835,-1],[1896,0.494,554,1866,-1],[1897,0.48200000000000004,417,1866,-1],[1898,0.48200000000000004,370,1835,-1],[1899,0.47600000000000003,25,1866,-1],[1900,0.47600000000000003,417,1835,-1],[1901,0.46299999999999997,370,1866,-1],[565,0.451,417,1835,-1],[1902,0.445,370,1866,-1],[1903,0.433,370,1835,-1],[1904,0.42700000000000005,370,1866,-1],[1905,0.42700000000000005,83,1835,-1],[599,0.42100000000000004,417,1835,-1],[1906,0.402,370,1835,-1],[1907,0.37799999999999995,370,1866,-1],[1908,0.354,370,1866,-1],[1909,0.354,83,1866,-1],[1910,0.341,370,1866,-1],[1911,0.335,66,1866,-1],[1912,0.32899999999999996,370,1866,-1],[1913,0.32899999999999996,370,1835,-1],[1914,0.317,370,1866,-1],[1915,0.317,370,1866,-1],[1916,0.3,370,1866,-1],[1917,0.293,370,1866,-1],[1918,0.293,370,1866,-1],[1919,0.293,370,1866,-1],[1920,0.287,25,1866,-1],[1921,0.287,417,1866,-1],[1922,0.287,83,1835,-1],[1923,0.268,370,1866,-1],[1924,0.25,370,1866,-1],[1925,0.25,83,1866,-1],[1926,0.244,370,1866,-1],[1927,0.244,370,1866,-1],[1928,0.226,83,1835,-1],[1929,0.195,370,1866,-1],[1930,0.183,370,1866,-1],[1931,0.177,370,1866,-1],[1932,0.177,83,1835,-1],[1933,0.171,370,1866,-1],[1934,0.159,370,1866,-1],[1935,0.159,370,1866,-1],[1936,0.146,370,1866,-1],[1937,0.146,370,1866,-1],[1938,0.122,15,1866,-1],[1939,0.122,370,1866,-1],[1940,0.11599999999999999,370,1866,-1],[1941,0.11,370,1866,-1],[1942,0.079,15,1866,-1],[1943,0.061,370,1866,-1],[1944,0.024,370,1866,-1],[1945,0.024,370,1835,-1]]},{"id":"humaneval-plus","name":"HumanEval+","group":"编程","unit":"%","metric":"pass@1","protocol":{"en":"EvalPlus published pass@1. HumanEval has 164 tasks; MBPP uses the 399-task EvalPlus subset. Base tests and expanded + tests remain separate.","zh":"EvalPlus 发布的 pass@1。HumanEval 为 164 道任务；MBPP 使用 EvalPlus 的 399 道题子集。原始测试与扩展 + 测试分别展示。"},"source":{"name":"EvalPlus","url":"https://evalplus.github.io/leaderboard.html","status":"current_snapshot","published_at":"2023-05-02"},"results":[[1834,0.89,370,1835,-1],[1836,0.89,370,1835,-1],[1837,0.872,15,1835,-1],[1533,0.872,195,1835,-1],[1838,0.866,25,1835,-1],[1840,0.866,15,1835,-1],[1839,0.835,25,1835,-1],[1842,0.835,15,1835,-1],[523,0.823,25,1835,-1],[1845,0.8170000000000001,1,1835,-1],[1846,0.8170000000000001,15,1835,-1],[1844,0.805,143,1835,-1],[1843,0.7929999999999999,15,1835,-1],[1841,0.7929999999999999,83,1835,-1],[1675,0.787,370,1835,-1],[1689,0.774,370,1835,-1],[1847,0.774,1,1835,-1],[1848,0.7559999999999999,83,1835,-1],[1849,0.75,25,1835,-1],[1699,0.738,370,1835,-1],[1851,0.738,370,1835,-1],[1850,0.732,370,1835,-1],[1743,0.726,370,1835,-1],[1852,0.72,66,1835,-1],[535,0.72,417,1835,-1],[1731,0.72,370,1835,-1],[1853,0.72,370,1835,-1],[1859,0.713,25,1835,-1],[1856,0.713,25,1835,-1],[1729,0.713,370,1835,-1],[1861,0.713,370,1835,-1],[1854,0.7070000000000001,15,1835,-1],[1860,0.7070000000000001,370,1835,-1],[1858,0.701,370,1835,-1],[1741,0.695,370,1835,-1],[1857,0.695,370,1835,-1],[1855,0.6890000000000001,1,1835,-1],[1868,0.677,370,1835,-1],[1864,0.677,370,1835,-1],[1739,0.6709999999999999,370,1866,-1],[1862,0.665,15,1835,-1],[1754,0.659,370,1835,-1],[1865,0.659,370,1835,-1],[1867,0.659,370,1835,-1],[1863,0.6459999999999999,370,1835,-1],[1869,0.64,1,1835,-1],[1870,0.628,66,1835,-1],[1875,0.628,370,1835,-1],[1871,0.622,417,1835,-1],[1872,0.616,1,1835,-1],[1873,0.61,83,1835,-1],[1876,0.604,25,1835,-1],[1874,0.604,370,1835,-1],[1877,0.597,370,1835,-1],[1878,0.591,554,1835,-1],[541,0.591,195,1835,-1],[1880,0.573,370,1835,-1],[1879,0.5670000000000001,370,1835,-1],[1882,0.5670000000000001,66,1835,-1],[1881,0.555,83,1835,-1],[1886,0.524,370,1835,-1],[1883,0.518,370,1835,-1],[1887,0.518,370,1835,-1],[1888,0.506,370,1866,-1],[1885,0.506,370,1835,-1],[1884,0.506,1,1835,-1],[1890,0.506,370,1835,-1],[1889,0.494,370,1835,-1],[1891,0.488,370,1835,-1],[1893,0.457,370,1866,-1],[1895,0.451,370,1835,-1],[1896,0.451,554,1866,-1],[1894,0.445,25,1866,-1],[1892,0.439,370,1866,-1],[1897,0.42100000000000004,417,1866,-1],[1900,0.42100000000000004,417,1835,-1],[1902,0.415,370,1866,-1],[1898,0.415,370,1835,-1],[1899,0.396,25,1866,-1],[565,0.396,417,1835,-1],[1904,0.384,370,1866,-1],[1901,0.37799999999999995,370,1866,-1],[1903,0.37200000000000005,370,1835,-1],[599,0.36,417,1835,-1],[1907,0.354,370,1866,-1],[1905,0.354,83,1835,-1],[1906,0.32899999999999996,370,1835,-1],[1913,0.305,370,1835,-1],[1908,0.299,370,1866,-1],[1911,0.293,66,1866,-1],[1910,0.293,370,1866,-1],[1909,0.287,83,1866,-1],[1912,0.28,370,1866,-1],[1915,0.27399999999999997,370,1866,-1],[1914,0.268,370,1866,-1],[1917,0.256,370,1866,-1],[1920,0.256,25,1866,-1],[1919,0.256,370,1866,-1],[1922,0.25,83,1835,-1],[1918,0.244,370,1866,-1],[1921,0.23800000000000002,417,1866,-1],[1916,0.23199999999999998,370,1866,-1],[1926,0.226,370,1866,-1],[1924,0.22,370,1866,-1],[1927,0.213,370,1866,-1],[1923,0.207,370,1866,-1],[1925,0.207,83,1866,-1],[1930,0.177,370,1866,-1],[1928,0.177,83,1835,-1],[1929,0.165,370,1866,-1],[1931,0.159,370,1866,-1],[1933,0.159,370,1866,-1],[1932,0.152,83,1835,-1],[1936,0.14,370,1866,-1],[1934,0.128,370,1866,-1],[1935,0.122,370,1866,-1],[1937,0.122,370,1866,-1],[1940,0.11599999999999999,370,1866,-1],[1938,0.11,15,1866,-1],[1939,0.11,370,1866,-1],[1941,0.091,370,1866,-1],[1942,0.067,15,1866,-1],[1943,0.061,370,1866,-1],[1944,0.024,370,1866,-1],[1945,0.018000000000000002,370,1835,-1]]},{"id":"mbpp-evalplus","name":"MBPP (EvalPlus subset)","group":"编程","unit":"%","metric":"pass@1","protocol":{"en":"EvalPlus published pass@1. HumanEval has 164 tasks; MBPP uses the 399-task EvalPlus subset. Base tests and expanded + tests remain separate.","zh":"EvalPlus 发布的 pass@1。HumanEval 为 164 道任务；MBPP 使用 EvalPlus 的 399 道题子集。原始测试与扩展 + 测试分别展示。"},"source":{"name":"EvalPlus","url":"https://evalplus.github.io/leaderboard.html","status":"current_snapshot","published_at":"2021-08-16"},"results":[[1836,0.955,370,1835,-1],[1834,0.9309999999999999,370,1835,-1],[1533,0.905,195,1835,-1],[1841,0.897,83,1835,-1],[1845,0.894,1,1835,-1],[523,0.894,25,1835,-1],[1847,0.894,1,1835,-1],[1839,0.8759999999999999,25,1835,-1],[1838,0.8759999999999999,25,1835,-1],[1837,0.8759999999999999,15,1835,-1],[1844,0.86,143,1835,-1],[1846,0.857,15,1835,-1],[1842,0.8540000000000001,15,1835,-1],[1848,0.847,83,1835,-1],[1869,0.836,1,1835,-1],[1854,0.825,15,1835,-1],[1852,0.823,66,1835,-1],[1689,0.82,370,1835,-1],[1743,0.807,370,1835,-1],[1849,0.804,25,1835,-1],[1851,0.802,370,1835,-1],[1855,0.802,1,1835,-1],[1675,0.794,370,1835,-1],[1729,0.794,370,1835,-1],[1865,0.794,370,1835,-1],[1874,0.78,370,1835,-1],[1857,0.77,370,1835,-1],[1731,0.765,370,1835,-1],[1860,0.762,370,1835,-1],[1867,0.759,370,1835,-1],[1881,0.754,83,1835,-1],[1856,0.752,25,1835,-1],[1863,0.7509999999999999,370,1835,-1],[1859,0.7490000000000001,25,1835,-1],[1741,0.7490000000000001,370,1835,-1],[1861,0.7490000000000001,370,1835,-1],[1879,0.743,370,1835,-1],[535,0.738,417,1835,-1],[1853,0.738,370,1835,-1],[1893,0.735,370,1866,-1],[1875,0.735,370,1835,-1],[1871,0.728,417,1835,-1],[1699,0.725,370,1835,-1],[541,0.725,195,1835,-1],[1899,0.72,25,1866,-1],[1868,0.706,370,1835,-1],[1880,0.706,370,1835,-1],[1883,0.7040000000000001,370,1835,-1],[1892,0.693,370,1866,-1],[1870,0.6829999999999999,66,1835,-1],[1858,0.672,370,1835,-1],[1887,0.667,370,1835,-1],[1878,0.659,554,1835,-1],[1876,0.653,25,1835,-1],[1902,0.6509999999999999,370,1866,-1],[1882,0.6459999999999999,66,1835,-1],[1885,0.643,370,1835,-1],[1890,0.642,370,1835,-1],[1896,0.64,554,1866,-1],[1864,0.638,370,1835,-1],[1904,0.635,370,1866,-1],[1911,0.614,66,1866,-1],[1907,0.595,370,1866,-1],[565,0.595,417,1835,-1],[1895,0.585,370,1835,-1],[1900,0.574,417,1835,-1],[1898,0.574,370,1835,-1],[1905,0.5710000000000001,83,1835,-1],[1920,0.569,25,1866,-1],[1914,0.5660000000000001,370,1866,-1],[1923,0.556,370,1866,-1],[1910,0.551,370,1866,-1],[1919,0.5479999999999999,370,1866,-1],[1912,0.542,370,1866,-1],[1918,0.529,370,1866,-1],[1909,0.526,83,1866,-1],[1891,0.524,370,1835,-1],[1921,0.519,417,1866,-1],[1917,0.508,370,1866,-1],[1906,0.503,370,1835,-1],[1924,0.484,370,1866,-1],[1922,0.47100000000000003,83,1835,-1],[1926,0.46299999999999997,370,1866,-1],[599,0.447,417,1835,-1],[1903,0.439,370,1835,-1],[1916,0.42100000000000004,370,1866,-1],[1925,0.418,83,1866,-1],[1928,0.298,83,1835,-1],[1945,0.098,370,1835,-1]]},{"id":"mbpp-plus","name":"MBPP+","group":"编程","unit":"%","metric":"pass@1","protocol":{"en":"EvalPlus published pass@1. HumanEval has 164 tasks; MBPP uses the 399-task EvalPlus subset. Base tests and expanded + tests remain separate.","zh":"EvalPlus 发布的 pass@1。HumanEval 为 164 道任务；MBPP 使用 EvalPlus 的 399 道题子集。原始测试与扩展 + 测试分别展示。"},"source":{"name":"EvalPlus","url":"https://evalplus.github.io/leaderboard.html","status":"current_snapshot","published_at":"2023-05-02"},"results":[[1836,0.802,370,1835,-1],[1834,0.7879999999999999,370,1835,-1],[1533,0.77,195,1835,-1],[523,0.7509999999999999,25,1835,-1],[1841,0.746,83,1835,-1],[1845,0.743,1,1835,-1],[1839,0.741,25,1835,-1],[1846,0.733,15,1835,-1],[1847,0.733,1,1835,-1],[1838,0.73,25,1835,-1],[1837,0.722,15,1835,-1],[1842,0.722,15,1835,-1],[1689,0.7140000000000001,370,1835,-1],[1849,0.701,25,1835,-1],[1854,0.6970000000000001,15,1835,-1],[1743,0.696,370,1835,-1],[1869,0.693,1,1835,-1],[1675,0.69,370,1835,-1],[1852,0.69,66,1835,-1],[1729,0.69,370,1835,-1],[1855,0.688,1,1835,-1],[1851,0.685,370,1835,-1],[1848,0.675,83,1835,-1],[1865,0.669,370,1835,-1],[1731,0.664,370,1835,-1],[1859,0.6559999999999999,25,1835,-1],[1844,0.6559999999999999,143,1835,-1],[1874,0.6509999999999999,370,1835,-1],[1857,0.648,370,1835,-1],[1860,0.6459999999999999,370,1835,-1],[1861,0.6459999999999999,370,1835,-1],[1867,0.644,370,1835,-1],[535,0.643,417,1835,-1],[1879,0.635,370,1835,-1],[1741,0.635,370,1835,-1],[1863,0.632,370,1835,-1],[1875,0.624,370,1835,-1],[1856,0.622,25,1835,-1],[1699,0.619,370,1835,-1],[541,0.616,195,1835,-1],[1881,0.614,83,1835,-1],[1853,0.614,370,1835,-1],[1893,0.608,370,1866,-1],[1868,0.601,370,1835,-1],[1871,0.595,417,1835,-1],[1880,0.59,370,1835,-1],[1899,0.5870000000000001,25,1866,-1],[1883,0.569,370,1835,-1],[1892,0.563,370,1866,-1],[1887,0.563,370,1835,-1],[1858,0.5579999999999999,370,1835,-1],[1870,0.556,66,1835,-1],[1876,0.5479999999999999,25,1835,-1],[1882,0.5479999999999999,66,1835,-1],[1864,0.545,370,1835,-1],[1878,0.542,554,1835,-1],[1885,0.542,370,1835,-1],[1896,0.542,554,1866,-1],[1904,0.526,370,1866,-1],[1902,0.524,370,1866,-1],[1911,0.516,66,1866,-1],[1890,0.506,370,1835,-1],[565,0.49700000000000005,417,1835,-1],[1895,0.495,370,1835,-1],[1898,0.48700000000000004,370,1835,-1],[1920,0.479,25,1866,-1],[1914,0.47100000000000003,370,1866,-1],[1907,0.46799999999999997,370,1866,-1],[1923,0.466,370,1866,-1],[1900,0.46399999999999997,417,1835,-1],[1910,0.461,370,1866,-1],[1891,0.45799999999999996,370,1835,-1],[1919,0.45799999999999996,370,1866,-1],[1912,0.455,370,1866,-1],[1905,0.45,83,1835,-1],[1909,0.434,83,1866,-1],[1917,0.429,370,1866,-1],[1921,0.42100000000000004,417,1866,-1],[1918,0.415,370,1866,-1],[1906,0.413,370,1835,-1],[1924,0.381,370,1866,-1],[599,0.37,417,1835,-1],[1922,0.368,83,1835,-1],[1903,0.36200000000000004,370,1835,-1],[1926,0.36,370,1866,-1],[1916,0.34700000000000003,370,1866,-1],[1925,0.341,83,1866,-1],[1928,0.233,83,1835,-1],[1945,0.075,370,1835,-1]]},{"id":"webarena","name":"WebArena","group":"Agent 与工具","unit":"%","metric":"Task success","protocol":{"en":"Official submission leaderboard; agent scaffolds and input modes vary and are retained. Scores are reported by submitting teams.","zh":"官方收录的提交榜单；Agent 框架与输入模式可能不同，随结果保留。成绩由提交团队报告。"},"source":{"name":"WebArena authors","url":"https://webarena.dev/og/","status":"current_snapshot","published_at":"2023-07-25"},"results":[[1946,0.861,370,1946,1947],[1948,0.743,25,1949,1950],[1951,0.716,370,1951,1952],[1953,0.7120000000000001,370,1954,1955],[1956,0.68,1,1957,1958],[1959,0.669,370,1959,-1],[1960,0.642,370,1960,-1],[1961,0.617,370,1961,1962],[1963,0.581,370,1964,1965],[1966,0.5710000000000001,370,1967,-1],[1968,0.563,551,1969,1970],[1971,0.546,370,1972,1973],[1974,0.53,370,1975,1976],[1977,0.521,370,1978,1979],[1980,0.48,370,1981,1982],[1983,0.457,370,1983,1984],[1985,0.42100000000000004,370,1986,1987],[1988,0.37200000000000005,370,1989,1990],[1991,0.358,370,1992,1993],[1994,0.355,370,1995,1996],[1997,0.335,370,1998,1999],[2000,0.261,370,2000,2001],[2002,0.235,370,2003,2004],[2005,0.22399999999999998,370,2006,2007],[2008,0.20199999999999999,15,2009,2010],[2011,0.192,15,2012,2013],[2014,0.182,370,2014,2015],[2016,0.163,370,2017,2018],[2019,0.149,15,2020,2021],[2022,0.131,15,2023,-1],[2019,0.11699999999999999,15,2023,2021],[2024,0.09359999999999999,195,2025,2026],[2027,0.08869999999999999,15,2028,2021],[2029,0.07139999999999999,195,50,2026],[2030,0.0712,83,2030,2021],[2031,0.0702,66,2032,-1],[2033,0.06280000000000001,370,2034,2035],[2036,0.053,370,2037,2038],[2039,0.046799999999999994,370,2039,2040],[2041,0.0406,370,2042,2038],[2043,0.0381,370,2044,2045],[2046,0.0332,66,2032,-1],[2047,0.023,370,2048,-1],[2049,0.016,370,2044,2045],[2050,0.0139,417,2050,2021],[2051,0.0074,370,2044,2045],[2052,0.0025,370,2052,2040],[2053,0.0,370,2054,-1]]},{"id":"visualwebarena","name":"VisualWebArena","group":"Agent 与工具","unit":"%","metric":"Task success","protocol":{"en":"Official submission leaderboard; agent scaffolds and input modes vary and are retained. Scores are reported by submitting teams.","zh":"官方收录的提交榜单；Agent 框架与输入模式可能不同，随结果保留。成绩由提交团队报告。"},"source":{"name":"VisualWebArena authors","url":"https://jykoh.com/vwa/","status":"current_snapshot","published_at":"2024-01-24"},"results":[[2055,0.54,83,2056,2057],[2058,0.529,15,2059,2060],[2061,0.48700000000000004,551,2062,1970],[2058,0.365,15,2063,-1],[1985,0.337,370,2064,1987],[2065,0.337,15,2066,2067],[2068,0.264,15,2069,2013],[2070,0.23600000000000002,15,2071,2072],[2073,0.23399999999999999,15,2074,2075],[353,0.1978,15,2076,2077],[2078,0.16699999999999998,66,2079,2013],[872,0.1637,15,2076,2077],[872,0.1505,15,2080,2077],[2081,0.1275,15,2082,2077],[2083,0.1198,83,2076,2077],[2084,0.11199999999999999,370,2085,2086],[2087,0.0978,66,2082,2077],[2088,0.08199999999999999,195,2074,2075],[2089,0.0725,15,2090,2077],[2091,0.0659,83,2076,2077],[2092,0.064,370,2085,2086],[2093,0.0604,83,2080,2077],[2093,0.0571,83,2076,2077],[2094,0.055,195,2085,2086],[2095,0.0385,83,2082,2077],[2096,0.0297,15,2082,2077],[823,0.028999999999999998,195,2074,2075],[2097,0.0275,15,2082,2077],[2098,0.022000000000000002,15,2090,2077],[2093,0.022000000000000002,83,2090,2077],[2099,0.0187,417,2082,2077],[2100,0.0176,417,2090,2077],[583,0.011000000000000001,66,2090,2077],[2101,0.009899999999999999,370,2076,2077],[2101,0.0077,370,2080,2077],[2102,0.0066,66,2082,2077],[2103,0.0033,370,2076,2077],[2103,0.0033,370,2080,2077]]},{"id":"androidworld","name":"AndroidWorld","group":"Agent 与工具","unit":"%","metric":"Task success (pass@1)","protocol":{"en":"Official community-submission board, not independently verified. Uses pass@1 only; trial counts, agents and screen representations are retained.","zh":"官方社区提交榜单，发布方未独立验证。仅使用 pass@1；保留试验次数、Agent 和屏幕表示。"},"source":{"name":"Google Research / AndroidWorld","url":"https://google-research.github.io/android_world/","status":"current_snapshot","published_at":"2024-05-23"},"results":[[2104,1.0,370,2105,-1],[2106,1.0,15,2107,-1],[2108,1.0,15,2109,2110],[2111,1.0,15,2112,2113],[2114,1.0,15,2115,2116],[2117,0.991,83,2118,-1],[2119,0.9740000000000001,83,2120,2121],[2122,0.9740000000000001,370,2123,-1],[2124,0.948,370,2125,-1],[2126,0.948,83,2127,2128],[2129,0.9309999999999999,83,2130,-1],[2131,0.914,66,2132,2133],[2134,0.871,370,2135,2136],[2137,0.862,370,2138,2139],[2140,0.802,370,2141,-1],[2142,0.7929999999999999,370,2143,-1],[2144,0.78,83,2145,-1],[2146,0.767,370,2147,-1],[2148,0.767,195,2149,2150],[2151,0.75,370,2152,-1],[2153,0.733,370,2154,-1],[2155,0.733,370,2147,-1],[2156,0.7070000000000001,370,2147,-1],[2157,0.6970000000000001,83,2158,-1],[2159,0.672,370,2160,-1],[2161,0.664,370,2162,-1],[2163,0.659,370,2164,2165],[2166,0.629,195,2167,-1],[2168,0.621,370,2169,-1],[2170,0.6,370,2171,-1],[2172,0.595,370,2173,-1],[2174,0.5429999999999999,370,2175,-1],[2176,0.491,370,2147,-1],[2177,0.491,370,2178,2165],[353,0.474,15,2179,-1],[353,0.46799999999999997,15,2180,-1],[2181,0.466,370,2182,-1],[2183,0.44799999999999995,15,2184,-1],[2185,0.44,15,2186,-1],[2187,0.44,370,2188,-1],[353,0.42200000000000004,15,2189,-1],[353,0.345,15,2190,-1],[2191,0.306,15,2192,-1],[2193,0.276,15,2194,-1],[2195,0.276,195,2196,-1],[2197,0.09,195,2198,-1]]},{"id":"bird-sql-test-single","name":"BIRD-SQL (single-model test)","group":"编程","unit":"%","metric":"Execution accuracy","protocol":{"en":"Single-model test EX track. Self-consistency settings vary and are retained; Dev, efficiency and Mini-Dev scores are excluded.","zh":"单模型 Test EX 赛道；保留各结果的自洽采样设置。不混入 Dev、效率分数或 Mini-Dev 成绩。"},"source":{"name":"BIRD","url":"https://bird-bench.github.io/","status":"current_snapshot","published_at":"2023-05-04"},"results":[[2199,0.8004000000000001,2200,2201,-1],[2202,0.7814,2203,2201,2204],[2205,0.777,2206,2201,-1],[2207,0.7714,2208,2201,2209],[2210,0.7707999999999999,2206,2201,-1],[2211,0.7686,2212,2201,-1],[2213,0.7646999999999999,2214,2201,2215],[2216,0.7568,2217,2218,2219],[2220,0.7484999999999999,2221,2218,2222],[2223,0.7479,2224,2201,-1],[2225,0.7451000000000001,2226,2201,-1],[2227,0.7429000000000001,2228,2201,-1],[2229,0.7412000000000001,2230,2201,-1],[2231,0.7401000000000001,2230,2201,-1],[2232,0.7384000000000001,2233,2218,2234],[2235,0.7373000000000001,2236,2201,-1],[2216,0.7356,2217,2237,2219],[2238,0.7345,2239,2201,-1],[2240,0.7278,2241,2218,-1],[2242,0.7254999999999999,15,2237,-1],[2243,0.7222,2233,2218,2234],[2244,0.72,2245,2201,2246],[2247,0.7166,2248,2237,-1],[2249,0.7116,2250,2201,-1],[2251,0.7093,2252,2201,-1],[2253,0.7093,2254,2255,2256],[2257,0.7071,2250,2201,-1],[2258,0.706,2259,2218,-1],[2260,0.7043,2233,2218,2234],[2261,0.7015,1,2237,-1],[2262,0.6903,936,2218,2263],[2264,0.6853,2233,2237,2265],[2266,0.6847,2236,2218,-1],[2267,0.6819,2233,2237,2265],[2268,0.6814,195,2237,-1],[2269,0.6797,2270,2218,-1],[2271,0.6685,1,2237,-1],[2272,0.6640999999999999,2270,2218,-1],[2273,0.6568,2274,2237,2275],[2276,0.6333,2277,2237,-1],[2278,0.6294,551,2237,-1],[2279,0.6265999999999999,2280,2237,2281],[769,0.6093,25,2237,-1],[2282,0.6042000000000001,2283,2237,2284],[2285,0.6037,2286,2237,2287],[2288,0.5987,21,2237,-1],[2289,0.5959,2290,2218,2291],[2292,0.5925,2286,2237,2287],[25,0.5668,25,2237,-1],[365,0.5584,417,2237,-1],[2089,0.5489,15,2237,-1],[2293,0.5154,2294,2218,-1],[2295,0.4902,1,2237,-1],[2296,0.4008,2297,2237,2298],[2299,0.39299999999999996,15,2237,-1],[2300,0.36469999999999997,370,2237,-1],[2301,0.33149999999999996,370,2218,2302],[2303,0.33039999999999997,83,2237,-1],[2304,0.2733,370,2237,2302],[2305,0.24050000000000002,370,2237,-1],[2306,0.2094,370,2237,-1],[2307,0.12890000000000001,370,2237,-1]]},{"id":"spider-2-lite","name":"Spider 2.0-Lite","group":"编程","unit":"%","metric":"Task success","protocol":{"en":"Official Lite submission track, 547 tasks. Model and agent names follow the authors’ table.","zh":"官方 Lite 提交赛道，共 547 道任务。模型与 Agent 名称取自作者的原表。"},"source":{"name":"Spider 2.0","url":"https://spider2-sql.github.io/","status":"current_snapshot","published_at":"2024-11-12"},"results":[[2308,0.7623000000000001,2309,2310,-1],[2311,0.7495,2312,2310,2313],[2314,0.7367,2315,2310,2316],[2317,0.7313,2318,2310,-1],[2319,0.7202,2320,2310,2321],[2322,0.6965,2323,2310,2324],[2325,0.6581,2326,2310,2327],[2328,0.6416,2329,2310,-1],[2330,0.619,2331,2310,2332],[2333,0.612,2334,2310,-1],[2335,0.6015,2309,2310,-1],[2336,0.5521,2337,2310,2338],[2339,0.5466,2340,2310,-1],[2341,0.5302,2342,2310,-1],[2343,0.5228,2344,2310,2345],[2346,0.5131,370,2310,-1],[2347,0.445,2348,2310,2349],[2350,0.424,2351,2310,2352],[2353,0.41859999999999997,2354,2310,-1],[2355,0.35600000000000004,2356,2310,-1],[2357,0.35600000000000004,2348,2310,2349],[2358,0.3309,370,2310,2359],[2360,0.3309,2344,2310,2361],[2362,0.3236,2363,2310,-1],[2364,0.3053,2344,2310,2361],[2365,0.3035,2337,2310,2338],[2366,0.2852,370,2310,-1],[2367,0.2779,370,2310,-1],[2368,0.2614,2344,2310,2361],[2369,0.2541,370,2310,-1],[2370,0.2486,370,2310,2359],[2371,0.23399999999999999,370,2310,-1],[2372,0.2303,370,2310,-1],[2373,0.21899999999999997,2351,2310,2352],[2374,0.1371,370,2310,-1],[2375,0.1316,370,2310,-1],[2376,0.1133,370,2310,-1],[2377,0.08960000000000001,2270,2310,-1],[2378,0.0832,370,2310,-1],[2379,0.058499999999999996,370,2310,-1],[2380,0.056799999999999996,370,2310,-1],[2381,0.0384,370,2310,-1],[2382,0.0146,370,2310,-1],[2285,0.0073,370,2310,-1]]},{"id":"screenspot-pro","name":"ScreenSpot-Pro","group":"Agent 与工具","unit":"%","metric":"Grounding accuracy","protocol":{"en":"Professional GUI grounding; published micro-average with greedy decoding. Zoom, routing and agent configurations are retained.","zh":"专业 GUI 定位；使用官方贪心解码的微平均成绩，保留缩放、路由和 Agent 配置。"},"source":{"name":"ScreenSpot-Pro","url":"https://gui-agent.github.io/grounding-leaderboard/","status":"current_snapshot","published_at":"2025-04-04"},"results":[[2383,0.8513598987982289,370,2384,-1],[2385,0.8500948766603416,370,2386,-1],[2387,0.8273244781783681,370,370,-1],[2388,0.8096141682479443,370,370,-1],[2389,0.809,370,2390,-1],[2391,0.8051865907653384,370,370,-1],[2392,0.7849462365591398,370,2392,-1],[2393,0.7746212121212122,370,2394,-1],[2395,0.7678684376976597,370,2396,-1],[2397,0.7640733712839974,370,370,-1],[2398,0.7520556609740671,370,2398,-1],[2399,0.7414664981036663,370,2400,-1],[2401,0.734977862112587,370,370,-1],[2402,0.7337128399746996,370,370,-1],[2403,0.7330803289057558,370,370,-1],[2404,0.7318153067678684,370,370,-1],[2405,0.7293,370,2406,-1],[2407,0.7242,370,2408,-1],[2409,0.7191650853889943,370,370,-1],[2410,0.7141049968374447,370,2410,-1],[2411,0.7084,370,2408,-1],[2412,0.7058823529411765,370,370,-1],[2413,0.7058823529411765,370,2413,-1],[2414,0.702719797596458,370,370,-1],[2415,0.6963946869070209,370,370,-1],[2416,0.6856419987349779,370,2416,-1],[2417,0.6843769765970904,370,370,-1],[2155,0.6786843769765971,370,370,-1],[2418,0.6704617330803289,370,370,-1],[2419,0.6641,370,2408,-1],[2420,0.6609740670461733,370,370,-1],[2156,0.6565464895635673,370,370,-1],[2421,0.6502525252525253,370,2400,-1],[2422,0.6363061353573688,370,370,-1],[2423,0.635,370,2408,-1],[2424,0.6325110689437066,370,370,-1],[2425,0.6318785578747628,370,370,-1],[2426,0.6280834914611005,370,370,-1],[2427,0.618595825426945,370,370,-1],[2428,0.6173,370,2408,-1],[2429,0.6160657811511702,370,370,-1],[2430,0.6160657811511702,370,370,-1],[2431,0.6091081593927894,370,370,-1],[2432,0.6078,370,370,-1],[2433,0.5958254269449715,370,370,-1],[2434,0.5888678051865908,370,370,-1],[2435,0.5844402277039848,370,370,-1],[2436,0.5793801391524351,370,370,-1],[2437,0.5774826059456041,370,2438,-1],[2439,0.5774826059456041,370,370,-1],[2176,0.5736875395319418,370,370,-1],[2440,0.5724225173940544,370,370,-1],[2441,0.556,370,2408,-1],[2442,0.5547122074636306,370,370,-1],[2443,0.5357368753953194,370,370,-1],[2444,0.5332068311195446,195,370,-1],[2445,0.528147,370,2446,-1],[2447,0.5253164556962026,370,370,-1],[2448,0.5186590765338394,370,370,-1],[2449,0.5148640101201771,370,370,-1],[2450,0.5098039215686274,21,370,-1],[2451,0.5079063883617964,370,370,-1],[2452,0.5009487666034156,370,370,-1],[2453,0.48007590132827327,195,370,-1],[2454,0.47881087919038584,370,2455,-1],[2456,0.47248576850094876,370,370,-1],[2457,0.4497153700189753,370,2455,-1],[2458,0.4459203036053131,370,370,-1],[2459,0.42188488298545224,370,370,-1],[2460,0.407337128399747,370,370,-1],[2461,0.38140417457305503,370,370,-1],[2462,0.3668564199873498,370,370,-1],[2463,0.3592662871600253,370,370,-1],[2464,0.35863377609108,370,370,-1],[2465,0.3573687539531942,370,370,-1],[2466,0.34535104364326374,370,370,-1],[2467,0.31562302340290954,370,370,-1],[2468,0.3105629348513599,370,370,-1],[2469,0.28716002530044277,370,370,-1],[2470,0.27703984819734345,370,370,-1],[2471,0.2675521821631879,195,370,-1],[2472,0.26628716002530045,370,370,-1],[2473,0.22896900695762176,370,370,-1],[2474,0.18912080961416824,370,370,-1],[2475,0.18532574320050602,15,370,-1],[2476,0.17101226993865032,1,370,-1],[2477,0.1650853889943074,370,370,-1],[2478,0.16065781151170144,195,370,-1],[2479,0.11321948134092347,370,370,-1],[2480,0.07716635041113219,370,370,-1],[2481,0.07716635041113219,370,370,-1],[2482,0.06008855154965212,15,370,-1],[2483,0.03731815306767868,370,370,-1],[2484,0.030360531309297913,370,370,-1],[2485,0.015812776723592662,195,370,-1],[2486,0.011385199240986717,370,370,-1],[2487,0.010120177103099304,195,370,-1],[353,0.008222643896268185,15,370,-1],[2094,0.0006325110689437065,195,370,-1]]},{"id":"mathvista-testmini","name":"MathVista (testmini)","group":"数学与科学","unit":"%","metric":"Accuracy","protocol":{"en":"1,000-question testmini cohort, not the full test split. Prompting and tool methods follow the official results.","zh":"1,000 道题的 testmini，不与完整测试集混用；保留官方成绩中的提示与工具方法。"},"source":{"name":"MathVista","url":"https://mathvista.github.io/","status":"current_snapshot","published_at":"2023-10-03"},"results":[[2488,0.852,370,2489,-1],[2490,0.8029999999999999,370,2489,-1],[2491,0.8009999999999999,370,2489,-1],[2492,0.8,21,2489,-1],[2493,0.795,370,2489,-1],[2494,0.7709999999999999,370,2495,-1],[2496,0.7490000000000001,21,2489,-1],[2497,0.7390000000000001,15,2489,-1],[362,0.737,66,2495,-1],[2498,0.732,370,2489,-1],[67,0.731,83,2495,-1],[2499,0.7140000000000001,370,2495,-1],[2500,0.705,195,2495,-1],[2501,0.6940000000000001,370,2495,-1],[2502,0.69,143,2495,-1],[2503,0.6809999999999999,143,2495,-1],[788,0.677,1,2495,-1],[844,0.675,370,2495,-1],[2504,0.6729999999999999,370,2495,-1],[2505,0.672,370,2495,-1],[2506,0.6679999999999999,370,2495,-1],[2507,0.662,370,2495,-1],[2508,0.6609999999999999,370,2495,-1],[2509,0.648,370,2495,-1],[2510,0.639,83,2495,-1],[353,0.638,15,2495,-1],[2511,0.624,554,2495,-1],[2512,0.599,370,2495,-1],[2513,0.584,83,2495,-1],[2514,0.581,15,2495,-1],[2515,0.58,370,2495,-1],[2516,0.5760000000000001,1820,2495,-1],[2517,0.53,83,2495,-1],[2518,0.528,143,2495,-1],[2519,0.521,83,2495,-1],[2520,0.505,1,2495,-1],[2521,0.499,15,2495,-1],[2522,0.479,1,2495,-1],[2523,0.47700000000000004,370,2495,-1],[2524,0.466,370,2495,-1],[2525,0.465,370,2495,-1],[2526,0.46399999999999997,1,2495,-1],[2527,0.452,83,2495,-1],[2528,0.445,554,2495,-1],[2529,0.439,554,2495,-1],[2530,0.433,370,2495,-1],[2531,0.433,195,2495,-1],[2532,0.423,370,2533,-1],[2534,0.418,370,2495,-1],[2535,0.409,370,2495,-1],[2536,0.406,370,2495,-1],[2537,0.39399999999999996,370,2495,-1],[2538,0.368,370,2533,-1],[2539,0.36700000000000005,370,2495,-1],[2540,0.359,370,2495,-1],[2541,0.355,370,2533,-1],[2542,0.349,370,2495,-1],[2543,0.348,370,2495,-1],[2544,0.34600000000000003,370,2495,-1],[2545,0.33899999999999997,370,2546,-1],[2547,0.332,370,2546,-1],[2548,0.332,370,2546,-1],[2549,0.332,370,2546,-1],[2550,0.326,370,2495,-1],[2551,0.32,370,2495,-1],[2552,0.306,83,2495,-1],[2553,0.276,370,2495,-1],[2554,0.275,370,2495,-1],[2555,0.273,83,2495,-1],[2556,0.268,370,2546,-1],[2557,0.264,370,2533,-1],[2558,0.261,370,2495,-1],[2559,0.253,370,2495,-1],[2560,0.252,370,2495,-1],[2561,0.239,66,2495,-1],[2562,0.231,370,2495,-1],[2563,0.222,370,2495,-1],[2564,0.198,370,2495,-1]]},{"id":"math-vision-full","name":"MATH-Vision (full test)","group":"数学与科学","unit":"%","metric":"Accuracy","protocol":{"en":"Full 3,040-question test set. Official board includes publisher and third-party submissions; tool use and third-party labels are retained in result names.","zh":"完整 3,040 道测试题。官方榜单包含模型发布方与第三方提交；结果名称保留工具使用和第三方标注。"},"source":{"name":"MATH-Vision","url":"https://mathllm.github.io/mathvision/","status":"current_snapshot","published_at":"2024-02-22"},"results":[[2565,0.961,15,2566,2567],[2568,0.9570000000000001,83,2566,2567],[2569,0.932,21,2566,2567],[2570,0.92,15,2566,2567],[2571,0.9059999999999999,83,2566,2572],[2573,0.898,370,2566,2572],[2574,0.898,83,2566,2567],[2575,0.888,370,2566,2572],[2576,0.8859999999999999,195,2566,2577],[2578,0.8740000000000001,21,2566,2567],[2579,0.866,83,2566,2580],[2581,0.8640000000000001,370,2566,2572],[2582,0.862,195,2566,2583],[2584,0.86,195,2566,2583],[2585,0.856,83,2566,2586],[2587,0.85,21,2566,2567],[2588,0.846,1,2566,2567],[244,0.8420000000000001,21,2566,2589],[2590,0.8390000000000001,195,2566,2583],[2591,0.83,15,2566,2592],[2593,0.8240000000000001,83,2566,2586],[2594,0.813,370,2566,2595],[2596,0.772,370,2566,2595],[2597,0.7709999999999999,1,2566,2592],[2598,0.7595000000000001,370,2566,2599],[2600,0.746,195,2566,2601],[2602,0.733,83,2566,2603],[2058,0.72,15,2566,2604],[2605,0.7190000000000001,15,2566,2583],[2606,0.7120000000000001,1,2566,2567],[2607,0.711,1,2566,2583],[2608,0.7081000000000001,370,2566,2599],[2609,0.6964,370,2566,2610],[2168,0.687,370,2566,2611],[2612,0.66,1,2566,2601],[2613,0.66,195,2566,2601],[2614,0.6559999999999999,551,2566,2615],[2616,0.648,370,2566,2617],[2618,0.639,370,2566,2619],[2620,0.635,551,2566,2621],[2622,0.604,370,2566,2623],[2497,0.603,15,2566,1550],[2624,0.5965,370,2566,2625],[2626,0.596,195,2566,2601],[2627,0.595,83,2566,2586],[2628,0.586,1,2566,2629],[2630,0.58,15,2566,2631],[2632,0.579,370,2566,2625],[2450,0.569,21,2566,2633],[2491,0.5660000000000001,370,2566,2634],[2635,0.557,370,2566,2636],[2637,0.5539000000000001,370,2566,2638],[2639,0.544,551,2566,2640],[2641,0.5405,551,2566,2621],[2642,0.54,370,2566,2643],[2644,0.539,370,2566,2645],[2646,0.539,370,2566,2647],[2648,0.5329,21,2566,2649],[2650,0.526,370,2566,2651],[2652,0.526,370,2566,2629],[2653,0.524,83,2566,2586],[2654,0.5205,370,2566,2643],[2655,0.518,370,2566,2656],[2657,0.5029,370,2566,2658],[2659,0.49700000000000005,370,2566,2660],[2661,0.48619999999999997,370,2566,2662],[2663,0.48100000000000004,83,2566,2664],[342,0.473,15,2566,2665],[2666,0.46,83,2566,2586],[2667,0.46,370,2566,2647],[2668,0.45799999999999996,15,2566,2601],[2669,0.4493,370,2566,2670],[2671,0.441,370,2566,2672],[2673,0.439,370,2566,2674],[2675,0.431,370,2566,2676],[2677,0.429,370,2566,2631],[2678,0.42729999999999996,370,2566,2679],[2680,0.413,83,2566,1593],[2681,0.408,370,2566,2682],[2683,0.405,370,2566,2684],[2685,0.401,370,2566,2686],[2687,0.4,370,2566,2643],[2688,0.386,21,2566,2689],[2690,0.384,370,2566,2691],[2166,0.381,195,2566,-1],[2692,0.3799,1,2566,2693],[2694,0.374,370,2566,2695],[2696,0.374,370,2566,2647],[2697,0.37,370,2566,2676],[2698,0.368,21,2566,2699],[2700,0.36200000000000004,554,2566,2701],[2702,0.359,370,2566,2703],[2704,0.35100000000000003,370,2566,2705],[2706,0.345,370,2566,2676],[2707,0.32899999999999996,370,2566,2708],[2709,0.3282,370,2566,2658],[2710,0.322,370,2566,2711],[2712,0.319,370,2566,2713],[2714,0.318,370,2566,2715],[2716,0.315,370,2566,2705],[2717,0.314,370,2566,2718],[2719,0.309,370,2566,2672],[353,0.3039,15,2566,1609],[2191,0.30260000000000004,15,2566,2720],[2721,0.302,370,2566,2722],[2723,0.302,370,2566,2724],[2725,0.301,370,2566,2713],[2726,0.299,370,2566,2727],[2728,0.29,370,2566,2676],[2729,0.28,370,2566,2730],[2731,0.276,370,2566,2732],[2733,0.2713,1,2566,2734],[2735,0.26899999999999996,370,2566,2736],[2737,0.26899999999999996,195,2566,2738],[2739,0.267,370,2566,2740],[2741,0.265,370,2566,2742],[2743,0.264,370,2566,2744],[2745,0.261,370,2566,2746],[2747,0.259,370,2566,2748],[2749,0.259,370,2566,2713],[2750,0.259,195,2566,2751],[2752,0.253,370,2566,2753],[2754,0.251,195,2566,-1],[2509,0.25,370,2566,2755],[2756,0.247,370,2566,2757],[2758,0.244,370,2566,2730],[2759,0.2398,370,2566,2760],[2761,0.23600000000000002,370,2566,2727],[870,0.23399999999999999,370,2566,2762],[2763,0.231,370,2566,2764],[2765,0.22760000000000002,15,2566,2760],[2766,0.217,370,2566,2767],[2768,0.217,370,2566,2746],[864,0.217,370,2566,2769],[2770,0.214,21,2566,2633],[2771,0.212,195,2566,-1],[2772,0.207,370,2566,2773],[2774,0.19699999999999998,370,2566,2775],[2776,0.195,370,2566,2777],[2778,0.1924,83,2566,2779],[2780,0.188,370,2566,2781],[2505,0.18780000000000002,370,2566,2713],[2782,0.18600000000000003,370,2566,2783],[2784,0.179,370,2566,2785],[2786,0.177,370,2566,2713],[2787,0.177,195,2566,2738],[2030,0.1766,83,2566,2788],[2789,0.1697,370,2566,2790],[2485,0.163,195,2566,2751],[2524,0.15689999999999998,370,2566,2791],[889,0.1559,195,2566,2792],[2793,0.147,370,2566,2794],[2795,0.1454,1820,2566,2796],[2797,0.14400000000000002,370,2566,2798],[2799,0.131,15,2566,2760],[2800,0.124,195,2566,2751],[2801,0.1188,370,2566,2802],[2803,0.1112,370,2566,2804],[2531,0.1072,195,2566,2792],[2805,0.10529999999999999,370,2566,2802],[2539,0.09699999999999999,370,2566,2806],[2807,0.0852,370,2566,2804]]},{"id":"charxiv-reasoning","name":"CharXiv (reasoning)","group":"视觉与视频","unit":"%","metric":"Accuracy","protocol":{"en":"Validation split; reasoning accuracy. Descriptive and reasoning questions are separate cohorts; human/random baselines are excluded.","zh":"验证集的推理准确率；两类问题分开统计，不含人类和随机基线。"},"source":{"name":"CharXiv","url":"https://charxiv.github.io/","status":"current_snapshot","published_at":"2024-06-26"},"results":[[1457,0.7859999999999999,370,2808,-1],[2809,0.72,370,2808,-1],[2810,0.642,1,2808,-1],[788,0.602,1,2808,-1],[2811,0.568,15,2808,-1],[2812,0.5670000000000001,15,2808,-1],[2813,0.5539999999999999,15,2808,-1],[2814,0.551,370,2808,-1],[2815,0.544,370,2808,-1],[2816,0.526,15,2808,-1],[2817,0.505,15,2808,-1],[2818,0.49700000000000005,195,2808,-1],[2819,0.47100000000000003,15,2808,-1],[2820,0.46399999999999997,370,2808,-1],[2821,0.46,370,2808,-1],[343,0.433,83,2808,-1],[2822,0.431,370,2808,-1],[2823,0.43,195,2808,-1],[2824,0.425,195,2808,-1],[2825,0.424,370,2808,-1],[2826,0.424,370,2808,-1],[2827,0.424,370,2808,-1],[2828,0.405,15,2808,-1],[2829,0.39799999999999996,370,2808,-1],[2830,0.389,370,2808,-1],[2831,0.38,370,2808,-1],[2832,0.376,370,2808,-1],[2833,0.375,66,2808,-1],[872,0.371,15,2808,-1],[2834,0.368,370,2808,-1],[2835,0.365,370,2808,-1],[2836,0.364,370,2808,-1],[2837,0.359,370,2808,-1],[2838,0.34600000000000003,195,2808,-1],[2839,0.341,15,2808,-1],[830,0.33899999999999997,83,2808,-1],[2840,0.33399999999999996,370,2808,-1],[2841,0.32899999999999996,370,2808,-1],[2842,0.327,554,2808,-1],[2522,0.322,1,2808,-1],[2526,0.318,1,2808,-1],[2843,0.316,554,2808,-1],[2844,0.313,195,2808,-1],[2845,0.312,66,2808,-1],[2846,0.31,370,2808,-1],[2520,0.302,1,2808,-1],[2847,0.292,370,2808,-1],[2848,0.29100000000000004,551,2808,-1],[2849,0.289,370,2808,-1],[2850,0.284,370,2808,-1],[2851,0.28300000000000003,370,2808,-1],[2852,0.282,370,2808,-1],[2853,0.273,370,2808,-1],[2854,0.271,370,2808,-1],[2855,0.266,370,2808,-1],[2856,0.264,370,2808,-1],[2857,0.25,1820,2808,-1],[2858,0.25,370,2808,-1],[2859,0.249,370,2808,-1],[2860,0.249,370,2808,-1],[2861,0.247,195,2808,-1],[2862,0.245,195,2808,-1],[2863,0.243,370,2808,-1],[2864,0.24,370,2808,-1],[2865,0.23600000000000002,370,2808,-1],[2866,0.235,370,2808,-1],[2527,0.228,83,2808,-1],[2867,0.225,370,2808,-1],[2868,0.213,370,2808,-1],[2869,0.21,370,2808,-1],[2870,0.19,370,2808,-1],[2871,0.19,370,2808,-1],[2872,0.188,370,2808,-1],[2873,0.187,1820,2808,-1],[2874,0.185,370,2808,-1],[2875,0.182,370,2808,-1],[2876,0.18,370,2808,-1],[2877,0.17800000000000002,370,2808,-1],[2878,0.175,370,2808,-1],[2879,0.171,25,2808,-1],[2880,0.17,370,2808,-1],[2881,0.161,370,2808,-1],[2882,0.16,195,2808,-1],[2883,0.156,370,2808,-1],[2884,0.14300000000000002,370,2808,-1],[2885,0.142,370,2808,-1],[2886,0.139,370,2808,-1],[2887,0.125,370,2808,-1],[2888,0.11699999999999999,370,2808,-1],[2889,0.11699999999999999,370,2808,-1],[2890,0.091,370,2808,-1],[2891,0.08800000000000001,370,2808,-1],[2892,0.083,370,2808,-1],[2893,0.057,370,2808,-1],[2894,0.039,370,2808,-1]]},{"id":"charxiv-descriptive","name":"CharXiv (descriptive)","group":"视觉与视频","unit":"%","metric":"Accuracy","protocol":{"en":"Validation split; descriptive accuracy. Descriptive and reasoning questions are separate cohorts; human/random baselines are excluded.","zh":"验证集的描述准确率；两类问题分开统计，不含人类和随机基线。"},"source":{"name":"CharXiv","url":"https://charxiv.github.io/","status":"current_snapshot","published_at":"2024-06-26"},"results":[[1457,0.95,370,2895,-1],[2809,0.943,370,2895,-1],[2813,0.9,15,2895,-1],[2814,0.889,370,2895,-1],[2811,0.884,15,2895,-1],[2812,0.879,15,2895,-1],[2816,0.8745,15,2895,-1],[2818,0.8740000000000001,195,2895,-1],[2820,0.872,370,2895,-1],[2821,0.851,370,2895,-1],[2819,0.8445,15,2895,-1],[788,0.843,1,2895,-1],[2815,0.843,370,2895,-1],[2826,0.823,370,2895,-1],[2822,0.8220000000000001,370,2895,-1],[2823,0.8134999999999999,195,2895,-1],[872,0.7992,15,2895,-1],[2825,0.7959999999999999,370,2895,-1],[2829,0.7683,370,2895,-1],[2835,0.7545000000000001,370,2895,-1],[2830,0.7517,370,2895,-1],[2839,0.7492,15,2895,-1],[2828,0.7390000000000001,15,2895,-1],[2824,0.7390000000000001,195,2895,-1],[2522,0.7365,1,2895,-1],[2832,0.736,370,2895,-1],[2837,0.735,370,2895,-1],[2831,0.725,370,2895,-1],[343,0.7197,83,2895,-1],[2836,0.7182999999999999,370,2895,-1],[2520,0.7155,1,2895,-1],[2841,0.6859999999999999,370,2895,-1],[2833,0.6842,66,2895,-1],[2827,0.6812,370,2895,-1],[2526,0.6507999999999999,1,2895,-1],[2834,0.6392,370,2895,-1],[2850,0.626,370,2895,-1],[2840,0.624,370,2895,-1],[2859,0.617,370,2895,-1],[2854,0.6162,370,2895,-1],[2843,0.6048,554,2895,-1],[2852,0.6015,370,2895,-1],[2853,0.5972999999999999,370,2895,-1],[2845,0.5935,66,2895,-1],[2860,0.5927,370,2895,-1],[2844,0.586,195,2895,-1],[2847,0.585,370,2895,-1],[2838,0.58,195,2895,-1],[2848,0.5761999999999999,551,2895,-1],[2846,0.5705,370,2895,-1],[2855,0.5645,370,2895,-1],[2849,0.556,370,2895,-1],[2842,0.5502,554,2895,-1],[2851,0.547,370,2895,-1],[2857,0.5465,1820,2895,-1],[2527,0.5437,83,2895,-1],[2856,0.5273,370,2895,-1],[2858,0.5268,370,2895,-1],[2867,0.5105,370,2895,-1],[2868,0.49700000000000005,370,2895,-1],[2865,0.48729999999999996,370,2895,-1],[2869,0.47100000000000003,370,2895,-1],[2879,0.45799999999999996,25,2895,-1],[2863,0.4567,370,2895,-1],[2862,0.44549999999999995,195,2895,-1],[2871,0.44420000000000004,370,2895,-1],[2877,0.4155,370,2895,-1],[2861,0.41479999999999995,195,2895,-1],[2876,0.408,370,2895,-1],[2873,0.3875,1820,2895,-1],[2864,0.38670000000000004,370,2895,-1],[2870,0.384,370,2895,-1],[2880,0.374,370,2895,-1],[2872,0.363,370,2895,-1],[2874,0.3577,370,2895,-1],[2886,0.354,370,2895,-1],[2866,0.33649999999999997,370,2895,-1],[2875,0.32770000000000005,370,2895,-1],[2883,0.32770000000000005,370,2895,-1],[2881,0.3025,370,2895,-1],[2882,0.2893,195,2895,-1],[2878,0.287,370,2895,-1],[2890,0.257,370,2895,-1],[2891,0.214,370,2895,-1],[2887,0.213,370,2895,-1],[2893,0.1932,370,2895,-1],[2885,0.1923,370,2895,-1],[2884,0.1898,370,2895,-1],[2888,0.1693,370,2895,-1],[2892,0.16149999999999998,370,2895,-1],[2889,0.1547,370,2895,-1],[2894,0.1245,370,2895,-1]]},{"id":"ocrbench-v2-en","name":"OCRBench v2 (English)","group":"视觉与视频","unit":"score","metric":"Average task score","protocol":{"en":"Official language-specific task average on a 0–100 scale. Heterogeneous task metrics are retained as a composite score, outside accuracy saturation groups.","zh":"对应语言各任务的官方平均分，保留 0–100 综合分数；各任务计分不同，不参与准确率饱和分组。"},"source":{"name":"OCRBench v2","url":"https://huggingface.co/spaces/ling99/OCRBench-v2-leaderboard","status":"current_snapshot","published_at":"2024-12-31"},"results":[[2896,60.2,66,2897,-1],[2898,52.6,370,2897,-1],[2093,51.9,83,2897,-1],[2485,51.4,195,2897,-1],[2763,49.4,370,2897,-1],[2899,49.0,370,2897,-1],[872,47.7,15,2897,-1],[2900,47.7,370,2897,-1],[2901,46.8,370,2897,-1],[2754,46.7,195,2897,-1],[2902,46.7,370,2897,-1],[353,46.5,15,2897,-1],[2903,45.2,1,2897,-1],[2904,45.1,370,2897,-1],[2774,45.0,370,2897,-1],[2905,44.8,370,2897,-1],[2906,43.3,25,2897,-1],[354,43.0,15,2897,-1],[2907,42.9,551,2897,-1],[2908,42.6,551,2897,-1],[2909,41.1,21,2897,-1],[2910,40.8,370,2897,-1],[2911,40.3,370,2897,-1],[2515,40.3,370,2897,-1],[2912,39.6,370,2897,-1],[2913,36.4,370,2897,-1],[2914,35.2,554,2897,-1],[2915,34.7,370,2897,-1],[2916,34.5,370,2897,-1],[2917,32.2,370,2897,-1],[2918,31.5,370,2897,-1],[2919,27.9,370,2897,-1],[2920,27.5,370,2897,-1],[2921,27.2,25,2897,-1],[874,25.1,370,2897,-1],[2894,23.9,370,2897,-1],[2922,23.6,370,2897,-1],[2923,23.4,195,2897,-1],[2924,23.3,195,2897,-1],[2925,23.1,370,2897,-1],[2926,21.8,370,2897,-1],[2927,19.4,370,2897,-1],[2928,18.8,370,2897,-1],[2929,18.7,370,2897,-1],[2930,17.3,527,2897,-1],[2931,16.0,370,2897,-1],[2560,15.3,370,2897,-1],[2884,13.7,370,2897,-1]]},{"id":"ocrbench-v2-cn","name":"OCRBench v2 (Chinese)","group":"视觉与视频","unit":"score","metric":"Average task score","protocol":{"en":"Official language-specific task average on a 0–100 scale. Heterogeneous task metrics are retained as a composite score, outside accuracy saturation groups.","zh":"对应语言各任务的官方平均分，保留 0–100 综合分数；各任务计分不同，不参与准确率饱和分组。"},"source":{"name":"OCRBench v2","url":"https://huggingface.co/spaces/ling99/OCRBench-v2-leaderboard","status":"current_snapshot","published_at":"2024-12-31"},"results":[[2697,55.7,370,2897,-1],[2754,55.6,195,2897,-1],[2728,53.5,370,2897,-1],[2900,49.2,370,2897,-1],[2774,46.2,370,2897,-1],[2932,45.5,21,2897,-1],[2763,44.2,370,2897,-1],[2093,43.1,83,2897,-1],[2485,42.9,195,2897,-1],[2906,42.7,25,2897,-1],[2902,42.6,370,2897,-1],[2904,41.1,370,2897,-1],[872,40.1,15,2897,-1],[2903,39.6,1,2897,-1],[2907,39.4,551,2897,-1],[2896,37.9,66,2897,-1],[2901,37.7,370,2897,-1],[2908,36.6,551,2897,-1],[2912,36.3,370,2897,-1],[2905,34.5,370,2897,-1],[2911,33.9,370,2897,-1],[353,32.2,15,2897,-1],[2914,30.7,554,2897,-1],[354,25.0,15,2897,-1],[2910,21.1,370,2897,-1],[2913,17.8,370,2897,-1],[2894,15.8,370,2897,-1],[2919,15.5,370,2897,-1],[2515,14.6,370,2897,-1],[874,13.3,370,2897,-1],[2917,13.1,370,2897,-1],[2925,13.1,370,2897,-1],[2916,12.8,370,2897,-1],[2921,12.2,25,2897,-1],[2923,11.8,195,2897,-1],[2920,11.1,370,2897,-1],[2915,9.9,370,2897,-1],[2930,9.4,527,2897,-1],[2924,9.2,195,2897,-1],[2918,9.1,370,2897,-1],[2929,8.6,370,2897,-1],[2933,8.2,370,2897,-1],[2927,7.3,370,2897,-1],[2926,5.9,370,2897,-1],[2931,5.6,370,2897,-1],[2884,5.5,370,2897,-1],[2928,3.9,370,2897,-1],[2560,3.1,370,2897,-1]]},{"id":"simpleqa-verified-epoch","name":"SimpleQA Verified (Epoch)","group":"知识与推理","unit":"%","metric":"Mean accuracy","protocol":{"en":"Epoch primary evaluations using the anti-abstention prompt introduced on 2026-08-27. Earlier prompt runs and best-of-scorer scores are excluded.","zh":"Epoch 直接评测，使用 2026-08-27 起的防弃答提示；不混入旧提示或挑选最佳评分器的成绩。"},"source":{"name":"Epoch AI","url":"https://epoch.ai/benchmarks/simple-qa-verified","status":"current_snapshot","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"published_at":"2025-09-09"},"results":[[380,0.76,4,2934,-1],[369,0.756,15,2934,-1],[371,0.739,15,2934,-1],[372,0.722,1,2934,-1],[374,0.708,1,2934,-1],[403,0.697,4,2934,-1],[397,0.692,4,2934,-1],[2935,0.668,4,2934,-1],[412,0.662,4,2934,-1],[415,0.662,4,2934,-1],[383,0.63,15,2934,-1],[375,0.607,15,2934,-1],[2936,0.6030150753768844,173,2934,-1],[2937,0.5778894472361809,173,2934,-1],[373,0.573,15,2934,-1],[400,0.566,173,2934,-1],[426,0.56,143,2934,-1],[411,0.5576730190571715,195,2934,-1],[387,0.53,1,2934,-1],[410,0.5290581162324649,25,2934,-1],[2938,0.5200400801603207,195,2934,-1],[2939,0.517,1,2934,-1],[395,0.506,396,2934,-1],[423,0.501,15,2934,-1],[447,0.494,15,2934,-1],[2940,0.493,143,2934,-1],[407,0.489,143,2934,-1],[469,0.4874874874874875,195,2934,-1],[419,0.483,143,2934,-1],[2941,0.48,15,2934,-1],[409,0.473,195,2934,-1],[406,0.47,1,2934,-1],[431,0.46993987975951906,25,2934,-1],[377,0.465,1,2934,-1],[384,0.463,15,2934,-1],[391,0.458,195,2934,-1],[443,0.457,1,2934,-1],[389,0.451,15,2934,-1],[438,0.44144144144144143,195,2934,-1],[405,0.429,25,2934,-1],[398,0.425,25,2934,-1],[388,0.414,15,2934,-1],[478,0.411,15,2934,-1],[401,0.41,402,2934,-1],[767,0.4044044044044044,195,2934,-1],[446,0.403,199,2934,-1],[404,0.371,15,2934,-1],[425,0.365,396,2934,-1],[2942,0.343,15,2934,-1],[2943,0.343,396,2934,-1],[414,0.342,402,2934,-1],[440,0.34,402,2934,-1],[408,0.337,1,2934,-1],[418,0.33633633633633636,25,2934,-1],[434,0.332,143,2934,-1],[2944,0.329,1,2934,-1],[2945,0.328,1,2934,-1],[2946,0.328,15,2934,-1],[2947,0.327,15,2934,-1],[2948,0.322,402,2934,-1],[485,0.311,15,2934,-1],[2949,0.307,1,2934,-1],[433,0.302,143,2934,-1],[2950,0.294,15,2934,-1],[493,0.26,15,2934,-1],[2951,0.2535070140280561,195,2934,-1],[390,0.238,1,2934,-1],[482,0.2032032032032032,195,2934,-1],[416,0.2,417,2934,-1],[477,0.196,15,2934,-1],[430,0.191,199,2934,-1],[441,0.188,15,2934,-1],[476,0.15915915915915915,195,2934,-1],[470,0.153,15,2934,-1],[484,0.127,15,2934,-1],[2952,0.126,1,2934,-1],[2953,0.126,1,2934,-1],[432,0.117,15,2934,-1],[2954,0.104,4,2934,-1],[492,0.083,15,2934,-1],[2955,0.06,15,2934,-1]]},{"id":"livebench-2-20261025","name":"LiveBench 2.0 (2026-10-25)","group":"综合","unit":"score","metric":"Weighted overall","protocol":{"en":"2026-10-25 release. Overall is reproduced from the official category weights: Long Agentic 2, Coding and Design 1.5, others 1. Composite score, not accuracy.","zh":"2026-10-25 批次。按官方权重汇总：Long Agentic 为 2，Coding、Design 为 1.5，其余为 1。综合分数不等同于准确率。"},"source":{"name":"LiveBench","url":"https://livebench.ai/","status":"current_snapshot","published_at":"2026-10-25"},"results":[[2956,85.69444444444444,1,2957,-1],[2958,85.63888888888889,15,2957,-1],[2959,85.14444444444445,1,2957,-1],[2960,84.07222222222222,15,2957,-1],[2961,80.03888888888889,1,2957,-1],[2962,79.30000000000001,83,2963,-1],[2964,77.78888888888889,15,2957,-1],[2965,74.15,66,2957,-1],[2966,72.86666666666666,25,2957,-1],[2967,71.28888888888889,21,2957,-1],[2968,70.13888888888889,143,2969,-1],[2970,69.85,1,2957,-1],[2971,68.47777777777777,2972,2963,-1],[2973,67.78888888888889,15,2957,-1],[2974,67.75,143,2969,-1],[2975,66.89444444444445,775,2957,-1],[2976,64.58888888888889,15,2957,-1],[2977,64.57222222222222,21,2957,-1],[2978,63.49444444444445,417,2963,-1],[2979,63.03333333333333,66,2957,-1],[2980,62.06666666666667,2981,2957,-1],[2982,61.5611111111111,195,2957,-1],[2983,59.205555555555556,83,2963,-1],[2984,58.60555555555555,2985,2957,-1],[2986,58.38333333333334,195,2957,-1],[2987,51.62777777777777,66,2957,-1],[2988,50.30555555555556,2989,2957,-1],[2990,50.227777777777774,307,2957,-1],[203,47.05,199,2957,-1],[2991,46.5,307,2957,-1],[2992,43.900000000000006,417,2963,-1],[2585,38.72222222222222,83,2957,-1]]},{"id":"ifeval-opencompass","name":"IFEval (OpenCompass)","group":"指令与长文","unit":"%","metric":"Accuracy","protocol":{"en":"Current OpenCompass academic release (2026-03-09); strict prompt-level accuracy.","zh":"OpenCompass 官方直接评测。2026-03-09 批次，严格提示级准确率。"},"source":{"name":"OpenCompass","url":"https://rank.opencompass.org.cn/leaderboard-llm-academic","status":"current_snapshot","published_at":"2023-11-14"},"results":[[2993,0.946,15,2994,-1],[2995,0.946,826,2994,-1],[2996,0.943,2997,2994,-1],[2998,0.9390000000000001,396,2994,-1],[268,0.932,633,2994,-1],[2999,0.932,2985,2994,-1],[3000,0.93,15,2994,-1],[3001,0.9279999999999999,15,2994,-1],[3002,0.9279999999999999,83,2994,-1],[2288,0.924,396,2994,-1],[3003,0.922,15,2994,-1],[3004,0.917,25,2994,-1],[2576,0.915,50,2994,-1],[3005,0.9109999999999999,775,2994,-1],[2278,0.902,633,2994,-1],[3006,0.902,15,2994,-1],[3007,0.902,876,2994,-1],[3008,0.902,775,2994,-1],[1512,0.902,15,2994,-1],[764,0.9,83,2994,-1],[3009,0.897,25,2994,-1],[785,0.897,195,2994,-1],[3010,0.895,2972,2994,-1],[3011,0.895,195,2994,-1],[3012,0.889,3013,2994,-1],[3014,0.889,15,2994,-1],[3015,0.887,1188,2994,-1],[3016,0.887,633,2994,-1],[791,0.885,396,2994,-1],[3017,0.885,195,2994,-1],[3018,0.883,1,2994,-1],[771,0.883,195,2994,-1],[3019,0.882,15,2994,-1],[767,0.878,195,2994,-1],[3020,0.8759999999999999,195,2994,-1],[3021,0.871,66,2994,-1],[3022,0.86,195,2994,-1],[3023,0.856,195,2994,-1],[782,0.8540000000000001,633,2994,-1],[3024,0.8390000000000001,195,2994,-1],[3025,0.8370000000000001,3026,2994,-1],[3027,0.8240000000000001,195,2994,-1],[3028,0.812,3013,2994,-1],[3029,0.81,83,2994,-1],[773,0.8,25,2994,-1],[3030,0.789,3026,2994,-1],[3031,0.769,865,2994,-1],[3032,0.603,3033,2994,-1]]},{"id":"cmmlu-5shot","name":"CMMLU (5-shot)","group":"知识与推理","unit":"%","metric":"Average accuracy","protocol":{"en":"Author-maintained README, 5-shot average across subjects. Latest published entries are retained; this table is not a current-model census.","zh":"作者维护的 README，使用 5-shot 各科平均分；保留目前公开记录，该表不代表当前所有模型。"},"source":{"name":"CMMLU authors","url":"https://github.com/haonan-li/CMMLU","status":"current_snapshot","published_at":"2023-06-15"},"results":[[3034,0.9026000000000001,370,3035,-1],[3036,0.9016,370,3035,-1],[3037,0.9007,370,3035,-1],[3038,0.8965000000000001,195,3035,-1],[3039,0.8859,370,3035,-1],[547,0.8832,195,3035,-1],[3040,0.8739,370,3035,-1],[792,0.8567,195,3035,-1],[3041,0.8559,370,3035,-1],[3042,0.8371999999999999,370,3035,-1],[3043,0.8354,195,3035,-1],[3044,0.8284,370,3035,-1],[3045,0.8245999999999999,370,3035,-1],[3046,0.8225,195,3035,-1],[3047,0.8173,370,3035,-1],[3048,0.8027,370,3035,-1],[3049,0.7995,370,3035,-1],[3050,0.7426999999999999,370,3035,-1],[3051,0.7403,370,3035,-1],[3052,0.725,195,3035,-1],[3053,0.7112,370,3035,-1],[3054,0.711,370,3035,-1],[1396,0.7095,15,3035,-1],[3055,0.665,370,3035,-1],[3056,0.6640999999999999,370,3035,-1],[524,0.6437999999999999,66,3035,-1],[3057,0.6173,370,3035,-1],[3058,0.6004999999999999,370,3035,-1],[3059,0.5866,195,3035,-1],[3060,0.5717,370,3035,-1],[3061,0.5716,370,3035,-1],[3062,0.5582,3063,3035,-1],[2299,0.5550999999999999,15,3035,-1],[3064,0.488,551,3035,-1],[3065,0.4539,370,3035,-1],[3066,0.4443,3063,3035,-1],[3067,0.41450000000000004,370,3035,-1],[3068,0.39799999999999996,66,3035,-1],[3069,0.37479999999999997,551,3035,-1],[3070,0.3716,370,3035,-1],[3071,0.3704,370,3035,-1],[536,0.3685,66,3035,-1],[3072,0.32630000000000003,370,3035,-1],[3073,0.3188,370,3035,-1],[3074,0.2957,370,3035,-1]]},{"id":"gaia2-cli","name":"Gaia2-CLI","group":"Agent 与工具","unit":"%","metric":"pass@1","protocol":{"en":"Current CLI track, OpenClaw 2026.4.1. Scores are reported by submitting teams. Not interchangeable with GAIA 2023 or the original Gaia2 track.","zh":"当前 CLI 赛道，OpenClaw 2026.4.1。成绩由提交团队报告，不与 GAIA 2023 或原始 Gaia2 赛道混用。"},"source":{"name":"Meta / Gaia2","url":"https://huggingface.co/spaces/meta-agents-research-environments/leaderboard","status":"current_snapshot","published_at":"2025-09-22"},"results":[[3075,0.57,1,3076,-1],[3077,0.564,15,3076,-1],[3078,0.556,15,3076,-1],[3079,0.52,83,3076,-1],[3080,0.519,1,3076,-1],[3081,0.505,551,3082,-1],[3083,0.34,21,3082,-1]]},{"id":"arc-agi-3","name":"ARC-AGI-3","group":"Agent 与工具","unit":"score","metric":"Human-normalized efficiency score","protocol":{"en":"Semi-private interactive evaluation. Human-normalized action efficiency on a 0–100 scale, not task accuracy; excluded from accuracy saturation groups.","zh":"半私有交互评测，保留 0–100 的人类归一化动作效率分数；不等同于任务准确率，不参加准确率饱和分组。"},"source":{"name":"ARC Prize","url":"https://arcprize.org/leaderboard","status":"current_snapshot","published_at":"2026-03-24"},"results":[[3084,99.94571232125699,15,71,-1],[3085,98.55074288592294,15,71,-1],[3086,98.44155844155843,15,71,-1],[3087,98.43861309255745,15,71,-1],[3088,98.02838165829648,15,71,-1],[3089,97.4591718377776,83,71,-1],[3090,97.3448904168153,83,71,-1],[3091,97.30432688837197,83,71,-1],[3092,96.72021831785531,15,71,-1],[3093,96.37493342875182,15,71,-1],[3094,96.17739071933946,15,71,-1],[3095,94.99362560053225,15,71,-1],[3096,91.02346102095301,15,71,-1],[3097,82.7870673049169,15,71,-1],[3098,72.44170308802806,1,71,-1],[70,62.712802100606275,15,71,-1],[74,59.342954091460555,15,71,-1],[77,54.8190516361223,15,71,-1],[72,52.72904006820879,15,71,-1],[196,49.79401633943577,83,71,-1],[96,46.265668402958575,83,71,-1],[81,39.92932040733721,15,71,-1],[78,38.58757449733784,15,71,-1],[3099,35.47664601535195,1,71,-1],[165,35.177880317424,15,71,-1],[3100,35.00424335075243,83,71,-1],[93,30.159999999999997,1,71,-1],[80,26.717284844257062,15,71,-1],[107,24.31040800668143,1,71,-1],[3101,24.202718056957025,83,71,-1],[3102,23.028056734101494,15,71,-1],[100,17.452190401133805,15,71,-1],[3103,15.071935405559646,83,71,-1],[97,10.568764673656194,15,71,-1],[90,10.36627922392312,83,71,-1],[3104,10.047981644646995,143,71,-1],[3105,9.453334058864572,15,71,-1],[3106,8.373637088154913,143,71,-1],[76,7.779999999999999,15,71,-1],[3107,7.016486895838931,143,71,-1],[87,6.99,15,71,-1],[115,5.989560493697216,83,71,-1],[3108,5.855663760579018,15,71,-1],[3109,4.930246053628116,15,71,-1],[3110,4.887554909249332,1,71,-1],[88,4.623568186131146,15,71,-1],[110,3.9878093830617267,83,71,-1],[120,3.9243846587314173,15,71,-1],[3111,3.264386796625269,25,71,-1],[3112,3.1518486978853844,25,71,-1],[3113,2.819365772805844,1,71,-1],[3114,2.6843916288028566,25,71,-1],[99,2.15,15,71,-1],[142,2.11,143,71,-1],[155,1.8162526536637038,143,71,-1],[114,1.7976269602783517,15,71,-1],[3115,1.7619154311646181,15,71,-1],[169,1.7332202934667684,143,71,-1],[168,1.7056643767835524,143,71,-1],[128,1.52,1,71,-1],[3116,1.2059164341155284,15,71,-1],[141,1.203076515291301,1,71,-1],[144,1.0699999999999998,15,71,-1],[134,0.8994389911152361,15,71,-1],[125,0.8516646657220222,25,71,-1],[106,0.8,15,71,-1],[3117,0.7739582026314942,143,71,-1],[122,0.6499999999999999,15,71,-1],[3118,0.6244567651378714,1,71,-1],[3119,0.5868386891554969,15,71,-1],[3120,0.51,1,71,-1],[3121,0.5072886313635385,15,71,-1],[145,0.49,15,71,-1],[139,0.4846664966204641,25,71,-1],[3122,0.4830935313474596,25,71,-1],[3123,0.4513711877557703,1,71,-1],[109,0.43,15,71,-1],[116,0.42,83,71,-1],[171,0.418612306617149,15,71,-1],[3124,0.41798234834263187,173,71,-1],[222,0.4047513036164488,1,71,-1],[209,0.3961083341784243,1,71,-1],[3125,0.37847731587690087,15,71,-1],[159,0.34514247315937174,25,71,-1],[327,0.3431694493873915,25,71,-1],[193,0.33,15,71,-1],[184,0.32,143,71,-1],[3126,0.31612009238177263,15,71,-1],[183,0.3,143,71,-1],[316,0.2837742350643006,15,71,-1],[207,0.26,143,71,-1],[234,0.24696281908456535,143,71,-1],[3127,0.2374807559887363,15,71,-1],[140,0.21,15,71,-1],[232,0.19415284639354707,15,71,-1],[211,0.18024216987988248,15,71,-1],[166,0.18,15,71,-1],[3128,0.18,1,71,-1],[266,0.16999999999999998,15,71,-1],[254,0.16999999999999998,15,71,-1],[197,0.16413333960356602,15,71,-1],[172,0.13621973920554634,173,71,-1],[210,0.12649234250931635,15,71,-1],[167,0.10360997760319558,15,71,-1],[216,0.1,15,71,-1],[3129,0.09,143,71,-1],[202,0.08,15,71,-1],[3130,0.04644044435594966,15,71,-1],[273,0.03291205821227175,15,71,-1],[358,0.027919455042839488,15,71,-1],[187,0.02,15,71,-1],[230,0.01,15,71,-1]]},{"id":"mmlu-deepseek-v4-base","name":"MMLU (DeepSeek base, 5-shot)","group":"知识与推理","unit":"%","metric":"Exact match","protocol":{"en":"Base models; 5-shot exact match.","zh":"Base 模型；5-shot 精确匹配。"},"source":{"name":"DeepSeek model report","url":"https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro","status":"current_snapshot","published_at":"2026-10-15"},"results":[[3131,0.9009999999999999,25,3132,3133],[3134,0.887,25,3132,3133],[3135,0.878,25,3132,3133]]},{"id":"ceval-deepseek-v4-base","name":"C-Eval (DeepSeek base, 5-shot)","group":"知识与推理","unit":"%","metric":"Exact match","protocol":{"en":"Base models; 5-shot exact match.","zh":"Base 模型；5-shot 精确匹配。"},"source":{"name":"DeepSeek model report","url":"https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro","status":"current_snapshot","published_at":"2026-10-15"},"results":[[3131,0.9309999999999999,25,3132,3133],[3134,0.9209999999999999,25,3132,3133],[3135,0.904,25,3132,3133]]},{"id":"gsm8k-deepseek-v4-base","name":"GSM8K (DeepSeek base, 8-shot)","group":"数学与科学","unit":"%","metric":"Exact match","protocol":{"en":"Base models; 8-shot exact match.","zh":"Base 模型；8-shot 精确匹配。"},"source":{"name":"DeepSeek model report","url":"https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro","status":"current_snapshot","published_at":"2026-10-15"},"results":[[3131,0.9259999999999999,25,3136,3133],[3135,0.9109999999999999,25,3136,3133],[3134,0.9079999999999999,25,3136,3133]]},{"id":"browsecomp-deepseek-v4","name":"BrowseComp (DeepSeek report)","group":"Agent 与工具","unit":"%","metric":"Pass@1","protocol":{"en":"DeepSeek V4 model-report comparison, pass@1. Model and reasoning labels follow the report.","zh":"DeepSeek V4 模型报告中的对比成绩，使用 pass@1；模型和推理模式名称沿用报告。"},"source":{"name":"DeepSeek model report","url":"https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro","status":"current_snapshot","published_at":"2026-10-15"},"results":[[3137,0.8590000000000001,83,3138,3133],[3139,0.8370000000000001,1,3140,3133],[3141,0.8340000000000001,25,3140,3133],[3142,0.8320000000000001,21,3143,3133],[3144,0.8270000000000001,15,3145,3133],[3146,0.7929999999999999,551,3143,3133]]},{"id":"chinese-simpleqa-deepseek-v4","name":"Chinese SimpleQA (DeepSeek report)","group":"知识与推理","unit":"%","metric":"Pass@1","protocol":{"en":"DeepSeek V4 model-report comparison, pass@1. Model and reasoning labels follow the report.","zh":"DeepSeek V4 模型报告中的对比成绩，使用 pass@1；模型和推理模式名称沿用报告。"},"source":{"name":"DeepSeek model report","url":"https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro","status":"current_snapshot","published_at":"2026-10-15"},"results":[[3137,0.8590000000000001,83,3138,3133],[3141,0.8440000000000001,25,3140,3133],[3144,0.768,15,3145,3133],[3139,0.764,1,3140,3133],[3142,0.759,21,3143,3133],[3146,0.75,551,3143,3133]]},{"id":"paperbenchx","name":"PaperBenchX (Pass rate)","group":"数学与科学","unit":"%","metric":"Full","source":{"name":"UniPat AI","url":"https://unipat.ai/benchmarks/PaperBenchX","status":"current_snapshot","published_at":"2026-10-08"},"protocol":{"en":"Pass rate across 93 tasks: a task passes only if its rubric score is strictly above 90% after clean replay. Missing outcomes count as zero.","zh":"通过率：93 道任务中，经干净重放后评分严格高于 90% 的任务比例；缺失结果计零。"},"results":[[3147,0.1398,15,3148,-1],[3149,0.129,1,3150,-1],[3151,0.1183,1,3150,-1],[3152,0.0968,15,3148,-1],[3153,0.0753,25,3150,-1],[3154,0.0753,551,3150,-1],[3155,0.0753,21,3150,-1],[3156,0.0753,1,3150,-1],[3157,0.0538,195,3150,-1],[3158,0.0215,3033,3150,-1]]},{"id":"paperbenchx-partial","name":"PaperBenchX (Average completion)","group":"数学与科学","unit":"score","metric":"Partial","source":{"name":"UniPat AI","url":"https://unipat.ai/benchmarks/PaperBenchX","status":"current_snapshot","published_at":"2026-10-08"},"protocol":{"en":"Average completion across all 93 tasks, measured by mean scientific rubric credit. Missing outcomes count as zero.","zh":"平均完成度：全部 93 道任务的科学评分细则得分取平均；缺失结果计零。"},"score_max":100,"results":[[3147,59.08,15,3148,-1],[3149,58.17,1,3150,-1],[3151,57.96,1,3150,-1],[3152,55.07,15,3148,-1],[3156,52.52,1,3150,-1],[3155,49.24,21,3150,-1],[3157,46.76,195,3150,-1],[3153,45.47,25,3150,-1],[3154,40.78,551,3150,-1],[3158,27.53,3033,3150,-1]]},{"id":"saas-bench-v1-1-resolved","name":"SaaS-Bench v1.1 (Resolved)","group":"Agent 与工具","unit":"%","metric":"Resolved","source":{"name":"UniPat AI","url":"https://unipat.ai/benchmarks/SaaS-Bench","status":"current_snapshot","published_at":"2026-06-12"},"protocol":{"en":"106 tasks across 23 SaaS systems. End-to-end task completion rate. Each model/harness pair is a separate result.","zh":"23 个 SaaS 系统中的 106 道任务。使用端到端任务完成率。各模型与运行框架组合分别记录。"},"results":[[3156,0.311,1,3159,-1],[3160,0.292,15,3161,-1],[3162,0.27399999999999997,370,3163,-1],[2977,0.245,21,3164,-1],[3165,0.23600000000000002,195,3166,-1],[3156,0.217,1,3167,-1],[3160,0.179,15,3167,-1],[2977,0.17,21,3167,-1],[3165,0.085,195,3167,-1]]},{"id":"saas-bench-v1-1-checkpoint","name":"SaaS-Bench v1.1 (Checkpoint)","group":"Agent 与工具","unit":"score","metric":"Checkpoint","source":{"name":"UniPat AI","url":"https://unipat.ai/benchmarks/SaaS-Bench","status":"current_snapshot","published_at":"2026-06-12"},"protocol":{"en":"106 tasks across 23 SaaS systems. Partial checkpoint credit. Each model/harness pair is a separate result.","zh":"23 个 SaaS 系统中的 106 道任务。使用检查点部分得分。各模型与运行框架组合分别记录。"},"score_max":100,"results":[[3162,78.3,370,3163,-1],[3156,74.3,1,3159,-1],[3160,71.1,15,3161,-1],[3160,69.8,15,3167,-1],[3165,65.9,195,3166,-1],[3156,64.7,1,3167,-1],[2977,56.4,21,3164,-1],[2977,56.2,21,3167,-1],[3165,31.1,195,3167,-1]]},{"id":"monthly-swebench-2026-06","name":"Monthly-SWEBench (2026-06)","group":"编程","unit":"%","metric":"Pass@1","source":{"name":"UniPat AI","url":"https://unipat.ai/benchmarks/MonthlySWEBench","status":"current_snapshot","published_at":"2026-06-30"},"protocol":{"en":"100 tasks, one attempt; 2026-06 release. The publisher counts missing/unclean required runs as zero under its provisional publication policy.","zh":"2026-06 批次，100 道任务、单次尝试。官方临时发布口径将缺失或尚未完成干净重跑的必需记录计零。"},"results":[[3168,0.76,15,3169,-1],[3170,0.69,1,3169,-1],[3171,0.65,633,3169,-1],[3172,0.58,775,3169,-1],[3173,0.57,25,3169,-1],[3174,0.53,3026,3169,-1],[3175,0.51,195,3169,-1],[3176,0.47,195,3169,-1],[3177,0.45,25,3169,-1],[3178,0.38,3033,3169,-1],[3179,0.29,83,3169,-1],[3180,0.22,2985,3169,-1]]},{"id":"roadmapbench-resolved","name":"RoadmapBench (Resolved)","group":"编程","unit":"%","metric":"Resolved","source":{"name":"UniPat AI","url":"https://unipat.ai/benchmarks/RoadmapBench","status":"current_snapshot","published_at":"2026-05-15"},"protocol":{"en":"OpenHands; 115 version-upgrade tasks. Task passes only when all targets pass.","zh":"OpenHands；115 道版本升级任务。所有目标均通过时才算完成任务。"},"results":[[3181,0.391,1,3182,-1],[3183,0.322,1,3182,-1],[3184,0.29600000000000004,15,3182,-1],[502,0.209,83,3182,-1],[3185,0.183,25,3182,-1],[3186,0.183,633,3182,-1],[3187,0.14800000000000002,21,3182,-1],[3188,0.139,2972,3182,-1],[3189,0.122,195,3182,-1],[2998,0.113,21,3182,-1],[3190,0.10400000000000001,775,3182,-1],[3191,0.096,195,3182,-1],[3192,0.052000000000000005,3026,3182,3193]]},{"id":"roadmapbench-completion","name":"RoadmapBench (Completion)","group":"编程","unit":"score","metric":"Completion","source":{"name":"UniPat AI","url":"https://unipat.ai/benchmarks/RoadmapBench","status":"current_snapshot","published_at":"2026-05-15"},"protocol":{"en":"OpenHands; 115 version-upgrade tasks. Original 0–1 phase-completion score.","zh":"OpenHands；115 道版本升级任务。保留 0–1 的阶段完成分数。"},"score_max":1,"results":[[3181,0.692,1,3182,-1],[3183,0.627,1,3182,-1],[3184,0.497,15,3182,-1],[3185,0.486,25,3182,-1],[3186,0.453,633,3182,-1],[3188,0.44,2972,3182,-1],[502,0.439,83,3182,-1],[3187,0.432,21,3182,-1],[3189,0.424,195,3182,-1],[3191,0.383,195,3182,-1],[2998,0.378,21,3182,-1],[3190,0.332,775,3182,-1],[3192,0.177,3026,3182,3193]]},{"id":"evocodebench-dataset","name":"EvoCode-Bench (Dataset score)","group":"编程","unit":"score","metric":"Dataset score","source":{"name":"UniPat AI","url":"https://github.com/UniPat-AI/EvoCodeBench","status":"current_snapshot","published_at":"2026-06-25"},"protocol":{"en":"Clean 2026-06-25 release; terminus-2, one attempt, 26 tasks. Mean per-task fraction of passing rounds, scaled to 100.","zh":"2026-06-25 干净重跑批次；terminus-2、单次尝试、26 道任务。先计算每道任务的轮次通过比例，再平均并乘以 100。"},"score_max":100,"results":[[3194,59.1,1,3195,-1],[3196,29.5,15,3197,-1],[3198,23.4,775,3199,-1],[227,16.2,551,3200,-1],[3185,14.1,25,3197,-1],[3187,13.2,21,3200,-1],[3201,12.2,25,3197,-1],[3202,11.9,195,3200,-1],[3189,9.7,195,3200,-1],[3203,7.8,21,3200,-1],[3204,6.7,370,3197,-1],[3186,5.9,551,3200,-1],[3190,5.1,775,3205,-1]]},{"id":"evocodebench-case","name":"EvoCode-Bench (Case score)","group":"编程","unit":"score","metric":"Case score","source":{"name":"UniPat AI","url":"https://github.com/UniPat-AI/EvoCodeBench","status":"current_snapshot","published_at":"2026-06-25"},"protocol":{"en":"Clean 2026-06-25 release; terminus-2, one attempt, 26 tasks. Mean test-case credit across rounds and tasks, scaled to 100.","zh":"2026-06-25 干净重跑批次；terminus-2、单次尝试、26 道任务。各轮测试用例得分按任务平均，换算为 0–100 分。"},"score_max":100,"results":[[3194,96.6,1,3195,-1],[3196,81.8,15,3197,-1],[3189,67.7,195,3200,-1],[3202,67.4,195,3200,-1],[3187,65.7,21,3200,-1],[3185,61.7,25,3197,-1],[3198,61.5,775,3199,-1],[3201,58.9,25,3197,-1],[3186,52.5,551,3200,-1],[227,47.5,551,3200,-1],[3203,45.4,21,3200,-1],[3190,44.9,775,3205,-1],[3204,42.0,370,3197,-1]]},{"id":"evocodebench-perfect","name":"EvoCode-Bench (Perfect tasks)","group":"编程","unit":"%","metric":"Perfect tasks","source":{"name":"UniPat AI","url":"https://github.com/UniPat-AI/EvoCodeBench","status":"current_snapshot","published_at":"2026-06-25"},"protocol":{"en":"Clean 2026-06-25 release; terminus-2, one attempt, 26 tasks. Fraction of tasks passing every round.","zh":"2026-06-25 干净重跑批次；terminus-2、单次尝试、26 道任务。每一轮均通过的完整任务比例。"},"results":[[3194,0.34615384615384615,1,3195,-1],[3198,0.07692307692307693,775,3199,-1],[3185,0.038461538461538464,25,3197,-1],[227,0.038461538461538464,551,3200,-1],[3201,0.0,25,3197,-1],[3204,0.0,370,3197,-1],[3186,0.0,551,3200,-1],[3196,0.0,15,3197,-1],[3187,0.0,21,3200,-1],[3203,0.0,21,3200,-1],[3190,0.0,775,3205,-1],[3189,0.0,195,3200,-1],[3202,0.0,195,3200,-1]]},{"id":"clawbench-v2-lenient","name":"ClawBench V2 (Lenient)","group":"Agent 与工具","unit":"%","metric":"Lenient judged task success","source":{"name":"ClawBench / UniPat AI","url":"https://claw-bench.com/leaderboard","status":"current_snapshot","published_at":"2026-06-19"},"protocol":{"en":"V2, 130 tasks; intercepted payload plus DeepSeek V4 Pro judgment using the lenient rubric. Harnesses are retained; the published API snapshot is 2026-05-20.","zh":"V2、130 道任务；拦截提交内容后由 DeepSeek V4 Pro 按宽松量规判定。保留运行框架；官方 API 快照日期为 2026-05-20。"},"results":[[3206,0.4462,1,3207,-1],[3208,0.3538,15,3207,-1],[440,0.3462,551,3207,-1],[3209,0.3385,25,3207,-1],[3210,0.0233,25,3207,-1],[3211,0.0231,370,3207,-1],[3212,0.0154,775,3207,-1],[440,0.0,551,3213,-1],[3214,0.0,370,3207,-1]]},{"id":"clawbench-v2-strict","name":"ClawBench V2 (Strict)","group":"Agent 与工具","unit":"%","metric":"Strict judged task success","source":{"name":"ClawBench / UniPat AI","url":"https://claw-bench.com/leaderboard","status":"current_snapshot","published_at":"2026-06-19"},"protocol":{"en":"V2, 130 tasks; intercepted payload plus DeepSeek V4 Pro judgment using the strict rubric. Harnesses are retained; the published API snapshot is 2026-05-20.","zh":"V2、130 道任务；拦截提交内容后由 DeepSeek V4 Pro 按严格量规判定。保留运行框架；官方 API 快照日期为 2026-05-20。"},"results":[[3206,0.2462,1,3215,-1],[3208,0.1846,15,3215,-1],[440,0.1769,551,3215,-1],[3209,0.1231,25,3215,-1],[3211,0.0077,370,3215,-1],[3210,0.0,25,3215,-1],[440,0.0,551,3216,-1],[3212,0.0,775,3215,-1],[3214,0.0,370,3215,-1]]},{"id":"echo-overall","name":"Echo (Overall)","group":"预测与决策","unit":"elo","metric":"Elo","source":{"name":"UniPat AI","url":"https://echo.unipat.ai/leaderboard","status":"current_snapshot","published_at":"2026-04-24"},"protocol":{"en":"Overall prediction Elo from the public rankings API. Market/human baselines are excluded; this is a manual snapshot of a changing ranking.","zh":"公开榜单 API 的总体预测 Elo；不含市场／人类基线。本页保留本次手动更新的快照。"},"results":[[3217,1025.6,3218,3219,-1],[3220,1018.4,25,3219,-1],[3187,1017.4,21,3219,-1],[3183,1008.6,1,3219,-1],[502,1006.3,83,3219,-1],[3186,1005.6,633,3219,-1],[3221,1003.2,143,3219,-1],[3222,1001.8,775,3219,-1],[340,1001.4,15,3219,-1],[3223,1001.3,143,3219,-1],[3184,999.3,15,3219,-1],[3192,996.2,3026,3219,-1],[2576,986.0,936,3219,-1],[268,983.9,633,3219,-1],[3224,979.7,775,3219,-1],[3225,966.5,936,3219,-1],[3226,964.3,25,3219,-1],[2998,920.2,21,3219,-1]]},{"id":"babyvision","name":"BabyVision","group":"视觉与视频","unit":"%","metric":"Accuracy","source":{"name":"UniPat AI","url":"https://unipat.ai/benchmarks/BabyVision","status":"current_snapshot","published_at":"2026-01-10"},"protocol":{"en":"Published visual-reasoning scores retain their tool and reasoning settings. Appendix Table 6 supplies additional tool-free Qwen3VL results.","zh":"官方视觉推理成绩，保留工具与推理设置。论文附录表 6 补充了 Qwen3VL 的无工具成绩。"},"results":[[3227,0.9129999999999999,195,3228,-1],[3229,0.905,1,2408,-1],[3230,0.889,15,2408,-1],[3231,0.857,21,3228,-1],[3232,0.836,15,2408,-1],[3157,0.82,195,3228,-1],[3233,0.812,1,2408,-1],[3234,0.763,66,2408,-1],[3235,0.737,3026,2408,-1],[3236,0.7040000000000001,195,2408,-1],[3237,0.685,21,3228,-1],[3238,0.606,3026,2408,-1],[502,0.516,83,2408,-1],[3239,0.49700000000000005,15,2408,-1],[3240,0.49700000000000005,83,2408,-1],[2576,0.433,195,3228,-1],[2998,0.365,21,3228,-1],[340,0.344,15,2408,-1],[2594,0.302,3026,2408,-1],[3241,0.222,195,3228,-1],[3242,0.195,195,3243,3244],[3245,0.192,3246,3228,-1],[3247,0.192,195,2408,-1],[3248,0.17600000000000002,633,3228,-1],[3249,0.174,195,3243,3244],[3250,0.162,143,2408,-1],[3251,0.151,2972,3228,-1],[3183,0.14800000000000002,1,2408,-1],[3252,0.147,2985,3228,-1],[3253,0.146,195,3243,3244],[3254,0.142,1,2408,-1],[3255,0.131,195,3243,3244],[3256,0.124,21,3228,-1]]},{"id":"sciencevision-3d","name":"ScienceVision (3D)","group":"数学与科学","unit":"score","metric":"Mean answer score","source":{"name":"UniPat AI","url":"https://unipat.ai/blog/ScienceVision","status":"current_snapshot","published_at":"2026-10-30"},"protocol":{"en":"3D; mean answer-component score on 0–1 over 250 questions, up to three runs; ineligible trials count as zero. GPT rows use Codex, others Claude Code; source segments span different runs.","zh":"3D 源文件；250 道题、最多三次运行的回答分量平均分，保留 0–1 原单位，无效试验计零。GPT 使用 Codex，其他模型使用 Claude Code；来源拼接不同运行批次。"},"score_max":1,"results":[[2958,0.5291,15,3257,-1],[2960,0.4893,15,3257,-1],[2956,0.4619,1,3258,-1],[2959,0.4539,1,3258,-1],[2962,0.4278,83,3258,-1],[3259,0.383,195,3258,-1],[3260,0.3616,195,3258,-1],[2977,0.3609,21,3258,-1],[2980,0.2846,633,3258,-1],[3261,0.2652,25,3258,-1],[3262,0.2435,3026,3258,-1],[3263,0.1598,3033,3258,-1]]},{"id":"sciencevision-mcp","name":"ScienceVision (3D + MCP)","group":"数学与科学","unit":"score","metric":"Mean answer score","source":{"name":"UniPat AI","url":"https://unipat.ai/blog/ScienceVision","status":"current_snapshot","published_at":"2026-10-30"},"protocol":{"en":"3D + MCP; mean answer-component score on 0–1 over 250 questions, up to three runs; ineligible trials count as zero. GPT rows use Codex, others Claude Code; source segments span different runs.","zh":"3D 源文件与 MCP 检视工具；250 道题、最多三次运行的回答分量平均分，保留 0–1 原单位，无效试验计零。GPT 使用 Codex，其他模型使用 Claude Code；来源拼接不同运行批次。"},"score_max":1,"results":[[2958,0.5282,15,3264,-1],[2960,0.511,15,3264,-1],[2956,0.4631,1,3265,-1],[2959,0.4343,1,3265,-1],[2962,0.4321,83,3265,-1],[3259,0.3736,195,3265,-1],[3260,0.3681,195,3265,-1],[2977,0.3369,21,3265,-1],[2980,0.2858,633,3265,-1],[3261,0.2678,25,3265,-1],[3262,0.2527,3026,3265,-1],[3263,0.1738,3033,3265,-1]]},{"id":"sciencevision-2d-6","name":"ScienceVision (2D, six views)","group":"数学与科学","unit":"score","metric":"Mean answer score","source":{"name":"UniPat AI","url":"https://unipat.ai/blog/ScienceVision","status":"current_snapshot","published_at":"2026-10-30"},"protocol":{"en":"2D, six views; mean answer-component score on 0–1 over 250 questions, up to three runs; ineligible trials count as zero. GPT rows use Codex, others Claude Code; source segments span different runs.","zh":"六幅 2D 固定视图；250 道题、最多三次运行的回答分量平均分，保留 0–1 原单位，无效试验计零。GPT 使用 Codex，其他模型使用 Claude Code；来源拼接不同运行批次。"},"score_max":1,"results":[[2956,0.2965,1,3266,-1],[2958,0.2498,15,3267,-1],[3260,0.2468,195,3266,-1],[3259,0.245,195,3266,-1],[2959,0.2383,1,3266,-1],[2977,0.2148,21,3266,-1],[2960,0.2044,15,3267,-1],[3261,0.1661,25,3266,-1],[2980,0.1521,633,3266,-1],[2962,0.1462,83,3266,-1],[3262,0.1102,3026,3266,-1],[3263,0.0824,3033,3266,-1]]},{"id":"deepterminalbench","name":"DeepTerminalBench (Terminal-X)","group":"编程","unit":"%","metric":"Pass@1","source":{"name":"UniPat AI","url":"https://unipat.ai/blog/TerminalX","status":"current_snapshot","published_at":"2026-05-02"},"protocol":{"en":"50 terminal tasks; terminus-2, one attempt. Passing requires hidden verification reward = 1; the separate pass@4 calibration is not the ranking score.","zh":"50 道终端任务；terminus-2、单次尝试。隐藏验证奖励为 1 才算通过；不使用任务校准的 pass@4 作为排名成绩。"},"results":[[3181,0.34,1,3268,-1],[3185,0.32,25,3268,-1],[3269,0.32,15,3268,-1],[3187,0.32,21,3268,-1],[3270,0.32,195,3268,-1],[3186,0.3,551,3268,-1],[3271,0.24,195,3268,-1],[502,0.16,83,3268,-1],[3190,0.1,775,3268,-1]]},{"id":"vibe-coding-arena","name":"Vibe-Coding Arena (14-task panel)","group":"编程","unit":"score","metric":"Mean process grade","source":{"name":"UniPat AI","url":"https://unipat.ai/blog/Vibe-Coding-Arena","status":"current_snapshot","published_at":"2026-07-25"},"protocol":{"en":"Four anonymous models, the same 14 tasks. Human process grades on the original 0–5 scale; identities are undisclosed. This panel is distinct from the larger task inventory.","zh":"四个匿名模型在相同的 14 道任务上，由人类评定过程表现；保留 0–5 原量纲。模型身份未公开，此面板不等于完整任务库。"},"score_max":5,"results":[[3272,4.21,370,3273,-1],[3274,3.57,370,3273,-1],[3275,2.93,370,3273,-1],[3276,1.79,370,3273,-1]]},{"id":"surge-riemann","name":"Riemann-bench","group":"数学与科学","unit":"%","metric":"Accuracy","protocol":{"en":"Published research-mathematics accuracy; reasoning settings are retained per result.","zh":"官方研究级数学正确率；逐条保留推理设置。"},"source":{"name":"Surge AI","url":"https://surgehq.ai/benchmarks/riemann-bench","status":"current_snapshot","published_at":"2026-03-24"},"results":[[3277,0.7440000000000001,15,370,-1],[3278,0.72,15,370,-1],[3279,0.696,1,370,-1],[3280,0.68,1,370,-1],[3281,0.664,15,370,-1],[3282,0.6559999999999999,1,370,-1],[3283,0.648,15,370,-1],[3284,0.6,1,370,-1],[3285,0.5920000000000001,15,370,-1],[3286,0.552,15,370,-1],[3287,0.512,83,370,-1],[3288,0.47200000000000003,1,370,-1],[3289,0.41600000000000004,15,370,-1],[3290,0.392,83,370,-1],[3291,0.384,25,370,-1],[3292,0.384,3293,370,-1],[3294,0.376,15,370,-1],[3295,0.376,3296,370,-1],[3297,0.368,83,370,-1],[3298,0.336,83,370,-1],[3299,0.32799999999999996,1,370,-1],[3300,0.304,83,370,-1],[3301,0.28,66,370,-1],[3302,0.272,1,370,-1],[3303,0.24,25,370,-1],[3304,0.24,3305,370,-1],[3306,0.23199999999999998,25,370,-1],[3307,0.23199999999999998,66,370,-1],[3308,0.20800000000000002,66,370,-1],[3309,0.152,370,370,-1],[3310,0.152,50,370,-1],[3311,0.152,50,370,-1],[3312,0.136,66,370,-1],[3313,0.12,48,370,-1],[3314,0.12,3296,370,-1],[3315,0.11199999999999999,1,370,-1],[3316,0.11199999999999999,83,370,-1],[3317,0.10400000000000001,25,370,-1],[3318,0.10400000000000001,48,370,-1],[3319,0.08800000000000001,48,370,-1],[3320,0.08,25,370,-1],[3321,0.07200000000000001,3296,370,-1],[3322,0.055999999999999994,25,370,-1],[3323,0.048,554,370,-1]]},{"id":"surge-chartography","name":"Chartography","group":"视觉与视频","unit":"%","metric":"Accuracy","protocol":{"en":"Professional graphical reasoning, official overall accuracy; model reasoning settings retained.","zh":"专业图表推理，官方总体正确率；保留模型推理设置。"},"source":{"name":"Surge AI","url":"https://surgehq.ai/benchmarks/chartography","status":"current_snapshot","published_at":"2026-08-11"},"results":[[3324,0.716,83,370,-1],[3278,0.71,15,370,-1],[3279,0.6629999999999999,1,370,-1],[3283,0.536,15,370,-1],[3282,0.462,1,370,-1],[3277,0.45,15,370,-1],[3287,0.409,83,370,-1],[3290,0.40399999999999997,83,370,-1],[3284,0.348,1,370,-1],[3281,0.34,15,370,-1],[3325,0.34,83,370,-1],[3307,0.321,66,370,-1],[3326,0.314,15,370,-1],[3286,0.31,15,370,-1],[3289,0.29600000000000004,15,370,-1],[3285,0.29100000000000004,15,370,-1],[3311,0.29100000000000004,50,370,-1],[3301,0.276,66,370,-1],[3280,0.273,1,370,-1],[3295,0.266,3296,370,-1],[3298,0.261,83,370,-1],[3308,0.244,66,370,-1],[3327,0.205,3293,370,-1],[3328,0.174,50,370,-1],[3312,0.172,66,370,-1],[3292,0.17,3293,370,-1],[3329,0.166,1,370,-1],[3299,0.165,1,370,-1],[3319,0.163,48,370,-1],[3288,0.159,1,370,-1],[3330,0.159,50,370,-1],[3331,0.158,50,370,-1],[3332,0.147,3293,370,-1],[3333,0.129,3293,370,-1],[3314,0.126,3296,370,-1],[3321,0.122,3296,370,-1],[3303,0.11599999999999999,25,370,-1],[3334,0.11199999999999999,25,370,-1],[3335,0.102,83,370,-1],[3336,0.1,370,370,-1],[3337,0.091,370,370,-1],[3338,0.09,365,370,-1]]},{"id":"surge-enterprise","name":"EnterpriseBench: CoreCraft","group":"专业工作","unit":"score","metric":"Mean task score","protocol":{"en":"Official CoreCraft overall task score on the original 0–100 scale; partial credit is not a task pass rate.","zh":"CoreCraft 官方总体任务得分，保留 0–100 原量纲；部分得分不等于任务通过率。"},"source":{"name":"Surge AI","url":"https://surgehq.ai/benchmarks/enterprisebench-corecraft","status":"current_snapshot","published_at":"2026-02-01"},"score_max":100,"results":[[3282,77.4,1,370,-1],[3279,74.4,1,370,-1],[3284,70.3,1,370,-1],[3280,68.7,1,370,-1],[3332,68.2,3293,370,-1],[3327,65.6,3293,370,-1],[3304,64.6,3305,370,-1],[3301,61.9,66,370,-1],[3287,58.5,83,370,-1],[3328,57.4,50,370,-1],[3290,55.9,83,370,-1],[3291,53.3,25,370,-1],[3288,52.3,1,370,-1],[3313,51.8,48,370,-1],[3312,51.8,66,370,-1],[3329,51.3,1,370,-1],[3286,51.3,15,370,-1],[3297,50.8,83,370,-1],[3319,46.7,48,370,-1],[3277,46.2,15,370,-1],[3334,45.6,25,370,-1],[3285,43.1,15,370,-1],[3283,43.1,15,370,-1],[3295,43.1,3296,370,-1],[3294,42.6,15,370,-1],[3278,42.6,15,370,-1],[3339,42.1,48,370,-1],[3303,41.5,25,370,-1],[3340,41.0,3305,370,-1],[3300,39.5,83,370,-1],[3289,36.4,15,370,-1],[3299,35.9,1,370,-1],[3302,30.8,1,370,-1],[3341,30.8,25,370,-1],[3298,27.2,83,370,-1],[3310,26.2,50,370,-1],[3321,24.6,3296,370,-1],[3320,24.1,25,370,-1],[3342,20.5,3293,370,-1],[3343,20.1,15,370,-1],[3344,20.0,83,370,-1],[3345,17.4,48,370,-1],[3346,16.4,1,370,-1],[3347,14.4,83,370,-1],[3330,11.3,50,370,-1],[3348,10.3,307,370,-1],[3349,8.9,720,370,-1],[3314,8.7,3296,370,-1],[3338,3.6,365,370,-1],[3350,3.6,50,370,-1]]},{"id":"surge-hemingway","name":"Hemingway-bench","group":"生成与创作","unit":"elo","metric":"Elo","protocol":{"en":"Published writing-preference Elo; not a percentage or task completion rate.","zh":"官方写作偏好 Elo；不属于百分制或任务通过率。"},"source":{"name":"Surge AI","url":"https://surgehq.ai/benchmarks/hemingway-bench","status":"current_snapshot","published_at":"2026-04-15"},"results":[[3351,1110.0,1,370,-1],[3352,1105.0,83,370,-1],[3353,1095.0,1,370,-1],[3354,1093.0,83,370,-1],[3325,1075.0,83,370,-1],[3298,1073.0,83,370,-1],[3355,1071.0,83,370,-1],[3295,1069.0,3296,370,-1],[3356,1057.0,15,370,-1],[3347,1056.0,83,370,-1],[3313,1053.0,48,370,-1],[3344,1052.0,83,370,-1],[3357,1047.0,1,370,-1],[3358,1046.0,25,370,-1],[3359,1045.0,1,370,-1],[3360,1044.0,1,370,-1],[3361,1042.0,1,370,-1],[3362,1038.0,1,370,-1],[3363,1036.0,48,370,-1],[3364,1035.0,15,370,-1],[3319,1034.0,48,370,-1],[3321,1021.0,3296,370,-1],[3365,1016.0,1,370,-1],[3304,1014.0,3305,370,-1],[3366,1013.0,66,370,-1],[3311,1009.0,50,370,-1],[3346,1001.0,1,370,-1],[3367,999.0,25,370,-1],[3368,994.0,15,370,-1],[3335,990.0,83,370,-1],[3330,990.0,50,370,-1],[3314,988.0,3296,370,-1],[3369,985.0,15,370,-1],[3370,985.0,66,370,-1],[3371,983.0,3293,370,-1],[3372,981.0,25,370,-1],[3292,972.0,3293,370,-1],[3373,971.0,3305,370,-1],[3374,969.0,25,370,-1],[3375,954.0,15,370,-1],[3376,948.0,50,370,-1],[3328,929.0,50,370,-1],[3377,916.0,3293,370,-1],[3378,913.0,66,370,-1],[3348,888.0,307,370,-1],[58,883.0,3296,370,-1],[362,803.0,66,370,-1],[3379,749.0,720,370,-1]]},{"id":"apex-agents-mercor","name":"APEX-Agents v1.1 (Mercor)","group":"专业工作","unit":"%","metric":"pass-1","protocol":{"en":"Mercor global leaderboard; complete task pass@1. Each model/reasoning/harness configuration remains separate.","zh":"Mercor 总榜；完整任务 pass@1。各模型、推理与 Agent 配置分别记录。"},"source":{"name":"Mercor","url":"https://www.mercor.com/apex/","status":"current_snapshot","published_at":"2026-09-08"},"results":[[3380,0.8420000000000001,83,3381,-1],[2962,0.8240000000000001,83,3381,-1],[3382,0.755,1,3383,-1],[3384,0.735,1,3383,-1],[3385,0.698,25,3383,-1],[2967,0.6940000000000001,3296,3383,-1],[3149,0.6859999999999999,1,3383,-1],[2964,0.6829999999999999,15,3383,-1],[3386,0.6779999999999999,83,3381,-1],[2965,0.6609999999999999,66,3383,-1],[3156,0.6579999999999999,1,3383,-1],[2974,0.653,143,3387,-1],[2958,0.647,15,3383,-1],[2983,0.643,83,3381,-1],[3151,0.636,1,3383,-1],[3157,0.633,195,3387,-1],[2960,0.6,15,3383,-1],[3149,0.597,1,3381,-1],[2979,0.586,66,3387,-1],[2973,0.5820000000000001,15,3383,-1],[3388,0.574,2972,3389,-1],[3154,0.5660000000000001,3390,3383,-1],[3391,0.562,143,3381,-1],[3392,0.561,1,3383,-1],[3201,0.5529999999999999,25,3383,-1],[3196,0.551,15,3387,-1],[2968,0.546,143,3387,-1],[3393,0.545,1,3383,-1],[2971,0.544,2972,3389,-1],[3394,0.5429999999999999,15,3383,-1],[3395,0.539,775,3383,-1],[3396,0.528,3390,3383,-1],[3384,0.525,1,3397,-1],[3184,0.524,15,3387,-1],[2992,0.516,365,3381,-1],[762,0.515,15,3383,-1],[3398,0.502,3399,3381,-1],[3400,0.49200000000000005,1,3383,-1],[3401,0.489,1,3383,-1],[3402,0.488,224,3383,-1],[2977,0.478,3296,3383,-1],[2979,0.47600000000000003,66,3383,-1],[3403,0.475,195,3387,-1],[3404,0.473,25,3383,-1],[3405,0.469,83,3381,-1],[3406,0.46299999999999997,1,3383,-1],[3407,0.45399999999999996,66,3387,-1],[227,0.452,3390,3383,-1],[3382,0.446,1,3397,-1],[2976,0.44299999999999995,15,3383,-1],[3408,0.43,15,3383,-1],[3409,0.43,1,3381,-1],[3186,0.409,3390,3410,-1],[3153,0.395,25,3383,-1],[3198,0.377,775,3381,-1],[3411,0.376,3296,3381,-1],[3412,0.35100000000000003,83,3381,-1],[203,0.33799999999999997,199,3381,-1],[2987,0.33399999999999996,66,3387,-1],[3413,0.318,66,3387,-1],[2990,0.295,307,3381,-1],[3414,0.293,83,3381,-1],[3415,0.275,83,3381,-1],[3416,0.27399999999999997,307,3381,-1],[3417,0.249,195,3410,-1],[3418,0.22699999999999998,307,3381,-1],[3009,0.213,25,3410,-1],[3419,0.044000000000000004,15,3381,-1]]},{"id":"apex-v1-mercor","name":"APEX-1","group":"专业工作","unit":"score","metric":"mean-score","protocol":{"en":"Mercor global leaderboard; mean rubric score on 0–100. Each model/reasoning/harness configuration remains separate.","zh":"Mercor 总榜；0–100 的量规平均分。各模型、推理与 Agent 配置分别记录。"},"source":{"name":"Mercor","url":"https://www.mercor.com/apex/","status":"current_snapshot","published_at":"2025-09-30"},"score_max":100,"results":[[2962,81.1,83,3420,-1],[3380,80.0,83,3420,-1],[3407,75.6,66,3421,-1],[2965,75.1,66,3422,-1],[2979,73.9,66,3421,-1],[2987,73.9,66,3421,-1],[3149,73.8,1,3422,-1],[3413,73.8,66,3421,-1],[762,73.1,15,3422,-1],[2979,73.0,66,3422,-1],[2973,69.5,15,3422,-1],[2958,69.3,15,3421,-1],[2960,69.3,15,3422,-1],[2983,68.8,83,3420,-1],[3156,68.6,1,3421,-1],[3196,68.4,15,3421,-1],[2977,68.4,3296,3422,-1],[3385,68.3,25,3422,-1],[2974,67.7,143,3421,-1],[3386,67.0,83,3420,-1],[3408,66.6,15,3422,-1],[3404,66.0,25,3422,-1],[3151,66.0,1,3422,-1],[3405,66.0,83,3420,-1],[3423,65.9,15,3420,-1],[3406,65.8,1,3422,-1],[3384,65.8,1,3422,-1],[3184,65.6,15,3421,-1],[3153,65.5,25,3422,-1],[2968,65.4,143,3421,-1],[3406,65.3,1,3420,-1],[3157,65.3,195,3421,-1],[3395,65.0,775,3422,-1],[340,64.7,15,3420,-1],[3392,64.7,1,3422,-1],[2964,64.2,15,3422,-1],[3201,63.8,25,3422,-1],[3394,63.6,15,3422,-1],[3184,63.2,15,3420,-1],[3154,63.0,3390,3422,-1],[3401,62.7,1,3422,-1],[3403,62.6,195,3421,-1],[3415,62.2,83,3420,-1],[3400,62.2,1,3422,-1],[2976,62.0,15,3422,-1],[3412,61.8,83,3420,-1],[3391,61.8,143,3420,-1],[203,61.6,199,3420,-1],[227,61.5,3390,3422,-1],[3424,60.9,83,3420,-1],[26,60.9,83,3420,-1],[3402,60.6,224,3422,-1],[1464,60.5,15,3420,-1],[3411,60.1,3296,3420,-1],[3198,59.8,775,3420,-1],[3185,59.7,25,3422,-1],[2992,59.6,365,3420,-1],[3396,58.7,3390,3422,-1],[2990,57.6,307,3420,-1],[3425,57.5,143,3426,-1],[3427,57.3,1,3420,-1],[3428,57.1,143,3429,-1],[3418,56.1,307,3420,-1],[3409,55.1,1,3420,-1],[3414,54.7,83,3420,-1],[3382,53.9,1,3422,-1],[3416,52.7,307,3420,-1],[3186,52.4,3390,3429,-1],[1485,52.4,3296,3420,-1],[3009,51.5,25,3429,-1],[3417,50.3,195,3429,-1],[3393,49.6,1,3422,-1],[2585,48.1,83,3429,-1],[3419,46.4,15,3420,-1],[3190,45.8,775,3420,-1]]},{"id":"apex-swe-mercor","name":"APEX-SWE","group":"编程","unit":"%","metric":"pass-1","protocol":{"en":"Mercor global leaderboard; complete task pass@1. Each model/reasoning/harness configuration remains separate.","zh":"Mercor 总榜；完整任务 pass@1。各模型、推理与 Agent 配置分别记录。"},"source":{"name":"Mercor","url":"https://www.mercor.com/apex/","status":"current_snapshot","published_at":"2026-01-13"},"results":[[3384,0.6759999999999999,1,3430,-1],[3382,0.664,1,3430,-1],[3156,0.637,1,3430,-1],[3149,0.636,1,3430,-1],[3151,0.589,1,3430,-1],[2974,0.5770000000000001,143,3431,-1],[2974,0.564,143,3432,-1],[3384,0.5589999999999999,1,3433,-1],[3392,0.551,1,3430,-1],[3151,0.5483,1,3434,-1],[3156,0.5467,1,3434,-1],[2965,0.546,66,3430,-1],[2971,0.542,2972,3435,-1],[3391,0.536,143,3432,-1],[2968,0.534,143,3431,-1],[3157,0.523,195,3431,-1],[3153,0.519,25,3430,-1],[3391,0.5117,143,3436,-1],[3403,0.507,195,3431,-1],[2962,0.49200000000000005,83,3432,-1],[3385,0.491,25,3430,-1],[2958,0.486,15,3430,-1],[762,0.48,15,3430,-1],[2977,0.48,3296,3430,-1],[3400,0.47600000000000003,1,3430,-1],[3401,0.473,1,3436,-1],[2960,0.469,15,3430,-1],[3201,0.467,25,3430,-1],[3393,0.46399999999999997,1,3430,-1],[3396,0.461,3390,3430,-1],[3404,0.45899999999999996,25,3430,-1],[762,0.45799999999999996,15,3431,-1],[3154,0.456,3390,3430,-1],[3394,0.45,15,3430,-1],[2977,0.44670000000000004,3296,3434,-1],[3382,0.44,1,3433,-1],[3401,0.439,1,3430,-1],[3393,0.4365,1,3436,-1],[3398,0.431,3399,3432,-1],[3386,0.42,83,3432,-1],[3406,0.418,1,3430,-1],[3402,0.41600000000000004,224,3430,-1],[3423,0.415,15,3436,-1],[3400,0.41340000000000005,1,3436,-1],[3196,0.40840000000000004,15,3437,-1],[3407,0.405,66,3431,-1],[3411,0.40299999999999997,3296,3432,-1],[762,0.3967,15,3437,-1],[3405,0.3967,83,3436,-1],[3405,0.39399999999999996,83,3432,-1],[2979,0.391,66,3430,-1],[3413,0.389,66,3431,-1],[2976,0.38799999999999996,15,3430,-1],[227,0.374,3390,3430,-1],[227,0.3732,3390,3438,-1],[2992,0.37,365,3432,-1],[3196,0.366,15,3431,-1],[2983,0.364,83,3432,-1],[2979,0.364,66,3431,-1],[3415,0.361,83,3432,-1],[3198,0.34600000000000003,775,3432,-1],[3186,0.345,3390,3439,-1],[3409,0.345,1,3432,-1],[3184,0.34299999999999997,15,3431,-1],[3412,0.33899999999999997,83,3432,-1],[3185,0.33799999999999997,25,3430,-1],[3415,0.325,83,3436,-1],[3440,0.3248,3296,3438,-1],[3408,0.321,15,3430,-1],[3441,0.317,370,3438,-1],[3414,0.312,83,3432,-1],[2973,0.311,15,3430,-1],[3442,0.31,1,3436,-1],[3443,0.308,15,3436,-1],[244,0.303,3296,3438,-1],[3444,0.29,15,3436,-1],[3412,0.28800000000000003,83,3436,-1],[3417,0.27899999999999997,195,3439,-1],[26,0.268,83,3436,-1],[203,0.261,199,3432,-1],[3009,0.256,25,3439,-1],[2990,0.223,307,3432,-1],[3190,0.215,775,3432,-1],[3428,0.21,143,3438,-1],[1485,0.174,3296,3432,-1],[3009,0.158,25,3438,-1],[3418,0.152,307,3432,-1],[1485,0.11199999999999999,3296,3438,-1],[3419,0.033,15,3432,-1]]},{"id":"frontierswe-v2","name":"FrontierSWE v2","group":"编程","unit":"score","metric":"Mean@5 score","protocol":{"en":"34 tasks; proximus; five trials and a 20-hour budget per task. Absolute mean@5 score (0–100), including partial-credit performance tasks; not best@5 or relative-to-frontier normalization.","zh":"34 道任务；proximus；每题五次试验、每次 20 小时。使用 0–100 的绝对 mean@5 得分，包含性能任务的部分得分；不使用 best@5 或相对最优模型归一化。"},"source":{"name":"Proximal","url":"https://www.frontierswe.com/","status":"current_snapshot","published_at":"2026-09-01"},"score_max":100,"results":[[2958,65.50757455882352,15,3445,3446],[2959,62.33098823529411,1,3445,3447],[2961,61.889299411764696,1,3445,3448],[3380,61.19949235294118,83,3445,3449],[2956,56.29078999999999,1,3445,3450],[2962,54.96118588235292,83,3445,3451],[755,52.01176764705882,1,3445,3452],[3453,46.96449352941177,1,3445,3454],[2970,43.76868294117648,1,3445,3455],[762,32.20246705882353,15,3445,3456],[3154,30.181864117647052,551,3445,3457],[2968,29.50476117647059,143,3445,3458],[2977,25.865032058823534,21,3445,3459],[2979,25.80214823529412,66,3445,3460],[2974,25.286195882352946,143,3445,3461],[3386,20.260590588235296,83,3445,3462],[2983,19.626902941176464,83,3445,3463],[2980,18.125205882352944,551,3445,3464],[3465,17.809649411764706,195,3445,3466],[3157,15.809350588235294,195,3445,3467],[3468,14.831252647058824,25,3445,3469],[3407,11.970256470588238,66,3445,3470],[203,4.118886470588236,3471,3445,3472]]},{"id":"turing-sdlc-all-core","name":"SDLCbench (All-core pass)","group":"编程","unit":"%","metric":"All-core pass rate","protocol":{"en":"Private enterprise software-development tasks; all core checks must pass. Harnesses are retained separately; soft-gated rewards are not used as pass rates.","zh":"私有企业软件开发任务；所有核心检查通过才算成功。分别保留 Agent；不将软门控奖励当作通过率。"},"source":{"name":"Turing","url":"https://bench.turing.com/sdlc/","status":"current_snapshot","published_at":"2026-04-06"},"results":[[3473,0.4,1,3474,-1],[3208,0.25,15,3475,-1],[3476,0.2,1,3477,-1],[3208,0.2,15,3477,-1],[413,0.0,83,3477,-1],[433,0.0,143,3477,-1]]},{"id":"turing-companybench","name":"CompanyBench","group":"专业工作","unit":"%","metric":"All-core pass rate","protocol":{"en":"64 tasks, ten independent trials per model; a trial passes only when every check passes. Published June 2026 evaluation cohort.","zh":"64 道任务，每模型十次独立试验；每项检查均通过时才算成功。官方 2026 年 6 月评测批次。"},"source":{"name":"Turing","url":"https://bench.turing.com/companybench/leaderboard/","status":"current_snapshot","published_at":"2026-06-15"},"results":[[3196,0.4438,15,3478,-1],[3479,0.3594,1,3478,-1],[3412,0.1828,83,3478,-1],[3415,0.0766,83,3478,-1]]},{"id":"turing-scicode-plus-plus","name":"SciCode++","group":"数学与科学","unit":"%","metric":"Main problem pass@8 (publisher label)","protocol":{"en":"The shared 3,451-problem grid, eight trials per model; official Main problem pass@8 column. Both published models use high reasoning. Distinct from SciCode.","zh":"共同完成的 3,451 道题，每模型八次试验；沿用官方 Main problem pass@8 列。两模型均使用 high 推理。与 SciCode 分开。"},"source":{"name":"Turing","url":"https://bench.turing.com/scicode/leaderboard/","status":"current_snapshot","published_at":"2024-07-18"},"results":[[3480,0.1771,15,3481,-1],[3482,0.1464,83,3481,-1]]},{"id":"skillsbench-with-skills","name":"SkillsBench (With Skills)","group":"Agent 与工具","unit":"%","metric":"Task pass rate","protocol":{"en":"Official paired leaderboard, with skills; model and agent pairs remain distinct. Uses pass rate, not normalized gain.","zh":"官方配对评测榜单；使用 Skills；分别记录模型与 Agent。使用通过率，不使用归一化增益。"},"source":{"name":"BenchFlow","url":"https://www.skillsbench.ai/","status":"current_snapshot","published_at":"2026-02-13"},"results":[[3196,0.6729999999999999,15,3483,-1],[3196,0.665,15,3484,-1],[3400,0.612,3485,3486,-1],[3412,0.608,3487,3488,-1],[3489,0.584,48,3483,-1],[3158,0.5589999999999999,3305,3486,-1],[3424,0.546,3487,3488,-1],[3401,0.541,3485,3483,-1],[3490,0.54,641,3483,-1],[3400,0.531,3485,3483,-1],[3491,0.53,6,3483,-1],[3412,0.528,3487,3483,-1],[340,0.517,15,3484,-1],[3406,0.502,3485,3486,-1],[3492,0.501,25,3483,-1],[3427,0.49,3485,3486,-1],[3415,0.48200000000000004,3487,3483,-1],[3409,0.47200000000000003,3485,3483,-1],[3261,0.447,25,3483,-1],[3493,0.41700000000000004,3293,3483,-1],[3494,0.414,15,3483,-1],[3442,0.36200000000000004,3485,3486,-1],[3495,0.349,6,3483,-1],[3496,0.301,3485,3486,-1],[3497,0.201,3487,3483,-1]]},{"id":"skillsbench-without-skills","name":"SkillsBench (Without Skills)","group":"Agent 与工具","unit":"%","metric":"Task pass rate","protocol":{"en":"Official paired leaderboard, without skills; model and agent pairs remain distinct. Uses pass rate, not normalized gain.","zh":"官方配对评测榜单；不使用 Skills；分别记录模型与 Agent。使用通过率，不使用归一化增益。"},"source":{"name":"BenchFlow","url":"https://www.skillsbench.ai/","status":"current_snapshot","published_at":"2026-02-13"},"results":[[3196,0.515,15,3498,-1],[3196,0.46799999999999997,15,3499,-1],[3401,0.457,3485,3498,-1],[3400,0.43,3485,3500,-1],[3400,0.42100000000000004,3485,3498,-1],[3415,0.41100000000000003,3487,3498,-1],[3412,0.36,3487,3501,-1],[3424,0.342,3487,3501,-1],[3412,0.33799999999999997,3487,3498,-1],[3406,0.337,3485,3500,-1],[3409,0.335,3485,3498,-1],[3490,0.33399999999999996,641,3498,-1],[3489,0.327,48,3498,-1],[3494,0.299,15,3498,-1],[340,0.297,15,3499,-1],[3491,0.297,6,3498,-1],[3261,0.275,25,3498,-1],[3492,0.26899999999999996,25,3498,-1],[3427,0.23800000000000002,3485,3500,-1],[3493,0.228,3293,3498,-1],[3495,0.18100000000000002,6,3498,-1],[3442,0.16699999999999998,3485,3500,-1],[3497,0.16,3487,3498,-1],[3496,0.08800000000000001,3485,3500,-1]]},{"id":"finance-agent-v2-1","name":"Finance Agent v2.1","group":"专业工作","unit":"score","metric":"Overall rubric score","protocol":{"en":"Reviewed FAB v2.1 dataset and FAB v2 harness; overall published rubric credit on 0–100.","zh":"审校后的 FAB v2.1 数据集与 FAB v2 Agent；总体量规得分，保留 0–100。"},"source":{"name":"Vals AI","url":"https://www.vals.ai/benchmarks/fabv2-1","status":"current_snapshot","published_at":"2026-05-24"},"score_max":100,"results":[[2959,67.85,1,3502,3503],[2962,67.66,83,3502,3504],[3505,67.58,66,3502,3506],[2961,67.28,1,3502,3507],[3380,67.16,83,3502,3508],[2983,65.76,83,3502,3509],[3510,65.55,66,3502,3511],[3512,65.43,25,3502,3513],[2970,65.37,1,3502,3514],[3515,65.0,551,3502,3516],[2971,64.26,2972,3502,3517],[3518,63.32,2972,3502,3519],[2966,62.93,25,3502,3520],[2960,61.82,15,3502,3521],[3522,61.72,417,3502,3523],[3524,61.62,3033,3502,3525],[2967,61.49,21,3502,3526],[2992,61.38,417,3502,3527],[2987,60.66,66,3502,3528],[3529,60.59,551,3502,3530],[2968,60.59,143,3502,3531],[2958,60.43,15,3502,3532],[2977,60.01,21,3502,3533],[3534,58.9,2985,3502,3535],[3395,58.83,775,3502,3536],[3537,58.83,195,3502,3538],[3539,57.12,25,3502,3540],[2976,56.84,15,3502,3541],[2964,55.96,15,3502,3542],[3398,54.71,3399,3502,3543],[3198,54.34,775,3502,3544],[3545,51.83,224,3502,3546],[203,49.87,3471,3502,3547],[2990,47.73,370,3502,3548],[3549,44.78,3471,3502,3550]]},{"id":"tax-agent-bench","name":"Tax Agent Bench","group":"专业工作","unit":"%","metric":"All-pass accuracy","protocol":{"en":"Research-grade US corporate tax questions; all-pass accuracy under the published Tax Agent Bench protocol.","zh":"研究级美国企业税务问题；按 Tax Agent Bench 官方协议计算完整通过率。"},"source":{"name":"Vals AI","url":"https://www.vals.ai/benchmarks/tax_agent_bench","status":"current_snapshot","published_at":"2026-09-01"},"results":[[3380,0.5178,83,3551,3508],[3537,0.4966,195,3551,3538],[2956,0.4919,1,3551,3552],[2959,0.4557,1,3551,3503],[755,0.45549999999999996,1,3551,3553],[2962,0.4456,83,3551,3504],[2966,0.4435,25,3551,3520],[3505,0.43,66,3551,3506],[3554,0.4244,66,3551,3555],[2961,0.41960000000000003,1,3551,3507],[3515,0.397,551,3551,3516],[3453,0.38280000000000003,1,3551,3556],[3510,0.3627,66,3551,3511],[2979,0.35969999999999996,66,3551,3557],[2968,0.342,143,3551,3531],[2977,0.3337,21,3551,3533],[2974,0.33140000000000003,143,3551,3558],[3407,0.32799999999999996,66,3551,3559],[3524,0.3262,3033,3551,3525],[2971,0.3239,2972,3551,3517],[3395,0.3214,775,3551,3536],[2983,0.3212,83,3551,3509],[3560,0.3209,195,3551,3561],[2970,0.31920000000000004,1,3551,3514],[3529,0.3155,551,3551,3530],[762,0.3109,15,3551,3562],[3563,0.31079999999999997,370,3551,3564],[3403,0.3089,195,3551,3565],[3522,0.30670000000000003,417,3551,3523],[2992,0.3039,417,3551,3527],[2973,0.3004,15,3551,3566],[3479,0.3,1,3551,3567],[3568,0.2937,1,3551,3569],[3518,0.2851,2972,3551,3519],[3570,0.2807,25,3551,3571],[3534,0.2727,2985,3551,3535],[3539,0.2676,25,3551,3540],[3512,0.2663,25,3551,3513],[2987,0.2632,66,3551,3528],[3572,0.2584,1,3551,3573],[3574,0.25170000000000003,25,3551,3575],[2967,0.2487,21,3551,3526],[3391,0.2486,143,3551,3576],[3408,0.2482,15,3551,3577],[3578,0.23739999999999997,1,3551,3579],[3386,0.228,83,3551,3580],[3581,0.21760000000000002,15,3551,3582],[2960,0.2175,15,3551,3521],[3415,0.2165,83,3551,3583],[2958,0.2072,15,3551,3532],[3584,0.20600000000000002,1188,3551,3585],[3198,0.2059,775,3551,3544],[2976,0.19949999999999998,15,3551,3541],[3398,0.18170000000000003,3399,3551,3543],[3405,0.1804,83,3551,3586],[2964,0.17079999999999998,15,3551,3542],[3587,0.1583,195,3551,3588],[3549,0.1539,3471,3551,3550],[203,0.15289999999999998,3471,3551,3547],[3394,0.15,15,3551,3589],[3490,0.124,21,3551,3590],[3591,0.1184,83,3551,3592],[2990,0.1155,370,3551,3548],[3593,0.115,2972,3551,3594],[3595,0.1047,195,3551,3596],[3597,0.0929,83,3551,3598],[3545,0.08810000000000001,224,3551,3546],[3599,0.08810000000000001,15,3551,3600],[3601,0.07769999999999999,2972,3551,3602],[3603,0.0768,195,3551,3604],[3414,0.0725,83,3551,3605],[148,0.0695,143,3551,3606],[3190,0.0687,775,3551,3607],[3493,0.061799999999999994,143,3551,3608],[3609,0.058600000000000006,1,3551,3610],[3611,0.049699999999999994,15,3551,3612],[3613,0.044800000000000006,417,3551,3614],[3615,0.0311,83,3551,3616],[3617,0.0259,370,3551,3618],[3619,0.024900000000000002,3399,3551,3620]]},{"id":"excel-modeling-v1-1","name":"Excel Modeling Benchmark v1.1","group":"专业工作","unit":"score","metric":"Overall modeling score","protocol":{"en":"Template and scratch Excel modeling; exact cell checks and partial-credit rubrics; overall score on 0–100.","zh":"模板填写与从零构建 Excel 模型；精确单元格检查与部分得分量规，总体分数保留 0–100。"},"source":{"name":"Vals AI","url":"https://www.vals.ai/benchmarks/emb-v1-1","status":"current_snapshot","published_at":"2026-10-01"},"score_max":100,"results":[[2956,76.73,1,3621,3552],[3380,76.47,83,3621,3508],[2961,75.9,1,3621,3507],[2959,75.84,1,3621,3503],[3505,75.0,66,3621,3506],[2962,74.83,83,3621,3504],[2964,73.33,15,3621,3542],[2983,72.87,83,3621,3509],[2958,72.68,15,3621,3532],[3510,72.5,66,3621,3511],[3537,71.98,195,3621,3538],[2968,71.5,143,3621,3531],[2977,71.26,21,3621,3533],[2960,71.09,15,3621,3521],[2967,70.0,21,3621,3526],[2976,69.95,15,3621,3541],[3534,69.55,2985,3621,3535],[2966,69.17,25,3621,3520],[3515,69.11,551,3621,3516],[2992,68.97,417,3621,3527],[3524,68.95,3033,3621,3525],[3518,68.88,2972,3621,3519],[3522,68.67,417,3621,3523],[3395,67.84,775,3621,3536],[3512,67.76,25,3621,3513],[2971,67.35,2972,3621,3517],[3529,66.25,551,3621,3530],[3560,65.99,195,3621,3561],[2970,65.48,1,3621,3514],[3539,65.13,25,3621,3540],[3545,64.73,224,3621,3546],[2987,61.98,66,3621,3528],[3198,59.75,775,3621,3544],[3398,57.88,3399,3621,3543],[2990,52.77,370,3621,3548],[203,40.97,3471,3621,3547],[3549,40.82,3471,3621,3550]]},{"id":"harvey-lab-vals-v1-1","name":"Harvey LAB (Vals v1.1)","group":"专业工作","unit":"%","metric":"Task pass rate","protocol":{"en":"120-task reviewed holdout, HLAB-Vals-1.1; all criteria must pass. Distinct from the AA evaluator cohort and archived Vals v1.","zh":"120 道经审校的保留任务，HLAB-Vals-1.1；全部标准通过才算成功。与 AA 评测及 Vals v1 分开。"},"source":{"name":"Vals AI","url":"https://www.vals.ai/benchmarks/hlab","status":"current_snapshot","published_at":"2026-05-06"},"results":[[2962,0.2708,83,3622,3504],[2987,0.25,66,3622,3528],[3510,0.2375,66,3622,3511],[3505,0.23329999999999998,66,3622,3506],[2968,0.1833,143,3622,3531],[3380,0.175,83,3622,3508],[2992,0.175,417,3622,3527],[3522,0.1583,417,3622,3523],[3537,0.1542,195,3622,3538],[3515,0.1458,551,3622,3516],[2966,0.1417,25,3622,3520],[2971,0.1375,2972,3622,3517],[3518,0.1333,2972,3622,3519],[2977,0.1292,21,3622,3533],[3534,0.1292,2985,3622,3535],[2983,0.1208,83,3622,3509],[2967,0.1167,21,3622,3526],[3539,0.10830000000000001,25,3622,3540],[2961,0.1042,1,3622,3507],[3529,0.1042,551,3622,3530],[2959,0.0917,1,3622,3503],[3524,0.0875,3033,3622,3525],[3395,0.075,775,3622,3536],[2958,0.0708,15,3622,3532],[2960,0.0625,15,3622,3521],[3398,0.05,3399,3622,3543],[3545,0.0458,224,3622,3546],[2970,0.0417,1,3622,3514],[2976,0.0417,15,3622,3541],[203,0.0375,3471,3622,3547],[2990,0.0375,370,3622,3548],[3549,0.0292,3471,3622,3550],[2964,0.0167,15,3622,3542]]},{"id":"phare-average-safety","name":"Phare (Average safety)","group":"安全与可靠性","unit":"score","metric":"Average safety score","protocol":{"en":"Average across hallucination, harm, bias and jailbreak resistance modules and evaluated languages. Composite 0–100 score, excluded from task-saturation bins.","zh":"幻觉、有害内容、偏见、越狱抵抗四个模块及受测语言的平均值；0–100 综合分，不进入任务饱和度分组。"},"source":{"name":"Giskard","url":"https://phare.giskard.ai/","status":"current_snapshot","published_at":"2025-05-16"},"score_max":100,"results":[[3623,83.16,1,3624,3625],[3626,82.38,1,3624,3627],[3628,78.1,1,3624,3629],[2271,77.6,1,3624,3630],[3631,76.87,1,3624,3632],[3633,76.42,66,3624,3634],[3490,76.06,641,3624,3635],[3587,75.57,370,3624,3636],[7,74.13,1,3624,3637],[3595,74.06,370,3624,3638],[3639,74.03,1,3624,3640],[3641,73.31,83,3624,3642],[46,73.22,15,3624,3643],[3644,72.82,15,3624,3645],[3646,71.03,15,3624,3647],[362,70.84,66,3624,3648],[3649,70.65,15,3624,3650],[3482,70.63,83,3624,3651],[3581,70.53,15,3624,3652],[2811,70.47,15,3624,3653],[3654,69.92,48,3624,3655],[3656,69.53,15,3624,3657],[2810,69.48,1,3624,3658],[3659,69.08,1,3624,3660],[3661,68.85,370,3624,3662],[3493,68.63,370,3624,3663],[3497,67.45,83,3624,3664],[3665,67.39,83,3624,3666],[3667,67.2,15,3624,3668],[2812,67.11,15,3624,3669],[244,67.05,641,3624,3670],[3671,66.55,370,3624,3672],[3673,66.14,15,3624,3674],[3675,65.93,370,3624,3676],[3677,64.84,370,3624,3678],[3679,64.52,370,3624,3680],[3613,64.32,370,3624,3681],[67,64.19,83,3624,3682],[3683,63.88,370,3624,3684],[363,63.82,66,3624,3685],[3261,63.68,370,3624,3686],[2055,63.64,83,3624,3687],[3688,62.46,66,3624,3689],[3690,62.26,370,3624,3691],[3692,62.24,370,3624,3693],[3694,62.22,370,3624,3695],[3428,62.13,370,3624,3696],[3697,61.98,66,3624,3698],[3338,60.93,370,3624,3699],[2273,60.92,644,3624,3700],[3701,60.87,370,3624,3702],[2602,60.55,83,3624,3703],[3704,59.96,370,3624,3705],[3706,59.33,370,3624,3707],[3708,58.91,83,3624,3709],[3492,58.85,370,3624,3710],[800,58.82,370,3624,3711],[3712,58.55,370,3624,3713],[3714,58.32,15,3624,3715],[3716,58.11,370,3624,3717],[2828,57.45,15,3624,3718],[3719,57.44,83,3624,3720],[3721,57.31,83,3624,3722],[3723,55.94,370,3624,3724],[3725,55.73,83,3624,3726],[3727,54.58,370,3624,3728],[360,53.45,370,3624,3729]]},{"id":"vending-bench-2","name":"Vending-Bench 2","group":"预测与决策","unit":"usd","metric":"Mean final money balance (USD)","protocol":{"en":"Arithmetic mean final bank balance in US dollars, including all model settings behind the Show more control. Trial counts and standard errors are preserved.","zh":"以美元计的期末银行余额算术平均数，包含展开榜单后的所有模型设置。逐行保留试验次数与标准误。"},"source":{"name":"Andon Labs","url":"https://andonlabs.com/evals/vending-bench-2","status":"current_snapshot","published_at":"2025-11-18"},"results":[[3380,16484.20666666667,83,3730,-1],[2958,15514.701666666668,15,3730,-1],[3394,14427.848333333333,15,3730,-1],[2962,13718.158333333333,83,3730,-1],[3731,12291.558333333334,66,3730,-1],[755,11181.87333333333,1,3730,-1],[2961,11165.27833333333,1,3730,-1],[3578,10936.763333333334,1,3730,-1],[2968,10536.831666666667,143,3730,-1],[762,9619.367999999999,15,3730,-1],[3732,9432.18,66,3730,-1],[2959,9235.248333333331,1,3730,-1],[2964,9107.458333333334,15,3730,-1],[2974,9047.032500000001,143,3730,-1],[227,8313.778333333335,551,3730,-1],[3154,8163.609999999998,551,3730,-1],[2261,8017.594,1,3730,-1],[3196,7523.843999999999,15,3730,-1],[2973,7343.206,15,3730,-1],[3572,7204.139999999999,1,3730,-1],[3413,6520.474999999999,66,3730,-1],[3568,6377.701666666665,1,3730,-1],[3490,6204.570000000001,21,3730,-1],[3184,6144.176,15,3730,-1],[3733,5940.124000000002,15,3730,-1],[3734,5787.428,1,3730,-1],[3735,5680.258,1,3730,-1],[3186,5634.406,551,3730,-1],[26,5478.158000000001,83,3730,-1],[2956,5421.556666666666,1,3730,-1],[3415,5396.418333333334,83,3730,-1],[3736,5165.035000000001,21,3730,-1],[3603,5114.872000000001,195,3730,-1],[2983,5093.793333333332,83,3730,-1],[3411,5082.936,21,3730,-1],[3737,5018.519999999999,1,3730,-1],[3738,4967.056666666668,1,3730,-1],[3739,4966.639999999998,1,3730,-1],[3740,4906.963333333332,21,3730,-1],[3741,4662.846666666667,143,3730,-1],[3742,4529.944000000001,1,3730,-1],[268,4432.122000000001,551,3730,-1],[3743,4339.812500000001,1,3730,-1],[3744,4254.1866666666665,195,3730,-1],[3408,4094.7120000000004,15,3730,-1],[3391,3887.4259999999986,143,3730,-1],[283,3838.7400000000002,1,3730,-1],[3745,3774.254,83,3730,-1],[3424,3634.715999999999,83,3730,-1],[2970,3602.4633333333345,1,3730,-1],[340,3591.3259999999996,15,3730,-1],[3746,3284.519999999999,25,3730,-1],[3747,2992.344,1,3730,-1],[2278,2376.816,551,3730,-1],[3198,2157.7716666666665,775,3730,-1],[3748,1473.432,15,3730,-1],[244,1198.4579999999999,21,3730,-1],[3749,1106.6280000000002,143,3730,-1],[3009,1034.0040000000001,25,3730,-1],[3412,911.2139999999997,83,3730,-1],[2602,573.6420000000003,83,3730,-1],[2055,548.836,83,3730,-1],[3750,462.68500000000023,195,3730,-1],[325,458.88599999999985,1,3730,-1],[3751,201.97599999999994,195,3730,-1],[3008,160.5959999999999,775,3730,-1],[3752,71.56500000000018,195,3730,-1],[3493,35.26166666666665,143,3730,-1],[3753,0.5440000000000417,195,3730,-1],[3754,-11.340000000000078,195,3730,-1],[3755,-21.532000000000053,15,3730,-1],[3005,-23.15999999999999,775,3730,-1],[3756,-31.18399999999995,15,3730,-1]]},{"id":"swe-rebench","name":"SWE-rebench (2026-05–06)","group":"编程","unit":"%","metric":"Resolved rate","protocol":{"en":"Official default window 2026-05-15 to 2026-07-01: 111 problems, 65 repositories; resolved rate (pass@1), not pass@5. Publisher-flagged contamination-risk rows are excluded.","zh":"官方默认窗口 2026-05-15 至 2026-07-01，111 道题、65 个仓库；解决率（pass@1），不使用 pass@5；排除发布方标记有污染风险的行。"},"source":{"name":"Nebius","url":"https://swe-rebench.com/","status":"current_snapshot","published_at":"2025-05-26"},"results":[[3757,0.618,3757,3758,-1],[3759,0.604,1,3758,-1],[2300,0.58,15,3758,-1],[3760,0.517,3760,3758,-1],[3593,0.465,3761,3758,-1],[3762,0.402,25,3758,-1],[3763,0.312,50,3758,-1],[3764,0.247,50,3758,-1],[2590,0.171,50,3758,-1]]},{"id":"multi-swe-bench-java","name":"Multi-SWE-bench (Java)","group":"编程","unit":"%","metric":"Resolved rate","protocol":{"en":"Java Verified subset; verified leaderboard submissions, latest dated run per model/agent. Different language task sets are not averaged together.","zh":"Java Verified 子集；采用榜单已验证提交，各模型与 Agent 使用最新日期记录。不同语言任务集不混合平均。"},"source":{"name":"ByteDance Seed","url":"https://multi-swe-bench.github.io/","status":"current_snapshot","published_at":"2025-04-03"},"results":[[764,0.2890625,83,3765,3766],[3767,0.234375,1,3765,3768],[769,0.2265625,25,3769,3770],[3767,0.21875,1,3771,3772],[764,0.21875,83,3769,3773],[3774,0.21875,15,3765,3775],[3774,0.2109375,15,3769,3776],[3777,0.203125,1,3765,3778],[3779,0.1640625,15,3765,3780],[3777,0.1484375,1,3769,3781],[3777,0.1484375,1,3771,3782],[1583,0.1484375,66,3769,3783],[3767,0.140625,1,3769,3784],[3785,0.1328125,370,3769,3786],[3787,0.125,15,3765,3788],[764,0.125,83,3771,3789],[1573,0.1171875,25,3765,3790],[3785,0.1171875,370,3765,3791],[3787,0.1171875,15,3769,3792],[3785,0.109375,370,3771,3793],[1651,0.109375,195,3769,3794],[3779,0.1015625,15,3771,3795],[769,0.09375,25,3765,3796],[1573,0.09375,25,3771,3797],[3787,0.09375,15,3771,3798],[769,0.0859375,25,3771,3799],[1573,0.0703125,25,3769,3800],[2661,0.0703125,370,3765,3801],[1583,0.0625,66,3765,3802],[1583,0.0625,66,3771,3803],[2661,0.0546875,370,3769,3804],[3779,0.0546875,15,3769,3805],[3774,0.0390625,15,3771,3806],[1651,0.03125,195,3771,3807],[1651,0.0234375,195,3765,3808],[2661,0.0078125,370,3771,3809]]},{"id":"multi-swe-bench-typescript","name":"Multi-SWE-bench (TypeScript)","group":"编程","unit":"%","metric":"Resolved rate","protocol":{"en":"TypeScript Verified subset; verified leaderboard submissions, latest dated run per model/agent. Different language task sets are not averaged together.","zh":"TypeScript Verified 子集；采用榜单已验证提交，各模型与 Agent 使用最新日期记录。不同语言任务集不混合平均。"},"source":{"name":"ByteDance Seed","url":"https://multi-swe-bench.github.io/","status":"current_snapshot","published_at":"2025-04-03"},"results":[[3777,0.27232142857142855,1,3810,3811],[764,0.22321428571428573,83,3771,3812],[3777,0.11607142857142858,1,3771,3813],[764,0.11607142857142858,83,3769,3814],[3767,0.11160714285714286,1,3765,3815],[1583,0.09375,66,3769,3816],[764,0.08928571428571429,83,3765,3817],[3777,0.08035714285714286,1,3765,3818],[1583,0.08035714285714286,66,3771,3819],[3785,0.07589285714285714,370,3769,3820],[3785,0.07142857142857142,370,3765,3821],[1573,0.06696428571428571,25,3769,3822],[769,0.0625,25,3769,3823],[769,0.05803571428571429,25,3765,3824],[3774,0.05803571428571429,15,3769,3825],[3785,0.05357142857142857,370,3771,3826],[3777,0.049107142857142856,1,3769,3827],[3779,0.049107142857142856,15,3765,3828],[1583,0.044642857142857144,66,3765,3829],[1651,0.044642857142857144,195,3769,3830],[3774,0.04017857142857143,15,3765,3831],[3767,0.03571428571428571,1,3769,3832],[1573,0.026785714285714284,25,3765,3833],[3767,0.022321428571428572,1,3771,3834],[2661,0.022321428571428572,370,3769,3835],[3787,0.022321428571428572,15,3769,3836],[2661,0.017857142857142856,370,3765,3837],[1573,0.013392857142857142,25,3771,3838],[769,0.004464285714285714,25,3771,3839],[3787,0.004464285714285714,15,3765,3840],[3774,0.004464285714285714,15,3771,3841],[3779,0.004464285714285714,15,3769,3842],[3779,0.004464285714285714,15,3771,3843],[2661,0.0,370,3771,3844],[3787,0.0,15,3771,3845],[1651,0.0,195,3765,3846],[1651,0.0,195,3771,3847]]},{"id":"multi-swe-bench-javascript","name":"Multi-SWE-bench (JavaScript)","group":"编程","unit":"%","metric":"Resolved rate","protocol":{"en":"JavaScript Verified subset; verified leaderboard submissions, latest dated run per model/agent. Different language task sets are not averaged together.","zh":"JavaScript Verified 子集；采用榜单已验证提交，各模型与 Agent 使用最新日期记录。不同语言任务集不混合平均。"},"source":{"name":"ByteDance Seed","url":"https://multi-swe-bench.github.io/","status":"current_snapshot","published_at":"2025-04-03"},"results":[[764,0.16292134831460675,83,3771,3848],[3785,0.09550561797752809,370,3771,3849],[764,0.08707865168539326,83,3769,3850],[764,0.07584269662921349,83,3765,3851],[3785,0.056179775280898875,370,3769,3852],[3767,0.05056179775280899,1,3771,3853],[1583,0.05056179775280899,66,3769,3854],[3774,0.05056179775280899,15,3769,3855],[3767,0.047752808988764044,1,3765,3856],[1583,0.047752808988764044,66,3771,3857],[769,0.0449438202247191,25,3769,3858],[3777,0.042134831460674156,1,3765,3859],[3774,0.042134831460674156,15,3765,3860],[3779,0.042134831460674156,15,3765,3861],[3774,0.03651685393258427,15,3771,3862],[1573,0.033707865168539325,25,3769,3863],[3779,0.033707865168539325,15,3771,3864],[3779,0.028089887640449437,15,3769,3865],[769,0.025280898876404494,25,3771,3866],[1573,0.025280898876404494,25,3765,3867],[1583,0.02247191011235955,66,3765,3868],[3777,0.019662921348314606,1,3769,3869],[3777,0.019662921348314606,1,3771,3870],[3767,0.019662921348314606,1,3769,3871],[3787,0.019662921348314606,15,3771,3872],[3785,0.016853932584269662,370,3765,3873],[769,0.014044943820224719,25,3765,3874],[2661,0.014044943820224719,370,3765,3875],[3787,0.014044943820224719,15,3769,3876],[1573,0.011235955056179775,25,3771,3877],[2661,0.011235955056179775,370,3769,3878],[2661,0.011235955056179775,370,3771,3879],[3787,0.008426966292134831,15,3765,3880],[1651,0.008426966292134831,195,3769,3881],[1651,0.008426966292134831,195,3771,3882],[1651,0.0056179775280898875,195,3765,3883]]},{"id":"multi-swe-bench-go","name":"Multi-SWE-bench (Go)","group":"编程","unit":"%","metric":"Resolved rate","protocol":{"en":"Go Verified subset; verified leaderboard submissions, latest dated run per model/agent. Different language task sets are not averaged together.","zh":"Go Verified 子集；采用榜单已验证提交，各模型与 Agent 使用最新日期记录。不同语言任务集不混合平均。"},"source":{"name":"ByteDance Seed","url":"https://multi-swe-bench.github.io/","status":"current_snapshot","published_at":"2025-04-03"},"results":[[764,0.1261682242990654,83,3771,3884],[764,0.09813084112149532,83,3765,3885],[3767,0.07476635514018691,1,3771,3886],[3777,0.06775700934579439,1,3771,3887],[3785,0.06074766355140187,370,3771,3888],[764,0.06074766355140187,83,3769,3889],[3777,0.05841121495327103,1,3765,3890],[3767,0.05841121495327103,1,3769,3891],[1583,0.056074766355140186,66,3771,3892],[3767,0.053738317757009345,1,3765,3893],[1573,0.053738317757009345,25,3769,3894],[3777,0.0514018691588785,1,3769,3895],[3774,0.04672897196261682,15,3765,3896],[1573,0.04439252336448598,25,3765,3897],[3785,0.04439252336448598,370,3769,3898],[3774,0.04439252336448598,15,3769,3899],[3785,0.04205607476635514,370,3765,3900],[3779,0.0397196261682243,15,3765,3901],[3779,0.0397196261682243,15,3769,3902],[769,0.037383177570093455,25,3769,3903],[3774,0.037383177570093455,15,3771,3904],[3787,0.035046728971962614,15,3771,3905],[1583,0.035046728971962614,66,3769,3906],[3787,0.028037383177570093,15,3769,3907],[1583,0.028037383177570093,66,3765,3908],[3787,0.02336448598130841,15,3765,3909],[3779,0.02336448598130841,15,3771,3910],[769,0.02102803738317757,25,3765,3911],[2661,0.02102803738317757,370,3765,3912],[2661,0.02102803738317757,370,3769,3913],[2661,0.016355140186915886,370,3771,3914],[1651,0.014018691588785047,195,3769,3915],[1651,0.014018691588785047,195,3771,3916],[1573,0.007009345794392523,25,3771,3917],[1651,0.004672897196261682,195,3765,3918],[769,0.0,25,3771,3919]]},{"id":"multi-swe-bench-rust","name":"Multi-SWE-bench (Rust)","group":"编程","unit":"%","metric":"Resolved rate","protocol":{"en":"Rust Verified subset; verified leaderboard submissions, latest dated run per model/agent. Different language task sets are not averaged together.","zh":"Rust Verified 子集；采用榜单已验证提交，各模型与 Agent 使用最新日期记录。不同语言任务集不混合平均。"},"source":{"name":"ByteDance Seed","url":"https://multi-swe-bench.github.io/","status":"current_snapshot","published_at":"2025-04-03"},"results":[[3767,0.1589958158995816,1,3771,3920],[764,0.14644351464435146,83,3771,3921],[3777,0.12133891213389121,1,3771,3922],[764,0.100418410041841,83,3765,3923],[3779,0.0794979079497908,15,3769,3924],[3785,0.07112970711297072,370,3769,3925],[3774,0.07112970711297072,15,3769,3926],[3777,0.06694560669456066,1,3765,3927],[3767,0.06694560669456066,1,3765,3928],[769,0.06694560669456066,25,3769,3929],[1583,0.06276150627615062,66,3769,3930],[1573,0.058577405857740586,25,3765,3931],[3787,0.058577405857740586,15,3769,3932],[3767,0.05439330543933055,1,3769,3933],[764,0.05439330543933055,83,3769,3934],[3777,0.0502092050209205,1,3769,3935],[1573,0.0502092050209205,25,3769,3936],[1583,0.0502092050209205,66,3771,3937],[3779,0.0502092050209205,15,3765,3938],[3779,0.0502092050209205,15,3771,3939],[769,0.04602510460251046,25,3771,3940],[1573,0.04602510460251046,25,3771,3941],[2661,0.04184100418410042,370,3769,3942],[3774,0.04184100418410042,15,3765,3943],[3785,0.03347280334728033,370,3765,3944],[3785,0.03347280334728033,370,3771,3945],[3787,0.03347280334728033,15,3771,3946],[1583,0.02510460251046025,66,3765,3947],[3774,0.02510460251046025,15,3771,3948],[1651,0.02510460251046025,195,3769,3949],[769,0.02092050209205021,25,3765,3950],[3787,0.02092050209205021,15,3765,3951],[2661,0.016736401673640166,370,3765,3952],[1651,0.016736401673640166,195,3771,3953],[2661,0.008368200836820083,370,3771,3954],[1651,0.0041841004184100415,195,3765,3955]]},{"id":"multi-swe-bench-c","name":"Multi-SWE-bench (C)","group":"编程","unit":"%","metric":"Resolved rate","protocol":{"en":"C Verified subset; verified leaderboard submissions, latest dated run per model/agent. Different language task sets are not averaged together.","zh":"C Verified 子集；采用榜单已验证提交，各模型与 Agent 使用最新日期记录。不同语言任务集不混合平均。"},"source":{"name":"ByteDance Seed","url":"https://multi-swe-bench.github.io/","status":"current_snapshot","published_at":"2025-04-03"},"results":[[764,0.09375,83,3765,3956],[764,0.09375,83,3769,3957],[3767,0.0859375,1,3765,3958],[3767,0.0859375,1,3771,3959],[3767,0.0625,1,3769,3960],[764,0.0546875,83,3771,3961],[3777,0.046875,1,3765,3962],[3785,0.046875,370,3769,3963],[3785,0.0390625,370,3771,3964],[1583,0.0390625,66,3769,3965],[3774,0.0390625,15,3765,3966],[3777,0.03125,1,3771,3967],[1573,0.03125,25,3769,3968],[1573,0.03125,25,3771,3969],[1583,0.03125,66,3771,3970],[3774,0.03125,15,3769,3971],[3774,0.03125,15,3771,3972],[769,0.0234375,25,3771,3973],[1573,0.0234375,25,3765,3974],[2661,0.0234375,370,3765,3975],[3779,0.0234375,15,3765,3976],[3779,0.0234375,15,3769,3977],[3787,0.015625,15,3765,3978],[3787,0.015625,15,3769,3979],[3779,0.015625,15,3771,3980],[1651,0.015625,195,3765,3981],[1651,0.015625,195,3769,3982],[3777,0.0078125,1,3769,3983],[769,0.0078125,25,3765,3984],[769,0.0078125,25,3769,3985],[2661,0.0078125,370,3769,3986],[3785,0.0078125,370,3765,3987],[1651,0.0078125,195,3771,3988],[2661,0.0,370,3771,3989],[3787,0.0,15,3771,3990],[1583,0.0,66,3765,3991]]},{"id":"multi-swe-bench-cpp","name":"Multi-SWE-bench (C++)","group":"编程","unit":"%","metric":"Resolved rate","protocol":{"en":"C++ Verified subset; verified leaderboard submissions, latest dated run per model/agent. Different language task sets are not averaged together.","zh":"C++ Verified 子集；采用榜单已验证提交，各模型与 Agent 使用最新日期记录。不同语言任务集不混合平均。"},"source":{"name":"ByteDance Seed","url":"https://multi-swe-bench.github.io/","status":"current_snapshot","published_at":"2025-04-03"},"results":[[3767,0.14728682170542637,1,3771,3992],[3777,0.12403100775193798,1,3771,3993],[3767,0.11627906976744186,1,3765,3994],[764,0.09302325581395349,83,3771,3995],[764,0.08527131782945736,83,3765,3996],[1573,0.07751937984496124,25,3765,3997],[1573,0.07751937984496124,25,3771,3998],[3777,0.06976744186046512,1,3765,3999],[3787,0.06976744186046512,15,3769,4000],[1583,0.06976744186046512,66,3765,4001],[3779,0.06976744186046512,15,3771,4002],[769,0.06201550387596899,25,3765,4003],[2661,0.06201550387596899,370,3765,4004],[3785,0.05426356589147287,370,3771,4005],[1583,0.05426356589147287,66,3769,4006],[3774,0.05426356589147287,15,3769,4007],[3779,0.05426356589147287,15,3765,4008],[769,0.046511627906976744,25,3771,4009],[3785,0.046511627906976744,370,3765,4010],[3777,0.03875968992248062,1,3769,4011],[3785,0.03875968992248062,370,3769,4012],[3787,0.03875968992248062,15,3771,4013],[3774,0.03875968992248062,15,3765,4014],[3774,0.03875968992248062,15,3771,4015],[3767,0.031007751937984496,1,3769,4016],[769,0.031007751937984496,25,3769,4017],[2661,0.031007751937984496,370,3771,4018],[1583,0.031007751937984496,66,3771,4019],[3787,0.023255813953488372,15,3765,4020],[764,0.023255813953488372,83,3769,4021],[1651,0.023255813953488372,195,3771,4022],[1573,0.015503875968992248,25,3769,4023],[3779,0.015503875968992248,15,3769,4024],[1651,0.007751937984496124,195,3769,4025],[2661,0.0,370,3769,4026],[1651,0.0,195,3765,4027]]},{"id":"explorationbench-aliencode","name":"ExplorationBench (AlienCode)","group":"数学与科学","unit":"%","metric":"Mean@3 held-out accuracy","protocol":{"en":"AlienCode: held-out accuracy after four exploration rounds (M4), averaged across three independent trajectories; each task is answered three times. Highest reasoning setting. Not the best@3 column.","zh":"AlienCode：四轮探索后（M4）的保留题正确率，三条独立轨迹取平均，每题回答三次；使用最高推理设置，不读取 best@3 列。"},"source":{"name":"Tencent Hunyuan","url":"https://explorationbench.com/","status":"current_snapshot","published_at":"2026-09-24"},"results":[[762,0.795,15,4028,-1],[755,0.735,1,4028,-1],[3157,0.64,195,4028,-1],[2974,0.532,143,4028,-1],[3153,0.46299999999999997,25,4028,-1],[2983,0.397,83,4028,-1],[2977,0.387,21,4028,-1],[4029,0.302,370,4028,-1],[4030,0.248,370,4028,-1],[3185,0.073,25,4028,-1]]},{"id":"explorationbench-alienlogic","name":"ExplorationBench (AlienLogic)","group":"数学与科学","unit":"%","metric":"Mean@3 held-out accuracy","protocol":{"en":"AlienLogic: held-out accuracy after four exploration rounds (M4), averaged across three independent trajectories; each task is answered three times. Highest reasoning setting. Not the best@3 column.","zh":"AlienLogic：四轮探索后（M4）的保留题正确率，三条独立轨迹取平均，每题回答三次；使用最高推理设置，不读取 best@3 列。"},"source":{"name":"Tencent Hunyuan","url":"https://explorationbench.com/","status":"current_snapshot","published_at":"2026-09-24"},"results":[[755,0.759,1,4028,-1],[762,0.7509999999999999,15,4028,-1],[3157,0.743,195,4028,-1],[2974,0.721,143,4028,-1],[2977,0.6940000000000001,21,4028,-1],[4029,0.6920000000000001,370,4028,-1],[3185,0.6629999999999999,25,4028,-1],[3153,0.606,25,4028,-1],[4030,0.598,370,4028,-1],[2983,0.59,83,4028,-1]]},{"id":"google-facts-v1","name":"FACTS Benchmark Suite (v1)","group":"知识与推理","unit":"score","metric":"FACTS composite score","protocol":{"en":"Official Kaggle board, benchmark version 1; current mapped task version 108. Average across FACTS components, shown on 0–100 and excluded from saturation bins. Requires all four currently mapped component versions; older component mixtures are omitted.","zh":"官方 Kaggle 榜单，评测版本 1，当前映射的任务版本 108。FACTS 分项平均，显示为 0–100 综合分，不进入饱和度分组；须具备四个当前分项版本的成绩，排除旧版分项混用。"},"source":{"name":"Google DeepMind / Kaggle","url":"https://www.kaggle.com/benchmarks/google/facts/leaderboard","status":"current_snapshot","published_at":"2025-12-11"},"score_max":100,"results":[[2958,75.39661520133869,15,4031,-1],[2960,74.60844650396427,15,4031,-1],[2983,68.47424679788553,83,4031,-1],[2959,64.0159443872485,1,4031,-1],[2976,62.34009939071006,15,4031,-1]]},{"id":"google-facts-grounding-v3","name":"FACTS Grounding (v3)","group":"知识与推理","unit":"%","metric":"Accuracy","protocol":{"en":"Official Kaggle board, benchmark version 3; current mapped task version 1109072. Published fractional accuracy; missing model results are omitted.","zh":"官方 Kaggle 榜单，评测版本 3，当前映射的任务版本 1109072。官方正确率；未公布成绩的模型不填分。"},"source":{"name":"Google DeepMind / Kaggle","url":"https://www.kaggle.com/benchmarks/google/facts-grounding/leaderboard","status":"current_snapshot","published_at":"2024-12-17"},"results":[[2958,0.8011661807580175,15,4031,-1],[2960,0.780466472303207,15,4031,-1],[2593,0.7778425655976676,83,4031,-1],[2976,0.7670553935860058,15,4031,-1],[2585,0.7583090379008747,83,4031,-1],[3560,0.7379008746355685,4032,4031,-1],[2962,0.7335276967930029,83,4031,-1],[2968,0.6860058309037901,143,4031,-1],[2983,0.6816326530612244,83,4031,-1],[2959,0.6244897959183674,1,4031,-1],[2973,0.6052478134110787,15,4031,-1],[2970,0.5696793002915452,1,4031,-1],[3482,0.5661807580174927,83,4031,-1],[3396,0.560932944606414,48,4031,-1],[62,0.20145772594752187,15,4031,-1]]},{"id":"google-facts-multimodal-v1","name":"FACTS Multimodal (v1)","group":"知识与推理","unit":"%","metric":"Accuracy","protocol":{"en":"Official Kaggle board, benchmark version 1; current mapped task version 40126. Published fractional accuracy; missing model results are omitted.","zh":"官方 Kaggle 榜单，评测版本 1，当前映射的任务版本 40126。官方正确率；未公布成绩的模型不填分。"},"source":{"name":"Google DeepMind / Kaggle","url":"https://www.kaggle.com/benchmarks/google/facts-multimodal/leaderboard","status":"current_snapshot","published_at":"2025-12-11"},"results":[[2983,0.5069953364423717,83,4031,-1],[3386,0.49833666001330673,83,4031,-1],[2960,0.48462566844919786,15,4031,-1],[2959,0.48367346938775513,1,4031,-1],[2958,0.4773030707610147,15,4031,-1],[3405,0.4720744680851064,83,4031,-1],[2602,0.46860820194999997,83,4031,-1],[3196,0.4576948700866089,15,4031,-1],[2058,0.44078298395000004,15,4031,-1],[3394,0.43057409879839786,15,4031,-1],[3748,0.418010359,15,4031,-1],[2055,0.4160572294460662,83,4031,-1],[3756,0.4147483514,15,4031,-1],[3415,0.4143904063957361,83,4031,-1],[3482,0.4134228187919463,83,4031,-1],[4033,0.4128256513026052,83,4031,-1],[3184,0.40975935828877,15,4031,-1],[345,0.4054397825191458,15,4031,-1],[1464,0.40204709409064726,15,4031,-1],[3479,0.397719651240778,1,4031,-1],[3738,0.39721688358286694,1,4031,-1],[340,0.3966442953020134,15,4031,-1],[4034,0.3935957304869913,83,4031,-1],[2261,0.3675385647216633,1,4031,-1],[3578,0.3584277148567621,1,4031,-1],[4035,0.3583776595744681,83,4031,-1],[3408,0.35809806835066865,15,4031,-1],[4036,0.35432962139987784,15,4031,-1],[2976,0.342456608811749,15,4031,-1],[4037,0.3331880154823186,1,4031,-1],[3572,0.3255033557046979,1,4031,-1],[4038,0.3106767038,1,4031,-1],[323,0.28625754285,1,4031,-1],[3428,0.2570295860890035,143,4031,-1],[2585,0.2471685542971352,83,4031,-1],[4039,0.2463069933,83,4031,-1],[2593,0.2385076615589607,83,4031,-1],[4040,0.21170987614999998,417,4031,-1],[4041,0.2080870178,83,4031,-1],[4042,0.1773702850016814,143,4031,-1],[4043,0.1363804586,83,4031,-1]]},{"id":"google-facts-parametric-v2","name":"FACTS Parametric (v2)","group":"知识与推理","unit":"%","metric":"Accuracy","protocol":{"en":"Official Kaggle board, benchmark version 2; current mapped task version 1118505. Published fractional accuracy; missing model results are omitted.","zh":"官方 Kaggle 榜单，评测版本 2，当前映射的任务版本 1118505。官方正确率；未公布成绩的模型不填分。"},"source":{"name":"Google DeepMind / Kaggle","url":"https://www.kaggle.com/benchmarks/google/facts-parametric/leaderboard","status":"current_snapshot","published_at":"2025-12-11"},"results":[[2958,0.8382446134347276,15,4031,-1],[2960,0.8376108998732572,15,4031,-1],[762,0.8109949302915082,15,4031,-1],[2983,0.7870722433460076,83,4031,-1],[3394,0.7612484157160964,15,4031,-1],[3405,0.7389100126742713,83,4031,-1],[4033,0.7240177439797212,83,4031,-1],[2962,0.7224334600760456,83,4031,-1],[2959,0.5872940430925222,1,4031,-1],[2973,0.5758871989860583,15,4031,-1],[2976,0.5698669201520913,15,4031,-1],[2968,0.5427756653992395,143,4031,-1],[3408,0.5063371356147022,15,4031,-1],[755,0.48748415716096327,1,4031,-1],[2974,0.44470849176172367,143,4031,-1],[4035,0.43029150823827633,83,4031,-1],[3391,0.4245880861850444,143,4031,-1],[2961,0.35107731305449935,1,4031,-1],[3568,0.2046894803548796,1,4031,-1],[2970,0.1308618504435995,1,4031,-1]]},{"id":"google-facts-search-v2","name":"FACTS Search V2","group":"知识与推理","unit":"%","metric":"Accuracy","protocol":{"en":"Official Kaggle board, benchmark version 2; current mapped task version 41923. Published fractional accuracy; missing model results are omitted.","zh":"官方 Kaggle 榜单，评测版本 2，当前映射的任务版本 41923。官方正确率；未公布成绩的模型不填分。"},"source":{"name":"Google DeepMind / Kaggle","url":"https://www.kaggle.com/benchmarks/google/facts-search/leaderboard","status":"current_snapshot","published_at":"2025-12-11"},"results":[[2958,0.8991507430997877,15,4031,-1],[762,0.8848195329087049,15,4031,-1],[2960,0.8816348195329087,15,4031,-1],[2959,0.8651804670912951,1,4031,-1],[3394,0.8614649681528662,15,4031,-1],[755,0.8527379053694844,1,4031,-1],[2962,0.8401924104756815,83,4031,-1],[3482,0.8370488322717622,83,4031,-1],[2973,0.8174097664543525,15,4031,-1],[2976,0.8142250530785563,15,4031,-1],[4033,0.8092948717948718,83,4031,-1],[3386,0.8089171974522293,83,4031,-1],[3479,0.79596174282678,1,4031,-1],[3391,0.7943676939426142,143,4031,-1],[3408,0.7935244161358811,15,4031,-1],[2970,0.7749469214437368,1,4031,-1],[2983,0.7632696390658175,83,4031,-1],[3405,0.6819148936170213,83,4031,-1],[2585,0.6814044213263979,83,4031,-1],[3415,0.6427795874049945,83,4031,-1],[4035,0.6342887473460722,83,4031,-1],[2593,0.6031317494600432,83,4031,-1],[4044,0.5335106382978724,4032,4031,-1]]},{"id":"tau3-banking-sierra","name":"τ³-Banking (Sierra)","group":"Agent 与工具","unit":"%","metric":"Pass@1","protocol":{"en":"Sierra standard submissions, banking_knowledge, v1.0.1 re-grading. Pass@1; retrieval, reasoning and user-simulator settings remain per result.","zh":"Sierra 标准提交，banking_knowledge、v1.0.1 重评分；使用 pass@1，逐条保留检索方式、推理与用户模拟器版本。"},"source":{"name":"Sierra","url":"https://taubench.com/leaderboard?benchmark=knowledge","status":"current_snapshot","published_at":"2026-03-18"},"results":[[3560,0.5515463917525774,50,4045,-1],[755,0.48711340206185566,1,4046,-1],[3391,0.4793814432989691,143,4047,-1],[4048,0.46907216494845355,15,4045,-1],[3196,0.44587628865979384,15,4045,-1],[3413,0.40463917525773196,66,4045,-1],[3578,0.4020618556701031,1,4046,-1],[3453,0.3969072164948454,1,4046,-1],[3479,0.3969072164948454,1,4046,-1],[3184,0.3943,15,4045,-1],[227,0.3711340206185567,48,4045,-1],[2977,0.3711340206185567,21,4046,-1],[340,0.3222,15,4047,-1],[2261,0.2732,1,4046,-1],[3424,0.2732,83,4049,-1],[3482,0.26030000000000003,83,4047,-1],[283,0.2526,1,4050,-1],[203,0.25,3471,4046,-1],[3738,0.24739999999999998,1,4047,-1],[26,0.1804,83,4049,-1],[4051,0.1804,143,4047,-1],[4052,0.1572,143,4047,-1],[2602,0.1366,83,4047,-1],[4053,0.13140000000000002,143,4047,-1],[340,0.1263,15,4054,-1],[268,0.09789999999999999,633,4055,-1],[2576,0.09789999999999999,936,4055,-1]]},{"id":"workarena-l1","name":"WorkArena (L1)","group":"Agent 与工具","unit":"%","metric":"Task success","protocol":{"en":"Official BrowserGym submissions for WorkArena L1. Latest dated result per model/agent/configuration. Submitted protocol changes, including step budgets, remain in result details.","zh":"官方 BrowserGym 提交，WorkArena L1；各模型、Agent 与配置采用最新日期记录。提交者声明的步骤预算等协议变化保留于成绩详情。"},"source":{"name":"ServiceNow","url":"https://huggingface.co/spaces/ServiceNow/browsergym-leaderboard","status":"current_snapshot","published_at":"2024-02-15"},"results":[[4056,0.903,83,4057,4058],[2058,0.7909999999999999,15,4059,4060],[4061,0.633,1,4062,4063],[4064,0.606,15,4062,4065],[4066,0.564,1,4062,4067],[4068,0.518,15,4062,4069],[4070,0.515,195,4071,4072],[4073,0.509,15,4062,4074],[4075,0.48200000000000004,15,4062,4076],[353,0.455,15,4062,4077],[4078,0.433,66,4062,4079],[4080,0.406,15,4062,4081],[4082,0.385,15,4062,4083],[4084,0.3829,370,4062,4085],[4086,0.27899999999999997,66,4062,4087],[354,0.27,15,4062,4088]]},{"id":"workarena-l2","name":"WorkArena++ (L2)","group":"Agent 与工具","unit":"%","metric":"Task success","protocol":{"en":"Official BrowserGym submissions for WorkArena L2. Latest dated result per model/agent/configuration. Submitted protocol changes, including step budgets, remain in result details.","zh":"官方 BrowserGym 提交，WorkArena L2；各模型、Agent 与配置采用最新日期记录。提交者声明的步骤预算等协议变化保留于成绩详情。"},"source":{"name":"ServiceNow","url":"https://huggingface.co/spaces/ServiceNow/browsergym-leaderboard","status":"current_snapshot","published_at":"2024-07-07"},"results":[[2058,0.6940000000000001,15,4062,4060],[4064,0.47700000000000004,15,4062,4065],[4061,0.40399999999999997,1,4062,4063],[4066,0.391,1,4062,4067],[4068,0.149,15,4062,4069],[4073,0.115,15,4062,4074],[4070,0.106,195,4089,4072],[353,0.085,15,4062,4077],[4078,0.07200000000000001,66,4062,4079],[4080,0.034,15,4062,4081],[4084,0.0298,370,4062,4085],[4082,0.026000000000000002,15,4062,4083],[4086,0.021,66,4062,4087],[354,0.013000000000000001,15,4062,4088]]},{"id":"workarena-l3","name":"WorkArena++ (L3)","group":"Agent 与工具","unit":"%","metric":"Task success","protocol":{"en":"Official BrowserGym submissions for WorkArena L3. Latest dated result per model/agent/configuration. Submitted protocol changes, including step budgets, remain in result details.","zh":"官方 BrowserGym 提交，WorkArena L3；各模型、Agent 与配置采用最新日期记录。提交者声明的步骤预算等协议变化保留于成绩详情。"},"source":{"name":"ServiceNow","url":"https://huggingface.co/spaces/ServiceNow/browsergym-leaderboard","status":"current_snapshot","published_at":"2024-07-07"},"results":[[2058,0.115,15,4090,4060],[4066,0.004,1,4062,4067],[4084,0.0,370,4062,4085],[353,0.0,15,4062,4077],[354,0.0,15,4062,4088],[4068,0.0,15,4062,4069],[4078,0.0,66,4062,4079],[4086,0.0,66,4062,4087]]},{"id":"paperbench-baselines","name":"PaperBench (2025)","group":"专业工作","unit":"score","metric":"Replication score","protocol":{"en":"Latest published author table contains the 2025-04-02 paper baselines. Default-budget full replication runs; longer-budget and Code-Dev variants are excluded. Mean rubric score on a 0–100 scale.","zh":"作者目前公开表格仍为 2025-04-02 论文基线。使用默认预算的完整复现成绩；排除更长预算与 Code-Dev。保留 0–100 的平均量规分数。"},"source":{"name":"OpenAI / PaperBench","url":"https://github.com/openai/frontier-evals/tree/main/project/paperbench","status":"paper_baselines","published_at":"2025-04-02"},"public_history":true,"results":[[4091,21.0,1,4092,-1],[4093,16.1,1,4092,-1],[4094,13.2,15,4092,-1],[4095,8.5,15,4092,-1],[4096,6.0,25,4092,-1],[4097,4.1,15,4092,-1],[4098,3.2,83,4092,-1],[4099,2.6,15,4092,-1]]},{"id":"ruler-v1-128k-history","name":"RULER v1 · 128K (2024–2025)","group":"指令与长文","unit":"%","metric":"128K accuracy","public_history":true,"protocol":{"en":"Historical 2024–2025 results. Original RULER v1 leaderboard, 128K context column; model sizes/settings remain in the original labels. Distinct from RULER v2.","zh":"2024–2025 年历史成绩。RULER v1 原榜单的 128K 上下文列；模型规模与设置沿用来源标签。与 RULER v2 分开。"},"source":{"name":"NVIDIA","url":"https://github.com/NVIDIA/RULER","status":"dated_release","published_at":"2024-04-09"},"results":[[4100,0.951,370,4101,-1],[4102,0.9440000000000001,83,4101,-1],[4103,0.922,195,4101,-1],[4104,0.9059999999999999,195,4101,-1],[4105,0.9,370,4101,-1],[4106,0.882,370,4101,-1],[4107,0.856,195,4101,-1],[4108,0.851,195,4101,-1],[4109,0.8440000000000001,195,4101,-1],[4110,0.8370000000000001,370,4101,-1],[4111,0.831,551,4101,-1],[4112,0.816,370,4101,-1],[4113,0.812,15,4101,-1],[4114,0.792,195,4101,-1],[4115,0.774,195,4101,-1],[4116,0.773,370,4101,-1],[4117,0.77,66,4101,-1],[4118,0.76,370,4101,-1],[4119,0.747,370,4101,-1],[4120,0.721,370,4101,-1],[4121,0.695,370,4101,-1],[4122,0.6890000000000001,1820,4101,-1],[4123,0.667,370,4101,-1],[4124,0.6659999999999999,66,4101,-1],[4125,0.66,195,4101,-1],[4126,0.65,370,4101,-1],[4127,0.6459999999999999,370,4101,-1],[4128,0.631,370,4101,-1],[4129,0.537,195,4101,-1],[4130,0.48100000000000004,417,4101,-1],[4131,0.461,370,4101,-1],[4132,0.445,417,4101,-1],[4133,0.42,551,4101,-1],[4134,0.317,417,4101,-1],[4135,0.271,370,4101,-1],[4136,0.237,417,4101,-1],[4137,0.19,417,4101,-1],[4138,0.138,417,4101,-1],[4139,0.0,370,4101,-1],[4140,0.0,370,4101,-1],[4141,0.0,370,4101,-1],[4142,0.0,195,4101,-1],[4143,0.0,370,4101,-1]]},{"id":"u-math-paper","name":"U-MATH (2024)","group":"数学与科学","unit":"%","metric":"Overall accuracy","public_history":true,"protocol":{"en":"Historical 2024 results. Paper Table 4, overall U-MATH accuracy; text-only and multimodal model settings remain separate.","zh":"2024 年历史成绩。论文表 4 的 U-MATH 总体正确率；纯文本与多模态设置分别保留。"},"source":{"name":"Toloka","url":"https://github.com/Toloka/u-math","status":"paper_baselines","published_at":"2024-12-04"},"results":[[4144,0.4889,83,4145,-1],[4146,0.41159999999999997,195,4147,-1],[353,0.3653,15,4145,-1],[4148,0.35200000000000004,195,4147,-1],[792,0.3404,195,4147,-1],[4149,0.33240000000000003,195,4147,-1],[797,0.3218,195,4147,-1],[4150,0.2862,195,4147,-1],[2750,0.2693,195,4145,-1],[542,0.2613,66,4147,-1],[2485,0.17329999999999998,195,4145,-1],[568,0.1636,370,4147,-1],[4151,0.1564,370,4145,-1],[587,0.1547,66,4147,-1],[4152,0.14400000000000002,370,4145,-1]]},{"id":"financebench-paper","name":"FinanceBench (2023)","group":"专业工作","unit":"%","metric":"Reported accuracy","public_history":true,"protocol":{"en":"Historical 2023 results. Original 150-question human evaluation. Table 2 reports rounded percentages for eight configurations. Figures 2(ii) and 5 give eight more as correct answers out of 150. Long context and oracle use context-first unless marked context-last.","zh":"2023 年历史成绩。原始 150 题人工评测。表 2 的八种配置保留原表舍入百分比，图 2(ii) 与图 5 的另八种配置按答对题数除以 150。长上下文与 oracle 设置默认将上下文前置，后置设置逐条标明。"},"source":{"name":"Patronus AI","url":"https://arxiv.org/html/2311.11944v1#S5.T2","status":"paper_baselines","published_at":"2023-11-20"},"results":[[522,0.8933333333333333,15,4153,4154],[522,0.85,15,4155,-1],[2089,0.84,15,4155,4156],[522,0.79,15,4157,-1],[2089,0.7866666666666666,15,4153,4156],[4158,0.76,1,4157,-1],[522,0.5,15,4159,-1],[2089,0.42,15,4159,4156],[2042,0.41,66,4159,-1],[4158,0.37333333333333335,1,4160,4154],[522,0.24666666666666667,15,4160,4154],[522,0.19,15,4161,-1],[2042,0.19,66,4161,-1],[2089,0.16666666666666666,15,4161,4156],[522,0.09,15,4162,-1],[2089,0.04666666666666667,15,4162,4156]]},{"id":"crmarena-pro-b2b-single-turn-paper","name":"CRMArena-Pro · B2B, single-turn (2025)","group":"专业工作","unit":"%","metric":"Overall task success","public_history":true,"protocol":{"en":"Historical 2025 results. B2B, single-turn; original paper overall column. B2B/B2C and dialogue settings are not pooled.","zh":"2025 年历史成绩。B2B、单轮；原论文总体列。B2B、B2C 与不同对话设置不混合。"},"source":{"name":"Salesforce","url":"https://github.com/SalesforceAIResearch/CRMArena","status":"paper_baselines","published_at":"2025-05-24"},"results":[[457,0.541,83,4163,-1],[2816,0.475,15,4163,-1],[4164,0.419,83,4163,-1],[4165,0.331,83,4163,-1],[4166,0.326,66,4163,-1],[4167,0.287,66,4163,-1],[4168,0.267,15,4163,-1],[4169,0.252,66,4163,-1],[4170,0.213,15,4163,-1]]},{"id":"crmarena-pro-b2c-single-turn-paper","name":"CRMArena-Pro · B2C, single-turn (2025)","group":"专业工作","unit":"%","metric":"Overall task success","public_history":true,"protocol":{"en":"Historical 2025 results. B2C, single-turn; original paper overall column. B2B/B2C and dialogue settings are not pooled.","zh":"2025 年历史成绩。B2C、单轮；原论文总体列。B2B、B2C 与不同对话设置不混合。"},"source":{"name":"Salesforce","url":"https://github.com/SalesforceAIResearch/CRMArena","status":"paper_baselines","published_at":"2025-05-24"},"results":[[457,0.583,83,4171,-1],[2816,0.495,15,4171,-1],[4164,0.467,83,4171,-1],[4166,0.375,66,4171,-1],[4165,0.365,83,4171,-1],[4168,0.292,15,4171,-1],[4167,0.265,66,4171,-1],[4169,0.249,66,4171,-1],[4170,0.18600000000000003,15,4171,-1]]},{"id":"crmarena-pro-b2b-multi-turn-paper","name":"CRMArena-Pro · B2B, multi-turn (2025)","group":"专业工作","unit":"%","metric":"Overall task success","public_history":true,"protocol":{"en":"Historical 2025 results. B2B, multi-turn; original paper overall column. B2B/B2C and dialogue settings are not pooled.","zh":"2025 年历史成绩。B2B、多轮；原论文总体列。B2B、B2C 与不同对话设置不混合。"},"source":{"name":"Salesforce","url":"https://github.com/SalesforceAIResearch/CRMArena","status":"paper_baselines","published_at":"2025-05-24"},"results":[[457,0.35100000000000003,83,4172,-1],[2816,0.325,15,4172,-1],[4164,0.301,83,4172,-1],[4165,0.26899999999999996,83,4172,-1],[4166,0.237,66,4172,-1],[4167,0.21,66,4172,-1],[4168,0.19699999999999998,15,4172,-1],[4170,0.193,15,4172,-1],[4169,0.163,66,4172,-1]]},{"id":"crmarena-pro-b2c-multi-turn-paper","name":"CRMArena-Pro · B2C, multi-turn (2025)","group":"专业工作","unit":"%","metric":"Overall task success","public_history":true,"protocol":{"en":"Historical 2025 results. B2C, multi-turn; original paper overall column. B2B/B2C and dialogue settings are not pooled.","zh":"2025 年历史成绩。B2C、多轮；原论文总体列。B2B、B2C 与不同对话设置不混合。"},"source":{"name":"Salesforce","url":"https://github.com/SalesforceAIResearch/CRMArena","status":"paper_baselines","published_at":"2025-05-24"},"results":[[2816,0.304,15,4173,-1],[457,0.3,83,4173,-1],[4165,0.23399999999999999,83,4173,-1],[4164,0.23399999999999999,83,4173,-1],[4167,0.2,66,4173,-1],[4168,0.183,15,4173,-1],[4170,0.158,15,4173,-1],[4169,0.151,66,4173,-1],[4166,0.149,66,4173,-1]]},{"id":"windows-agent-arena-paper","name":"Windows Agent Arena (2024)","group":"Agent 与工具","unit":"%","metric":"Task success","public_history":true,"protocol":{"en":"Historical 2024 results. 154 Windows tasks; Navi with UIA tree + Omniparser. Paper Table 4 Total column; human baseline and Mind2Web results excluded.","zh":"2024 年历史成绩。154 道 Windows 任务；Navi，UIA tree 与 Omniparser。论文表 4 Total 列；不含人类基线或 Mind2Web 成绩。"},"source":{"name":"Microsoft","url":"https://microsoft.github.io/WindowsAgentArena","status":"paper_baselines","published_at":"2024-09-12"},"results":[[4174,0.195,15,4175,-1],[353,0.086,15,4175,-1],[354,0.042,15,4175,-1],[4176,0.035,370,4175,-1]]},{"id":"mle-bench-2026-04-board","name":"MLE-bench (2026-04)","group":"编程","unit":"%","metric":"Mean any-medal rate","public_history":true,"protocol":{"en":"Historical 2026-04 results. Main all-complexity leaderboard, mean any-medal rate. The publisher paused submissions on 2026-04-24. Run budgets remain per result; non-comparable additional submissions are excluded.","zh":"2026-04 年历史成绩。主榜单全部难度的平均获奖牌率。发布方于 2026-04-24 暂停收录；逐条保留运行预算，不纳入另列的不可比提交。"},"source":{"name":"OpenAI","url":"https://github.com/openai/mle-bench","status":"archived","published_at":"2026-04-15"},"results":[[3002,0.6444,83,4177,-1],[3183,0.6311,1,4178,-1],[3002,0.6267,83,4179,-1],[3002,0.6133,83,4180,-1],[3002,0.6133,83,4181,-1],[764,0.5956,83,4177,-1],[4182,0.5644,25,4183,-1],[3002,0.56,83,4184,-1],[3002,0.52,83,4185,-1],[3002,0.5067,83,4186,-1],[4187,0.4844,15,4188,-1],[764,0.44,83,4189,-1],[764,0.43560000000000004,83,4190,-1],[4191,0.3956,15,4192,-1],[764,0.38670000000000004,83,4193,-1],[4194,0.3644,25,4195,-1],[4196,0.35109999999999997,15,4197,-1],[4198,0.3422,370,4199,-1],[1464,0.316,15,4200,-1],[4201,0.30219999999999997,370,4202,-1],[4194,0.2933,25,4203,-1],[772,0.22399999999999998,15,4202,-1],[772,0.17120000000000002,15,4204,-1],[493,0.0863,15,4204,-1],[4205,0.0756,1,4204,-1],[493,0.0489,15,4206,-1],[4207,0.0333,66,4204,-1],[493,0.016,15,4208,-1]]},{"id":"swe-lancer-diamond-paper","name":"SWE-Lancer Diamond (2025)","group":"编程","unit":"%","metric":"Pass@1","public_history":true,"protocol":{"en":"Historical 2025 results. Original 502-task Diamond set, IC + manager tasks, paper v4 Tables 1 and 11 pass@1. The two open-weight runs use the same scaffold, with one seed and preliminary status. The July 198-task offline IC release is separate.","zh":"2025 年历史成绩。原始 502 题 Diamond 集，IC 与管理任务合计，论文 v4 表 1 与表 11 的 pass@1。两项开放权重模型沿用相同框架，各运行一个种子，属初步结果。7 月的 198 题离线 IC 另为一批。"},"source":{"name":"OpenAI","url":"https://arxiv.org/html/2502.12115v4","status":"paper_baselines","published_at":"2025-02-17"},"results":[[4209,0.361,370,4210,-1],[2816,0.297,15,4211,-1],[796,0.235,66,4212,4213],[353,0.233,15,4210,-1],[4214,0.231,25,4212,4213]]},{"id":"swe-lancer-full-paper","name":"SWE-Lancer Full (2025)","group":"编程","unit":"%","metric":"Pass@1","public_history":true,"protocol":{"en":"Historical 2025 results. Original 1488-task Full set, IC + manager tasks, paper v4 Tables 1 and 11 pass@1. The two open-weight runs use the same scaffold, with one seed and preliminary status. The July 198-task offline IC release is separate.","zh":"2025 年历史成绩。原始 1488 题 Full 集，IC 与管理任务合计，论文 v4 表 1 与表 11 的 pass@1。两项开放权重模型沿用相同框架，各运行一个种子，属初步结果。7 月的 198 题离线 IC 另为一批。"},"source":{"name":"OpenAI","url":"https://arxiv.org/html/2502.12115v4","status":"paper_baselines","published_at":"2025-02-17"},"results":[[4209,0.337,370,4215,-1],[2816,0.32899999999999996,15,4216,-1],[4214,0.254,25,4217,4213],[796,0.245,66,4217,4213],[353,0.233,15,4215,-1]]},{"id":"clawsbench-openclaw-paper","name":"ClawsBench (2026-04)","group":"Agent 与工具","unit":"%","metric":"Non-safety task success rate","public_history":true,"protocol":{"en":"Historical 2026-04 results. Paper Table 4: OpenClaw, domain skills and meta prompt both on. Non-safety task success requires task score ≥0.8; this is TSR, not unsafe-action rate or safe-completion rate.","zh":"2026-04 年历史成绩。论文表 4：OpenClaw，领域 Skills 与元提示均开启。非安全任务得分 ≥0.8 计成功；使用 TSR，不使用不安全动作率或安全任务完成率。"},"source":{"name":"BenchFlow","url":"https://benchflow-ai.github.io/ClawsBench/","status":"paper_baselines","published_at":"2026-04-06"},"results":[[2261,0.63,1,4218,-1],[268,0.6,551,4218,-1],[3412,0.58,83,4218,-1],[3572,0.56,1,4218,-1],[3184,0.53,15,4218,-1],[4219,0.39,83,4218,-1]]},{"id":"saas-bench-paper-resolved","name":"SaaS-Bench (Resolved, May 2026 paper)","group":"Agent 与工具","unit":"%","metric":"Resolved","source":{"name":"UniPat AI","url":"https://arxiv.org/html/2605.15777v2#S4.T2","status":"paper_baselines","published_at":"2026-05-15"},"protocol":{"en":"Paper Figure 1, Browser-Use. Eleven models ran all 106 tasks. The three marked text-only ran 74 tasks. A task counts as resolved when every checkpoint passes.","zh":"论文图 1 使用 Browser-Use。11 个模型评测全部 106 道任务，标有「仅文本」的 3 个模型评测 74 道纯文本任务。任务的所有检查点均通过才算完成。"},"public_history":true,"results":[[3578,0.038,1,4220,4221,"full"],[4222,0.038,15,4220,4223,"full"],[2261,0.019,1,4220,4224,"full"],[4225,0.019,3026,4220,4226,"full"],[4227,0.019,15,4220,4228,"full"],[3603,0.019,195,4220,4229,"full"],[3492,0.013999999999999999,25,4230,4231,"text_only"],[3572,0.009000000000000001,1,4220,4232,"full"],[4233,0.009000000000000001,83,4220,4234,"full"],[3490,0.009000000000000001,21,4220,4235,"full"],[3186,0.0,551,4230,4236,"text_only"],[3412,0.0,83,4220,4237,"full"],[244,0.0,21,4220,4238,"full"],[3495,0.0,775,4230,4239,"text_only"]]},{"id":"saas-bench-paper-checkpoint","name":"SaaS-Bench (Checkpoint, May 2026 paper)","group":"Agent 与工具","unit":"score","metric":"Checkpoint","source":{"name":"UniPat AI","url":"https://arxiv.org/html/2605.15777v2#S4.T2","status":"paper_baselines","published_at":"2026-05-15"},"protocol":{"en":"Paper Figure 1, Browser-Use. Eleven models ran all 106 tasks. The three marked text-only ran 74 tasks. Scores are weighted checkpoint credit on a 0–100 scale.","zh":"论文图 1 使用 Browser-Use。11 个模型评测全部 106 道任务，标有「仅文本」的 3 个模型评测 74 道纯文本任务。检查点按权重计分，满分 100。"},"public_history":true,"score_max":100,"results":[[3578,43.9,1,4220,4221,"full"],[4227,43.8,15,4220,4228,"full"],[2261,43.2,1,4220,4224,"full"],[4222,37.0,15,4220,4223,"full"],[3490,34.1,21,4220,4235,"full"],[3603,29.9,195,4220,4229,"full"],[244,27.7,21,4220,4238,"full"],[4225,27.1,3026,4220,4226,"full"],[3412,27.1,83,4220,4237,"full"],[3572,23.3,1,4220,4232,"full"],[4233,23.3,83,4220,4234,"full"],[3492,21.5,25,4230,4231,"text_only"],[3186,17.4,551,4230,4236,"text_only"],[3495,15.8,775,4230,4239,"text_only"]]},{"id":"osworld-2-20260624-full","name":"OSWorld 2.0 (2026.06.24, full)","group":"Agent 与工具","unit":"%","metric":"Binary task completion","protocol":{"en":"Official end-to-end model runs, 500-step budget. Score is full task completion, not partial reward; reasoning settings are retained.","zh":"官方端到端模型运行，最多 500 步。使用完整任务完成率，不使用部分奖励；保留推理设置。"},"source":{"name":"OSWorld / XLANG Lab","url":"https://osworld-v2.xlang.ai/","status":"dated_release","published_at":"2026-06-24"},"public_history":true,"results":[[3479,0.20600000000000002,1,4240,-1],[3479,0.1852,1,4241,-1],[3578,0.182,1,4240,-1],[3578,0.139,1,4241,-1],[3196,0.13,15,759,-1],[3572,0.09300000000000001,1,4242,-1],[3572,0.083,1,4241,-1],[4243,0.046,21,4244,-1],[3491,0.046,775,4244,-1],[4245,0.027999999999999997,195,4246,-1]]},{"id":"monthly-swebench-2026-03","name":"Monthly-SWEBench (2026-03)","group":"编程","unit":"%","metric":"Pass@1","source":{"name":"UniPat AI","url":"https://unipat.ai/benchmarks/MonthlySWEBench","status":"dated_release","published_at":"2026-03-31"},"protocol":{"en":"2026-03 Overall, 112 tasks. The harness is recorded per result.","zh":"2026-03 总榜共 112 道任务，各行保留执行框架。"},"public_history":true,"results":[[3183,0.75,1,4247,-1],[3186,0.6696,633,4247,-1],[4248,0.643,15,4247,-1],[3270,0.5714,195,4247,-1],[502,0.5446,83,4247,-1],[268,0.5,633,4247,-1],[2998,0.5,21,4247,-1],[4249,0.4643,2972,4247,-1],[4250,0.4643,775,4247,-1],[3271,0.4375,195,4247,-1]]},{"id":"monthly-swebench-2026-04","name":"Monthly-SWEBench (2026-04)","group":"编程","unit":"%","metric":"Pass@1","source":{"name":"UniPat AI","url":"https://unipat.ai/benchmarks/MonthlySWEBench","status":"dated_release","published_at":"2026-04-30"},"protocol":{"en":"2026-04 Overall, 90 tasks. The harness is recorded per result. Includes only runs covering all 90 tasks.","zh":"2026-04 总榜共 90 道任务，各行保留执行框架。四月仅列跑完 90 题的记录。"},"public_history":true,"results":[[3168,0.6444,15,4251,-1],[4252,0.6333,633,4251,-1],[4253,0.6222,1,4251,-1],[3173,0.6222,25,4251,-1],[3175,0.6111,195,4251,-1],[4254,0.5778,195,4251,-1],[4255,0.5556,83,4251,-1],[3179,0.5556,83,4251,-1]]},{"id":"monthly-swebench-2026-05","name":"Monthly-SWEBench (2026-05)","group":"编程","unit":"%","metric":"Pass@1","source":{"name":"UniPat AI","url":"https://unipat.ai/benchmarks/MonthlySWEBench","status":"dated_release","published_at":"2026-05-31"},"protocol":{"en":"2026-05 Overall, 100 tasks. The harness is recorded per result.","zh":"2026-05 总榜共 100 道任务，各行保留执行框架。"},"public_history":true,"results":[[3168,0.82,15,4256,-1],[3179,0.81,83,4256,-1],[3170,0.8,1,4256,-1],[3174,0.75,3026,4256,-1],[3171,0.75,633,4256,-1],[4257,0.73,21,4256,-1],[3175,0.72,195,4256,-1],[4252,0.71,633,4256,-1],[3178,0.71,3033,4256,-1],[3177,0.69,25,4256,-1],[4258,0.69,21,4256,-1],[4253,0.67,1,4256,-1],[3173,0.67,25,4256,-1],[4259,0.67,2972,4256,-1],[3172,0.67,775,4256,-1],[4255,0.62,83,4256,-1],[3180,0.53,2985,4256,-1],[4260,0.52,3026,4256,-1],[4261,0.5,775,4256,-1]]},{"id":"roadmapbench-terminus-resolved","name":"RoadmapBench (Resolved, Terminus 2, 2026 paper)","group":"编程","unit":"%","metric":"Resolved","source":{"name":"UniPat AI","url":"https://arxiv.org/html/2605.15846v2#S4.T2","status":"paper_baselines","published_at":"2026-05-15"},"protocol":{"en":"Terminus 2 on 115 version-upgrade tasks, one trial per model.","zh":"Terminus 2 执行 115 道版本升级任务，每个模型运行一次。"},"public_history":true,"results":[[3181,0.38299999999999995,1,4262,4263],[3183,0.313,1,4262,4264],[3186,0.209,551,4262,4265],[3189,0.165,195,4262,4266],[3187,0.157,21,4262,4267],[3185,0.10400000000000001,25,4262,4268],[3188,0.10400000000000001,2972,4262,4269],[3191,0.10400000000000001,195,4262,4270],[2998,0.078,21,4262,4271],[3190,0.043,775,4262,4272],[3192,0.026000000000000002,3026,4262,4273]]},{"id":"roadmapbench-terminus-completion","name":"RoadmapBench (Completion, Terminus 2, 2026 paper)","group":"编程","unit":"score","metric":"Completion","source":{"name":"UniPat AI","url":"https://arxiv.org/html/2605.15846v2#S4.T2","status":"paper_baselines","published_at":"2026-05-15"},"protocol":{"en":"Terminus 2 on 115 version-upgrade tasks, one trial per model.","zh":"Terminus 2 执行 115 道版本升级任务，每个模型运行一次。"},"public_history":true,"score_max":1,"results":[[3181,0.681,1,4262,4263],[3183,0.666,1,4262,4264],[3186,0.512,551,4262,4265],[3189,0.508,195,4262,4266],[3187,0.409,21,4262,4267],[3185,0.395,25,4262,4268],[2998,0.36,21,4262,4271],[3188,0.344,2972,4262,4269],[3191,0.337,195,4262,4270],[3190,0.279,775,4262,4272],[3192,0.135,3026,4262,4273]]},{"id":"clawbench-v1-openclaw","name":"ClawBench V1 (OpenClaw, 2026 paper)","group":"Agent 与工具","unit":"%","metric":"Overall success rate","source":{"name":"ClawBench / UniPat AI","url":"https://unipat.ai/benchmarks/ClawBench","status":"paper_baselines","published_at":"2026-05-20"},"protocol":{"en":"Original OpenClaw evaluation on the 153-task V1 corpus.","zh":"原版 OpenClaw 框架，V1 共 153 道任务。"},"public_history":true,"results":[[3572,0.33299999999999996,1,4274,-1],[4275,0.261,195,4274,-1],[268,0.242,551,4274,-1],[3424,0.19,83,4274,-1],[325,0.183,1,4274,-1],[3412,0.098,83,4274,-1],[3184,0.065,15,4274,-1],[3497,0.033,83,4274,-1]]},{"id":"clawbench-v1-hermes","name":"ClawBench V1 (Hermes, 2026-05-20)","group":"Agent 与工具","unit":"%","metric":"Published V1 reward","source":{"name":"ClawBench / UniPat AI","url":"https://claw-bench.com/leaderboard","status":"dated_release","published_at":"2026-05-20"},"protocol":{"en":"Published V1 Hermes reward on 153 tasks.","zh":"V1 题集共 153 道任务，使用发布方记录的 Hermes reward。"},"public_history":true,"results":[[4276,0.6144,1,4277,-1],[4278,0.5686,1,4277,-1],[4279,0.3007,1,4277,-1],[4280,0.2549,15,4277,-1],[4281,0.2484,15,4277,-1],[2943,0.1765,21,4277,-1]]},{"id":"ceval-test-2023-0-shot-ao","name":"C-Eval (2023 paper, 0-shot answer-only)","group":"知识与推理","unit":"%","metric":"Mean subject accuracy","protocol":{"en":"C-Eval paper, mean accuracy across 52 test subjects. 0-shot answer-only.","zh":"C-Eval 原论文，取测试集的 52 门学科的平均准确率。0-shot AO。"},"source":{"name":"C-Eval authors","url":"https://arxiv.org/html/2305.08322v3#S3.T3","status":"paper_baselines","published_at":"2023-05-15"},"public_history":true,"results":[[2089,0.664,15,4282,4283],[2299,0.51,15,4282,4283],[4284,0.505,1,4282,4283],[4285,0.44299999999999995,4286,4287,4283],[4288,0.44,4289,4287,4283],[4290,0.406,1,4282,4283],[3069,0.389,4289,4287,4283],[3068,0.34700000000000003,66,4287,4283],[4291,0.331,4292,4287,4283],[4293,0.309,4294,4287,4283],[3072,0.29600000000000004,4294,4287,4283]]},{"id":"ceval-test-2023-5-shot-ao","name":"C-Eval (2023 paper, 5-shot answer-only)","group":"知识与推理","unit":"%","metric":"Mean subject accuracy","protocol":{"en":"C-Eval paper, mean accuracy across 52 test subjects. 5-shot answer-only.","zh":"C-Eval 原论文，取测试集的 52 门学科的平均准确率。5-shot AO。"},"source":{"name":"C-Eval authors","url":"https://arxiv.org/html/2305.08322v3#S3.T4","status":"paper_baselines","published_at":"2023-05-15"},"public_history":true,"results":[[2089,0.687,15,4295,4296],[2299,0.544,15,4295,4296],[4284,0.542,1,4295,4296],[4290,0.45899999999999996,1,4295,4296],[4288,0.40299999999999997,4289,4297,4296],[4285,0.39,4286,4297,4296],[3068,0.38799999999999996,66,4297,4296],[3069,0.345,4289,4297,4296],[3072,0.33299999999999996,4294,4297,4296],[4291,0.311,4292,4297,4296],[4293,0.267,4294,4297,4296]]},{"id":"ceval-test-2023-5-shot-cot","name":"C-Eval (2023 paper, 5-shot CoT)","group":"知识与推理","unit":"%","metric":"Mean subject accuracy","protocol":{"en":"C-Eval paper, mean accuracy across 52 test subjects. 5-shot CoT. The authors shortened the five-example prompt when it exceeded the context window.","zh":"C-Eval 原论文，取测试集的 52 门学科的平均准确率。5-shot CoT。 五个示例超出模型上下文窗口时，作者会减少示例。"},"source":{"name":"C-Eval authors","url":"https://arxiv.org/html/2305.08322v3#S3.T5","status":"paper_baselines","published_at":"2023-05-15"},"public_history":true,"results":[[2089,0.6829999999999999,15,4298,4299],[4284,0.542,1,4298,4299],[2299,0.5,15,4298,4299],[4290,0.445,1,4298,4299],[3069,0.345,4289,4300,4299],[4291,0.312,4292,4300,4299],[3068,0.303,66,4300,4299],[4288,0.28800000000000003,4289,4300,4299],[3072,0.254,4294,4300,4299]]},{"id":"ceval-hard-2023-0-shot-ao","name":"C-Eval Hard (2023 paper, 0-shot answer-only)","group":"知识与推理","unit":"%","metric":"Mean subject accuracy","protocol":{"en":"C-Eval paper, mean accuracy across eight hard test subjects. 0-shot answer-only.","zh":"C-Eval 原论文，取测试集的 8 门高难学科的平均准确率。0-shot AO。"},"source":{"name":"C-Eval authors","url":"https://arxiv.org/html/2305.08322v3#S3.T6","status":"paper_baselines","published_at":"2023-05-15"},"public_history":true,"results":[[2089,0.5329999999999999,15,4301,4302],[4284,0.376,1,4301,4302],[2299,0.36700000000000005,15,4301,4302],[4290,0.321,1,4301,4302],[4285,0.308,4286,4303,4302],[4288,0.307,4289,4303,4302],[3068,0.298,66,4303,4302],[3069,0.292,4289,4303,4302],[4291,0.284,4292,4303,4302],[3072,0.275,4294,4303,4302],[4293,0.244,4294,4303,4302]]},{"id":"ceval-hard-2023-5-shot-ao","name":"C-Eval Hard (2023 paper, 5-shot answer-only)","group":"知识与推理","unit":"%","metric":"Mean subject accuracy","protocol":{"en":"C-Eval paper, mean accuracy across eight hard test subjects. 5-shot answer-only.","zh":"C-Eval 原论文，取测试集的 8 门高难学科的平均准确率。5-shot AO。"},"source":{"name":"C-Eval authors","url":"https://arxiv.org/html/2305.08322v3#S3.T6","status":"paper_baselines","published_at":"2023-05-15"},"public_history":true,"results":[[2089,0.5489999999999999,15,4304,4302],[2299,0.414,15,4304,4302],[4284,0.39,1,4304,4302],[4290,0.355,1,4304,4302],[3068,0.317,66,4305,4302],[4285,0.304,4286,4305,4302],[4288,0.303,4289,4305,4302],[3072,0.273,4294,4305,4302],[4293,0.271,4294,4305,4302],[4291,0.24,4292,4305,4302],[3069,0.231,4289,4305,4302]]},{"id":"ceval-hard-2023-5-shot-cot","name":"C-Eval Hard (2023 paper, 5-shot CoT)","group":"知识与推理","unit":"%","metric":"Mean subject accuracy","protocol":{"en":"C-Eval paper, mean accuracy across eight hard test subjects. 5-shot CoT. The authors shortened the five-example prompt when it exceeded the context window.","zh":"C-Eval 原论文，取测试集的 8 门高难学科的平均准确率。5-shot CoT。 五个示例超出模型上下文窗口时，作者会减少示例。"},"source":{"name":"C-Eval authors","url":"https://arxiv.org/html/2305.08322v3#S3.T6","status":"paper_baselines","published_at":"2023-05-15"},"public_history":true,"results":[[2089,0.568,15,4306,4302],[4284,0.392,1,4306,4302],[2299,0.35,15,4306,4302],[4290,0.33399999999999996,1,4306,4302],[3069,0.261,4289,4307,4302],[4288,0.226,4289,4307,4302],[4291,0.21600000000000003,4292,4307,4302],[3068,0.214,66,4307,4302],[3072,0.154,4294,4307,4302]]},{"id":"ceval-validation-2023-0-shot-ao","name":"C-Eval (2023 paper, validation, 0-shot answer-only)","group":"知识与推理","unit":"%","metric":"Mean subject accuracy","protocol":{"en":"C-Eval paper, mean accuracy across 52 validation subjects. 0-shot answer-only.","zh":"C-Eval 原论文，取验证集的 52 门学科的平均准确率。0-shot AO。"},"source":{"name":"C-Eval authors","url":"https://arxiv.org/html/2305.08322v3#S3.T7","status":"paper_baselines","published_at":"2023-05-15"},"public_history":true,"results":[[2089,0.667,15,4308,4309],[4284,0.521,1,4308,4309],[2299,0.508,15,4308,4309],[4285,0.45899999999999996,4286,4310,4309],[4288,0.442,4289,4310,4309],[4290,0.43200000000000005,1,4308,4309],[3069,0.397,4289,4310,4309],[3068,0.386,66,4310,4309],[4291,0.35100000000000003,4292,4310,4309],[4293,0.32,4294,4310,4309],[3072,0.294,4294,4310,4309]]},{"id":"ceval-validation-2023-5-shot-ao","name":"C-Eval (2023 paper, validation, 5-shot answer-only)","group":"知识与推理","unit":"%","metric":"Mean subject accuracy","protocol":{"en":"C-Eval paper, mean accuracy across 52 validation subjects. 5-shot answer-only.","zh":"C-Eval 原论文，取验证集的 52 门学科的平均准确率。5-shot AO。"},"source":{"name":"C-Eval authors","url":"https://arxiv.org/html/2305.08322v3#S3.T7","status":"paper_baselines","published_at":"2023-05-15"},"public_history":true,"results":[[2089,0.6990000000000001,15,4311,4309],[4284,0.555,1,4311,4309],[2299,0.535,15,4311,4309],[4290,0.474,1,4311,4309],[4288,0.408,4289,4312,4309],[3068,0.39799999999999996,66,4312,4309],[4285,0.38,4286,4312,4309],[3069,0.371,4289,4312,4309],[3072,0.331,4294,4312,4309],[4291,0.289,4292,4312,4309],[4293,0.272,4294,4312,4309]]},{"id":"gsm8k-wei-2022-standard-1-order","name":"GSM8K (Wei et al. 2022, 8-shot standard, one exemplar order)","group":"数学与科学","unit":"%","metric":"Test accuracy","protocol":{"en":"1,319 GSM8K test questions, eight examples and greedy decoding. One exemplar order.","zh":"GSM8K 测试集 1,319 题，使用八个示例和贪心解码。使用一种示例顺序。"},"source":{"name":"Wei et al. / Google Research","url":"https://arxiv.org/html/2201.11903v6#A2.T2","status":"dated_release","published_at":"2022-01-28"},"public_history":true,"results":[[4313,0.19699999999999998,15,4314,4315],[4316,0.179,83,4314,4315],[4317,0.156,15,4314,4315],[4318,0.096,83,4314,4315],[4319,0.049,83,4314,4315],[4320,0.040999999999999995,83,4314,4315],[4321,0.04,15,4314,4315],[4322,0.024,15,4314,4315],[4323,0.022000000000000002,15,4314,4315]]},{"id":"gsm8k-wei-2022-cot-1-order","name":"GSM8K (Wei et al. 2022, 8-shot CoT, one exemplar order)","group":"数学与科学","unit":"%","metric":"Test accuracy","protocol":{"en":"1,319 GSM8K test questions, eight examples and greedy decoding. One exemplar order.","zh":"GSM8K 测试集 1,319 题，使用八个示例和贪心解码。使用一种示例顺序。"},"source":{"name":"Wei et al. / Google Research","url":"https://arxiv.org/html/2201.11903v6#A2.T2","status":"dated_release","published_at":"2022-01-28"},"public_history":true,"results":[[4313,0.631,15,4324,4315],[4316,0.569,83,4324,4315],[4317,0.469,15,4324,4315],[4318,0.299,83,4324,4315],[4320,0.044000000000000004,83,4324,4315],[4319,0.040999999999999995,83,4324,4315],[4321,0.024,15,4324,4315],[4323,0.005,15,4324,4315],[4322,0.005,15,4324,4315]]},{"id":"mmlu-helm","name":"MMLU (HELM 2025, v1.13.0, 5-shot)","group":"知识与推理","unit":"%","metric":"Mean subject exact match","protocol":{"en":"HELM v1.13.0 (2025-01-10), five-shot. Mean exact match across all 57 subjects, 14,042 test instances.","zh":"HELM v1.13.0（2025-01-10），五样本。共 14,042 道测试题，取 57 门学科精确匹配率的平均值。"},"source":{"name":"Stanford CRFM / HELM","url":"https://crfm.stanford.edu/helm/mmlu/latest/","status":"dated_release","published_at":"2025-01-10"},"public_history":true,"results":[[4325,0.8727304053073446,1,4326,4327],[4328,0.8720324162906944,25,4326,4327],[4329,0.8687880474746731,83,4326,4327],[4330,0.8654078253932308,1,4326,4327],[4331,0.8457152123680285,1,4326,4327],[4332,0.8453174958174847,66,4326,4327],[507,0.8433429854407875,15,4326,4327],[510,0.8417289823699189,15,4326,4327],[4333,0.8337220858502318,195,4326,4327],[4334,0.8270842647299882,83,4326,4327],[4335,0.8243666982319177,15,4326,4327],[4336,0.8235031827495974,195,4326,4327],[4337,0.8199706336653506,370,4326,4327],[4338,0.8127981488830059,83,4326,4327],[4339,0.8127187978959117,15,4326,4327],[4340,0.8095085603987299,83,4326,4327],[4341,0.8026595003973722,66,4326,4327],[4342,0.8010355128348475,66,4326,4327],[4343,0.7999764497205896,417,4326,4327],[4344,0.7973847335945918,83,4326,4327],[4345,0.7955114613915927,15,4326,4327],[4346,0.7933654187607483,66,4326,4327],[4347,0.7932309410233613,370,4326,4327],[4348,0.790602761302675,66,4326,4327],[4349,0.7861802691450123,83,4326,4327],[4350,0.78602059579238,83,4326,4327],[4351,0.781820884292191,370,4326,4327],[4352,0.7789853346850054,83,4326,4327],[4353,0.7781994679340385,417,4326,4327],[4354,0.7776077742585416,83,4326,4327],[4355,0.7764959772338222,370,4326,4327],[4356,0.774517567943956,554,4326,4327],[4142,0.7739168514623904,195,4326,4327],[4357,0.7701970132651439,370,4326,4327],[4358,0.767842526875325,195,4326,4327],[4359,0.7672266349885158,15,4326,4327],[4116,0.7623282062667585,370,4326,4327],[4360,0.7591643944489032,1,4326,4327],[4361,0.7570433943977638,83,4326,4327],[4362,0.7567775597029733,554,4326,4327],[4363,0.7443285147738198,195,4326,4327],[4364,0.7431516832098404,1,4326,4327],[4365,0.7410120567563482,370,4326,4327],[4366,0.7386012759260526,83,4326,4327],[4367,0.7379963704791742,1,4326,4327],[4368,0.7347573300097616,1,4326,4327],[4369,0.7287077954964658,195,4326,4327],[4370,0.725042055452075,25,4326,4327],[4371,0.7213434388300994,83,4326,4327],[4372,0.7172533474856093,417,4326,4327],[4373,0.7076395165491024,370,4326,4327],[4374,0.6999548452855813,83,4326,4327],[4375,0.6990912864332881,370,4326,4327],[4376,0.6954627064646919,66,4326,4327],[4377,0.693531399017232,370,4326,4327],[4378,0.692092729805366,83,4326,4327],[4379,0.6888082681118407,15,4326,4327],[4380,0.6879381069508809,1,4326,4327],[4381,0.6878312763571534,417,4326,4327],[4382,0.6872982985284851,417,4326,4327],[4383,0.6864380513233806,195,4326,4327],[4384,0.676569761285256,370,4326,4327],[4385,0.672681361696075,15,4326,4327],[4386,0.6676368046780581,66,4326,4327],[4387,0.6611956672419421,83,4326,4327],[4388,0.6588924296974432,370,4326,4327],[4389,0.6532946316803779,417,4326,4327],[4390,0.6520083694568076,370,4326,4327],[4391,0.6400594608704304,370,4326,4327],[4392,0.6257777441280399,195,4326,4327],[4393,0.5991250371958482,417,4326,4327],[4394,0.5657513230766065,417,4326,4327],[4395,0.5650818312394074,66,4326,4327],[4396,0.5606310290011183,66,4326,4327],[4397,0.5541548609350062,66,4326,4327],[4398,0.4581089294358622,66,4326,4327]]},{"id":"gsm8k-helm","name":"GSM8K (HELM Lite 2025, v1.13.0, 5-shot)","group":"数学与科学","unit":"%","metric":"Final-answer exact match","protocol":{"en":"HELM Lite v1.13.0 (2025-01-10), five-shot. Final-answer exact match on 1,000 GSM8K test instances.","zh":"HELM Lite v1.13.0（2025-01-10），五样本。取 1,000 道 GSM8K 测试题的最终答案精确匹配率。"},"source":{"name":"Stanford CRFM / HELM","url":"https://crfm.stanford.edu/helm/lite/latest/","status":"dated_release","published_at":"2025-01-10"},"public_history":true,"results":[[4325,0.956,1,4399,4400],[4330,0.949,1,4399,4400],[4332,0.949,66,4399,4400],[4344,0.946,83,4399,4400],[4348,0.942,66,4399,4400],[4328,0.94,25,4399,4400],[4342,0.938,66,4399,4400],[4341,0.936,66,4399,4400],[4335,0.932,15,4399,4400],[4331,0.924,1,4399,4400],[4336,0.92,195,4399,4400],[4343,0.912,417,4399,4400],[507,0.909,15,4399,4400],[4360,0.907,1,4399,4400],[510,0.905,15,4399,4400],[4338,0.905,83,4399,4400],[4333,0.9,195,4399,4400],[4356,0.878,554,4399,4400],[4355,0.871,370,4399,4400],[4337,0.87,370,4399,4400],[4351,0.846,370,4399,4400],[4359,0.843,15,4399,4400],[4334,0.836,83,4399,4400],[4350,0.831,83,4399,4400],[4349,0.831,83,4399,4400],[4369,0.83,195,4399,4400],[4357,0.829,370,4399,4400],[4339,0.824,15,4399,4400],[4395,0.823,66,4399,4400],[4329,0.817,83,4399,4400],[4401,0.816,83,4399,4400],[4364,0.815,1,4399,4400],[4358,0.815,195,4399,4400],[4402,0.812,83,4399,4400],[4346,0.805,66,4399,4400],[4353,0.8,417,4399,4400],[4142,0.799,195,4399,4400],[4396,0.798,66,4399,4400],[4370,0.795,25,4399,4400],[4373,0.794,370,4399,4400],[4352,0.785,83,4399,4400],[4403,0.784,1,4399,4400],[4389,0.782,417,4399,4400],[4363,0.773,195,4399,4400],[4384,0.768,370,4399,4400],[4404,0.762,83,4399,4400],[4377,0.738,370,4399,4400],[4405,0.735,83,4399,4400],[4382,0.734,417,4399,4400],[4380,0.721,1,4399,4400],[4406,0.706,417,4399,4400],[4367,0.699,1,4399,4400],[4381,0.694,417,4399,4400],[4383,0.693,195,4399,4400],[4375,0.691,370,4399,4400],[4347,0.69,370,4399,4400],[4365,0.671,370,4399,4400],[4388,0.67,370,4399,4400],[4345,0.668,15,4399,4400],[4116,0.648,370,4399,4400],[4372,0.622,417,4399,4400],[4407,0.615,15,4399,4400],[4378,0.61,83,4399,4400],[4368,0.604,1,4399,4400],[4392,0.6,195,4399,4400],[4408,0.583,1,4399,4400],[4409,0.581,554,4399,4400],[4376,0.567,66,4399,4400],[4387,0.559,83,4399,4400],[4390,0.551,370,4399,4400],[4393,0.538,417,4399,4400],[4379,0.501,15,4399,4400],[4386,0.499,66,4399,4400],[4410,0.489,66,4399,4400],[4411,0.479,15,4399,4400],[4412,0.452,370,4399,4400],[4394,0.377,417,4399,4400],[4391,0.375,370,4399,4400],[4366,0.328,83,4399,4400],[4413,0.267,370,4399,4400],[4397,0.266,66,4399,4400],[4414,0.239,370,4399,4400],[4415,0.159,370,4399,4400],[4398,0.154,66,4399,4400],[4416,0.149,370,4399,4400],[4417,0.137,370,4399,4400],[4418,0.075,370,4399,4400],[4419,0.055,370,4399,4400],[4420,0.044,370,4399,4400],[4421,0.028,370,4399,4400]]},{"id":"swe-bench-multilingual","name":"SWE-bench Multilingual","group":"编程","unit":"%","metric":"Resolved","protocol":{"en":"Multilingual track with one mini-SWE-agent attempt. Each result records the harness version.","zh":"多语言赛道，mini-SWE-agent 单次尝试。各条成绩保留 Agent 版本。"},"source":{"name":"SWE-bench","url":"https://www.swebench.com/","status":"current_snapshot","published_at":"2025-04-04"},"results":[[3424,0.727,4,4422,-1],[7,0.72,1,4422,-1],[3626,0.7070000000000001,1,4422,-1],[4423,0.6970000000000001,48,4422,-1],[26,0.687,4,4422,-1],[4424,0.6829999999999999,6,4422,-1],[244,0.6729999999999999,21,4422,-1],[2271,0.67,1,4422,-1],[14,0.667,15,4422,-1],[16,0.6629999999999999,15,2,-1],[3623,0.647,1,4422,-1],[4425,0.59,25,4422,-1],[46,0.397,15,4422,-1]]},{"id":"frontiermath-erdos","name":"FrontierMath Erdős","group":"数学与科学","unit":"%","metric":"Mean verified solution rate","protocol":{"en":"68 conjectures that were open in August 2026. A Lean proof or disproof must pass the official verifier. Runs retain the published model and reasoning settings.","zh":"68 个在 2026 年 8 月仍未解决的猜想。模型提交的 Lean 证明或反证须通过官方验证器。保留各条成绩的模型与推理设置。"},"source":{"name":"Epoch AI","url":"https://epoch.ai/benchmarks/frontiermath-erdos","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"status":"current_snapshot","published_at":"2026-08-31"},"results":[[371,0.04411764705882353,15,370,-1],[369,0.029411764705882353,15,370,-1],[372,0.0294,1,370,-1],[377,0.029,1,370,-1],[380,0.014705882352941176,4,370,-1],[375,0.014705882352941176,15,370,-1],[374,0.0,1,370,-1],[379,0.0,1,370,-1],[382,0.0,1,370,-1],[403,0.0,4,370,-1],[383,0.0,15,370,-1],[376,0.0,15,370,-1],[426,0.0,143,370,-1],[395,0.0,396,370,-1]]},{"id":"longbench-v2-direct","name":"LongBench v2 (direct)","group":"指令与长文","unit":"%","metric":"Overall direct-answer accuracy","protocol":{"en":"Overall accuracy with direct answers. Each run retains its published context limit.","zh":"直接作答的总体准确率。每项结果保留原榜注明的上下文上限。"},"source":{"name":"LongBench / THUDM","url":"https://longbench2.github.io/","status":"current_snapshot","published_at":"2024-12-19"},"results":[[774,0.529,775,4426,-1],[507,0.501,15,4427,-1],[778,0.493,83,4428,-1],[781,0.474,83,4429,-1],[771,0.467,195,4430,-1],[780,0.46,15,4427,-1],[791,0.449,21,4427,-1],[790,0.44299999999999995,783,4427,-1],[792,0.42100000000000004,195,4427,-1],[788,0.41,1,4431,-1],[784,0.40399999999999997,195,4427,-1],[786,0.38799999999999996,195,4427,-1],[341,0.37799999999999995,15,4427,-1],[794,0.344,417,4427,-1],[793,0.326,195,4427,-1],[795,0.316,66,4427,-1],[798,0.31,799,4427,-1],[802,0.308,803,4432,-1],[552,0.302,783,4427,-1],[806,0.3,66,4427,-1],[797,0.3,195,4427,-1],[796,0.298,66,4427,-1],[801,0.293,15,4427,-1],[805,0.278,644,4427,-1],[800,0.266,417,4427,-1]]},{"id":"video-mme-with-subs","name":"Video-MME (with subtitles)","group":"视觉与视频","unit":"%","metric":"Overall accuracy with subtitles","protocol":{"en":"Overall accuracy with subtitles. Each result retains the source frame-sampling setting.","zh":"提供字幕时的总体准确率。各模型保留原表中的采样帧设置。"},"source":{"name":"Video-MME","url":"https://video-mme.github.io/home_page.html#leaderboard","status":"current_snapshot","published_at":"2024-05-31"},"results":[[807,0.816,808,4433,-1],[343,0.813,83,4434,-1],[811,0.7959999999999999,812,4435,-1],[825,0.7909999999999999,826,4436,-1],[823,0.778,195,4437,-1],[353,0.772,15,4438,-1],[828,0.769,829,4439,-1],[830,0.75,83,4434,-1],[840,0.7490000000000001,841,4440,-1],[820,0.74,821,4439,-1],[814,0.74,815,4439,-1],[817,0.737,818,4440,-1],[837,0.735,838,4441,-1],[834,0.721,835,4442,-1],[842,0.72,843,4441,-1],[831,0.7170000000000001,832,4443,-1],[850,0.715,851,4444,-1],[861,0.703,862,4445,-1],[846,0.703,195,4446,-1],[860,0.7,307,4435,-1],[848,0.7,849,4439,-1],[853,0.6970000000000001,854,4447,-1],[844,0.696,829,4448,-1],[801,0.6890000000000001,15,4449,-1],[866,0.6890000000000001,812,4435,-1],[857,0.688,858,4450,-1],[874,0.6809999999999999,195,4440,-1],[864,0.679,865,4439,-1],[867,0.647,195,4448,-1],[870,0.637,865,4439,-1],[872,0.633,15,4451,-1],[788,0.629,1,4452,-1],[875,0.628,876,4440,-1],[868,0.624,821,4453,-1],[882,0.61,883,4440,-1],[881,0.5920000000000001,878,4448,-1],[877,0.5870000000000001,878,4453,-1],[879,0.5760000000000001,880,4439,-1],[884,0.574,885,4444,-1],[886,0.5710000000000001,720,4439,-1],[887,0.5429999999999999,888,4440,-1],[891,0.524,821,4451,-1],[889,0.512,195,4454,-1],[895,0.479,849,4439,-1],[892,0.47200000000000003,893,4455,-1],[897,0.45899999999999996,898,4439,-1],[901,0.43799999999999994,821,4453,-1],[899,0.436,821,4453,-1],[902,0.423,898,4439,-1],[896,0.419,195,4454,-1],[900,0.41600000000000004,898,4455,-1]]},{"id":"chinese-simpleqa","name":"Chinese SimpleQA (2024 paper baselines)","group":"知识与推理","unit":"%","metric":"Correct (CO)","protocol":{"en":"Original 2024 paper, Table 3. Correct answers across all 3,000 questions (CO), not F-score or accuracy conditional on answering.","zh":"2024 年原论文表 3 的基线成绩。使用全部 3,000 道题的答对比例 CO，不使用 F-score 或已作答题准确率。"},"source":{"name":"Chinese SimpleQA authors","url":"https://arxiv.org/html/2411.07140v2#S3.T3","status":"paper_baselines","published_at":"2024-11-11"},"public_history":true,"results":[[772,0.638,15,370,-1],[4456,0.619,370,370,-1],[353,0.593,15,370,-1],[790,0.5870000000000001,551,370,-1],[4457,0.544,83,370,-1],[1633,0.541,25,370,-1],[4458,0.541,195,370,-1],[4459,0.48700000000000004,21,370,-1],[792,0.484,195,370,-1],[1711,0.473,527,370,-1],[4066,0.462,1,370,-1],[4460,0.456,15,370,-1],[2089,0.45399999999999996,15,370,-1],[4461,0.452,3063,370,-1],[4462,0.435,25,370,-1],[341,0.395,15,370,-1],[4463,0.38799999999999996,195,370,-1],[4464,0.38299999999999995,66,370,-1],[801,0.376,15,370,-1],[4465,0.36700000000000005,370,370,-1],[4466,0.354,195,370,-1],[4467,0.337,25,370,-1],[4468,0.315,1820,370,-1],[4469,0.309,527,370,-1],[2098,0.297,15,370,-1],[535,0.273,417,370,-1],[797,0.266,195,370,-1],[4470,0.259,551,370,-1],[4471,0.247,1820,370,-1],[4472,0.23199999999999998,25,370,-1],[565,0.204,417,370,-1],[4473,0.191,3063,370,-1],[4474,0.182,527,370,-1],[4475,0.16899999999999998,66,370,-1],[4476,0.162,195,370,-1],[4477,0.159,527,370,-1],[599,0.15,417,370,-1],[4478,0.125,3063,370,-1],[4479,0.11199999999999999,551,370,-1],[4480,0.111,195,370,-1],[4481,0.053,1820,370,-1]]},{"id":"swe-bench-pro-v2-hard","name":"SWE-bench Pro V2 (Hard)","group":"编程","unit":"%","metric":"Resolved","protocol":{"en":"Scale V2 Hard subset. Each result keeps the model, agent and reasoning setting published on the board.","zh":"Scale V2 Hard 子集。各项成绩保留榜单注明的模型、Agent 和推理设置。"},"source":{"name":"Scale AI","url":"https://labs.scale.com/leaderboard/swe_bench_pro_public_v2","status":"current_snapshot","published_at":"2026-09-22"},"results":[[1424,0.98,1425,4482,-1],[1429,0.922,1425,4482,-1],[1427,0.9216,1428,4482,-1],[4483,0.902,1428,4482,-1],[1433,0.902,1428,4482,-1],[1430,0.882,1431,4482,-1],[1442,0.882,1425,4482,-1],[4484,0.863,1428,4482,-1],[1436,0.843,1437,4482,-1],[4485,0.8240000000000001,1428,4482,-1],[1439,0.8234999999999999,1428,4482,-1],[1435,0.7059000000000001,1425,4482,-1],[1440,0.588,1441,4482,-1],[1444,0.569,1445,4482,-1],[1446,0.5294,1425,4482,-1],[4486,0.255,1425,4482,-1]]},{"id":"bigcodebench-hard-complete","name":"BigCodeBench Hard (Complete)","group":"编程","unit":"%","metric":"Calibrated pass@1","protocol":{"en":"Hard-Complete, calibrated pass@1 with greedy decoding. Each run retains its prompt and prefilling settings.","zh":"Hard-Complete，采用贪心解码的校准 pass@1。各条结果保留提示与预填设置。"},"source":{"name":"BigCodeBench","url":"https://bigcode-bench.github.io/","status":"current_snapshot","published_at":"2024-06-18"},"results":[[769,0.405,25,4487,1565],[1573,0.405,25,4487,1574],[778,0.405,83,4487,1582],[1547,0.392,1,4487,1548],[1594,0.392,143,4487,1558],[1569,0.392,15,4487,1550],[1549,0.385,15,4487,1550],[1555,0.37799999999999995,370,4487,1556],[1551,0.37799999999999995,15,4487,1546],[1559,0.37799999999999995,15,4487,1546],[1544,0.37799999999999995,15,4487,1546],[512,0.37200000000000005,370,4488,1588],[1577,0.37200000000000005,15,4487,1550],[1610,0.365,15,4488,1611],[1592,0.365,83,4488,1593],[503,0.365,83,4487,1566],[1578,0.358,25,4487,1579],[1571,0.358,195,4487,1572],[1603,0.35100000000000003,1,4488,1601],[1570,0.35100000000000003,15,4488,1383],[1560,0.35100000000000003,370,4487,1561],[1600,0.345,1,4487,1601],[1580,0.345,15,4487,1581],[1636,0.345,15,4487,1587],[1552,0.33799999999999997,1,4487,1548],[1553,0.33799999999999997,15,4487,1350],[1627,0.33799999999999997,83,4488,1593],[1634,0.33799999999999997,83,4488,1593],[1533,0.33799999999999997,195,4488,1585],[1562,0.331,370,4488,1564],[1598,0.331,15,4487,1599],[1602,0.331,1,4487,1601],[1604,0.331,25,4488,1359],[1607,0.324,25,4487,1359],[511,0.324,83,4487,1591],[1595,0.324,83,4487,1582],[1554,0.318,15,4487,1350],[1575,0.318,15,4487,1350],[1590,0.318,83,4488,1591],[1669,0.311,83,4488,1591],[1670,0.311,83,4488,1593],[1557,0.311,143,4487,1558],[1619,0.304,83,4488,1582],[1620,0.304,83,4488,1582],[1642,0.304,66,4488,1643],[1621,0.304,554,4488,1622],[1637,0.297,1,4487,1638],[523,0.297,25,4487,1617],[1612,0.297,83,4487,1591],[1618,0.297,83,4488,1582],[1628,0.297,143,4487,1558],[1629,0.297,143,4487,1630],[517,0.297,417,4487,1644],[1631,0.297,195,4488,1632],[1596,0.297,370,4488,1597],[1623,0.29100000000000004,25,4488,1624],[1633,0.29100000000000004,25,4488,1359],[1608,0.29100000000000004,15,4488,1609],[1685,0.29100000000000004,66,4488,1686],[1583,0.29100000000000004,66,4487,1584],[1651,0.29100000000000004,195,4488,1652],[1714,0.284,370,4488,1715],[1657,0.284,66,4488,1658],[516,0.284,66,4488,1576],[1567,0.284,370,4487,1568],[1615,0.284,370,4488,1616],[1655,0.27699999999999997,370,4488,1656],[524,0.27699999999999997,66,4488,1635],[1605,0.27699999999999997,370,4488,1606],[1681,0.27,370,4488,1682],[1625,0.27,15,4488,1626],[1613,0.27,83,4488,1614],[536,0.27,66,4488,1641],[1645,0.27,195,4488,1646],[1586,0.27,15,4487,1587],[1661,0.264,1,4487,1638],[1695,0.264,554,4488,1696],[519,0.264,195,4488,1716],[1589,0.257,25,4488,-1],[1639,0.25,83,4488,1640],[1707,0.243,370,4488,1708],[1699,0.243,370,4488,1700],[1664,0.243,370,4488,1665],[1668,0.23600000000000002,83,4488,1640],[4489,0.23600000000000002,370,4490,4491],[1679,0.223,370,4488,1680],[1701,0.223,370,4487,1680],[1693,0.223,83,4488,1694],[1683,0.21600000000000003,15,4488,1684],[1758,0.21600000000000003,417,4488,1759],[1691,0.209,25,4488,1692],[1653,0.209,25,4488,1654],[1677,0.209,417,4488,1678],[1851,0.209,370,4488,4492],[1659,0.209,195,4488,1660],[1697,0.209,370,4488,1698],[1662,0.203,25,4488,1663],[1713,0.203,15,4488,1383],[1666,0.203,195,4488,1667],[567,0.196,1,4487,1638],[4493,0.196,25,4490,4494],[1649,0.196,417,4488,1650],[1745,0.196,195,4488,1746],[1711,0.196,527,4488,1712],[549,0.18899999999999997,417,4488,1768],[1689,0.18899999999999997,370,4488,1690],[1762,0.18899999999999997,370,4488,1763],[1709,0.182,370,4488,1710],[569,0.182,25,4488,1706],[1671,0.182,370,4488,1672],[1717,0.182,370,4488,1718],[1764,0.17600000000000002,527,4488,1765],[1702,0.16899999999999998,66,4487,1703],[1731,0.16899999999999998,370,4488,1732],[1741,0.16899999999999998,370,4488,1742],[1743,0.162,370,4488,1744],[4495,0.162,370,4490,4496],[1687,0.162,370,4488,1688],[1748,0.162,195,4488,1749],[1673,0.155,370,4488,1674],[1754,0.155,370,4488,1755],[1893,0.155,370,4490,4497],[1675,0.155,370,4488,1676],[1773,0.155,25,4488,1774],[1647,0.155,25,4488,1648],[1719,0.155,370,4488,1720],[1769,0.155,554,4488,1770],[1739,0.155,370,4488,1740],[1752,0.149,370,4488,1753],[1733,0.149,370,4488,1734],[1735,0.149,370,4488,1736],[1766,0.149,370,4488,1767],[4498,0.149,66,4490,4499],[1737,0.149,417,4488,1738],[1777,0.149,195,4488,1778],[1760,0.149,370,4488,1761],[1750,0.149,370,4488,1751],[805,0.142,370,4488,1727],[4500,0.142,417,4490,4501],[1795,0.142,370,4488,1796],[1802,0.142,554,4488,1696],[1771,0.142,554,4488,1772],[1725,0.135,370,4488,1726],[1798,0.135,370,4488,1799],[4502,0.135,25,4490,4503],[562,0.135,66,4488,1728],[555,0.135,195,4488,1779],[1789,0.128,370,4488,1790],[4504,0.128,370,4490,4505],[1721,0.128,370,4488,1722],[1729,0.128,370,4488,1730],[1812,0.128,417,4488,1738],[1704,0.128,554,4488,1705],[1787,0.128,195,4488,1788],[1791,0.122,370,4488,1792],[541,0.122,195,4488,1747],[1901,0.122,370,4490,4506],[557,0.122,527,4488,1782],[590,0.115,25,4490,4507],[1793,0.10800000000000001,370,4488,1794],[4508,0.10800000000000001,370,4490,4509],[4510,0.10800000000000001,370,4490,4511],[4512,0.10099999999999999,370,4490,4513],[1775,0.10099999999999999,83,4488,1776],[1756,0.10099999999999999,370,4488,1757],[4514,0.095,370,4490,4515],[1800,0.095,370,4488,1801],[4469,0.095,527,4490,4516],[543,0.095,527,4488,1797],[1785,0.08800000000000001,370,4488,1786],[1780,0.08800000000000001,370,4488,1781],[1908,0.08800000000000001,370,4490,4517],[4474,0.08800000000000001,527,4490,4518],[4519,0.081,370,4490,4520],[4521,0.081,370,4488,4522],[572,0.081,66,4488,1803],[4523,0.07400000000000001,370,4490,4524],[1819,0.07400000000000001,1820,4488,1821],[1783,0.068,370,4488,1784],[578,0.068,527,4488,1813],[1818,0.061,25,4488,1403],[4525,0.061,66,4490,4526],[1810,0.061,417,4488,1811],[1808,0.061,195,4488,1809],[4527,0.054000000000000006,370,4490,4528],[4529,0.054000000000000006,370,4488,4530],[1804,0.054000000000000006,66,4488,1805],[4531,0.047,417,4490,4532],[1826,0.047,195,4488,1827],[1915,0.047,370,4490,4533],[1814,0.040999999999999995,370,4488,1815],[4477,0.040999999999999995,527,4490,4534],[4535,0.034,25,4490,4536],[1822,0.034,25,4488,1823],[1806,0.034,370,4488,1807],[1816,0.02,25,4488,1817],[1830,0.02,195,4488,1831],[4537,0.013999999999999999,370,4490,4538],[1828,0.013999999999999999,66,4488,1829]]},{"id":"bigcodebench-full-instruct","name":"BigCodeBench Full (Instruct)","group":"编程","unit":"%","metric":"Calibrated pass@1","protocol":{"en":"Full-Instruct, calibrated pass@1 with greedy decoding. Each run retains its prompt and prefilling settings.","zh":"Full-Instruct，采用贪心解码的校准 pass@1。各条结果保留提示与预填设置。"},"source":{"name":"BigCodeBench","url":"https://bigcode-bench.github.io/","status":"current_snapshot","published_at":"2024-06-18"},"results":[[1608,0.511,15,4539,1609],[1573,0.5,25,4540,1574],[1583,0.49700000000000005,66,4540,1584],[1555,0.496,370,4540,1556],[1595,0.49200000000000005,83,4540,1582],[1533,0.49,195,4539,1585],[1607,0.489,25,4540,1359],[1554,0.489,15,4540,1350],[1589,0.486,25,4539,-1],[523,0.48200000000000004,25,4540,1617],[1570,0.48200000000000004,15,4539,1383],[1631,0.48200000000000004,195,4539,1632],[1580,0.48,15,4540,1581],[512,0.47200000000000003,370,4539,1588],[778,0.47,83,4540,1582],[516,0.469,66,4539,1576],[1602,0.46799999999999997,1,4540,1601],[1562,0.462,370,4539,1564],[1600,0.461,1,4540,1601],[1625,0.461,15,4539,1626],[524,0.461,66,4539,1635],[1713,0.46,15,4539,1383],[1634,0.45899999999999996,83,4539,1593],[1651,0.45799999999999996,195,4539,1652],[1664,0.456,370,4539,1665],[1637,0.455,1,4540,1638],[1621,0.455,554,4539,1622],[1620,0.45399999999999996,83,4539,1582],[1649,0.45299999999999996,417,4539,1650],[1605,0.451,370,4539,1606],[1645,0.45,195,4539,1646],[1596,0.449,370,4539,1597],[1603,0.446,1,4539,1601],[1615,0.446,370,4539,1616],[1623,0.439,25,4539,1624],[1669,0.43799999999999994,83,4539,1591],[536,0.436,66,4539,1641],[1639,0.435,83,4539,1640],[1689,0.43200000000000005,370,4539,1690],[1693,0.428,83,4539,1694],[1685,0.428,66,4539,1686],[1661,0.42700000000000005,1,4540,1638],[1671,0.424,370,4539,1672],[1691,0.42,25,4539,1692],[1699,0.418,370,4539,1700],[1697,0.41700000000000004,370,4539,1698],[1709,0.40700000000000003,370,4539,1710],[1754,0.40700000000000003,370,4539,1755],[1677,0.406,417,4539,1678],[1647,0.40399999999999997,25,4539,1648],[1666,0.40399999999999997,195,4539,1667],[1673,0.4,370,4539,1674],[1723,0.39899999999999997,370,4539,1724],[1659,0.39799999999999996,195,4539,1660],[1675,0.396,370,4539,1676],[1687,0.396,370,4539,1688],[567,0.39399999999999996,1,4540,1638],[1683,0.391,15,4539,1684],[1657,0.387,66,4539,1658],[519,0.385,195,4539,1716],[1679,0.382,370,4539,1680],[1739,0.382,370,4539,1740],[1653,0.381,25,4539,1654],[1764,0.381,527,4539,1765],[1743,0.38,370,4539,1744],[1760,0.377,370,4539,1761],[1711,0.377,527,4539,1712],[1769,0.376,554,4539,1770],[1748,0.376,195,4539,1749],[1750,0.376,370,4539,1751],[1714,0.37200000000000005,370,4539,1715],[1725,0.371,370,4539,1726],[1731,0.371,370,4539,1732],[1717,0.37,370,4539,1718],[1707,0.368,370,4539,1708],[569,0.368,25,4539,1706],[1695,0.368,554,4539,1696],[1762,0.368,370,4539,1763],[1729,0.36200000000000004,370,4539,1730],[1791,0.361,370,4539,1792],[1719,0.361,370,4539,1720],[549,0.361,417,4539,1768],[1777,0.361,195,4539,1778],[1733,0.359,370,4539,1734],[1773,0.355,25,4539,1774],[1662,0.353,25,4539,1663],[1745,0.35,195,4539,1746],[1785,0.349,370,4539,1786],[1775,0.34700000000000003,83,4539,1776],[557,0.345,527,4539,1782],[1756,0.34,370,4539,1757],[1741,0.33899999999999997,370,4539,1742],[543,0.33899999999999997,527,4539,1797],[805,0.33799999999999997,370,4539,1727],[541,0.332,195,4539,1747],[562,0.32799999999999996,66,4539,1728],[1704,0.32799999999999996,554,4539,1705],[1798,0.32299999999999995,370,4539,1799],[1787,0.32299999999999995,195,4539,1788],[1789,0.322,370,4539,1790],[1737,0.321,417,4539,1738],[572,0.319,66,4539,1803],[1771,0.311,554,4539,1772],[1812,0.3,417,4539,1738],[1802,0.29600000000000004,554,4539,1696],[1800,0.293,370,4539,1801],[555,0.29100000000000004,195,4539,1779],[1793,0.29,370,4539,1794],[1783,0.285,370,4539,1784],[1780,0.284,370,4539,1781],[1795,0.281,370,4539,1796],[1808,0.27,195,4539,1809],[1819,0.258,1820,4539,1821],[578,0.256,527,4539,1813],[1806,0.253,370,4539,1807],[1804,0.23399999999999999,66,4539,1805],[1816,0.228,25,4539,1817],[1814,0.21899999999999997,370,4539,1815],[1824,0.205,370,4539,1825],[1826,0.203,195,4539,1827],[1810,0.195,417,4539,1811],[1818,0.175,25,4539,1403],[1822,0.106,25,4539,1823],[1830,0.08800000000000001,195,4539,1831],[1828,0.08199999999999999,66,4539,1829],[1832,0.07,25,4539,1833]]},{"id":"bigcodebench-full-complete","name":"BigCodeBench Full (Complete)","group":"编程","unit":"%","metric":"Calibrated pass@1","protocol":{"en":"Full-Complete, calibrated pass@1 with greedy decoding. Each run retains its prompt and prefilling settings.","zh":"Full-Complete，采用贪心解码的校准 pass@1。各条结果保留提示与预填设置。"},"source":{"name":"BigCodeBench","url":"https://bigcode-bench.github.io/","status":"current_snapshot","published_at":"2024-06-18"},"results":[[778,0.624,83,4541,1582],[1573,0.622,25,4541,1574],[1583,0.614,66,4541,1584],[1608,0.611,15,4542,1609],[1555,0.606,370,4541,1556],[1634,0.599,83,4542,1593],[523,0.597,25,4541,1617],[1607,0.594,25,4541,1359],[1595,0.593,83,4541,1582],[1600,0.59,1,4541,1601],[1580,0.589,15,4541,1581],[1602,0.586,1,4541,1601],[1570,0.5820000000000001,15,4542,1383],[1620,0.581,83,4542,1582],[1533,0.58,195,4542,1585],[1603,0.575,1,4542,1601],[1669,0.575,83,4542,1591],[516,0.575,66,4542,1576],[1637,0.574,1,4541,1638],[1625,0.574,15,4542,1626],[1713,0.5720000000000001,15,4542,1383],[512,0.568,370,4542,1588],[1631,0.5670000000000001,195,4542,1632],[1562,0.561,370,4542,1564],[1651,0.5589999999999999,195,4542,1652],[1664,0.556,370,4542,1665],[1621,0.5539999999999999,554,4542,1622],[1639,0.551,83,4542,1640],[1623,0.5489999999999999,25,4542,1624],[524,0.5479999999999999,66,4542,1635],[1714,0.545,370,4542,1715],[536,0.545,66,4542,1641],[1615,0.544,370,4542,1616],[1685,0.541,66,4542,1686],[519,0.54,195,4542,1716],[1661,0.5379999999999999,1,4541,1638],[1589,0.532,25,4542,-1],[1671,0.529,370,4542,1672],[1697,0.526,370,4542,1698],[1709,0.525,370,4542,1710],[1699,0.525,370,4542,1700],[1693,0.525,83,4542,1694],[1645,0.523,195,4542,1646],[1659,0.522,195,4542,1660],[1605,0.512,370,4542,1606],[1691,0.511,25,4542,1692],[1596,0.511,370,4542,1597],[1851,0.51,370,4542,4492],[1689,0.509,370,4542,1690],[1683,0.506,15,4542,1684],[1649,0.504,417,4542,1650],[1677,0.502,417,4542,1678],[567,0.501,1,4541,1638],[1662,0.499,25,4542,1663],[4489,0.499,370,4543,4491],[1754,0.496,370,4542,1755],[1679,0.496,370,4542,1680],[1673,0.49,370,4542,1674],[1647,0.49,25,4542,1648],[1764,0.49,527,4542,1765],[1666,0.488,195,4542,1667],[1769,0.48700000000000004,554,4542,1770],[1653,0.484,25,4542,1654],[1657,0.48200000000000004,66,4542,1658],[1762,0.478,370,4542,1763],[569,0.47600000000000003,25,4542,1706],[1729,0.47600000000000003,370,4542,1730],[1707,0.475,370,4542,1708],[1739,0.475,370,4542,1740],[1711,0.47200000000000003,527,4542,1712],[1743,0.47100000000000003,370,4542,1744],[1777,0.46799999999999997,195,4542,1778],[1760,0.46799999999999997,370,4542,1761],[4493,0.466,25,4543,4494],[549,0.466,417,4542,1768],[1791,0.465,370,4542,1792],[1717,0.46299999999999997,370,4542,1718],[1695,0.462,554,4542,1696],[1748,0.461,195,4542,1749],[1893,0.456,370,4543,4497],[4500,0.45299999999999996,417,4543,4501],[1725,0.452,370,4542,1726],[1750,0.451,370,4542,1751],[1733,0.446,370,4542,1734],[1687,0.446,370,4542,1688],[1731,0.446,370,4542,1732],[1719,0.444,370,4542,1720],[1745,0.444,195,4542,1746],[4495,0.44,370,4543,4496],[1773,0.43799999999999994,25,4542,1774],[543,0.43799999999999994,527,4542,1797],[1741,0.43700000000000006,370,4542,1742],[1675,0.436,370,4542,1676],[4498,0.433,66,4543,4499],[1702,0.431,66,4541,1703],[557,0.424,527,4542,1782],[555,0.42100000000000004,195,4542,1779],[1756,0.42,370,4542,1757],[1787,0.42,195,4542,1788],[805,0.419,370,4542,1727],[1789,0.418,370,4542,1790],[4502,0.418,25,4543,4503],[1737,0.413,417,4542,1738],[1775,0.406,83,4542,1776],[1802,0.406,554,4542,1696],[562,0.405,66,4542,1728],[1780,0.405,370,4542,1781],[541,0.40299999999999997,195,4542,1747],[4469,0.4,527,4543,4516],[4521,0.397,370,4542,4522],[1771,0.39399999999999996,554,4542,1772],[1798,0.39299999999999996,370,4542,1799],[1785,0.38799999999999996,370,4542,1786],[1704,0.385,554,4542,1705],[4510,0.384,370,4543,4511],[1901,0.384,370,4543,4506],[4512,0.38299999999999995,370,4543,4513],[1812,0.38299999999999995,417,4542,1738],[4504,0.371,370,4543,4505],[1795,0.371,370,4542,1796],[572,0.369,66,4542,1803],[4519,0.35700000000000004,370,4543,4520],[1793,0.35600000000000004,370,4542,1794],[4474,0.355,527,4543,4518],[1800,0.354,370,4542,1801],[578,0.33899999999999997,527,4542,1813],[1819,0.331,1820,4542,1821],[1808,0.327,195,4542,1809],[4514,0.32,370,4543,4515],[1783,0.317,370,4542,1784],[4529,0.314,370,4542,4530],[590,0.306,25,4543,4507],[1806,0.298,370,4542,1807],[1816,0.29600000000000004,25,4542,1817],[4525,0.28800000000000003,66,4543,4526],[4527,0.287,370,4543,4528],[1804,0.28300000000000003,66,4542,1805],[1908,0.27699999999999997,370,4543,4517],[4477,0.275,527,4543,4534],[1810,0.273,417,4542,1811],[1826,0.271,195,4542,1827],[1814,0.257,370,4542,1815],[4508,0.256,370,4543,4509],[1818,0.249,25,4542,1403],[4523,0.239,370,4543,4524],[4531,0.235,417,4543,4532],[4535,0.222,25,4543,4536],[1915,0.214,370,4543,4533],[4537,0.2,370,4543,4538],[1822,0.153,25,4542,1823],[1828,0.113,66,4542,1829],[1830,0.113,195,4542,1831],[1832,0.079,25,4542,1833]]},{"id":"bird-sql-test-overall","name":"BIRD-SQL (overall test)","group":"编程","unit":"%","metric":"Execution accuracy","protocol":{"en":"Full BIRD test execution accuracy on the overall submission board. Each result records the oracle-knowledge setting.","zh":"BIRD 综合提交榜的全量测试集执行准确率。每项成绩注明是否使用 oracle 知识。"},"source":{"name":"BIRD","url":"https://bird-bench.github.io/","status":"current_snapshot","published_at":"2023-05-04"},"results":[[4544,0.8340000000000001,4545,4546,4547],[4548,0.8295,4549,4546,-1],[4550,0.8239,4551,4546,4552],[4553,0.8228,2206,4546,-1],[4554,0.8167,1188,4546,4555],[4556,0.8133,4557,4546,-1],[4558,0.8083,4559,4546,-1],[4560,0.7981999999999999,4561,4546,-1],[4562,0.7909,2280,4546,4563],[4564,0.787,2270,4546,-1],[4565,0.7842,2280,4546,4566],[2220,0.7831,4567,4546,4568],[4569,0.7753,4570,4546,-1],[4571,0.7686,4572,4546,-1],[4573,0.768,4574,4546,-1],[4575,0.7663,4576,4546,-1],[2213,0.7646999999999999,2214,4546,2215],[4577,0.7641,2270,4546,-1],[4578,0.7636,4579,4546,-1],[4580,0.7619,4581,4546,-1],[4582,0.7608,2270,4546,-1],[4583,0.7602,4584,4546,4585],[4586,0.7595999999999999,4587,4546,-1],[4588,0.7591,4589,4546,-1],[4590,0.758,4591,4546,-1],[4592,0.7574,4593,4546,-1],[4594,0.7563,4595,4546,-1],[4596,0.7563,936,4546,2263],[4597,0.7535,4598,4546,-1],[4599,0.7535,2270,4546,-1],[4600,0.7529,2270,4546,-1],[4601,0.7513,4602,4546,-1],[4603,0.7451000000000001,4604,4546,-1],[2225,0.7451000000000001,2226,4546,-1],[4605,0.7406,4606,4546,-1],[4607,0.7373000000000001,2236,4546,-1],[4608,0.7367,4609,4546,4610],[4611,0.7334,2230,4546,-1],[4612,0.7317,4613,4546,-1],[4614,0.7278,4615,4546,4616],[2242,0.7254999999999999,15,4546,-1],[4617,0.7228,4618,4546,4619],[4620,0.7228,936,4546,4621],[4622,0.7216,4623,4546,-1],[4624,0.7204999999999999,4625,4546,4626],[4627,0.7182999999999999,4628,4546,4629],[4630,0.7182999999999999,4631,4546,4632],[4633,0.7172,4609,4546,4610],[4634,0.716,4635,4546,-1],[4636,0.7116,4637,4546,-1],[4638,0.711,4639,4546,4640],[2258,0.706,2259,4546,-1],[4641,0.7049,4609,4546,4642],[4643,0.7026,2280,4546,4644],[4645,0.7026,4646,4546,-1],[4647,0.7021,4648,4546,-1],[2261,0.7015,1,4546,-1],[4649,0.6959000000000001,4650,4546,-1],[4651,0.6926000000000001,2270,4546,-1],[4652,0.6926000000000001,4653,4546,-1],[4654,0.6920000000000001,4655,4546,-1],[4656,0.6903,4584,4546,-1],[2262,0.6903,936,4546,2263],[4657,0.6887000000000001,4658,4546,-1],[4659,0.687,2270,4546,2361],[4660,0.6847,2236,4546,-1],[2268,0.6814,195,4546,-1],[4661,0.6803,2270,4546,-1],[4662,0.6797,4625,4546,4626],[4663,0.6786,4613,4546,-1],[4664,0.6740999999999999,4545,4665,4547],[2271,0.6685,1,4546,-1],[4666,0.6668999999999999,4639,4546,4640],[4667,0.6646,4650,4546,-1],[4668,0.6629,4669,4546,-1],[4670,0.6620999999999999,4671,4546,-1],[4672,0.659,4673,4546,-1],[2273,0.6568,2274,4546,2275],[4674,0.6551,2270,4546,2361],[4675,0.6545000000000001,4676,4546,4677],[4678,0.6523,4679,4546,-1],[4680,0.6495000000000001,936,4546,-1],[4681,0.6462,4682,4546,-1],[4683,0.6451,4648,4546,-1],[4684,0.64,4685,4546,-1],[4686,0.635,1352,4546,-1],[4687,0.6339,2270,4546,-1],[2276,0.6333,2277,4546,-1],[4688,0.6322,4689,4546,-1],[2278,0.6294,551,4546,-1],[2279,0.6265999999999999,2280,4546,2281],[4690,0.6182,4609,4546,4642],[4691,0.6098,4692,4546,-1],[769,0.6093,25,4546,-1],[4693,0.6071,4694,4546,-1],[2282,0.6042000000000001,2283,4546,2284],[2285,0.6037,2286,4546,2287],[4695,0.6031,4696,4546,4697],[2288,0.5987,21,4546,-1],[4698,0.5981000000000001,4669,4546,-1],[4699,0.5959,4700,4546,4701],[2289,0.5959,2290,4665,-1],[2292,0.5925,2286,4546,2287],[4702,0.5914,4631,4546,4703],[4704,0.5740999999999999,4705,4546,4706],[4663,0.5713,4613,4665,-1],[25,0.5668,25,4546,-1],[4707,0.5589999999999999,4696,4546,4708],[365,0.5584,417,4546,-1],[2089,0.5489,15,4546,-1],[4709,0.5411,4710,4665,4711],[2293,0.5154,2294,4546,-1],[2295,0.4902,1,4546,-1],[4712,0.47740000000000005,2270,4546,-1],[2296,0.4008,2297,4546,2298],[2299,0.39299999999999996,15,4546,-1],[2300,0.36469999999999997,370,4546,-1],[2301,0.33149999999999996,370,4546,2302],[2303,0.33039999999999997,83,4546,-1],[2296,0.2895,2297,4665,2298],[2304,0.2733,370,4546,2302],[2299,0.2677,15,4665,-1],[2300,0.2486,370,4665,-1],[2305,0.24050000000000002,370,4546,-1],[2306,0.2094,370,4546,-1],[2307,0.12890000000000001,370,4546,-1],[2305,0.1117,370,4665,-1],[2306,0.1038,370,4665,-1],[2307,0.0706,370,4665,-1]]},{"id":"spider-2-snow","name":"Spider 2.0-Snow","group":"编程","unit":"%","metric":"Task success","protocol":{"en":"Official Snow submission track, 547 tasks. Model and agent names follow the authors’ table.","zh":"官方 Snow 提交赛道，共 547 道任务。模型与 Agent 名称取自作者的原表。"},"source":{"name":"Spider 2.0","url":"https://spider2-sql.github.io/","status":"current_snapshot","published_at":"2024-11-12"},"results":[[4713,0.9670000000000001,4714,4715,4716],[2211,0.9653,4717,4715,-1],[4718,0.9415,2326,4715,-1],[4719,0.9397,4720,4715,-1],[4721,0.9048999999999999,4722,4715,4723],[4724,0.8848,4714,4715,4716],[4725,0.8628,4726,4715,-1],[4727,0.8628,2326,4715,-1],[4728,0.841,4729,4715,-1],[4730,0.8373,4731,4715,-1],[4732,0.8336,4714,4715,4716],[4733,0.83,1188,4715,-1],[4734,0.8281000000000001,4735,4715,-1],[4736,0.8281000000000001,370,4715,-1],[4737,0.8262999999999999,4722,4715,4723],[4738,0.7989,1188,4715,-1],[4739,0.7514,4740,4715,-1],[4741,0.7404000000000001,4742,4715,-1],[4743,0.7313,4744,4715,4745],[4746,0.7038,4747,4715,-1],[4748,0.7020000000000001,4749,4715,-1],[4750,0.6947,4751,4715,4752],[4753,0.6728000000000001,4754,4715,4755],[4756,0.6636,4757,4715,-1],[4758,0.6563,370,4715,-1],[4759,0.6545000000000001,4696,4715,4760],[2336,0.6289,2337,4715,2338],[4761,0.6143,4762,4715,-1],[4763,0.6033,4764,4715,-1],[4765,0.5484,2344,4715,4766],[4767,0.5046,4768,4715,-1],[4769,0.5027,4770,4715,-1],[4771,0.4534,4772,4715,-1],[4773,0.4515,4774,4715,-1],[4775,0.40950000000000003,4776,4715,-1],[4777,0.40399999999999997,2342,4715,-1],[4778,0.3839,4779,4715,-1],[4780,0.3803,370,4715,-1],[4781,0.37799999999999995,4782,4715,-1],[2365,0.3126,2337,4715,2338],[4783,0.31079999999999997,370,4715,-1],[2367,0.25780000000000003,370,4715,-1],[2369,0.245,370,4715,-1],[2366,0.24309999999999998,370,4715,-1],[2372,0.23579999999999998,370,4715,-1],[4784,0.2321,370,4715,-1],[2371,0.192,370,4715,-1],[4785,0.19010000000000002,4786,4715,-1],[4787,0.1389,370,4715,-1],[2373,0.131,2351,4715,2352],[2375,0.1298,370,4715,-1],[2374,0.10550000000000001,370,4715,-1],[4788,0.0968,4789,4715,-1],[2376,0.08960000000000001,370,4715,-1],[4790,0.08779999999999999,370,4715,-1],[2379,0.0548,370,4715,-1],[4791,0.022000000000000002,370,4715,-1],[2381,0.0128,370,4715,-1],[2382,0.0,370,4715,-1],[2285,0.0,370,4715,-1]]},{"id":"spider-2-dbt","name":"Spider 2.0-DBT","group":"编程","unit":"%","metric":"Task success","protocol":{"en":"Official DBT submission track, 68 tasks. Model and agent names follow the authors’ table.","zh":"官方 DBT 提交赛道，共 68 道任务。模型与 Agent 名称取自作者的原表。"},"source":{"name":"Spider 2.0","url":"https://spider2-sql.github.io/","status":"current_snapshot","published_at":"2024-11-12"},"results":[[4792,0.6559999999999999,4793,4794,4795],[2322,0.6029,2323,4794,2324],[4796,0.4118,4797,4794,-1],[4798,0.3971,4799,4794,-1],[4800,0.3971,4801,4794,-1],[4750,0.375,4751,4794,4752],[4802,0.3529,4754,4794,4755],[4803,0.3529,370,4794,-1],[4804,0.3529,2323,4794,-1],[4805,0.1765,2326,4794,2327],[2369,0.147,370,4794,-1],[2372,0.1324,370,4794,-1],[4806,0.10289999999999999,370,4794,-1],[4807,0.0735,370,4794,-1],[4808,0.0441,370,4794,-1],[4809,0.0294,370,4794,-1]]},{"id":"cmmlu-0shot","name":"CMMLU (0-shot)","group":"知识与推理","unit":"%","metric":"Average accuracy","protocol":{"en":"Author-maintained zero-shot table. Each score is the published average across subjects.","zh":"作者零样本榜单，各项成绩为原表中的跨学科平均准确率。"},"source":{"name":"CMMLU authors","url":"https://github.com/haonan-li/CMMLU","status":"current_snapshot","published_at":"2023-06-15"},"results":[[3037,0.9097,370,4810,-1],[3036,0.9048999999999999,370,4810,-1],[3034,0.9007,370,4810,-1],[547,0.8764,195,4810,-1],[3038,0.8747,195,4810,-1],[792,0.847,195,4810,-1],[3041,0.8468000000000001,370,4810,-1],[4811,0.8273,370,4810,-1],[3045,0.8244,370,4810,-1],[3043,0.8181,195,4810,-1],[3046,0.8147,195,4810,-1],[3042,0.8119,370,4810,-1],[4812,0.8062,370,4810,-1],[3049,0.7995,370,4810,-1],[3044,0.792,370,4810,-1],[3048,0.7707999999999999,370,4810,-1],[3050,0.754,370,4810,-1],[3051,0.7384999999999999,370,4810,-1],[3047,0.7273999999999999,370,4810,-1],[3052,0.7105,195,4810,-1],[3054,0.7059000000000001,370,4810,-1],[3053,0.6995999999999999,370,4810,-1],[524,0.6901,66,4810,-1],[1396,0.6890000000000001,15,4810,-1],[3056,0.6688,370,4810,-1],[536,0.6674,66,4810,-1],[3057,0.6122,370,4810,-1],[3055,0.6103000000000001,370,4810,-1],[3058,0.6051,370,4810,-1],[3060,0.6037,370,4810,-1],[3059,0.5757,195,4810,-1],[3061,0.5714,370,4810,-1],[3062,0.5463,3063,4810,-1],[2299,0.5322,15,4810,-1],[3065,0.5079,370,4810,-1],[3064,0.49950000000000006,551,4810,-1],[3071,0.428,370,4810,-1],[3066,0.4233,3063,4810,-1],[3069,0.4079,551,4810,-1],[3070,0.3851,370,4810,-1],[3067,0.385,370,4810,-1],[3068,0.3488,66,4810,-1],[3073,0.3006,370,4810,-1],[3072,0.2734,370,4810,-1],[3074,0.2688,370,4810,-1]]},{"id":"gaia2-vanilla-2025","name":"Gaia2 (Vanilla, 2025)","group":"Agent 与工具","unit":"%","metric":"pass@1","protocol":{"en":"Meta’s September 2025 reference runs on original Gaia2, including noise and agent-to-agent scenarios. Each score is the published macro success rate.","zh":"Meta 于 2025 年 9 月公布的原始 Gaia2 基线，含噪声与 Agent 间交互场景。成绩取各场景的宏平均成功率。"},"source":{"name":"Meta / Gaia2","url":"https://huggingface.co/spaces/meta-agents-research-environments/leaderboard","status":"dated_release","published_at":"2025-09-22"},"public_history":true,"results":[[4813,0.42100000000000004,15,4814,-1],[4815,0.37799999999999995,1,4814,-1],[4061,0.348,1,4814,-1],[4816,0.34600000000000003,15,4814,-1],[764,0.258,83,4814,-1],[4817,0.231,25,4814,-1],[4818,0.21899999999999997,25,4814,-1],[4819,0.201,396,4814,-1],[4820,0.182,15,4814,-1],[3250,0.157,143,4814,-1],[4821,0.157,195,4814,-1],[4822,0.13699999999999998,15,4814,-1],[4823,0.11599999999999999,195,4814,-1],[353,0.07400000000000001,15,4814,-1],[362,0.07400000000000001,66,4814,-1],[4824,0.044000000000000004,66,4814,-1]]},{"id":"mmlu-qwen25-base-2024","name":"MMLU (Qwen2.5 base comparison 2024, 5-shot)","group":"知识与推理","unit":"%","metric":"Accuracy","protocol":{"en":"Qwen2.5 base-model evaluation, 5-shot. The four model-size tables use one published protocol.","zh":"Qwen2.5 发布时的基础模型对比，5 样本。四张按模型规模分组的表沿用同一评测设置。"},"source":{"name":"Qwen Team","url":"https://qwenlm.github.io/blog/qwen2.5-llm/#qwen25-base-language-model-evaluation","status":"dated_release","published_at":"2024-09-19"},"public_history":true,"results":[[792,0.861,195,4825,4826],[4827,0.852,66,4825,4826],[3038,0.8420000000000001,195,4825,4826],[4463,0.833,195,4825,4826],[4466,0.797,195,4825,4826],[540,0.795,66,4825,4826],[4828,0.778,417,4825,4826],[4469,0.772,527,4825,4826],[1777,0.765,195,4825,4826],[4829,0.752,83,4825,4826],[3046,0.743,195,4825,4826],[797,0.742,195,4825,4826],[4830,0.713,83,4825,4826],[574,0.703,195,4825,4826],[4831,0.6659999999999999,66,4825,4826],[4476,0.6559999999999999,195,4825,4826],[1921,0.642,417,4825,4826],[4480,0.609,195,4825,4826],[612,0.5589999999999999,195,4825,4826],[4832,0.522,83,4825,4826],[4833,0.475,195,4825,4826],[620,0.44299999999999995,195,4825,4826]]},{"id":"gsm8k-qwen25-base-2024","name":"GSM8K (Qwen2.5 base comparison 2024, 4-shot)","group":"数学与科学","unit":"%","metric":"Accuracy","protocol":{"en":"Qwen2.5 base-model evaluation, 4-shot. The four model-size tables use one published protocol.","zh":"Qwen2.5 发布时的基础模型对比，4 样本。四张按模型规模分组的表沿用同一评测设置。"},"source":{"name":"Qwen Team","url":"https://qwenlm.github.io/blog/qwen2.5-llm/#qwen25-base-language-model-evaluation","status":"dated_release","published_at":"2024-09-19"},"public_history":true,"results":[[4463,0.929,195,4834,4826],[792,0.915,195,4834,4826],[4466,0.902,195,4834,4826],[4827,0.89,66,4834,4826],[3038,0.89,195,4834,4826],[797,0.8540000000000001,195,4834,4826],[4828,0.8370000000000001,417,4834,4826],[4469,0.8170000000000001,527,4834,4826],[4829,0.8109999999999999,83,4834,4826],[1777,0.807,195,4834,4826],[574,0.802,195,4834,4826],[4476,0.7909999999999999,195,4834,4826],[3046,0.785,195,4834,4826],[540,0.7759999999999999,66,4834,4826],[4830,0.7070000000000001,83,4834,4826],[4480,0.685,195,4834,4826],[4831,0.5529999999999999,66,4834,4826],[612,0.469,195,4834,4826],[4833,0.41600000000000004,195,4834,4826],[620,0.364,195,4834,4826],[1921,0.36200000000000004,417,4834,4826],[4832,0.303,83,4834,4826]]},{"id":"apex-accounting-mercor","name":"APEX-Accounting","group":"专业工作","unit":"score","metric":"mean-score","protocol":{"en":"Mercor global leaderboard; mean rubric score on 0–100. Each model/reasoning/harness configuration remains separate.","zh":"Mercor 总榜；0–100 的量规平均分。各模型、推理与 Agent 配置分别记录。"},"source":{"name":"Mercor","url":"https://www.mercor.com/apex/apex-accounting-leaderboard/","status":"current_snapshot","published_at":"2026-04-16"},"score_max":100,"results":[[2962,71.5,83,4835,-1],[3380,70.7,83,4835,-1],[2965,67.1,66,4836,-1],[3149,66.7,1,4836,-1],[3384,65.5,1,4836,-1],[2979,64.9,66,4836,-1],[2958,64.7,15,4836,-1],[3382,64.5,1,4836,-1],[2960,64.3,15,4836,-1],[3156,64.3,1,4836,-1],[2964,63.7,15,4836,-1],[2967,62.4,3296,4836,-1],[762,62.2,15,4836,-1],[3392,62.2,1,4836,-1],[3151,61.0,1,4836,-1],[2974,60.3,143,4837,-1],[3395,60.0,775,4836,-1],[2973,59.8,15,4836,-1],[3394,59.7,15,4836,-1],[3154,59.5,3390,4836,-1],[3385,59.3,25,4836,-1],[2983,58.8,83,4835,-1],[2987,58.0,66,4837,-1],[3391,57.4,143,4835,-1],[3386,57.2,83,4835,-1],[3157,57.2,195,4837,-1],[2977,56.7,3296,4836,-1],[2992,56.5,365,4835,-1],[3400,56.5,1,4836,-1],[3196,56.3,15,4837,-1],[3393,55.7,1,4836,-1],[2968,55.3,143,4837,-1],[3408,54.3,15,4836,-1],[2976,53.9,15,4836,-1],[3396,53.7,3390,4836,-1],[3404,53.4,25,4836,-1],[3201,53.3,25,4836,-1],[3405,52.5,83,4835,-1],[3398,50.8,3399,4835,-1],[3153,50.1,25,4836,-1],[3198,41.8,775,4835,-1],[2990,38.1,307,4835,-1],[3416,35.7,307,4835,-1],[3415,31.4,83,4835,-1],[203,31.2,199,4835,-1],[3418,29.4,307,4835,-1],[3414,5.2,83,4835,-1]]},{"id":"ace-mercor","name":"AI Consumer Index (ACE)","group":"专业工作","unit":"score","metric":"Mean rubric score","protocol":{"en":"Mercor ACE v1, 400 held-out consumer tasks with web search. Mean hurdle- and grounding-adjusted rubric score on 0–100.","zh":"Mercor ACE v1 的 400 道消费场景保留题，允许网络搜索。按目标达成与来源依据评分，取 0–100 量规分的平均值。"},"source":{"name":"Mercor","url":"https://www.mercor.com/blog/introducing-the-ai-consumer-index/","status":"current_snapshot","published_at":"2025-12-04"},"score_max":100,"results":[[2058,56.1,15,4838,-1],[4839,55.2,15,4838,-1],[3748,55.1,15,4838,-1],[1464,52.9,15,4838,-1],[340,51.5,15,4838,-1],[26,45.7,83,4838,-1],[3427,38.3,1,4838,-1],[3424,36.1,83,4838,-1],[4840,35.7,83,4841,-1],[3442,35.5,1,4841,-1],[4842,33.8,1,4841,-1],[4843,31.9,83,4841,-1]]},{"id":"vending-bench-v1-history","name":"Vending-Bench (2025)","group":"预测与决策","unit":"usd","metric":"Mean final net worth (USD)","protocol":{"en":"The publisher’s retired original Vending-Bench. Mean final net worth includes bank balance and inventory, in US dollars. API configurations remain separate.","zh":"发布方已归档的原版 Vending-Bench。期末平均净资产包含银行余额与库存，以美元计。每个 API 配置分别保留。"},"source":{"name":"Andon Labs","url":"https://andonlabs.com/evals/vending-bench","status":"archived","published_at":"2025-02-20"},"public_history":true,"results":[[3428,4694.15,143,4844,-1],[26,4387.93,83,4844,-1],[2058,3578.9,15,4844,-1],[283,2465.02,1,4844,-1],[3748,2379.88,15,4844,-1],[788,2217.93,1,4844,-1],[322,2077.41,1,4844,-1],[1464,1843.11,15,4844,-1],[2810,1567.9,1,4844,-1],[36,1295.8,21,4845,-1],[4846,975.68,83,4844,-1],[323,968.31,1,4844,-1],[4075,906.86,15,4844,-1],[4847,789.34,83,4844,-1],[36,648.93,21,4848,-1],[784,615.52,195,4844,-1],[62,607.97,15,4844,-1],[343,594.02,83,4844,-1],[801,582.33,15,4844,-1],[4064,580.77,15,4844,-1],[830,571.85,83,4844,-1],[4849,373.36,1,4844,-1],[67,338.08,83,4844,-1],[353,335.46,15,4844,-1],[4850,273.7,83,4844,-1],[362,157.33,66,4844,-1]]},{"id":"tau3-banking-mercor","name":"τ³-Banking (Mercor)","group":"Agent 与工具","unit":"%","metric":"Pass@1","protocol":{"en":"Mercor evaluation, τ² v1.0.1 task set, 97 tasks × 3 trials. Half-duplex agent, offline BM25 grep, GPT-5.2 low user simulator, Gemini 3.5 Flash judge. The 194-sample result is outside this complete 291-sample batch.","zh":"Mercor 独立评测，τ² v1.0.1 题集，97 题各运行 3 次。半双工 Agent、离线 BM25 检索、GPT-5.2 low 用户模拟器与 Gemini 3.5 Flash 裁判。本批只纳入 291 个样本完整的成绩。"},"source":{"name":"Mercor","url":"https://www.mercor.com/apex/partner-benchmarks/tau-3-banking-leaderboard/","status":"current_snapshot","published_at":"2026-04-23"},"results":[[2967,0.615,3296,4851,-1],[2962,0.581,83,4852,-1],[3380,0.557,83,4852,-1],[3384,0.55,1,4851,-1],[3149,0.54,1,4851,-1],[3382,0.54,1,4851,-1],[2979,0.522,66,4853,-1],[2965,0.519,66,4851,-1],[2983,0.488,83,4852,-1],[2958,0.467,15,4851,-1],[3157,0.467,195,4853,-1],[2974,0.457,143,4853,-1],[3156,0.44,1,4851,-1],[762,0.43,15,4851,-1],[3394,0.43,15,4851,-1],[2960,0.43,15,4851,-1],[3392,0.43,1,4851,-1],[3385,0.41600000000000004,25,4851,-1],[3154,0.405,3390,4851,-1],[3398,0.405,3399,4852,-1],[3404,0.402,25,4851,-1],[2976,0.402,15,4851,-1],[2973,0.38799999999999996,15,4851,-1],[3153,0.37799999999999995,25,4851,-1],[3396,0.37799999999999995,3390,4851,-1],[2964,0.371,15,4851,-1],[3395,0.361,775,4851,-1],[3408,0.35700000000000004,15,4851,-1],[2977,0.34700000000000003,3296,4851,-1],[3402,0.33,224,4851,-1],[2968,0.326,143,4853,-1],[3198,0.21,775,4852,-1],[2990,0.192,307,4852,-1],[3414,0.18600000000000003,83,4852,-1],[3418,0.18600000000000003,307,4852,-1],[3416,0.165,307,4852,-1],[203,0.155,199,4852,-1]]},{"id":"crmarena-original-paper","name":"CRMArena (2025)","group":"专业工作","unit":"score","metric":"Mean task score","public_history":true,"protocol":{"en":"Historical 2025 results. Original CRMArena, paper v2 Table 2. The ALL column averages F1 for knowledge QA and exact match for the other eight tasks. Act, ReAct and function calling are preserved per result.","zh":"2025 年历史成绩。原版 CRMArena 论文 v2 表 2。ALL 是九项任务的平均分，知识问答按 F1、另八项按完全匹配计分。各行保留 Act、ReAct 或函数调用配置。"},"source":{"name":"Salesforce","url":"https://arxiv.org/html/2411.02305v2#S3.T2","status":"paper_baselines","published_at":"2024-11-04"},"results":[[2816,64.3,15,4854,-1],[2816,57.7,15,4855,-1],[4168,54.4,15,4856,-1],[4167,51.3,66,4854,-1],[4857,41.8,1,4856,-1],[4169,41.1,66,4854,-1],[4168,38.2,15,4855,-1],[4857,37.4,1,4858,-1],[4194,35.1,25,4855,-1],[4857,34.3,1,4855,-1],[4167,33.8,66,4855,-1],[4168,29.4,15,4858,-1],[4170,28.3,15,4855,-1],[4169,27.8,66,4855,-1],[4167,22.2,66,4858,-1],[4170,19.5,15,4856,-1],[4169,18.6,66,4858,-1],[4859,17.3,1,4855,-1],[4170,16.7,15,4858,-1],[4859,16.6,1,4858,-1],[4859,15.1,1,4856,-1],[4194,9.0,25,4854,-1],[4860,3.1,66,4855,-1],[4860,2.5,66,4858,-1],[4860,0.0,66,4854,-1]]},{"id":"google-facts-2025-paper","name":"FACTS Benchmark Suite (2025)","group":"知识与推理","unit":"score","metric":"FACTS composite score","public_history":true,"protocol":{"en":"Historical 2025 results. December 2025 paper Table 1. Published four-component average with Grounding v2 and the paper versions of Multimodal, Parametric and Search. The original 0–100 points are retained.","zh":"2025 年历史成绩。2025 年 12 月论文表 1。原表四项平均分使用 Grounding v2，以及论文当时的 Multimodal、Parametric、Search 版本。保留原表 0–100 综合分。"},"source":{"name":"Google DeepMind / Google Research","url":"https://arxiv.org/html/2512.10791v1#S1.T1","status":"paper_baselines","published_at":"2025-12-11"},"results":[[26,68.8,83,4861,-1],[2602,62.1,83,4861,-1],[3667,61.8,15,4861,-1],[3428,53.6,143,4861,-1],[4862,52.0,15,4861,-1],[3626,51.3,1,4861,-1],[2812,50.5,15,4861,-1],[2055,50.4,83,4861,-1],[3644,49.4,15,4861,-1],[4863,49.1,1,4861,-1],[3631,46.5,1,4861,-1],[46,45.9,15,4861,-1],[4864,42.8,1,4861,-1],[4865,37.6,15,4861,-1],[4866,36.0,143,4861,-1]]},{"id":"scicode-authors-main","name":"SciCode (authors’ leaderboard, main problems) (2024–2025)","group":"编程","unit":"%","metric":"Main problem resolve rate","public_history":true,"protocol":{"en":"Historical 2024–2025 results. Original SciCode authors’ published main-problem leaderboard. Each reasoning setting keeps its source label. The separate subproblem column is omitted.","zh":"2024–2025 年历史成绩。SciCode 原作者榜单的完整问题解决率。各推理设置沿用原榜标签，分步子题另列，未计入本项。"},"source":{"name":"SciCode authors","url":"https://scicode-bench.github.io/leaderboard/","status":"dated_release","published_at":"2024-07-18"},"results":[[4867,0.10800000000000001,15,4868,-1],[4869,0.092,15,4868,-1],[4870,0.092,15,4868,-1],[4871,0.077,15,4868,-1],[2692,0.046,1,4868,-1],[4872,0.046,1,4868,-1],[4214,0.046,25,4868,-1],[4873,0.031,25,4868,-1],[4874,0.031,25,4868,-1],[2733,0.015,1,4868,-1],[4875,0.015,1,4868,-1],[522,0.015,15,4868,-1],[353,0.015,15,4868,-1],[343,0.015,83,4868,-1],[4876,0.015,66,4868,-1],[4877,0.015,15,4868,-1],[4878,0.015,195,4868,-1],[4879,0.0,66,4868,-1],[4880,0.0,66,4868,-1],[1677,0.0,417,4868,-1]]},{"id":"scicode-original-matched-paper","name":"SciCode (matched original rerun) (2026-08)","group":"编程","unit":"%","metric":"Main problem pass@1","public_history":true,"protocol":{"en":"Historical 2026-08 results. Paper Table 1, 64 whole problems with background, pass@1 and two-environment OR grading. The authors reran the original test data under the matched protocol.","zh":"2026-08 年历史成绩。论文表 1，64 道完整题，提供背景，pass@1；两个环境中有一个通过即算通过。作者按相同协议重新运行原始题集。"},"source":{"name":"SciCode-Verified authors","url":"https://arxiv.org/html/2608.04975v1#S3.T1","status":"paper_baselines","published_at":"2026-08-05"},"results":[[4881,0.266,370,4882,-1],[3479,0.25,1,4882,-1],[3453,0.23399999999999999,1,4882,-1],[762,0.23399999999999999,15,4882,-1],[3413,0.23399999999999999,66,4882,-1],[2977,0.21899999999999997,21,4882,-1],[3196,0.203,15,4882,-1],[3261,0.188,25,4882,-1],[4883,0.172,25,4882,-1],[3415,0.172,83,4882,-1],[227,0.156,551,4882,-1],[4884,0.094,25,4882,-1]]},{"id":"scicode-verified-main-paper","name":"SciCode-Verified (2026-08)","group":"编程","unit":"%","metric":"Main problem pass@1","public_history":true,"protocol":{"en":"Historical 2026-08 results. Paper Table 1, 64 whole problems with background, pass@1 and two-environment OR grading. Corrected test data, main harness; two starred earlier-harness means are excluded.","zh":"2026-08 年历史成绩。论文表 1，64 道完整题，提供背景，pass@1；两个环境中有一个通过即算通过。修订题集及主评测框架，排除两项标星的旧框架多次运行均值。"},"source":{"name":"SciCode-Verified authors","url":"https://arxiv.org/html/2608.04975v1#S3.T1","status":"paper_baselines","published_at":"2026-08-05"},"results":[[762,0.922,15,4882,-1],[3196,0.9059999999999999,15,4882,-1],[3415,0.875,83,4882,-1],[3261,0.8440000000000001,25,4882,-1],[3453,0.812,1,4882,-1],[4881,0.797,370,4882,-1],[3479,0.7809999999999999,1,4882,-1],[2977,0.7659999999999999,21,4882,-1],[3413,0.7659999999999999,66,4882,-1],[227,0.703,551,4882,-1]]},{"id":"browsecomp-kaggle-v1","name":"BrowseComp (Kaggle v1)","group":"Agent 与工具","unit":"%","metric":"Score","protocol":{"en":"Kaggle’s OpenAI collection, BrowseComp v1. The Score task reports one attempt and selects the latest run for each model version. The leaderboard does not specify a browsing tool or agent.","zh":"Kaggle OpenAI 名下的 BrowseComp v1。Score 任务记录每个模型版本的一次尝试，采用最近一次运行。榜单未标明浏览工具或智能体。"},"source":{"name":"Kaggle Benchmarks / OpenAI collection","url":"https://www.kaggle.com/benchmarks/openai/browsecomp/leaderboard","status":"current_snapshot","published_at":"2025-07-15"},"results":[[3482,0.3118536197295147,83,4885,4886],[2058,0.2006319115323854,15,4887,4886],[4033,0.1691699604743083,83,4888,4886],[4889,0.0781990521327014,83,4890,4886],[2816,0.0631911532385466,15,4891,4886],[345,0.0355450236966824,15,4892,4886],[4036,0.0331753554502369,370,4893,4886],[769,0.0308056872037914,25,4894,4886],[4895,0.026066350710900472,25,4896,4886],[4897,0.0252764612954186,83,4898,4886],[4899,0.0173775671406003,143,4900,4886],[62,0.0157977883096366,15,4901,4886],[4902,0.0150078988941548,370,4903,4886],[1573,0.0126382306477093,25,4904,4886],[360,0.0126382306477093,143,4905,4886],[4906,0.0126382306477093,15,4907,4886],[4034,0.0110584518167456,83,4908,4886],[322,0.0102685624012638,1,4909,4886],[4037,0.0102685624012638,1,4910,4886],[3727,0.0102685624012638,143,4911,4886],[788,0.0094786729857819,1,4912,4886],[4913,0.0094786729857819,370,4914,4886],[2810,0.0086887835703001,1,4915,4886],[353,0.0071090047393364,15,4916,4886],[323,0.0063191153238546,1,4917,4886],[4918,0.0063191153238546,370,4919,4886],[4920,0.0063191153238546,15,4921,4886],[801,0.0063191153238546,15,4922,4886],[67,0.0055292259083728,83,4923,4886],[4039,0.0055292259083728,83,4924,4886],[800,0.0055292259083728,417,4925,4886],[4041,0.0047393364928909,83,4926,4886],[4927,0.0047393364928909,417,4928,4886],[4929,0.0047393364928909,195,4930,4886],[830,0.0039494470774091,83,4931,4886],[343,0.0039494470774091,83,4932,4886],[4043,0.0039494470774091,83,4933,4886],[4934,0.0039494470774091,370,4935,4886],[4849,0.0031595576619273,1,4936,4886],[4937,0.0031595576619273,370,4938,4886],[4939,0.0015797788309636,370,4940,4886],[4941,0.0007898894154818,83,4942,4886]]},{"id":"rsi-bench","name":"RSI Bench","group":"Agent 与工具","unit":"score","metric":"Mean reward","protocol":{"en":"The leaderboard currently scores one task. Agents choose biomedical images and captions to train TinyCLIP. The trained model earns up to 50 points for matching images and captions and 50 for classifying images from text descriptions. Rankings use the mean across valid runs.","zh":"当前总榜计入一道任务。Agent 为 TinyCLIP 挑选生物医学图文训练数据。训练后的 TinyCLIP 要把图像和图注配对，还要根据文字描述识别图像类别，两项各占 50 分。排名取每个参评模型有效运行的平均分。"},"source":{"name":"Scale AI","url":"https://rsi-benchmark.com/tasks","status":"current_snapshot","published_at":"2026-09-15"},"score_max":100,"score_decimals":2,"results":[[4943,38.86,15,4944,4945],[4946,36.97,15,4944,4945],[4947,35.24,1,4948,4945],[4949,35.22,1,4948,4945],[4950,33.36,195,4951,4945],[4952,32.75,21,4953,4945],[4954,32.28,551,4953,4945],[4955,32.15,25,4953,4945]]},{"id":"euroexec-2026-paper","name":"EuroExec (2026-08, paper results)","group":"专业工作","unit":"%","metric":"Solve Rate","protocol":{"en":"Human-rated executive decisions across 413 cases.","zh":"人工评审 413 个高管决策案例。"},"source":{"name":"EuroExec authors","url":"https://arxiv.org/abs/2608.04549v2","status":"paper_baselines","published_at":"2026-08-05"},"public_history":true,"results":[[3151,0.569,370,370,-1],[3196,0.513,15,370,-1],[3401,0.341,370,370,-1],[3412,0.26899999999999996,83,370,-1],[227,0.21100000000000002,551,370,-1],[4956,0.184,417,370,-1]]},{"id":"gb-t-bench-2026-paper","name":"GB/T-Bench (2026-08, paper results)","group":"专业工作","unit":"%","metric":"Diagnosis recall","protocol":{"en":"Document errors whose section, review dimension and error type were all identified. Each backbone was tested alone and with GB/T-Reviewer.","zh":"以章节、审查维度和错误类型均匹配为命中。每个模型都接受了独立审查和搭配 GB/T-Reviewer 的测试。"},"source":{"name":"GB/T-Bench authors","url":"https://arxiv.org/abs/2608.06312v1","status":"paper_baselines","published_at":"2026-08-06"},"public_history":true,"results":[[4048,0.5203,15,4957,-1],[3196,0.5052,15,4957,-1],[3184,0.4287,15,4957,-1],[4958,0.3052,15,4957,-1],[4959,0.362,15,4957,-1],[3194,0.2904,1,4957,-1],[4960,0.4993,1,4957,-1],[2129,0.4087,83,4957,-1],[4961,0.4351,195,4957,-1],[4962,0.3961,25,4957,-1],[4963,0.3456,25,4957,-1],[3190,0.2516,775,4957,-1],[2444,0.2731,195,4957,-1],[771,0.2551,195,4957,-1],[4048,0.6906,15,4964,-1],[3196,0.7314,15,4964,-1],[3184,0.6642,15,4964,-1],[4958,0.5221,15,4964,-1],[4959,0.6014,15,4964,-1],[3194,0.5694,1,4964,-1],[4960,0.7008,1,4964,-1],[2129,0.5296,83,4964,-1],[4961,0.6704,195,4964,-1],[4962,0.5647,25,4964,-1],[4963,0.5894,25,4964,-1],[3190,0.466,775,4964,-1],[2444,0.4921,195,4964,-1],[771,0.5258,195,4964,-1]]},{"id":"ehr2dial-triage-2026-paper","name":"EHR2Dial-Triage (2026-08, paper results)","group":"医疗","unit":"score","metric":"ESI macro-AUC","protocol":{"en":"Emergency Severity Index prediction from the same completed triage dialogue for every model.","zh":"各模型根据同一份完整分诊对话预测急诊严重程度指数（ESI）。"},"source":{"name":"EHR2Dial-Triage authors","url":"https://arxiv.org/abs/2608.09024v1","status":"paper_baselines","published_at":"2026-08-10"},"score_max":1,"public_history":true,"score_decimals":3,"results":[[4965,0.574,195,370,-1],[4966,0.559,551,370,-1],[4967,0.611,15,370,-1],[4968,0.668,83,370,-1],[4969,0.517,195,370,-1],[4070,0.63,195,370,-1],[3763,0.668,195,370,-1],[3764,0.671,195,370,-1],[4970,0.621,370,370,-1],[4971,0.711,15,4972,-1],[3453,0.763,1,370,-1]]},{"id":"safescenereason-2026-paper","name":"SafeSceneReason (2026-08, paper results)","group":"视觉与视频","unit":"%","metric":"Overall accuracy","protocol":{"en":"Micro-accuracy across 1,000 construction-safety questions.","zh":"按全部 1,000 道施工安全题计算准确率。"},"source":{"name":"SafeSceneReason authors","url":"https://arxiv.org/abs/2608.09230v1","status":"paper_baselines","published_at":"2026-08-10"},"public_history":true,"results":[[3160,0.887,15,370,-1],[3196,0.879,15,370,-1],[3412,0.8029999999999999,83,370,-1],[3415,0.787,83,370,-1],[3453,0.741,1,370,-1],[3490,0.7070000000000001,21,370,-1],[4973,0.409,195,370,-1],[4070,0.374,195,4974,-1],[4070,0.7509999999999999,195,4975,-1],[4070,0.89,195,4976,-1]]},{"id":"sre-bench-2026-paper","name":"SRE-Bench (2026-08, paper results)","group":"编程","unit":"%","metric":"Fully solved on gradable runs","protocol":{"en":"Models used their highest reasoning setting, up to 500 steps and six hours. The full-solve rate uses each model’s gradable runs, excluding refusals and context failures.","zh":"模型使用最高推理强度，最多运行 500 步、6 小时。完全解决率以各模型可评分的运行为分母，排除拒答和上下文溢出。"},"source":{"name":"SRE-Bench authors","url":"https://arxiv.org/abs/2608.11469v2","status":"paper_baselines","published_at":"2026-08-11"},"public_history":true,"results":[[4977,0.823,15,4978,-1],[4979,0.315,15,4980,-1],[4981,0.26899999999999996,1,4982,-1],[4983,0.125,1,4984,-1],[3196,0.038,15,4985,-1],[3153,0.008,25,4985,-1],[4986,0.009000000000000001,143,4987,-1],[227,0.0,551,4985,-1]]},{"id":"dig-bench-publisher","name":"DiG-bench","group":"Agent 与工具","unit":"%","metric":"Tier 6 game win rate","protocol":{"en":"Tier 6 contains ten discovery games. The authors average each game’s win rate across games. All basic and agentic configurations ran this tier.","zh":"第六档包含十款需自行探索规则的游戏。先计算每款游戏的胜率，再取平均。所有基础框架和代理框架均测了这一档。"},"source":{"name":"DiG-bench authors","url":"https://digbench.ai/","status":"current_snapshot","published_at":"2026-08-12"},"results":[[755,0.6,1,4988,-1],[3196,0.2,15,4988,-1],[2977,0.0,21,4989,-1],[3482,0.0,83,4988,-1],[227,0.0,551,4989,-1],[3570,0.0,25,4990,-1],[755,0.5,1,4991,-1],[3453,0.4,1,4992,-1],[4883,0.0,25,4990,-1],[762,0.3,15,4993,-1],[2977,0.1,21,4994,-1],[4995,0.0,195,4990,-1],[3482,0.0,83,4996,-1]]},{"id":"frontierfinance-publisher","name":"FrontierFinance","group":"专业工作","unit":"%","metric":"All-rubric qualification rate","protocol":{"en":"Macro-average of the share of satisfied rubrics on 220 public finance queries. The official leaderboard was updated on 28 October.","zh":"220 道公开金融分析题，按每题满足评分标准的比例取宏平均。官方榜单于 10 月 28 日更新。"},"source":{"name":"FrontierFinance authors","url":"https://samaya.ai/benchmarks/frontier-finance","status":"current_snapshot","published_at":"2026-07-09"},"saturation_excluded":true,"results":[[4997,0.589,370,370,-1],[4998,0.56,370,4999,-1],[2956,0.5589999999999999,1,5000,-1],[2987,0.537,66,5001,-1],[4998,0.529,370,370,-1],[2965,0.526,66,5002,-1],[2959,0.509,1,5000,-1],[2983,0.5,83,5003,-1],[3453,0.49200000000000005,1,5000,-1],[5004,0.46799999999999997,15,5003,-1],[3584,0.466,370,5002,-1],[2977,0.46399999999999997,21,5005,-1],[3405,0.46299999999999997,83,5003,-1],[3479,0.45,1,5000,-1],[3581,0.435,15,5003,-1],[3654,0.428,551,5005,-1],[3492,0.405,25,5000,-1],[3479,0.33,1,5006,-1],[3490,0.32299999999999995,21,5007,-1],[3412,0.307,83,5006,-1],[3412,0.305,83,5000,-1],[3581,0.207,15,5008,-1]]},{"id":"oeis-open-2026-paper","name":"OEIS Open (2026-08, paper results)","group":"数学与科学","unit":"%","metric":"LITE conjectures resolved","protocol":{"en":"Base agent on 100 OEIS Open Lite conjectures with a $200 cap per attempt. Either a verified proof or disproof resolves the conjecture.","zh":"基础代理在 100 道 OEIS Open Lite 猜想上每次最多花费 200 美元。通过验证的证明或反证都算解出。"},"source":{"name":"OEIS Open authors","url":"https://arxiv.org/abs/2608.11941v2","status":"paper_baselines","published_at":"2026-08-12"},"public_history":true,"results":[[3479,0.39,1,370,-1],[3196,0.36,15,370,-1],[3415,0.29,83,370,-1],[3453,0.44,1,370,-1],[762,0.43,15,370,-1]]},{"id":"polycomp-2026-paper","name":"PolyComp (2026-08, paper results)","group":"视觉与视频","unit":"%","metric":"Accuracy","protocol":{"en":"120 spatial problems in three visual presentations each, 360 responses per model.","zh":"120 道空间推理题各有三种图像呈现方式，每个模型作答 360 次。"},"source":{"name":"PolyComp authors","url":"https://arxiv.org/abs/2608.14741v2","status":"paper_baselines","published_at":"2026-08-13"},"public_history":true,"results":[[762,0.5,15,5009,-1],[3453,0.39399999999999996,1,5009,-1],[3482,0.275,83,5010,-1]]},{"id":"quotebench-2026-paper","name":"QuoteBench (2026-08, paper results)","group":"编程","unit":"%","metric":"All-56 matched-nested success","protocol":{"en":"Best observed reasoning setting per model on 56 core shell tasks, using the matched nested command path and one stored trial per task.","zh":"56 道核心 Shell 任务使用匹配的嵌套命令路径。每题取一次已记录尝试，每个模型展示实测表现最好的设置。"},"source":{"name":"QuoteBench authors","url":"https://arxiv.org/abs/2608.13547v1","status":"paper_baselines","published_at":"2026-08-07"},"public_history":true,"results":[[3196,1.0,15,5011,-1],[5012,1.0,370,5011,-1],[5013,1.0,370,5009,-1],[4048,0.982,15,5010,-1],[5014,0.946,370,5009,-1],[502,0.9109999999999999,83,5015,-1],[5016,0.643,370,5010,-1],[2129,0.643,83,5017,-1],[5018,0.375,370,5017,-1],[2584,0.304,195,5019,-1],[4969,0.214,195,5020,-1],[4070,0.179,195,5019,-1],[5021,0.14300000000000002,83,5022,-1]]},{"id":"vals-index-v2-publisher","name":"Vals Index v2 (2026-08, Vals AI release)","group":"综合","unit":"%","metric":"Vals Index v2","protocol":{"en":"Archived Vals v2 index, using the original GDP weights and component evaluations.","zh":"按 Vals v2 原始 GDP 权重和组成评测计算的历史指数。"},"source":{"name":"Vals AI","url":"https://www.vals.ai/benchmarks/vals_index_v2","status":"dated_release","published_at":"2026-08-13"},"saturation_excluded":true,"scale_label":{"en":"Index","zh":"指数"},"public_history":true,"results":[[5023,0.6968899999999999,1,370,-1],[5024,0.69219,1,370,-1],[5025,0.68825,1,370,-1],[5026,0.6721299999999999,1,370,-1],[5027,0.66608,15,5028,-1],[5029,0.6603600000000001,1,370,-1],[5030,0.64526,66,5028,-1],[5031,0.63709,15,5028,-1],[5032,0.6256900000000001,15,5028,-1],[5033,0.6225200000000001,83,5034,-1],[5035,0.6090800000000001,1,370,-1],[5036,0.6031,66,5037,-1],[5038,0.6021500000000001,143,5037,-1],[5039,0.59881,15,5028,-1],[5040,0.5961,1,370,-1],[5041,0.59588,15,5028,-1],[5042,0.59584,370,370,-1],[5043,0.5947,370,370,-1],[5044,0.59307,83,5034,-1],[5045,0.59167,143,5034,-1],[5046,0.5845,15,5028,-1],[5047,0.5786,25,5034,-1],[5048,0.57813,21,370,-1],[5049,0.57411,15,5037,-1],[5050,0.57053,66,5037,-1],[5051,0.5697,551,5028,-1],[5052,0.56114,1,370,-1],[5053,0.55402,370,370,-1],[5054,0.55354,83,5034,-1],[5055,0.54754,66,5037,-1],[5056,0.5356799999999999,25,5034,-1],[5057,0.53122,551,370,-1],[5058,0.53079,83,5034,-1],[5059,0.52368,25,5028,-1],[5060,0.51844,195,370,-1],[5061,0.51528,143,5034,-1],[5062,0.50593,1,370,-1],[5063,0.48485,195,5037,-1],[5064,0.47216,551,5028,-1],[5065,0.44769,195,370,-1],[5066,0.43465000000000004,21,370,-1],[5067,0.42888,25,5028,-1],[5068,0.42719,775,370,-1],[5069,0.41903,83,5034,-1],[5070,0.40974,370,370,-1],[5071,0.39909999999999995,370,370,-1],[5072,0.39632,15,5037,-1],[5073,0.38645,195,370,-1],[5074,0.36711,83,5034,-1],[5075,0.34102,370,5076,-1],[5077,0.32997,15,5034,-1],[5078,0.32553,370,370,-1],[5079,0.31979,195,370,-1],[5080,0.31861999999999996,370,5076,-1],[5081,0.29444,83,5034,-1],[5082,0.27394999999999997,370,370,-1],[5083,0.26300999999999997,21,370,-1],[5084,0.24556,775,370,-1],[5085,0.24294000000000002,143,5034,-1],[5086,0.22898,1,370,-1],[5087,0.21699000000000002,370,370,-1],[5088,0.17947,417,5034,-1],[5089,0.17552,143,370,-1],[5090,0.15457,83,5034,-1],[5091,0.12952,370,5034,-1],[5092,0.11494,370,370,-1]]},{"id":"vals-rsi-index-publisher","name":"Vals RSI Index","group":"数学与科学","unit":"score","metric":"RSI Index","protocol":{"en":"The index averages four research tasks: compression, language-model training, agent harness engineering and post-training. Each score is anchored at the starter (0), a published reference (0.5) and the theoretical best (1).","zh":"压缩、语言模型训练、智能体工具开发和后训练四项研究任务等权计分。各项以初始基线为 0、已发表的参考结果为 0.5、理论最优为 1。"},"source":{"name":"Vals RSI Index authors","url":"https://www.vals.ai/benchmarks/rsi_index","status":"current_snapshot","published_at":"2026-08-13"},"score_max":1,"score_decimals":4,"results":[[2959,0.3731,1,5093,-1],[2956,0.3609,1,5093,-1],[755,0.3302,1,5093,-1],[2962,0.3055,83,5094,-1],[3394,0.2811,15,5095,-1],[2958,0.2707,15,5095,-1],[2960,0.2662,15,5095,-1],[2974,0.2546,5096,5097,-1],[3505,0.2443,66,5098,-1],[3453,0.2432,1,5093,-1],[3479,0.2398,1,5093,-1],[762,0.2388,15,5095,-1],[3515,0.2291,633,5093,-1],[3560,0.2176,195,5099,-1],[2977,0.2088,21,4994,-1],[2968,0.202,5096,5097,-1],[2983,0.1993,83,5094,-1],[3554,0.1964,66,5098,-1],[3578,0.1908,1,5093,-1],[3386,0.1827,83,5094,-1],[3581,0.1648,15,5100,-1],[3646,0.158,15,5100,-1],[5101,0.1548,15,5100,-1],[5102,0.1482,1,4992,-1],[244,0.0685,21,4994,-1]]},{"id":"fm-bench-2026-paper","name":"FM-Bench (2026-08, paper results)","group":"预测与决策","unit":"raw","metric":"Solo final score","protocol":{"en":"Three seeds per model in a 20-year football management simulation against fixed scripted opponents. The final score is uncapped.","zh":"每个模型在三个随机种子下对阵固定脚本对手，经营俱乐部 20 个赛季。最终分数无上限。"},"source":{"name":"FM-Bench authors","url":"https://arxiv.org/abs/2608.18423v2","status":"paper_baselines","published_at":"2026-08-19"},"public_history":true,"results":[[3473,90.94,1,370,-1],[420,88.49,21,370,-1],[5103,86.66,15,370,-1],[4972,86.4,15,370,-1],[2937,83.19,66,370,-1],[5104,83.17,551,370,-1],[5105,81.82,143,370,-1],[411,80.66,195,370,-1],[3209,79.15,25,370,-1],[5106,79.06,83,370,-1],[5107,75.75,1,370,-1],[5108,75.02,1,370,-1],[5109,74.59,83,370,-1],[5110,68.37,775,370,-1],[5111,36.9,1,370,-1]]},{"id":"gamexpert-bench-2026-paper","name":"GameXpert-Bench (2026-08, paper results)","group":"生成与创作","unit":"score","metric":"GAMEGEN overall score","protocol":{"en":"Game generation track. The authors average completeness, richness, player experience and visual quality on a 100-point scale.","zh":"游戏生成赛道按完整性、丰富度、玩家体验和视觉质量四项取平均，满分 100。"},"source":{"name":"GameXpert-Bench authors","url":"https://arxiv.org/abs/2608.21833v1","status":"paper_baselines","published_at":"2026-08-22"},"score_max":100,"public_history":true,"results":[[4983,79.7,1,370,-1],[4960,75.8,1,370,-1],[3155,71.3,21,370,-1],[3194,69.9,1,370,-1],[5112,63.4,15,370,-1],[5113,63.4,551,370,-1],[3201,62.9,25,370,-1],[5114,62.1,370,370,-1],[3181,62.0,1,370,-1],[3198,59.6,775,370,-1],[5115,59.1,83,370,-1],[5116,59.0,15,370,-1],[5117,58.0,551,370,-1],[3202,54.4,195,370,-1],[5118,48.7,370,370,-1]]},{"id":"earthverse-2026-paper","name":"EarthVerse (2026-08, paper results)","group":"数学与科学","unit":"%","metric":"EarthVerse Core","protocol":{"en":"Earth-science investigations. Core averages answer and process scores. Failures, timeouts and malformed responses score zero.","zh":"地球科学调查任务的 Core 分取答案与过程两项平均；执行失败、超时或格式错误均计零。"},"source":{"name":"EarthVerse authors","url":"https://arxiv.org/abs/2608.23525v1","status":"paper_baselines","published_at":"2026-08-10"},"saturation_excluded":true,"public_history":true,"results":[[5119,0.3079,370,5119,-1],[3196,0.6835,15,5120,-1],[5121,0.11720000000000001,370,5122,-1],[5123,0.1315,370,5122,-1],[5124,0.1321,370,5122,-1],[5125,0.24739999999999998,370,5122,-1],[5126,0.2502,195,5122,-1],[5127,0.311,370,5122,-1],[4823,0.3331,195,5122,-1],[3408,0.3566,15,2300,-1],[353,0.3923,15,5122,-1],[3495,0.4471,775,5122,-1],[3497,0.5966,83,5122,-1],[5128,0.6299,370,5122,-1],[3595,0.7034999999999999,195,5122,-1],[325,0.7223,1,5122,-1],[5129,0.7277,370,5122,-1],[3261,0.7354,25,5122,-1],[244,0.7532,21,5122,-1],[3572,0.7615999999999999,1,5122,-1],[3196,0.7806000000000001,15,5122,-1],[227,0.7917000000000001,551,5122,-1],[2973,0.8006,15,2300,-1],[762,0.8168000000000001,15,2300,-1],[3453,0.8497,1,3759,-1]]},{"id":"mobilepa-bench-publisher","name":"MobilePA-Bench","group":"Agent 与工具","unit":"%","metric":"Overall weighted success","protocol":{"en":"Public leaderboard with 20 models and the Qwen-Planner-Agent system. Tool use, memory, skills and subagent collaboration receive weights of 50%, 20%, 20% and 10%.","zh":"官方榜单包括 20 个模型和 Qwen-Planner-Agent 系统。工具使用、记忆、技能和子代理协作分别占 50%、20%、20% 和 10%。"},"source":{"name":"MobilePA-Bench authors","url":"https://tongyi-mai.github.io/MobilePA-Bench/#leaderboard","status":"current_snapshot","published_at":"2026-08-24"},"results":[[5130,0.7705,195,5131,-1],[4977,0.7687,15,5132,-1],[4983,0.7570999999999999,1,5132,-1],[4960,0.7453,1,5132,-1],[3154,0.7387999999999999,633,5132,-1],[3385,0.7197,25,5132,-1],[5133,0.7177,195,5132,-1],[5134,0.7147,66,5132,-1],[5135,0.7141,195,5132,-1],[3155,0.6964,396,5132,-1],[5136,0.6960999999999999,83,5132,-1],[3396,0.6831,633,5132,-1],[502,0.6816,83,5132,-1],[5137,0.6722,195,5132,-1],[227,0.6606000000000001,633,5132,-1],[3194,0.6605,1,5132,-1],[5138,0.6489,195,5132,-1],[3235,0.6489,3026,5132,-1],[5139,0.6254,195,5132,-1],[3196,0.6194,15,5132,-1],[4979,0.6021,15,5132,-1]]},{"id":"healthbench-psych-publisher","name":"HealthBench-Psych (v1, 610 prompts)","group":"医疗","unit":"score","metric":"Mean three-judge rubric score","protocol":{"en":"All 23 candidates answer the original 610 mental-health conversations. This is the mean of the three published judge scores, with equal weight for Claude Haiku 4.5, Gemini 2.5 Flash and GPT-4.1.","zh":"23 款模型回答初版的 610 道心理健康对话题。Claude Haiku 4.5、Gemini 2.5 Flash 和 GPT-4.1 分别按细则评分，三名裁判的分数取等权均值。"},"source":{"name":"HealthBench-Psych authors","url":"https://github.com/mindbench-ai/healthbench-psych","status":"current_snapshot","published_at":"2026-08-25"},"score_max":1,"score_decimals":6,"results":[[3473,0.590858,1,5140,-1],[5141,0.6207826666666667,1,5140,-1],[4279,0.4407493333333333,1,5140,-1],[5142,0.6195926666666667,1,5140,-1],[5107,0.5331486666666667,1,5140,-1],[1321,0.538134,25,5140,-1],[3209,0.5542493333333334,25,5140,-1],[4164,0.456839,83,5140,-1],[457,0.5268093333333334,83,5140,-1],[5143,0.5782543333333333,83,5140,-1],[5144,0.56816,83,5140,-1],[5145,0.175891,15,5140,-1],[485,0.512313,15,5140,-1],[3208,0.623864,15,5140,-1],[4972,0.6095886666666667,15,5140,-1],[4943,0.5848256666666667,15,5140,-1],[5105,0.6121633333333333,143,5140,-1],[420,0.6269513333333333,21,5140,-1],[4952,0.5680473333333333,21,5140,-1],[5146,0.5435623333333334,417,5140,-1],[5147,0.362702,417,5140,-1],[5148,0.445818,195,5140,-1],[5149,0.5517883333333333,195,5140,-1]]},{"id":"healthbench-psych-publisher-v2","name":"HealthBench-Psych (v2, 611 prompts)","group":"医疗","unit":"score","metric":"Mean three-judge rubric score","protocol":{"en":"The 611-conversation v2 set, using the equally weighted clipped means from Claude Haiku 4.5, Gemini 2.5 Flash and GPT-4.1.","zh":"v2 共 611 段对话。Claude Haiku 4.5、Gemini 2.5 Flash 和 GPT-4.1 的三组截断分数等权平均。"},"source":{"name":"HealthBench-Psych authors","url":"https://github.com/mindbench-ai/healthbench-psych","status":"current_snapshot","published_at":"2026-08-25"},"score_max":1,"score_decimals":6,"results":[[3473,0.59107033,1,370,-1],[5141,0.62105967,1,370,-1],[4279,0.44095967,1,370,-1],[5142,0.61973267,1,370,-1],[4947,0.64246867,1,370,-1],[5107,0.53326233,1,370,-1],[4949,0.66356267,1,370,-1],[1321,0.538508,25,370,-1],[3209,0.55432,25,370,-1],[4164,0.45705233,83,370,-1],[457,0.52697933,83,370,-1],[5143,0.57835267,83,370,-1],[5144,0.56837167,83,370,-1],[5145,0.176023,15,370,-1],[485,0.51239767,15,370,-1],[3208,0.62395533,15,370,-1],[4972,0.609699,15,370,-1],[4943,0.58490933,15,370,-1],[5150,0.53033233,15,370,-1],[2114,0.49698433,15,370,-1],[4946,0.60022567,15,370,-1],[5105,0.61245833,143,370,-1],[420,0.62697467,21,370,-1],[4952,0.568251,21,370,-1],[5146,0.543676,417,370,-1],[5147,0.36277133,417,370,-1],[5148,0.445768,195,370,-1],[5149,0.551972,195,370,-1]]},{"id":"apiflow-bench-publisher","name":"APIFlow-Bench","group":"Agent 与工具","unit":"%","metric":"All-task pass rate","protocol":{"en":"Postman’s APIFlow-Bench 1.0 leaderboard covers 467 REST API tasks with five trials scheduled per task. It records 56,037 trials across 24 models.","zh":"Postman 的 APIFlow-Bench 1.0 共 467 道 REST API 任务，每题计划执行五次。榜单记录了 24 个模型的 56,037 次运行。"},"source":{"name":"APIFlow-Bench authors","url":"https://www.postman.com/ai/apiflow-leaderboard/overview-all/","status":"current_snapshot","published_at":"2026-07-21"},"results":[[4952,0.899,21,370,-1],[4278,0.89,1,370,-1],[3476,0.8859999999999999,1,370,-1],[1321,0.884,25,370,-1],[5151,0.879,21,370,-1],[3208,0.878,15,370,-1],[5107,0.878,1,370,-1],[5152,0.8759999999999999,21,370,-1],[4972,0.873,15,370,-1],[5153,0.873,195,370,-1],[5154,0.871,551,370,-1],[3209,0.867,25,370,-1],[4279,0.86,1,370,-1],[5155,0.86,15,370,-1],[5156,0.8590000000000001,21,370,-1],[5157,0.855,195,370,-1],[5158,0.855,775,370,-1],[5103,0.8540000000000001,15,370,-1],[62,0.82,15,370,-1],[4906,0.809,15,370,-1],[5159,0.8059999999999999,15,370,-1],[5160,0.685,15,370,-1],[5110,0.327,775,370,-1],[3473,0.145,1,370,-1]]},{"id":"cwe-bench-v0-publisher","name":"CWE-bench v0","group":"安全与可靠性","unit":"%","metric":"Pass@1","protocol":{"en":"Programmatic exploit and regression tests for the v0 task set.","zh":"以 v0 版本的漏洞验证和回归测试计算单次通过率。"},"source":{"name":"CWE-bench","url":"https://cwe-bench.com/v0","status":"current_snapshot","published_at":"2026-09-01"},"results":[[3453,0.478,1,3759,-1],[5161,0.47200000000000003,83,5162,-1],[762,0.442,15,2300,-1],[3386,0.44,83,5162,-1],[3479,0.42,1,3759,-1],[2974,0.382,143,5163,-1],[3157,0.375,195,5164,-1],[4029,0.33799999999999997,370,5164,-1],[3154,0.311,551,5164,-1],[3201,0.304,25,5164,-1],[3407,0.303,66,5164,-1],[3491,0.22699999999999998,775,5164,-1],[2977,0.225,21,5164,-1],[5165,0.185,370,5164,-1],[203,0.156,370,5164,-1]]},{"id":"protein-design-library-ranking-2026-paper","name":"Protein Design library ranking (2026-09, paper results)","group":"数学与科学","unit":"score","metric":"Chance-normalized average precision","protocol":{"en":"Revised 83-problem library ranking cohort.","zh":"83 道题的修订版蛋白设计库排序任务。"},"source":{"name":"Anthropic","url":"https://www-cdn.anthropic.com/fc1b44717c85dc068bc6ba5024219938094694bd/Claude%20Opus%205.5%20System%20Card.pdf#page=219","status":"paper_baselines","published_at":"2026-09-01"},"public_history":true,"results":[[5166,56.9,1,370,-1],[2959,56.0,1,370,-1],[755,53.5,1,370,-1],[3568,44.0,1,370,-1]]},{"id":"tax-agent-bench-publisher","name":"Tax Agent Bench","group":"专业工作","unit":"%","metric":"Overall score","protocol":{"en":"Research-grade US tax questions, using the Tax Agent Bench harness.","zh":"按 Tax Agent Bench 的流程回答美国税务研究问题。"},"source":{"name":"Vals AI","url":"https://www.vals.ai/benchmarks/tax_agent_bench","status":"current_snapshot","published_at":"2026-09-04"},"saturation_excluded":true,"scale_label":{"en":"Rubric score","zh":"量表得分"},"results":[[5167,0.5178499999999999,83,5034,-1],[5168,0.49662999999999996,195,370,-1],[5025,0.49194000000000004,1,370,-1],[5023,0.45573,1,370,-1],[5026,0.45548,1,370,-1],[5169,0.4456,83,5034,-1],[5170,0.4435,25,5028,-1],[5171,0.42997,66,5028,-1],[5030,0.42435,66,5028,-1],[5024,0.41959,1,370,-1],[5051,0.39698,551,5028,-1],[5029,0.38281,1,370,-1],[5172,0.36268999999999996,66,5028,-1],[5036,0.35973999999999995,66,5037,-1],[5173,0.34399,370,370,-1],[5038,0.34197000000000005,143,5037,-1],[5048,0.33368000000000003,21,5028,-1],[5045,0.33141,143,5034,-1],[5050,0.32798,66,5037,-1],[5053,0.32616999999999996,370,370,-1],[5043,0.32388,370,370,-1],[5174,0.32136000000000003,775,370,-1],[5033,0.32124,83,5034,-1],[5060,0.32095,195,370,-1],[5175,0.31922,1,370,-1],[5064,0.3155,551,5028,-1],[5031,0.31088,15,5028,-1],[5176,0.31077,370,370,-1],[5063,0.30894,195,5037,-1],[5177,0.30668,417,5034,-1],[5178,0.30390999999999996,417,5034,-1],[5041,0.30035,15,5028,-1],[5035,0.29999,1,370,-1],[5040,0.29374,1,370,-1],[5042,0.28507,370,370,-1],[5056,0.28069,25,5034,-1],[5179,0.27273,370,370,-1],[5059,0.26756,25,5028,-1],[5047,0.2663,25,5028,-1],[5180,0.26321,66,5037,-1],[5062,0.2584,1,370,-1],[5067,0.25170000000000003,25,5028,-1],[5181,0.2487,21,5028,-1],[5061,0.24855,143,5034,-1],[5039,0.24815,15,5028,-1],[5052,0.23738,1,370,-1],[5044,0.22798,83,5034,-1],[5049,0.21762,15,5037,-1],[5182,0.21754,15,5028,-1],[5058,0.21652000000000002,83,5034,-1],[5027,0.20717,15,5028,-1],[5078,0.20599,370,370,-1],[5068,0.20588,775,370,-1],[5046,0.19946999999999998,15,5028,-1],[5183,0.18169000000000002,370,5034,-1],[5054,0.18038,83,5034,-1],[5184,0.17081,15,5028,-1],[5065,0.15832000000000002,195,370,-1],[5080,0.15389,370,5076,-1],[5075,0.15291,370,5076,-1],[5032,0.15002000000000001,15,5028,-1],[5066,0.12404,21,370,-1],[5081,0.11843,83,5034,-1],[5185,0.11554,370,5034,-1],[5070,0.11503000000000001,370,370,-1],[5073,0.10465999999999999,195,370,-1],[5069,0.09295,83,5034,-1],[5186,0.08807999999999999,370,370,-1],[5072,0.08807999999999999,15,5037,-1],[5071,0.07772,370,370,-1],[5079,0.07679,195,370,-1],[5074,0.07254,83,5034,-1],[5089,0.06952,143,370,-1],[5084,0.06872,775,370,-1],[5085,0.061790000000000005,143,5034,-1],[5086,0.058550000000000005,1,370,-1],[5077,0.04974,15,5034,-1],[5088,0.04482,417,5034,-1],[5090,0.03109,83,5034,-1],[5092,0.025910000000000002,370,370,-1],[5091,0.02487,370,5034,-1]]},{"id":"autoresearchexam-publisher","name":"AutoResearchExam","group":"数学与科学","unit":"score","metric":"Hidden-test AUARC at 24 hours","protocol":{"en":"Over 24 hours, agents repeatedly improve solutions to 29 machine-learning research tasks. The score integrates hidden-test performance over elapsed time, rewarding improvements made earlier.","zh":"智能体用 24 小时反复改进 29 项机器学习研究任务的解法。分数取隐藏测试成绩在整个时段的曲线下面积并按时长归一，越早取得进展，贡献越大。"},"source":{"name":"AutoResearchExam authors","url":"https://benchmarks.bespokelabs.ai/autoresearchexam/","status":"current_snapshot","published_at":"2026-09-09"},"score_max":1,"score_decimals":6,"results":[[2959,0.609250526746918,1,5187,-1],[2956,0.6024326804582145,1,5188,-1],[2958,0.6000894530946282,15,5188,-1],[2962,0.5943556684453349,83,5187,-1],[755,0.5790323080032505,1,5188,-1],[2968,0.5227267004954371,143,5187,-1],[762,0.5131796443845165,15,5188,-1],[2974,0.4931355237085956,143,5188,-1],[2983,0.48350995545208975,83,5188,-1],[2977,0.43545335727965423,21,5188,-1],[2982,0.4234121630633122,195,5188,-1],[2979,0.37885642205704395,66,5189,-1]]},{"id":"cursorbench-4-0-publisher","name":"CursorBench 4.0","group":"编程","unit":"%","metric":"Task score","protocol":{"en":"Cursor’s 4.0 development-task leaderboard, with each published reasoning setting kept separately.","zh":"Cursor 4.0 开发任务榜单，各推理设置分别保留。"},"source":{"name":"Cursor","url":"https://cursor.com/cursorbench","status":"current_snapshot","published_at":"2026-09-10"},"results":[[5190,0.578,370,5191,-1],[5192,0.56,370,5191,-1],[5193,0.56,370,5191,-1],[5194,0.555,370,5191,-1],[5195,0.531,370,5191,-1],[5196,0.529,83,5191,-1],[5197,0.528,83,5191,-1],[5198,0.525,370,5191,-1],[5199,0.518,370,5191,-1],[5200,0.516,370,5191,-1],[5201,0.506,83,5191,-1],[5202,0.504,66,5191,-1],[5203,0.49200000000000005,370,5191,-1],[5204,0.484,370,5191,-1],[5205,0.48200000000000004,66,5191,-1],[5206,0.478,370,5191,-1],[5207,0.47700000000000004,25,5191,-1],[5208,0.46799999999999997,370,5191,-1],[5209,0.466,370,5191,-1],[5210,0.46299999999999997,143,5191,-1],[5211,0.46299999999999997,25,5191,-1],[5212,0.461,370,5191,-1],[5213,0.461,66,5191,-1],[5214,0.456,25,5191,-1],[5215,0.452,66,5191,-1],[5216,0.451,370,5191,-1],[5217,0.449,25,5191,-1],[5218,0.447,370,5191,-1],[5219,0.44299999999999995,370,5191,-1],[5220,0.439,143,5191,-1],[5221,0.43700000000000006,370,5191,-1],[5222,0.433,370,5191,-1],[5223,0.426,551,5191,-1],[5224,0.423,370,5191,-1],[5225,0.41700000000000004,15,5191,-1],[5226,0.41700000000000004,66,5191,-1],[5227,0.41600000000000004,143,5191,-1],[3554,0.41600000000000004,66,5191,-1],[5228,0.414,143,5191,-1],[5229,0.413,15,5191,-1],[5230,0.40700000000000003,370,5191,-1],[5231,0.40399999999999997,143,5191,-1],[5232,0.396,83,5191,-1],[5233,0.392,370,5191,-1],[5234,0.38,551,5191,-1],[5235,0.377,15,5191,-1],[5236,0.375,66,5191,-1],[5237,0.373,83,5191,-1],[5238,0.369,370,5191,-1],[5239,0.368,551,5191,-1],[5240,0.361,143,5191,-1],[5241,0.359,15,5191,-1],[5242,0.358,370,5191,-1],[5243,0.35700000000000004,15,5191,-1],[5244,0.342,66,5191,-1],[5245,0.341,370,5191,-1],[5246,0.336,15,5191,-1],[5247,0.33399999999999996,143,5191,-1],[5248,0.33399999999999996,66,5191,-1],[5249,0.33299999999999996,551,5191,-1],[5250,0.331,143,5191,-1],[5251,0.33,15,5191,-1],[5252,0.326,66,5191,-1],[5253,0.32,370,5191,-1],[5254,0.311,15,5191,-1],[5255,0.311,551,5191,-1],[5256,0.309,370,5191,-1],[5257,0.308,370,5191,-1],[5258,0.307,15,5191,-1],[5259,0.294,15,5191,-1],[5260,0.293,66,5191,-1],[5261,0.28,370,5191,-1],[5262,0.27699999999999997,370,5191,-1],[5263,0.276,15,5191,-1],[5264,0.26899999999999996,551,5191,-1],[5265,0.252,15,5191,-1],[5266,0.24600000000000002,15,5191,-1],[5267,0.243,66,5191,-1],[5268,0.24100000000000002,370,5191,-1],[5269,0.222,15,5191,-1],[5270,0.16,15,5191,-1]]},{"id":"crypto-accounting-bench-2026-paper","name":"Crypto Accounting Bench (2026-09, paper results)","group":"专业工作","unit":"%","metric":"Mean rubric score","protocol":{"en":"Mean weighted rubric score across 118 accounting tasks and three independent attempts per model and task.","zh":"118 道会计分录任务，每个模型每题独立尝试三次，计算全部尝试的加权评分均值。"},"source":{"name":"Crypto Accounting Bench authors","url":"https://arxiv.org/abs/2609.14811v1","status":"paper_baselines","published_at":"2026-09-13"},"saturation_excluded":true,"public_history":true,"results":[[3386,0.7743000000000001,83,5271,-1],[3151,0.7739,370,5011,-1],[2977,0.7707999999999999,21,5009,-1],[2974,0.7633,143,5010,-1],[762,0.7598999999999999,15,5011,-1],[2973,0.7593000000000001,15,5011,-1],[3568,0.7565999999999999,1,5011,-1],[3407,0.7562000000000001,66,5011,-1],[3391,0.7532,143,5010,-1],[755,0.7428,1,5011,-1],[5272,0.7282,25,5009,-1],[3408,0.7192000000000001,15,5011,-1]]},{"id":"arxivmath-august-2026-publisher","name":"ArXivMath, August 2026","group":"数学与科学","unit":"%","metric":"Accuracy","protocol":{"en":"The August set contains 57 research-math questions. MathArena runs models in their recommended harnesses with offline Python and SageMath. It reports separate rows where it tested more than one reasoning level.","zh":"8 月版收录 57 道研究数学题。MathArena 使用各模型推荐的运行工具，可离线调用 Python 和 SageMath。同一模型测过多档推理强度时，分别列出成绩。"},"source":{"name":"ArXivMath, August 2026 authors","url":"https://matharena.ai/arxiv_august/","status":"current_snapshot","published_at":"2026-09-15"},"results":[[2960,0.9649,15,5273,-1],[3394,0.9298000000000001,15,5273,-1],[2958,0.8859999999999999,15,5273,-1],[4981,0.8772,1,5274,-1],[5275,0.8595999999999999,1,5274,-1],[2958,0.8509,15,5276,-1],[2962,0.8245999999999999,83,5277,-1],[4981,0.7895,1,5276,-1],[3732,0.6842,173,5277,-1],[2967,0.6491,21,5277,-1],[3157,0.5789,50,5277,-1],[3385,0.5614,25,5278,-1],[2968,0.46490000000000004,143,5279,-1],[2979,0.4561,173,5277,-1],[3153,0.4386,25,5278,-1],[2983,0.4035,83,5277,-1],[2977,0.386,21,5280,-1],[2987,0.2982,173,5277,-1],[2992,0.24559999999999998,365,5277,-1]]},{"id":"parallel-search-capability-publisher","name":"Parallel Search Capability","group":"知识与推理","unit":"score","metric":"Search Intelligence Score","protocol":{"en":"Equal-weight index of DeepSearchQA, HLE and WISER with Parallel Search Fast and Extract enabled.","zh":"启用 Parallel Search Fast 和 Extract，将 DeepSearchQA、HLE 和 WISER 三项成绩等权汇总。"},"source":{"name":"Parallel","url":"https://parallel.ai/leaderboard","status":"current_snapshot","published_at":"2026-09-15"},"results":[[2959,75.4,1,5281,-1],[2967,74.2,21,5281,-1],[2961,74.1,1,5281,-1],[2956,73.4,1,5281,-1],[5282,72.5,370,5281,-1],[2962,72.2,83,5281,-1],[2965,71.3,66,5281,-1],[2958,70.8,15,5281,-1],[2960,70.4,15,5281,-1],[755,70.0,1,5281,-1],[2964,69.7,15,5281,-1],[5283,69.2,370,5281,-1],[5284,68.9,25,5281,-1],[2975,68.8,775,5281,-1],[5285,68.6,370,5281,-1],[3563,67.9,370,5281,-1],[762,67.7,15,5281,-1],[2987,67.2,66,5281,-1],[3386,66.8,83,5281,-1],[3394,66.6,15,5281,-1],[3568,66.6,1,5281,-1],[2979,66.1,66,5281,-1],[2982,66.0,195,5281,-1],[5227,66.0,143,5281,-1],[2983,65.1,83,5281,-1],[3524,64.6,370,5281,-1],[5286,64.4,417,5281,-1],[5287,64.4,370,5281,-1],[2977,64.2,21,5281,-1],[3398,63.6,370,5281,-1],[2970,62.9,1,5281,-1],[3512,62.7,25,5281,-1],[3515,62.7,551,5281,-1],[2976,61.9,15,5281,-1],[3534,61.1,370,5281,-1],[5288,61.1,370,5281,-1],[5289,60.9,370,5281,-1],[3529,60.8,551,5281,-1],[3408,60.7,15,5281,-1],[2990,59.1,370,5281,-1],[3424,58.6,83,5281,-1],[3492,58.4,25,5281,-1],[5290,58.2,370,5281,-1],[3402,56.0,370,5281,-1],[227,54.3,551,5281,-1],[5291,53.0,370,5281,-1],[3491,52.6,775,5281,-1],[5292,49.5,370,5281,-1],[5293,39.5,370,5281,-1],[3617,37.9,370,5281,-1],[5294,34.8,370,5281,-1]]},{"id":"mysterymechanism-publisher","name":"MysteryMechanism","group":"数学与科学","unit":"%","metric":"Overall score","protocol":{"en":"Agents use a fixed experiment budget to identify mathematical mechanisms.","zh":"在给定实验预算内识别隐藏的数学机制。"},"source":{"name":"Vals AI","url":"https://www.vals.ai/benchmarks/mysterymechanism","status":"current_snapshot","published_at":"2026-09-16"},"saturation_excluded":true,"scale_label":{"en":"Rubric score","zh":"量表得分"},"results":[[5027,0.53153,15,5028,-1],[5023,0.4955,1,370,-1],[5024,0.49099,1,370,-1],[5025,0.47747999999999996,1,370,-1],[5182,0.46396,15,5028,-1],[5169,0.45494999999999997,83,5034,-1],[5026,0.37387,1,370,-1],[5033,0.36485999999999996,83,5034,-1],[5030,0.36036,66,5028,-1],[5175,0.35135,1,370,-1],[5171,0.35135,66,5028,-1],[5031,0.33332999999999996,15,5028,-1],[5045,0.30631,143,5034,-1],[5032,0.3018,15,5028,-1],[5172,0.27477,66,5028,-1],[5181,0.26126,21,5028,-1],[5038,0.25225000000000003,143,5037,-1],[5060,0.23873999999999998,195,370,-1],[5051,0.22973,551,5028,-1],[5176,0.22522999999999999,370,370,-1],[5186,0.22072,370,370,-1],[5042,0.21622,370,370,-1],[5047,0.21170999999999998,25,5034,-1],[5069,0.20721,83,5034,-1],[5046,0.19369,15,5028,-1],[5179,0.18468,370,370,-1],[5043,0.15315,370,370,-1],[5039,0.14414,15,5028,-1],[5178,0.12162,417,5034,-1]]},{"id":"overclaimbench-2026-paper","name":"OverclaimBench (2026-09, paper results)","group":"安全与可靠性","unit":"%","metric":"Explicit overclaim rate","protocol":{"en":"Share of all file-review runs in which the agent claimed full coverage despite missing files. Across five scenarios the authors report 100 runs per model, except Gemini 3.1 Pro with 40. Lower is better.","zh":"在文件审查中，代理漏读文件却声称全部覆盖的次数占所有运行的比例。五个场景合计，每个模型运行 100 次，Gemini 3.1 Pro 为 40 次。数值越低越好。"},"source":{"name":"OverclaimBench authors","url":"https://arxiv.org/abs/2609.20812v3","status":"paper_baselines","published_at":"2026-09-17"},"higher_is_better":false,"public_history":true,"results":[[3568,0.48,1,3759,-1],[755,0.36,1,3759,-1],[3453,0.39,1,3759,-1],[5295,0.08,143,5296,-1],[5297,0.44,15,2300,-1],[5298,0.28,15,2300,-1],[4048,0.34,15,2300,-1],[3412,0.55,83,5299,-1],[3154,0.31,551,3759,-1],[3396,0.35,551,3759,-1],[3201,0.42,25,3759,-1],[5300,0.42,195,3759,-1]]},{"id":"vibe-code-bench-1-100-publisher","name":"Vibe Code Bench 1-100","group":"编程","unit":"%","metric":"Overall score","protocol":{"en":"A sequence of dependent changes to a working web application.","zh":"在已有网页应用上依次完成相互依赖的修改。"},"source":{"name":"Vals AI","url":"https://www.vals.ai/benchmarks/vcb-1-100","status":"current_snapshot","published_at":"2026-09-17"},"saturation_excluded":true,"scale_label":{"en":"Rubric score","zh":"量表得分"},"results":[[5171,0.33978,66,5301,-1],[5023,0.30361,1,5302,-1],[5026,0.28533000000000003,1,5302,-1],[5025,0.28,1,5302,-1],[5027,0.27643999999999996,15,5301,-1],[5172,0.2655,66,5301,-1],[5168,0.24833,195,5302,-1],[5039,0.22594,15,5301,-1],[5169,0.212,83,5303,-1],[5030,0.20461,66,5301,-1],[5175,0.20406,1,5302,-1],[5031,0.20010999999999998,15,5301,-1],[5051,0.19994,551,5301,-1],[5033,0.18772,83,5303,-1],[5048,0.18244,21,5301,-1],[5059,0.17533,25,5301,-1],[5047,0.16378,25,5303,-1],[5064,0.16033,551,5301,-1],[5041,0.14822,15,5301,-1],[5045,0.1475,143,5303,-1],[5178,0.14261,417,5303,-1],[5040,0.13821999999999998,1,5302,-1],[5060,0.12833,195,5302,-1],[5068,0.09222,775,5302,-1],[5075,0.07333,370,5304,-1],[5069,0.06694,83,5303,-1]]},{"id":"cua-bench-publisher","name":"CUA-bench","group":"Agent 与工具","unit":"%","metric":"Overall game performance","protocol":{"en":"Six commercial games played using keyboard and mouse.","zh":"使用键盘和鼠标完成六款商业游戏中的任务。"},"source":{"name":"Vals AI","url":"https://www.vals.ai/benchmarks/cua_bench","status":"current_snapshot","published_at":"2026-09-18"},"saturation_excluded":true,"scale_label":{"en":"Rubric score","zh":"量表得分"},"results":[[5027,0.19166667,15,5305,-1],[5025,0.13166667,1,5306,-1],[5026,0.09,1,5306,-1],[5031,0.08333333,15,5305,-1],[5033,0.04166667,83,5307,-1],[5030,0.058333329999999996,66,5308,-1],[5023,0.14,1,5306,-1],[5169,0.048333329999999994,83,5309,-1],[5172,0.07166667,66,5308,-1]]},{"id":"parametric-cad-bench-v3-publisher","name":"Parametric CAD Bench v3","group":"专业工作","unit":"%","metric":"Mean task reward","protocol":{"en":"Version 3 covers 100 FreeCAD tasks: 30 text-to-CAD, 30 create-and-edit and 40 drawing-to-CAD. Failed tasks receive zero.","zh":"V3 包含 100 道 FreeCAD 任务，含 30 道文本建模、30 道创建后编辑和 40 道工程图建模。失败任务计零。"},"source":{"name":"Parametric CAD Bench v3 authors","url":"https://cadbench.ai/","status":"current_snapshot","published_at":"2026-09-21"},"saturation_excluded":true,"results":[[2959,0.6103000000000001,1,5310,-1],[2959,0.5796,1,5311,-1],[2958,0.5687,15,5312,-1],[2956,0.5675,1,5313,-1],[755,0.5085999999999999,1,5313,-1],[2983,0.3619,83,5314,-1],[2968,0.3482,143,5315,-1],[762,0.2998,15,5312,-1],[2974,0.2772,143,5315,-1],[2983,0.2756,83,5316,-1],[2977,0.2492,21,5311,-1],[2979,0.20920000000000002,66,5311,-1],[2973,0.1724,15,5312,-1]]},{"id":"the-endless-exam-2026-paper","name":"The Endless Exam (2026-09, paper results)","group":"数学与科学","unit":"raw","metric":"Overall score","protocol":{"en":"Uncapped construction score across 69 math instances. Three configurations had code and web access.","zh":"69 道数学构造题按作者的无上限总分计分，其中三个设置允许使用代码和网页。"},"source":{"name":"The Endless Exam authors","url":"https://arxiv.org/abs/2609.24555v2","status":"paper_baselines","published_at":"2026-09-21"},"public_history":true,"results":[[4969,7.55,195,5010,-1],[4070,14.52,195,5010,-1],[2584,16.18,195,5010,-1],[5300,38.07,195,5010,-1],[3408,16.7,15,5017,-1],[3408,34.85,15,5010,-1],[3512,37.11,25,5015,-1],[3512,44.9,25,5010,-1],[2956,71.9,1,5017,-1],[2956,43.73,1,5010,-1],[2959,66.7,1,5017,-1],[2959,69.81,1,5010,-1],[2958,75.34,15,5017,-1],[2958,91.9,15,5010,-1],[2958,143.16,15,5317,-1],[3408,119.35,15,5317,-1],[2959,180.06,1,5317,-1]]},{"id":"biomedical-image-analysis-2026-paper","name":"Biomedical image analysis (2026-09, paper results)","group":"医疗","unit":"score","metric":"Reference-normalized score","protocol":{"en":"91 cases from 31 biomedical imaging tasks; 100 corresponds to the best published competition result.","zh":"31 项生物医学影像任务共 91 个病例，100 对应该任务已有的最佳竞赛成绩。"},"source":{"name":"Anthropic","url":"https://www-cdn.anthropic.com/fc1b44717c85dc068bc6ba5024219938094694bd/Claude%20Opus%205.5%20System%20Card.pdf#page=220","status":"paper_baselines","published_at":"2026-09-22"},"public_history":true,"results":[[2958,77.5,15,370,-1],[2959,71.4,1,370,-1],[5166,67.1,1,370,-1],[755,61.0,1,370,-1],[3568,42.3,1,370,-1]]},{"id":"de-novo-protein-binder-design-2026-paper","name":"De novo protein-binder design (2026-09, paper results)","group":"数学与科学","unit":"score","metric":"Mean design score","protocol":{"en":"15 protein targets, averaging five runs per target.","zh":"15 个蛋白靶点，每个靶点取五次运行的平均值。"},"source":{"name":"Anthropic","url":"https://www-cdn.anthropic.com/fc1b44717c85dc068bc6ba5024219938094694bd/Claude%20Opus%205.5%20System%20Card.pdf#page=219","status":"paper_baselines","published_at":"2026-09-22"},"public_history":true,"results":[[2959,82.6,1,370,-1],[5166,79.1,1,370,-1],[755,78.9,1,370,-1],[3568,72.6,1,370,-1]]},{"id":"hle-diamond-publisher","name":"HLE-Diamond (2026-09, CAIS / Scale AI release)","group":"知识与推理","unit":"%","metric":"Accuracy","protocol":{"en":"All 1,000 HLE-Diamond questions without tools, using high reasoning effort.","zh":"在全部 1,000 道 HLE-Diamond 题目上使用高推理力度，不使用工具。"},"source":{"name":"CAIS / Scale AI","url":"https://lastexam.ai/blog/hle-diamond","status":"dated_release","published_at":"2026-09-22"},"public_history":true,"results":[[2958,0.599,15,5318,-1],[2962,0.569,83,5318,-1],[2959,0.546,1,5318,-1],[2960,0.532,15,5318,-1],[2956,0.516,1,5318,-1],[3732,0.34700000000000003,66,5318,-1],[2961,0.331,1,5318,-1],[2967,0.295,21,5318,-1],[2968,0.228,143,5318,-1]]},{"id":"medicinal-chemistry-adme-2026-paper","name":"Medicinal chemistry (ADME) (2026-09, paper results)","group":"医疗","unit":"%","metric":"Accuracy","protocol":{"en":"504 compound-comparison questions, five attempts per question without tools.","zh":"504 道药物分子性质比较题，每题五次作答，不使用工具。"},"source":{"name":"Anthropic","url":"https://www-cdn.anthropic.com/fc1b44717c85dc068bc6ba5024219938094694bd/Claude%20Opus%205.5%20System%20Card.pdf#page=218","status":"paper_baselines","published_at":"2026-09-22"},"public_history":true,"results":[[2959,0.635,1,370,-1],[755,0.579,1,370,-1],[5166,0.5770000000000001,1,370,-1],[2958,0.5660000000000001,15,370,-1],[3568,0.413,1,370,-1]]},{"id":"morphology-to-molecule-matching-2026-paper","name":"Morphology-to-molecule matching (2026-09, paper results)","group":"数学与科学","unit":"%","metric":"Accuracy","protocol":{"en":"Match four groups of cell images to four shuffled perturbations in a closed sandbox.","zh":"在封闭沙箱中，将四组细胞图像与打乱顺序的四种扰动条件配对。"},"source":{"name":"Anthropic","url":"https://www-cdn.anthropic.com/fc1b44717c85dc068bc6ba5024219938094694bd/Claude%20Opus%205.5%20System%20Card.pdf#page=218","status":"paper_baselines","published_at":"2026-09-22"},"public_history":true,"results":[[2959,0.34,1,370,-1],[5166,0.258,1,370,-1],[755,0.25,1,370,-1],[3568,0.068,1,370,-1],[2958,0.228,15,370,-1]]},{"id":"dayjob-2026-paper","name":"DAYJOB (2026-09, paper results)","group":"专业工作","unit":"%","metric":"Finance strict pass@1","protocol":{"en":"September launch cohort. Each of 80 finance tasks was attempted five times. A passing attempt meets every criterion. Healthcare is scored separately.","zh":"9 月首发批次。80 道金融任务各运行五次，单次尝试必须满足全部标准才算通过。医疗任务单独评分。"},"source":{"name":"DAYJOB authors","url":"https://arxiv.org/abs/2610.01306v1","status":"paper_baselines","published_at":"2026-09-23"},"public_history":true,"results":[[2959,0.239,1,5319,-1],[2958,0.215,15,5009,-1],[2956,0.198,1,5319,-1],[2968,0.145,143,5011,-1],[2979,0.14800000000000002,66,5009,-1],[755,0.113,1,5319,-1],[2974,0.128,143,5011,-1],[3453,0.09,1,5319,-1],[3394,0.09300000000000001,15,5009,-1],[762,0.055,15,5009,-1],[3515,0.06,633,5009,-1],[3560,0.03,936,5011,-1],[2977,0.038,21,5009,-1],[3407,0.023,66,5011,-1],[2983,0.027999999999999997,83,5010,-1],[2973,0.018000000000000002,15,5011,-1],[3386,0.027999999999999997,83,5010,-1],[3529,0.023,633,5009,-1],[3568,0.023,1,5319,-1],[2976,0.023,15,5009,-1],[3408,0.018000000000000002,15,5009,-1],[3492,0.01,25,5009,-1],[5114,0.003,3033,5010,-1],[3261,0.0,25,5009,-1],[3412,0.0,83,5010,-1],[203,0.0,199,5009,-1],[3411,0.0,21,5009,-1],[5320,0.0,66,5011,-1],[3338,0.0,417,370,-1],[3418,0.0,307,370,-1]]},{"id":"mentalhealthbench-2026-paper","name":"MentalHealthBench (2026-09, paper results)","group":"医疗","unit":"score","metric":"Mean task-clipped rubric score","protocol":{"en":"Four responses per model per conversation. Scores are clipped at zero and averaged over 1,215 tasks, on the published 0–100 scale.","zh":"每个模型对每段对话作答四次。评分下限截断为零，再对 1,215 个任务取平均，沿用原文的 0–100 量表。"},"source":{"name":"OpenAI","url":"https://openai.com/index/introducing-mentalhealthbench/","status":"paper_baselines","published_at":"2026-09-23"},"public_history":true,"results":[[2958,57.3,15,370,-1],[3394,53.9,15,370,-1],[2959,52.4,1,370,-1],[2976,50.2,15,370,-1],[2979,48.6,66,370,-1],[5321,47.0,15,370,-1],[2956,46.4,1,370,-1],[5322,44.9,15,370,-1],[3568,44.5,1,370,-1],[5323,42.9,15,370,-1],[325,41.7,1,370,-1],[2968,41.3,143,370,-1],[2983,35.5,83,370,-1],[2055,33.5,83,370,-1],[5324,32.1,15,370,-1],[3412,32.1,83,370,-1],[2602,29.5,83,370,-1]]},{"id":"sciuniverse-level-1-publisher","name":"SciUniverse Level 1","group":"数学与科学","unit":"%","metric":"Pass@1","protocol":{"en":"Level 1 scientific tasks. Pass@1 as published by C5R.","zh":"C5R 发布的 Level 1 科学任务单次通过率。"},"source":{"name":"C5R","url":"https://c5r.net/sciuniverse/","status":"current_snapshot","published_at":"2026-09-24"},"results":[[5325,0.45299999999999996,1,370,-1],[5326,0.325,15,370,-1],[5327,0.305,1,370,-1],[5328,0.262,143,370,-1],[5329,0.146,83,370,-1],[5330,0.094,15,370,-1]]},{"id":"artificial-analysis-cyber-index-publisher","name":"Artificial Analysis Cyber Index","group":"安全与可靠性","unit":"score","metric":"Cyber Index","protocol":{"en":"Artificial Analysis held-out evaluation, using the published configuration for each model.","zh":"采用 Artificial Analysis 留出集及各模型公布的评测配置。"},"source":{"name":"Artificial Analysis","url":"https://artificialanalysis.ai/evaluations/artificial-analysis-cyber-index","status":"current_snapshot","published_at":"2026-09-25"},"results":[[5331,28.6636152579411,1,370,-1],[70,33.4198270126414,15,370,-1],[5332,25.0195230207051,1,370,-1],[2967,64.0651080825665,21,370,-1],[5333,56.432341352642,143,370,-1],[113,56.8116757365339,25,370,-1],[5334,36.3757005819883,551,370,-1],[90,25.1449496510851,83,370,-1],[5335,56.0656020403805,370,370,-1],[3534,47.7068149907409,370,370,-1],[167,52.6853594148008,15,370,-1],[3396,49.9235527532159,551,370,-1],[5336,50.8946787572742,417,370,-1],[5337,12.7243142724314,195,370,-1],[3198,22.3880159084502,775,370,-1],[3545,33.6992432071682,370,370,-1],[5338,13.0784136433712,370,370,-1],[5339,12.5980432572072,195,370,-1],[5340,68.6586024559712,15,370,-1]]},{"id":"cwe-bench-v1-publisher","name":"CWE-bench v1","group":"安全与可靠性","unit":"%","metric":"Pass@1","protocol":{"en":"Programmatic exploit and regression tests for the v1 task set.","zh":"以 v1 版本的漏洞验证和回归测试计算单次通过率。"},"source":{"name":"CWE-bench","url":"https://cwe-bench.com/v1","status":"current_snapshot","published_at":"2026-09-25"},"results":[[3380,0.71,83,5162,-1],[2968,0.68,143,5164,-1],[2962,0.68,83,5162,-1],[2958,0.68,15,2300,-1],[2959,0.67,1,3759,-1],[2956,0.58,1,3759,-1],[2974,0.57,143,5164,-1],[3153,0.55,25,5164,-1],[2979,0.55,66,5164,-1],[3524,0.53,370,5164,-1],[3394,0.52,15,2300,-1],[203,0.37,370,5164,-1]]},{"id":"cwe-bench-aa-publisher","name":"CWE-Bench-AA","group":"安全与可靠性","unit":"%","metric":"Pass@1","protocol":{"en":"Artificial Analysis held-out evaluation, using the published configuration for each model.","zh":"采用 Artificial Analysis 留出集及各模型公布的评测配置。"},"source":{"name":"Artificial Analysis","url":"https://artificialanalysis.ai/evaluations/cwe-bench-aa","status":"current_snapshot","published_at":"2026-09-25"},"results":[[5331,0.583333333333333,1,370,-1],[70,0.633333333333333,15,370,-1],[5332,0.508333333333333,1,370,-1],[2967,0.608333333333333,21,370,-1],[5333,0.683333333333333,143,370,-1],[113,0.691666666666667,25,370,-1],[5334,0.558333333333333,551,370,-1],[90,0.525,83,370,-1],[5335,0.633333333333333,370,370,-1],[3534,0.525,370,370,-1],[167,0.566666666666667,15,370,-1],[3396,0.558333333333333,551,370,-1],[5336,0.583333333333333,417,370,-1],[5337,0.266666666666667,195,370,-1],[3198,0.416666666666667,775,370,-1],[3545,0.491666666666667,370,370,-1],[5338,0.275,370,370,-1],[5339,0.283333333333333,195,370,-1],[5340,0.647222222222222,15,370,-1]]},{"id":"cybergym-e2e-aa-publisher","name":"CyberGym-E2E-AA","group":"安全与可靠性","unit":"%","metric":"Pass@1","protocol":{"en":"Artificial Analysis held-out evaluation, using the published configuration for each model.","zh":"采用 Artificial Analysis 留出集及各模型公布的评测配置。"},"source":{"name":"Artificial Analysis","url":"https://artificialanalysis.ai/evaluations/cybergym-e2e-aa","status":"current_snapshot","published_at":"2026-09-25"},"results":[[5331,0.007633587786259542,1,370,-1],[70,0.0,15,370,-1],[5332,0.007633587786259542,1,370,-1],[2967,0.9465648854961832,21,370,-1],[5333,0.7404580152671756,143,370,-1],[113,0.7633587786259542,25,370,-1],[5334,0.2900763358778626,551,370,-1],[90,0.08396946564885496,83,370,-1],[5335,0.7862595419847328,370,370,-1],[3534,0.7480916030534351,370,370,-1],[167,0.7786259541984732,15,370,-1],[3396,0.7404580152671756,551,370,-1],[5336,0.7938931297709924,417,370,-1],[5337,0.0,195,370,-1],[3198,0.24427480916030533,775,370,-1],[3545,0.45038167938931295,370,370,-1],[5338,0.10687022900763359,370,370,-1],[5339,0.0,195,370,-1],[5340,0.9312977099236641,15,370,-1]]},{"id":"deepsecbench-aa-publisher","name":"DeepsecBench-AA","group":"安全与可靠性","unit":"score","metric":"Median F2","protocol":{"en":"Artificial Analysis held-out evaluation, using the published configuration for each model.","zh":"采用 Artificial Analysis 留出集及各模型公布的评测配置。"},"source":{"name":"Artificial Analysis","url":"https://artificialanalysis.ai/evaluations/deepsecbench-aa","status":"current_snapshot","published_at":"2026-09-25"},"results":[[5331,26.894153661864,1,370,-1],[70,36.9261477045908,15,370,-1],[5332,23.461876950156,1,370,-1],[2967,36.705502364748,21,370,-1],[5333,26.917889197875198,143,370,-1],[113,24.9324826803397,25,370,-1],[5334,24.2861348248454,551,370,-1],[90,14.5379023883697,83,370,-1],[5335,26.237518589335,370,370,-1],[3534,15.8112846668792,370,370,-1],[167,23.5268161578884,15,370,-1],[3396,19.8915233995969,551,370,-1],[5336,14.961389961390001,417,370,-1],[5337,11.5062761506276,195,370,-1],[3198,1.0699001426533499,775,370,-1],[3545,6.892895015906681,370,370,-1],[5338,1.0482180293501,370,370,-1],[5339,9.4607964382884,195,370,-1],[5340,48.1238141533251,15,370,-1]]},{"id":"anestrace-2026-paper","name":"AnesTRACE (2026-09, paper results)","group":"医疗","unit":"score","metric":"L2 single-point decision average","protocol":{"en":"Single-point anesthetic management with standardized textual evidence. The reported average combines risk, diagnosis, intervention and reassessment scores on a 100-point scale.","zh":"模型根据标准化文本进行单步麻醉管理决策。平均分涵盖风险、诊断、干预和重新评估，满分 100。"},"source":{"name":"AnesTRACE authors","url":"https://arxiv.org/abs/2609.32740v2","status":"paper_baselines","published_at":"2026-09-26"},"score_max":100,"public_history":true,"results":[[4977,92.2,15,5341,-1],[5342,75.2,83,5341,-1],[4960,90.8,1,5341,-1],[3185,83.6,25,5341,-1],[3157,77.4,195,5341,-1],[2584,73.0,195,5341,-1],[5300,72.0,195,5341,-1],[5343,62.9,15,5341,-1],[4966,55.7,551,5341,-1],[4070,66.1,195,5341,-1],[5344,49.7,195,5341,-1],[5345,51.3,370,5341,-1],[5346,60.3,370,5341,-1],[5347,57.7,370,5341,-1],[5348,54.9,3063,5341,-1],[5349,57.7,370,5341,-1],[5350,41.1,370,5341,-1],[5351,44.7,370,5341,-1],[5352,50.8,370,5341,-1]]},{"id":"autopdebench-2026-paper","name":"AutoPDEBench (2026-09, paper results)","group":"数学与科学","unit":"raw","metric":"HighSpeedFlow RMSE","protocol":{"en":"Root mean squared error of the solver designed by each backbone for HighSpeedFlow. The agent, prompts and search space are held constant. Lower is better.","zh":"各模型在相同代理框架、提示和搜索空间下为 HighSpeedFlow 设计求解器，比较均方根误差。数值越低越好。"},"source":{"name":"AutoPDEBench authors","url":"https://arxiv.org/abs/2609.32245v1","status":"paper_baselines","published_at":"2026-09-26"},"higher_is_better":false,"public_history":true,"results":[[3208,5.066,15,370,-1],[4168,8.332,15,370,-1],[4972,5.537,15,370,-1],[4196,5.771,15,370,-1],[3473,6.766,1,370,-1],[3476,6.687,1,370,-1],[5105,5.299,143,370,-1],[413,9.3,83,370,-1],[3209,368.5,25,370,-1],[411,6.14,195,370,-1],[425,5.784,21,370,-1]]},{"id":"cua-swe-publisher","name":"CUA-SWE","group":"编程","unit":"%","metric":"Web single-attempt pass@1","protocol":{"en":"Single-attempt success on the same 36 web software-engineering tasks. Code-only and hybrid computer-use runs are listed separately.","zh":"在同一组 36 道网页软件工程题上比较单次成功率，纯代码和可操作图形界面的设置分列。"},"source":{"name":"CUA-SWE authors","url":"https://github.com/kingofspace0wzz/cua-swe","status":"current_snapshot","published_at":"2026-09-26"},"results":[[2958,0.2777777777777778,15,5353,-1],[2958,0.6666666666666667,15,5354,-1],[762,0.19444444444444442,15,5353,-1],[762,0.5833333333333334,15,5354,-1],[3408,0.1111111111111111,15,5353,-1],[3408,0.05555555555555555,15,5354,-1],[2973,0.1111111111111111,15,5353,-1],[2973,0.1388888888888889,15,5354,-1],[3156,0.2222222222222222,370,5353,-1],[3156,0.5555555555555556,370,5354,-1],[3401,0.16666666666666669,370,5353,-1],[3401,0.2777777777777778,370,5354,-1],[3393,0.08333333333333334,370,5353,-1],[3393,0.0,370,5354,-1],[3151,0.19444444444444442,370,5353,-1],[3151,0.4722222222222222,370,5354,-1],[2974,0.3055555555555556,143,5353,-1],[2974,0.5555555555555556,143,5354,-1],[762,0.5555555555555556,15,5355,-1],[3156,0.5555555555555556,370,5356,-1],[3384,0.2777777777777778,370,5353,-1],[3384,0.4722222222222222,370,5354,-1]]},{"id":"financialauditbench-2026-paper","name":"FinancialAuditBench (2026-09, paper results)","group":"专业工作","unit":"%","metric":"Strict pass rate","protocol":{"en":"Audit workpapers must satisfy every applicable criterion. The authors average across tasks and eight independent runs per model-task pair.","zh":"审计工作底稿须满足所有适用标准。每个模型在每项任务上独立运行八次，再按任务和运行取平均。"},"source":{"name":"FinancialAuditBench authors","url":"https://arxiv.org/abs/2609.32835v1","status":"paper_baselines","published_at":"2026-09-26"},"public_history":true,"results":[[2959,0.6917,1,5357,-1],[2956,0.6583,1,5357,-1],[2974,0.4722,143,5357,-1],[762,0.44170000000000004,15,5357,-1],[2958,0.4361,15,5357,-1],[2973,0.4028,15,5357,-1],[2979,0.3944,66,5357,-1],[3512,0.3306,25,5357,-1],[227,0.2944,551,5358,-1],[2977,0.2889,21,5357,-1],[2983,0.275,83,5357,-1]]},{"id":"discern-2026-paper","name":"DISCERN (2026-09, paper results)","group":"数学与科学","unit":"score","metric":"Overall life-science quality","protocol":{"en":"The overall score combines three levels of scientific investigation. Fable did not cover enough tasks for a standalone score. The Fable-to-Opus row replays recorded Opus 5 outputs wherever Fable refused.","zh":"总分涵盖三个层次的生命科学研究任务。Fable 的作答覆盖不足以计算独立总分。Fable 转 Opus 这一行在 Fable 拒答的任务上采用已记录的 Opus 5 输出。"},"source":{"name":"DISCERN authors","url":"https://arxiv.org/abs/2609.33357v2","status":"paper_baselines","published_at":"2026-09-27"},"score_max":100,"public_history":true,"results":[[2958,78.1,15,370,-1],[4048,75.8,15,370,-1],[755,75.6,1,370,-1],[2977,73.2,21,370,-1],[3154,70.5,551,370,-1],[227,67.8,551,370,-1],[3492,62.4,25,370,-1],[345,39.8,15,370,-1]]},{"id":"long-horizon-analog-design-bench-2026-paper","name":"Long-Horizon Analog Design Bench (2026-09, paper results)","group":"数学与科学","unit":"%","metric":"Full-specification pass@1","protocol":{"en":"50 analog circuit tasks, three two-hour attempts per configuration and task. Each passing design must satisfy every specification gate.","zh":"50 道模拟电路设计题，每个配置每题进行三次两小时尝试。电路满足全部规格才算通过。"},"source":{"name":"Long-Horizon Analog Design Bench authors","url":"https://arxiv.org/abs/2609.33356v1","status":"paper_baselines","published_at":"2026-09-27"},"public_history":true,"results":[[5359,0.08,370,5360,-1],[227,0.08,551,5093,-1],[5361,0.14,370,4992,-1],[3261,0.16,25,5093,-1],[2980,0.1867,551,5093,-1],[3560,0.3,195,5362,-1],[3408,0.31329999999999997,15,5095,-1],[3154,0.32,551,5093,-1],[3492,0.42,25,5093,-1],[2977,0.49329999999999996,21,5093,-1],[2973,0.49329999999999996,15,5095,-1],[3196,0.6067,15,5100,-1],[762,0.68,15,5095,-1],[755,0.6933,1,5093,-1],[3453,0.78,1,5093,-1]]},{"id":"slopbench-2026-paper","name":"SlopBench (2026-09, paper results)","group":"生成与创作","unit":"score","metric":"Overall Slop Index","protocol":{"en":"Overall index of writing patterns across email, essays, workplace chat and social posts. Lower scores mean fewer measured patterns.","zh":"综合邮件、文章、工作聊天和社交帖中的写作模式计算指数。分数越低，测得的模式越少。"},"source":{"name":"SlopBench authors","url":"https://arxiv.org/abs/2609.33905v1","status":"paper_baselines","published_at":"2026-09-27"},"score_max":100,"higher_is_better":false,"public_history":true,"results":[[4956,40.6,417,370,-1],[3453,35.6,1,370,-1],[2973,32.8,15,370,-1],[3479,31.8,1,370,-1],[762,31.7,15,370,-1],[3408,31.7,15,370,-1],[3654,31.2,551,370,-1],[3568,30.5,1,370,-1],[3587,28.7,195,370,-1],[3415,27.8,83,370,-1],[3494,26.8,15,370,-1],[325,24.9,1,370,-1],[3412,24.3,83,370,-1],[3391,24.1,143,370,-1],[3492,23.8,25,370,-1],[3491,23.1,775,370,-1],[3413,23.1,66,370,-1],[3490,21.1,21,370,-1]]},{"id":"apex-voice-2026-paper","name":"APEX-Voice (2026-09, paper results)","group":"专业工作","unit":"%","metric":"Text-control workflow pass@1","protocol":{"en":"Text-only comparison on the same work scenarios. Opus 5.5 and the other four text agents receive written exchanges. The paper tabulates voice systems separately.","zh":"五个文本代理在相同工作场景中接收书面交互。论文另表报告语音代理成绩。"},"source":{"name":"APEX-Voice authors","url":"https://arxiv.org/abs/2609.34973v1","status":"paper_baselines","published_at":"2026-09-28"},"public_history":true,"results":[[5363,0.5870000000000001,1,5364,-1],[5365,0.55,15,5364,-1],[3196,0.62,15,5364,-1],[5366,0.5429999999999999,83,5364,-1],[2977,0.5579999999999999,21,5364,-1]]},{"id":"arch-b-2026-paper","name":"ARCH-B (2026-09, paper results)","group":"视觉与视频","unit":"%","metric":"Overall accuracy","protocol":{"en":"All 354 architecture questions. Invalid responses and terminal inference failures count as incorrect.","zh":"全部 354 道建筑视觉题按题计算准确率。答案无效或推理最终失败均计错。"},"source":{"name":"ARCH-B authors","url":"https://arxiv.org/abs/2609.34047v1","status":"paper_baselines","published_at":"2026-09-28"},"public_history":true,"results":[[413,0.8390000000000001,83,370,-1],[5109,0.8305,83,370,-1],[3473,0.8023,1,370,-1],[1500,0.7797,15,370,-1],[3208,0.7655,15,5367,-1],[427,0.7598999999999999,83,370,-1],[5368,0.7429000000000001,15,5367,-1],[457,0.7147,83,370,-1],[3208,0.7090000000000001,15,370,-1],[5369,0.7034,1,370,-1],[5370,0.6525,1,370,-1],[5368,0.6441,15,370,-1],[433,0.5706,143,370,-1],[5371,0.5028,143,370,-1],[5160,0.5,15,5367,-1],[5160,0.4661,15,370,-1],[5372,0.3898,83,370,-1],[4278,0.3785,1,370,-1],[5373,0.3446,1,370,-1],[5374,0.32770000000000005,370,370,-1],[5375,0.29100000000000004,15,5367,-1],[5376,0.1751,1,370,-1],[4164,0.1751,83,370,-1],[5375,0.16949999999999998,15,370,-1],[5377,0.1045,83,370,-1]]},{"id":"cheatbench-publisher","name":"CheatBench","group":"安全与可靠性","unit":"%","metric":"Overall cheating rate","protocol":{"en":"Equal-weight mean over ten categories. The authors publish provider-harness and Terminus-2 results separately. Sycophancy uses the same direct-API measurement in both. Lower is better.","zh":"十类任务的作弊发生率取等权平均。作者分别公布官方代理框架与 Terminus-2 的结果；两个设置共用直接 API 测得的迎合指标。数值越低越好。"},"source":{"name":"CheatBench authors","url":"https://www.cheatbench.ai/","status":"current_snapshot","published_at":"2026-09-28"},"higher_is_better":false,"results":[[2968,0.7798802282191432,143,5296,-1],[2983,0.7522802122119212,83,5378,-1],[2983,0.8319468788785879,83,5379,-1],[3492,0.7506484205130559,25,5380,-1],[3492,0.7624905257762138,25,5379,-1],[2960,0.49299139693369864,15,2300,-1],[2960,0.42799139693369864,15,5379,-1],[3394,0.7194579484630416,15,2300,-1],[2977,0.7004622553169682,21,5381,-1],[2977,0.7332200644740071,21,5379,-1],[2958,0.47382567049806906,15,2300,-1],[2958,0.4498256704980691,15,5379,-1],[2956,0.4576383929949676,1,3759,-1],[2979,0.37009121647470716,66,5382,-1],[2979,0.8649245498080406,66,5379,-1],[2959,0.11216779681390487,1,3759,-1],[2959,0.07216779681390487,1,5379,-1],[2974,0.7908324742615997,143,5296,-1],[2974,0.7958324742615998,143,5379,-1],[762,0.7697188333491589,15,2300,-1],[762,0.7485521666824922,15,5379,-1],[755,0.4534750139491212,1,3759,-1],[755,0.5659750139491213,1,5379,-1],[2962,0.08552348846438015,83,5378,-1],[2962,0.2880234884643802,83,5379,-1],[3732,0.12663681357813156,66,5382,-1],[3732,0.7308034802447984,66,5379,-1],[2987,0.8417546506269905,66,5382,-1],[2987,0.8767546506269904,66,5379,-1]]},{"id":"painterbench-2026-paper","name":"PainterBench (2026-09, paper results)","group":"生成与创作","unit":"score","metric":"Rated creativity (1–5)","protocol":{"en":"Mean creativity rating for drawings made with the drawing agent, using the original five-point scale.","zh":"绘图代理作品的创造力评分均值，保留原始五分制。"},"source":{"name":"PainterBench authors","url":"https://arxiv.org/abs/2609.34195v1","status":"paper_baselines","published_at":"2026-09-28"},"score_max":5,"public_history":true,"score_decimals":2,"results":[[3453,2.79,1,5383,-1],[755,2.95,1,5383,-1],[3568,2.43,1,5383,-1],[2958,3.95,15,5383,-1],[3408,3.12,15,5383,-1],[762,3.45,15,5383,-1],[2983,3.55,83,5383,-1],[3386,3.4,83,5383,-1],[3414,2.12,83,5383,-1],[362,2.33,66,5383,-1],[2979,3.11,66,5383,-1],[3391,3.12,143,5383,-1],[5384,2.94,417,5383,-1],[4070,2.21,195,5383,-1]]},{"id":"revivebench-2026-paper","name":"ReviveBench (2026-09, paper results)","group":"编程","unit":"score","metric":"Reconstruction engines passed","protocol":{"en":"Number of industrial software engines meeting their task-specific clean-room reconstruction criteria, out of 13. The authors count outcomes from all valid recorded runs.","zh":"13 款工业软件引擎中，达到各自独立重建标准的数量。作者按所有有效运行的结果统计。"},"source":{"name":"ReviveBench authors","url":"https://arxiv.org/abs/2609.36161v1","status":"paper_baselines","published_at":"2026-09-28"},"score_max":13,"public_history":true,"results":[[3149,13.0,370,3759,-1],[3156,13.0,370,3759,-1],[3393,10.0,370,3759,-1],[3496,2.0,370,3759,-1],[3160,9.0,15,5385,-1],[5386,6.0,15,5385,-1],[5387,5.0,15,5385,-1],[5388,7.0,551,5385,-1]]},{"id":"rle-bench-publisher","name":"RLE-Bench","group":"Agent 与工具","unit":"score","metric":"RLE Index","protocol":{"en":"The index weights four robot-learning workflows equally. The authors’ scoring code averages task-family scores within each workflow and then averages the workflows. Scores are on a 100-point scale.","zh":"机器人学习指数涵盖四类工作。作者的计分程序先对每类工作中的任务族取平均，再对四类工作等权平均，满分 100。"},"source":{"name":"RLE-Bench authors","url":"https://rle-bench.github.io/","status":"current_snapshot","published_at":"2026-09-14"},"score_max":100,"results":[[2958,73.4,15,2300,-1],[2962,72.4,83,5299,-1],[2959,69.4,1,3759,-1],[2960,67.0,15,2300,-1],[2956,64.0,1,3759,-1],[755,53.4,1,3759,-1],[2967,51.8,21,3759,-1],[762,48.7,15,2300,-1],[3153,38.9,5389,3759,-1],[2974,29.0,143,5296,-1],[3386,26.8,83,5299,-1],[2980,26.5,48,3759,-1],[3479,24.3,1,3759,-1],[3408,20.3,15,2300,-1],[2976,17.7,15,2300,-1],[2973,16.3,15,2300,-1]]},{"id":"tcsalgbench-2026-paper","name":"TCSAlgBench (2026-09, paper results)","group":"数学与科学","unit":"%","metric":"First-run proof acceptance","protocol":{"en":"A single directly generated proof on each of 398 theoretical computer science challenges, using seed 1 and a majority vote of three verifiers.","zh":"每道理论计算机科学题直接生成一份证明，共 398 题。使用第一个随机种子，三个验证器多数同意即通过。"},"source":{"name":"TCSAlgBench authors","url":"https://arxiv.org/abs/2609.35606v1","status":"paper_baselines","published_at":"2026-09-28"},"public_history":true,"results":[[762,0.057999999999999996,15,5390,-1],[762,0.048,15,5391,-1],[762,0.048,15,5392,-1],[3196,0.033,15,5391,-1],[3196,0.02,15,5392,-1],[3401,0.008,370,5390,-1],[3401,0.003,370,5391,-1],[3401,0.003,370,5392,-1],[3151,0.068,370,5391,-1],[3151,0.045,370,5392,-1]]},{"id":"timeline-bench-2026-paper","name":"Timeline-Bench (2026-09, paper results)","group":"视觉与视频","unit":"%","metric":"Resolution rate","protocol":{"en":"One run on each of 56 video editing tasks, with the highest reasoning effort available for each model. Resolution requires passing the benchmark’s quality test as well as its other tests.","zh":"56 道视频剪辑任务各执行一次，使用模型可用的最高推理强度。成片须同时通过质量检查和其他测试。"},"source":{"name":"Timeline-Bench authors","url":"https://arxiv.org/abs/2609.35143v1","status":"paper_baselines","published_at":"2026-09-28"},"public_history":true,"results":[[2958,0.214,15,5393,-1],[2956,0.179,1,5393,-1],[755,0.179,1,5393,-1],[762,0.179,15,5393,-1],[2974,0.125,143,5393,-1],[2983,0.107,83,5393,-1],[3529,0.107,551,5393,-1],[5394,0.071,25,5393,-1],[3560,0.036000000000000004,195,5393,-1],[3482,0.018000000000000002,83,5393,-1],[755,0.23199999999999998,1,3759,-1],[2958,0.214,15,5395,-1],[2956,0.179,1,3759,-1],[762,0.08900000000000001,15,5395,-1],[2958,0.268,15,5396,-1],[2958,0.036000000000000004,15,5397,-1]]},{"id":"code4scene-2026-paper","name":"Code4Scene (2026-09, paper results)","group":"生成与创作","unit":"score","metric":"Public-set overall score","protocol":{"en":"The public-set score gives equal weight to text-to-scene (20 cases) and image-to-scene (75 cases). Each task rewards semantic or repair quality together with physical validity.","zh":"公开集将文本生成场景（20 例）和图像生成场景（75 例）等权计分，结合语义或修复质量与物理有效性。"},"source":{"name":"Code4Scene authors","url":"https://arxiv.org/abs/2609.36777v1","status":"paper_baselines","published_at":"2026-09-29"},"score_max":1,"public_history":true,"score_decimals":3,"results":[[2958,0.619,15,5009,-1],[2983,0.619,83,5010,-1],[2956,0.606,1,5009,-1],[755,0.593,1,5009,-1],[762,0.55,15,5010,-1],[2979,0.502,66,5017,-1],[2980,0.415,551,5009,-1],[3403,0.378,195,5398,-1],[2974,0.376,143,5010,-1],[3403,0.371,195,5399,-1],[203,0.312,370,5010,-1],[2585,0.298,83,5399,-1],[2585,0.285,83,5398,-1],[3512,0.227,25,5010,-1]]},{"id":"e2e-swe-2026-paper","name":"E2E-SWE (2026-09, paper results)","group":"编程","unit":"%","metric":"Pass@1","protocol":{"en":"Across four independent runs on 186 repository-building tasks, a run passes when every hidden test succeeds.","zh":"186 道代码仓库构建任务各运行四次。一次运行须通过全部隐藏测试才算成功。"},"source":{"name":"E2E-SWE authors","url":"https://arxiv.org/abs/2609.38335v1","status":"paper_baselines","published_at":"2026-09-29"},"public_history":true,"results":[[3453,0.653,1,5400,-1],[755,0.677,1,5401,-1],[3568,0.431,1,5401,-1],[3479,0.519,1,5401,-1],[2261,0.226,1,5400,-1],[762,0.612,15,5402,-1],[2973,0.518,15,5402,-1],[3196,0.461,15,5401,-1],[2979,0.452,66,5402,-1],[2983,0.36700000000000005,83,5400,-1],[3405,0.11699999999999999,83,5400,-1],[2977,0.423,21,5400,-1],[227,0.222,551,5400,-1]]},{"id":"klinikebench-2026-paper","name":"KlinikeBench (2026-09, paper results)","group":"医疗","unit":"%","metric":"Strict pass@1","protocol":{"en":"On 333 simulated clinical cases an attempt must reach the correct diagnosis, complete the required clinical actions and cover every must-ask item.","zh":"333 个模拟病例中，模型须同时给出正确诊断、完成规定临床操作并问完所有必问项。"},"source":{"name":"KlinikeBench authors","url":"https://arxiv.org/abs/2609.38480v2","status":"paper_baselines","published_at":"2026-09-29"},"public_history":true,"results":[[755,0.294,1,5403,-1],[2959,0.28800000000000003,1,5403,-1],[3479,0.168,1,5403,-1],[3568,0.138,1,5403,-1],[283,0.099,1,5403,-1],[3572,0.084,1,5403,-1],[325,0.027000000000000003,1,5403,-1],[5404,0.243,15,5403,-1],[4048,0.237,15,5403,-1],[3184,0.23399999999999999,15,5403,-1],[3196,0.168,15,5403,-1],[4958,0.159,15,5403,-1],[3748,0.153,15,5403,-1],[5298,0.084,15,5403,-1],[5297,0.075,15,5403,-1],[345,0.075,15,5403,-1],[353,0.039,15,5403,-1],[5405,0.036000000000000004,15,5403,-1],[5406,0.021,15,5403,-1],[5407,0.009000000000000001,15,5403,-1],[354,0.006,15,5403,-1],[5408,0.18899999999999997,83,5403,-1],[5115,0.156,83,5403,-1],[5409,0.10800000000000001,83,5403,-1],[5410,0.222,143,5403,-1],[5295,0.21899999999999997,143,5403,-1],[5411,0.213,143,5403,-1],[5412,0.081,25,5403,-1],[5413,0.039,25,5403,-1],[3187,0.23399999999999999,21,5403,-1],[3186,0.048,551,5403,-1]]},{"id":"librarydesignbench-2026-paper","name":"LibraryDesignBench (2026-09, paper results)","group":"编程","unit":"score","metric":"Library designer score","protocol":{"en":"A library designer is scored through the same three downstream implementers. The score combines test performance and the simplicity of each implementation.","zh":"各模型设计的库均交给同样的三个实现代理。得分结合了实现的测试通过情况和代码简洁度。"},"source":{"name":"LibraryDesignBench authors","url":"https://arxiv.org/abs/2609.36730v1","status":"paper_baselines","published_at":"2026-09-29"},"score_max":100,"public_history":true,"results":[[3492,31.2,25,5414,-1],[3149,47.5,370,5414,-1],[3149,39.9,370,3759,-1],[3515,41.8,551,5414,-1],[762,39.5,15,2300,-1],[2958,45.1,15,5414,-1],[2958,44.1,15,2300,-1],[3394,38.5,15,5414,-1],[2974,39.7,143,5414,-1],[2977,44.0,21,5414,-1],[3384,48.9,370,5414,-1]]},{"id":"zero2repo-2026-paper","name":"Zero2Repo (2026-09, paper results)","group":"编程","unit":"%","metric":"Pass rate","protocol":{"en":"One adopted run per model configuration and task on the 17-task release. Passing requires every hidden test to succeed.","zh":"17 道任务各取作者选定的一次运行，全部隐藏测试通过才算解决。"},"source":{"name":"Zero2Repo authors","url":"https://arxiv.org/abs/2609.38269v3","status":"paper_baselines","published_at":"2026-09-29"},"public_history":true,"results":[[2958,0.882,15,4993,-1],[2959,0.8240000000000001,1,5415,-1],[2958,0.706,15,5415,-1],[2959,0.647,1,4992,-1],[2961,0.588,1,4992,-1],[2961,0.529,1,5415,-1],[3394,0.529,15,4993,-1],[2962,0.529,83,5415,-1],[2968,0.47100000000000003,143,5416,-1],[5284,0.41200000000000003,25,5415,-1],[3394,0.353,15,5415,-1],[2982,0.353,195,5415,-1],[2977,0.294,21,5415,-1],[3154,0.294,551,5415,-1],[3524,0.235,370,5415,-1],[3492,0.11800000000000001,25,5415,-1],[3534,0.059000000000000004,370,5415,-1],[5417,0.059000000000000004,370,5415,-1]]},{"id":"drivingbench-2026-paper","name":"DrivingBench (2026-09, paper results)","group":"Agent 与工具","unit":"%","metric":"Best course progress","protocol":{"en":"Best progress over up to three attempts on a 127-metre physical driving course. GPT-6 Astra finished on attempt two.","zh":"各模型在 127 米实车赛道上最多尝试三次，取最佳完成比例。GPT-6 Astra 第二次即完赛。"},"source":{"name":"DrivingBench authors","url":"https://arxiv.org/abs/2609.38948v1","status":"paper_baselines","published_at":"2026-09-30"},"public_history":true,"results":[[2958,1.0,15,5418,-1],[2956,0.45,1,5419,-1],[2974,0.11,143,5420,-1],[762,0.06,15,5418,-1]]},{"id":"libero-agent-2026-paper","name":"LIBERO-Agent (2026-09, paper results)","group":"Agent 与工具","unit":"score","metric":"Stable performance score","protocol":{"en":"Weighted score on 30 robot tasks spanning perception, short-horizon manipulation and long-horizon manipulation, from 90 episodes per agent.","zh":"每个代理执行 90 个回合，按感知、短程操作和长程操作三类共 30 项机器人任务计算加权分。"},"source":{"name":"LIBERO-Agent authors","url":"https://arxiv.org/abs/2609.39507v1","status":"paper_baselines","published_at":"2026-09-30"},"score_max":100,"public_history":true,"results":[[2958,45.0,15,5421,-1],[762,10.8,15,5421,-1],[755,19.3,1,5422,-1],[2956,15.8,1,5422,-1],[3153,4.5,25,5423,-1],[2977,5.5,21,5424,-1],[3157,10.5,195,5425,-1]]},{"id":"osworld-science-2026-paper","name":"OSWorld-Science (2026-09, paper results)","group":"Agent 与工具","unit":"%","metric":"Mean task score","protocol":{"en":"Mean task-specific score across 146 scientific software tasks, including partial credit. Runs without a gradable outcome count as zero.","zh":"146 道科学软件任务按各自标准给分并取均值，可得部分分；无法评分的运行计零。"},"source":{"name":"OSWorld-Science authors","url":"https://arxiv.org/abs/2609.39903v1","status":"paper_baselines","published_at":"2026-09-30"},"saturation_excluded":true,"public_history":true,"results":[[2956,0.737,1,5426,-1],[755,0.579,1,5426,-1],[3568,0.263,1,5426,-1],[3160,0.501,15,5426,-1],[5387,0.318,15,5426,-1],[5386,0.226,15,5426,-1],[2958,0.574,15,5426,-1],[3412,0.172,83,5426,-1],[2977,0.302,21,5426,-1],[5427,0.158,195,5426,-1],[4961,0.152,195,5426,-1],[3491,0.13,775,5426,-1]]},{"id":"posttrainbench-v1-2-publisher","name":"PostTrainBench v1.2","group":"数学与科学","unit":"%","metric":"Weighted post-training performance","protocol":{"en":"Agents post-train four small base models. The authors recompute performance across six weighted evaluations under v1.2. Fable’s GPQA runs include documented Opus fallbacks.","zh":"智能体为四款小型基座模型做后训练。v1.2 按六项评测的权重重新计算综合表现。Fable 的部分 GPQA 任务触发了 Opus 回退。"},"source":{"name":"PostTrainBench v1.2 authors","url":"https://posttrainbench.com/blog/posttrainbench-1-2/","status":"current_snapshot","published_at":"2026-09-30"},"results":[[3151,0.37270000000000003,370,5428,-1],[3149,0.4465,370,5429,-1],[3654,0.304,551,5430,-1],[3515,0.3452,551,5431,-1],[3529,0.3679,551,5431,-1],[5101,0.1776,15,5432,-1],[3581,0.23870000000000002,15,5433,-1],[5434,0.31010000000000004,15,5435,-1],[5436,0.4188,15,5435,-1],[5437,0.38789999999999997,15,5435,-1],[3412,0.18109999999999998,83,5438,-1],[3391,0.2262,143,5439,-1],[2977,0.3148,21,5440,-1],[3400,0.2434,370,5441,-1],[3401,0.3066,370,5442,-1],[3401,0.31,370,5431,-1],[3156,0.3566,370,5443,-1],[3384,0.43799999999999994,370,5431,-1],[5444,0.41659999999999997,370,5445,-1]]},{"id":"odysseys2-publisher","name":"Odysseys2","group":"Agent 与工具","unit":"score","metric":"Mean web-task rubric score","protocol":{"en":"The Odysseys2 judge grades delivered files, screenshots and actions across 100 real web requests, giving zero credit for attempts it flags for reward hacking. Content filters left 98 graded attempts for Claude Fable 5. Every other agent has 100.","zh":"Odysseys2 裁判依据交付文件、截图和操作记录，为 100 个真实网页请求评分。判定为钻评分规则空子的尝试计零分。Claude Fable 5 有两次任务触发内容过滤，以 98 次有效尝试取均值，其余智能体各有 100 次。"},"source":{"name":"Odysseys2 authors","url":"https://odysseysbench.com/v2/","status":"current_snapshot","published_at":"2026-10"},"score_max":100,"results":[[2958,84.6,15,5446,-1],[2959,71.2,1,5447,-1],[2968,68.2,143,5447,-1],[762,62.0,15,5446,-1],[3453,61.5,1,5448,-1],[3408,59.7,15,5447,-1],[755,58.3,1,5446,-1],[3391,48.2,143,5446,-1],[2973,44.6,15,5446,-1],[3568,42.3,1,5446,-1],[5449,24.1,15,5446,-1],[5450,10.8,83,5446,-1],[2983,10.7,83,5447,-1],[3386,5.8,83,5446,-1],[5451,1.6,83,5446,-1]]},{"id":"d2k-bench-2026-paper","name":"D2K-Bench (2026-10, paper results)","group":"编程","unit":"raw","metric":"Relative kernel performance score","protocol":{"en":"Geometric mean over 26 GPU kernel tasks. Each task is measured against its Fable 5 first-correct baseline. A failed check or invalid timing receives 0.5. The table reports guided and unguided final submissions.","zh":"26 道 GPU 内核题按几何均值计分，以 Fable 5 首个正确内核的耗时为基准。未通过验证或无法有效计时的任务得 0.5。表中区分有无专家设计指导。"},"source":{"name":"D2K-Bench authors","url":"https://arxiv.org/abs/2610.03226v1","status":"paper_baselines","published_at":"2026-10-02"},"public_history":true,"results":[[4977,2.68,15,5452,-1],[4977,3.33,15,5453,-1],[3194,1.38,1,5454,-1],[3194,2.31,1,5455,-1],[4979,1.31,15,5452,-1],[4979,2.01,15,5453,-1],[3154,1.18,551,5454,-1],[3154,1.34,551,5455,-1],[3155,1.16,21,5454,-1],[3155,1.38,21,5455,-1]]},{"id":"minteval-2026-paper","name":"MintEval (2026-10, paper results)","group":"编程","unit":"%","metric":"ActionMatch on compiled runs","protocol":{"en":"Open specification, 200 shared tasks stratified by strategy memory horizon. ActionMatch is the fraction of active bars where the generated and reference positions agree, averaged over compiled runs.","zh":"开放式任务按策略记忆跨度分层选取 200 题。ActionMatch 在有仓位或目标仓位的时间段比较生成策略与参考策略的动作，仅对成功运行的策略取均值。"},"source":{"name":"MintEval authors","url":"https://arxiv.org/abs/2610.03080v1","status":"paper_baselines","published_at":"2026-10-02"},"public_history":true,"results":[[5456,0.889,1,5457,-1],[5160,0.575,15,5457,-1],[5458,0.42,1,5457,-1],[5459,0.181,195,5457,-1],[5460,0.076,195,5457,-1]]},{"id":"agentpersonabench-2026-paper","name":"AgentPersonaBench (2026-10, paper results)","group":"Agent 与工具","unit":"%","metric":"Web full-pass rate","protocol":{"en":"Full pass on all checked persona traits for 707 web tasks at medium reasoning effort.","zh":"在 707 道网页任务中，模型须符合全部受检人格特征。推理强度为中档。"},"source":{"name":"AgentPersonaBench authors","url":"https://arxiv.org/abs/2610.04379","status":"paper_baselines","published_at":"2026-10-03"},"public_history":true,"results":[[4943,0.861,15,5461,-1],[2114,0.7859999999999999,15,5461,-1],[5150,0.5770000000000001,15,5461,-1],[5462,0.72,15,5461,-1],[5463,0.652,15,5461,-1],[5464,0.56,15,5461,-1],[5465,0.853,370,5461,-1],[5466,0.767,370,5461,-1],[5467,0.72,370,5461,-1],[5468,0.85,83,5461,-1],[5469,0.835,83,5461,-1],[5470,0.7070000000000001,143,5461,-1],[5471,0.6559999999999999,551,5461,-1],[5472,0.6,551,5461,-1],[5473,0.68,195,5461,-1],[5474,0.669,195,5461,-1],[4952,0.75,21,5461,-1],[5475,0.6859999999999999,25,5461,-1],[3209,0.616,25,5461,-1],[1321,0.44299999999999995,25,5461,-1]]},{"id":"medimageosworld-2026-paper","name":"MedImageOSWorld (Provider APIs) (2026-10, paper results)","group":"医疗","unit":"%","metric":"Weighted success rate, API cohort","protocol":{"en":"Provider APIs, first five assets per task. Success is weighted by capability level across 49 imaging-console tasks.","zh":"两个模型均通过供应商 API 运行，每题取前五个素材。49 项影像设备操作任务按能力等级加权。"},"source":{"name":"MedImageOSWorld authors","url":"https://arxiv.org/abs/2610.04800","status":"paper_baselines","published_at":"2026-10-03"},"public_history":true,"results":[[3394,0.345,15,5476,-1],[2959,0.341,1,5477,-1]]},{"id":"medimageosworld-2026-paper-local-models","name":"MedImageOSWorld (Local models) (2026-10, paper results)","group":"医疗","unit":"%","metric":"Weighted success rate, local models","protocol":{"en":"Eleven locally served models on 49 imaging-console tasks. The score weights each task by capability level. The study records the actual assets and seeds run for each condition.","zh":"11 个本地模型在 49 项影像设备操作任务上测试，按任务的能力等级加权。各组实际使用的素材和随机种子见论文记录。"},"source":{"name":"MedImageOSWorld authors","url":"https://arxiv.org/abs/2610.04800","status":"paper_baselines","published_at":"2026-10-03"},"public_history":true,"results":[[5300,0.25,195,5478,-1],[5479,0.20800000000000002,66,5478,-1],[5480,0.146,83,5478,-1],[5481,0.107,370,5478,-1],[5482,0.077,195,5478,-1],[5483,0.065,370,5478,-1],[5484,0.055,370,5478,-1],[2444,0.052000000000000005,195,5478,-1],[5485,0.037000000000000005,195,5478,-1],[5486,0.033,370,5478,-1],[5487,0.03,370,5478,-1]]},{"id":"holdembench-publisher","name":"HoldemBench","group":"预测与决策","unit":"raw","metric":"All-in-adjusted big blinds per hand","protocol":{"en":"Duplicate heads-up no-limit Hold’em. The author estimates each model’s return against the field over 7,729 hands.","zh":"双人无限注德州扑克，每副牌交换座位重打。作者根据全场 7,729 手估算每个模型每手赢得的大盲注数。"},"source":{"name":"HoldemBench authors","url":"https://github.com/ChrystianSchutz/poker-holdembench","status":"current_snapshot","published_at":"2026-10-05"},"results":[[3512,5.4,25,5488,-1],[5489,4.3,66,5488,-1],[2960,3.4,15,5490,-1],[2976,3.0,15,5488,-1],[2959,2.7,1,5490,-1],[3529,2.5,551,5488,-1],[3518,2.4,370,5488,-1],[5491,2.0,370,5492,-1],[5493,-0.8,370,5492,-1],[5494,-3.2,370,5492,-1],[5495,-4.0,370,5492,-1],[5496,-8.4,370,5497,-1]]},{"id":"tasteval-2026-paper","name":"TasteVal (Seven shared tasks) (2026-10, paper results)","group":"数学与科学","unit":"raw","metric":"Compute multiplier vs. expert baseline","protocol":{"en":"Seven shared research tasks. The authors recomputed the expert baseline and both reported model multipliers on this set.","zh":"在相同的七项研究任务上，作者重新计算了人类专家基准和这两个模型的算力倍率。"},"source":{"name":"TasteVal authors","url":"https://arxiv.org/abs/2610.06824","status":"paper_baselines","published_at":"2026-10-05"},"public_history":true,"results":[[3384,2.25,370,5009,-1],[3149,1.68,370,5009,-1]]},{"id":"tasteval-2026-paper-eight-tasks","name":"TasteVal (Eight-task cohort) (2026-10, paper results)","group":"数学与科学","unit":"raw","metric":"Compute multiplier vs. expert baseline","protocol":{"en":"Eight AI research tasks, 40 H100 GPU-hours per task attempt. Each model’s compute multiplier is relative to the per-task expert benchmark.","zh":"模型执行全部八项 AI 研究任务，每次尝试有 40 个 H100 GPU 小时。算力倍率以专家在各项任务上的成绩为参照。"},"source":{"name":"TasteVal authors","url":"https://arxiv.org/abs/2610.06824","status":"paper_baselines","published_at":"2026-10-05"},"public_history":true,"results":[[5498,2.11,370,5499,-1],[2958,1.6,15,5009,-1],[5498,1.29,370,5009,-1],[5498,0.97,370,5017,-1],[3401,0.84,370,5009,-1],[3515,0.59,551,5009,-1],[762,0.59,15,5009,-1],[5498,0.58,370,5015,-1],[3400,0.5,370,5009,-1],[3196,0.47,15,5011,-1],[2977,0.45,21,5500,-1],[3406,0.4,370,5009,-1],[340,0.28,15,5010,-1],[3184,0.21,15,5011,-1],[3427,0.12,370,5501,-1],[2058,0.12,15,5010,-1],[244,0.09,21,5502,-1],[1464,0.08,15,5010,-1],[2816,0.07,15,5010,-1],[353,0.06,15,2022,-1],[2089,0.03,15,2019,-1],[2098,0.01,15,5503,-1],[795,0.01,66,370,-1]]},{"id":"biostudybench-2026-paper","name":"BioStudyBench (2026-10, paper results)","group":"医疗","unit":"%","metric":"Pass rate with data and tools","protocol":{"en":"Three runs on each of 25 biomedical analyses with source data and tools. Timeouts and missing answers count as failures.","zh":"每个模型使用数据和工具完成 25 项生物医学分析，重复三轮。超时或没有提交答案均计作失败。"},"source":{"name":"BioStudyBench authors","url":"https://arxiv.org/abs/2610.07614","status":"paper_baselines","published_at":"2026-10-06"},"public_history":true,"results":[[2958,0.9470000000000001,15,370,-1],[2983,0.9470000000000001,83,370,-1],[2959,0.9329999999999999,1,370,-1],[2976,0.747,15,370,-1],[3515,0.813,551,370,-1],[3512,0.76,25,370,-1],[2971,0.667,370,370,-1],[2986,0.56,195,370,-1]]},{"id":"learn2play-bench-2026-paper","name":"Learn2Play Bench (2026-10, paper results)","group":"知识与推理","unit":"raw","metric":"Peak normalized game score (Max)","protocol":{"en":"OpenCode on 20 hidden-rule games. The peak score for each trial is divided by the game reference, multiplied by 100, then averaged using the paper’s game weights.","zh":"OpenCode 在 20 个隐藏规则游戏中运行。每轮峰值除以该游戏的参考分并乘以 100，再按论文的游戏权重汇总。"},"source":{"name":"Learn2Play Bench authors","url":"https://arxiv.org/abs/2610.08215","status":"paper_baselines","published_at":"2026-09-29"},"public_history":true,"results":[[2959,80.1,1,5393,-1],[2958,76.2,15,5393,-1],[755,74.5,1,5393,-1],[3560,72.5,195,5393,-1],[3415,72.0,83,5393,-1],[2977,68.5,21,5393,-1],[3386,58.3,83,5393,-1],[5504,57.0,15,5393,-1],[3568,55.9,1,5393,-1],[3492,54.2,25,5393,-1],[5505,34.0,15,5393,-1]]},{"id":"agenttime-publisher","name":"AgentTime","group":"Agent 与工具","unit":"%","metric":"Duration-following on-time rate","protocol":{"en":"AgentTime v1.1. Each of 200 tasks has a short and a long requested duration. A run is on time within 5% of the request.","zh":"AgentTime v1.1 对 200 项任务分别给出较短和较长的用时要求。实际用时与要求相差不超过 5% 即计为准时。"},"source":{"name":"AgentTime authors","url":"https://agenttimebench.com/","status":"current_snapshot","published_at":"2026-09-26"},"results":[[5506,0.645,15,5507,-1],[5508,0.605,15,5507,-1],[5004,0.3725,15,5507,-1],[2962,0.105,83,5509,-1],[2959,0.05527638190954774,1,5510,-1],[2965,0.0463768115942029,66,5511,-1],[2956,0.043037974683544304,1,5510,-1],[2971,0.03759398496240601,370,5512,-1],[5513,0.0325,370,5512,-1]]},{"id":"roboquest-2026-paper","name":"RoboQuest (2026-10, paper results)","group":"Agent 与工具","unit":"%","metric":"Overall success rate","protocol":{"en":"Ten robot manipulation tasks, with 50 episodes per task and model. A successful episode completes the full task.","zh":"10 项机器人操作任务，每个模型每项运行 50 次，按完整完成任务的比例计分。"},"source":{"name":"RoboQuest authors","url":"https://arxiv.org/abs/2610.10388","status":"paper_baselines","published_at":"2026-10-05"},"public_history":true,"results":[[2958,0.23199999999999998,15,5514,-1],[2959,0.138,1,5514,-1],[2960,0.122,15,5514,-1],[2956,0.114,1,5514,-1],[2983,0.02,83,5515,-1]]},{"id":"robotworld-2026-paper","name":"RobotWorld (2026-10, paper results)","group":"Agent 与工具","unit":"%","metric":"Overall task success rate","protocol":{"en":"One retained episode per model and task, across 84 simulated robot tasks.","zh":"每个模型在 84 项仿真机器人任务上各保留一次运行，按任务完成率计分。"},"source":{"name":"RobotWorld authors","url":"https://arxiv.org/abs/2610.10409","status":"paper_baselines","published_at":"2026-10-07"},"public_history":true,"results":[[2958,0.19,15,370,-1],[2959,0.155,1,370,-1],[2977,0.024,21,370,-1],[3512,0.012,25,370,-1],[2983,0.012,83,370,-1]]},{"id":"rt-safe-2026-paper","name":"RT-Safe (2026-10, paper results)","group":"安全与可靠性","unit":"%","metric":"Safe real-time navigation success rate","protocol":{"en":"108 real-time navigation episodes per model at provider-default reasoning. Easy, medium and hard conditions receive equal weight.","zh":"每个模型在供应商默认推理档位下运行 108 次实时导航，三个难度权重相同，统计安全完成导航的比例。"},"source":{"name":"RT-Safe authors","url":"https://arxiv.org/abs/2610.09294","status":"paper_baselines","published_at":"2026-10-07"},"public_history":true,"results":[[2958,0.0,15,5516,-1],[762,0.019,15,5517,-1],[2956,0.0,1,5518,-1],[3568,0.037000000000000005,1,5518,-1],[5519,0.0,83,5520,-1],[5521,0.0,25,5522,-1],[203,0.0,370,5522,-1],[5295,0.0,143,5522,-1]]},{"id":"storebench-2026-paper","name":"StoreBench (2026-10, paper results)","group":"Agent 与工具","unit":"score","metric":"Mean store-operation composite","protocol":{"en":"Goose harness. Eleven store-management tasks, three seeds and three attempts per seed. The score combines business performance, reliability and continuity.","zh":"使用 Goose 执行 11 项门店经营任务，每项用三个随机种子，每个种子运行三次。综合分包括经营业绩、可靠性和业务连续性。"},"source":{"name":"StoreBench authors","url":"https://arxiv.org/abs/2610.10942","status":"paper_baselines","published_at":"2026-10-07"},"score_max":1,"public_history":true,"score_decimals":3,"results":[[3185,0.7,25,5523,-1],[3479,0.668,1,5524,-1],[3453,0.629,1,5524,-1],[3157,0.587,195,5523,-1],[2983,0.552,83,5524,-1],[762,0.538,15,5524,-1],[3407,0.387,66,5523,-1]]},{"id":"datasense-bench-2026-paper","name":"DataSense-Bench (2026-10, paper results)","group":"Agent 与工具","unit":"score","metric":"TBLite post-training reward (0–100)","protocol":{"en":"Each agent ranks groups of 1,000 training trajectories. The score is the downstream model’s mean TBLite reward after training on the selector’s top-choice group.","zh":"各模型对每组 1,000 条训练轨迹排序。取其首选组训练下游模型，再按下游模型在 TBLite 上的平均奖励计分。"},"source":{"name":"DataSense-Bench authors","url":"https://arxiv.org/abs/2610.12190","status":"paper_baselines","published_at":"2026-10-08"},"score_max":100,"public_history":true,"results":[[3492,12.54,25,5525,-1],[3149,12.5,370,5525,-1],[3415,12.07,83,5525,-1],[2958,11.95,15,5525,-1],[2977,11.94,21,5525,-1],[3196,11.87,15,5525,-1],[3156,11.6,370,5525,-1],[3393,11.55,370,5525,-1],[762,11.26,15,5525,-1],[3151,11.0,370,5525,-1]]},{"id":"mine-odyssey-2026-paper","name":"Mine Odyssey (2026-10, paper results)","group":"Agent 与工具","unit":"%","metric":"Overall task success rate","protocol":{"en":"The published Minecraft agents complete 180 tasks, including 125 outdoors and 55 indoors.","zh":"各模型在 Minecraft 中完成 180 项任务，其中 125 项在室外，55 项在室内。"},"source":{"name":"Mine Odyssey authors","url":"https://arxiv.org/abs/2610.11328","status":"paper_baselines","published_at":"2026-10-07"},"public_history":true,"results":[[2958,0.8556,15,5017,-1],[2959,0.7389,1,5009,-1],[755,0.5056,1,5009,-1],[755,0.44439999999999996,1,5010,-1],[755,0.3944,1,5017,-1],[755,0.35,1,5015,-1],[762,0.4389,15,5009,-1],[2983,0.2667,83,5010,-1],[2974,0.1722,143,5011,-1],[3153,0.2389,25,5009,-1],[3396,0.15560000000000002,551,5009,-1]]},{"id":"safe-teen-vals-publisher","name":"SAFE-Teen (Vals)","group":"安全与可靠性","unit":"%","metric":"Urgent safety failure rate","protocol":{"en":"Failure rate in the urgent safety track of the publisher’s model comparison. Lower is better.","zh":"发布方模型比较中的紧急安全场景失败率，越低越好。"},"source":{"name":"Vals AI","url":"https://www.vals.ai/benchmarks/mental-health","status":"current_snapshot","published_at":"2026-10-08"},"higher_is_better":false,"results":[[2960,0.15,15,370,-1],[3408,0.23,15,370,-1],[2961,0.23,1,370,-1],[5526,0.26,66,370,-1],[3560,0.26,195,370,-1],[2959,0.27,1,370,-1],[3512,0.31,25,370,-1],[5527,0.33,417,370,-1],[5528,0.33,66,370,-1],[5287,0.4,370,370,-1],[2977,0.44,21,370,-1],[2968,0.44,143,370,-1],[3391,0.5,143,370,-1],[3515,0.51,551,370,-1],[3405,0.52,83,370,-1],[2962,0.53,83,370,-1],[3491,0.54,775,370,-1],[2983,0.56,83,370,-1]]},{"id":"drylab-bench-2026-paper","name":"DRYLAB-Bench (2026-10, paper results)","group":"安全与可靠性","unit":"raw","metric":"Mean biological risk score, refusal floor","protocol":{"en":"S2 adaptive design, up to four rounds. Each scenario receives equal weight and refusals receive the assay-specific floor. A lower score indicates less biological risk.","zh":"S2 自适应设计最多进行四轮。各情景等权，拒答按对应实验的风险下限计分。分数越低，设计的生物风险越低。"},"source":{"name":"DRYLAB-Bench authors","url":"https://arxiv.org/abs/2610.13116","status":"paper_baselines","published_at":"2026-10-08"},"higher_is_better":false,"public_history":true,"results":[[3155,0.637,21,370,-1],[3154,0.497,551,370,-1],[3492,0.483,25,370,-1],[3391,0.478,143,370,-1],[2982,0.445,195,370,-1],[3261,0.416,25,370,-1],[3405,0.414,83,370,-1],[2983,0.356,83,370,-1],[3408,0.3,15,370,-1],[762,0.293,15,370,-1],[2973,0.159,15,370,-1],[2958,0.101,15,370,-1],[3479,0.086,1,370,-1],[755,0.072,1,370,-1],[3453,0.025,1,370,-1]]},{"id":"mathvis-2026-paper","name":"MathVis (2026-10, paper results)","group":"数学与科学","unit":"%","metric":"Strict pass@1","protocol":{"en":"One figure-generation attempt on each of 512 mathematics questions, scored by the benchmark’s strict verification rules.","zh":"512 道数学作图题，每题取第一次作答，按基准的严格核验规则判定是否通过。"},"source":{"name":"MathVis authors","url":"https://arxiv.org/abs/2610.12954","status":"paper_baselines","published_at":"2026-10-09"},"public_history":true,"results":[[4977,0.3125,15,370,-1],[3453,0.2656,1,370,-1],[3482,0.2285,83,370,-1],[4048,0.2148,15,370,-1],[3479,0.209,1,370,-1],[5361,0.209,370,370,-1],[3654,0.1895,551,5529,-1],[4225,0.1816,370,370,-1],[3654,0.17579999999999998,551,5530,-1],[3572,0.168,1,370,-1],[3570,0.17190000000000003,25,370,-1],[5531,0.16210000000000002,370,370,-1],[3415,0.1602,83,370,-1],[5532,0.1582,370,370,-1],[5533,0.1133,370,370,-1],[325,0.1035,1,370,-1]]},{"id":"physimcode-2026-paper","name":"PhySimCode (2026-10, paper results)","group":"数学与科学","unit":"%","metric":"Parameter estimation accuracy within ±20%","protocol":{"en":"The 10-example hard subset contains 77 ground-truth physical parameters. A prediction passes when it is within 20% of its reference.","zh":"困难子集有 10 个样例、77 个物理参数。预测值与参考值的偏差在 20% 以内即算正确。"},"source":{"name":"PhySimCode authors","url":"https://arxiv.org/abs/2610.15722","status":"paper_baselines","published_at":"2026-10-08"},"public_history":true,"results":[[2956,0.1818,1,370,-1],[2958,0.0519,15,370,-1],[2974,0.0909,143,370,-1],[2977,0.12990000000000002,21,370,-1],[5485,0.039,195,370,-1],[5534,0.039,370,370,-1],[5535,0.039,83,370,-1],[5536,0.039,551,370,-1],[4151,0.039,370,370,-1],[5505,0.039,15,370,-1],[2602,0.0519,83,370,-1],[3572,0.0519,1,370,-1],[4042,0.0649,143,370,-1],[5537,0.026000000000000002,370,370,-1]]},{"id":"rob-2-ai-benchmark-publisher","name":"RoB 2 AI Benchmark","group":"医疗","unit":"%","metric":"Overall risk-of-bias agreement","protocol":{"en":"ROBoto2 manual, 185 papers. Each model receives the full paper and the same 22 signalling questions. Compare its overall risk-of-bias classification with the independent manual assessment.","zh":"ROBoto2 人工评估组共 185 篇论文。模型阅读全文并回答相同的 22 个偏倚风险问题，按总体判断与独立人工评估一致的比例计分。"},"source":{"name":"RoB 2 AI Benchmark authors","url":"https://github.com/CoreSR/rob2-ai-benchmark","status":"current_snapshot","published_at":"2026-10-12"},"results":[[2956,0.48100000000000004,1,5538,-1],[2958,0.368,15,5538,-1],[3492,0.4,25,5538,-1],[2977,0.373,21,5538,-1]]},{"id":"aa-omniscience-decision-publisher","name":"AA-Omniscience-Decision","group":"知识与推理","unit":"%","metric":"Accuracy","protocol":{"en":"Artificial Analysis decision evaluation, preserving the reported model and inference configuration.","zh":"按 Artificial Analysis 决策评测的原始模型及推理配置计分。"},"source":{"name":"Artificial Analysis","url":"https://artificialanalysis.ai/evaluations/artificial-analysis-decision-index","status":"current_snapshot","published_at":"2026-10-13"},"results":[[3123,0.5760000000000001,1,370,-1],[212,0.5720000000000001,1,370,-1],[292,0.524,1,370,-1],[248,0.586,1,370,-1],[5539,0.586,1,370,-1],[5331,0.934,1,370,-1],[5540,0.934,1,370,-1],[5541,0.11599999999999999,5542,5543,-1],[5544,0.37799999999999995,5545,5543,-1],[5546,0.39,5547,5543,-1],[5548,0.326,5547,5543,-1],[5549,0.36,5547,5543,-1],[5550,0.47600000000000003,83,370,-1],[5551,0.364,5552,5543,-1],[5553,0.276,5552,5543,-1],[5494,0.35,3399,5543,-1],[5554,0.41,5555,5543,-1],[5556,0.316,5555,5543,-1],[5557,0.276,5558,5543,-1],[5559,0.376,5558,5543,-1],[5560,0.27399999999999997,5558,5543,-1],[72,0.91,15,370,-1],[120,0.93,15,370,-1],[167,0.736,15,370,-1],[5561,0.406,15,5543,-1],[211,0.696,15,370,-1],[273,0.66,15,370,-1],[232,0.7120000000000001,15,370,-1],[5562,0.45399999999999996,15,370,-1],[5563,0.282,554,5543,-1],[5564,0.35200000000000004,5565,5543,-1],[5566,0.39799999999999996,5565,5543,-1],[5567,0.39399999999999996,5565,5543,-1],[5568,0.20199999999999999,5569,5543,-1],[5491,0.34600000000000003,5570,5543,-1],[5571,0.33799999999999997,5570,5543,-1]]},{"id":"artificial-analysis-decision-index-publisher","name":"Artificial Analysis Decision Index","group":"预测与决策","unit":"score","metric":"Decision Index","protocol":{"en":"Artificial Analysis decision evaluation, preserving the reported model and inference configuration.","zh":"按 Artificial Analysis 决策评测的原始模型及推理配置计分。"},"source":{"name":"Artificial Analysis","url":"https://artificialanalysis.ai/evaluations/artificial-analysis-decision-index","status":"current_snapshot","published_at":"2026-10-13"},"results":[[3123,68.36475873224768,1,370,-1],[212,62.55462486332752,1,370,-1],[292,60.58568626557774,1,370,-1],[248,61.39217154925252,1,370,-1],[5539,64.4193030242969,1,370,-1],[5331,79.0075318761743,1,370,-1],[5540,75.42214106805352,1,370,-1],[5541,64.6916698767591,5542,5543,-1],[5544,52.292281662967724,5545,5543,-1],[5546,48.61266368225613,5547,5543,-1],[5548,45.80872726695699,5547,5543,-1],[5549,44.25935869244778,5547,5543,-1],[5550,55.52364035466043,83,370,-1],[5551,53.90622624105178,5552,5543,-1],[5553,65.01239362403462,5552,5543,-1],[5494,52.72371648704283,3399,5543,-1],[5554,50.292464197464454,5555,5543,-1],[5556,65.24809097459318,5555,5543,-1],[5557,63.73590557818611,5558,5543,-1],[5559,50.72643132996479,5558,5543,-1],[5560,57.88898444363282,5558,5543,-1],[72,70.91112171604452,15,370,-1],[120,71.46536806721173,15,370,-1],[167,60.77068937873605,15,370,-1],[5561,48.48606776391437,15,5543,-1],[211,59.08458632172907,15,370,-1],[273,55.18427890385554,15,370,-1],[232,58.15418508167304,15,370,-1],[5562,49.5923400266625,15,370,-1],[5563,53.75552448107142,554,5543,-1],[5564,50.153108044043115,5565,5543,-1],[5566,53.06349517938014,5565,5543,-1],[5567,56.17404470463419,5565,5543,-1],[5568,64.99541059577241,5569,5543,-1],[5491,57.503981359365326,5570,5543,-1],[5571,57.073921605209755,5570,5543,-1]]},{"id":"ma-bench-2026-paper","name":"MA-Bench (2026-10, paper results)","group":"专业工作","unit":"score","metric":"Severity-weighted mission brief score","protocol":{"en":"Ten mission-analysis scenarios, medium reasoning, 3,000-word limit. The production judge scores requirements and subtracts penalties for wrong actions.","zh":"10 个任务分析情景，推理强度中档，简报限 3,000 词。主评审按要求完成情况给分，并对错误行动扣分。"},"source":{"name":"MA-Bench authors","url":"https://arxiv.org/abs/2610.19138","status":"paper_baselines","published_at":"2026-10-13"},"score_max":1,"public_history":true,"score_decimals":3,"results":[[2979,0.574,66,5572,-1],[2974,0.54,143,5572,-1],[755,0.514,1,5572,-1],[3154,0.505,551,5572,-1],[2958,0.485,15,5573,-1],[2977,0.481,21,5572,-1],[762,0.46,15,5572,-1],[3408,0.452,15,5572,-1],[3418,0.434,370,5572,-1],[3491,0.43,775,5572,-1],[2956,0.428,1,5573,-1],[3157,0.41,195,5572,-1],[3568,0.393,1,5572,-1],[2983,0.371,83,5572,-1],[3184,0.37,15,5572,-1],[3185,0.359,25,5572,-1],[3412,0.34,83,5573,-1],[3613,0.338,417,5572,-1],[3201,0.307,25,5572,-1],[2585,0.302,83,5572,-1]]},{"id":"nepsee-a-double-annotated-benchmark-for-nepali-sentiment-analysis-2026-paper","name":"NEPSEE: A Double-Annotated Benchmark for Nepali Sentiment Analysis (Zero-shot LLMs) (2026-10, paper results)","group":"指令与长文","unit":"score","metric":"Three-class sentiment macro-F1","protocol":{"en":"Zero-shot sentiment classification on the same 479 Nepali sentences. Macro-F1 gives the three sentiment classes equal weight.","zh":"五个模型在同一组 479 句尼泊尔语文本上做零样本情感分类。宏平均 F1 对三个情感类别等权。"},"source":{"name":"NEPSEE: A Double-Annotated Benchmark for Nepali Sentiment Analysis authors","url":"https://arxiv.org/abs/2610.19190","status":"paper_baselines","published_at":"2026-10-13"},"score_max":1,"public_history":true,"score_decimals":3,"results":[[2959,0.896,1,5574,-1],[2983,0.891,83,5574,-1],[3412,0.882,83,5574,-1],[2576,0.867,195,5574,-1],[4070,0.79,195,5574,-1]]},{"id":"nepsee-a-double-annotated-benchmark-for-nepali-sentiment-analysis-2026-paper-fine-tuned-encoders","name":"NEPSEE: A Double-Annotated Benchmark for Nepali Sentiment Analysis (Fine-tuned encoders) (2026-10, paper results)","group":"指令与长文","unit":"score","metric":"Three-class sentiment macro-F1","protocol":{"en":"Fine-tuned Nepali and multilingual encoders on the 479-sentence test set. Macro-F1 averages the three classes, and each reported score averages three training seeds.","zh":"尼泊尔语与多语种编码器经微调后，在 479 句测试文本上分类。宏平均 F1 对三个类别等权，每项成绩取三个训练种子的平均值。"},"source":{"name":"NEPSEE: A Double-Annotated Benchmark for Nepali Sentiment Analysis authors","url":"https://arxiv.org/abs/2610.19190","status":"paper_baselines","published_at":"2026-10-13"},"score_max":1,"public_history":true,"score_decimals":3,"results":[[5575,0.877,370,5576,-1],[5577,0.867,370,5578,-1],[5579,0.858,370,5578,-1],[5580,0.847,370,5578,-1],[5581,0.84,370,5578,-1],[5582,0.831,370,5578,-1],[5583,0.817,370,5584,-1],[5575,0.793,370,5584,-1],[5585,0.788,370,5576,-1],[5586,0.771,370,5587,-1],[5585,0.739,370,5584,-1],[5583,0.721,370,5576,-1]]},{"id":"robo-bench-publisher","name":"Robo-bench","group":"Agent 与工具","unit":"%","metric":"Overall task performance","protocol":{"en":"Control tasks in real-time robot simulations.","zh":"实时物理模拟中的机器人控制任务。"},"source":{"name":"Vals AI","url":"https://www.vals.ai/benchmarks/robo_bench","status":"current_snapshot","published_at":"2026-10-13"},"saturation_excluded":true,"scale_label":{"en":"Rubric score","zh":"量表得分"},"results":[[5182,0.28332999999999997,15,5305,-1],[5027,0.23332999999999998,15,5305,-1],[5023,0.14583000000000002,1,5306,-1],[5171,0.10417,66,5308,-1],[5025,0.05833,1,5306,-1],[5169,0.008329999999999999,83,5309,-1],[5038,0.00208,143,5588,-1]]},{"id":"j-tau-2026-paper","name":"J-tau v0.2 (airline and retail expansion) (2026-10, paper results)","group":"Agent 与工具","unit":"%","metric":"Airline task success rate","protocol":{"en":"Japanese-language airline tasks in J-tau. Each model runs each task three times; the table reports mean success.","zh":"在 J-tau 的日语航空客服任务上，每个模型每题运行三次，取平均成功率。"},"source":{"name":"J-tau authors","url":"https://arxiv.org/abs/2610.19638","status":"paper_baselines","published_at":"2026-08-13"},"public_history":true,"results":[[762,0.847,15,370,-1],[3412,0.84,83,370,-1],[3479,0.813,1,370,-1],[3453,0.907,1,370,-1],[3155,0.7,21,370,-1],[5589,0.893,551,370,-1],[62,0.733,15,370,-1],[4906,0.507,15,370,-1],[5590,0.833,195,370,-1],[2584,0.847,195,370,-1],[4070,0.493,195,370,-1],[4969,0.24,195,370,-1],[2954,0.513,83,370,-1],[5591,0.413,83,370,-1],[5592,0.387,83,370,-1],[5593,0.207,83,370,-1],[5594,0.647,370,370,-1],[5595,0.56,370,370,-1]]},{"id":"veld-qa-2026-paper","name":"VELD-QA (2026-10, paper results)","group":"指令与长文","unit":"%","metric":"Accuracy with gold evidence pages","protocol":{"en":"All models receive the human-annotated evidence pages for each test question.","zh":"对测试集中的每道问题，所有模型均可查看人工标注的证据页。"},"source":{"name":"VELD-QA authors","url":"https://arxiv.org/abs/2610.20964","status":"paper_baselines","published_at":"2026-10-14"},"public_history":true,"results":[[5596,0.391,83,370,-1],[5597,0.7390000000000001,66,370,-1],[2055,0.705,83,370,-1],[2602,0.74,83,370,-1],[3741,0.6459999999999999,143,370,-1],[3196,0.76,15,370,-1],[2958,0.802,15,370,-1],[2961,0.812,1,370,-1],[2959,0.8170000000000001,1,370,-1]]},{"id":"fde-gym-2026-paper","name":"FDE-Gym (2026-10, paper results)","group":"专业工作","unit":"raw","metric":"L3 delivery score","protocol":{"en":"All 95 tasks at L3, starting from the sponsor’s original request. The score combines business improvement, incidents and stakeholder contact costs relative to the project baseline.","zh":"全部 95 项任务均从客户最初的要求出发（L3）。相对项目基线评估业务收益，并计入事故和联系相关人员的成本。"},"source":{"name":"FDE-Gym authors","url":"https://arxiv.org/abs/2610.22350","status":"paper_baselines","published_at":"2026-10-06"},"public_history":true,"results":[[2959,0.56,1,370,-1],[2961,0.49,1,370,-1],[2958,0.38,15,370,-1],[2960,0.37,15,370,-1],[2980,0.22,551,370,-1],[3512,0.21,25,370,-1],[2983,0.19,83,370,-1],[5598,0.07,195,370,-1],[3196,0.03,15,370,-1],[2976,-0.12,15,370,-1],[5599,-0.32,417,370,-1],[325,-0.33,1,370,-1],[325,-0.35,1,5600,-1]]},{"id":"preference-integrity-benchmark-pib-v1-0-publisher","name":"Preference Integrity Benchmark (PIB) v1.0","group":"安全与可靠性","unit":"score","metric":"Preference integrity index","protocol":{"en":"Six preference questions across controlled presentations and conversations. The index averages presentation invariance, conformity resistance, pushback resistance, context stability and choice coherence.","zh":"六个偏好问题在不同展示顺序和对话条件下重复提问。指数取展示不变性、抵抗迎合、抵抗质疑、上下文稳定性和选择一致性的平均值。"},"source":{"name":"Preference Integrity Benchmark (PIB) v1.0 authors","url":"https://huggingface.co/datasets/PAOP123/preference-integrity-bench","status":"current_snapshot","published_at":"2026-10-15"},"score_max":1,"score_decimals":6,"results":[[2959,0.8772244883410967,1,4947,-1],[2960,0.8736150419167661,15,4946,-1],[2958,0.8715877719598011,15,4943,-1],[2961,0.8504668747124471,1,4949,-1],[2585,0.7826059376619453,83,5601,-1],[4906,0.7625681424272527,15,4906,-1],[5602,0.7589330194608181,370,5603,-1],[2970,0.7561958624708625,1,5604,-1],[5320,0.754149435219729,66,5605,-1],[3154,0.7479692581927877,551,4954,-1],[2983,0.7429422763230209,83,5606,-1],[203,0.7420447985774908,370,5607,-1],[5608,0.7395223608716865,370,5609,-1],[3534,0.7380658465620854,370,5610,-1],[2976,0.7379249103460641,15,5150,-1],[2977,0.7378094205402441,21,4952,-1],[3418,0.7259237541775626,370,5611,-1],[5289,0.6658237462955149,370,5612,-1],[5293,0.6640845540848389,370,5613,-1]]},{"id":"premisesql-2026-paper","name":"PremiseSQL (2026-10, paper results)","group":"编程","unit":"%","metric":"Spontaneous premise flaw detection (DET)","protocol":{"en":"Base prompt, 1,800 flawed text-to-SQL requests. DET is the mean binary judgment that the model identified the flaw.","zh":"基础提示下共 1,800 个含前提错误的文本转 SQL 请求。DET 统计模型识别出错误的比例。"},"source":{"name":"PremiseSQL authors","url":"https://arxiv.org/abs/2610.22158","status":"paper_baselines","published_at":"2026-10-15"},"public_history":true,"results":[[5614,0.228,551,370,-1],[2261,0.281,1,370,-1],[2959,0.492,1,370,-1],[3572,0.263,1,370,-1],[4425,0.422,25,370,-1],[3512,0.247,25,370,-1],[4225,0.305,370,370,-1],[3482,0.272,83,370,-1],[5615,0.147,66,370,-1],[3184,0.261,15,370,-1],[5616,0.312,15,370,-1],[5617,0.386,15,370,-1],[5618,0.417,15,370,-1],[5619,0.399,195,370,-1],[5620,0.351,195,370,-1],[3191,0.315,195,370,-1]]},{"id":"system1-arena-2026-paper","name":"System1-Arena (2026-10, paper results)","group":"预测与决策","unit":"%","metric":"Non-agentic overall accuracy","protocol":{"en":"The static decision benchmark covers 13 domains. Accuracy averages across tasks, with unanswered questions scored as wrong.","zh":"静态决策评测涵盖 13 个领域，正确率按任务取平均，未作答的问题计错。"},"source":{"name":"System1-Arena authors","url":"https://arxiv.org/abs/2610.21244","status":"paper_baselines","published_at":"2026-10-15"},"public_history":true,"results":[[2959,0.8114,1,5621,-1],[2958,0.8007,15,5622,-1],[2968,0.775,143,5623,-1],[3394,0.7739,15,5624,-1],[762,0.7667,15,5625,-1],[5480,0.7391,83,5626,-1],[5627,0.7303,370,5628,-1],[5491,0.7257,370,5629,-1],[62,0.7178,15,5630,-1],[2976,0.7116,15,5631,-1],[5563,0.7089,370,5632,-1],[5300,0.7016,195,5633,-1],[3408,0.6998,15,5634,-1],[5635,0.6989,83,5636,-1],[5637,0.6965,370,5638,-1],[4906,0.6869,15,5639,-1],[5640,0.6861,370,5641,-1],[5642,0.6705,370,5643,-1],[4070,0.6532,195,5644,-1],[5645,0.6514,370,5646,-1],[786,0.6496,195,5647,-1],[5648,0.6481,66,5649,-1],[5650,0.6282,83,5651,-1],[5344,0.5925,195,5652,-1],[587,0.5314,66,5653,-1],[5654,0.5029,370,5655,-1],[5656,0.4447,370,5657,-1],[5658,0.4143,370,5659,-1],[5660,0.3924,195,5661,-1]]},{"id":"toolbottleneckbench-2026-paper","name":"ToolBottleneckBench (2026-10, paper results)","group":"Agent 与工具","unit":"%","metric":"Baseline verified task success (VTS)","protocol":{"en":"All 21 model configurations on 36 task families. The baseline pools the raw and linked tool menus with eight visible schemas and a low working-set demand.","zh":"21 个模型配置均测试 36 类任务。基线汇总原始和链接两种工具菜单，均展示八个工具描述，并保持较低的工作集要求。"},"source":{"name":"ToolBottleneckBench authors","url":"https://arxiv.org/abs/2610.21098","status":"paper_baselines","published_at":"2026-10-13"},"public_history":true,"results":[[2970,0.9440000000000001,1,5662,-1],[755,0.9440000000000001,1,5662,-1],[2959,0.972,1,5662,-1],[3568,0.9309999999999999,1,5662,-1],[2961,0.9440000000000001,1,5662,-1],[3408,0.958,15,5662,-1],[762,1.0,15,5662,-1],[2973,0.972,15,5662,-1],[2976,0.9309999999999999,15,5662,-1],[2960,1.0,15,5662,-1],[3405,0.9309999999999999,83,5662,-1],[2983,0.875,83,5662,-1],[5663,0.889,195,5662,-1],[3260,0.667,195,5662,-1],[5664,0.736,195,5662,-1],[3570,0.9309999999999999,25,5662,-1],[3539,0.9440000000000001,25,5662,-1],[3512,0.6940000000000001,25,5662,-1],[5665,0.278,3033,5662,-1],[3154,0.9309999999999999,48,5662,-1],[2980,0.958,48,5662,-1]]},{"id":"personacuabench-2026-paper","name":"PersonaCUABench (2026-10, paper results)","group":"Agent 与工具","unit":"%","metric":"Task requirements met, expert personas","protocol":{"en":"Expert-persona requests across all 74 tasks. Completion measures the share of the task’s independently checked requirements that the agent fulfilled.","zh":"在全部 74 项任务中，由专家人设发起请求。完成度按模型满足的经核验任务要求所占比例计算。"},"source":{"name":"PersonaCUABench authors","url":"https://arxiv.org/abs/2610.22990","status":"paper_baselines","published_at":"2026-10-16"},"public_history":true,"results":[[3491,0.447,775,5666,-1],[2977,0.555,21,5666,-1],[5320,0.413,66,5666,-1],[2974,0.543,143,5666,-1],[3403,0.554,195,5666,-1],[2983,0.476,83,5666,-1],[2958,0.504,15,5666,-1],[3512,0.485,25,5666,-1],[2959,0.55,1,5666,-1]]},{"id":"vals-science-index-publisher","name":"Vals Science Index","group":"数学与科学","unit":"%","metric":"Science index","protocol":{"en":"Equal weighting across the component scientific research tasks.","zh":"将组成该指数的科学研究任务等权汇总。"},"source":{"name":"Vals AI","url":"https://www.vals.ai/benchmarks/science_index","status":"current_snapshot","published_at":"2026-10-16"},"saturation_excluded":true,"scale_label":{"en":"Index","zh":"指数"},"results":[[5027,0.6509,15,5028,-1],[5182,0.59628,15,5028,-1],[5024,0.59118,1,370,-1],[5023,0.5865100000000001,1,370,-1],[5169,0.55359,83,5034,-1],[5171,0.48536999999999997,66,5028,-1],[5026,0.4793,1,370,-1],[5032,0.44998,15,5028,-1],[5181,0.4437,21,5028,-1],[5031,0.41481,15,5028,-1],[5172,0.39124000000000003,66,5028,-1],[5175,0.38502000000000003,1,370,-1],[5045,0.38094,143,5034,-1],[5033,0.3576,83,5034,-1],[5038,0.34828000000000003,143,5037,-1],[5176,0.31758,370,370,-1],[5042,0.31722,370,370,-1],[5047,0.2965,25,5034,-1],[5179,0.29577000000000003,370,370,-1],[5046,0.28379,15,5028,-1],[5069,0.27383,83,5034,-1],[5039,0.25298,15,5028,-1],[5186,0.22030999999999998,370,370,-1]]},{"id":"hermes3000-writing-benchmark-publisher","name":"hermes3000 Writing Benchmark (Preliminary R2)","group":"生成与创作","unit":"elo","metric":"German literary prose rating","protocol":{"en":"Preliminary R2, all 14 entrants. Blind literary-prose comparisons yield Bradley–Terry ratings on an Elo scale centered at 1,000.","zh":"第二轮初赛共 14 个模型。对德语文学散文进行匿名比较，用 Bradley–Terry 方法计算以 1,000 为中心的 Elo 量表评分。"},"source":{"name":"hermes3000 Writing Benchmark authors","url":"https://github.com/HybridAIOne/hermes3000-writing-benchmark","status":"current_snapshot","published_at":"2026-10-17"},"results":[[2959,1710.0,1,5667,-1],[2960,1706.0,15,5667,-1],[2961,1574.0,1,5667,-1],[2976,1128.0,15,5667,-1],[2962,1067.0,83,5667,-1],[2970,1011.0,1,5667,-1],[2977,965.0,21,5667,-1],[2983,927.0,83,5668,-1],[3512,895.0,25,5667,-1],[5449,836.0,15,5667,-1],[3154,778.0,551,5667,-1],[3493,668.0,143,5667,-1],[2585,435.0,83,5667,-1],[3403,298.0,195,5667,-1]]},{"id":"hermes3000-writing-benchmark-publisher-final","name":"hermes3000 Writing Benchmark (Final round)","group":"生成与创作","unit":"elo","metric":"German literary prose rating","protocol":{"en":"Seven finalists, three samples per task and model. Blind judgments of German literary prose are converted to Bradley–Terry ratings on a separately fitted Elo scale centered at 1,000.","zh":"七个决赛模型每项任务各生成三个样本。评审匿名比较德语文学散文，再为决赛单独拟合以 1,000 为中心的 Elo 量表评分。"},"source":{"name":"hermes3000 Writing Benchmark authors","url":"https://github.com/HybridAIOne/hermes3000-writing-benchmark","status":"current_snapshot","published_at":"2026-10-17"},"results":[[2960,1382.0,15,5669,-1],[2959,1325.0,1,5669,-1],[2961,1105.0,1,5669,-1],[2976,935.0,15,5669,-1],[2962,799.0,83,5669,-1],[2977,733.0,21,5669,-1],[2970,722.0,1,5669,-1]]},{"id":"code-migration-v1-1-publisher","name":"Code Migration v1.1","group":"编程","unit":"%","metric":"Overall code migration score","protocol":{"en":"Reimplement working programs in a different language, using the v1.1 tests.","zh":"用另一门编程语言重写可运行的程序，按 v1.1 测试计分。"},"source":{"name":"Vals AI","url":"https://www.vals.ai/benchmarks/code-migration-v1-1","status":"current_snapshot","published_at":"2026-10-19"},"saturation_excluded":true,"scale_label":{"en":"Rubric score","zh":"量表得分"},"results":[[5024,0.86875,1,370,-1],[5023,0.81012,1,370,-1],[5027,0.8057899999999999,15,5028,-1],[5182,0.7732399999999999,15,5028,-1],[5172,0.72355,66,5028,-1],[5169,0.70796,83,5034,-1],[5170,0.6752400000000001,25,5028,-1],[5171,0.6279,66,5028,-1],[5047,0.58936,25,5028,-1],[5168,0.57187,195,370,-1],[5051,0.55072,551,5028,-1],[5038,0.53217,143,5037,-1],[5175,0.5236,1,370,-1],[5059,0.51987,25,5028,-1],[5064,0.51647,551,5028,-1],[5043,0.48786,370,370,-1],[5179,0.47003,370,370,-1],[5046,0.46669,15,5028,-1],[5053,0.45364,370,370,-1],[5042,0.39552,370,370,-1],[5183,0.39264000000000004,370,5034,-1],[5033,0.38646,83,5034,-1],[5048,0.28587,21,5028,-1],[5180,0.26893,66,5037,-1],[5186,0.22846,370,370,-1],[5068,0.20898,775,370,-1],[5185,0.16444,370,5034,-1],[5080,0.10425000000000001,370,5076,-1],[5075,0.08821,370,5076,-1]]},{"id":"excel-modeling-benchmark-v1-1-publisher","name":"Excel Modeling Benchmark v1.1","group":"专业工作","unit":"%","metric":"Overall rubric score","protocol":{"en":"Excel financial modeling, including the corrected template-scoring rules in v1.1.","zh":"按 v1.1 修正后的模板评分规则评估 Excel 财务建模。"},"source":{"name":"Vals AI","url":"https://www.vals.ai/benchmarks/emb-v1-1","status":"current_snapshot","published_at":"2026-10-19"},"saturation_excluded":true,"scale_label":{"en":"Rubric score","zh":"量表得分"},"results":[[5025,0.76728,1,370,-1],[5167,0.76468,83,5034,-1],[5024,0.75899,1,370,-1],[5023,0.75836,1,370,-1],[5171,0.7499800000000001,66,5028,-1],[5169,0.7482800000000001,83,5034,-1],[5184,0.7333,15,5028,-1],[5033,0.72871,83,5034,-1],[5027,0.7268300000000001,15,5028,-1],[5172,0.72503,66,5028,-1],[5168,0.71979,195,370,-1],[5038,0.7149800000000001,143,5037,-1],[5048,0.7126399999999999,21,5028,-1],[5182,0.71088,15,5028,-1],[5181,0.7,21,5028,-1],[5046,0.6995399999999999,15,5028,-1],[5179,0.69552,370,370,-1],[5170,0.6917300000000001,25,5028,-1],[5051,0.69112,551,5028,-1],[5178,0.6897199999999999,417,5034,-1],[5053,0.68946,370,370,-1],[5042,0.6888200000000001,370,370,-1],[5177,0.6867300000000001,417,5034,-1],[5174,0.6784300000000001,775,370,-1],[5047,0.67758,25,5028,-1],[5043,0.6734699999999999,370,370,-1],[5064,0.66252,551,5028,-1],[5173,0.6617799999999999,370,370,-1],[5060,0.65986,195,370,-1],[5175,0.6547700000000001,1,370,-1],[5059,0.65128,25,5028,-1],[5186,0.6473300000000001,370,370,-1],[5180,0.61983,66,5037,-1],[5068,0.59754,775,370,-1],[5183,0.5788,370,5034,-1],[5185,0.52768,370,5034,-1],[5075,0.40970999999999996,370,5076,-1],[5080,0.40823,370,5076,-1]]},{"id":"finance-agent-v2-1-publisher","name":"Finance Agent v2.1","group":"专业工作","unit":"%","metric":"Overall score","protocol":{"en":"Financial analyst tasks, scored with the FAB v2 harness on the v2.1 dataset.","zh":"用 FAB v2 评测流程考查 v2.1 数据集上的金融分析任务。"},"source":{"name":"Vals AI","url":"https://www.vals.ai/benchmarks/fabv2-1","status":"current_snapshot","published_at":"2026-10-19"},"saturation_excluded":true,"scale_label":{"en":"Rubric score","zh":"量表得分"},"results":[[5023,0.6784600000000001,1,370,-1],[5169,0.67658,83,5034,-1],[5171,0.6757500000000001,66,5028,-1],[5024,0.6727500000000001,1,370,-1],[5167,0.6716200000000001,83,5034,-1],[5033,0.6576099999999999,83,5034,-1],[5172,0.65545,66,5028,-1],[5047,0.65428,25,5028,-1],[5175,0.65373,1,370,-1],[5051,0.6500400000000001,551,5028,-1],[5173,0.64705,370,370,-1],[5043,0.64259,370,370,-1],[5042,0.6331600000000001,370,370,-1],[5170,0.62933,25,5028,-1],[5182,0.61817,15,5028,-1],[5177,0.61723,417,5034,-1],[5053,0.6162299999999999,370,370,-1],[5181,0.61488,21,5028,-1],[5178,0.61383,417,5034,-1],[5180,0.60665,66,5037,-1],[5064,0.60587,551,5028,-1],[5038,0.60585,143,5037,-1],[5027,0.6042799999999999,15,5028,-1],[5048,0.6001299999999999,21,5028,-1],[5179,0.589,370,370,-1],[5168,0.58829,195,370,-1],[5174,0.58826,775,370,-1],[5059,0.57116,25,5028,-1],[5046,0.56841,15,5028,-1],[5184,0.55958,15,5028,-1],[5183,0.54708,370,5034,-1],[5068,0.54338,775,370,-1],[5186,0.51827,370,370,-1],[5075,0.49872999999999995,370,5076,-1],[5185,0.47729,370,5034,-1],[5080,0.44776000000000005,370,5076,-1]]},{"id":"gods-eye-bench-publisher","name":"gods-eye-bench","group":"Agent 与工具","unit":"%","metric":"Verified run accuracy","protocol":{"en":"Verified runs on the frozen God’s Eye View snapshot, 55 public questions, three trials per question, prompt v2.","zh":"对固定的 God’s Eye View 快照提问，使用第 2 版提示词。作者核验了 55 道公开问题，每题运行三次。"},"source":{"name":"gods-eye-bench authors","url":"https://github.com/Alpha-W0lf/gods-eye-bench","status":"current_snapshot","published_at":"2026-10-19"},"results":[[2956,1.0,1,3759,-1],[2959,1.0,1,3759,-1],[2961,1.0,1,3759,-1],[762,0.97,15,5670,-1],[325,0.95,1,3759,-1],[2968,0.94,143,5670,-1],[3391,0.92,143,5670,-1],[5262,0.92,370,5670,-1],[2593,0.88,83,5671,-1],[2983,0.87,83,5670,-1],[2585,0.84,83,5671,-1],[4035,0.82,83,5671,-1],[4219,0.81,83,5671,-1],[5672,0.72,195,5673,-1],[5674,0.71,195,5673,-1],[5675,0.57,370,5673,-1],[2653,0.56,83,5673,-1]]},{"id":"movebench-publisher","name":"MoveBench","group":"医疗","unit":"score","metric":"Overall normalized rubric score","protocol":{"en":"Release 1 uses 80 Chinese exercise and health questions, including 50 open-ended and 30 multiple-choice questions.","zh":"首批公开测试包含 80 道中文运动与健康题，其中 50 道开放题、30 道选择题。"},"source":{"name":"MoveBench authors","url":"https://github.com/Keep-Tech/MoveBench/releases/tag/release-1","status":"current_snapshot","published_at":"2026-10-19"},"score_max":1,"score_decimals":3,"results":[[2959,0.897,1,370,-1],[3157,0.886,195,370,-1],[3154,0.878,551,370,-1],[3198,0.864,775,370,-1],[3155,0.86,21,370,-1],[3158,0.858,370,370,-1],[3204,0.857,370,370,-1],[5676,0.848,15,370,-1],[5677,0.838,83,370,-1],[5335,0.837,370,370,-1]]},{"id":"spatialscout-2026-paper","name":"SpatialScout (2026-10, paper results)","group":"视觉与视频","unit":"%","metric":"All-target localization rate at IoU > 0.5","protocol":{"en":"Object-level localization in 80 scenes. A detection counts when the overlap of its predicted box with the reference exceeds 0.5. The denominator includes every target.","zh":"在 80 个场景中逐个统计目标。预测框与参考框的交并比超过 0.5 即为定位成功，分母包括所有目标。"},"source":{"name":"SpatialScout authors","url":"https://arxiv.org/abs/2610.26659","status":"paper_baselines","published_at":"2026-10-19"},"public_history":true,"results":[[2959,0.47229999999999994,1,5011,-1],[2961,0.336,1,5011,-1],[2958,0.3515,15,5011,-1],[2958,0.3155,15,5017,-1],[3394,0.1895,15,5011,-1],[3155,0.0277,21,5011,-1],[2982,0.0452,195,5011,-1],[3512,0.0092,25,5011,-1]]},{"id":"claimbench-publisher","name":"ClaimBench","group":"专业工作","unit":"%","metric":"Accuracy","protocol":{"en":"Classify each claim as supported, contradicted or unaddressed by its 10-K excerpt.","zh":"判断年报摘录支持、反驳还是未提及给定主张。"},"source":{"name":"Vals AI","url":"https://www.vals.ai/benchmarks/claimbench","status":"current_snapshot","published_at":"2026-10-20"},"results":[[5678,1.0,370,370,-1],[5024,1.0,1,370,-1],[5175,1.0,1,370,-1],[5023,1.0,1,370,-1],[5182,1.0,15,5028,-1],[5679,0.99694,370,370,-1],[5680,0.9918399999999999,370,370,-1],[5046,0.9918399999999999,15,5028,-1],[5681,0.9908199999999999,370,370,-1],[5682,0.98776,370,370,-1],[5683,0.98673,370,370,-1],[5684,0.98214,370,370,-1],[5685,0.9806100000000001,370,370,-1],[5686,0.9801000000000001,370,370,-1],[5687,0.97653,370,370,-1],[5688,0.97296,370,370,-1],[5689,0.9688800000000001,15,370,-1],[5690,0.9653100000000001,370,370,-1],[5691,0.9510200000000001,370,370,-1],[5692,0.94388,370,370,-1],[5693,0.8979600000000001,370,370,-1],[5694,0.8770399999999999,370,370,-1],[5695,0.85,370,370,-1],[5696,0.84847,370,370,-1],[5697,0.82449,370,370,-1]]},{"id":"selfdx-2026-paper","name":"SelfDx (2026-10, paper results)","group":"医疗","unit":"%","metric":"Appropriate caution on condition cases","protocol":{"en":"The 20-model current-generation panel. Each case pairs a proposed self-diagnosis with grounds for caution. Values are the authors’ percentage point estimates.","zh":"20 个当代模型接受同一组病例。患者提出自我诊断，病例中另有需要警惕的临床线索；统计模型作出恰当提醒的比例。"},"source":{"name":"SelfDx authors","url":"https://arxiv.org/abs/2610.27518","status":"paper_baselines","published_at":"2026-10-20"},"public_history":true,"results":[[2982,0.57,195,370,-1],[3156,0.47,370,370,-1],[3405,0.29,83,370,-1],[2977,0.33,21,370,-1],[5698,0.67,15,370,-1],[5298,0.64,15,370,-1],[5699,0.76,15,370,-1],[4048,0.77,15,370,-1],[5297,0.73,15,370,-1],[3391,0.53,143,370,-1],[3393,0.5,370,370,-1],[227,0.17,551,370,-1],[3151,0.31,370,370,-1],[3491,0.25,775,370,-1],[3401,0.47,370,370,-1],[3763,0.27,195,370,-1],[3196,0.72,15,370,-1],[3492,0.24,25,370,-1],[3412,0.13,83,370,-1],[5700,0.48,370,370,-1]]},{"id":"vals-router-index-publisher","name":"Vals Router Index","group":"综合","unit":"%","metric":"Router index","protocol":{"en":"Overall scores for routing systems and the published individual-model baselines.","zh":"列出路由系统及单模型基线的综合成绩。"},"source":{"name":"Vals AI","url":"https://www.vals.ai/benchmarks/vals_router_index","status":"current_snapshot","published_at":"2026-10-20"},"saturation_excluded":true,"scale_label":{"en":"Index","zh":"指数"},"results":[[5701,0.52195,370,370,-1],[5702,0.56458,370,370,-1],[5703,0.60019,370,370,-1],[5704,0.57795,370,370,-1],[5705,0.55652,370,370,-1],[5706,0.48113999999999996,370,370,-1],[5175,0.5870799999999999,1,370,-1],[5023,0.64156,1,370,-1],[5024,0.69028,1,370,-1],[5059,0.52974,25,370,-1],[5170,0.61257,25,370,-1],[5033,0.55951,83,370,-1],[5169,0.6923699999999999,83,370,-1],[5038,0.58564,143,370,-1],[5048,0.52955,21,370,-1],[5172,0.64877,66,370,-1],[5171,0.6678700000000001,66,370,-1],[5174,0.53781,775,370,-1],[5027,0.65023,15,370,-1],[5046,0.52322,15,370,-1],[5182,0.63944,15,370,-1],[5053,0.5320900000000001,370,370,-1],[5075,0.28961,370,370,-1],[5080,0.27588,370,370,-1],[5042,0.54826,370,370,-1],[5043,0.57813,370,370,-1],[5051,0.59729,551,370,-1],[5064,0.50861,551,370,-1]]},{"id":"vfs-bench-publisher","name":"VFS-Bench","group":"Agent 与工具","unit":"%","metric":"Toolathlon pass@1 with AgentVFS","protocol":{"en":"VFS-Bench 1.0, Toolathlon tasks with the AgentVFS interface. Each row reports the task pass rate for one model.","zh":"VFS-Bench 1.0 的 Toolathlon 任务，统一使用 AgentVFS 虚拟文件系统，按任务通过率计分。"},"source":{"name":"VFS-Bench authors","url":"https://agentvfs.org/blogs/agentvfs-1","status":"current_snapshot","published_at":"2026-10-12"},"results":[[2959,0.86,1,5707,-1],[2960,0.75,15,5707,-1],[3408,0.73,15,5707,-1],[2970,0.74,1,5707,-1],[2983,0.52,83,5707,-1]]},{"id":"frontiercs-2-2026-paper","name":"FrontierCS 2 (2026-10, paper results)","group":"编程","unit":"%","metric":"Pass rate against the research reference","protocol":{"en":"One run per agent on each of 48 research coding tasks. A pass requires beating the reference implementation on every metric and held-out workload.","zh":"48 项科研编程任务每项运行一次。提交的实现在每项指标和每个留出工作负载上均须胜过原论文的实现才算通过。"},"source":{"name":"FrontierCS 2 authors","url":"https://arxiv.org/abs/2610.29932","status":"paper_baselines","published_at":"2026-10-21"},"public_history":true,"results":[[2959,0.58,1,5708,-1],[2958,0.44,15,5709,-1],[2962,0.4,83,5710,-1],[2960,0.35,15,5709,-1],[3515,0.25,551,5711,-1],[2977,0.21,21,5712,-1],[2965,0.21,66,5713,-1],[3512,0.15,25,5714,-1],[2979,0.12,66,5715,-1],[3560,0.12,195,5716,-1],[3529,0.12,551,5711,-1],[3534,0.08,370,5717,-1]]},{"id":"vals-finance-index-publisher","name":"Vals Finance Index","group":"专业工作","unit":"%","metric":"Finance index","protocol":{"en":"Finance benchmarks weighted by subsector shares of US GDP.","zh":"按各子行业占美国 GDP 的比例汇总金融评测成绩。"},"source":{"name":"Vals AI","url":"https://www.vals.ai/benchmarks/finance_index","status":"current_snapshot","published_at":"2026-10-21"},"saturation_excluded":true,"scale_label":{"en":"Index","zh":"指数"},"results":[[5167,0.6704600000000001,83,5034,-1],[5023,0.6558700000000001,1,370,-1],[5169,0.6489,83,5034,-1],[5171,0.64551,66,5028,-1],[5024,0.64533,1,370,-1],[5168,0.61656,195,370,-1],[5172,0.61225,66,5028,-1],[5170,0.6088600000000001,25,5028,-1],[5051,0.60544,551,5028,-1],[5033,0.60461,83,5034,-1],[5038,0.58459,143,5037,-1],[5173,0.5805,370,370,-1],[5048,0.57955,21,5028,-1],[5043,0.57847,370,370,-1],[5053,0.57507,370,370,-1],[5175,0.57448,1,370,-1],[5042,0.57149,370,370,-1],[5047,0.5707800000000001,25,5028,-1],[5177,0.56977,417,5034,-1],[5178,0.5689500000000001,417,5034,-1],[5181,0.56012,21,5028,-1],[5174,0.55906,775,370,-1],[5064,0.55832,551,5028,-1],[5182,0.5581,15,5028,-1],[5027,0.55642,15,5028,-1],[5179,0.55428,370,370,-1],[5184,0.53319,15,5028,-1],[5046,0.53052,15,5028,-1],[5180,0.5299,66,5037,-1],[5059,0.5294,25,5028,-1],[5068,0.48366,775,370,-1],[5183,0.47217,370,5034,-1],[5186,0.46501,370,370,-1],[5185,0.41036,370,5034,-1],[5075,0.38248,370,5076,-1],[5080,0.36273000000000005,370,5076,-1]]},{"id":"celltrack-bench-2026-paper","name":"CellTrack-Bench (2026-10, paper results)","group":"视觉与视频","unit":"score","metric":"Mean task F1","protocol":{"en":"The frozen c1 hard test averages F1 for linking, repair and inspection.","zh":"在固定的 c1 困难测试集上，取细胞连线、修复和检查三项 F1 的均值。"},"source":{"name":"CellTrack-Bench authors","url":"https://arxiv.org/abs/2610.31566","status":"paper_baselines","published_at":"2026-10-22"},"score_max":1,"public_history":true,"score_decimals":3,"results":[[3480,0.752,15,5718,-1],[5719,0.724,15,5718,-1],[3453,0.758,1,5718,-1],[3196,0.682,15,5718,-1],[755,0.676,1,5718,-1],[3479,0.627,1,5718,-1],[2585,0.505,83,5720,-1],[5721,0.57,15,5718,-1],[325,0.503,1,5718,-1],[2576,0.484,195,5720,-1],[2582,0.443,195,5720,-1],[5722,0.385,551,5720,-1],[5723,0.392,370,5720,-1],[3491,0.333,775,5720,-1],[3568,0.409,1,5718,-1],[3763,0.326,195,5720,-1],[2590,0.264,195,5720,-1],[2977,0.276,21,5720,-1],[2642,0.246,370,5720,-1],[5724,0.27,417,5720,-1],[5725,0.204,370,5720,-1],[363,0.275,66,5720,-1]]},{"id":"compact-bench-2026-paper","name":"Compact-Bench (2026-10, paper results)","group":"编程","unit":"%","metric":"Stable task damage rate","protocol":{"en":"Of the 22 repairs solved with full history, this is the share that failed in at least two of three continuations after compaction. Lower is better. The authors round rates to whole percentages.","zh":"在保留完整历史时能修好的 22 个问题中，换用摘要后重试三次，至少两次失败的比例。越低越好，原表取整数百分比。"},"source":{"name":"Compact-Bench authors","url":"https://arxiv.org/abs/2610.31710","status":"paper_baselines","published_at":"2026-10-22"},"higher_is_better":false,"public_history":true,"results":[[3479,0.05,1,5726,-1],[3155,0.18,21,5726,-1],[5616,0.18,15,5726,-1],[4979,0.27,15,5726,-1],[3453,0.36,1,5726,-1],[2956,0.36,1,5726,-1],[4977,0.41,15,5726,-1],[3157,0.45,195,5726,-1],[5727,0.45,15,5726,-1],[2959,0.45,1,5726,-1],[3154,0.5,551,5726,-1],[3185,0.55,25,5726,-1],[5728,0.59,370,5726,-1],[5729,0.64,25,5726,-1]]},{"id":"pointerbench-2026-paper","name":"PointerBench (2026-10, paper results)","group":"视觉与视频","unit":"%","metric":"Mean GUI grounding success","protocol":{"en":"Sheets, Text and Pro each contain 500 screenshots. The score averages the three success rates. GUI specialists use their native grounding prompts and parsers.","zh":"表格、文字和专业界面各有 500 张截图。成绩取三组定位成功率的均值，专用 GUI 模型使用各自的提示和坐标解析方式。"},"source":{"name":"PointerBench authors","url":"https://arxiv.org/abs/2610.31274","status":"paper_baselines","published_at":"2026-07-09"},"public_history":true,"results":[[3453,0.8320000000000001,1,5730,-1],[755,0.8109999999999999,1,5730,-1],[762,0.777,15,5730,-1],[3408,0.727,15,5730,-1],[2973,0.723,15,5730,-1],[3572,0.722,1,5730,-1],[3479,0.6890000000000001,1,5730,-1],[5731,0.653,15,5730,-1],[3568,0.639,1,5730,-1],[3578,0.618,1,5730,-1],[3196,0.599,15,5730,-1],[5732,0.48700000000000004,15,5730,-1],[3184,0.462,15,5730,-1],[5733,0.447,15,5730,-1],[2058,0.23,15,5730,-1],[3391,0.102,143,5730,-1],[3741,0.087,143,5730,-1],[244,0.057,396,5730,-1],[5505,0.044000000000000004,15,5730,-1],[3490,0.037000000000000005,396,5730,-1],[5734,0.031,50,5730,-1],[3386,0.031,83,5730,-1],[2982,0.027999999999999997,50,5730,-1],[3491,0.025,775,5730,-1],[5735,0.024,50,5730,-1],[5736,0.021,50,5730,-1],[4070,0.016,50,5730,-1],[3415,0.015,83,5730,-1],[5737,0.015,50,5730,-1],[5738,0.013000000000000001,15,5730,-1],[3411,0.01,396,5730,-1],[3493,0.004,143,5730,-1],[3482,0.002,83,5730,-1],[5739,0.001,83,5730,-1],[5740,0.001,5741,5730,-1],[5742,0.0,143,5730,-1],[3601,0.0,2972,5730,-1],[5743,0.593,370,5744,-1],[5745,0.5489999999999999,370,5744,-1],[5746,0.545,370,5744,-1],[5747,0.513,370,5744,-1],[5748,0.512,370,5744,-1],[5749,0.51,370,5744,-1],[5750,0.509,370,5744,-1],[5751,0.507,195,5744,-1],[5752,0.505,370,5744,-1],[5753,0.501,370,5744,-1],[5754,0.473,370,5744,-1],[5755,0.467,370,5744,-1],[5756,0.462,370,5744,-1],[5757,0.457,370,5744,-1],[5758,0.44,370,5744,-1],[5759,0.436,370,5744,-1],[5760,0.429,195,5744,-1],[5761,0.423,370,5744,-1],[5762,0.41700000000000004,370,5744,-1],[5763,0.406,370,5744,-1],[5764,0.396,370,5744,-1],[5765,0.379,370,5744,-1],[5766,0.375,370,5744,-1],[5767,0.37200000000000005,370,5744,-1],[5768,0.371,370,5744,-1],[5769,0.369,370,5744,-1],[5770,0.358,370,5744,-1],[5771,0.355,370,5744,-1],[5772,0.337,370,5744,-1],[5773,0.337,370,5744,-1],[5774,0.33399999999999996,370,5744,-1],[5775,0.33299999999999996,370,5744,-1],[5776,0.332,370,5744,-1],[5777,0.315,370,5744,-1],[5778,0.307,370,5744,-1],[5779,0.3,370,5744,-1],[5780,0.293,370,5744,-1],[5781,0.272,370,5744,-1],[5782,0.253,370,5744,-1]]},{"id":"swe-clar-2026-paper","name":"SWE-Clar (2026-10, paper results)","group":"编程","unit":"%","metric":"Implementation pass rate","protocol":{"en":"After clarifying the client’s requirements, the model implements the requested change. Its score is the share of 100 deliveries that pass the acceptance tests.","zh":"模型先向客户澄清需求，再实现修改。分数是 100 项交付中通过验收测试的比例。"},"source":{"name":"SWE-Clar authors","url":"https://arxiv.org/abs/2610.31854","status":"paper_baselines","published_at":"2026-10-22"},"public_history":true,"results":[[2958,0.67,15,5783,-1],[755,0.63,1,5783,-1],[2956,0.6,1,5783,-1],[762,0.57,15,5783,-1],[5784,0.55,195,5783,-1],[3154,0.54,551,5783,-1],[2977,0.49,21,5783,-1],[3408,0.49,15,5783,-1],[2980,0.39,551,5783,-1]]},{"id":"benchbench-cloning-2026-paper","name":"BenchBench-Cloning (2026-10, paper results)","group":"数学与科学","unit":"%","metric":"Task-weighted pass rate","protocol":{"en":"Each of the 50 cloning-protocol tasks and 15 construct-design tasks has equal weight. Each system runs at its highest tested reasoning effort.","zh":"50 项克隆实验方案和 15 项构建设计任务各占相同权重。每个系统使用论文测试的最高推理强度。"},"source":{"name":"BenchBench-Cloning authors","url":"https://arxiv.org/abs/2610.33355","status":"paper_baselines","published_at":"2026-10-23"},"public_history":true,"results":[[2958,0.306,15,5785,-1],[2960,0.222,15,5785,-1],[2959,0.205,1,5785,-1],[2962,0.188,83,4999,-1],[5166,0.16,1,5785,-1],[2968,0.109,143,5786,-1],[2977,0.08800000000000001,21,4999,-1],[2983,0.086,83,4999,-1]]},{"id":"dmaic-bench-2026-paper","name":"DMAIC-Bench (2026-10, paper results)","group":"专业工作","unit":"score","metric":"Mean task reward","protocol":{"en":"Each model attempts 97 analytical tasks three times under the default instruction. The reward combines method choice, numerical accuracy and decisions about whether to proceed.","zh":"每个模型按默认指令独立完成 97 项分析任务各三次。平均奖励分综合方法选择、计算结果和是否继续执行的判断。"},"source":{"name":"DMAIC-Bench authors","url":"https://arxiv.org/abs/2610.32305","status":"paper_baselines","published_at":"2026-10-23"},"score_max":1,"public_history":true,"score_decimals":3,"results":[[5404,0.662,15,5787,-1],[4048,0.649,15,5787,-1],[5788,0.603,21,5787,-1],[5789,0.599,1,5787,-1],[5413,0.599,25,5787,-1],[5412,0.592,25,5787,-1],[3154,0.582,551,5787,-1],[5297,0.564,15,5787,-1],[4981,0.557,1,5787,-1],[3396,0.556,551,5787,-1],[4983,0.542,1,5787,-1],[5790,0.404,195,5787,-1],[5791,0.379,195,5787,-1]]},{"id":"swe-shift-2026-paper","name":"SWE-Shift (2026-10, paper results)","group":"编程","unit":"%","metric":"Paired success rate (PSR)","protocol":{"en":"The same saved work is continued with and without a change. Paired success credits both completed branches or a verified preemptive success, across 120 scenarios and nine intervention positions. Each continuation has a one-million-token budget.","zh":"同一工作进度分别在原环境和变更环境中续跑。两个分支都成功，或变更前已完成且通过核验，才计入配对成功率。120 个场景各设九个变更时点，每次续跑限 100 万 token。"},"source":{"name":"SWE-Shift authors","url":"https://arxiv.org/abs/2610.32196","status":"paper_baselines","published_at":"2026-10-23"},"public_history":true,"results":[[2958,0.2034,15,5792,-1],[2956,0.16879999999999998,1,5793,-1],[755,0.1478,1,5793,-1],[762,0.113,15,5792,-1],[3155,0.08199999999999999,21,5794,-1],[5795,0.0883,25,5794,-1],[5796,0.076,25,5797,-1],[3187,0.071,21,5798,-1],[3198,0.062,775,5797,-1],[3396,0.053,551,5797,-1],[3190,0.057,775,5798,-1],[5799,0.047,370,5797,-1],[5800,0.045,370,5801,-1],[5300,0.0434,195,5794,-1],[5802,0.026000000000000002,370,5794,-1],[793,0.009000000000000001,195,5798,-1],[5803,0.021,195,5798,-1]]},{"id":"vals-legal-index-publisher","name":"Vals Legal Index","group":"专业工作","unit":"%","metric":"Legal index","protocol":{"en":"Legal research and drafting benchmarks weighted by US law-office revenue shares.","zh":"按美国律师事务所各业务领域的收入占比汇总法律研究和起草成绩。"},"source":{"name":"Vals AI","url":"https://www.vals.ai/benchmarks/legal_index","status":"current_snapshot","published_at":"2026-10-23"},"saturation_excluded":true,"scale_label":{"en":"Index","zh":"指数"},"results":[[5171,0.77863,66,5028,-1],[5169,0.76717,83,5034,-1],[5167,0.7593000000000001,83,5034,-1],[5172,0.75259,66,5028,-1],[5051,0.74575,551,5028,-1],[5168,0.73901,195,370,-1],[5038,0.72596,143,5037,-1],[5023,0.7233499999999999,1,370,-1],[5043,0.7214100000000001,370,370,-1],[5170,0.71863,25,5028,-1],[5048,0.7172499999999999,21,5028,-1],[5053,0.71699,370,370,-1],[5064,0.71051,551,5028,-1],[5173,0.70552,370,370,-1],[5175,0.70294,1,370,-1],[5179,0.70162,370,370,-1],[5033,0.69784,83,5034,-1],[5059,0.69012,25,5028,-1],[5042,0.6899,370,370,-1],[5182,0.68718,15,5028,-1],[5180,0.68512,66,5037,-1],[5027,0.6838599999999999,15,5028,-1],[5024,0.68021,1,370,-1],[5177,0.67838,417,5034,-1],[5174,0.67069,775,370,-1],[5181,0.67041,21,5028,-1],[5178,0.65613,417,5034,-1],[5183,0.62963,370,5034,-1],[5046,0.6220300000000001,15,5028,-1],[5186,0.61799,370,370,-1],[5184,0.5875699999999999,15,5028,-1],[5080,0.5783,370,5076,-1],[5185,0.55366,370,5034,-1],[5075,0.5416,370,5076,-1]]},{"id":"secprbench-2026-paper","name":"SecPRBench (2026-10, paper results)","group":"安全与可靠性","unit":"%","metric":"Vulnerability recall","protocol":{"en":"Each system reviews the same 265 buggy pull requests. Recall is the share where it identifies the reference vulnerability.","zh":"每个系统审查相同的 265 个含漏洞的拉取请求。召回率是成功发现指定漏洞的比例。"},"source":{"name":"SecPRBench authors","url":"https://arxiv.org/abs/2610.33509","status":"paper_baselines","published_at":"2026-10-24"},"public_history":true,"results":[[2974,0.7809999999999999,143,3759,-1],[3515,0.747,551,3759,-1],[2959,0.74,1,3759,-1],[3568,0.5920000000000001,1,3759,-1],[3491,0.325,775,3759,-1],[5804,0.762,15,5805,-1],[3160,0.5579999999999999,15,5805,-1],[5806,0.555,15,5805,-1],[3160,0.475,15,5807,-1]]},{"id":"behavior-spatial-benchmark-2026-paper","name":"Behavior Spatial Benchmark (2026-10, paper results)","group":"视觉与视频","unit":"%","metric":"Single-frame VQA accuracy","protocol":{"en":"Models answer spatial questions about a single frame, choosing one of four options. Accuracy pools all single-frame questions.","zh":"模型看单帧画面回答四选一空间问题。准确率按所有单帧题合计。"},"source":{"name":"Behavior Spatial Benchmark authors","url":"https://arxiv.org/abs/2610.34051","status":"paper_baselines","published_at":"2026-10-25"},"public_history":true,"results":[[2958,0.821,15,5808,-1],[5809,0.778,83,5808,-1],[5810,0.705,1,5808,-1],[2058,0.647,15,5808,-1],[5811,0.599,83,5808,-1],[5725,0.518,370,5808,-1],[4070,0.506,195,5808,-1],[4969,0.46799999999999997,195,5808,-1],[5812,0.455,370,5808,-1],[5813,0.36200000000000004,370,5808,-1],[5814,0.348,370,5808,-1],[5815,0.344,83,5808,-1],[5816,0.33399999999999996,195,5808,-1],[5817,0.314,25,5808,-1],[5818,0.096,370,5819,-1],[5820,0.009000000000000001,370,5819,-1]]},{"id":"braillebench-2026-paper","name":"BrailleBench (2026-10, paper results)","group":"视觉与视频","unit":"%","metric":"Grade-1 image exact match","protocol":{"en":"The model must transcribe the entire image correctly. All 13 systems are scored on the same 215 Grade-1 Braille photographs.","zh":"13 个系统识别同一组 215 张一级盲文照片。整张图的转录与参考答案完全一致才计为正确。"},"source":{"name":"BrailleBench authors","url":"https://arxiv.org/abs/2610.36195","status":"paper_baselines","published_at":"2026-10-26"},"public_history":true,"results":[[3196,0.0,15,5821,-1],[3490,0.0,21,5821,-1],[3479,0.009300000000000001,1,5821,-1],[3415,0.0279,83,5821,-1],[5822,0.004699999999999999,83,5821,-1],[762,0.1116,15,5823,-1],[2960,0.5395,15,5821,-1],[2959,0.8837,1,4992,-1],[762,0.6279,15,2300,-1],[5824,0.0326,370,5825,-1],[5826,0.004699999999999999,66,5827,-1],[5828,0.8698,195,5829,-1],[5830,0.8512000000000001,370,5829,-1]]},{"id":"embodiedlab-2026-paper","name":"EmbodiedLab (2026-10, paper results)","group":"数学与科学","unit":"%","metric":"T0 experiment success rate","protocol":{"en":"The agent operates a simulated physics apparatus through its base interface. The 30 experiments receive equal weight. Success requires all six milestones, including measurement and cleanup.","zh":"智能体通过基础接口操作模拟物理仪器。30 项实验等权计分，须完成包括测量和归位在内的全部六步才算成功。"},"source":{"name":"EmbodiedLab authors","url":"https://arxiv.org/abs/2610.35385","status":"paper_baselines","published_at":"2026-10-26"},"public_history":true,"results":[[2958,0.4,15,5418,-1],[2960,0.28300000000000003,15,5418,-1],[2959,0.433,1,5419,-1],[3385,0.1,25,5831,-1],[2977,0.1,21,5832,-1],[3396,0.0,551,5833,-1]]},{"id":"kw-bench-2026-paper","name":"kw-bench (2026-10, paper results)","group":"专业工作","unit":"score","metric":"VAT investigation score","protocol":{"en":"Six model–harness runs solve the same synthetic VAT investigation (variant b), graded against the generated answer key. Each row represents one run.","zh":"六组模型与工具组合分别完成同一道模拟增值税取证题（b 版），按生成的标准答案评分。每组各运行一次。"},"source":{"name":"kw-bench authors","url":"https://arxiv.org/abs/2610.36405","status":"paper_baselines","published_at":"2026-10-26"},"score_max":1,"public_history":true,"score_decimals":4,"results":[[5834,0.542,25,5393,-1],[5834,0.528,25,5835,-1],[5834,0.0,25,2300,-1],[5834,0.0,25,3759,-1],[2959,0.9986,1,5835,-1],[2959,1.0,1,3759,-1]]},{"id":"3dcodeedit-2026-paper","name":"3DCodeEdit (2026-10, paper results)","group":"编程","unit":"%","metric":"Single-shot all-pass rate","protocol":{"en":"All 44 timeline models edit Blender programs on the same 312-task subset. A task passes when the edit executes, renders and meets every checklist item.","zh":"44 个模型在相同的 312 道题上一次性修改 Blender 程序。修改后能运行、能渲染，并满足检查表的全部要求，才算通过。"},"source":{"name":"3DCodeEdit authors","url":"https://arxiv.org/abs/2610.38231","status":"paper_baselines","published_at":"2026-10-28"},"public_history":true,"results":[[5836,0.09,195,5837,-1],[513,0.09,370,5838,-1],[786,0.096,195,5839,-1],[323,0.17300000000000001,1,5840,-1],[2055,0.157,83,5841,-1],[62,0.14400000000000002,15,5842,-1],[5843,0.11900000000000001,195,5844,-1],[283,0.205,1,5845,-1],[325,0.18600000000000003,1,5846,-1],[3738,0.304,1,5847,-1],[3424,0.27899999999999997,83,5848,-1],[5803,0.087,195,5849,-1],[2261,0.34299999999999997,1,5850,-1],[3572,0.36200000000000004,1,5851,-1],[3412,0.353,83,5852,-1],[2584,0.16,195,5853,-1],[2590,0.115,195,5853,-1],[5828,0.003,195,5854,-1],[4969,0.096,195,5854,-1],[4070,0.077,195,5854,-1],[3184,0.401,15,5855,-1],[2585,0.16,83,5856,-1],[3764,0.135,195,5857,-1],[3763,0.196,195,5858,-1],[3196,0.516,15,5859,-1],[3415,0.38799999999999996,83,5860,-1],[3479,0.39399999999999996,1,5861,-1],[3568,0.28800000000000003,1,5862,-1],[5297,0.263,15,5863,-1],[4048,0.532,15,5864,-1],[2977,0.46799999999999997,21,5865,-1],[3405,0.413,83,5866,-1],[755,0.522,1,5867,-1],[3386,0.452,83,5868,-1],[5300,0.26,195,5869,-1],[2956,0.5379999999999999,1,5870,-1],[2983,0.5770000000000001,83,5871,-1],[5404,0.6759999999999999,15,5872,-1],[2959,0.6759999999999999,1,5873,-1],[5874,0.369,15,5873,-1],[2961,0.506,1,5875,-1],[5876,0.647,15,5877,-1],[2970,0.423,1,5878,-1],[2962,0.5379999999999999,83,5879,-1]]},{"id":"ai-agent-building-kit-publisher","name":"AI Agent Building Kit","group":"安全与可靠性","unit":"score","metric":"Cases passing all three trials (of 43)","protocol":{"en":"The same guarded agents handle 43 HR, support and research cases, including 25 attacks. A case counts only when all three trials pass.","zh":"同一套防护机制下，智能体处理 43 个人事、客服和研究案例，其中 25 个含攻击。每个案例连续三次测试均通过才计数。"},"source":{"name":"AI Agent Building Kit authors","url":"https://github.com/ji-hun-git/ai-agent-building-kit/tree/main/benchmarks","status":"current_snapshot","published_at":"2026-10-28"},"score_max":43,"results":[[2970,43.0,1,5880,-1],[2958,43.0,15,5881,-1],[2976,43.0,15,5881,-1],[2959,42.0,1,5880,-1],[2979,42.0,66,5882,-1],[5883,29.0,195,5884,-1]]},{"id":"immigration-bench-publisher","name":"Immigration Bench","group":"专业工作","unit":"%","metric":"Overall rubric score","protocol":{"en":"US immigration law questions, graded against the rubric for each question.","zh":"美国移民法问题按逐题评分标准计分。"},"source":{"name":"Vals AI","url":"https://www.vals.ai/benchmarks/immigration","status":"current_snapshot","published_at":"2026-10-28"},"saturation_excluded":true,"scale_label":{"en":"Rubric score","zh":"量表得分"},"results":[[5025,0.75444,1,370,-1],[5023,0.7214,1,370,-1],[5171,0.71221,66,5028,-1],[5060,0.70139,195,370,-1],[5172,0.69616,66,5028,-1],[5024,0.68962,1,370,-1],[5048,0.6825600000000001,21,5028,-1],[5169,0.6807500000000001,83,5034,-1],[5038,0.68049,143,5037,-1],[5182,0.6780700000000001,15,5028,-1],[5027,0.67691,15,5028,-1],[5043,0.67501,370,370,-1],[5051,0.66731,551,5028,-1],[5050,0.66542,66,5037,-1],[5045,0.65982,143,5034,-1],[5047,0.6563800000000001,25,5028,-1],[5036,0.64997,66,5037,-1],[5030,0.64929,66,5028,-1],[5178,0.63708,417,5034,-1],[5033,0.63405,83,5034,-1],[5046,0.62529,15,5028,-1],[5175,0.61791,1,370,-1],[5059,0.6165999999999999,25,5028,-1],[5068,0.57526,775,370,-1],[5074,0.55661,83,5034,-1]]},{"id":"treasuryqa-bench-2026-paper","name":"TreasuryQA Bench (2026-10, paper results)","group":"专业工作","unit":"%","metric":"Pass@1","protocol":{"en":"Each agent answers 70 numerical questions from historical U.S. Treasury reports three times. Pass@1 estimates the chance that one attempt yields the correct answer.","zh":"每个智能体从美国财政部历史报告中回答 70 道数值题，每题做三次。pass@1 估计单次作答正确的概率。"},"source":{"name":"TreasuryQA Bench authors","url":"https://arxiv.org/abs/2610.38924","status":"paper_baselines","published_at":"2026-10-28"},"public_history":true,"results":[[2959,0.648,1,5885,-1],[2962,0.6,83,5886,-1],[2960,0.514,15,5887,-1],[2983,0.51,83,5886,-1],[2961,0.419,1,5885,-1],[3512,0.371,25,5888,-1],[2977,0.276,21,5888,-1],[3529,0.271,551,5889,-1],[2968,0.18100000000000002,143,5890,-1],[2973,0.16699999999999998,15,5887,-1]]},{"id":"trusstutor-bench-2026-paper","name":"TrussTutor-Bench (2026-10, paper results)","group":"数学与科学","unit":"%","metric":"Baseline problem-solving accuracy","protocol":{"en":"Each model receives only the statement and diagram for 200 planar truss problems. Every requested quantity must match the verified solution.","zh":"模型仅凭题目和结构图解答 200 道平面桁架题。所有要求求出的量都与核验答案相符，才算答对。"},"source":{"name":"TrussTutor-Bench authors","url":"https://arxiv.org/abs/2610.38917","status":"paper_baselines","published_at":"2026-10-28"},"public_history":true,"results":[[2958,0.97,15,5891,-1],[762,0.955,15,5891,-1],[2962,0.95,83,5892,-1],[2983,0.93,83,5892,-1],[3412,0.92,83,5892,-1],[755,0.915,1,5893,-1],[2956,0.86,1,5894,-1],[2973,0.83,15,5891,-1],[3396,0.74,551,5895,-1],[3568,0.715,1,5893,-1],[3512,0.66,25,5896,-1],[3198,0.65,775,5897,-1],[3408,0.645,15,5891,-1]]},{"id":"dynarev-2026-paper","name":"DynaRev (2026-10, paper results)","group":"Agent 与工具","unit":"score","metric":"Mean constraint satisfaction score","protocol":{"en":"The agent receives revised instructions while working. The score averages final constraint satisfaction across the dynamic task subsets.","zh":"智能体在执行途中收到修改后的指令。分数取各动态任务中最终满足约束的平均程度。"},"source":{"name":"DynaRev authors","url":"https://arxiv.org/abs/2610.40133","status":"paper_baselines","published_at":"2026-10-29"},"score_max":1,"public_history":true,"score_decimals":3,"results":[[3153,0.698,25,5898,-1],[5618,0.598,15,5898,-1],[5519,0.735,83,5898,-1],[5899,0.604,195,5898,-1],[5900,0.681,1,5898,-1]]},{"id":"preflight-bench-2026-paper","name":"preflight-bench (2026-10, paper results)","group":"Agent 与工具","unit":"%","metric":"Stale-click detection recall","protocol":{"en":"The two hosted models inspect every test pair. Recall measures detection of 1,064 stale clicks. Haiku also flagged 95% of valid clicks and Opus flagged 12%. Open models ran on smaller balanced subsets.","zh":"两款托管模型检查全部测试样本，召回率表示是否识别出 1,064 次目标已移动的点击。对仍然有效的点击，Haiku 的误报率为 95%，Opus 为 12%。开放模型另测平衡抽样子集。"},"source":{"name":"preflight-bench authors","url":"https://arxiv.org/abs/2610.39282","status":"paper_baselines","published_at":"2026-10-29"},"public_history":true,"results":[[2970,0.98,1,5901,-1],[2959,0.83,1,5902,-1]]},{"id":"fontvibe-ai-design-benchmark-publisher","name":"FontVibe AI Design Benchmark","group":"生成与创作","unit":"score","metric":"Mean design quality","protocol":{"en":"Seven models completed designs for all 12 briefs using the same image generator. Finished designs are scored on legibility (20%), lettering and finish (35%), information and layout (25%) and appeal (20%). Opus 5.5 finished 18 of 36 runs on eight briefs, so its 7.97 is reported separately.","zh":"七款模型在全部 12 份简报上均有作品交付，使用同一图像生成器。完成的作品按易读性（20%）、字艺与做工（35%）、信息与排版（25%）、吸引力（20%）四项盲评加权。Opus 5.5 在 36 次运行中完成 18 次，覆盖八份简报，所以其 7.97 分单列。"},"source":{"name":"FontVibe AI Design Benchmark authors","url":"https://fontvibe.ai/blog/gpt-6-1-terra-vs-claude-gemini-design-test","status":"current_snapshot","published_at":"2026-10-30"},"score_max":10,"score_min":1,"score_decimals":2,"results":[[2976,7.69,15,5903,-1],[2973,7.58,15,5904,-1],[2962,7.56,83,5903,-1],[2961,7.43,1,5903,-1],[2960,7.35,15,5903,-1],[2983,7.33,83,5903,-1],[2964,7.08,15,5903,-1]]},{"id":"reconspatial-2026-paper","name":"RECONSPATIAL (2026-11, paper results)","group":"视觉与视频","unit":"%","metric":"Overall spatial score","protocol":{"en":"Geometric perception, spatial relations, scene reconstruction and navigation each have equal weight. The navigation scorer normalizes path scale.","zh":"几何感知、空间关系、场景重建和导航各占四分之一。导航得分按经过尺度归一化的路径计算。"},"source":{"name":"RECONSPATIAL authors","url":"https://arxiv.org/abs/2611.02026","status":"paper_baselines","published_at":"2026-11-01"},"public_history":true,"results":[[4977,0.595,15,5905,-1],[3157,0.484,195,5905,-1],[3155,0.451,21,5905,-1],[5519,0.441,83,5905,-1],[4979,0.44,15,5905,-1],[4983,0.439,1,5905,-1],[4960,0.425,1,5905,-1],[5906,0.41100000000000003,83,5905,-1],[3396,0.40700000000000003,551,5905,-1],[3196,0.39799999999999996,15,5905,-1],[3204,0.395,370,5905,-1],[5295,0.39399999999999996,143,5905,-1],[502,0.34600000000000003,83,5905,-1],[2129,0.345,83,5905,-1],[4961,0.332,195,5905,-1],[3194,0.298,1,5905,-1],[3181,0.282,1,5905,-1],[3198,0.264,775,5905,-1]]},{"id":"agent-money-benchmark-publisher","name":"Agent Money Benchmark","group":"Agent 与工具","unit":"usd","metric":"Wasted USD per $100","protocol":{"en":"Normalized waste per $100, averaging two equally weighted $50 tracks. Lower is better.","zh":"每 100 美元中的浪费金额，两组任务各按 50 美元等权计入。金额越低越好。"},"source":{"name":"BlockRun","url":"https://agentmoneybench.com/","status":"current_snapshot","published_at":"2026-11-02"},"higher_is_better":false,"results":[[2959,25.0,1,370,-1],[2960,27.9,15,370,-1],[2958,33.3,15,370,-1],[2962,34.3,83,370,-1],[2956,37.9,1,370,-1],[2961,56.3,1,370,-1],[3515,58.3,48,370,-1],[3492,65.3,25,370,-1],[2967,66.3,396,370,-1],[2976,71.0,15,370,-1]]},{"id":"collaborative-intelligence-benchmark-2026-paper","name":"Collaborative Intelligence Benchmark (2026-11, paper results)","group":"Agent 与工具","unit":"%","metric":"Overall collaborative test score","protocol":{"en":"Each model works with a scripted teammate. The score averages eight dimensions, using only tasks that the same model solved alone. Each team test runs three times.","zh":"模型与按脚本行动的队友协作。各模型仅在自己能独立解决的题目上计分，每项协作测试运行三次，最终取八个维度的平均分。"},"source":{"name":"Collaborative Intelligence Benchmark authors","url":"https://arxiv.org/abs/2611.02633","status":"paper_baselines","published_at":"2026-11-02"},"public_history":true,"results":[[2962,0.7659999999999999,83,5907,-1],[2958,0.765,15,5908,-1],[2960,0.745,15,5909,-1],[2959,0.71,1,5910,-1],[2956,0.653,1,5911,-1],[2961,0.652,1,5912,-1],[2968,0.556,143,5913,-1],[2970,0.517,1,5914,-1],[2983,0.515,83,5915,-1],[2992,0.445,417,5916,-1],[2976,0.434,15,5917,-1],[2967,0.599,21,5918,-1],[3512,0.508,25,5919,-1],[2977,0.45899999999999996,21,5914,-1],[3154,0.45799999999999996,551,5920,-1],[3492,0.39399999999999996,25,5921,-1],[3418,0.306,370,5922,-1],[62,0.191,15,5923,-1]]},{"id":"distdebug-bench-2026-paper","name":"DistDebug-Bench (2026-11, paper results)","group":"编程","unit":"%","metric":"Mean block coverage","protocol":{"en":"How much of the reference fault location does a diagnosis cover? The authors average the first three valid attempts for each incident, then average across the same 120 distributed-system incidents.","zh":"诊断报告定位到了多少参考故障代码块？每起事故先取前三次有效诊断的平均覆盖率，再对同一组 120 起分布式系统事故取平均。"},"source":{"name":"DistDebug-Bench authors","url":"https://arxiv.org/abs/2611.03523","status":"paper_baselines","published_at":"2026-11-02"},"public_history":true,"results":[[2958,0.677,15,5924,-1],[2959,0.664,1,5924,-1],[3196,0.6509999999999999,15,5924,-1],[3184,0.631,15,5924,-1],[3733,0.597,15,5924,-1],[5925,0.585,370,5924,-1],[3490,0.583,21,5924,-1],[5292,0.58,370,5924,-1],[5926,0.564,370,5924,-1],[5927,0.564,15,5924,-1],[3158,0.516,370,5924,-1],[5928,0.475,83,5924,-1],[5929,0.39299999999999996,15,5924,-1],[5927,0.465,15,5395,-1],[5929,0.344,15,5395,-1],[5927,0.5,15,5393,-1],[5929,0.335,15,5393,-1]]},{"id":"memshift-2026-paper","name":"MemShift (2026-11, paper results)","group":"指令与长文","unit":"%","metric":"Repair accuracy after source revision","protocol":{"en":"Given the same 165 changed definition items from ten test worlds, each writer re-derives the value from its own 6,000-token record selection. The score is the share repaired correctly.","zh":"源记录修订后，三款模型从各自检索到的 6,000 个 token 以内的记录重新推导取值。分数统计十个测试世界中同一组 165 个已变化定义项的修正正确率。"},"source":{"name":"MemShift authors","url":"https://arxiv.org/abs/2611.02338","status":"paper_baselines","published_at":"2026-11-02"},"public_history":true,"results":[[2961,0.455,1,5930,-1],[2959,0.43,1,5930,-1],[4070,0.127,195,5931,-1]]},{"id":"nitibench-2026-paper","name":"NitiBench+ (2026-11, paper results)","group":"专业工作","unit":"score","metric":"Thai Bar-standard score","protocol":{"en":"For each of 500 held-out legal questions, the model can search the Thai law in force at the time of the decision. Answers are graded on the Thai Bar’s 0–10 scale.","zh":"模型回答 500 道留出法律题，可检索裁判当时生效的泰国法律。答案按泰国律师资格考试的 0 至 10 分标准评分。"},"source":{"name":"NitiBench authors","url":"https://arxiv.org/abs/2611.03576","status":"paper_baselines","published_at":"2026-11-02"},"score_max":10,"public_history":true,"score_decimals":2,"results":[[2956,7.78,1,5932,-1],[2962,7.54,83,5932,-1],[2960,7.28,15,5932,-1],[2959,7.03,1,5932,-1],[2979,6.37,66,5932,-1],[2964,6.14,15,5932,-1],[2961,5.97,1,5932,-1],[5933,5.43,551,5932,-1],[2970,4.62,1,5932,-1],[3512,6.39,25,5932,-1],[3261,5.71,25,5932,-1],[5320,4.93,66,5932,-1],[3529,4.8,551,5932,-1],[2585,4.57,83,5932,-1],[5934,3.68,83,5932,-1],[5935,2.47,370,5932,-1],[2986,5.48,195,5932,-1],[5936,5.1,370,5932,-1],[5937,1.62,370,5932,-1],[5938,2.45,370,5932,-1],[5939,0.87,370,5932,-1],[5940,3.23,370,5932,-1],[5941,2.88,370,5932,-1],[5344,0.96,195,5932,-1],[5942,1.54,370,5932,-1],[5943,1.46,370,5932,-1],[5944,1.1,370,5932,-1],[5945,0.87,370,5932,-1]]},{"id":"sweetbench-2026-paper","name":"SweetBench (2026-11, paper results)","group":"医疗","unit":"score","metric":"COGI-4-8 glucose-control score","protocol":{"en":"The same agent controller makes hourly decisions in simulated 72-hour glucose-control episodes. Higher scores reflect more time in range, less hypoglycemia and lower variability. The reported means cover completed episodes.","zh":"同一控制器在为期 72 小时的血糖模拟中每小时决策一次。综合分衡量达标时间、低血糖和波动程度。成绩按完成的模拟取平均。"},"source":{"name":"SweetBench authors","url":"https://arxiv.org/abs/2611.03601","status":"paper_baselines","published_at":"2026-11-02"},"score_max":100,"public_history":true,"results":[[2959,73.54,1,5946,-1],[5506,70.58,15,5946,-1],[5947,68.61,25,5946,-1],[2967,67.15,21,5946,-1],[3529,66.25,551,5946,-1]]},{"id":"agenticfinbench-2026-paper","name":"AgenticFinBench (2026-11, paper results)","group":"专业工作","unit":"score","metric":"Mean financial answer quality (Q)","protocol":{"en":"All 18 model–agent configurations answer the same 230 financial questions. Each has equal weight. The quality score combines the task rubric with a factual audit of the answer’s claims.","zh":"18 组模型与智能体配置回答相同的 230 道金融题，各题等权。质量分综合任务评分细则和对答案中事实陈述的核查结果。"},"source":{"name":"AgenticFinBench authors","url":"https://arxiv.org/abs/2611.04813","status":"paper_baselines","published_at":"2026-11-03"},"score_max":100,"public_history":true,"results":[[762,89.2,15,5395,-1],[3512,88.79,25,5395,-1],[762,88.62,15,5948,-1],[3512,87.84,25,5948,-1],[2960,87.68,15,5395,-1],[2958,87.27,15,5395,-1],[2977,86.74,21,5949,-1],[3512,85.82,25,5950,-1],[2962,85.09,83,5378,-1],[2959,84.92,1,5948,-1],[2959,84.38,1,5951,-1],[762,84.03,15,5950,-1],[2959,83.13,1,5395,-1],[2959,80.68,1,5950,-1],[2976,78.69,15,5395,-1],[5118,77.95,370,5952,-1],[3512,71.27,25,5953,-1],[5954,65.42,370,5955,-1]]},{"id":"kierston-kierstonbench-2026-paper","name":"Kierston & KierstonBench (2026-11, paper results)","group":"Agent 与工具","unit":"score","metric":"Mean score across all 32 tasks","protocol":{"en":"Each model attempts 22 technical tasks, seven real-time investigations and three deception tasks. Correct claims earn credit and wrong claims incur penalties. Abstentions count as zero in the mean over all 32 tasks.","zh":"每款模型各做 22 项技术任务、七项实时调查和三项欺骗识别任务。答对得分，错误陈述扣分，弃答按零分计入全部 32 项的均值。"},"source":{"name":"Kierston & KierstonBench authors","url":"https://arxiv.org/abs/2611.04785","status":"paper_baselines","published_at":"2026-11-03"},"score_max":1,"score_min":-1,"public_history":true,"score_decimals":2,"results":[[755,0.36,1,5956,-1],[2956,0.52,1,5956,-1],[2959,0.39,1,5956,-1],[2961,0.25,1,5956,-1],[2958,0.43,15,5956,-1],[762,0.36,15,5956,-1],[3394,0.55,15,5956,-1],[2960,0.46,15,5956,-1],[2976,0.15,15,5956,-1],[3491,0.22,775,5956,-1],[3512,0.35,25,5956,-1],[2983,0.28,83,5956,-1],[3515,0.31,551,5956,-1],[2977,0.29,21,5956,-1],[3560,0.34,195,5956,-1],[2968,0.2,143,5956,-1]]},{"id":"extractbench-value-f1","name":"ExtractBench (Value F1)","group":"专业工作","unit":"score","score_max":100,"score_decimals":2,"saturation_excluded":true,"scale_label":{"en":"Value F1 / 100","zh":"字段 F1 / 100"},"metric":"Value F1","protocol":{"en":"370 enterprise documents. Official ranked model and coding-agent configurations; specialized extraction API products excluded. Deterministic value F1, with equal document weights. Failed, missing and over-30-minute documents score zero. ExtractBench Pro is a separate dataset.","zh":"370 份企业文档，采用官方公开排名中的模型及编码 Agent 配置，不含专用抽取 API 产品。字段 F1 由程序计算，各文档等权；失败、缺失或超过 30 分钟的文档计零。ExtractBench Pro 为独立题集。"},"source":{"name":"ExtractBench authors","url":"https://github.com/run-llama/ExtractBench","status":"current_snapshot","license":{"name":"Apache-2.0","url":"https://github.com/run-llama/ExtractBench/blob/4064262a67eb19ee6d4213754a2efc108733e46a/LICENSE"},"published_at":"2026-07-31"},"results":[[3479,86.84,1,5957,5958],[2959,86.06,1,5959,5958],[2961,94.68,1,5960,5958],[3196,93.35,15,5961,5958],[3408,91.16,15,5962,5958],[762,93.77,15,5963,5958],[2973,92.26,15,5964,5958],[2976,91.01,15,5965,5958],[3394,94.57,15,5966,5958],[2960,95.31,15,5967,5958],[2964,94.28,15,5968,5958],[3512,87.11,25,5969,5958],[5970,45.39,83,5971,5958],[5972,70.27,83,5973,5958],[5974,51.85,83,5975,5958],[5976,69.64,83,5977,5958],[5978,45.19,551,5979,5958],[2980,80.75,551,5980,5958],[5981,79.84,83,5982,5958],[5983,80.71,83,5984,5958],[5985,66.35,370,5986,5958],[5987,64.63,370,5988,5958],[2977,83.17,21,5989,5958],[5990,57.77,21,5991,5958],[3184,92.97,15,5992,5958],[5993,78.43,370,5994,5958],[5995,45.85,370,5996,5958],[5997,49.42,370,5998,5958],[5999,81.02,370,6000,5958],[6001,74.9,15,6002,5958],[6003,91.91,15,6004,5958],[6005,46.63,195,6006,5958],[6007,63.4,195,6008,5958],[6009,88.0,195,6010,5958],[6011,82.43,195,6012,5958],[6013,86.1,195,6014,5958],[6015,88.11,195,6016,5958],[2986,89.75,195,6017,5958],[6018,89.88,195,6019,5958]]},{"id":"naturebench-full-90-surpass-sota-2026-paper-v2","name":"NatureBench (Full 90, Surpass-SOTA, 2026-07 paper v2)","group":"数学与科学","unit":"%","metric":"Surpass-SOTA (g > 0.1)","protocol":{"en":"All 90 tasks; fraction with a judge-valid SOTA-normalized gap g > 0.1. Four-hour solve budget per task, paused during evaluation; web search disabled; harness-default reasoning and task-matched compute. Claude Sonnet 4.6 audits solution validity. Fixed July 2026 paper v2 results; each result names its harness.","zh":"分母为全部 90 项任务，统计通过有效性审核且相对论文指标差距 g > 0.1 的比例。每题四小时求解预算，评分期间暂停计时；关闭网页搜索，使用 Agent 默认推理设置及按任务分配的算力。Claude Sonnet 4.6 审核解法有效性。固定采用 2026 年 7 月论文 v2 成绩，各结果注明 Agent。"},"source":{"name":"NatureBench authors","url":"https://arxiv.org/abs/2606.24530v2","status":"paper_baselines","published_at":"2026-06-23"},"public_history":true,"results":[[3578,0.178,1,6020,6021],[227,0.156,551,6020,6021],[3415,0.156,83,6022,6021],[3196,0.144,15,6023,6021],[2261,0.122,1,6020,6021],[3198,0.111,775,6020,6021],[3587,0.1,195,6020,6021],[3490,0.089,21,6020,6021],[3184,0.089,15,6023,6021],[3186,0.078,551,6020,6021],[3185,0.044,25,6020,6021],[3190,0.011,775,6020,6021]]},{"id":"deepswe-v1-1-publisher","name":"DeepSWE v1.1","group":"编程","unit":"%","metric":"Pass@1","protocol":{"en":"113 tasks, mini-swe-agent, four runs. Each model uses the reasoning setting selected in the official Best panel. Pass@1 counts successful scored attempts; context limits and timeouts count as failures, while provider, verifier and network errors are excluded. All listed models have scored attempts on every task.","zh":"113 题，使用 mini-swe-agent，运行四轮。每个模型采用官方 Best 面板选出的推理档位。Pass@1 为通过次数占有效尝试次数的比例；上下文超限、超时计失败，供应商、评分器和网络错误剔除。各入榜模型均覆盖全部 113 题。"},"source":{"name":"Datacurve","url":"https://deepswe.datacurve.ai/","status":"current_snapshot","published_at":"2026-06-15"},"results":[[4943,0.7411504424778761,15,6024,-1],[5144,0.738255033557047,4,6025,-1],[5142,0.7364864864864865,1,6026,-1],[4972,0.7266666666666667,15,6027,-1],[3473,0.6991150442477876,1,6028,-1],[4954,0.6895787139689579,48,6029,-1],[4952,0.6851441241685144,21,6030,-1],[6031,0.6747787610619469,143,6032,-1],[5155,0.671875,15,6033,-1],[3208,0.6703539823008849,15,6034,-1],[6035,0.6548672566371682,4,6036,-1],[6037,0.6339285714285714,48,6038,-1],[3209,0.6283185840707964,25,6039,-1],[4950,0.5746102449888641,195,6040,-1],[6041,0.5486725663716814,66,6042,-1],[5107,0.5384615384615384,1,6043,-1],[1321,0.5331858407079646,25,6044,-1],[5143,0.4668141592920354,4,6045,-1],[5104,0.43777777777777777,48,6046,-1],[5109,0.3606194690265487,4,6047,-1]]},{"id":"crm-agent-bench-2026-paper","name":"CRM-Agent-Bench (2026-10, paper results)","group":"专业工作","unit":"%","metric":"Strict task success","public_history":true,"protocol":{"en":"Formal first round: 555 tasks across nine domains, one trial per task. The graph-routed Burr ReAct agent has up to 30 turns and 1,200 seconds. GPT-5-mini serves as simulated user and semantic judge (v6); strict success requires every task-specific check to pass. All 555 tasks were scored for every model.","zh":"论文正式首轮成绩：9 个业务领域、555 题，每题一次。使用 Burr ReAct 多节点代理，最多 30 轮、1,200 秒。GPT-5-mini 模拟用户并判定语义要求（v6）；每题的全部检查均通过才计成功。所有模型均有 555 题的成绩。"},"source":{"name":"Salesforce CRM-Agent-Bench","url":"https://crm-agent-bench.github.io/","status":"paper_baselines","published_at":"2026-10-23"},"results":[[2983,0.836036036036036,4,6048,6049],[755,0.827027027027027,1,6050,6049],[3572,0.8198198198198198,1,6051,6049],[2973,0.8162162162162162,15,6052,6049],[3568,0.8126126126126126,1,6053,6049],[3196,0.8126126126126126,15,6053,6049],[762,0.8126126126126126,15,6053,6049],[3408,0.8072072072072072,15,6054,6049],[6055,0.7963963963963964,3471,6056,6049],[3482,0.790990990990991,4,6057,6049],[6058,0.790990990990991,15,6057,6049],[5320,0.7873873873873873,66,6059,6049],[325,0.7603603603603604,1,6060,6049],[6061,0.6558558558558558,307,6062,6049],[6063,0.6432432432432432,307,6064,6049],[6065,0.6432432432432432,15,6064,6049]]},{"id":"long-horizon-terminal-bench-2-0-publisher","name":"Long-Horizon-Terminal-Bench 2.0","group":"编程","unit":"%","metric":"Solved (R ≥ 0.95)","protocol":{"en":"77 tasks (46 original and 31 new). Terminus-2 runs one three-hour trial per task with a verifier checkpoint every 30 minutes. The LHTB Harbor runner gives only a binary rejection and resumes until success or timeout. A task is solved at reward ≥0.95. Unscored cases and provider refusals contribute zero to the 77-task denominator.","zh":"共 77 题（原版 46 题、新增 31 题）。Terminus-2 每题运行一次，限时 3 小时，每 30 分钟检查一次。LHTB 版 Harbor 只反馈未通过，代理继续执行至通过或超时。奖励达到 0.95 算完成；未评分题和供应商拒绝题在 77 题分母中计零。"},"source":{"name":"Long-Horizon-Terminal-Bench authors","url":"https://github.com/zli12321/LHTB#lhtb-20-results-october-2026","status":"current_snapshot","published_at":"2026-10-12"},"results":[[5404,0.4805194805194805,15,6066,6067],[4048,0.4155844155844156,15,6068,6067],[2977,0.22077922077922077,21,6069,6067],[6070,0.19480519480519481,3033,6071,6067],[2983,0.19480519480519481,4,6072,6067],[3154,0.15584415584415584,48,6073,6067],[3261,0.12987012987012986,25,6074,6067],[5295,0.12987012987012986,143,6074,6067],[3492,0.11688311688311688,25,6075,6067],[227,0.1038961038961039,48,6076,6067],[5361,0.1038961038961039,3026,6076,6067],[3560,0.09090909090909091,195,6077,6067],[3491,0.06493506493506493,775,6078,6067],[3412,0.06493506493506493,4,6079,6067],[3490,0.025974025974025976,21,6080,6067]]},{"id":"thinkingbox-bench-2026-paper","name":"ThinkingBox-Bench (2026-10, paper results)","group":"Agent 与工具","unit":"%","metric":"Pass@1","public_history":true,"protocol":{"en":"Paper v5: 507 workflows, 20 attempts per task (10,140 trials per model), at most 10 simulated-user follow-ups. GPT-5.4-mini is the fixed user and response judge; the main check compares final backend state. Pass@1 counts successful attempts, including API and system errors as failures.","zh":"论文 v5：507 项工作流，每项独立运行 20 次，每个模型共 10,140 次，最多承接模拟用户的 10 轮追问。GPT-5.4-mini 模拟用户并评判回答，主要依据是后端最终状态。Pass@1 为单次尝试成功率，API 和系统错误计失败。"},"source":{"name":"ThinkingBox-Bench authors","url":"https://arxiv.org/abs/2608.19741v5","status":"paper_baselines","published_at":"2026-08-20"},"results":[[755,0.665,1,6081,6082],[3184,0.6536,15,6083,6082],[4048,0.6191,15,6084,6082],[3572,0.5919,1,6085,6082],[2958,0.5831000000000001,15,6086,6082],[3155,0.5737,21,6087,6082],[5300,0.517,195,6088,6082],[340,0.4628,15,6089,6082],[3185,0.4326,25,6090,6082],[3187,0.3766,21,6091,6082],[3186,0.3319,48,6092,6082],[3763,0.32939999999999997,195,6093,6082],[2261,0.3209,1,6094,6082],[6095,0.1931,15,6096,6082],[5411,0.1438,143,6097,6082],[4070,0.0565,195,6098,6082],[6099,0.0466,417,6100,6082],[3005,0.0019,775,6101,6082]]},{"id":"culturalriddles-closed-book-2026-paper","name":"CulturalRiddles (Closed book, 2026-10 paper results)","group":"知识与推理","unit":"%","metric":"Total score-3 accuracy, closed book","protocol":{"en":"Original-language riddles from the 57 communities with human answer evaluation: 5,690 riddles per configuration. Total accuracy is the share of scored answers judged fully correct by the published rules, native speakers or two agreeing model judges. Unresolved answers are excluded from each model’s denominator. All rows are closed book.","zh":"闭卷回答原语言谜语。作者主表取有人工评估的 57 个社区，每个配置对应 5,690 题。正确率为已评分回答中获得满分（3 分）的比例；评分来自规则、母语者或两名意见一致的模型评审。尚未判定的回答从各模型分母中排除。"},"source":{"name":"CulturalRiddles authors and contributors","url":"https://arxiv.org/abs/2610.29968v3","status":"paper_baselines","license":{"name":"CulturalRiddles custom attribution and evaluation-only data terms","url":"https://github.com/Cohere-Labs-Community/CulturalRiddles/blob/084597ac4951d9874fdd41d4de5497d98ae628ec/data/LICENSE"},"published_at":"2026-10-21"},"public_history":true,"results":[[3482,0.794,83,6102,-1],[6103,0.763,15,6104,-1],[6105,0.66,1,6106,-1],[6107,0.638,25,6106,-1],[6108,0.597,1,6102,-1],[6109,0.588,25,6102,-1],[2055,0.501,83,6102,-1],[6110,0.396,83,6102,-1],[3261,0.392,25,6102,-1],[3338,0.385,417,6111,-1],[3708,0.336,83,6102,-1],[6112,0.335,6113,6102,-1],[6114,0.28,195,6102,-1],[6115,0.196,6116,6111,-1],[6117,0.152,6116,6111,-1],[6118,0.15,6119,6111,-1],[4357,0.147,720,6111,-1],[6120,0.113,6121,6102,-1],[6122,0.10400000000000001,195,6111,-1],[6123,0.10400000000000001,6116,6111,-1],[4373,0.083,720,6111,-1],[6124,0.076,6116,6111,-1],[6125,0.07,6126,6111,-1],[6127,0.069,6126,6111,-1],[6128,0.068,6126,6111,-1],[6129,0.068,6126,6111,-1],[6005,0.004,195,6102,-1]]},{"id":"video-ifbench-tisr-2026-paper","name":"Video-IFBench (Overall TISR, 2026-08 paper results)","group":"视觉与视频","unit":"%","metric":"Overall task-gated instruction satisfaction rate (TISR)","protocol":{"en":"A response passes only if it addresses all active video tasks and satisfies every checklist item. Table 1 uses the paper scorer and Qwen3.5-397B-A17B-Instruct for judged items. Gemini and Doubao use 1 frame/s; GPT-5.4 uses 50 frames. Omni models receive video with audio; vision-only models receive frames plus timestamped Gemini-3-Pro transcripts. Open-model inference follows the model’s recommended setup.","zh":"回答必须覆盖视频指令中实际生效的全部任务，并满足每项约束才算通过。主表用论文评分器，语义检查由 Qwen3.5-397B-A17B-Instruct 判定。Gemini 和豆包每秒取 1 帧，GPT-5.4 取 50 帧；全模态模型读取带音频的视频，纯视觉模型读取图像帧及 Gemini-3-Pro 生成的带时间戳字幕。开放模型采用各自推荐的推理设置。"},"source":{"name":"Video-IFBench authors","url":"https://arxiv.org/abs/2608.25529v1","status":"paper_baselines","published_at":"2026-08-26"},"public_history":true,"results":[[6130,0.545,83,6131,6132],[6133,0.495,83,6131,6132],[6134,0.408,3026,6131,6132],[3184,0.3,15,6135,6132],[6136,0.086,195,6137,6132],[6138,0.135,195,6137,6132],[6139,0.18,195,6137,6132],[6140,0.195,83,6141,6132],[6142,0.297,83,6141,6132],[6143,0.354,83,6141,6132],[6144,0.16,3246,6141,6132],[6145,0.166,3246,6141,6132],[6146,0.185,3246,6141,6132],[6147,0.20800000000000002,3246,6141,6132],[6148,0.22699999999999998,3246,6141,6132],[6149,0.16399999999999998,195,6141,6132],[6150,0.196,195,6141,6132],[5485,0.18600000000000003,195,6141,6132],[2613,0.258,195,6141,6132],[6151,0.17300000000000001,195,6141,6132],[6152,0.196,195,6141,6132],[6153,0.252,195,6141,6132],[6154,0.226,195,6141,6132],[6155,0.253,195,6141,6132],[6156,0.304,195,6141,6132],[6157,0.262,195,6158,6132],[6159,0.22,195,6160,6132],[6161,0.335,195,6160,6132],[6162,0.165,3246,6160,6132],[6163,0.18100000000000002,3246,6160,6132],[6164,0.187,3246,6160,6132],[6165,0.191,3246,6160,6132],[6166,0.223,3246,6160,6132],[6167,0.322,195,6160,6132],[6168,0.375,195,6160,6132],[6169,0.37200000000000005,195,6160,6132],[6170,0.41700000000000004,195,6160,6132],[6171,0.461,195,6160,6132]]},{"id":"science-last-exam","name":"Science’s Last Exam (v1)","group":"数学与科学","unit":"score","score_max":100,"score_decimals":2,"saturation_excluded":true,"scale_label":{"en":"Composite / 100","zh":"综合分 / 100"},"metric":"Six-assessment composite","protocol":{"en":"Each of the six assessments contributes one sixth of the v1 composite score. All eight models scored zero on the end-to-end research task.","zh":"v1 综合分取六项评测得分的算术平均值。八个模型在端到端研究任务上均得零分。"},"source":{"name":"SLE Team","url":"https://sciences-last-exam.org/leaderboard","status":"current_snapshot","published_at":"2026-11-04"},"results":[[755,37.50000000000001,1,6172,-1],[2960,36.15333333333333,15,6173,-1],[2967,28.923333333333332,21,3759,-1],[3512,26.825000000000003,25,6172,-1],[5335,26.401666666666667,2972,3759,-1],[2982,23.296666666666667,195,3759,-1],[2980,21.333333333333332,551,6172,-1],[3198,16.823333333333334,775,6172,-1]]},{"id":"tandembench-decision-accuracy-2026-paper","name":"TandemBench (decision accuracy, 2026 paper)","group":"知识与推理","metric":"Decision accuracy","unit":"%","public_history":true,"protocol":{"en":"Decision accuracy on 1,200 multi-agent cooking questions. Each of 40 native decision models chooses a team plan; the published score averages four option rotations. General-purpose language models in Table 2 are a separate cohort and are not mixed in.","zh":"1,200 道多智能体协作选题上的决策准确率。40 个决策模型从表内候选团队方案中作答，分数为四个选项轮换的平均。表 2 的通用语言模型是另一组，不并入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.05827v1","status":"paper_baselines","published_at":"2026-11-04"},"results":[[6174,0.625,370,6175,6176],[6177,0.6075,370,6175,6176],[6178,0.6075,370,6175,6176],[6179,0.6038,370,6175,6176],[6180,0.6038,370,6175,6176],[6181,0.5943999999999999,370,6175,6176],[6182,0.5893999999999999,370,6175,6176],[6183,0.5869,370,6175,6176],[6184,0.5869,370,6175,6176],[6185,0.5813,370,6175,6176],[6186,0.5744,370,6175,6176],[6187,0.5556,370,6175,6176],[6188,0.5294,370,6175,6176],[6189,0.5294,370,6175,6176],[6190,0.5275,370,6175,6176],[6191,0.5219,370,6175,6176],[6192,0.5063,370,6175,6176],[6193,0.505,370,6175,6176],[6194,0.5025,370,6175,6176],[6195,0.4863,370,6175,6176],[6196,0.475,370,6175,6176],[6197,0.4744,370,6175,6176],[6198,0.4738,370,6175,6176],[6199,0.4719,370,6175,6176],[6200,0.4675,370,6175,6176],[6201,0.4631,370,6175,6176],[6202,0.4425,370,6175,6176],[6203,0.4388,370,6175,6176],[6204,0.435,370,6175,6176],[5546,0.4094,370,6175,6176],[6205,0.4069,370,6175,6176],[6206,0.3944,370,6175,6176],[6207,0.3906,370,6175,6176],[6208,0.39,370,6175,6176],[5548,0.35810000000000003,370,6175,6176],[6209,0.3456,370,6175,6176],[6210,0.2944,370,6175,6176],[6211,0.2588,370,6175,6176],[6212,0.2181,370,6175,6176],[6213,0.15439999999999998,370,6175,6176]]},{"id":"abench-judge-average-2026-paper","name":"ABench (judge average, 2026 paper)","group":"知识与推理","metric":"Judge average","unit":"score","public_history":true,"protocol":{"en":"Mean score assigned by the paper’s judge over valid ABench samples. Macro-average, invalid counts and score tails are not substituted for this column.","zh":"论文裁判对 ABench 有效样本给出的平均分。不改用宏平均、无效样本数或高分尾部。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.27611v1","status":"paper_baselines","published_at":"2026-10-20"},"score_max":100,"saturation_excluded":true,"score_decimals":2,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"results":[[6214,86.26,370,6175,6215],[2288,56.08,370,6175,6215],[3008,47.04,370,6175,6215],[6216,43.87,370,6175,6215],[3748,41.72,370,6175,6215],[769,39.05,370,6175,6215],[6217,35.77,370,6175,6215],[3002,31.92,370,6175,6215],[6218,31.22,370,6175,6215],[3016,30.98,370,6175,6215],[3009,29.42,370,6175,6215],[6219,29.03,370,6175,6215],[793,25.55,370,6175,6215],[6220,24.92,370,6175,6215],[6221,23.78,370,6175,6215],[6222,23.41,370,6175,6215]]},{"id":"slackbench-overall-2026-paper","name":"SlackBench (overall accuracy, 2026 paper)","group":"Agent 与工具","metric":"Overall accuracy","unit":"%","public_history":true,"protocol":{"en":"Unweighted overall accuracy on SlackBench. The same model may appear more than once when the paper reports a different harness; the harness stays in the configuration and does not create another model.","zh":"SlackBench 各类查询的未加权总准确率。同一模型的不同 harness 分列，记在配置中，不另算一个模型。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.05512v1","status":"paper_baselines","published_at":"2026-11-03"},"results":[[2962,0.8025,370,6223,6224],[3732,0.7373999999999999,370,6225,6224],[2956,0.7367,370,6226,6224],[2959,0.7265,370,6226,6224],[3732,0.7071999999999999,370,6226,6224],[3515,0.6596,370,6226,6224],[2968,0.6547,370,6227,6224],[2979,0.6362,370,6225,6224],[6228,0.6112,370,6229,6224],[5508,0.6048,370,6229,6224],[5506,0.5819,370,6229,6224],[3578,0.5649000000000001,370,6226,6224],[3196,0.5189,370,6230,6224],[3196,0.48310000000000003,370,6231,6224],[3482,0.4628,370,6232,6224],[3572,0.45909999999999995,370,6226,6224],[2261,0.4587,370,6230,6224],[2261,0.4578,370,6231,6224],[3196,0.45520000000000005,370,6229,6224],[3490,0.446,370,6226,6224],[2261,0.4169,370,6226,6224],[3482,0.4052,370,6231,6224],[3492,0.3978,370,6226,6224],[3482,0.38799999999999996,370,6230,6224],[3572,0.3868,370,6231,6224],[6233,0.3822,370,6226,6224],[3186,0.38079999999999997,370,6226,6224],[3184,0.37439999999999996,370,6229,6224],[6234,0.374,370,6229,6224],[4033,0.31129999999999997,370,6232,6224],[325,0.2751,370,6226,6224],[6235,0.2704,370,6229,6224],[6235,0.1282,370,6231,6224]]},{"id":"plan-bench-overall-2026-paper","name":"Plan-Bench (overall, 2026 paper)","group":"知识与推理","metric":"Overall accuracy","unit":"%","public_history":true,"protocol":{"en":"Overall accuracy across Plan-Bench spatial, temporal and multi-step questions. The random baseline is omitted. Category scores are not substituted for the overall column.","zh":"Plan-Bench 空间、时间与多步题目的总准确率。不含随机基线，也不用分类分数代替总列。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.18836v1","status":"paper_baselines","published_at":"2026-10-13"},"results":[[3157,0.75,370,6175,6236],[3155,0.7020000000000001,370,6175,6236],[4048,0.6579999999999999,370,6175,6236],[3194,0.514,370,6175,6236],[6237,0.46399999999999997,370,6175,6236],[5300,0.622,370,6175,6236],[2584,0.5589999999999999,370,6175,6236],[2590,0.495,370,6175,6236],[6143,0.435,370,6175,6236],[6238,0.2,370,6175,6236],[5534,0.17600000000000002,370,6175,6236],[4070,0.49200000000000005,370,6175,6236],[6239,0.359,370,6175,6236],[6240,0.285,370,6175,6236],[6241,0.27,370,6175,6236],[2654,0.196,370,6175,6236],[2646,0.196,370,6175,6236],[6242,0.18600000000000003,370,6175,6236],[2632,0.183,370,6175,6236],[6243,0.179,370,6175,6236],[6150,0.179,370,6175,6236],[6244,0.163,370,6175,6236],[2900,0.156,370,6175,6236],[4969,0.43200000000000005,370,6175,6236],[5828,0.20199999999999999,370,6175,6236],[6245,0.132,370,6175,6236],[6246,0.126,370,6175,6236],[6247,0.083,370,6175,6236]]},{"id":"game2real-overall-2026-paper","name":"Game2Real-Bench (overall, 2026 paper)","group":"视觉与视频","metric":"Overall score","unit":"%","public_history":true,"protocol":{"en":"Overall score, in percent, on Game2Real-Bench. Paired toy-game and real-world cells in the capability columns are not averaged or substituted for the published overall.","zh":"Game2Real-Bench 的总分（百分数）。能力列里成对的玩具游戏与真实场景分数不另作平均，也不代替已发表的总分。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.18834v1","status":"paper_baselines","published_at":"2026-10-13"},"results":[[4979,0.698,370,6175,6248],[6237,0.596,370,6175,6248],[6249,0.537,370,6175,6248],[5300,0.526,370,6175,6248],[2590,0.45,370,6175,6248],[2584,0.449,370,6175,6248],[6250,0.385,370,6175,6248],[5485,0.218,370,6175,6248],[5534,0.21,370,6175,6248],[4070,0.35700000000000004,370,6175,6248],[6239,0.242,370,6175,6248],[6251,0.23399999999999999,370,6175,6248],[6241,0.222,370,6175,6248],[2646,0.20199999999999999,370,6175,6248],[6252,0.18899999999999997,370,6175,6248],[2654,0.188,370,6175,6248],[6253,0.185,370,6175,6248],[6150,0.18,370,6175,6248],[6254,0.172,370,6175,6248],[6242,0.17,370,6175,6248],[4969,0.317,370,6175,6248],[5828,0.193,370,6175,6248],[6247,0.132,370,6175,6248],[6245,0.126,370,6175,6248],[6246,0.125,370,6175,6248]]},{"id":"cyberarena-expected-score-2026-paper","name":"CyberArena (expected score, 2026 paper)","group":"安全与可靠性","metric":"Posterior mean expected score","unit":"score","public_history":true,"protocol":{"en":"Posterior mean expected score on the 0–100 CyberArena scale. The harness is part of the configuration. Costs and credible intervals are not used as the score.","zh":"CyberArena 0–100 尺度上的后验期望分数。Harness 记在配置中。费用和可信区间不作为分数。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.19005v1","status":"paper_baselines","published_at":"2026-10-13"},"score_max":100,"saturation_excluded":true,"score_decimals":2,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"results":[[762,78.6,370,6255,6256],[755,73.8,370,6226,6256],[3154,68.5,370,6257,6256],[3386,67.1,370,6232,6256],[2974,65.5,370,6258,6256],[3479,65.0,370,6226,6256],[2982,62.3,370,6259,6256],[3415,60.6,370,6232,6256],[3539,59.5,370,6260,6256],[3491,59.1,370,6257,6256],[2977,57.0,370,6261,6256],[5665,56.3,370,6257,6256]]},{"id":"declinecraft-accuracy-2026-paper","name":"DeclineCraft (accuracy, 2026 paper)","group":"知识与推理","metric":"Accuracy","unit":"%","public_history":true,"protocol":{"en":"Accuracy under the paper’s single-shot hybrid retrieval setting with k=5. The point estimate is used and the published interval is not. A thinking row of the same model stays a configuration of that model.","zh":"论文单次混合检索、k=5 设置下的准确率。采用点估计，不用区间。同一模型的 thinking 行记为该模型的配置。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.28340v1","status":"paper_baselines","published_at":"2026-10-20"},"results":[[6262,0.693,370,6175,6263],[6262,0.696,370,5600,6263],[6264,0.695,370,6175,6263],[6264,0.72,370,5600,6263],[4043,0.535,370,6175,6263],[4041,0.625,370,6175,6263],[4039,0.649,370,6175,6263],[796,0.6809999999999999,370,6175,6263],[6265,0.687,370,6175,6263],[787,0.648,370,6175,6263],[787,0.7240000000000001,370,5600,6263],[6266,0.628,370,6175,6263],[6267,0.601,370,6175,6263],[6267,0.643,370,5600,6263],[6268,0.644,370,6175,6263],[6268,0.685,370,5600,6263],[6269,0.65,370,6175,6263],[6269,0.675,370,5600,6263],[6270,0.643,370,6175,6263],[6270,0.693,370,5600,6263],[1513,0.662,370,6175,6263],[1513,0.706,370,5600,6263],[6271,0.684,370,6175,6263],[6271,0.7190000000000001,370,5600,6263],[6272,0.72,370,6175,6263],[6114,0.772,370,6175,6263],[6114,0.763,370,5600,6263],[6273,0.75,370,6175,6263],[6273,0.726,370,5600,6263],[6274,0.7559999999999999,370,6175,6263],[6274,0.7559999999999999,370,5600,6263],[6275,0.7290000000000001,370,6175,6263],[6276,0.608,370,6175,6263],[3756,0.7390000000000001,370,6175,6263],[6277,0.7120000000000001,370,6175,6263],[6278,0.716,370,6175,6263],[2055,0.568,370,6175,6263],[2055,0.6970000000000001,370,5600,6263],[6279,0.632,370,6175,6263],[3424,0.762,370,6175,6263]]},{"id":"interactivebench-pass-2026-paper","name":"InteractiveBench (pass rate, 2026 paper)","group":"Agent 与工具","metric":"Average pass rate","unit":"%","public_history":true,"protocol":{"en":"Average pass rate: the share of instances in which every objective is met. Mean completion rate and per-category columns are not substituted.","zh":"平均通过率，即全部目标都达成的题目比例。不改用平均完成率或分类列。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.33966v1","status":"paper_baselines","published_at":"2026-10-25"},"results":[[2956,0.41700000000000004,370,6175,6280],[755,0.33299999999999996,370,6175,6280],[3413,0.317,370,6175,6280],[762,0.308,370,6175,6280],[3196,0.3,370,6175,6280],[3529,0.3,370,6175,6280],[3479,0.28300000000000003,370,6175,6280],[2977,0.242,370,6175,6280],[3560,0.225,370,6175,6280],[3412,0.20800000000000002,370,6175,6280],[3408,0.2,370,6175,6280],[3512,0.125,370,6175,6280]]},{"id":"ruletrek-average-2026-paper","name":"RuleTrek (average, 2026 paper)","group":"知识与推理","metric":"Average accuracy","unit":"%","public_history":true,"protocol":{"en":"Average accuracy, in percent, across RuleTrek’s six domains and its identification, deduction and end-to-end tasks. Domain columns are not substituted for the average.","zh":"RuleTrek 六个领域、识别、推导与端到端任务的平均准确率（百分数）。不用领域列代替平均列。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.19698v1","status":"paper_baselines","published_at":"2026-10-14"},"results":[[5618,0.6247,370,6175,6281],[5410,0.6186,370,6175,6281],[3153,0.5921,370,6175,6281],[3185,0.5825,370,6175,6281],[3396,0.5653,370,6175,6281],[5906,0.5527000000000001,370,6175,6281],[3196,0.602,370,6175,6281],[5899,0.5502,370,6175,6281],[3158,0.5097,370,6175,6281],[5617,0.5085999999999999,370,6175,6281],[766,0.3335,370,6175,6281]]},{"id":"featureloop-fdac-2026-paper","name":"FeatureLoop (FDAC, 2026 paper)","group":"编程","metric":"FDAC","unit":"%","public_history":true,"protocol":{"en":"FDAC as published in the main FeatureLoop table, in percent. StrictPass and message counts are not used. The pooled row is not a model.","zh":"FeatureLoop 主表发表的 FDAC（百分数）。不采用 StrictPass 或消息数。合计行不是模型。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.35488v1","status":"paper_baselines","published_at":"2026-10-26"},"results":[[2958,0.465,370,6175,6282],[762,0.45399999999999996,370,6175,6282],[2977,0.442,370,6175,6282],[5728,0.42700000000000005,370,6175,6282],[3261,0.41200000000000003,370,6175,6282],[3512,0.408,370,6175,6282],[3515,0.405,370,6175,6282],[5598,0.373,370,6175,6282],[3529,0.37200000000000005,370,6175,6282],[2974,0.371,370,6175,6282],[3492,0.36700000000000005,370,6175,6282],[5114,0.212,370,6175,6282]]},{"id":"verifierbench-ba-norm-2026-paper","name":"VerifierBench (balanced-accuracy norm, 2026 paper)","group":"Agent 与工具","metric":"Balanced-accuracy norm","unit":"%","public_history":true,"protocol":{"en":"Balanced-accuracy norm, TPR + TNR − 100, for the common Pi harness on the same 222 tasks. True-positive and true-negative rates are not entered as separate rankings. This index stays outside percentage saturation.","zh":"同一 Pi harness、222 道题上的平衡准确率指数，即真正例率加真负例率再减 100。真正例率和真负例率不另开排名。该指数不进入百分制饱和分档。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.28679v1","status":"paper_baselines","published_at":"2026-10-19"},"saturation_excluded":true,"scale_label":{"en":"Balanced-accuracy norm","zh":"平衡准确率指数"},"results":[[5404,0.621,370,6175,6283],[3479,0.54,370,6175,6283],[2977,0.534,370,6175,6283],[4048,0.529,370,6175,6283],[5298,0.529,370,6175,6283],[227,0.488,370,6175,6283],[5361,0.46799999999999997,370,6175,6283],[3196,0.466,370,6175,6283],[3492,0.44799999999999995,370,6175,6283],[3261,0.405,370,6175,6283],[5734,0.402,370,6175,6283],[3415,0.35700000000000004,370,6175,6283],[3490,0.35200000000000004,370,6175,6283],[6284,0.297,370,6175,6283],[5533,0.268,370,6175,6283],[4225,0.228,370,6175,6283],[3748,0.222,370,6175,6283],[2602,0.142,370,6175,6283],[5128,0.106,370,6175,6283]]},{"id":"frontierphysics-pass1-2026-paper","name":"FrontierPhysics (Pass@1, 2026 paper)","group":"数学与科学","metric":"Pass@1","unit":"%","public_history":true,"protocol":{"en":"Pass@1 estimate over 56 tasks and 3 trials. Table 7 rows with any slot that had no rollout or incomplete scoring are omitted, because those missing runs are folded into the rate. Gemini 3.8 Flash and Mistral Large 4 remain: Table 7 records zero such slots, so their published 0 is an observed pass rate. Effort levels stay configurations of the same model.","zh":"56 题、每题 3 次的 Pass@1。表 7 中只要有缺失运行或未完成评分的配置就整行不收，因为缺失被折进了通过率。Gemini 3.8 Flash 与 Mistral Large 4 保留：表 7 记其缺失格为 0，故 0 分是实际通过率。不同推理档位记为同一模型的配置。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.06910v1","status":"paper_baselines","published_at":"2026-11-04"},"results":[[2959,0.27399999999999997,370,6285,6286],[2958,0.23199999999999998,370,6287,6286],[2958,0.196,370,6288,6286],[2960,0.196,370,6285,6286],[2964,0.14300000000000002,370,6285,6286],[3394,0.14300000000000002,370,6285,6286],[2956,0.113,370,6285,6286],[755,0.095,370,6285,6286],[2962,0.083,370,6285,6286],[762,0.071,370,6287,6286],[2967,0.065,370,6285,6286],[2968,0.042,370,6285,6286],[2970,0.042,370,6285,6286],[762,0.042,370,6288,6286],[3154,0.042,370,6285,6286],[2973,0.036000000000000004,370,6287,6286],[3408,0.036000000000000004,370,6287,6286],[2977,0.024,370,6285,6286],[2974,0.024,370,6285,6286],[2971,0.012,370,6285,6286],[2973,0.012,370,6288,6286],[3492,0.006,370,6285,6286],[2983,0.0,370,6285,6286],[2992,0.0,370,6285,6286]]},{"id":"mcphallu-overall-2026-paper","name":"MCPHallu (overall, 2026 paper)","group":"Agent 与工具","metric":"Overall","unit":"%","public_history":true,"protocol":{"en":"Overall score on a published 0–1 scale for 14 LLMs. Subtype and difficulty columns are not used. The stored value is the fraction printed in Table 5.","zh":"14 个语言模型在表 5 的总分，论文刻度是 0–1。不采用分项或难度列。存入的是表中的分数本身。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.07689v1","status":"paper_baselines","published_at":"2026-11-05"},"results":[[353,0.599,370,6175,6289],[6290,0.659,370,6175,6289],[283,0.71,370,6175,6289],[2261,0.728,370,6175,6289],[3738,0.696,370,6175,6289],[3572,0.702,370,6175,6289],[325,0.639,370,6175,6289],[3424,0.583,370,6175,6289],[6291,0.692,370,6175,6289],[6292,0.655,370,6175,6289],[6293,0.606,370,6175,6289],[3191,0.589,370,6175,6289],[3186,0.539,370,6175,6289],[3187,0.595,370,6175,6289]]},{"id":"zhenhuanbench-all-2026-paper","name":"ZhenhuanBench (public-set average, 2026 paper)","group":"知识与推理","metric":"All","unit":"%","public_history":true,"protocol":{"en":"Mean score over T1–T4 on the public set, in percent. Both panels of Table 1 are included, including the paper’s 8B reference row. Task columns are not substituted for All.","zh":"公开集上 T1 至 T4 的平均分（百分数）。表 1 左右两栏都收录，包括论文的 8B 参照行。不用分项列代替 All。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.26059v1","status":"paper_baselines","published_at":"2026-10-19"},"results":[[2977,0.6679999999999999,370,6175,6294],[6295,0.342,370,6175,6294],[3492,0.625,370,6175,6294],[6241,0.332,370,6175,6294],[3187,0.62,370,6175,6294],[4969,0.32899999999999996,370,6175,6294],[5480,0.596,370,6175,6294],[2728,0.318,370,6175,6294],[3201,0.568,370,6175,6294],[6296,0.311,370,6175,6294],[6297,0.524,370,6175,6294],[6298,0.303,370,6175,6294],[5722,0.474,370,6175,6294],[787,0.301,370,6175,6294],[6299,0.444,370,6175,6294],[2904,0.294,370,6175,6294],[786,0.441,370,6175,6294],[6300,0.28300000000000003,370,6175,6294],[4070,0.41600000000000004,370,6175,6294],[2754,0.28300000000000003,370,6175,6294],[6301,0.391,370,6175,6294],[5828,0.26,370,6175,6294],[2958,0.735,370,6175,6294],[2058,0.523,370,6175,6294],[3386,0.723,370,6175,6294],[353,0.493,370,6175,6294],[3157,0.625,370,6175,6294],[5018,0.444,370,6175,6294],[5016,0.589,370,6175,6294],[4064,0.381,370,6175,6294],[5021,0.586,370,6175,6294],[354,0.308,370,6175,6294]]},{"id":"utilitybench-answer-accuracy-2026-paper","name":"UtilityBench (answer accuracy, 2026 paper)","group":"知识与推理","metric":"Answer accuracy","unit":"%","public_history":true,"protocol":{"en":"Answer accuracy under the shared Refusal Hint, binary scores and CR = 2. Family labels in the table are part of the model name. UtilityAUC and refusal accuracy are not used.","zh":"统一 Refusal Hint、二元计分、CR=2 下的回答准确率。表中的系列名并入模型名。不采用 UtilityAUC 或拒绝准确率。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.25924v1","status":"paper_baselines","published_at":"2026-10-19"},"results":[[2058,0.6083,370,6302,6303],[3748,0.6759999999999999,370,6302,6303],[3494,0.5979,370,6302,6303],[3408,0.6132,370,6302,6303],[762,0.7497,370,6302,6303],[6304,0.7191,370,6302,6303],[6305,0.6061,370,6302,6303],[6306,0.6172,370,6302,6303],[325,0.6598999999999999,370,6302,6303],[3568,0.7003,370,6302,6303],[4425,0.5467,370,6302,6303],[6307,0.5776,370,6302,6303],[6308,0.5571,370,6302,6303],[6309,0.6095,370,6302,6303],[6310,0.6779000000000001,370,6302,6303],[268,0.5891,370,6302,6303],[227,0.6284000000000001,370,6302,6303],[3601,0.6049,370,6302,6303],[3593,0.6459,370,6302,6303],[3490,0.6744,370,6302,6303]]},{"id":"m261102536-2026-paper","name":"FORGE (Overall-Core, 2026 paper)","group":"编程","metric":"Overall-Core","unit":"score","public_history":true,"protocol":{"en":"Overall-Core is the published mean of six automatic metrics on the fixed task suite. Table dashes denote zero credit under Appendix D.5. This composite is kept on its native 0–100 scale.","zh":"Overall-Core 是固定题集上六项自动指标的原表均值。附录 D.5 将表内横线定义为零分。此综合分保留原生 0–100 刻度。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.02536v1","status":"paper_baselines","published_at":"2026-11-02"},"score_max":100,"saturation_excluded":true,"score_decimals":2,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"results":[[6311,20.31,370,6175,6312],[5535,11.17,370,6175,6312],[6313,9.53,370,6175,6312],[6314,19.7,370,6175,6312],[6315,20.3,370,6175,6312],[562,10.48,370,6175,6312],[3494,28.22,370,6175,6312],[3572,30.86,370,6175,6312],[3189,23.43,370,6175,6312],[3415,46.44,370,6175,6312],[6316,28.56,370,6175,6312],[6317,14.51,370,6175,6312],[6318,7.46,370,6175,6312],[6319,72.75,370,6175,6312]]},{"id":"m261102017-2026-paper","name":"Switch-SWE (Overall, 2026 paper)","group":"编程","metric":"Overall","unit":"%","public_history":true,"protocol":{"en":"Table 1 task success in Uninterrupted and Switch conditions. Harness and reasoning effort are retained for each configuration.","zh":"表 1 的 Uninterrupted 和 Switch 任务成功率。每个配置保留智能体框架和推理档位。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.02017v1","status":"paper_baselines","published_at":"2026-11-01"},"results":[[3153,0.465,370,6320,6321],[3153,0.315,370,6322,6321],[3154,0.49,370,6320,6321],[3154,0.43,370,6322,6321],[2977,0.48,370,6323,6321],[2977,0.345,370,6324,6321],[6325,0.41,370,6326,6321],[6325,0.28,370,6327,6321],[6328,0.525,370,6326,6321],[6328,0.38,370,6327,6321],[6329,0.535,370,6330,6321],[6329,0.415,370,6331,6321],[3386,0.295,370,6332,6321],[3386,0.23,370,6333,6321]]},{"id":"m261101145-2026-paper","name":"CivicBench (Exact match, 2026 paper)","group":"知识与推理","metric":"Exact match","unit":"%","public_history":true,"protocol":{"en":"Exact match on the same 400 tasks for nine models, with ReAct and TSM as configurations. Printed zeros are measured outcomes of completed runs. Mean task score is a separate metric.","zh":"九个模型在同一套 400 题上的精确匹配率，ReAct 和 TSM 是配置。表上的 0 是已完成运行的实测结果。平均任务分是另一指标。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.01145v1","status":"paper_baselines","published_at":"2026-10-31"},"results":[[3261,0.0,370,6334,6335],[3261,0.005,370,6336,6335],[4042,0.01,370,6334,6335],[4042,0.015,370,6336,6335],[6337,0.0025,370,6334,6335],[6337,0.0,370,6336,6335],[6338,0.0075,370,6334,6335],[6338,0.0075,370,6336,6335],[3764,0.0,370,6334,6335],[3764,0.0,370,6336,6335],[227,0.0,370,6334,6335],[227,0.0,370,6336,6335],[3203,0.0,370,6334,6335],[3203,0.005,370,6336,6335],[5298,0.0,370,6334,6335],[5298,0.0,370,6336,6335],[3568,0.0075,370,6334,6335],[3568,0.0,370,6336,6335]]},{"id":"m261036384-2026-paper","name":"SOMA (Avg., 2026 paper)","group":"知识与推理","metric":"Avg.","unit":"%","public_history":true,"protocol":{"en":"Table 2 Avg. across SOMA tasks for all 37 named model checkpoints. CBS is one component and is not substituted for Avg. Human and random rows are controls.","zh":"表 2 中 37 个具名模型检查点的 Avg. 总平均。CBS 是其中一项，不替代总平均。人类和随机行作为对照保留在原表。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.36384v1","status":"paper_baselines","published_at":"2026-10-26"},"results":[[3568,0.40299999999999997,370,6175,6339],[3479,0.44799999999999995,370,6175,6339],[3408,0.434,370,6175,6339],[3748,0.40299999999999997,370,6175,6339],[3415,0.47,370,6175,6339],[2602,0.469,370,6175,6339],[2055,0.348,370,6175,6339],[5300,0.47,370,6175,6339],[3763,0.5,370,6175,6339],[6340,0.317,370,6175,6339],[5828,0.327,370,6175,6339],[4969,0.431,370,6175,6339],[4070,0.44299999999999995,370,6175,6339],[2584,0.495,370,6175,6339],[6341,0.311,370,6175,6339],[6298,0.341,370,6175,6339],[6342,0.371,370,6175,6339],[6343,0.442,370,6175,6339],[6344,0.309,370,6175,6339],[6246,0.33399999999999996,370,6175,6339],[2687,0.33,370,6175,6339],[2654,0.33399999999999996,370,6175,6339],[6345,0.382,370,6175,6339],[2642,0.39,370,6175,6339],[4968,0.348,370,6175,6339],[5480,0.431,370,6175,6339],[6241,0.354,370,6175,6339],[5722,0.38799999999999996,370,6175,6339],[6346,0.325,370,6175,6339],[6347,0.363,370,6175,6339],[5812,0.368,370,6175,6339],[5725,0.406,370,6175,6339],[6348,0.311,370,6175,6339],[6349,0.32299999999999995,370,6175,6339],[6350,0.308,370,6175,6339],[6351,0.33299999999999996,370,6175,6339],[6352,0.358,370,6175,6339]]},{"id":"m261032429-2026-paper","name":"Async-RBench (Task success, 2026 paper)","group":"Agent 与工具","metric":"Task success","unit":"%","public_history":true,"protocol":{"en":"Table 4 task success for nine models under both Batched and Async scheduling, recorded as configurations.","zh":"表 4 九个模型在 Batched 和 Async 调度下的任务成功率，两种调度记为配置。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.32429v1","status":"paper_baselines","published_at":"2026-10-23"},"results":[[3568,0.423,370,6353,6354],[3568,0.382,370,6355,6354],[2973,0.413,370,6353,6354],[2973,0.38799999999999996,370,6355,6354],[5734,0.213,370,6353,6354],[5734,0.187,370,6355,6354],[6356,0.255,370,6353,6354],[6356,0.20800000000000002,370,6355,6354],[3408,0.402,370,6353,6354],[3408,0.34299999999999997,370,6355,6354],[3492,0.35700000000000004,370,6353,6354],[3492,0.34700000000000003,370,6355,6354],[6114,0.19699999999999998,370,6353,6354],[6114,0.17,370,6355,6354],[244,0.23199999999999998,370,6353,6354],[244,0.192,370,6355,6354],[6357,0.183,370,6353,6354],[6357,0.12300000000000001,370,6355,6354]]},{"id":"m261028622-2026-paper","name":"MindView (Overall, 2026 paper)","group":"视觉与视频","metric":"Overall","unit":"score","public_history":true,"protocol":{"en":"Published Overall for all twelve named checkpoints. The VLM-judge Quality component is not the overall score.","zh":"原表十二个具名检查点的 Overall。VLM 裁判的 Quality 是分项。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.28622v1","status":"paper_baselines","published_at":"2026-10-20"},"score_max":100,"saturation_excluded":true,"score_decimals":2,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"results":[[6358,40.0,370,6175,6359],[6360,49.3,370,6175,6359],[6361,47.9,370,6175,6359],[6362,45.0,370,6175,6359],[6363,40.0,370,6175,6359],[6243,43.7,370,6175,6359],[6364,51.3,370,6175,6359],[6365,49.7,370,6175,6359],[6366,53.4,370,6175,6359],[6367,53.3,370,6175,6359],[6368,57.9,370,6175,6359],[6369,64.4,370,6175,6359]]},{"id":"m261026394-2026-paper","name":"PACE (Success rate, 2026 paper)","group":"Agent 与工具","metric":"Success rate","unit":"%","public_history":true,"protocol":{"en":"Table 2 success rate on all 43 cases. Harnesses are configurations. Qwen3.8-27B completed the evaluation with zero successes.","zh":"表 2 在全部 43 个案例上的成功率。智能体框架记为配置。Qwen3.8-27B 已完成评测，成功数为 0。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.26394v1","status":"paper_baselines","published_at":"2026-10-19"},"results":[[4979,0.14,370,5302,6370],[227,0.09300000000000001,370,5302,6370],[3202,0.07,370,5302,6370],[3187,0.047,370,5302,6370],[5014,0.07,370,5302,6370],[6237,0.023,370,5302,6370],[3185,0.09300000000000001,370,5302,6370],[3190,0.023,370,5302,6370],[5300,0.0,370,5302,6370],[4979,0.14,370,2300,6370],[3202,0.11599999999999999,370,6371,6370]]},{"id":"m261025096-2026-paper","name":"RetailBridge (average accuracy, 2026 paper)","group":"知识与推理","metric":"Average accuracy","unit":"%","public_history":true,"protocol":{"en":"Avg. Acc for all ten named models. The LLM-judged column and the Level-2 heterogeneous subset remain separate metrics.","zh":"原表十个具名模型的 Avg. Acc。LLM 裁判列和 Level-2 异构子集属于其他指标。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.25096v1","status":"paper_baselines","published_at":"2026-10-18"},"results":[[3764,0.169,370,6372,6373],[2998,0.193,370,6372,6373],[6374,0.333,370,6372,6373],[6375,0.298,370,6372,6373],[3201,0.302,370,6372,6373],[3185,0.319,370,6372,6373],[6376,0.246,370,6372,6373],[6377,0.302,370,6372,6373],[3189,0.349,370,6372,6373],[3184,0.345,370,6372,6373]]},{"id":"m261020731-2026-paper","name":"WorldCast (Overall, 2026 paper)","group":"视觉与视频","metric":"Overall","unit":"score","public_history":true,"protocol":{"en":"Published Overall for all twelve named models. Adaptive Task Completion is a component.","zh":"原表十二个具名模型的 Overall。Adaptive Task Completion 是分项。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.20731v1","status":"paper_baselines","published_at":"2026-10-14"},"score_max":100,"saturation_excluded":true,"score_decimals":2,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"results":[[6243,20.1,370,6175,6378],[6362,12.3,370,6175,6378],[6358,6.8,370,6175,6378],[6379,5.6,370,6175,6378],[6380,72.8,370,6175,6378],[6368,66.2,370,6175,6378],[6366,42.8,370,6175,6378],[6381,33.6,370,6175,6378],[6382,16.0,370,6175,6378],[6383,47.2,370,6175,6378],[6384,41.6,370,6175,6378],[6385,39.9,370,6175,6378]]},{"id":"m261020690-2026-paper","name":"ImpactBench (overall positive pass rate, 2026 paper)","group":"Agent 与工具","metric":"Overall positive pass rate","unit":"%","public_history":true,"protocol":{"en":"Table 1 Overall positive-metric pass rate for all ten named models. Negative-metric pass rate is a different measure and is retained in the audit.","zh":"表 1 十个具名模型的 Overall 正向指标通过率。负向指标通过率属于另一指标，保留在审计记录中。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.20690v1","status":"paper_baselines","published_at":"2026-10-14"},"results":[[2973,0.7509999999999999,370,6386,6387],[3408,0.7490000000000001,370,6386,6387],[3568,0.73,370,6386,6387],[755,0.715,370,6386,6387],[2974,0.705,370,6386,6387],[3386,0.685,370,6386,6387],[2986,0.6779999999999999,370,6386,6387],[2585,0.634,370,6386,6387],[227,0.627,370,6386,6387],[3261,0.5920000000000001,370,6386,6387]]},{"id":"m261019564-2026-paper","name":"FrameEvidence (matched success DIS, 2026 paper)","group":"视觉与视频","metric":"Matched success DIS","unit":"%","public_history":true,"protocol":{"en":"Table 4 matched success, DIS only. GEM and NTS are different measures. Base, LoRA and shot counts are configurations; every printed zero is retained. Classification recall is a separate column.","zh":"表 4 的匹配成功 DIS 列。GEM 和 NTS 是其他指标。Base、LoRA 和示例数记为配置，保留所有实测 0。分类召回率是另一列。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.19564v1","status":"paper_baselines","published_at":"2026-10-14"},"results":[[2754,0.4375,370,4974,6388],[2728,0.20829999999999999,370,4974,6388],[6342,0.3958,370,4974,6388],[6389,0.375,370,4974,6388],[5995,0.4792,370,4974,6388],[6390,0.0833,370,4974,6388],[6391,0.3542,370,4974,6388],[6345,0.25,370,4974,6388],[6392,0.3542,370,4974,6388],[6346,0.3958,370,4974,6388],[2754,0.125,370,6393,6388],[2728,0.0833,370,6393,6388],[4971,0.3542,370,6394,6388],[4971,0.4167,370,6395,6388],[4971,0.4375,370,6396,6388],[4971,0.4792,370,6397,6388],[2956,0.4167,370,6394,6388],[2956,0.375,370,6395,6388],[2956,0.375,370,6396,6388],[2956,0.3958,370,6397,6388]]},{"id":"m261012823-2026-paper","name":"MisVisAgentBench (overall PRA, 2026 paper)","group":"Agent 与工具","metric":"Overall PRA","unit":"%","public_history":true,"protocol":{"en":"Overall Evaluation PRA from the main table for all thirteen named models. Failure-attribution AFR_c is a separate diagnostic.","zh":"主表十三个具名模型的 Overall Evaluation PRA。故障归因 AFR_c 作为另一诊断指标保留在原表。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.12823v1","status":"paper_baselines","published_at":"2026-10-09"},"results":[[3184,0.35700000000000004,370,6175,6398],[3196,0.34299999999999997,370,6175,6398],[5504,0.493,370,6175,6398],[6399,0.27899999999999997,370,6175,6398],[26,0.457,370,6175,6398],[325,0.34299999999999997,370,6175,6398],[3572,0.34299999999999997,370,6175,6398],[2261,0.414,370,6175,6398],[3578,0.45,370,6175,6398],[3479,0.507,370,6175,6398],[3753,0.4,370,6175,6398],[3603,0.379,370,6175,6398],[3490,0.379,370,6175,6398]]},{"id":"ausclinbench-blind-2026-paper","name":"AusClinBench (blind accuracy, 2026 paper)","group":"医疗","metric":"Blind accuracy","unit":"%","public_history":true,"protocol":{"en":"Blind accuracy on version 1.4.1: the question alone, no sources. Figure 1 of the PDF. October-route rows used a Claude Code subscription and are marked as that configuration. Item counts vary from 135 to 137 as accounted in section 3.10; no missing cell was filled with zero.","zh":"1.4.1 版的盲测准确率，只给题目、不提供资料，取自 PDF 图 1。十月路径的三行走 Claude Code 订阅，记为该配置。题数在 135 到 137 之间，第 3.10 节逐项说明，没有把缺失格填成 0。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.31993v1","status":"paper_baselines","published_at":"2026-10-22"},"results":[[6400,0.927,370,6401,6402],[2958,0.92,370,6401,6402],[2959,0.861,370,6403,6402],[2956,0.8390000000000001,370,6401,6402],[755,0.8320000000000001,370,6401,6402],[3453,0.8220000000000001,370,6401,6402],[2983,0.7959999999999999,370,6401,6402],[2977,0.7809999999999999,370,6401,6402],[3482,0.779,370,6401,6402],[2973,0.772,370,6401,6402],[3391,0.772,370,6401,6402],[2974,0.7659999999999999,370,6401,6402],[2961,0.743,370,6403,6402],[3408,0.737,370,6401,6402],[3479,0.728,370,6401,6402],[3568,0.701,370,6401,6402],[3539,0.693,370,6401,6402],[3414,0.664,370,6401,6402],[3587,0.647,370,6401,6402],[3492,0.632,370,6401,6402],[3490,0.618,370,6401,6402],[3261,0.606,370,6401,6402],[4956,0.596,370,6401,6402],[3560,0.574,370,6401,6402],[2970,0.547,370,6403,6402],[5598,0.5329999999999999,370,6401,6402],[6404,0.5329999999999999,370,6401,6402],[362,0.515,370,6401,6402]]},{"id":"deltatau-sr-2026-paper","name":"DeltaTau-Bench (success rate, 2026 paper)","group":"Agent 与工具","metric":"Success rate","unit":"%","public_history":true,"protocol":{"en":"Task success rate from Table 1. Codex-GPT-5.5 is GPT-5.5 with the Codex harness, kept as a configuration beside the plain GPT-5.5 row. Retrieval F1 and path-efficiency columns are not included.","zh":"表 1 的任务成功率。Codex-GPT-5.5 是 GPT-5.5 加 Codex harness，与单独的 GPT-5.5 行记为不同配置。检索 F1 和路径效率不并入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.34238v1","status":"paper_baselines","published_at":"2026-10-25"},"results":[[3196,0.4227,370,6405,6406],[3405,0.4165,370,6175,6406],[3196,0.31739999999999996,370,6175,6406],[3568,0.29969999999999997,370,6175,6406],[3492,0.3329,370,6175,6406],[227,0.32280000000000003,370,6175,6406],[3560,0.301,370,6175,6406],[2977,0.28300000000000003,370,6175,6406],[6375,0.27,370,6175,6406]]},{"id":"spaceact-tiered-score-2026-paper","name":"SpaceAct (tiered score, 2026 paper)","group":"Agent 与工具","metric":"Tiered score","unit":"score","public_history":true,"protocol":{"en":"Tiered score from Table 3, averaged over the 41 graded instances. Success rate and category columns are not included. Random, myopic and reference-solver baselines are omitted. The authors already fold abandoned runs into this published score as unsolved; no extra zero was added.","zh":"表 3 的分级分，对 41 个有评分的实例取平均。成功率和分类列不并入。随机、贪心和参考求解器基线不计入。作者已把中止的运行按未解决计入这个公布分数，这里没有再补 0。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.38405v1","status":"paper_baselines","published_at":"2026-10-28"},"score_max":100,"score_decimals":1,"results":[[2959,79.1,370,6175,6407],[2958,65.0,370,6175,6407],[2962,42.1,370,6175,6407],[2965,29.0,370,6175,6407],[5361,24.7,370,6175,6407],[3560,21.5,370,6175,6407],[3512,19.5,370,6175,6407],[2977,10.4,370,6175,6407],[2971,6.1,370,6175,6407],[3534,4.8,370,6175,6407]]},{"id":"trigeobench-gen-acc-2026-paper","name":"TriGeoBench (generation accuracy, 2026 paper)","group":"视觉与视频","metric":"Generation accuracy","unit":"%","public_history":true,"protocol":{"en":"Generation accuracy from the cited table. Image-metric and TikZ columns are not included.","zh":"所引表的生成准确率。图像指标和 TikZ 列不并入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.27626v1","status":"paper_baselines","published_at":"2026-10-20"},"results":[[6408,0.805,370,6175,6409],[6410,0.764,370,6175,6409],[2444,0.935,370,6175,6409],[2752,0.688,370,6175,6409],[6411,0.784,370,6175,6409],[6412,0.882,370,6175,6409],[3192,0.93,370,6175,6409],[502,0.961,370,6175,6409],[3184,0.956,370,6175,6409]]},{"id":"agentheal-task-success-2026-paper","name":"AgentHeal (task success, 2026 paper)","group":"Agent 与工具","metric":"Task success","unit":"%","public_history":true,"protocol":{"en":"Task success in the counterfactual sandbox. The no-barrier column and the real-web replay are not included. The average row is omitted.","zh":"反事实沙箱中的任务成功率。无障碍列和真实网页重放不并入。平均行不计入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.39234v1","status":"paper_baselines","published_at":"2026-10-29"},"results":[[5618,0.5579999999999999,370,6413,6414],[5617,0.47,370,6413,6414],[5727,0.45799999999999996,370,6413,6414],[354,0.258,370,6413,6414],[6415,0.569,370,6413,6414],[3153,0.578,370,6413,6414],[3396,0.5489999999999999,370,6413,6414],[6416,0.271,370,6413,6414],[6417,0.527,370,6413,6414],[1621,0.28300000000000003,370,6413,6414],[5344,0.349,370,6413,6414],[4070,0.35600000000000004,370,6413,6414],[5300,0.46,370,6413,6414]]},{"id":"plc-direct-exec-2026-paper","name":"PLC scan evaluation (direct execution, 2026 paper)","group":"编程","metric":"Execution pass","unit":"%","public_history":true,"protocol":{"en":"Direct-execution pass rate. Compilation rate and the other harness columns are not included.","zh":"直接执行通过率。编译率和其余 harness 列不并入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.28675v1","status":"paper_baselines","published_at":"2026-10-19"},"results":[[3196,0.8439,370,6418,6419],[353,0.7276,370,6418,6419],[6420,0.8271999999999999,370,6418,6419],[3496,0.7807,370,6418,6419],[227,0.7973,370,6418,6419],[5803,0.5847,370,6418,6419],[3005,0.711,370,6418,6419],[3763,0.6977,370,6418,6419]]},{"id":"rtl-repair-nospec-2026-paper","name":"RTL repair (no-spec pass, 2026 paper)","group":"编程","metric":"Complete-case pass","unit":"%","public_history":true,"protocol":{"en":"Complete-case pass rate when no specification is given. The with-spec columns are not included.","zh":"不提供规格说明时的完整案例通过率。有规格说明的列不并入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.28595v1","status":"paper_baselines","published_at":"2026-10-20"},"results":[[3196,0.573,370,6421,6422],[3479,0.34299999999999997,370,6421,6422],[4958,0.29600000000000004,370,6421,6422],[345,0.273,370,6421,6422],[354,0.19,370,6421,6422],[2261,0.095,370,6421,6422]]},{"id":"rostrans-pass1-2026-paper","name":"ROSTrans (pass@1, 2026 paper)","group":"编程","metric":"pass@1","unit":"%","public_history":true,"protocol":{"en":"pass@1. The printed interval is not stored. pass@3, pass@5 and the by-level columns are not included.","zh":"pass@1。区间不入库。pass@3、pass@5 和分项不并入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.41168v1","status":"paper_baselines","published_at":"2026-10-30"},"results":[[2261,0.109,370,6175,6423],[3196,0.1,370,6175,6423],[3412,0.07200000000000001,370,6175,6423],[3753,0.067,370,6175,6423],[6424,0.065,370,6175,6423],[3492,0.057,370,6175,6423]]},{"id":"cxbench-both-pass-2026-paper","name":"cxbench (both-pass rate, 2026 paper)","group":"Agent 与工具","metric":"Both-pass rate","unit":"%","public_history":true,"protocol":{"en":"Both-pass rate under the gpt-5.6-sol judge. The gpt-6-astra judge column is not mixed in.","zh":"gpt-5.6-sol 裁判下的双通过率。gpt-6-astra 裁判列不并入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.25954v1","status":"paper_baselines","published_at":"2026-10-19"},"results":[[5368,0.20829999999999999,370,6425,6426],[5160,0.15,370,6425,6426],[4170,0.025,370,6425,6426],[440,0.2167,370,6425,6426],[438,0.15,370,6425,6426],[2951,0.0833,370,6425,6426]]},{"id":"terminal-math-oeval-2026-paper","name":"Terminal math modeling (O-Eval, 2026 paper)","group":"数学与科学","metric":"O-Eval","unit":"%","public_history":true,"protocol":{"en":"Outcome evaluation percentage. The harness after the plus sign, and xhigh effort, are configurations. The interval is not stored.","zh":"结果评估百分比。加号后的 harness 和 xhigh 档位是配置。区间不入库。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.08237v1","status":"paper_baselines","published_at":"2026-11-05"},"results":[[2958,0.6457999999999999,370,6427,6428],[3149,0.6311,370,6429,6428],[5729,0.5981000000000001,370,6430,6428],[3196,0.5975,370,6431,6428],[2977,0.5822999999999999,370,6430,6428],[3154,0.5778,370,6430,6428],[3260,0.5266,370,6430,6428]]},{"id":"codesign-strict-2026-paper","name":"Co-Design Bench (strict completion, 2026 paper)","group":"编程","metric":"Strict completion","unit":"score","public_history":true,"protocol":{"en":"Mean strict completion for the ME block on the published 0–1 scale. Graded completion and the later block are not included.","zh":"ME 组的严格完成均值，按原文 0–1。分级完成分和后面一组不并入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.35138v1","status":"paper_baselines","published_at":"2026-10-26"},"score_max":1,"score_decimals":3,"results":[[4943,0.754,370,6432,6433],[5466,0.695,370,6432,6433],[6434,0.667,370,6432,6433],[4952,0.662,370,6432,6433],[6435,0.642,370,6432,6433],[5467,0.593,370,6432,6433],[4972,0.583,370,6432,6433],[5104,0.572,370,6432,6433],[6037,0.538,370,6432,6433],[6031,0.467,370,6432,6433]]},{"id":"vissearch-acc-2026-paper","name":"VisSearch-Bench (accuracy, 2026 paper)","group":"Agent 与工具","metric":"Accuracy average","unit":"%","public_history":true,"protocol":{"en":"Micro-average accuracy. Direct answering, ReAct and DeerFlow are configurations of the same model. Per-domain columns and the process score are not included. A printed 0.00 is the cell in the table.","zh":"微平均准确率。直接作答、ReAct 和 DeerFlow 是同一模型的配置。分领域列和过程分不并入。表里印出的 0.00 按原格保留。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.04894v1","status":"paper_baselines","published_at":"2026-11-03"},"results":[[3764,0.0098,370,6436,6437],[3763,0.0,370,6436,6437],[2576,0.0196,370,6436,6437],[5722,0.0049,370,6436,6437],[3198,0.0245,370,6436,6437],[2600,0.0147,370,6436,6437],[6150,0.0049,370,6436,6437],[6149,0.0,370,6436,6437],[3396,0.0392,370,6436,6437],[6438,0.034300000000000004,370,6436,6437],[5300,0.0245,370,6436,6437],[3153,0.0196,370,6436,6437],[3155,0.0588,370,6436,6437],[3415,0.0637,370,6436,6437],[3479,0.049,370,6436,6437],[3196,0.0637,370,6436,6437],[3764,0.0735,370,6439,6437],[3196,0.25489999999999996,370,6439,6437],[2576,0.15689999999999998,370,6439,6437],[3396,0.201,370,6439,6437],[3155,0.2647,370,6439,6437],[3153,0.2255,370,6439,6437],[5300,0.2108,370,6439,6437],[3764,0.10779999999999999,370,6440,6437],[3196,0.201,370,6440,6437],[2576,0.10289999999999999,370,6440,6437]]},{"id":"iprefbench-sr-2026-paper","name":"iPrefBench (success rate, 2026 paper)","group":"Agent 与工具","metric":"Success rate","unit":"%","public_history":true,"protocol":{"en":"Overall personalized-task success rate. Infer and Ask splits are not included.","zh":"个性化任务的总成功率。Infer 和 Ask 分项不并入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.34735v1","status":"paper_baselines","published_at":"2026-10-26"},"results":[[755,0.47700000000000004,370,6175,6441],[3568,0.392,370,6175,6441],[3412,0.36200000000000004,370,6175,6441],[3386,0.3,370,6175,6441],[2977,0.36200000000000004,370,6175,6441],[5300,0.262,370,6175,6441],[6438,0.315,370,6175,6441]]},{"id":"codecap-video-success-2026-paper","name":"CodeCap-Bench (video production success, 2026 paper)","group":"视觉与视频","metric":"Video production success","unit":"%","public_history":true,"protocol":{"en":"Video production success rate. Agentic and non-agentic settings, and the harness, are configurations. Visual-similarity columns are not included. A printed 100 is the cell in the table.","zh":"视频生成成功率。智能体与非智能体、以及 harness，都是配置。视觉相似列不并入。表里印出的 100 按原格保留。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.37341v1","status":"paper_baselines","published_at":"2026-10-27"},"results":[[5404,0.9940000000000001,370,6442,6443],[5404,0.997,370,6444,6443],[5012,0.978,370,6444,6443],[5012,0.9940000000000001,370,6445,6443],[5365,1.0,370,6442,6443],[5790,0.932,370,6444,6443],[5408,0.988,370,6444,6443],[5365,0.991,370,6444,6443],[6446,0.9159999999999999,370,6444,6443],[4048,0.991,370,6444,6443],[4048,0.997,370,6442,6443],[5404,0.985,370,6447,6443],[5365,0.9570000000000001,370,6447,6443],[4048,0.9690000000000001,370,6447,6443],[5408,0.972,370,6447,6443],[5790,0.82,370,6447,6443]]},{"id":"cle-completion-2026-paper","name":"CLE (Trial 1 completion, 2026 paper)","group":"Agent 与工具","metric":"Completion","unit":"%","public_history":true,"protocol":{"en":"Trial 1 completion. The harness is a configuration. Trial 2 and the per-level columns are not included.","zh":"第一次提交的完成率。Harness 是配置。第二次提交和分级列不并入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.33956v1","status":"paper_baselines","published_at":"2026-10-25"},"results":[[755,0.4217,370,6448,6449],[3568,0.27,370,6448,6449],[4979,0.2783,370,6450,6449],[5727,0.2783,370,6450,6449],[5616,0.20329999999999998,370,6450,6449],[3386,0.25329999999999997,370,6451,6449],[6452,0.20329999999999998,370,6451,6449],[3157,0.25670000000000004,370,6451,6449],[5114,0.20329999999999998,370,6451,6449],[3492,0.2183,370,6451,6449],[3515,0.225,370,6451,6449],[3386,0.1867,370,6453,6449],[6452,0.145,370,6453,6449],[3157,0.18170000000000003,370,6453,6449],[5114,0.0983,370,6453,6449],[3492,0.1133,370,6453,6449],[3515,0.0617,370,6453,6449]]},{"id":"goal-success-2026-paper","name":"SABench (goal success, 2026 paper)","group":"Agent 与工具","metric":"Goal","unit":"%","public_history":true,"protocol":{"en":"Goal satisfaction on SABench, introduced in this paper. Collateral damage is not included. This is not a re-score of an older public goal benchmark.","zh":"本篇论文提出的 SABench 上的目标达成率。附带损害不并入。不是旧公开目标榜的重测。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.34167v1","status":"paper_baselines","published_at":"2026-10-25"},"results":[[3753,0.905,370,6175,6454],[4425,0.8759999999999999,370,6175,6454],[244,0.9,370,6175,6454],[4424,0.856,370,6175,6454],[4958,0.895,370,6175,6454],[2261,0.8759999999999999,370,6175,6454],[3424,0.955,370,6175,6454],[3572,0.92,370,6175,6454],[268,0.89,370,6175,6454],[3184,0.8809999999999999,370,6175,6454],[3412,0.856,370,6175,6454]]},{"id":"vibevideo-spec-2026-paper","name":"VibeVideoBench (spec compliance, 2026 paper)","group":"视觉与视频","metric":"Spec compliance","unit":"%","public_history":true,"protocol":{"en":"Share of tasks whose video meets the spec. The harness is a configuration. Judge scores and Elo are not included.","zh":"视频符合规格的任务比例。Harness 是配置。裁判分和 Elo 不并入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.35864v1","status":"paper_baselines","published_at":"2026-10-26"},"results":[[2958,0.937,370,6455,6456],[2959,0.996,370,6457,6456],[2961,0.946,370,6457,6456],[2962,0.7659999999999999,370,6458,6456],[3153,0.946,370,6459,6456],[2977,0.858,370,6460,6456],[3396,0.669,370,6461,6456],[762,0.845,370,6455,6456],[2983,0.774,370,6458,6456]]},{"id":"hardeval-pass-2026-paper","name":"HardEval (verified pass rate, 2026 paper)","group":"安全与可靠性","metric":"Pass rate","unit":"%","public_history":true,"protocol":{"en":"Pass rate on all verified applicable rules, the percent in parentheses, with the fixed 151-rule denominator.","zh":"全部已核适用规则上的通过率，取括号中的百分数，分母固定为 151。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.23685v1","status":"paper_baselines","published_at":"2026-10-16"},"results":[[3578,0.8079999999999999,370,6462,6463],[3479,0.768,370,6462,6463],[283,0.755,370,6462,6463],[3738,0.728,370,6462,6463],[353,0.662,370,6462,6463],[3568,0.589,370,6462,6463],[325,0.503,370,6462,6463],[6464,0.364,370,6462,6463],[6465,0.35100000000000003,370,6462,6463],[6466,0.212,370,6462,6463],[6467,0.159,370,6462,6463],[6468,0.166,370,6462,6463],[6469,0.139,370,6462,6463],[6470,0.126,370,6462,6463]]},{"id":"visualworld-mcq-2026-paper","name":"VisualWorldBench (MCQ, 2026 paper)","group":"视觉与视频","metric":"MCQ accuracy","unit":"%","public_history":true,"protocol":{"en":"Multiple-choice accuracy only. Grid selection and localization are not included.","zh":"只收录选择题准确率。网格选择和定位不并入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.20083v1","status":"paper_baselines","published_at":"2026-10-14"},"results":[[3412,0.86,370,6471,6472],[6473,0.7509999999999999,370,6471,6472],[6474,0.655,370,6471,6472],[3184,0.679,370,6471,6472],[340,0.655,370,6471,6472],[2998,0.6859999999999999,370,6471,6472],[2584,0.591,370,6471,6472],[5480,0.605,370,6471,6472],[4070,0.524,370,6471,6472],[6342,0.52,370,6471,6472],[5635,0.56,370,6471,6472],[4969,0.441,370,6471,6472],[6298,0.478,370,6471,6472],[2654,0.355,370,6471,6472],[2687,0.331,370,6471,6472],[6341,0.28,370,6471,6472],[6475,0.7070000000000001,370,6471,6472],[6476,0.768,370,6471,6472],[6477,0.8,370,6471,6472],[6478,0.693,370,6471,6472]]},{"id":"decision-pair-accuracy-2026-paper","name":"Decision models (pair accuracy, 2026 paper)","group":"预测与决策","metric":"Pair accuracy","unit":"%","public_history":true,"protocol":{"en":"Pair accuracy, macro-averaged over 13 task families. The interval is not stored. Level-3 columns are not included.","zh":"13 个任务族上的配对准确率宏平均。区间不入库。Level-3 列不并入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.28522v1","status":"paper_baselines","published_at":"2026-10-20"},"results":[[6479,0.8690000000000001,370,6175,6480],[6481,0.754,370,6175,6480],[6482,0.713,370,6175,6480],[6483,0.701,370,6175,6480],[6484,0.514,370,6175,6480],[6485,0.442,370,6175,6480],[6486,0.442,370,6175,6480],[6487,0.433,370,6175,6480],[6488,0.433,370,6175,6480],[6489,0.38,370,6175,6480],[6490,0.31,370,6175,6480],[6491,0.273,370,6175,6480],[6492,0.262,370,6175,6480],[6493,0.252,370,6175,6480],[5546,0.249,370,6175,6480],[6494,0.205,370,6175,6480],[6207,0.19699999999999998,370,6175,6480],[6495,0.196,370,6175,6480],[6496,0.183,370,6175,6480],[6497,0.16399999999999998,370,6175,6480],[6498,0.161,370,6175,6480],[6499,0.133,370,6175,6480],[6500,0.129,370,6175,6480],[6501,0.125,370,6175,6480],[6502,0.11800000000000001,370,6175,6480],[6503,0.11800000000000001,370,6175,6480],[6504,0.11699999999999999,370,6175,6480],[6505,0.105,370,6175,6480],[6506,0.098,370,6175,6480],[6507,0.07400000000000001,370,6175,6480],[6508,0.071,370,6175,6480],[6509,0.068,370,6175,6480],[6510,0.062,370,6175,6480],[6511,0.06,370,6175,6480],[6512,0.055,370,6175,6480],[6513,0.044000000000000004,370,6175,6480],[6514,0.040999999999999995,370,6175,6480],[6515,0.037000000000000005,370,6175,6480],[6516,0.036000000000000004,370,6175,6480],[6517,0.035,370,6175,6480]]},{"id":"intentdelta-travelplanner-2026-paper","name":"IntentDelta (TravelPlanner environment, intention recovery, 2026)","group":"Agent 与工具","metric":"Intention recovery","unit":"%","public_history":true,"protocol":{"en":"Intention recovery on the 638-trajectory IntentDelta benchmark released with this paper, TravelPlanner environment only. This is not a new score release of the original TravelPlanner leaderboard.","zh":"本篇论文发布的 IntentDelta（638 条轨迹）里、TravelPlanner 环境上的意图恢复率。不是原 TravelPlanner 榜的新分数。"},"source":{"name":"IntentDelta authors","url":"https://arxiv.org/abs/2610.33806v1","status":"paper_baselines","published_at":"2026-10-24"},"results":[[4972,0.384,370,6518,6519],[5155,0.382,370,6518,6519],[5108,0.373,370,6518,6519],[5107,0.391,370,6518,6519],[4194,0.23399999999999999,370,6518,6519],[6031,0.373,370,6518,6519],[2943,0.36200000000000004,370,6518,6519],[6520,0.312,370,6518,6519],[6521,0.256,370,6518,6519],[6522,0.304,370,6518,6519],[6523,0.275,370,6518,6519]]},{"id":"intentdelta-webshop-2026-paper","name":"IntentDelta (WebShop environment, intention recovery, 2026)","group":"Agent 与工具","metric":"Intention recovery","unit":"%","public_history":true,"protocol":{"en":"Intention recovery on IntentDelta, WebShop environment only. This is not a new score release of the original WebShop leaderboard.","zh":"IntentDelta 的 WebShop 环境上的意图恢复率。不是原 WebShop 榜的新分数。"},"source":{"name":"IntentDelta authors","url":"https://arxiv.org/abs/2610.33806v1","status":"paper_baselines","published_at":"2026-10-24"},"results":[[4972,0.506,370,6524,6519],[5155,0.494,370,6524,6519],[5108,0.52,370,6524,6519],[5107,0.506,370,6524,6519],[4194,0.35600000000000004,370,6524,6519],[6031,0.514,370,6524,6519],[2943,0.493,370,6524,6519],[6520,0.401,370,6524,6519],[6521,0.325,370,6524,6519],[6522,0.40399999999999997,370,6524,6519],[6523,0.401,370,6524,6519]]},{"id":"openhome-overall-2026-paper","name":"OpenHome (overall accuracy, 2026 paper)","group":"视觉与视频","metric":"Overall accuracy","unit":"%","public_history":true,"protocol":{"en":"Overall accuracy. Input mode and reasoning setting are configurations.","zh":"总准确率。输入方式和推理设置是配置。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.31986v1","status":"paper_baselines","published_at":"2026-10-22"},"results":[[2962,0.812,370,6525,6526],[2958,0.7659999999999999,370,6527,6526],[2960,0.736,370,6527,6526],[2983,0.7040000000000001,370,6525,6526],[2965,0.6829999999999999,370,6528,6526],[3412,0.638,370,6525,6526],[3415,0.623,370,6529,6526],[2602,0.586,370,6530,6526],[5479,0.579,370,6531,6526],[2976,0.5760000000000001,370,6532,6526],[2968,0.5720000000000001,370,6533,6526],[4035,0.552,370,6529,6526],[6534,0.547,370,6535,6526],[353,0.535,370,6536,6526],[2576,0.52,370,6537,6526],[3756,0.518,370,6538,6526],[345,0.518,370,6536,6526],[6438,0.517,370,6537,6526],[5300,0.51,370,6537,6526],[2055,0.509,370,6539,6526],[2582,0.506,370,6537,6526],[3764,0.499,370,6537,6526],[2584,0.49200000000000005,370,6537,6526],[6279,0.48200000000000004,370,6539,6526],[4070,0.478,370,6537,6526],[6540,0.466,370,6541,6526],[6542,0.46399999999999997,370,6537,6526],[6543,0.46,370,6541,6526],[6275,0.45799999999999996,370,6536,6526],[6544,0.43200000000000005,370,6541,6526],[6545,0.43,370,6541,6526],[6546,0.415,370,6537,6526]]},{"id":"refcraft-overall-2026-paper","name":"RefCraft (overall, 2026 paper)","group":"Agent 与工具","metric":"Overall","unit":"score","public_history":true,"protocol":{"en":"Overall score on the 300-task RefCraft benchmark. Component scores are not included.","zh":"RefCraft 300 项任务的总分。分项不并入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.26826v1","status":"paper_baselines","published_at":"2026-10-19"},"score_max":100,"saturation_excluded":true,"score_decimals":2,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"results":[[2958,52.12,370,6175,6547],[3157,48.95,370,6175,6547],[2974,48.83,370,6175,6547],[2977,47.55,370,6175,6547],[3453,44.61,370,6175,6547],[3154,42.6,370,6175,6547],[3185,37.09,370,6175,6547],[4030,32.06,370,6175,6547],[3412,19.01,370,6175,6547]]},{"id":"vibench-pass1-2026-paper","name":"ViBench (pass@1, 2026 paper)","group":"编程","metric":"pass@1","unit":"%","public_history":true,"protocol":{"en":"pass@1. Plan-pass rate and cost are not included. A printed zero is kept as printed.","zh":"pass@1。计划通过率和费用不并入。表里印出的 0 按原格保留。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.05620v1","status":"paper_baselines","published_at":"2026-11-03"},"results":[[2960,0.531,370,6175,6548],[2964,0.469,370,6175,6548],[3394,0.312,370,6175,6548],[762,0.031,370,6175,6548],[2976,0.0,370,6175,6548],[3196,0.0,370,6175,6548],[3184,0.0,370,6175,6548],[2973,0.0,370,6175,6548],[3408,0.0,370,6175,6548]]},{"id":"rota-success-2026-paper","name":"RoTA-Bench (success, 2026 paper)","group":"Agent 与工具","metric":"Success","unit":"%","public_history":true,"protocol":{"en":"Success rate on the 100 test tasks. The confidence interval is not stored.","zh":"100 道测试任务的成功率。置信区间不入库。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.17665v1","status":"paper_baselines","published_at":"2026-10-13"},"results":[[762,0.16,370,6549,6550],[3405,0.1,370,6175,6550],[4979,0.17,370,6175,6550],[4977,0.1,370,6175,6550],[5727,0.08,370,6175,6550],[5616,0.04,370,6175,6550],[762,0.02,370,6551,6550],[3595,0.0,370,6175,6550],[3568,0.0,370,6175,6550],[755,0.0,370,6175,6550],[6552,0.0,370,6175,6550]]},{"id":"agentvqa-avg-2026-paper","name":"AgentVQA (average, 2026 paper)","group":"视觉与视频","metric":"Average","unit":"%","public_history":true,"protocol":{"en":"Average across five domains. Thinking effort is a configuration.","zh":"五个领域的平均。思考档位是配置。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.31494v1","status":"paper_baselines","published_at":"2026-10-22"},"results":[[6553,0.55,370,6554,6555],[6553,0.382,370,6556,6555],[6557,0.479,370,6554,6555],[6557,0.32799999999999996,370,6556,6555],[353,0.314,370,6175,6555],[6558,0.557,370,6175,6555],[6559,0.585,370,6175,6555],[3479,0.44,370,6175,6555],[3568,0.434,370,6175,6555],[6560,0.418,370,6175,6555],[6561,0.429,370,6175,6555],[6562,0.354,370,6175,6555],[6563,0.363,370,6175,6555],[2977,0.483,370,6175,6555],[1583,0.28,370,6175,6555],[2986,0.509,370,6175,6555],[5734,0.486,370,6175,6555],[5663,0.55,370,6175,6555],[6564,0.478,370,6175,6555],[6565,0.46,370,6175,6555],[6566,0.479,370,6175,6555],[6114,0.467,370,6175,6555]]},{"id":"multistep-swe-solve-2026-paper","name":"Multi-step SWE (solve rate, 2026 paper)","group":"编程","metric":"Solve rate","unit":"%","public_history":true,"protocol":{"en":"Overall solve rate on the 69 chained tasks. Family columns and pass@4 are not included. The interval is not stored.","zh":"69 个链式任务的总解决率。分族和 pass@4 不并入。区间不入库。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.07166v1","status":"paper_baselines","published_at":"2026-11-04"},"results":[[2959,0.185,370,6175,6567],[2956,0.17800000000000002,370,6175,6567],[2962,0.10099999999999999,370,6175,6567],[2960,0.10099999999999999,370,6175,6567],[2958,0.069,370,6175,6567],[3512,0.065,370,6175,6567],[3560,0.04,370,6175,6567],[6568,0.028999999999999998,370,6175,6567],[2968,0.013999999999999999,370,6175,6567]]},{"id":"omnlife-full-correct-2026-paper","name":"Omni-Life (full correct, 2026 paper)","group":"Agent 与工具","metric":"Full correct","unit":"%","public_history":true,"protocol":{"en":"Full-correct rate. The human baseline is omitted. A printed zero is the cell in the table.","zh":"完全正确率。人类基线不计入。表里印出的 0 按原格保留。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.02835v1","status":"paper_baselines","published_at":"2026-11-02"},"results":[[6569,0.0,370,6175,6570],[6571,0.0,370,6175,6570],[6572,0.0,370,6175,6570],[6573,0.0069,370,6175,6570],[6574,0.0138,370,6175,6570],[6575,0.0207,370,6175,6570],[6576,0.0207,370,6175,6570],[6577,0.044800000000000006,370,6175,6570],[6578,0.0207,370,6175,6570],[3260,0.0276,370,6175,6570],[6579,0.031,370,6175,6570],[6580,0.0379,370,6175,6570]]},{"id":"pianobench-midi-2026-paper","name":"PianoBench (MIDI-text accuracy, 2026 paper)","group":"知识与推理","metric":"MIDI-text accuracy","unit":"%","public_history":true,"protocol":{"en":"Accuracy with MIDI-text input. The no-music control and other audio conditions are not included. Thinking effort is a configuration. A dash is a missing run and is not stored as zero.","zh":"MIDI 文本输入下的准确率。无音乐对照和其他音频条件不并入。思考档位是配置。破折号表示没有这次运行，不记成 0。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.03839v1","status":"paper_baselines","published_at":"2026-11-02"},"results":[[6581,0.47600000000000003,370,6582,6583],[6581,0.47200000000000003,370,6584,6583],[6585,0.584,370,6582,6583],[6586,0.456,370,6175,6583],[6587,0.488,370,6175,6583],[6588,0.688,370,6589,6583],[203,0.664,370,6589,6583],[3482,0.892,370,6589,6583],[6590,0.5720000000000001,370,6175,6583],[3407,0.868,370,6589,6583],[3386,0.9,370,6589,6583],[2983,0.888,370,6589,6583],[5672,0.7,370,6582,6583],[5672,0.5,370,6584,6583],[6591,0.852,370,6554,6583],[5004,0.7040000000000001,370,6556,6583],[5004,0.856,370,6589,6583],[5004,0.872,370,6554,6583],[5004,0.88,370,6288,6583],[5508,0.932,370,6554,6583]]},{"id":"ptc-bench-accuracy-2026-paper","name":"PTC-Bench (accuracy, 2026 paper)","group":"Agent 与工具","metric":"Accuracy","unit":"%","public_history":true,"protocol":{"en":"Accuracy on PTC-Bench. Tool-only, PTC and I-PTC are protocols of the same model, not separate models.","zh":"PTC-Bench 的准确率。Tool-only、PTC 和 I-PTC 是同一模型的协议，不是不同模型。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.01446v1","status":"paper_baselines","published_at":"2026-11-01"},"results":[[5101,0.18600000000000003,370,6592,6593],[5101,0.715,370,6594,6593],[5101,0.8,370,6595,6593],[3599,0.10099999999999999,370,6592,6593],[3599,0.591,370,6594,6593],[3599,0.68,370,6595,6593],[3611,0.069,370,6592,6593],[3611,0.373,370,6594,6593],[3611,0.46399999999999997,370,6595,6593],[3412,0.679,370,6592,6593],[3412,0.828,370,6594,6593],[3412,0.8290000000000001,370,6595,6593],[4219,0.10800000000000001,370,6592,6593],[4219,0.713,370,6594,6593],[4219,0.728,370,6595,6593],[2602,0.315,370,6592,6593],[2602,0.5660000000000001,370,6594,6593],[2602,0.6,370,6595,6593],[3578,0.618,370,6592,6593],[3578,0.8420000000000001,370,6594,6593],[3578,0.8290000000000001,370,6595,6593],[3572,0.54,370,6592,6593],[3572,0.8390000000000001,370,6594,6593],[3572,0.8059999999999999,370,6595,6593],[6596,0.435,370,6592,6593],[6596,0.753,370,6594,6593],[6596,0.718,370,6595,6593],[244,0.281,370,6592,6593],[244,0.755,370,6594,6593],[244,0.789,370,6595,6593]]},{"id":"interactionbench-spoken-2026-paper","name":"InteractionBench (spoken-task accuracy, 2026 paper)","group":"知识与推理","metric":"Spoken-task accuracy","unit":"%","public_history":true,"protocol":{"en":"Spoken-task accuracy. Timing, whisper, groundedness and coherence are not included.","zh":"口语任务准确率。时机、耳语、依据和连贯性不并入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.02430v1","status":"paper_baselines","published_at":"2026-11-02"},"results":[[6597,0.68,370,6175,6598],[6599,0.7,370,6175,6598],[6600,0.72,370,6175,6598],[6601,0.06,370,6175,6598],[6602,0.18,370,6175,6598],[6603,0.08,370,6175,6598],[6604,0.76,370,6175,6598]]},{"id":"composet-accuracy-2026-paper","name":"CompoSET (accuracy, 2026 paper)","group":"视觉与视频","metric":"CompoSET accuracy","unit":"%","public_history":true,"protocol":{"en":"Accuracy on CompoSET only. The other columns are different benchmarks and are not included. Chance is omitted.","zh":"只收录 CompoSET 上的准确率。其他列是别的基准，不并入。随机水平不计入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.41118v1","status":"paper_baselines","published_at":"2026-10-30"},"results":[[6605,0.044000000000000004,370,6175,6606],[6607,0.078,370,6175,6606],[6608,0.084,370,6175,6606],[6609,0.102,370,6175,6606],[6610,0.135,370,6175,6606],[6611,0.113,370,6175,6606],[6612,0.136,370,6175,6606],[6613,0.139,370,6175,6606],[6614,0.07,370,6175,6606],[6615,0.055,370,6175,6606],[6616,0.13699999999999998,370,6175,6606],[6617,0.10800000000000001,370,6175,6606],[6618,0.086,370,6175,6606],[6619,0.08900000000000001,370,6175,6606]]},{"id":"foresight-swe-easy-2026-paper","name":"Foresight-SWE (easy resolve rate, 2026 paper)","group":"编程","metric":"Resolve rate","unit":"%","public_history":true,"protocol":{"en":"Resolve rate on Easy tasks. The other difficulties are separate cohorts. FRR is not included.","zh":"Easy 难度上的解决率。其他难度另成成绩组。FRR 不并入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.01181v1","status":"paper_baselines","published_at":"2026-10-31"},"results":[[6620,0.9570000000000001,370,6621,6622],[3154,0.7659999999999999,370,6621,6622],[6293,0.6809999999999999,370,6621,6622],[762,0.9359999999999999,370,6621,6622],[2973,0.851,370,6621,6622],[3408,0.872,370,6621,6622],[3156,0.851,370,6621,6622],[3393,0.7659999999999999,370,6621,6622],[3191,0.46799999999999997,370,6621,6622],[5300,0.532,370,6621,6622],[2584,0.38299999999999995,370,6621,6622]]},{"id":"foresight-swe-medium-2026-paper","name":"Foresight-SWE (medium resolve rate, 2026 paper)","group":"编程","metric":"Resolve rate","unit":"%","public_history":true,"protocol":{"en":"Resolve rate on Medium tasks. The other difficulties are separate cohorts. FRR is not included.","zh":"Medium 难度上的解决率。其他难度另成成绩组。FRR 不并入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.01181v1","status":"paper_baselines","published_at":"2026-10-31"},"results":[[6620,0.9740000000000001,370,6623,6622],[3154,0.805,370,6623,6622],[6293,0.597,370,6623,6622],[762,0.818,370,6623,6622],[2973,0.792,370,6623,6622],[3408,0.7659999999999999,370,6623,6622],[3156,0.74,370,6623,6622],[3393,0.5710000000000001,370,6623,6622],[3191,0.23399999999999999,370,6623,6622],[5300,0.273,370,6623,6622],[2584,0.23399999999999999,370,6623,6622]]},{"id":"foresight-swe-hard-2026-paper","name":"Foresight-SWE (hard resolve rate, 2026 paper)","group":"编程","metric":"Resolve rate","unit":"%","public_history":true,"protocol":{"en":"Resolve rate on Hard tasks. The other difficulties are separate cohorts. FRR is not included.","zh":"Hard 难度上的解决率。其他难度另成成绩组。FRR 不并入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.01181v1","status":"paper_baselines","published_at":"2026-10-31"},"results":[[6620,0.852,370,6624,6622],[3154,0.83,370,6624,6622],[6293,0.523,370,6624,6622],[762,0.818,370,6624,6622],[2973,0.648,370,6624,6622],[3408,0.648,370,6624,6622],[3156,0.773,370,6624,6622],[3393,0.534,370,6624,6622],[3191,0.068,370,6624,6622],[5300,0.091,370,6624,6622],[2584,0.034,370,6624,6622]]},{"id":"onescene-counting-complete-2026-paper","name":"OneScene-Bench (counting, complete, 2026 paper)","group":"视觉与视频","metric":"Counting accuracy","unit":"%","public_history":true,"protocol":{"en":"Counting accuracy on complete evidence. Subset, complementary, consistency and spatial reasoning are not mixed in.","zh":"完整证据条件下的计数准确率。子集、互补、一致性和空间推理不并入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.38926v1","status":"paper_baselines","published_at":"2026-10-28"},"results":[[6625,0.7,370,6626,6627],[6374,0.61,370,6626,6627],[6628,0.72,370,6626,6627],[6629,0.555,370,6626,6627],[6630,0.67,370,6626,6627],[6631,0.54,370,6626,6627],[6241,0.57,370,6626,6627],[6632,0.66,370,6626,6627],[353,0.615,370,6626,6627],[3196,0.7,370,6626,6627],[3386,0.815,370,6626,6627],[325,0.455,370,6626,6627]]},{"id":"uthcd-c-clean-2026-paper","name":"uTHCD-C (clean accuracy, 2026 paper)","group":"视觉与视频","metric":"Clean accuracy","unit":"%","public_history":true,"protocol":{"en":"Mean clean-data accuracy under 5-fold cross-validation. Corruption results are not included.","zh":"五折交叉验证下的干净数据平均准确率。损坏条件不并入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.36850v1","status":"paper_baselines","published_at":"2026-10-27"},"results":[[6633,0.9782,370,6634,6635],[6636,0.9777,370,6634,6635],[6637,0.9773000000000001,370,6634,6635],[6638,0.9756999999999999,370,6634,6635],[6639,0.9727,370,6634,6635],[6640,0.9711,370,6634,6635],[6641,0.97,370,6634,6635],[6642,0.9632999999999999,370,6634,6635],[6643,0.9573999999999999,370,6634,6635]]},{"id":"atlas-bench-success-k-2026-paper","name":"ATLAS-Bench (Success@k, 2026 paper)","group":"智能体","metric":"Success@k","unit":"%","public_history":true,"protocol":{"en":"Success@k on ATLAS-Bench. Think and no-think are separate configurations of the same model. CSR and efficiency are not included.","zh":"ATLAS-Bench 的 Success@k。思考与不思考是同一模型的两种配置。CSR 和效率不并入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.39438v1","status":"paper_baselines","published_at":"2026-10-29"},"results":[[4969,0.54,370,6644,6645],[4969,0.5770000000000001,370,6646,6645],[3764,0.731,370,6644,6645],[3764,0.7390000000000001,370,6646,6645],[5300,0.754,370,6644,6645],[5300,0.8290000000000001,370,6646,6645],[5519,0.762,370,6647,6645],[6648,0.8079999999999999,370,6649,6645],[3155,0.8240000000000001,370,3140,6645],[5616,0.69,370,6623,6645],[5617,0.6729999999999999,370,6650,6645],[3157,0.84,370,3140,6645],[3153,0.847,370,3143,6645],[3154,0.853,370,3140,6645]]},{"id":"agentdecept-rate-2026-paper","name":"AgentDecept (deception rate, 2026 paper)","group":"智能体","metric":"Deception rate","unit":"%","public_history":true,"protocol":{"en":"Share of decisive judgments labeled deceptive. A printed zero is kept. Higher means more deception.","zh":"在已判定样本中被标为欺骗的比例。表中的 0 保留。越高表示欺骗越多。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.39377v1","status":"paper_baselines","published_at":"2026-10-29"},"higher_is_better":false,"results":[[2958,0.0173,370,6651,6652],[3512,0.0215,370,6651,6652],[2983,0.16449999999999998,370,6651,6652],[3412,0.2484,370,6653,6652],[6654,0.1419,370,6651,6652],[762,0.14980000000000002,370,6655,6652],[755,0.0,370,6655,6652],[5598,0.062400000000000004,370,6651,6652],[283,0.1266,370,6656,6652]]},{"id":"streamvibe-explicit-f1-2026-paper","name":"StreamVibe-Bench (explicit F1, 2026 paper)","group":"视觉与视频","metric":"Explicit F1","unit":"score","public_history":true,"protocol":{"en":"Pointwise explicit-event F1 at a ±2 second tolerance. Implicit and dynamic columns, and the trajectory protocol, are separate and not included. Native 0–100 scale. This is not an accuracy percentage and is kept out of the saturation groups.","zh":"逐点评估下显式事件的 F1，容差 ±2 秒。隐式、动态和轨迹协议不并入。 原生 0–100 尺度，不是准确率百分比，不进入饱和分组。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.34662v1","status":"paper_baselines","published_at":"2026-10-26"},"score_max":100,"scale_label":{"en":"Explicit F1 / 100","zh":"Explicit F1 / 100"},"results":[[502,65.28,370,6657,6658],[6659,64.93,370,6657,6658],[5519,67.81,370,6657,6658],[6342,63.16,370,6657,6658],[4070,67.04,370,6657,6658],[6660,66.34,370,6657,6658],[6661,59.88,370,6662,6658],[6661,60.88,370,6663,6658]]},{"id":"bac-bench-effective-2026-paper","name":"BAC-Bench (effective success, 2026 paper)","group":"编程","metric":"Effective success","unit":"%","public_history":true,"protocol":{"en":"End-to-end effective success of the base model. Applicability, proof-of-concept and functional stages are not mixed in. The macro-average row is omitted.","zh":"基础模型的端到端有效成功率。适用性、概念验证和功能阶段不并入。宏平均行不计入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.34290v1","status":"paper_baselines","published_at":"2026-10-25"},"results":[[6664,0.36229999999999996,370,279,6665],[3202,0.38409999999999994,370,279,6665],[3479,0.3043,370,279,6665],[3196,0.6087,370,279,6665],[3415,0.3043,370,279,6665],[227,0.413,370,279,6665],[6666,0.2609,370,279,6665],[6667,0.3768,370,279,6665]]},{"id":"boardsafety-direct-2026-paper","name":"BoardSafety (no rule, direct answer, 2026 paper)","group":"安全与可靠性","metric":"Accuracy","unit":"%","public_history":true,"protocol":{"en":"Direct-answer accuracy when the rule text is withheld. The explicit-rule condition and the mean row are not included.","zh":"不提供规则文本时的直接作答准确率。显式规则条件和平均行不并入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.33448v1","status":"paper_baselines","published_at":"2026-10-24"},"results":[[6668,0.4732,370,6669,6670],[6671,0.37579999999999997,370,6669,6670],[587,0.4597,370,6669,6670],[613,0.3691,370,6669,6670],[5660,0.3691,370,6669,6670],[5126,0.40270000000000006,370,6669,6670]]},{"id":"sci-agent-exam-final-2026-paper","name":"SCI-AGENT-EXAM (final score, 2026 paper)","group":"数学与科学","metric":"Final score","unit":"score","public_history":true,"protocol":{"en":"Final score on 120 tasks with the Claude Code harness. Programmatic and judge columns are components of this published final score, not extra cohorts.","zh":"120 道题、Claude Code 工具环境下的最终分。程序分和评审分是这个已发表最终分的组成部分。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.32441v1","status":"paper_baselines","published_at":"2026-10-23"},"score_max":100,"saturation_excluded":true,"score_decimals":2,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"results":[[2958,80.9,370,3759,6672],[3453,67.3,370,3759,6672],[755,64.2,370,3759,6672],[2982,62.7,370,3759,6672],[2977,62.6,370,3759,6672],[762,59.6,370,3759,6672],[3479,57.1,370,3759,6672],[3154,55.9,370,3759,6672],[5295,52.4,370,3759,6672],[3185,50.9,370,3759,6672]]},{"id":"rxnex-c1-f1-2026-paper","name":"RxnExBench (all levels, criterion 1 F1, 2026 paper)","group":"数学与科学","metric":"F1","unit":"score","public_history":true,"protocol":{"en":"F1 on the full figure set under criterion 1. Other criteria and difficulty levels are not mixed in. A printed near-zero is kept. Native 0–100 scale. This is not an accuracy percentage and is kept out of the saturation groups.","zh":"全部图、标准 1 下的 F1。其他标准和难度不并入。表上接近 0 的分数保留。 原生 0–100 尺度，不是准确率百分比，不进入饱和分组。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.32221v1","status":"paper_baselines","published_at":"2026-10-23"},"score_max":100,"scale_label":{"en":"F1 / 100","zh":"F1 / 100"},"results":[[2261,2.52,370,6673,6674],[3412,19.47,370,6673,6674],[3196,23.16,370,6673,6674],[353,0.09,370,6673,6674],[4225,19.81,370,6673,6674],[6675,5.08,370,6673,6674],[6676,2.88,370,6673,6674],[362,2.27,370,6673,6674],[6297,1.68,370,6673,6674]]},{"id":"pcfbench-geometry-familiarity-2026-paper","name":"PCFBench (geometry familiarity, 2026 paper)","group":"视觉与视频","metric":"Geometry familiarity accuracy","unit":"%","public_history":true,"protocol":{"en":"Geometry familiarity accuracy, printed as a proportion. Counting, topology and the loss column use different metrics and are not included.","zh":"几何熟悉度准确率，按表中的比例录入。计数、拓扑和损失列是不同指标，不并入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.32485v1","status":"paper_baselines","published_at":"2026-10-23"},"results":[[6677,0.045,370,6678,6679],[2754,0.096,370,6680,6679],[6342,0.192,370,6681,6679],[6682,0.054,370,6681,6679],[6683,0.093,370,6681,6679],[6684,0.031,370,6685,6679],[6668,0.085,370,6686,6679],[2515,0.133,370,6686,6679],[6687,0.115,370,6688,6679],[6689,0.125,370,6690,6679],[6238,0.115,370,6691,6679],[6391,0.18,370,6692,6679],[2750,0.155,370,6693,6679],[2166,0.185,370,6693,6679],[6694,0.025,370,6695,6679],[1702,0.105,370,6696,6679],[6697,0.06,370,6698,6679],[5843,0.225,370,6699,6679],[4070,0.175,370,6700,6679],[2584,0.205,370,6690,6679],[2582,0.242,370,6701,6679],[1583,0.19,370,6702,6679],[889,0.21,370,6703,6679],[354,0.13,370,6703,6679],[353,0.185,370,6703,6679],[5406,0.305,370,6703,6679],[345,0.36,370,6703,6679],[4064,0.385,370,6703,6679],[2058,0.455,370,6703,6679],[5929,0.335,370,6703,6679],[1464,0.42,370,6703,6679],[4066,0.065,370,6703,6679],[6704,0.265,370,6703,6679],[6705,0.395,370,6703,6679],[6706,0.305,370,6703,6679],[6707,0.445,370,6703,6679],[6133,0.668,370,6703,6679],[6708,0.264,370,6703,6679],[3250,0.215,370,6703,6679],[6709,0.24,370,6703,6679],[5722,0.145,370,6703,6679]]},{"id":"attack-stage-accuracy-2026-paper","name":"SecLog-Bench (accuracy, 2026 paper)","group":"安全与可靠性","metric":"Accuracy","unit":"%","public_history":true,"protocol":{"en":"Attack-stage accuracy. Macro-F1 and per-label recall are not included. The mean row is omitted.","zh":"攻击阶段准确率。宏 F1 和各类召回不并入。平均行不计入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.31000v1","status":"paper_baselines","published_at":"2026-10-22"},"results":[[5344,0.23149999999999998,370,6175,6710],[786,0.2639,370,6175,6710],[562,0.2407,370,6175,6710],[769,0.2731,370,6175,6710],[3189,0.33799999999999997,370,6175,6710],[3482,0.375,370,6175,6710],[3572,0.44439999999999996,370,6175,6710],[3184,0.3333,370,6175,6710]]},{"id":"environment-duels-solve-2026-paper","name":"Environment Duels (solve, 2026 paper)","group":"知识与推理","metric":"Solve","unit":"%","public_history":true,"protocol":{"en":"Solver score. Difficulty and hint lift are separate authoring measures and are not included.","zh":"求解分。难度和提示提升是出题侧指标，不并入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.30116v1","status":"paper_baselines","published_at":"2026-10-21"},"results":[[4983,0.9440000000000001,370,6711,6712],[4979,0.892,370,6711,6712],[3153,0.8690000000000001,370,6711,6712],[5300,0.831,370,6711,6712],[3154,0.828,370,6711,6712],[5295,0.8240000000000001,370,6711,6712],[5519,0.718,370,6711,6712],[3155,0.643,370,6711,6712],[5723,0.5479999999999999,370,6711,6712]]},{"id":"evaomni-overall-accuracy-2026-paper","name":"EvaOmni (overall accuracy, 2026 paper)","group":"视觉与视频","metric":"Overall accuracy","unit":"%","public_history":true,"protocol":{"en":"Overall accuracy across the three capability levels. Level scores and the human-expert row are not included.","zh":"三个能力层的总准确率。分层分数和人类专家行不并入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.28444v1","status":"paper_baselines","published_at":"2026-10-20"},"results":[[6138,0.16399999999999998,370,6713,6714],[6715,0.252,370,6713,6714],[6716,0.248,370,6713,6714],[6717,0.242,370,6713,6714],[6718,0.568,370,6713,6714],[6719,0.536,370,6713,6714],[6720,0.518,370,6713,6714],[6721,0.5539999999999999,370,6713,6714],[6722,0.7090000000000001,370,6713,6714],[6723,0.377,370,6713,6714],[6724,0.49,370,6713,6714],[6725,0.5820000000000001,370,6713,6714],[6726,0.48100000000000004,370,6713,6714],[6727,0.397,370,6713,6714],[6728,0.71,370,6713,6714],[6729,0.456,370,6713,6714],[6730,0.478,370,6713,6714],[6731,0.723,370,6713,6714],[6732,0.685,370,6713,6714],[6733,0.483,370,6713,6714],[6734,0.7440000000000001,370,6713,6714],[6735,0.794,370,6713,6714],[6736,0.78,370,6713,6714],[6737,0.614,370,6713,6714]]},{"id":"dsni-media-accuracy-2026-paper","name":"DSNI (media accuracy, 2026 paper)","group":"数学与科学","metric":"Media accuracy","unit":"%","public_history":true,"protocol":{"en":"Media accuracy on the DSNI test set. Temperature, pH and oxygen columns are not included. Gating is a configuration of Transformer and of RNABERT, not a separate model.","zh":"DSNI 测试集上的培养基准确率。温度、pH 和氧气列不并入。 门控是 Transformer 和 RNABERT 的配置，不是另一个模型。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.28310v1","status":"paper_baselines","published_at":"2026-10-20"},"results":[[6738,0.382,370,6175,6739],[6740,0.415,370,6175,6739],[6741,0.46299999999999997,370,6175,6739],[6742,0.43799999999999994,370,6175,6739],[6743,0.447,370,6175,6739],[6744,0.483,370,6745,6739],[6744,0.526,370,6746,6739],[6747,0.514,370,6175,6739],[6747,0.5479999999999999,370,6746,6739]]},{"id":"terranostra-overall-accuracy-2026-paper","name":"TerraNostra-Bench (overall accuracy, 2026 paper)","group":"预测与决策","metric":"Overall accuracy","unit":"%","public_history":true,"protocol":{"en":"Overall accuracy, printed as a proportion. The land-use-change columns are not included.","zh":"总准确率，按表中的比例录入。土地利用变化列不并入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.27793v1","status":"paper_baselines","published_at":"2026-10-20"},"results":[[6748,0.8058,370,6713,6749],[6750,0.8193,370,6713,6749],[6751,0.8072,370,6713,6749],[6752,0.8399,370,6713,6749],[6753,0.7923,370,6713,6749],[6754,0.8455,370,6713,6749]]},{"id":"wildaigc-raw-recall-2026-paper","name":"WildAIGC-Bench (raw recall, 2026 paper)","group":"视觉与视频","metric":"Recall","unit":"%","public_history":true,"protocol":{"en":"Recall on the raw WildAIGC-Bench images. External recall, false-positive rate and the cropped subset are not included. PE-336 and PE-518 are two input sizes of one detector.","zh":"未裁剪 WildAIGC-Bench 上的召回率。外部召回、假阳率和裁剪子集不并入。 PE-336 和 PE-518 是同一检测器的两种输入尺寸。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.28087v1","status":"paper_baselines","published_at":"2026-10-20"},"results":[[6755,0.2948,370,6756,6757],[6758,0.3538,370,6756,6757],[6759,0.145,370,6756,6757],[6760,0.33659999999999995,370,6756,6757],[6761,0.2531,370,6756,6757],[6762,0.1376,370,6756,6757],[6763,0.2973,370,6756,6757],[6764,0.30469999999999997,370,6756,6757],[6765,0.1941,370,6766,6757],[6765,0.2236,370,6767,6757],[6768,0.5283,370,6756,6757],[6769,0.2973,370,6756,6757],[6770,0.4644,370,6756,6757],[6771,0.38079999999999997,370,6756,6757],[5300,0.5528,370,6756,6757],[3396,0.6658,370,6756,6757]]},{"id":"pangaea-ai4farms-miou-2026-paper","name":"Pangaea-Latents (AI4Farms mIoU, 2026 paper)","group":"视觉与视频","metric":"mIoU","unit":"score","public_history":true,"protocol":{"en":"Mean intersection over union on AI4Farms. The other eight datasets are separate tasks and are not averaged in. Native 0–100 scale. This is not an accuracy percentage and is kept out of the saturation groups. GeoMixture-Naive, Sum and MoE are configurations of one encoder, not three models. The random or architecture baseline row is not a model checkpoint and is omitted.","zh":"AI4Farms 上的平均交并比。另外八个数据集是不同任务，不平均并入。 原生 0–100 尺度，不是准确率百分比，不进入饱和分组。 GeoMixture 的 Naive、Sum、MoE 是同一编码器的三种配置，不是三个模型。 随机行或架构基线不是模型检查点，不录入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.28313v1","status":"paper_baselines","published_at":"2026-10-20"},"score_max":100,"scale_label":{"en":"mIoU / 100","zh":"mIoU / 100"},"results":[[6772,27.59,370,6773,6774],[6775,27.11,370,6773,6774],[6776,26.87,370,6773,6774],[6777,22.52,370,6773,6774],[6778,26.98,370,6773,6774],[6779,27.05,370,6780,6774],[6779,27.05,370,6781,6774],[6779,26.95,370,6782,6774]]},{"id":"lift-bench-avg-all-2026-paper","name":"Lift-Bench (average of structured and flat, 2026 paper)","group":"指令与长文","metric":"Mean score","unit":"score","public_history":true,"protocol":{"en":"The paper average of the structured and flat means. Individual length splits are not mixed into another score.","zh":"论文给出的结构化平均和扁平平均的等权平均。各长度划分不另合成分数。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.28819v1","status":"paper_baselines","published_at":"2026-10-20"},"score_max":1,"saturation_excluded":true,"score_decimals":2,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"results":[[3196,0.84,370,6783,6784],[762,0.8,370,6783,6784],[6277,0.65,370,6783,6784],[3453,0.76,370,6783,6784],[3479,0.81,370,6783,6784],[6664,0.87,370,6783,6784],[3154,0.73,370,6783,6784],[2977,0.9,370,6783,6784],[6785,0.79,370,6783,6784]]},{"id":"omnilingual-msasr-es-parse-2026-paper","name":"Omnilingual-MsASR (Spanish parse rate, 2026 paper)","group":"其他","metric":"Parse rate","unit":"score","public_history":true,"protocol":{"en":"Spanish format-parse rate. Diarization error and the other five languages are not included. Native 0–100 scale. This is not an accuracy percentage and is kept out of the saturation groups.","zh":"西班牙语的格式解析率。说话人误差和其他五种语言不并入。 原生 0–100 尺度，不是准确率百分比，不进入饱和分组。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.29455v1","status":"paper_baselines","published_at":"2026-10-21"},"score_max":100,"scale_label":{"en":"Parse rate / 100","zh":"Parse rate / 100"},"results":[[6786,90.0,370,6787,6788],[6789,84.0,370,6787,6788],[6790,98.0,370,6787,6788],[6791,76.0,370,6787,6788],[6792,100.0,370,6787,6788],[6793,100.0,370,6787,6788],[6794,86.0,370,6787,6788],[6795,80.0,370,6787,6788],[6796,100.0,370,6787,6788]]},{"id":"cloudir-azure-bleu4-2026-paper","name":"CloudIR-Bench (Azure BLEU-4, top 1, 2026 paper)","group":"专业工作","metric":"BLEU-4","unit":"score","public_history":true,"protocol":{"en":"Top-1 BLEU-4 on the Azure incidents. Other datasets and ROUGE, METEOR and BERTScore are not included. Native 0–100 scale. This is not an accuracy percentage and is kept out of the saturation groups.","zh":"Azure 事件上的 Top-1 BLEU-4。其他数据集以及 ROUGE、METEOR、BERTScore 不并入。 原生 0–100 尺度，不是准确率百分比，不进入饱和分组。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.29185v1","status":"paper_baselines","published_at":"2026-10-21"},"score_max":100,"scale_label":{"en":"BLEU-4 / 100","zh":"BLEU-4 / 100"},"results":[[340,1.57,370,6797,6798],[345,2.25,370,6797,6798],[1623,1.41,370,6797,6798],[786,2.17,370,6797,6798],[793,1.15,370,6797,6798],[1822,1.61,370,6797,6798],[562,2.09,370,6797,6798],[580,0.69,370,6797,6798]]},{"id":"quant-arena-score-2026-paper","name":"Quant-Arena (score, 2026 paper)","group":"预测与决策","metric":"Score","unit":"score","public_history":true,"protocol":{"en":"Column S. Reasoning efforts are configurations of the same model. In, out and ARR columns are not included.","zh":"S 列。推理档位是同一模型的配置。内部、外部和 ARR 列不并入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.30113v1","status":"paper_baselines","published_at":"2026-10-21"},"score_max":100,"saturation_excluded":true,"score_decimals":2,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"results":[[2958,47.22,370,5015,6799],[2958,47.17,370,5017,6799],[2958,43.7,370,5010,6799],[2958,51.29,370,5011,6799],[2958,50.63,370,5009,6799],[3394,38.51,370,5015,6799],[3394,49.8,370,5017,6799],[3394,42.89,370,5010,6799],[3394,51.14,370,5011,6799],[3394,45.31,370,5009,6799],[2973,33.09,370,6800,6799],[2973,41.19,370,5015,6799],[2973,39.5,370,5017,6799],[2973,41.54,370,5010,6799],[2973,44.77,370,5011,6799],[2973,47.58,370,5009,6799],[2959,30.15,370,5015,6799],[2959,44.28,370,5017,6799],[2959,39.89,370,5010,6799],[2959,41.92,370,5011,6799],[2959,41.39,370,5009,6799],[2983,41.46,370,5015,6799],[2983,43.04,370,5017,6799],[2983,35.39,370,5010,6799],[2961,34.97,370,5015,6799],[2961,39.42,370,5017,6799],[2961,40.5,370,5010,6799],[2961,40.1,370,5011,6799],[2961,40.93,370,5009,6799],[3154,38.67,370,5010,6799],[3154,38.91,370,5009,6799],[2977,38.31,370,5015,6799],[2977,31.43,370,5010,6799],[2977,31.86,370,5009,6799],[3512,22.0,370,6800,6799],[3512,36.45,370,5015,6799],[3512,35.97,370,5010,6799],[3512,35.9,370,5009,6799]]},{"id":"code2video-t2v-rank-2026-paper","name":"Code2Video-Compass (T2V average rank, 2026 paper)","group":"生成与创作","metric":"Average rank","unit":"raw","public_history":true,"protocol":{"en":"Average rank on the T2V track only. Rank 1 is best. This is not an overall Compass score and not an accuracy percentage.","zh":"只收录 T2V 轨道的平均名次。名次 1 最好。这不是 Compass 总分，也不是准确率。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.28416v1","status":"paper_baselines","published_at":"2026-10-20"},"higher_is_better":false,"results":[[2959,1.6,370,6801,6802],[2961,1.9,370,6801,6802],[2960,2.6,370,6801,6802],[2958,4.7,370,6801,6802],[2983,5.7,370,6801,6802],[3157,6.25,370,6801,6802],[2976,7.4,370,6801,6802],[2970,10.6,370,6801,6802],[5899,12.8,370,6801,6802],[3153,7.35,370,6801,6802],[5335,8.6,370,6801,6802],[2977,9.3,370,6801,6802],[3396,12.9,370,6801,6802],[5300,13.3,370,6801,6802],[3491,15.0,370,6801,6802]]},{"id":"whatif-wql-skill-2026-paper","name":"WhatIf-Bench (wQL skill, 2026 paper)","group":"预测与决策","metric":"wQL skill","unit":"%","public_history":true,"protocol":{"en":"wQL skill is relative error reduction against the last-value reference, not task accuracy. Medium and High are configurations; the Naive reference is retained in the audit.","zh":"wQL skill 是相对末值基线的误差降低比例。Medium 和 High 是配置，Naive 参考行保留在审计记录中。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.26582v1","status":"paper_baselines","published_at":"2026-10-19"},"saturation_excluded":true,"scale_label":{"en":"wQL skill (%)","zh":"wQL 技能分（%）"},"results":[[3194,0.725,370,3138,6803],[3194,0.703,370,6623,6803],[502,0.7,370,3138,6803],[502,0.5820000000000001,370,6623,6803],[6804,0.645,370,6175,6803],[6805,0.616,370,6175,6803],[6806,0.613,370,6175,6803],[6807,0.5920000000000001,370,6175,6803],[6808,0.508,370,6175,6803],[6809,0.508,370,6175,6803],[6810,0.467,370,6175,6803],[6811,0.429,370,6175,6803],[6812,0.262,370,6175,6803],[6813,0.254,370,6175,6803]]},{"id":"stebbench-t1-accuracy-2026-paper","name":"StebBench (T1 accuracy, 2026 paper)","group":"知识与推理","metric":"T1 accuracy","unit":"%","public_history":true,"protocol":{"en":"Exact-match accuracy on diagnostic task T1. T2, T3, T4 and the full plan are not included. Temperature and think mode are configurations. The random or architecture baseline row is not a model checkpoint and is omitted.","zh":"诊断任务 T1 的精确匹配准确率。T2、T3、T4 和完整计划不并入。温度和思考模式是配置。 随机行或架构基线不是模型检查点，不录入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.26679v1","status":"paper_baselines","published_at":"2026-10-19"},"results":[[4048,0.757,370,6175,6814],[3424,0.7879999999999999,370,6815,6814],[3424,0.774,370,6816,6814],[6817,0.679,370,6175,6814],[2273,0.655,370,6175,6814],[1513,0.654,370,5020,6814],[1513,0.627,370,6175,6814],[4039,0.62,370,6175,6814],[6269,0.539,370,6175,6814],[6818,0.5429999999999999,370,6175,6814],[6268,0.282,370,6175,6814],[806,0.21600000000000003,370,6175,6814],[6267,0.424,370,6819,6814],[6267,0.254,370,6175,6814]]},{"id":"sham-sentence-f1-2026-paper","name":"SHAM-Bench (sentence-level F1, 2026 paper)","group":"知识与推理","metric":"Sentence-level F1","unit":"score","public_history":true,"protocol":{"en":"Sentence-level F1 on the printed 0–1 scale. Span-level F1 is a different task and is not included. This is not an accuracy percentage.","zh":"句子级 F1，按表中的 0–1 录入。跨度级 F1 是另一任务，不并入。这不是准确率百分比。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.27370v1","status":"paper_baselines","published_at":"2026-10-19"},"score_max":1,"scale_label":{"en":"Sentence-level F1 / 1","zh":"Sentence-level F1 / 1"},"score_decimals":3,"results":[[6820,0.632,370,6821,6822],[6823,0.595,370,6824,6822],[6825,0.484,370,6826,6822],[6827,0.638,370,6828,6822],[6829,0.638,370,6830,6822],[6831,0.54,370,6832,6822],[5505,0.691,370,6833,6822]]},{"id":"genolympiad-text-score-2026-paper","name":"GenOlympiad (text score, 2026 paper)","group":"生成与创作","metric":"Text score","unit":"score","public_history":true,"protocol":{"en":"Text score only. Fact, symbol, world and physics columns are not included. Input size is a configuration of the same model. Printed zeros are kept.","zh":"只收录文本分。事实、符号、世界和物理列不并入。输入尺寸是同一模型的配置。表上的 0 保留。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.26850v1","status":"paper_baselines","published_at":"2026-10-19"},"score_max":100,"scale_label":{"en":"Text score / 100","zh":"Text score / 100"},"results":[[2958,65.4,370,6834,6835],[762,62.5,370,6834,6835],[5004,62.2,370,6836,6835],[6837,26.9,370,6836,6835],[6366,41.5,370,6838,6835],[6380,30.4,370,6838,6835],[6381,19.0,370,6838,6835],[6839,19.5,370,6838,6835],[6368,20.8,370,6838,6835],[6367,10.1,370,6838,6835],[6365,7.8,370,6838,6835],[6840,7.1,370,6838,6835],[6841,11.4,370,6838,6835],[6842,3.0,370,6838,6835],[6843,1.4,370,6844,6835],[6845,0.0,370,6700,6835],[6845,0.0,370,6678,6835],[6846,1.7,370,6844,6835],[6847,1.6,370,6844,6835],[6848,0.0,370,6844,6835],[6849,0.0,370,6844,6835],[6850,0.0,370,6844,6835],[6851,0.0,370,6852,6835],[6851,0.0,370,6853,6835],[6851,1.5,370,6854,6835]]},{"id":"deepsea-math-en-2026-paper","name":"DeepSEA (English mathematical reasoning, 2026 paper)","group":"数学与科学","metric":"Mathematical reasoning accuracy","unit":"%","public_history":true,"protocol":{"en":"English mathematical-reasoning accuracy. High- and low-resource languages and the other tasks are not included.","zh":"英语数学推理准确率。高资源、低资源语言和其他任务不并入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.26155v1","status":"paper_baselines","published_at":"2026-10-19"},"results":[[5906,0.992,370,6855,6856],[6130,0.98,370,6855,6856],[6133,0.988,370,6855,6856],[5616,0.9640000000000001,370,6855,6856],[5789,0.988,370,6855,6856],[6689,0.96,370,6855,6856],[6668,0.9279999999999999,370,6855,6856],[6857,0.932,370,6855,6856],[6858,0.8759999999999999,370,6855,6856],[6859,0.96,370,6855,6856],[6860,0.9279999999999999,370,6855,6856]]},{"id":"perspectivebench-1hop-precision-2026-paper","name":"PerspectiveBench (1-hop precision, base, 2026 paper)","group":"知识与推理","metric":"Precision","unit":"%","public_history":true,"protocol":{"en":"Base-setting precision on 1-hop items. Oracle and the other hops are not included. Different parameter counts stay separate models.","zh":"基础设置下 1-hop 的精确率。Oracle 和其他跳数不并入。不同参数量仍是不同模型。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.25087v1","status":"paper_baselines","published_at":"2026-10-18"},"results":[[2058,0.4437,370,6861,6862],[345,0.515,370,6861,6862],[6704,0.4387,370,6861,6862],[6863,0.4325,370,6861,6862],[786,0.4667,370,6861,6862],[6301,0.525,370,6861,6862],[5344,0.4854,370,6861,6862],[6864,0.4354,370,6861,6862],[6865,0.4604,370,6861,6862],[6866,0.3063,370,6861,6862]]},{"id":"vista-judge-score-2026-paper","name":"VISTA (judge score, 2026 paper)","group":"视觉与视频","metric":"Judge score","unit":"score","public_history":true,"protocol":{"en":"Zero-shot judge score on the published 0–5 scale. Different parameter counts stay separate models. Rubric parts are not mixed in.","zh":"零样本评审分，按表中的 0–5 录入。不同参数量仍是不同模型。细则分项不并入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.22822v1","status":"paper_baselines","published_at":"2026-10-16"},"score_max":5,"scale_label":{"en":"Judge score / 5","zh":"Judge score / 5"},"score_decimals":2,"results":[[2780,2.39,370,5574,6867],[2766,2.75,370,5574,6867],[2728,3.16,370,5574,6867],[2731,3.13,370,5574,6867],[2697,3.33,370,5574,6867],[6349,1.76,370,5574,6867],[6341,2.65,370,5574,6867],[6298,3.22,370,5574,6867],[6342,3.31,370,5574,6867],[5815,2.73,370,5574,6867],[6868,3.07,370,5574,6867],[6869,3.06,370,5574,6867],[6870,3.42,370,5574,6867],[4219,3.75,370,5574,6867]]},{"id":"nmr-burden-accuracy-2026-paper","name":"NMR-Bench (burden counting accuracy, 2026 paper)","group":"视觉与视频","metric":"Accuracy","unit":"%","public_history":true,"protocol":{"en":"Overall accuracy on burden counting. Path tracing and MAE are not included. The printed 0.71 for ViGoRL is kept. The delta row is omitted.","zh":"计数负担的总准确率。路径追踪和 MAE 不并入。ViGoRL 的 0.71 保留。差值行不计入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.25247v1","status":"paper_baselines","published_at":"2026-10-18"},"results":[[3198,0.047599999999999996,370,6871,6872],[3194,0.0643,370,6871,6872],[3187,0.0929,370,6871,6872],[5139,0.1881,370,6871,6872],[3196,0.2214,370,6871,6872],[502,0.27140000000000003,370,6871,6872],[6873,0.0143,370,6871,6872],[6874,0.0070999999999999995,370,6871,6872],[1464,0.069,370,6871,6872],[6875,0.019,370,6871,6872],[6876,0.021400000000000002,370,6871,6872],[6877,0.035699999999999996,370,6871,6872],[6878,0.019,370,6871,6872],[6879,0.023799999999999998,370,6871,6872],[6880,0.0286,370,6871,6872],[6881,0.0738,370,6871,6872],[6882,0.6643000000000001,370,6871,6872]]},{"id":"membubble-distinct-n-2026-paper","name":"MemBubble (no memory, adapted distinct-n, 2026 paper)","group":"Agent 与工具","metric":"Adapted distinct-n","unit":"raw","public_history":true,"protocol":{"en":"Adapted distinct-n with no memory. Other memory conditions and class entropy are not included.","zh":"无记忆条件下的改编 distinct-n。其他记忆条件和类别熵不并入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.25127v1","status":"paper_baselines","published_at":"2026-10-18"},"results":[[6883,3.67,370,6884,6885],[5616,3.56,370,6884,6885],[325,3.58,370,6884,6885],[3386,2.3,370,6884,6885],[6886,3.05,370,6884,6885],[6887,2.82,370,6884,6885]]},{"id":"vtf900-frontal-auc-2026-paper","name":"VTF900 (frontal AUC, 2026 paper)","group":"视觉与视频","metric":"AUC","unit":"score","public_history":true,"protocol":{"en":"Frontal-to-gallery AUC on the published 0–100 scale. Equal-error rate and the other view are not included. This is not an accuracy percentage.","zh":"正脸到图库的 AUC，按表中的 0–100 录入。等错误率和其他视角不并入。这不是准确率百分比。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.24832v1","status":"paper_baselines","published_at":"2026-10-17"},"score_max":100,"scale_label":{"en":"AUC / 100","zh":"AUC / 100"},"results":[[6888,59.61,370,6889,6890],[6891,81.07,370,6889,6890],[6892,90.24,370,6889,6890],[6893,96.41,370,6889,6890],[6894,96.21,370,6889,6890],[6895,97.51,370,6889,6890],[6896,97.52,370,6889,6890]]},{"id":"senri-direct-total-2026-paper","name":"SENRI-Bench (direct total accuracy, 2026 paper)","group":"视觉与视频","metric":"Total accuracy","unit":"%","public_history":true,"protocol":{"en":"Total accuracy where the method and the inference model are the same system. Wrapper methods are not included.","zh":"方法名和推理模型相同的直接评测总准确率。外包一层的方法不并入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.25309v1","status":"paper_baselines","published_at":"2026-10-18"},"results":[[3412,0.474,370,5529,6897],[3415,0.498,370,5529,6897],[3386,0.522,370,5529,6897],[2058,0.47200000000000003,370,5529,6897],[3196,0.48200000000000004,370,5529,6897],[2977,0.414,370,5529,6897],[4070,0.41600000000000004,370,5529,6897],[2584,0.43799999999999994,370,5529,6897],[3763,0.43799999999999994,370,5529,6897],[3764,0.456,370,5529,6897],[5300,0.41,370,5529,6897]]},{"id":"stride-nuscenes-mcq-2026-paper","name":"STRIDE (nuScenes MCQ, 2026 paper)","group":"预测与决策","metric":"MCQ accuracy","unit":"%","public_history":true,"protocol":{"en":"Multiple-choice accuracy on the nuScenes split. Waymo, open-ended answers and displacement errors are not included. A missing MCQ cell is omitted.","zh":"nuScenes 划分上的选择题准确率。Waymo、开放问答和位移误差不并入。缺测的选择题空着，不补 0。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.22795v1","status":"paper_baselines","published_at":"2026-10-16"},"results":[[6898,0.185,370,6899,6900],[3763,0.175,370,6899,6900],[2626,0.23399999999999999,370,6899,6900],[5482,0.268,370,6899,6900],[3764,0.316,370,6899,6900],[5789,0.294,370,6899,6900],[4979,0.385,370,6899,6900],[502,0.37799999999999995,370,6899,6900],[5136,0.389,370,6899,6900],[3196,0.39299999999999996,370,6899,6900],[4977,0.455,370,6899,6900]]},{"id":"mobileinject-uc-2026-paper","name":"MobileInject (UC, 2026 paper)","group":"安全与可靠性","metric":"UC","unit":"%","public_history":true,"protocol":{"en":"UC under the MobAgent framework. The other safety columns are different metrics and are not included. Different model versions stay separate.","zh":"MobAgent 框架下的 UC。其他安全列是不同指标，不并入。不同模型版本仍分开。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.23307v1","status":"paper_baselines","published_at":"2026-10-16"},"results":[[6901,0.6667000000000001,370,6902,6903],[5722,0.44439999999999996,370,6902,6903],[2166,0.75,370,6902,6903],[5620,1.0,370,6902,6903],[6771,0.44439999999999996,370,6902,6903],[766,0.7778,370,6902,6903],[6133,0.8889,370,6902,6903],[345,1.0,370,6902,6903],[6904,0.6667000000000001,370,6902,6903]]},{"id":"visualtrust-noclaim-2026-paper","name":"VisualTrust (no claim, 2026 paper)","group":"视觉与视频","metric":"Accuracy","unit":"%","public_history":true,"protocol":{"en":"Accuracy when no claim is shown. Accurate-claim and misleading-claim conditions, and the trap-flip rate, are not included.","zh":"不展示主张时的准确率。准确主张、误导主张和陷阱翻转率不并入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.21159v1","status":"paper_baselines","published_at":"2026-10-15"},"results":[[4070,0.8909999999999999,370,6905,6906],[2584,0.925,370,6905,6906],[2590,0.9129999999999999,370,6905,6906],[2582,0.9279999999999999,370,6905,6906],[2576,0.9309999999999999,370,6905,6906],[5635,0.7859999999999999,370,6905,6906],[5480,0.8390000000000001,370,6905,6906],[6412,0.927,370,6905,6906],[4881,0.934,370,6905,6906],[2129,0.925,370,6905,6906],[6237,0.909,370,6905,6906]]},{"id":"gridbench-pred-accuracy-2026-paper","name":"Grid-Bench (prediction cell accuracy, 2026 paper)","group":"知识与推理","metric":"Cell accuracy","unit":"%","public_history":true,"protocol":{"en":"Prediction cell accuracy at T=64. Copy and zero columns are different conditions and are not included.","zh":"T=64 的预测格准确率。抄写列和零列是不同条件，不并入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.22312v1","status":"paper_baselines","published_at":"2026-10-15"},"results":[[762,0.816,370,6907,6908],[3628,0.905,370,6907,6908],[3494,0.561,370,6907,6908],[268,0.624,370,6907,6908],[244,0.556,370,6907,6908],[4424,0.524,370,6907,6908]]},{"id":"chemargus-full-problem-2026-paper","name":"ChemArgus (full problem, 2026 paper)","group":"数学与科学","metric":"Full-problem score","unit":"score","public_history":true,"protocol":{"en":"Full-problem rubric score. Sequential and oracle settings are not included. Input type is the configuration. A missing input is omitted.","zh":"完整题目的量表分。分步和提示脚手架不并入。输入类型是配置。没有该输入的不补 0。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.20012v1","status":"paper_baselines","published_at":"2026-10-14"},"score_max":100,"saturation_excluded":true,"score_decimals":2,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"results":[[4979,23.2,370,6909,6910],[2983,48.5,370,6909,6910],[755,53.8,370,6909,6910],[3157,29.3,370,6909,6910],[2977,27.5,370,6909,6910],[2974,33.8,370,6909,6910],[3491,27.3,370,6909,6910],[3492,25.0,370,6911,6910],[227,23.5,370,6911,6910],[4979,29.8,370,6911,6910],[2983,46.5,370,6911,6910],[755,61.5,370,6911,6910],[3157,36.0,370,6911,6910],[2977,34.5,370,6911,6910],[2974,33.4,370,6911,6910],[3491,26.2,370,6911,6910],[3492,36.7,370,6912,6910],[227,52.0,370,6912,6910],[4979,31.1,370,6912,6910],[2983,33.8,370,6912,6910],[755,46.2,370,6912,6910],[3157,27.9,370,6912,6910],[2977,32.3,370,6912,6910],[2974,28.6,370,6912,6910],[3491,27.2,370,6912,6910]]},{"id":"april-ssa-2026-paper","name":"APRiL (SSA, 2026 paper)","group":"知识与推理","metric":"SSA","unit":"%","public_history":true,"protocol":{"en":"Social-situation SSA. Thinking and box type are configurations. Average rows are omitted. Different parameter counts stay separate.","zh":"社会情境 SSA。思考与黑盒或白盒是配置。平均行不计入。不同参数量仍分开。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.20785v1","status":"paper_baselines","published_at":"2026-10-14"},"results":[[3748,0.8256999999999999,370,6913,6914],[6915,0.8359000000000001,370,6916,6914],[6915,0.8144,370,6917,6914],[2998,0.7492,370,6916,6914],[6918,0.7239,370,6916,6914],[6130,0.7698999999999999,370,6913,6914],[5519,0.7448999999999999,370,6913,6914],[2998,0.6931,370,6917,6914],[6918,0.6957,370,6917,6914],[6687,0.7918999999999999,370,6919,6914],[6682,0.7674,370,6919,6914],[6920,0.5603,370,6919,6914],[6301,0.5349,370,6919,6914],[5344,0.3392,370,6919,6914],[5126,0.38270000000000004,370,6919,6914]]},{"id":"lephil-support-court-2026-paper","name":"LePhilBench (support court, no profile, 2026 paper)","group":"知识与推理","metric":"Support court","unit":"%","public_history":true,"protocol":{"en":"Share of answers supporting the court when no profile is given. This is a stance share, not a capability score, and it is excluded from accuracy saturation. The dissent column and the overall row are not included.","zh":"不提供人物画像时，回答里支持法院的比例。这是立场比例，不是能力分数，不进入准确率饱和分组。异议列和总平均行不并入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.19427v1","status":"paper_baselines","published_at":"2026-10-14"},"saturation_excluded":true,"scale_label":{"en":"Court-support share","zh":"支持法院的比例"},"results":[[4979,0.9515,370,6921,6922],[5727,0.9273,370,6921,6922],[5616,0.9152,370,6921,6922],[755,0.9667,370,6921,6922],[3568,0.9091,370,6921,6922],[3202,0.9242,370,6921,6922],[4961,0.8515,370,6921,6922],[6923,0.7667,370,6921,6922],[3386,0.9484999999999999,370,6921,6922],[2979,0.9273,370,6921,6922]]},{"id":"linteval-pass1-2026-paper","name":"LintEval (Pass@1, 2026 paper)","group":"编程","metric":"Pass@1","unit":"%","public_history":true,"protocol":{"en":"Pass@1. Different coder sizes stay separate models. The copy and rule-based rows are the printed baselines.","zh":"Pass@1。不同参数量的编码器仍是不同模型。复制和规则修复是表上的基线。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.20027v1","status":"paper_baselines","published_at":"2026-10-14"},"results":[[340,0.6709999999999999,370,6924,6925],[4958,0.6829999999999999,370,6924,6925],[6926,0.5760000000000001,370,6924,6925],[5836,0.617,370,6924,6925],[6467,0.53,370,6924,6925],[6927,0.52,370,6924,6925],[6928,0.488,370,6924,6925],[613,0.38799999999999996,370,6924,6925],[623,0.337,370,6924,6925]]},{"id":"cswbench-focused-d1-2026-paper","name":"CSWBench (focused D1, 2026 paper)","group":"生成与创作","metric":"Focused D1","unit":"score","public_history":true,"protocol":{"en":"Focused-narrative dimension D1 on the printed 0–5 scale. Other dimensions and the extended narrative are not included.","zh":"聚焦叙事的 D1，按表中的 0–5 录入。其他维度和扩展叙事不并入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.20272v1","status":"paper_baselines","published_at":"2026-10-14"},"score_max":5,"scale_label":{"en":"Focused D1 / 5","zh":"Focused D1 / 5"},"score_decimals":3,"results":[[4979,4.998,370,6929,6930],[5727,4.997,370,6929,6930],[3196,4.999,370,6929,6930],[2129,4.995,370,6929,6930],[4961,4.989,370,6929,6930],[3202,4.99,370,6929,6930],[3185,4.996,370,6929,6930],[3201,4.992,370,6929,6930],[3005,4.941,370,6929,6930],[3190,4.938,370,6929,6930],[2576,4.992,370,6929,6930],[2590,4.985,370,6929,6930]]},{"id":"banglaplan-em-2026-paper","name":"BANGLAPLAN (exact match, 2026 paper)","group":"知识与推理","metric":"Exact match","unit":"%","public_history":true,"protocol":{"en":"Exact match on all 1,000 instances. Think mode is a configuration of the same 27B weights. The constant baseline is omitted.","zh":"全部 1000 条上的精确匹配。思考模式是同一 27B 权重的配置。恒定基线不计入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.19704v1","status":"paper_baselines","published_at":"2026-10-14"},"results":[[2976,0.851,370,6175,6931],[2960,0.987,370,6175,6931],[5828,0.033,370,6175,6931],[4969,0.141,370,6175,6931],[4070,0.196,370,6175,6931],[2584,0.309,370,6175,6931],[2584,0.8029999999999999,370,5020,6931],[4041,0.085,370,6175,6931],[6932,0.111,370,6175,6931],[6933,0.09699999999999999,370,6175,6931],[806,0.067,370,6175,6931]]},{"id":"swepatch-f1-2026-paper","name":"SWEPatchTest (patch-level F1, 2026 paper)","group":"编程","metric":"F1","unit":"score","public_history":true,"protocol":{"en":"Patch-level F1 on the published 0–100 scale. GRR, WDR and CAR are not included.","zh":"补丁级 F1，按表中的 0–100 录入。GRR、WDR 和 CAR 不并入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.18004v1","status":"paper_baselines","published_at":"2026-10-13"},"score_max":100,"scale_label":{"en":"F1 / 100","zh":"F1 / 100"},"score_decimals":3,"results":[[4048,64.08,370,6934,6935],[4981,71.37,370,6934,6935],[4983,75.13,370,6934,6935],[3155,53.95,370,6934,6935],[5790,71.38,370,6934,6935],[3185,61.77,370,6934,6935],[3201,57.42,370,6934,6935],[3154,65.81,370,6934,6935]]},{"id":"stou-action-2026-paper","name":"STOU (action accuracy, 2026 paper)","group":"安全与可靠性","metric":"Action accuracy","unit":"%","public_history":true,"protocol":{"en":"Action accuracy over all audio-to-text items. Harm is not included. Different parameter counts stay separate.","zh":"音频到文本全部题目的动作准确率。伤害列不并入。不同参数量仍分开。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.17728v1","status":"paper_baselines","published_at":"2026-10-13"},"results":[[864,0.565,370,6936,6937],[6574,0.7070000000000001,370,6936,6937],[6136,0.753,370,6936,6937],[6138,0.618,370,6936,6937],[6938,0.337,370,6936,6937],[6939,0.47700000000000004,370,6936,6937],[2627,0.76,370,6936,6937],[4056,0.8640000000000001,370,6936,6937],[3407,0.945,370,6936,6937]]},{"id":"mmta-agibot-swf1-2026-paper","name":"MMTA (AgiBot swF1, 2026 paper)","group":"视觉与视频","metric":"swF1","unit":"score","public_history":true,"protocol":{"en":"Sample-weighted F1 on AgiBot only, printed 0–1. Other datasets are not included. The blank Qwen3-VL-30B cell is omitted.","zh":"只收录 AgiBot 的样本加权 F1，0–1。其他数据集不并入。Qwen3-VL-30B 的空格不补 0。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.17856v1","status":"paper_baselines","published_at":"2026-10-13"},"score_max":1,"scale_label":{"en":"swF1 / 1","zh":"swF1 / 1"},"score_decimals":3,"results":[[2687,0.034,370,6940,6941],[2654,0.057,370,6940,6941],[6150,0.108,370,6940,6941],[6138,0.074,370,6940,6941],[6942,0.052,370,6940,6941],[4064,0.116,370,6940,6941],[6943,0.126,370,6940,6941],[6944,0.844,370,6940,6941]]},{"id":"cseg-exact-2026-paper","name":"CSEG-Bench (exact match, 2026 paper)","group":"知识与推理","metric":"Exact match","unit":"score","public_history":true,"protocol":{"en":"Exact match on the printed 0–1 scale. Think effort is a configuration. Different parameter counts stay separate.","zh":"精确匹配，0–1。思考档位是配置。不同参数量仍分开。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.17827v1","status":"paper_baselines","published_at":"2026-10-13"},"score_max":1,"scale_label":{"en":"Exact match / 1","zh":"Exact match / 1"},"score_decimals":4,"results":[[5126,0.3828,370,5010,6945],[5344,0.4233,370,5010,6945],[5343,0.494,370,5015,6945],[5343,0.5293,370,5017,6945],[5343,0.4164,370,5010,6945],[5479,0.4595,370,5010,6945],[5480,0.3009,370,6800,6945],[5480,0.219,370,5010,6945],[6946,0.2491,370,6800,6945],[6946,0.5233,370,5010,6945],[3419,0.5957,370,5015,6945],[3419,0.525,370,5017,6945],[3419,0.5009,370,5010,6945],[3201,0.3931,370,6800,6945],[3201,0.6422,370,5010,6945],[5114,0.6388,370,6800,6945],[5114,0.7543,370,5015,6945],[5114,0.7569,370,5010,6945],[3396,0.8664,370,5009,6945],[227,0.6371,370,6800,6945],[3185,0.5155,370,6800,6945],[3185,0.7595,370,5009,6945],[6947,0.5466,370,5020,6945],[6948,0.7353,370,5020,6945],[6949,0.8388,370,5020,6945],[6950,0.5647,370,5020,6945],[6951,0.7672,370,5020,6945],[6952,0.8328,370,5020,6945]]},{"id":"medharness-healthadmin-pass3-2026-paper","name":"MedHarnessBench (HealthAdmin Pass@3, 2026 paper)","group":"医疗","metric":"Pass@3","unit":"score","public_history":true,"protocol":{"en":"Native Pass@3 on HealthAdminBench only, printed 0–1. PhysicianBench is not included.","zh":"只收录 HealthAdminBench 的原生 Pass@3，0–1。PhysicianBench 不并入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.15714v1","status":"paper_baselines","published_at":"2026-10-11"},"score_max":1,"scale_label":{"en":"Pass@3 / 1","zh":"Pass@3 / 1"},"score_decimals":3,"results":[[3568,0.341,370,6953,6954],[3184,0.267,370,6953,6954],[3202,0.252,370,6953,6954],[3196,0.244,370,6953,6954],[3411,0.244,370,6953,6954],[3198,0.222,370,6953,6954],[3201,0.193,370,6953,6954],[3424,0.156,370,6953,6954]]},{"id":"describeevent-github-f1-2026-paper","name":"DescribeEvent (GitHub repo semantic F1, 2026 paper)","group":"知识与推理","metric":"F1","unit":"score","public_history":true,"protocol":{"en":"Semantic-event F1 on GitHub repositories, printed 0–1. Symbolic events and other datasets are not included.","zh":"GitHub 仓库的语义事件 F1，0–1。符号事件和其他数据集不并入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.19021v1","status":"paper_baselines","published_at":"2026-10-13"},"score_max":1,"scale_label":{"en":"F1 / 1","zh":"F1 / 1"},"score_decimals":3,"results":[[4969,0.226,370,6955,6956],[4070,0.29,370,6955,6956],[2584,0.349,370,6955,6956],[5505,0.332,370,6955,6956],[6709,0.342,370,6955,6956],[6957,0.351,370,6955,6956]]},{"id":"linguaguard-bacc-2026-paper","name":"LinguaGuardBench (balanced accuracy, 2026 paper)","group":"安全与可靠性","metric":"Balanced accuracy","unit":"score","score_max":1,"score_decimals":3,"public_history":true,"scale_label":{"en":"Balanced accuracy / 1","zh":"平衡准确率 / 1"},"protocol":{"en":"Original-wording balanced accuracy. Scanner control is the configuration named in the Gemini row. Gemma 4 E2B is omitted because the paper excludes it. ShieldAgent, AgentDoG and Safiron are omitted because the paper says their step-label rates are not comparable.","zh":"原词操作点的平衡准确率。扫描对照是 Gemini 那一行括号里的配置。Gemma 4 E2B 按论文排除。ShieldAgent、AgentDoG 和 Safiron 因论文写明与逐步标签不可比而不并入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.22244v1","status":"paper_baselines","published_at":"2026-10-15"},"results":[[2983,0.983,370,6958,6959],[2983,0.983,370,6960,6959],[3568,0.958,370,5490,6959],[6961,0.91,370,6962,6959],[62,0.899,370,6963,6959],[6585,0.849,370,6964,6959],[3654,0.842,370,5490,6959],[5934,0.829,370,6963,6959],[3403,0.825,370,6963,6959],[6965,0.82,370,6963,6959],[362,0.818,370,6963,6959],[5674,0.809,370,6964,6959],[5672,0.801,370,6964,6959],[6966,0.737,370,6964,6959],[6967,0.684,370,6964,6959]]},{"id":"threedverifier-overall-ba-2026-paper","name":"3DVerifierBench (overall binary balanced accuracy, 2026 paper)","group":"视觉与视频","metric":"Balanced accuracy","unit":"%","public_history":true,"protocol":{"en":"Overall column. The other columns are separate tasks. Motion is the only pair: its first cell is four-class exact match and its second cell is binary balanced accuracy. Human and the random row are omitted. A dash is omitted.","zh":"总列。其他列是分开的任务。只有 Motion 跨两列：前一格是四分类精确匹配，后一格是二分类平衡准确率。人类和随机行不计入。破折号不补 0。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.22360v1","status":"paper_baselines","published_at":"2026-10-15"},"results":[[2958,0.6940000000000001,370,6713,6968],[5519,0.684,370,6713,6968],[4979,0.649,370,6713,6968],[5480,0.626,370,6713,6968],[2582,0.616,370,6713,6968],[3764,0.595,370,6713,6968],[2642,0.5670000000000001,370,6713,6968],[6969,0.55,370,6713,6968],[6970,0.52,370,6713,6968],[2471,0.502,370,6713,6968]]},{"id":"multimodal-sea-zh-d2-2026-paper","name":"Multimodal-SEA (Chinese D2, verification coverage, 2026 paper)","group":"安全与可靠性","metric":"Verification coverage","unit":"%","public_history":true,"protocol":{"en":"Verification coverage on the Chinese task set. D1, D3, D4, D5 and D6 are different metrics and are not included. SP is a configuration of the same model.","zh":"中文任务集上的核验覆盖率。D1、D3、D4、D5、D6 是不同指标，不并入。SP 是同一模型的配置。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.08281v1","status":"paper_baselines","published_at":"2026-11-05"},"results":[[755,0.6559999999999999,370,6175,6971],[2983,0.701,370,6175,6971],[3198,0.509,370,6175,6971],[2977,0.607,370,6175,6971],[5598,0.8190000000000001,370,6175,6971],[4979,0.535,370,6175,6971],[3529,0.772,370,6175,6971],[3198,0.613,370,6972,6971],[4979,0.7809999999999999,370,6972,6971]]},{"id":"multimodal-sea-en-d2-2026-paper","name":"Multimodal-SEA (English D2, verification coverage, 2026 paper)","group":"安全与可靠性","metric":"Verification coverage","unit":"%","public_history":true,"protocol":{"en":"Verification coverage on the English task set. The other D columns are not included. SP is a configuration of the same model.","zh":"英文任务集上的核验覆盖率。其他 D 列不并入。SP 是同一模型的配置。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.08281v1","status":"paper_baselines","published_at":"2026-11-05"},"results":[[755,0.638,370,6175,6971],[2983,0.7040000000000001,370,6175,6971],[3198,0.645,370,6175,6971],[2977,0.655,370,6175,6971],[5598,0.8,370,6175,6971],[4979,0.462,370,6175,6971],[3529,0.784,370,6175,6971],[3198,0.67,370,6972,6971],[4979,0.789,370,6972,6971]]},{"id":"scamworld-detected-2026-paper","name":"ScamWorld (detected rate, 2026 paper)","group":"安全与可靠性","metric":"Detected rate","unit":"%","public_history":true,"protocol":{"en":"Detected rate during task execution. Deceived rate is a separate lower-is-better column and is not included. Short names are paired with the logo in the same table cell and with the backbone sentence in section 4.1: openai.svg + “5.6 Luna” is GPT-5.6 Luna; claude-color.svg + “Haiku 5.5” is Claude Haiku 5.5; qwen-color.svg + “3.8 Flash” is Qwen3.8 Flash; kimi.svg + “K3” is Kimi K3; gemini-color.svg + “3.8 Flash” is Gemini 3.8 Flash; deepseek-color.svg + “V4.1 Flash” is DeepSeek V4.1 Flash.","zh":"任务执行中的发现率。受骗率是另一列、越低越好，不并入。短名与同一格的厂商标记以及 4.1 节骨干句对应：openai.svg 加“5.6 Luna”是 GPT-5.6 Luna；claude-color.svg 加“Haiku 5.5”是 Claude Haiku 5.5；qwen-color.svg 加“3.8 Flash”是 Qwen3.8 Flash；kimi.svg 加“K3”是 Kimi K3；gemini-color.svg 加“3.8 Flash”是 Gemini 3.8 Flash；deepseek-color.svg 加“V4.1 Flash”是 DeepSeek V4.1 Flash。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.04879v1","status":"paper_baselines","published_at":"2026-11-03"},"results":[[3408,0.79,370,6973,6974],[2970,0.79,370,6973,6974],[3260,0.81,370,6973,6974],[2977,0.76,370,6973,6974],[2983,0.72,370,6973,6974],[3512,0.65,370,6973,6974]]},{"id":"magar-overall-2026-paper","name":"MAGAR-Bench (overall, 2026 paper)","group":"Agent 与工具","metric":"Overall","unit":"score","public_history":true,"protocol":{"en":"Published overall score. Assembly, atomic-action and rework columns are not included. Dashes in other columns are not filled.","zh":"论文给出的总分。装配、原子动作和返工列不并入。其他列的破折号不补 0。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.02472v1","status":"paper_baselines","published_at":"2026-11-02"},"score_max":100,"saturation_excluded":true,"score_decimals":2,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"results":[[6150,14.05,370,6975,6976],[6977,3.16,370,6975,6976],[2774,10.46,370,6975,6976],[6632,10.79,370,6975,6976],[2911,9.32,370,6975,6976],[6389,4.53,370,6975,6976],[5725,14.67,370,6975,6976],[6978,17.33,370,6975,6976],[6235,10.3,370,6975,6976],[6979,14.59,370,6975,6976],[766,19.62,370,6975,6976],[6980,12.7,370,6975,6976],[5487,8.51,370,6975,6976],[6981,41.39,370,6975,6976]]},{"id":"plancontext-pdms-2026-paper","name":"PlanContext (PDMS, 2026 paper)","group":"预测与决策","metric":"PDMS","unit":"score","public_history":true,"protocol":{"en":"PDMS only, on the published 0–100 score. The other driving columns are not included. This is not an accuracy percentage.","zh":"只收录 PDMS，按表中的 0–100。其他驾驶列不并入。这不是准确率百分比。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.03067v1","status":"paper_baselines","published_at":"2026-11-02"},"score_max":100,"scale_label":{"en":"PDMS / 100","zh":"PDMS / 100"},"score_decimals":2,"results":[[6982,93.96,370,6983,6984],[6985,88.31,370,6983,6984],[6986,83.39,370,6983,6984],[6987,92.22,370,6983,6984],[6988,88.05,370,6983,6984],[6989,65.52,370,6983,6984],[6990,94.48,370,6983,6984],[6991,90.95,370,6983,6984],[6992,88.92,370,6983,6984],[6993,95.21,370,6983,6984],[6994,86.0,370,6983,6984],[6995,90.46,370,6983,6984],[6996,85.69,370,6983,6984],[6997,92.06,370,6983,6984],[6998,87.02,370,6983,6984],[6999,88.56,370,6983,6984],[7000,89.2,370,6983,6984],[7001,90.27,370,6983,6984],[7002,90.87,370,6983,6984],[7003,88.22,370,6983,6984],[7004,90.97,370,6983,6984],[7005,93.69,370,6983,6984],[7006,87.2,370,6983,6984]]},{"id":"rescope-bp-2026-paper","name":"ReScopeBench (baseline performance, 2026 paper)","group":"Agent 与工具","metric":"Baseline performance","unit":"score","score_max":100,"score_decimals":2,"public_history":true,"scale_label":{"en":"Baseline performance / 100","zh":"基线表现 / 100"},"protocol":{"en":"Baseline performance without experience. The paper scores BP, EP, DP and the experience metrics from 0 to 100. The wide numeric range is that scale, not a mixture of units. EP, DP and coverage columns are not included.","zh":"无经验时的基线表现。论文把 BP、EP、DP 和经验指标都记为 0 到 100。数值跨度大仍是这一尺度。EP、DP 和覆盖率列不并入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.05450v1","status":"paper_baselines","published_at":"2026-11-03"},"results":[[4070,0.51,195,7007,7008],[2590,0.82,195,7007,7008],[5300,20.07,195,7007,7008],[3184,0.09,15,7007,7008],[3198,5.61,775,7007,7008],[2976,20.83,15,7007,7008],[3196,20.55,15,7007,7008],[3204,18.2,3026,7007,7008],[3155,22.23,396,7007,7008],[3404,19.67,25,7007,7008],[3396,22.35,48,7007,7008],[2960,24.84,15,7007,7008],[5519,25.7,83,7007,7008],[5335,24.6,2972,7007,7008],[3153,19.98,25,7007,7008],[3157,24.09,195,7007,7008],[3154,23.78,48,7007,7008],[2958,25.86,15,7007,7008],[2959,21.94,1,7007,7008],[5677,25.86,83,7007,7008]]},{"id":"babytheorist-single-a-2026-paper","name":"BabyTheorist (Single-A score, 2026 paper)","group":"知识与推理","metric":"Score","unit":"raw","public_history":true,"protocol":{"en":"Single-A score from the appendix table. The score sums 100 times exact-match accuracy over reached levels and can exceed 100. The other three modes are not included. A dash is omitted.","zh":"附录表的 Single-A 分数。它把到达层的精确匹配准确率乘以 100 后相加，可以超过 100。其他三种模式不并入。破折号不补 0。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.08177v1","status":"paper_baselines","published_at":"2026-11-05"},"results":[[6278,59.0,370,7009,7010],[3751,8.0,370,7009,7010],[7011,6.7,370,7009,7010],[7012,15.9,370,7009,7010],[7013,52.5,370,7009,7010],[7014,16.8,370,7009,7010],[7015,136.8,370,7009,7010]]},{"id":"gitrace-pgs-2026-paper","name":"GI-TraceBench (PGS, 2026 paper)","group":"知识与推理","metric":"PGS","unit":"score","public_history":true,"protocol":{"en":"PGS on the paper’s 0–100 scale. The other columns, including the 0–5 judge, are not included.","zh":"PGS，按论文的 0–100。其他列和 0–5 的评审分不并入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.02545v1","status":"paper_baselines","published_at":"2026-11-02"},"score_max":100,"scale_label":{"en":"PGS / 100","zh":"PGS / 100"},"score_decimals":2,"results":[[7016,10.29,370,6175,7017],[5535,12.55,370,6175,7017],[7018,0.97,370,6175,7017],[7019,12.41,370,6175,7017],[562,11.08,370,6175,7017],[7020,4.16,370,6175,7017],[4070,8.88,370,6175,7017],[3572,21.57,370,6175,7017],[3415,16.41,370,6175,7017],[4958,10.28,370,6175,7017],[3189,10.08,370,6175,7017],[7021,9.12,370,6175,7017],[7022,7.67,370,6175,7017],[7023,9.38,370,6175,7017],[7024,10.05,370,6175,7017],[7025,72.72,370,6175,7017],[7026,73.17,370,6175,7017],[7027,79.55,370,6175,7017],[7028,84.03,370,6175,7017]]},{"id":"waypop-unknown-pq-2026-paper","name":"WayPOP (unknown PQ, Objaverse v1.0, 2026 paper)","group":"视觉与视频","metric":"Unknown PQ","unit":"score","score_max":100,"score_decimals":1,"public_history":true,"scale_label":{"en":"Unknown PQ / 100","zh":"未知类 PQ / 100"},"protocol":{"en":"Unknown panoptic quality on the Objaverse v1.0 / WayPOP-v1 split, from PDF Tables 2 and 3. Known PQ and the XL split are not included. Printed values below 1 are kept. This is not an accuracy percentage.","zh":"Objaverse v1.0 / WayPOP-v1 上的未知类全景质量，取自 PDF 表 2 和表 3。已知类和 XL 划分不并入。小于 1 的印刷值保留。这不是准确率百分比。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.05801v1","status":"paper_baselines","published_at":"2026-11-04"},"results":[[7029,0.1,370,7030,7031],[7032,0.2,370,7033,7031],[7034,0.7,370,7033,7031],[7035,1.0,370,7036,7031],[7037,30.9,370,7038,7031],[7039,9.3,370,6175,7031],[7040,27.5,370,6175,7031],[7041,31.9,370,6175,7031],[7042,34.6,370,6175,7031]]},{"id":"cosmio-euclid-roman-psnr-2026-paper","name":"COSMIO (Euclid to Roman PSNR, 2026 paper)","group":"视觉与视频","metric":"PSNR","unit":"raw","public_history":true,"protocol":{"en":"PSNR in decibels for Euclid-to-Roman imputation only. Other survey pairs are separate. RMSE and CRPS are not included. A slash is not stored as zero.","zh":"只收录 Euclid 到 Roman 的 PSNR，单位是分贝。其他巡天对另计。RMSE 和 CRPS 不并入。斜杠不记成 0。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.04139v1","status":"paper_baselines","published_at":"2026-11-03"},"results":[[7043,33.758,370,7044,7045],[7046,36.838,370,7044,7045],[7047,34.533,370,7044,7045],[7048,34.153,370,7044,7045],[7049,30.157,370,7044,7045],[7050,33.567,370,7044,7045],[7051,35.727,370,7044,7045],[7052,37.423,370,7044,7045],[7053,35.507,370,7044,7045],[7054,38.737,370,7044,7045],[7055,39.19,370,7044,7045]]},{"id":"serverbench-gemma-e4b-dollars-2026-paper","name":"ServerBench (Gemma E4B dollars per dollar, 2026 paper)","group":"编程","metric":"Dollars per dollar","unit":"raw","public_history":true,"protocol":{"en":"Latency-qualified dollars per dollar on the Gemma E4B, 1×L40 deployment. The rows are the coding-agent models named in the abstract and section 4.1, not serving engines. Gemma E4B is the model being served. vLLM and SGLang are baselines and are not rows. The other two deployments are not included. Not passed is omitted.","zh":"Gemma E4B、1×L40 上的延迟合格美元比。行是摘要和 4.1 节写明的编程智能体模型，不是推理服务引擎。Gemma E4B 是被服务的模型。vLLM 和 SGLang 是基线，不是这些行。另外两个部署不并入。Not passed 不记成 0。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.06841v1","status":"paper_baselines","published_at":"2026-11-04"},"results":[[2959,2.47,370,7056,7057],[2961,2.97,370,7056,7057],[5508,1.75,370,7056,7057],[5506,1.59,370,7056,7057],[755,1.15,370,7056,7057],[7058,0.55,370,7056,7057],[2977,0.52,370,7056,7057],[3465,0.87,370,7056,7057],[2971,0.22,370,7056,7057]]},{"id":"motoverse-erma-full-2026-paper","name":"MotoVerse (ERMA full-set accuracy, 2026 paper)","group":"视觉与视频","metric":"Accuracy","unit":"%","public_history":true,"protocol":{"en":"ERMA accuracy on the full test set at τ=0.5. Trajectory error is a different metric. Alpamayo-1.5-10B is omitted because the table scores only its parsed answers, not the full set. Different parameter counts stay separate models.","zh":"τ=0.5 时完整测试集上的 ERMA 准确率。轨迹误差是另一指标。Alpamayo-1.5-10B 只统计可解析答案，不并入完整集。不同参数量仍分开。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.06007v1","status":"paper_baselines","published_at":"2026-11-04"},"results":[[7059,0.452,370,7060,7061],[7062,0.5329999999999999,370,7060,7061],[7063,0.467,370,7060,7061],[7064,0.489,370,7060,7061],[2797,0.322,370,7060,7061],[7065,0.322,370,7060,7061],[6341,0.3,370,7060,7061],[6298,0.341,370,7060,7061],[7066,0.40700000000000003,370,7060,7061],[7067,0.337,370,7060,7061],[7068,0.226,370,7060,7061]]},{"id":"pixelbench-t2p-top1-16-2026-paper","name":"PixelBench (text-to-pixel Top-1, 16², 2026 paper)","group":"视觉与视频","metric":"Top-1 accuracy","unit":"%","public_history":true,"protocol":{"en":"500-way Top-1 at 16² for text-to-pixel generation. The published aggregate of six columns is not used. Other resolutions and the judge score are not included.","zh":"文生像素在 16² 上的 500 选 1。不用论文对六列做的平均。其他分辨率和评审分不并入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.02540v1","status":"paper_baselines","published_at":"2026-11-02"},"results":[[2958,0.87,370,7069,7070],[755,0.81,370,7069,7070],[762,0.84,370,7069,7070],[2977,0.81,370,7069,7070],[3405,0.87,370,7069,7070],[7071,0.7,370,7069,7070],[3492,0.49,370,7069,7070]]},{"id":"geobiaset-street-1km-2026-paper","name":"GeoBiaset (original, street 1 km, 2026 paper)","group":"视觉与视频","metric":"Accuracy","unit":"%","public_history":true,"protocol":{"en":"Street-level accuracy within 1 km on original backgrounds. Edited images are a separate condition. Section 4.2 names the four MLLMs: DeepSeek-VL-7B, InternVL3.5-8B-Instruct, Qwen3-VL-8B-Instruct and GLM-4.1V-9B. Larger checkpoints in the appendix are not mixed in. POPI is lower-is-better and is not included.","zh":"原图、1 公里街级准确率。编辑图是另一种条件。4.2 节写明四个多模态模型是 DeepSeek-VL-7B、InternVL3.5-8B-Instruct、Qwen3-VL-8B-Instruct 和 GLM-4.1V-9B。附录里更大的检查点不并入。POPI 越低越好，不并入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.06085v1","status":"paper_baselines","published_at":"2026-11-04"},"results":[[7072,0.111,370,7073,7074],[7075,0.08199999999999999,370,7073,7074],[7076,0.166,370,7073,7074],[7077,0.009000000000000001,370,7073,7074],[6144,0.009000000000000001,370,7073,7074],[6150,0.032,370,7073,7074],[6296,0.035,370,7073,7074]]},{"id":"vipbench-task1-r-2026-paper","name":"VIPBench (Task 1 Pearson r, 2026 paper)","group":"其他","metric":"Pearson r","unit":"score","score_max":1,"score_decimals":3,"public_history":true,"scale_label":{"en":"Pearson r / 1","zh":"Pearson r / 1"},"protocol":{"en":"Pearson r on Task 1, continuous same-speaker prediction. AUC and the other tasks are not included. This is a correlation, not an accuracy percentage.","zh":"任务 1 上的 Pearson r，连续的同说话人预测。AUC 和其他任务不并入。这是相关，不是准确率百分比。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.05638v1","status":"paper_baselines","published_at":"2026-11-03"},"results":[[7078,0.656,370,7079,7080],[7081,0.646,370,7079,7080],[7082,0.638,370,7079,7080],[7083,0.627,370,7079,7080],[7084,0.617,370,7079,7080],[7085,0.597,370,7079,7080],[7086,0.51,370,7079,7080],[7087,0.471,370,7079,7080],[7088,0.465,370,7079,7080],[7089,0.458,370,7079,7080],[7090,0.226,370,7079,7080]]},{"id":"ideataste-match-cite-2026-paper","name":"IdeaTaste (Match-Cite, 2026 paper)","group":"知识与推理","metric":"Match-Cite","unit":"%","public_history":true,"protocol":{"en":"Match-Cite from Table 2 for all four areas and the four agent methods. Match-Idea, Diversity and the human row are not included. No average across areas was added.","zh":"表 2 四个领域、四种方法的 Match-Cite。Match-Idea、Diversity 和人类行不并入。没有另算领域平均。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.03424v1","status":"paper_baselines","published_at":"2026-11-02"},"results":[[363,0.0344,370,7091,7092],[363,0.0387,370,7093,7092],[363,0.025099999999999997,370,7094,7092],[363,0.0276,370,7095,7092],[363,0.027200000000000002,370,7096,7092],[363,0.0329,370,7097,7092],[363,0.021,370,7098,7092],[363,0.031400000000000004,370,7099,7092],[363,0.0265,370,7100,7092],[363,0.0342,370,7101,7092],[363,0.0191,370,7102,7092],[363,0.0395,370,7103,7092],[363,0.0642,370,7104,7092],[363,0.054400000000000004,370,7105,7092],[363,0.051399999999999994,370,7106,7092],[363,0.0471,370,7107,7092],[325,0.044800000000000006,370,7091,7092],[325,0.0424,370,7093,7092],[325,0.0397,370,7094,7092],[325,0.0441,370,7095,7092],[325,0.039,370,7096,7092],[325,0.04,370,7097,7092],[325,0.0371,370,7098,7092],[325,0.039,370,7099,7092],[325,0.0437,370,7100,7092],[325,0.049400000000000006,370,7101,7092],[325,0.0429,370,7102,7092],[325,0.050300000000000004,370,7103,7092],[325,0.058600000000000006,370,7104,7092],[325,0.061399999999999996,370,7105,7092],[325,0.057999999999999996,370,7106,7092],[325,0.060700000000000004,370,7107,7092],[4035,0.0373,370,7091,7092],[4035,0.045899999999999996,370,7093,7092],[4035,0.036699999999999997,370,7094,7092],[4035,0.04650000000000001,370,7095,7092],[4035,0.037200000000000004,370,7096,7092],[4035,0.042300000000000004,370,7097,7092],[4035,0.0388,370,7098,7092],[4035,0.042300000000000004,370,7099,7092],[4035,0.0489,370,7100,7092],[4035,0.0533,370,7101,7092],[4035,0.0444,370,7102,7092],[4035,0.0528,370,7103,7092],[4035,0.0718,370,7104,7092],[4035,0.0658,370,7105,7092],[4035,0.0718,370,7106,7092],[4035,0.0634,370,7107,7092],[3201,0.0394,370,7091,7092],[3201,0.0463,370,7093,7092],[3201,0.0353,370,7094,7092],[3201,0.0478,370,7095,7092],[3201,0.0351,370,7096,7092],[3201,0.0463,370,7097,7092],[3201,0.0362,370,7098,7092],[3201,0.0458,370,7099,7092],[3201,0.04769999999999999,370,7100,7092],[3201,0.0516,370,7101,7092],[3201,0.042199999999999994,370,7102,7092],[3201,0.049800000000000004,370,7103,7092],[3201,0.064,370,7104,7092],[3201,0.07719999999999999,370,7105,7092],[3201,0.0533,370,7106,7092],[3201,0.0702,370,7107,7092],[5300,0.039599999999999996,370,7091,7092],[5300,0.0489,370,7093,7092],[5300,0.037000000000000005,370,7094,7092],[5300,0.049400000000000006,370,7095,7092],[5300,0.0444,370,7096,7092],[5300,0.0421,370,7097,7092],[5300,0.0347,370,7098,7092],[5300,0.050499999999999996,370,7099,7092],[5300,0.053200000000000004,370,7100,7092],[5300,0.0559,370,7101,7092],[5300,0.0473,370,7102,7092],[5300,0.0591,370,7103,7092],[5300,0.063,370,7104,7092],[5300,0.0717,370,7105,7092],[5300,0.0658,370,7106,7092],[5300,0.07719999999999999,370,7107,7092],[7108,0.0421,370,7091,7092],[7108,0.0444,370,7093,7092],[7108,0.0409,370,7094,7092],[7108,0.045899999999999996,370,7095,7092],[7108,0.0382,370,7096,7092],[7108,0.045599999999999995,370,7097,7092],[7108,0.0373,370,7098,7092],[7108,0.0424,370,7099,7092],[7108,0.0501,370,7100,7092],[7108,0.049699999999999994,370,7101,7092],[7108,0.0451,370,7102,7092],[7108,0.050499999999999996,370,7103,7092],[7108,0.0721,370,7104,7092],[7108,0.0704,370,7105,7092],[7108,0.0675,370,7106,7092],[7108,0.0682,370,7107,7092]]},{"id":"egoaction-score-2026-paper","name":"EgoActionBench (EgoActionScore, 2026 paper)","group":"视觉与视频","metric":"EgoActionScore","unit":"score","public_history":true,"protocol":{"en":"EgoActionScore from Table 3. The paper defines it as the mean of Temporal Overall and Semantic Overall, on a 0–100 scale. The human expert row is not included. Component columns are not included.","zh":"表 3 的 EgoActionScore。原文把它定义为时间总分和语义总分的平均，量程 0 到 100。人类专家行不并入。分项列不并入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.03646v1","status":"paper_baselines","published_at":"2026-11-02"},"score_max":100,"score_decimals":1,"results":[[762,49.8,370,7109,7110],[2958,59.6,370,7109,7110],[3482,34.8,370,7109,7110],[2444,24.7,370,7109,7110],[6625,29.2,370,7109,7110],[6150,25.2,370,7109,7110],[2654,20.4,370,7109,7110],[7111,19.4,370,7109,7110],[6349,14.2,370,7109,7110],[6149,25.9,370,7109,7110],[5725,18.8,370,7109,7110],[7112,24.7,370,7109,7110],[7113,10.8,370,7109,7110],[7114,32.4,370,7109,7110],[7115,20.0,370,7109,7110],[7116,23.0,370,7109,7110],[6352,20.8,370,7109,7110],[6350,15.4,370,7109,7110],[7117,13.9,370,7109,7110],[7118,37.8,370,7109,7110],[7119,47.9,370,7109,7110],[7120,41.8,370,7109,7110]]},{"id":"cast-single-overall-2026-paper","name":"CAST-Bench (single-shot overall, 2026 paper)","group":"生成与创作","metric":"Overall","unit":"score","public_history":true,"protocol":{"en":"The Overall column of the single-shot block in Table 2. The paper maps metrics onto 0–100 and then takes its weighted overall. The multi-shot block is a different subset and is not included. Component columns are not included.","zh":"表 2 单镜头块的 Overall 列。原文先把各项映到 0–100，再按固定权重合成。多镜头块是另一子集，不并入。分项列不并入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.05422v1","status":"paper_baselines","published_at":"2026-11-02"},"score_max":100,"score_decimals":2,"results":[[7121,78.11,370,7122,7123],[7124,78.03,370,7122,7123],[7125,77.97,370,7122,7123],[7126,77.41,370,7122,7123],[7127,74.52,370,7122,7123],[7128,73.66,370,7122,7123],[7129,73.64,370,7122,7123],[7130,70.43,370,7122,7123],[7131,69.81,370,7122,7123],[7132,68.74,370,7122,7123],[7133,67.75,370,7122,7123],[7134,62.34,370,7122,7123]]},{"id":"triphys-overall-acc-2026-paper","name":"TriPhysBench (overall ACC, 2026 paper)","group":"知识与推理","metric":"ACC","unit":"score","public_history":true,"protocol":{"en":"Overall ACC from Table 3. Section 4.1.1 defines ACC as the mean of question-level scores in [0, 1], with partial credit. SCS and OGS are not included. Adapted TriPhys checkpoints are not in this table.","zh":"表 3 的总体 ACC。4.1.1 节把它定义为题目得分在 0 到 1 上的平均，允许部分分。SCS 和 OGS 不并入。改编后的 TriPhys 检查点不在这张表里。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.01934v1","status":"paper_baselines","published_at":"2026-11-01"},"score_max":1,"score_decimals":4,"results":[[3196,0.4441,370,7135,7136],[5136,0.5038,370,7135,7136],[7137,0.3259,370,7135,7136],[4969,0.3922,370,7135,7136],[4070,0.4196,370,7135,7136],[3763,0.4378,370,7135,7136]]},{"id":"ai-video-detector-auroc-2026","name":"AI video detector benchmark (original AUROC, 2026)","group":"安全与可靠性","metric":"AUROC","unit":"score","public_history":true,"protocol":{"en":"AUROC on original video clips from the October 2026 video table. Messenger-copy AUROC is a separate condition. The image table is not included. AUROC is a 0–1 ranking score, not an accuracy percentage.","zh":"2026 年 10 月视频表里原片的 AUROC。信使压缩副本是另一种条件。图像表不并入。AUROC 是 0 到 1 的排序分，不是准确率。"},"source":{"name":"WasItAIGenerated","url":"https://www.wasitaigenerated.com/research/ai-image-video-detector-benchmark-2026","status":"paper_baselines","published_at":"2026-10-29"},"score_max":1,"score_decimals":3,"results":[[7138,0.987,370,7139,7140],[7141,0.869,370,7139,7140],[7142,0.839,370,7139,7140],[7143,0.813,370,7139,7140],[7144,0.77,370,7139,7140],[7145,0.677,370,7139,7140]]},{"id":"physcount-score-2026-paper","name":"PhysCount (Score, 2026 paper)","group":"视觉与视频","metric":"Score","unit":"score","public_history":true,"protocol":{"en":"Primary Score from Table 1, defined in equation 2 as 100 times the mean over groups of the product of per-family reliabilities. MAE and NAE are lower-is-better and are not included. The five-person human mean is not a model.","zh":"表 1 的主指标 Score。式 2 把它定义为各组内分项可靠度乘积的平均再乘 100。MAE 和 NAE 越低越好，不并入。五人人类均值不是模型。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.40530v1","status":"paper_baselines","published_at":"2026-10-29"},"score_max":100,"score_decimals":2,"results":[[3386,36.45,370,7146,7147],[762,21.73,370,7146,7147],[2958,20.77,370,7146,7147],[2973,20.42,370,7146,7147],[3408,16.87,370,7146,7147],[5300,28.84,370,7146,7147],[2576,27.95,370,7146,7147],[2582,26.03,370,7146,7147],[2590,25.7,370,7146,7147],[2584,26.32,370,7146,7147],[4070,23.54,370,7146,7147],[4969,18.57,370,7146,7147],[7148,26.32,370,7146,7147],[7149,23.79,370,7146,7147],[7150,19.03,370,7146,7147],[7151,2.93,370,7146,7147],[7152,4.76,370,7146,7147],[7153,21.04,370,7146,7147],[2642,15.55,370,7146,7147],[5534,17.76,370,7146,7147],[6345,15.2,370,7146,7147],[2654,14.29,370,7146,7147],[2687,9.53,370,7146,7147],[5479,29.61,370,7146,7147],[7154,20.74,370,7146,7147],[7155,17.43,370,7146,7147],[7156,14.9,370,7146,7147],[7157,12.4,370,7146,7147],[7158,14.69,370,7146,7147],[7159,12.08,370,7146,7147],[7160,9.27,370,7146,7147],[7161,14.93,370,7146,7147],[7162,3.8,370,7146,7147],[5812,14.96,370,7146,7147],[5725,13.13,370,7146,7147],[7163,13.08,370,7146,7147],[7164,11.25,370,7146,7147],[7165,10.51,370,7146,7147],[7166,9.02,370,7146,7147],[7167,5.87,370,7146,7147]]},{"id":"finforesight-overall-2026-paper","name":"FinForesight (T2 Overall, 2026 paper)","group":"预测与决策","metric":"Overall","unit":"score","public_history":true,"protocol":{"en":"Table 1 Overall on T2. The paper defines it as the unweighted mean of VAL, RANK, EVT and FUND. CF and CALC are excluded by that definition and are not added here. A harness is a configuration. A model absent from a harness is omitted.","zh":"表 1 的 T2 Overall。原文定义为 VAL、RANK、EVT、FUND 的不加权平均。CF 和 CALC 按这个定义不计入，这里也不另加。运行框架是配置。某个框架里没有的模型留空。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.01621v1","status":"paper_baselines","published_at":"2026-11-01"},"score_max":1,"score_decimals":4,"results":[[7168,0.591,370,7169,7170],[5406,0.6507,370,7169,7170],[1621,0.6665,370,7169,7170],[283,0.6797,370,7169,7170],[3261,0.6887,370,7169,7170],[4219,0.7049,370,7169,7170],[5406,0.6018,370,7171,7170],[7168,0.6089,370,7171,7170],[283,0.6807,370,7171,7170],[1621,0.6823,370,7171,7170],[3261,0.6827,370,7171,7170],[4219,0.6899,370,7171,7170],[7168,0.5603,370,7172,7170],[1621,0.6198,370,7172,7170],[283,0.6521,370,7172,7170],[5406,0.6566,370,7172,7170],[3261,0.7031,370,7172,7170],[4219,0.7099,370,7172,7170],[283,0.6292,370,3759,7170],[2261,0.6476,370,3759,7170],[5406,0.6189,370,2300,7170],[3184,0.6678,370,2300,7170]]},{"id":"astock-cr-2026-paper","name":"AStockArena (cumulative return, 2026 paper)","group":"预测与决策","metric":"Cumulative return","unit":"raw","public_history":true,"protocol":{"en":"Cumulative return from Table B6, in percent, including negative returns. GPT-5.1 made no trades; the paper records its return as zero. Market indexes and technical strategies in Table B6a are not models. Sharpe is not included.","zh":"表 B6 的累计收益，单位是百分比，包含负值。GPT-5.1 没有成交，原文把收益记为 0。表 B6a 的市场指数和技术策略不是模型。Sharpe 不并入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.00515v1","status":"paper_baselines","published_at":"2026-10-31"},"score_decimals":2,"results":[[3738,10.84,370,7173,7174],[7175,8.48,370,7173,7174],[4425,7.5,370,7173,7174],[3412,7.38,370,7173,7174],[325,6.0,370,7173,7174],[3752,5.46,370,7173,7174],[7176,4.53,370,7173,7174],[7177,2.64,370,7173,7174],[7178,1.41,370,7173,7174],[3748,0.0,370,7173,7174],[340,-1.43,370,7173,7174],[7179,-1.85,370,7173,7174],[2055,-3.59,370,7173,7174]]},{"id":"baoyan-agent-mean-2026","name":"Graduate admissions agents (mean score, 2026)","group":"专业工作","metric":"Mean score","unit":"score","public_history":true,"protocol":{"en":"Mean score out of 100 on 70 tasks. The agent client is the configuration. Claude Opus and Claude Sonnet are listed in the registry but have no row in this table, so they are omitted. The README says the eight runs are not fully uniform.","zh":"70 题的平均分，满分 100。智能体客户端是配置。注册表里的 Claude Opus 和 Claude Sonnet 没有出现在这张表，不补 0。README 写明这八次运行并不完全同一条件。"},"source":{"name":"Repository authors","url":"https://github.com/yeyuxin777/baoyan-agent-benchmark","status":"paper_baselines","published_at":"2026-10-12"},"score_max":100,"score_decimals":2,"results":[[7180,79.08,370,2300,7181],[7182,78.36,370,2300,7181],[3512,68.16,370,5950,7181],[3529,66.8,370,7183,7181],[3560,64.24,370,7184,7181],[2977,58.96,370,5381,7181],[2983,58.23,370,5162,7181],[3412,55.55,370,5162,7181]]},{"id":"llmdyno-score-2026","name":"LLMDyno (score, 2026)","group":"编程","metric":"Score","unit":"score","public_history":true,"protocol":{"en":"The Score column. Quantization and where experts run are configurations of the named checkpoint, not extra models. Correctness percent is a separate column and is not included.","zh":"Score 列。量化和专家放在 CPU 是具名检查点的配置，不是额外模型。正确率是另一列，不并入。"},"source":{"name":"Repository authors","url":"https://github.com/speak2jerome/llmdyno","status":"paper_baselines","published_at":"2026-10-25"},"score_max":100,"score_decimals":1,"results":[[2976,94.3,370,7185,7186],[2986,90.9,370,7187,7186],[2986,90.7,370,7188,7186],[2986,90.1,370,7189,7186],[7190,89.0,370,7191,7186],[5934,87.2,370,7187,7186],[7190,86.6,370,7188,7186],[5934,85.1,370,7188,7186],[6585,80.6,370,7192,7186],[6013,72.0,370,7192,7186],[6585,70.9,370,7193,7186],[4906,48.5,370,7194,7186],[2627,43.4,370,7192,7186],[2627,42.2,370,7193,7186],[2653,39.7,370,7192,7186]]},{"id":"dramavqa-mini-medium-2026-paper","name":"DramaVQA (Mini, Medium, 2026 paper)","group":"视觉与视频","metric":"Accuracy","unit":"%","public_history":true,"protocol":{"en":"Mini-set accuracy on the Medium track from Table 2. Full-set accuracy, in square brackets, and inference time are not included. Frame count and thinking are configurations. Language-only, random guess, and RAG wrappers are not separate models.","zh":"表 2 里 Mini 集、中等长度这一列的准确率。方括号里的全量准确率和括号里的时间不并入。帧数和思考开关是配置。纯语言、随机猜测和 RAG 包装不是另外的模型。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.00769v2","status":"paper_baselines","published_at":"2026-10-31"},"results":[[7195,0.732,370,7196,7197],[3196,0.672,370,7198,7197],[2129,0.775,370,7198,7197],[6969,0.51,370,7198,7197],[6969,0.5710000000000001,370,7196,7197],[7199,0.705,370,7196,7197],[6343,0.62,370,7196,7197],[2584,0.6940000000000001,370,7198,7197],[6342,0.409,370,7198,7197],[6342,0.542,370,7196,7197],[6343,0.51,370,7198,7197],[3763,0.6990000000000001,370,7198,7197],[6969,0.55,370,7200,7197],[6969,0.608,370,7201,7197],[7199,0.754,370,7201,7197],[6343,0.6609999999999999,370,7201,7197],[2584,0.722,370,7200,7197],[6342,0.41700000000000004,370,7200,7197],[6342,0.5720000000000001,370,7201,7197],[6343,0.5479999999999999,370,7200,7197],[3763,0.725,370,7200,7197],[7202,0.488,370,7196,7197],[7202,0.49,370,7201,7197],[7203,0.628,370,7204,7197],[7203,0.6609999999999999,370,7205,7197]]},{"id":"fontvibe-llm-overall-2026","name":"FontVibe LLM designer (overall, 2026-10)","group":"生成与创作","metric":"Overall","unit":"score","score_max":10,"score_decimals":2,"public_history":true,"protocol":{"en":"Overall from summary.csv of benchmark 2026-10-llm-designer, published 2026-10-30. It is 0.20 legibility + 0.35 lettering + 0.25 layout + 0.20 appeal, on a 1–10 scale. Unfinished runs are not filled with zero. The image-model sets have five and four models and are not included.","zh":"2026-10-30 发布的 2026-10-llm-designer 的 summary.csv 总分。它是可读性 0.20、字形 0.35、版面 0.25、吸引力 0.20，量程 1 到 10。没完成的运行不补 0。图像模型两组只有五个和四个，不并入。"},"source":{"name":"FontVibe","url":"https://doi.org/10.5281/zenodo.23803802","status":"paper_baselines","published_at":"2026-10-30"},"results":[[2959,7.97,370,7206,7207],[2976,7.69,370,7208,7207],[2973,7.58,370,7209,7207],[2962,7.56,370,7208,7207],[2961,7.43,370,7208,7207],[2960,7.35,370,7208,7207],[2983,7.33,370,7208,7207],[2964,7.08,370,7208,7207]]},{"id":"dgx-spark-fresh-pr-2026","name":"DGX Spark (Fresh-PR solo, 2026-10-25)","group":"编程","metric":"Fresh-PR","unit":"%","public_history":true,"protocol":{"en":"Fresh-PR solo from RESULTS.md dated 2026-10-25: tasks passed out of the same 18. The percent is that ratio. Laguna, Nemotron and Qwen3.6 are later rows in that file, each marked solo on 18 tasks, the same set as the 2026-10-21 deposit. SWE-bench Pro is a different 11-task subset; errored trials are not stored as zero. laguna-s-2.1-lcpp has only a 3-task bakeoff and is omitted.","zh":"RESULTS.md（数据截至 2026-10-25）的 Fresh-PR 单人：同一套 18 题的通过数。Laguna、Nemotron 和 Qwen3.6 是该文件里后补的行，都标明 18 题单人，和 10 月 21 日存档是同一题集。SWE-bench Pro 是另一套 11 题，出错的不记 0。laguna-s-2.1-lcpp 只有 3 题 bakeoff，不并入。"},"source":{"name":"Repository authors","url":"https://doi.org/10.5281/zenodo.23524590","status":"paper_baselines","published_at":"2026-10-25"},"results":[[7210,0.7222222222222223,370,7211,7212],[7213,0.4444444444444444,370,7214,7212],[7215,0.5555555555555556,370,7214,7212],[7216,0.4444444444444444,370,7214,7212],[6435,0.6666666666666667,370,7214,7212],[7217,0.7222222222222223,370,7218,7212],[7219,0.6111111111111112,370,7218,7212]]},{"id":"askstream-askf1-2026-paper","name":"AskStream (Ask-F1, multiple choice, 2026 paper)","group":"视觉与视频","metric":"Ask-F1","unit":"score","public_history":true,"protocol":{"en":"Ask-F1 from Table 2, multiple choice. Printed zeros are measured. Subtask columns are not included. The backbone named beside a streaming system is its configuration, not a second model.","zh":"表 2 选择题设定的 Ask-F1。印出的 0 是测到的。分项不并入。流式系统旁边的骨干是配置，不是第二个模型。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.38678v1","status":"paper_baselines","published_at":"2026-10-28"},"score_max":100,"saturation_excluded":true,"score_decimals":2,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"results":[[4979,20.5,370,7220,7221],[3568,22.48,370,7220,7221],[7222,52.18,370,7220,7221],[7223,14.66,370,7220,7221],[3490,16.52,370,7220,7221],[7224,17.6,370,7220,7221],[3153,19.69,370,7220,7221],[7225,0,370,7220,7221],[7226,0,370,7220,7221],[7227,0,370,7220,7221],[6574,0.12,370,7220,7221],[7228,4.18,370,7220,7221],[6660,4.91,370,7220,7221],[6571,7.22,370,7220,7221]]},{"id":"scigame-fixed-hist-esr-2026-paper","name":"SciGame (Fixed, historical identification ESR, 2026 paper)","group":"数学与科学","metric":"ESR","unit":"%","public_history":true,"protocol":{"en":"Episode success rate for method Fixed, Historical collection, Identification, original laws, from Table 2. Prior-only and Adaptive are other methods and are not included. Temporal and counterfactual cells are not included. A printed zero in those other cells is not copied here.","zh":"表 2 里 Fixed、历史集、识别、原定律的回合成功率。Prior-only 和 Adaptive 是别的方法，不并入。时间集和反事实格不并入。那些格里印出的 0 不抄到这里。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.37919v1","status":"paper_baselines","published_at":"2026-10-27"},"results":[[7229,0.917,370,7230,7231],[7232,0.7829999999999999,370,7230,7231],[5735,0.36700000000000005,370,7230,7231],[5734,0.8170000000000001,370,7230,7231],[3196,0.95,370,7230,7231],[3408,0.9,370,7230,7231],[2973,0.9329999999999999,370,7230,7231],[762,0.983,370,7230,7231],[2958,0.8170000000000001,370,7230,7231]]},{"id":"worldprobe-memory-2026-paper","name":"WorldProbe (memory overall, 2026 paper)","group":"视觉与视频","metric":"Memory overall","unit":"score","score_max":10,"score_decimals":2,"public_history":true,"protocol":{"en":"Memory Overall from Table 5, on the paper’s 0–10 scale. Scene, layout, entity and texture are the parts of that overall and are not entered separately. Evolution is a separate cohort.","zh":"表 5 的记忆总分，原文量程 0 到 10。场景、布局、个体和纹理是这个总分的分项，不另录。演化是另一组成绩。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.39721v1","status":"paper_baselines","published_at":"2026-10-29"},"results":[[7233,4.01,370,7234,7235],[7236,5.37,370,7234,7235],[7237,4.16,370,7234,7235],[7238,3.78,370,7234,7235],[7239,6.31,370,7234,7235],[7240,5.62,370,7234,7235],[7241,5.87,370,7234,7235],[7242,6.53,370,7234,7235],[7243,5.92,370,7234,7235],[7244,5.2,370,7234,7235]]},{"id":"worldprobe-evolution-2026-paper","name":"WorldProbe (evolution overall, 2026 paper)","group":"视觉与视频","metric":"Evolution overall","unit":"score","score_max":10,"score_decimals":2,"public_history":true,"protocol":{"en":"Evolution Overall from Table 5, on the paper’s 0–10 scale. Entity, scene, physics and causality are the parts of that overall and are not entered separately.","zh":"表 5 的演化总分，原文量程 0 到 10。个体、场景、物理和因果是这个总分的分项，不另录。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.39721v1","status":"paper_baselines","published_at":"2026-10-29"},"results":[[7233,3.56,370,7234,7235],[7236,3.57,370,7234,7235],[7237,2.51,370,7234,7235],[7238,3.77,370,7234,7235],[7239,3.29,370,7234,7235],[7240,4.06,370,7234,7235],[7241,5.17,370,7234,7235],[7242,4.41,370,7234,7235],[7243,4.24,370,7234,7235],[7244,3.9,370,7234,7235]]},{"id":"mimo-ab-iacc-real-2026-paper","name":"MIMO-Bench (attribute binding, real, 2026 paper)","group":"视觉与视频","metric":"IACC","unit":"%","public_history":true,"protocol":{"en":"Attribute-binding instance accuracy on the real split from Table 2. Synthetic images and numerical counting are separate and are not included. Reasoning effort is a configuration. Chance and human rows are not models. Cells marked with a dash are omitted.","zh":"表 2 真实图像上的属性绑定逐例准确率。合成图和计数是另外的指标，不并入。推理档位是配置。随机和人类行不是模型。标成横线的格子不补。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.38957v1","status":"paper_baselines","published_at":"2026-10-28"},"results":[[3415,0.455,370,7245,7246],[3482,0.465,370,7245,7246],[2055,0.513,370,5022,7246],[2602,0.491,370,5022,7246],[2058,0.32,370,7247,7246],[2058,0.528,370,5017,7246],[6901,0.503,370,5022,7246],[7153,0.47600000000000003,370,5022,7246],[2675,0.402,370,5022,7246],[6625,0.47700000000000004,370,5022,7246],[6150,0.424,370,5022,7246],[5485,0.366,370,5022,7246],[2444,0.386,370,5022,7246],[2453,0.384,370,5022,7246],[2471,0.355,370,5022,7246],[2642,0.409,370,5022,7246],[6345,0.321,370,5022,7246],[2654,0.386,370,5022,7246],[2706,0.423,370,5022,7246],[2697,0.359,370,5022,7246],[2728,0.344,370,5022,7246],[5536,0.34600000000000003,370,5022,7246]]},{"id":"physinsert-diou-text-2026-paper","name":"PhysInsert-Bench (text-guided DIoU, 2026 paper)","group":"视觉与视频","metric":"DIoU","unit":"score","public_history":true,"protocol":{"en":"Detection IoU from Table 2, text-guided, one reference image. Box-guided Table 3 is a different setting and is not included. PhysW is a fine-tune configuration of the named editor. Other columns are not included.","zh":"表 2 文本引导、一张参考图的检测 IoU。表 3 的框引导是另一种设置，不并入。PhysW 是具名编辑模型的微调配置。其他列不并入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.40453v1","status":"paper_baselines","published_at":"2026-10-29"},"score_max":100,"saturation_excluded":true,"score_decimals":2,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"results":[[6366,54.58,370,7248,7249],[6368,38.02,370,7248,7249],[7250,37.69,370,7251,7249],[7252,22.66,370,7251,7249],[7253,46.37,370,7248,7249],[6361,48.32,370,7248,7249],[7254,62.99,370,7248,7249]]},{"id":"culturalbind-recognition-2026-paper","name":"CulturalBind (recognition, 2026 paper)","group":"视觉与视频","metric":"Recognition","unit":"%","public_history":true,"protocol":{"en":"Recognition column of Table 3. Text description, encyclopedic QA and attribute QA are separate and are not included. Parameter size is part of the checkpoint name. Table 4 splits known and unknown concepts and is not included.","zh":"表 3 的识别列。文字描述、百科问答和属性问答不并入。参数规模属于检查点。表 4 的已知和未知拆分不并入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.39602v1","status":"paper_baselines","published_at":"2026-10-29"},"results":[[2687,0.18600000000000003,370,7255,7256],[2654,0.187,370,7255,7256],[6345,0.198,370,7255,7256],[2642,0.257,370,7255,7256],[6298,0.34600000000000003,370,7255,7256],[6342,0.349,370,7255,7256],[6343,0.363,370,7255,7256],[5815,0.319,370,7255,7256],[6868,0.382,370,7255,7256],[6869,0.431,370,7255,7256],[340,0.446,370,7255,7256]]},{"id":"indicipr-token-f1-2026-paper","name":"IndicIPR-QA (token F1, 2026 paper)","group":"专业工作","metric":"Token F1","unit":"score","score_max":100,"score_decimals":2,"public_history":true,"protocol":{"en":"Token-level F1 from Table VII, printed on a 0–100 scale. Exact match and character F1 are not included. Zero-shot and 3-shot are configurations. BM25 retrieval is a lexical method and is not included.","zh":"表 VII 的 token F1，原文按 0 到 100 印出。精确匹配和字符 F1 不并入。零样本和三样本是配置。BM25 是词法检索，不并入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.40518v1","status":"paper_baselines","published_at":"2026-10-29"},"results":[[7257,83.33,370,7258,7259],[7260,84.66,370,7258,7259],[7261,84.37,370,7258,7259],[7262,85.02,370,7258,7259],[7263,67.11,370,7264,7259],[7263,68.83,370,6397,7259],[806,74.78,370,7264,7259],[806,77.02,370,6397,7259],[7265,77.59,370,7264,7259],[7265,76.39,370,6397,7259]]},{"id":"infradev-resolved-2026-paper","name":"InfraDev (resolved, 2026 paper)","group":"编程","metric":"Resolved","unit":"%","public_history":true,"protocol":{"en":"Percent resolved from Table 2: every test passes. Percent of tests passed, time, cost and tokens are not included. Category rates in Table 3 are not included.","zh":"表 2 的解决率：该任务的测试全部通过。测试通过比例、时间、费用和 token 不并入。表 3 的分类通过率不并入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.36589v1","status":"paper_baselines","published_at":"2026-10-27"},"results":[[3394,0.389,370,7266,7267],[762,0.306,370,7266,7267],[3515,0.25,370,7266,7267],[7268,0.361,370,7266,7267],[3492,0.306,370,7266,7267],[2977,0.306,370,7266,7267]]},{"id":"openfrontiers-hq50-native-2026-paper","name":"OpenFrontiers HQ-50 (native closed-book, 2026 paper)","group":"数学与科学","metric":"Solved","unit":"%","public_history":true,"protocol":{"en":"Native protocol, closed-book column “all” from Table 3: items solved out of 40. A printed 0 is a measured miss. Continuation, frozen retrieval and live web are other protocols and are not included. The clean column uses a different denominator for each model and is not included.","zh":"表 3 原生协议、闭卷的 all 列：40 题里解出的题数。印出的 0 是没解出。续跑、冻结检索和实时网页是别的协议，不并入。clean 列每个模型的分母不同，不并入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.37739v1","status":"paper_baselines","published_at":"2026-10-27"},"results":[[2961,0.0,370,7269,7270],[5876,0.05,370,7269,7270],[5874,0.0,370,7269,7270],[3155,0.05,370,7269,7270],[3154,0.05,370,7269,7270],[3153,0.075,370,7269,7270],[5300,0.0,370,7269,7270]]},{"id":"geo-citation-rho-2026-paper","name":"GEO Citation Benchmark (Spearman rho, 2026 paper)","group":"其他","metric":"Spearman rho","unit":"score","score_max":1,"score_decimals":3,"public_history":true,"protocol":{"en":"Mean within-group Spearman rho from Table 1. Top-1 accuracy is not included. Other engines, source position and BM25 are references, not models. Seeds of one student are configurations. The three-seed ensemble is not included.","zh":"表 1 的组内平均 Spearman rho。Top-1 不并入。其他引擎、来源位置和 BM25 是参照，不是模型。同一学生的种子是配置。三种子集成不并入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.38090v1","status":"paper_baselines","published_at":"2026-10-28"},"results":[[7271,0.386,370,7264,7272],[7273,0.412,370,7264,7272],[7274,0.341,370,7264,7272],[7275,0.452,370,7264,7272],[7276,0.449,370,7264,7272],[5491,0.575,370,7277,7272],[6486,0.578,370,7278,7272],[7279,0.575,370,7278,7272],[6498,0.516,370,7278,7272],[7280,0.501,370,7278,7272],[7281,0.457,370,7282,7272],[7283,0.542,370,7284,7272],[7283,0.546,370,7285,7272],[7283,0.549,370,7286,7272],[7287,0.533,370,7282,7272],[7287,0.559,370,7285,7272]]},{"id":"moi-gpt4o-quality-2026-paper","name":"MoI-Bench (GPT-4o full-task overall, 2026 paper)","group":"生成与创作","metric":"Overall quality","unit":"score","score_max":1,"score_decimals":2,"public_history":true,"protocol":{"en":"GPT-4o column Q from Table 4. This is the full-task overall over generation and editing, not a generation-only quality score. Section 5.2 states that generation-only models stay low because unsupported editing tasks contribute zero under that definition. The Gemini 3.1 Pro column is a second judge and is not included. Criterion triples are not averaged again.","zh":"表 4 的 GPT-4o 列 Q。这是生成和编辑都计入的全任务总分，不是纯生成质量。5.2 节写明只生成的模型偏低，是因为按这个定义，不支持的编辑任务计 0。Gemini 3.1 Pro 是另一位评判，不并入。准则三元组不再另平均。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.37599v1","status":"paper_baselines","published_at":"2026-10-27"},"results":[[6839,0.95,370,7288,7289],[6381,0.95,370,7288,7289],[7290,0.97,370,7288,7289],[7291,0.12,370,7288,7289],[7292,0.43,370,7288,7289],[7293,0.51,370,7288,7289],[7294,0.59,370,7288,7289],[7295,0.62,370,7288,7289],[7296,0.64,370,7288,7289],[7297,0.67,370,7288,7289],[7298,0.67,370,7288,7289],[7299,0.69,370,7288,7289],[7300,0.83,370,7288,7289],[7301,0.08,370,7288,7289],[7302,0.1,370,7288,7289],[7303,0.1,370,7288,7289],[7304,0.1,370,7288,7289],[7305,0.62,370,7288,7289],[7306,0.66,370,7288,7289]]},{"id":"unitybench-mini3d-overall-2026-paper","name":"UnityBench-Mini3D (overall pass, 2026 paper)","group":"编程","metric":"Overall pass","unit":"%","public_history":true,"protocol":{"en":"Overall pass rate, column O.a., from Table 3. Category rates, difficulty splits, cost and time are not included. The harness is a configuration. Part (b) CUA rows use the same overall column. A printed zero in a category is not copied; the overall is the published rate.","zh":"表 3 的总通过率，列 O.a.。分类、难度、费用和时间不并入。运行框架是配置。(b) 的计算机使用行用同一总通过率。分类格里印出的 0 不另抄，这里用公布的总通过率。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.35980v1","status":"paper_baselines","published_at":"2026-10-26"},"results":[[755,0.6876000000000001,370,7307,7308],[5004,0.6233,370,7307,7308],[2977,0.662,370,7307,7308],[3492,0.6309,370,7307,7308],[3261,0.6172,370,7307,7308],[3491,0.5228,370,7307,7308],[3654,0.6245,370,7307,7308],[3560,0.6668000000000001,370,7307,7308],[2974,0.639,370,7307,7308],[5290,0.5104,370,7307,7308],[4881,0.2441,370,7307,7308],[2977,0.6196,370,7309,7308],[5004,0.6129,370,7309,7308],[3654,0.5195000000000001,370,7309,7308],[3261,0.3871,370,7309,7308],[5004,0.7249,370,7310,7308],[6591,0.6801999999999999,370,7310,7308],[7311,0.6638,370,7310,7308],[3581,0.6617000000000001,370,7310,7308],[755,0.7175,370,7312,7308],[3568,0.54,370,7312,7308],[2974,0.7062,370,7313,7308],[3391,0.7119,370,7313,7308],[3412,0.25239999999999996,370,7314,7308],[3386,0.3119,370,7314,7308],[3560,0.6231,370,7315,7308],[3587,0.478,370,7315,7308],[3595,0.3886,370,7315,7308],[2977,0.6635,370,7316,7308],[3411,0.415,370,7316,7308],[3261,0.6498,370,5380,7308],[3492,0.705,370,5380,7308],[2977,0.0504,370,7317,7308],[5004,0.2216,370,7318,7308],[2977,0.0877,370,7319,7308],[5004,0.2311,370,7320,7308]]},{"id":"skilllife-runtime-joint-2026-paper","name":"SkillLifeBench (runtime joint success, 2026 paper)","group":"Agent 与工具","metric":"Joint success","unit":"%","public_history":true,"protocol":{"en":"Runtime-track joint success from Table 2, 89 tasks. Lifecycle joint success is a separate cohort. Task-success columns are not included. Open or closed weights are recorded with the checkpoint.","zh":"表 2 运行轨道的联合成功率，89 题。生命周期联合成功率是另一组。任务成功率不并入。开放或封闭权重随检查点记下。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.35215v1","status":"paper_baselines","published_at":"2026-10-26"},"results":[[4979,0.82,370,7321,7322],[5727,0.809,370,7321,7322],[3393,0.764,370,7321,7322],[227,0.742,370,7323,7322],[3154,0.764,370,7323,7322],[3185,0.775,370,7323,7322],[3187,0.775,370,7323,7322],[3009,0.281,370,7323,7322],[3409,0.82,370,7321,7322],[4969,0.618,370,7323,7322],[4070,0.73,370,7323,7322],[3201,0.775,370,7323,7322],[5617,0.73,370,7321,7322],[3415,0.674,370,7321,7322]]},{"id":"skilllife-lifecycle-joint-2026-paper","name":"SkillLifeBench (lifecycle joint success, 2026 paper)","group":"Agent 与工具","metric":"Joint success","unit":"%","public_history":true,"protocol":{"en":"Lifecycle-track joint success from Table 2, 60 tasks. The paper ranks models by this column. Runtime joint success is separate and is not averaged with it.","zh":"表 2 生命周期轨道的联合成功率，60 题。原文按这一列排名。运行轨道不并入，也不与它平均。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.35215v1","status":"paper_baselines","published_at":"2026-10-26"},"results":[[4979,0.9670000000000001,370,7321,7322],[5727,0.9670000000000001,370,7321,7322],[3393,0.883,370,7321,7322],[227,0.883,370,7323,7322],[3154,0.8,370,7323,7322],[3185,0.767,370,7323,7322],[3187,0.767,370,7323,7322],[3009,0.6,370,7323,7322],[3409,0.5670000000000001,370,7321,7322],[4969,0.55,370,7323,7322],[4070,0.517,370,7323,7322],[3201,0.45,370,7323,7322],[5617,0.317,370,7321,7322],[3415,0.1,370,7321,7322]]},{"id":"prism-cri-2026-paper","name":"PRISM (calibration-aware CRI, 2026 paper)","group":"视觉与视频","metric":"CRI","unit":"score","score_max":1,"score_decimals":3,"public_history":true,"protocol":{"en":"The cri column of the authors’ reference CSV. It is their published composite, not a new average of the component columns. Eight pathology foundation models. Reported to three decimals.","zh":"作者参考 CSV 的 cri 列。这是他们公布的综合分，不是把其他列再平均一次。八个病理基础模型。保留三位小数。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.36517","status":"paper_baselines","published_at":"2026-10-27"},"results":[[7324,0.541,370,7325,7326],[7327,0.52,370,7325,7326],[7328,0.489,370,7325,7326],[7329,0.642,370,7325,7326],[7330,0.469,370,7325,7326],[7331,0.552,370,7325,7326],[7332,0.553,370,7325,7326],[7333,0.475,370,7325,7326]]},{"id":"probe-reference-f1-2026-paper","name":"PROBE (reference-set F1, 2026 paper)","group":"编程","metric":"F1","unit":"score","public_history":true,"protocol":{"en":"Table 3 F1 against the maintainer-fix reference set. Reasoning effort is the configuration. Muse Spark 1.3 and GPT-5.6 Luna at medium effort are the means the caption states, over 5 and 10 runs. Claude Opus 5 is named in the setup and has no row in this table, so it is omitted.","zh":"表 3 相对维护者修复参考集的 F1。推理力度是配置。Muse Spark 1.3 和中等力度的 GPT-5.6 Luna 用题注写明的五次和十次均值。设置里提到 Claude Opus 5，但这张表没有它的行，所以不录入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.35987","status":"paper_baselines","published_at":"2026-10-26"},"score_max":100,"saturation_excluded":true,"score_decimals":2,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"results":[[3512,10.5,370,7334,7335],[3529,9.7,370,7334,7335],[3184,8.5,370,7336,7335],[2973,8.3,370,7336,7335],[3415,7.8,370,7336,7335],[2979,7.4,370,7334,7335],[3568,6.7,370,7336,7335],[762,10.2,370,7334,7335],[762,9.6,370,7336,7335],[762,9.0,370,7337,7335],[3408,10.1,370,7338,7335],[3408,9.0,370,7334,7335],[3408,7.3,370,7336,7335],[3408,5.8,370,7337,7335]]},{"id":"medconst-medethicsqa-overall-2026-paper","name":"MedConstBench (MedEthicsQA overall, 2026 paper)","group":"医疗","metric":"Overall","unit":"score","score_max":2,"score_decimals":2,"public_history":true,"protocol":{"en":"Table III Overall on MedEthicsQA. Overall is the authors’ mean of the four principle columns. Those columns are not entered again. The four constitutions are configurations. MedDilemma is a separate cohort. The scale is 0 to 2.","zh":"表 III 在 MedEthicsQA 上的 Overall。Overall 是作者对四个原则列的平均，原则列不再另录。四种宪法是配置。MedDilemma 是另一组。刻度是 0 到 2。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.36513","status":"paper_baselines","published_at":"2026-10-27"},"results":[[6689,1.71,370,7339,7340],[6689,1.74,370,7341,7340],[6689,1.73,370,7342,7340],[6689,1.71,370,7343,7340],[7344,1.75,370,7339,7340],[7344,1.75,370,7341,7340],[7344,1.77,370,7342,7340],[7344,1.69,370,7343,7340],[4469,1.7,370,7339,7340],[4469,1.68,370,7341,7340],[4469,1.84,370,7342,7340],[4469,1.72,370,7343,7340],[7345,1.75,370,7339,7340],[7345,1.79,370,7341,7340],[7345,1.81,370,7342,7340],[7345,1.77,370,7343,7340],[1621,1.72,370,7339,7340],[1621,1.79,370,7341,7340],[1621,1.78,370,7342,7340],[1621,1.75,370,7343,7340],[588,1.7,370,7339,7340],[588,1.73,370,7341,7340],[588,1.72,370,7342,7340],[588,1.7,370,7343,7340],[4463,1.76,370,7339,7340],[4463,1.75,370,7341,7340],[4463,1.79,370,7342,7340],[4463,1.76,370,7343,7340],[7346,1.6,370,7339,7340],[7346,1.63,370,7341,7340],[7346,1.69,370,7342,7340],[7346,1.62,370,7343,7340],[2061,1.73,370,7339,7340],[2061,1.79,370,7341,7340],[2061,1.79,370,7342,7340],[2061,1.75,370,7343,7340],[7347,1.74,370,7339,7340],[7347,1.71,370,7341,7340],[7347,1.84,370,7342,7340],[7347,1.64,370,7343,7340]]},{"id":"medconst-meddilemma-overall-2026-paper","name":"MedConstBench (MedDilemma overall, 2026 paper)","group":"医疗","metric":"Overall","unit":"score","score_max":2,"score_decimals":2,"public_history":true,"protocol":{"en":"Table III Overall on MedDilemma, the authors’ mean of the four principle columns. Constitutions are configurations. MedEthicsQA is not averaged with this cohort.","zh":"表 III 在 MedDilemma 上的 Overall，是作者对四个原则列的平均。宪法是配置。不与 MedEthicsQA 平均。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.36513","status":"paper_baselines","published_at":"2026-10-27"},"results":[[6689,1.72,370,7339,7340],[6689,1.75,370,7341,7340],[6689,1.76,370,7342,7340],[6689,1.74,370,7343,7340],[7344,1.66,370,7339,7340],[7344,1.71,370,7341,7340],[7344,1.8,370,7342,7340],[7344,1.63,370,7343,7340],[4469,1.69,370,7339,7340],[4469,1.69,370,7341,7340],[4469,1.82,370,7342,7340],[4469,1.62,370,7343,7340],[7345,1.74,370,7339,7340],[7345,1.79,370,7341,7340],[7345,1.78,370,7342,7340],[7345,1.76,370,7343,7340],[1621,1.58,370,7339,7340],[1621,1.76,370,7341,7340],[1621,1.77,370,7342,7340],[1621,1.67,370,7343,7340],[588,1.57,370,7339,7340],[588,1.67,370,7341,7340],[588,1.25,370,7342,7340],[588,1.45,370,7343,7340],[4463,1.72,370,7339,7340],[4463,1.75,370,7341,7340],[4463,1.8,370,7342,7340],[4463,1.7,370,7343,7340],[7346,1.52,370,7339,7340],[7346,1.54,370,7341,7340],[7346,1.71,370,7342,7340],[7346,1.56,370,7343,7340],[2061,1.72,370,7339,7340],[2061,1.76,370,7341,7340],[2061,1.78,370,7342,7340],[2061,1.73,370,7343,7340],[7347,1.73,370,7339,7340],[7347,1.74,370,7341,7340],[7347,1.79,370,7342,7340],[7347,1.62,370,7343,7340]]},{"id":"worldfail-robotwin-tfe-2026-paper","name":"WorldFailBench (RoboTwin type-frequency error, 2026 paper)","group":"预测与决策","metric":"Type frequency error","unit":"score","score_max":1,"score_decimals":3,"higher_is_better":false,"public_history":true,"protocol":{"en":"Table 3 Mean on RoboTwin. The paper defines that mean as an equal weight of the five policy columns. Ref–Ref is a reference row and is omitted. Lower is better. Stage-frequency error is a different table and is not included.","zh":"表 3 在 RoboTwin 上的 Mean。原文把这一列定义为五个策略列的等权平均。Ref–Ref 是参照行，不录入。越低越好。阶段频率误差是另一张表，不并入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.33565","status":"paper_baselines","published_at":"2026-10-24"},"results":[[7348,0.211,370,7349,7350],[7351,0.236,370,7352,7350],[7353,0.335,370,7354,7350],[7355,0.171,370,7356,7350],[7357,0.159,370,7358,7350],[7359,0.224,370,7360,7350],[7361,0.184,370,7362,7350]]},{"id":"onecodereview-boolean-2026-paper","name":"OneCodeReviewBench (Boolean balanced accuracy, 2026 paper)","group":"编程","metric":"Balanced accuracy","unit":"%","public_history":true,"protocol":{"en":"Table IV Boolean balanced accuracy. Choice accuracy and score MAE are separate cohorts. Simple baselines are a control row and are omitted. Jev, Laya and Von are omitted: Table III does not isolate their hosted or SDK backends. The seven retained rows name gpt-6-luna, microsoft-decision-1, Qwen3.5-4B and the four Kev checkpoints.","zh":"表 IV 的布尔平衡准确率。选择题准确率和分数平均绝对误差是另外两组。Simple baselines 是对照行，不录入。 Jev、Laya 和 Von 不录入：表 III 没有把它们的托管或 SDK 后端拆开。保留的七行是 gpt-6-luna、microsoft-decision-1、Qwen3.5-4B 和四个 Kev 检查点。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.33389","status":"paper_baselines","published_at":"2026-10-24"},"results":[[7363,0.813,370,5150,7364],[7365,0.6409999999999999,370,7366,7364],[7367,0.526,370,7368,7364],[7369,0.502,370,7370,7364],[7371,0.546,370,7370,7364],[7372,0.565,370,7370,7364],[7373,0.797,370,7374,7364]]},{"id":"onecodereview-choice-2026-paper","name":"OneCodeReviewBench (choice accuracy, 2026 paper)","group":"编程","metric":"Choice accuracy","unit":"%","public_history":true,"protocol":{"en":"Table IV Choice accuracy, exact category match. It is not averaged with Boolean balanced accuracy. Jev, Laya and Von are omitted: Table III does not isolate their hosted or SDK backends. The seven retained rows name gpt-6-luna, microsoft-decision-1, Qwen3.5-4B and the four Kev checkpoints.","zh":"表 IV 的选择题准确率，类别完全匹配。不与布尔平衡准确率平均。 Jev、Laya 和 Von 不录入：表 III 没有把它们的托管或 SDK 后端拆开。保留的七行是 gpt-6-luna、microsoft-decision-1、Qwen3.5-4B 和四个 Kev 检查点。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.33389","status":"paper_baselines","published_at":"2026-10-24"},"results":[[7363,0.682,370,5150,7364],[7365,0.634,370,7366,7364],[7367,0.429,370,7368,7364],[7369,0.32899999999999996,370,7370,7364],[7371,0.479,370,7370,7364],[7372,0.5589999999999999,370,7370,7364],[7373,0.768,370,7374,7364]]},{"id":"onecodereview-score-mae-2026-paper","name":"OneCodeReviewBench (score MAE, 2026 paper)","group":"编程","metric":"Score MAE","unit":"score","score_max":3,"score_decimals":3,"higher_is_better":false,"public_history":true,"protocol":{"en":"Table IV score mean absolute error on the 0–3 rubric. Lower is better. Boolean and choice accuracy are separate. Jev, Laya and Von are omitted: Table III does not isolate their hosted or SDK backends. The seven retained rows name gpt-6-luna, microsoft-decision-1, Qwen3.5-4B and the four Kev checkpoints.","zh":"表 IV 在 0 到 3 量规上的分数平均绝对误差。越低越好。布尔和选择题准确率不并入。 Jev、Laya 和 Von 不录入：表 III 没有把它们的托管或 SDK 后端拆开。保留的七行是 gpt-6-luna、microsoft-decision-1、Qwen3.5-4B 和四个 Kev 检查点。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.33389","status":"paper_baselines","published_at":"2026-10-24"},"results":[[7363,0.516,370,5150,7364],[7365,0.43,370,7366,7364],[7367,0.727,370,7368,7364],[7369,0.899,370,7370,7364],[7371,0.723,370,7370,7364],[7372,0.571,370,7370,7364],[7373,0.367,370,7374,7364]]},{"id":"rvg-overall-miou-2026-paper","name":"RVG-Bench (overall mIoU, 2026 paper)","group":"视觉与视频","metric":"mIoU","unit":"score","public_history":true,"protocol":{"en":"Table 2 Overall mIoU. The caption says this column is sample-count-weighted. Subtype columns and IoU@0.3 are not entered. The human audit row is not a model. Visual or audio-visual input is the configuration.","zh":"表 2 的 Overall mIoU。题注写明这一列按样本数加权。子类型列和 IoU@0.3 不录入。人类审计行不是模型。视觉或音视频输入记为配置。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.32268","status":"paper_baselines","published_at":"2026-10-23"},"score_max":100,"saturation_excluded":true,"score_decimals":2,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"results":[[6725,49.6,370,7375,7376],[7377,47.0,370,7375,7376],[6130,44.4,370,7375,7376],[7378,41.9,370,7379,7376],[3196,38.5,370,7379,7376],[5899,38.2,370,7379,7376],[7380,23.4,370,7375,7376],[7381,10.7,370,7375,7376],[7382,5.1,370,7375,7376],[7383,5.1,370,7375,7376],[7384,4.5,370,7375,7376],[7385,2.2,370,7375,7376],[7386,30.1,370,7379,7376],[6342,26.3,370,7379,7376],[7387,21.8,370,7379,7376],[7388,19.7,370,7379,7376],[7389,14.1,370,7379,7376],[2754,13.4,370,7379,7376],[7390,11.9,370,7379,7376],[7391,9.4,370,7379,7376],[7392,6.4,370,7379,7376],[6254,1.7,370,7379,7376]]},{"id":"robotagent-standalone-average-2026-paper","name":"RobotAgentBench (standalone average, 2026 paper)","group":"Agent 与工具","metric":"Average","unit":"%","public_history":true,"protocol":{"en":"Table 1 Average for the eleven standalone policies. The caption defines Average as the mean of the five task-level scores. Open-ended is separate. Rank medals are not scores. Coding-agent rows are harnesses and are not included.","zh":"表 1 里十一个独立策略的 Average。题注把 Average 定义为五个任务分的平均。开放场景是另一组。名次圆点不是分数。编程智能体行是框架，不并入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.34649","status":"paper_baselines","published_at":"2026-10-26"},"saturation_excluded":true,"scale_label":{"en":"Task score (%)","zh":"任务分（%）"},"results":[[5820,0.11900000000000001,370,7393,7394],[7395,0.124,370,7393,7394],[7396,0.18,370,7393,7394],[7397,0.032,370,7393,7394],[7398,0.051,370,7393,7394],[7399,0.075,370,7393,7394],[7400,0.11,370,7393,7394],[7401,0.059000000000000004,370,7393,7394],[7402,0.043,370,7393,7394],[7403,0.086,370,7393,7394],[7404,0.039,370,7393,7394]]},{"id":"robotagent-standalone-open-2026-paper","name":"RobotAgentBench (standalone open-ended, 2026 paper)","group":"Agent 与工具","metric":"Open-ended","unit":"%","public_history":true,"protocol":{"en":"Table 1 Open-Ended column for the same eleven standalone policies. It is not averaged with the five-task Average.","zh":"表 1 同一批十一个独立策略的 Open-Ended 列。不与五任务 Average 平均。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.34649","status":"paper_baselines","published_at":"2026-10-26"},"saturation_excluded":true,"scale_label":{"en":"Task score (%)","zh":"任务分（%）"},"results":[[5820,0.011000000000000001,370,7393,7394],[7395,0.004,370,7393,7394],[7396,0.01,370,7393,7394],[7397,0.019,370,7393,7394],[7398,0.005,370,7393,7394],[7399,0.008,370,7393,7394],[7400,0.013000000000000001,370,7393,7394],[7401,0.003,370,7393,7394],[7402,0.005,370,7393,7394],[7403,0.02,370,7393,7394],[7404,0.004,370,7393,7394]]},{"id":"vast-asr-2026-paper","name":"VAST-Bench (attack success, 2026 paper)","group":"安全与可靠性","metric":"Attack success","unit":"%","higher_is_better":false,"public_history":true,"protocol":{"en":"Table 3 task-level attack success. Lower is better. Harnesses are configurations. A dash means that combination was not evaluated and is omitted, not scored as zero. Capability pass rate is a separate cohort.","zh":"表 3 的任务级攻击成功率。越低越好。框架是配置。破折号表示该组合没有评估，不记成 0。能力通过率是另一组。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.35188","status":"paper_baselines","published_at":"2026-10-26"},"results":[[4969,0.977,370,7405,7406],[4969,0.948,370,2300,7406],[4969,0.955,370,3759,7406],[4969,0.894,370,5393,7406],[4969,0.9440000000000001,370,5948,7406],[2584,0.924,370,7405,7406],[2584,0.898,370,2300,7406],[2584,0.882,370,3759,7406],[2584,0.81,370,5393,7406],[2584,0.851,370,5948,7406],[3202,0.784,370,7405,7406],[3202,0.732,370,2300,7406],[3202,0.7090000000000001,370,3759,7406],[3202,0.6970000000000001,370,5393,7406],[3202,0.758,370,5948,7406],[5300,0.478,370,7405,7406],[5300,0.519,370,2300,7406],[5300,0.428,370,3759,7406],[5300,0.469,370,5393,7406],[5300,0.447,370,5948,7406],[3157,0.198,370,7405,7406],[3157,0.243,370,2300,7406],[3157,0.33299999999999996,370,3759,7406],[3157,0.28800000000000003,370,5393,7406],[3157,0.187,370,5948,7406],[3196,0.684,370,7405,7406],[3196,0.7040000000000001,370,2300,7406],[3196,0.665,370,5393,7406],[3196,0.6409999999999999,370,5948,7406],[4979,0.608,370,7405,7406],[4979,0.636,370,2300,7406],[4979,0.565,370,5393,7406],[4979,0.574,370,5948,7406],[3183,0.652,370,7405,7406],[3183,0.528,370,3759,7406],[3183,0.508,370,5393,7406],[3183,0.5710000000000001,370,5948,7406],[3194,0.287,370,7405,7406],[3194,0.209,370,3759,7406],[3194,0.237,370,5393,7406],[3194,0.27699999999999997,370,5948,7406],[3154,0.284,370,7405,7406],[3154,0.77,370,2300,7406],[3154,0.42700000000000005,370,3759,7406],[3154,0.442,370,5393,7406],[3154,0.455,370,5948,7406],[3155,0.396,370,7405,7406],[3155,0.436,370,2300,7406],[3155,0.38,370,3759,7406],[3155,0.365,370,5393,7406],[3155,0.37,370,5948,7406]]},{"id":"vast-cap-2026-paper","name":"VAST-Bench (capability pass, 2026 paper)","group":"安全与可靠性","metric":"Capability pass","unit":"%","public_history":true,"protocol":{"en":"Table 3 capability-twin pass rate. Higher is better. The same unevaluated dashes are omitted. Attack success is not averaged with this rate.","zh":"表 3 的能力对照通过率。越高越好。同样的未评估破折号不录入。不与攻击成功率平均。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.35188","status":"paper_baselines","published_at":"2026-10-26"},"results":[[4969,0.623,370,7405,7406],[4969,0.552,370,2300,7406],[4969,0.519,370,3759,7406],[4969,0.58,370,5393,7406],[4969,0.675,370,5948,7406],[2584,0.887,370,7405,7406],[2584,0.882,370,2300,7406],[2584,0.882,370,3759,7406],[2584,0.868,370,5393,7406],[2584,0.8540000000000001,370,5948,7406],[3202,0.9620000000000001,370,7405,7406],[3202,0.8959999999999999,370,2300,7406],[3202,0.925,370,3759,7406],[3202,0.873,370,5393,7406],[3202,0.8959999999999999,370,5948,7406],[5300,0.925,370,7405,7406],[5300,0.929,370,2300,7406],[5300,0.92,370,3759,7406],[5300,0.92,370,5393,7406],[5300,0.915,370,5948,7406],[3157,0.958,370,7405,7406],[3157,0.972,370,2300,7406],[3157,0.948,370,3759,7406],[3157,0.934,370,5393,7406],[3157,0.958,370,5948,7406],[3196,0.986,370,7405,7406],[3196,0.972,370,2300,7406],[3196,0.958,370,5393,7406],[3196,0.986,370,5948,7406],[4979,0.986,370,7405,7406],[4979,0.972,370,2300,7406],[4979,0.948,370,5393,7406],[4979,0.991,370,5948,7406],[3183,0.934,370,7405,7406],[3183,0.92,370,3759,7406],[3183,0.8440000000000001,370,5393,7406],[3183,0.925,370,5948,7406],[3194,0.953,370,7405,7406],[3194,0.92,370,3759,7406],[3194,0.835,370,5393,7406],[3194,0.92,370,5948,7406],[3154,0.9620000000000001,370,7405,7406],[3154,0.9620000000000001,370,2300,7406],[3154,0.972,370,3759,7406],[3154,0.9390000000000001,370,5393,7406],[3154,0.943,370,5948,7406],[3155,0.976,370,7405,7406],[3155,0.953,370,2300,7406],[3155,0.9059999999999999,370,3759,7406],[3155,0.943,370,5393,7406],[3155,0.943,370,5948,7406]]},{"id":"local-agent-bench-score-2026","name":"Local Agent Bench (score v3, 2026)","group":"Agent 与工具","metric":"Score","unit":"%","public_history":true,"protocol":{"en":"The published score field of scoring version 3. Each dated run is a configuration and runs are not averaged. Runs with no score are omitted, not filled with zero. Component columns and tokens per second are not entered. The public repository was created on 2026-10-23. Scoring v3, dated 2026-10-23, is the first public score of those runs; the changelog says the runs were not repeated when the formula changed. The 2026-10-28 hardware series did not change this score.","zh":"评分版本 3 公布的 score 字段。每次带日期的运行是配置，不把多次运行平均。没有分数的运行不录入，不补 0。分项和每秒 token 不录入。 公开仓库创建于 2026-10-23。评分版本 3 也是这一天，是这些运行的首次公开分数；变更说明写明改公式时没有重跑。2026-10-28 的硬件系列没有改这个分数。"},"source":{"name":"Paper authors","url":"https://localagentbench.com","status":"paper_baselines","published_at":"2026-10-23"},"results":[[7407,0.266,370,7408,7409],[7410,0.292,370,7411,7409],[7410,0.276,370,7412,7409],[7413,0.9834,370,7408,7409],[7414,0.9972,370,7415,7409],[7416,0.9468000000000001,370,7417,7409],[7416,0.9407,370,7418,7409],[7419,0.9344,370,7420,7409],[7421,0.45280000000000004,370,7411,7409],[7421,0.4181,370,7417,7409],[7421,0.4292,370,7422,7409],[7423,0.5323,370,7411,7409],[7423,0.49829999999999997,370,7417,7409],[7423,0.5105,370,7422,7409],[7424,0.49829999999999997,370,7425,7409],[7424,0.5036999999999999,370,7417,7409],[7424,0.4941,370,7418,7409],[7426,0.602,370,7427,7409],[7426,0.5991,370,7428,7409],[7429,0.4239,370,7430,7409],[7431,0.28059999999999996,370,7432,7409],[7433,0.22,370,7411,7409],[7433,0.218,370,7422,7409],[7434,0.242,370,7408,7409],[7435,0.26,370,7425,7409],[7435,0.264,370,7417,7409],[7435,0.256,370,7418,7409],[7436,0.5925,370,7411,7409],[7436,0.6136,370,7408,7409],[7437,0.5856,370,7425,7409],[7437,0.5866,370,7417,7409],[7437,0.6229,370,7422,7409],[7438,0.8659,370,7415,7409],[7439,0.9165000000000001,370,7408,7409],[7439,0.8714,370,7412,7409],[7440,0.9495999999999999,370,7415,7409],[7441,0.9057999999999999,370,7408,7409],[7442,0.9523,370,7430,7409],[7443,0.9198999999999999,370,7430,7409],[7444,0.9062,370,7408,7409],[7445,0.9847,370,7432,7409],[7446,0.9285,370,7417,7409],[7446,0.9192,370,7422,7409],[7447,0.6061,370,7411,7409],[7447,0.6007,370,7417,7409],[7447,0.6267,370,7422,7409]]},{"id":"crowdcc-sha-mcmae-2026-paper","name":"CrowdCC (ShanghaiTech A corruption mcMAE, 2026 paper)","group":"视觉与视频","metric":"Corruption mcMAE","unit":"raw","score_decimals":1,"higher_is_better":false,"public_history":true,"protocol":{"en":"Table 4 per-dataset mcMAE from the corruption protocol introduced in arXiv:2610.36340v2 (28 Oct 2026). mcMAE is the authors’ published mean corruption error. FP32 and INT8 are configurations. The four image collections stay separate and are not averaged. Table 3 clean MAE is the older counting leaderboard and is not included. Lower is better. This is a count error, not a percentage.","zh":"表 4 的分数据集 mcMAE，来自 arXiv:2610.36340v2（2026-10-28）提出的损坏协议。mcMAE 是作者公布的平均损坏误差。FP32 和 INT8 是配置。四个图像集分开，不平均。表 3 的干净 MAE 是更早的计数榜，不录入。越低越好。这是计数误差，不是百分比。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.36340v2","status":"paper_baselines","published_at":"2026-10-26"},"results":[[7448,74.8,370,7449,7450],[7448,73.6,370,7451,7450],[7452,81.0,370,7449,7450],[7452,81.1,370,7451,7450],[7453,73.4,370,7449,7450],[7453,72.5,370,7451,7450],[7454,87.2,370,7449,7450],[7454,87.9,370,7451,7450],[7455,80.7,370,7449,7450],[7455,82.1,370,7451,7450],[7456,75.4,370,7449,7450],[7456,76.1,370,7451,7450],[7457,79.6,370,7449,7450],[7457,75.9,370,7451,7450],[7458,80.4,370,7449,7450],[7458,80.6,370,7451,7450],[7459,78.7,370,7449,7450],[7459,75.6,370,7451,7450],[7460,68.9,370,7449,7450],[7460,71.3,370,7451,7450],[7461,108.3,370,7449,7450],[7461,108.0,370,7451,7450],[7462,86.1,370,7449,7450],[7462,82.9,370,7451,7450],[7463,70.9,370,7449,7450],[7463,71.4,370,7451,7450],[7464,100.6,370,7449,7450],[7464,92.4,370,7451,7450],[7465,77.8,370,7449,7450],[7465,72.2,370,7451,7450],[7466,70.2,370,7449,7450],[7466,71.6,370,7451,7450],[7467,96.0,370,7449,7450],[7467,94.7,370,7451,7450],[7468,124.5,370,7449,7450],[7468,128.7,370,7451,7450],[7469,72.3,370,7449,7450],[7469,72.0,370,7451,7450],[7470,77.1,370,7449,7450],[7470,78.9,370,7451,7450],[7471,80.4,370,7449,7450],[7471,83.8,370,7451,7450],[7472,70.5,370,7449,7450],[7472,73.2,370,7451,7450],[7473,85.9,370,7449,7450],[7473,86.6,370,7451,7450],[7474,74.7,370,7449,7450],[7474,74.8,370,7451,7450],[7475,70.8,370,7449,7450],[7475,69.9,370,7451,7450],[7476,69.4,370,7449,7450],[7476,70.1,370,7451,7450]]},{"id":"crowdcc-shb-mcmae-2026-paper","name":"CrowdCC (ShanghaiTech B corruption mcMAE, 2026 paper)","group":"视觉与视频","metric":"Corruption mcMAE","unit":"raw","score_decimals":1,"higher_is_better":false,"public_history":true,"protocol":{"en":"Table 4 per-dataset mcMAE from the corruption protocol introduced in arXiv:2610.36340v2 (28 Oct 2026). mcMAE is the authors’ published mean corruption error. FP32 and INT8 are configurations. The four image collections stay separate and are not averaged. Table 3 clean MAE is the older counting leaderboard and is not included. Lower is better. This is a count error, not a percentage.","zh":"表 4 的分数据集 mcMAE，来自 arXiv:2610.36340v2（2026-10-28）提出的损坏协议。mcMAE 是作者公布的平均损坏误差。FP32 和 INT8 是配置。四个图像集分开，不平均。表 3 的干净 MAE 是更早的计数榜，不录入。越低越好。这是计数误差，不是百分比。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.36340v2","status":"paper_baselines","published_at":"2026-10-26"},"results":[[7448,17.4,370,7449,7450],[7448,16.8,370,7451,7450],[7452,16.3,370,7449,7450],[7452,15.9,370,7451,7450],[7453,14.2,370,7449,7450],[7453,14.6,370,7451,7450],[7454,26.5,370,7449,7450],[7454,26.1,370,7451,7450],[7455,18.7,370,7449,7450],[7455,18.4,370,7451,7450],[7456,18.5,370,7449,7450],[7456,18.8,370,7451,7450],[7457,15.0,370,7449,7450],[7457,15.5,370,7451,7450],[7458,18.8,370,7449,7450],[7458,17.2,370,7451,7450],[7459,14.4,370,7449,7450],[7459,15.8,370,7451,7450],[7460,14.0,370,7449,7450],[7460,13.4,370,7451,7450],[7461,26.7,370,7449,7450],[7461,25.9,370,7451,7450],[7462,17.6,370,7449,7450],[7462,19.1,370,7451,7450],[7463,12.1,370,7449,7450],[7463,11.9,370,7451,7450],[7464,19.7,370,7449,7450],[7464,23.4,370,7451,7450],[7465,12.3,370,7449,7450],[7465,14.2,370,7451,7450],[7466,12.9,370,7449,7450],[7466,13.8,370,7451,7450],[7467,33.3,370,7449,7450],[7467,29.0,370,7451,7450],[7468,41.9,370,7449,7450],[7468,52.4,370,7451,7450],[7469,13.8,370,7449,7450],[7469,13.7,370,7451,7450],[7470,15.3,370,7449,7450],[7470,15.3,370,7451,7450],[7471,15.0,370,7449,7450],[7471,14.9,370,7451,7450],[7472,12.5,370,7449,7450],[7472,13.7,370,7451,7450],[7473,14.5,370,7449,7450],[7473,14.9,370,7451,7450],[7474,17.4,370,7449,7450],[7474,17.9,370,7451,7450],[7475,17.9,370,7449,7450],[7475,18.4,370,7451,7450],[7476,17.2,370,7449,7450],[7476,17.9,370,7451,7450]]},{"id":"crowdcc-qnrf-mcmae-2026-paper","name":"CrowdCC (UCF-QNRF corruption mcMAE, 2026 paper)","group":"视觉与视频","metric":"Corruption mcMAE","unit":"raw","score_decimals":1,"higher_is_better":false,"public_history":true,"protocol":{"en":"Table 4 per-dataset mcMAE from the corruption protocol introduced in arXiv:2610.36340v2 (28 Oct 2026). mcMAE is the authors’ published mean corruption error. FP32 and INT8 are configurations. The four image collections stay separate and are not averaged. Table 3 clean MAE is the older counting leaderboard and is not included. Lower is better. This is a count error, not a percentage.","zh":"表 4 的分数据集 mcMAE，来自 arXiv:2610.36340v2（2026-10-28）提出的损坏协议。mcMAE 是作者公布的平均损坏误差。FP32 和 INT8 是配置。四个图像集分开，不平均。表 3 的干净 MAE 是更早的计数榜，不录入。越低越好。这是计数误差，不是百分比。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.36340v2","status":"paper_baselines","published_at":"2026-10-26"},"results":[[7448,107.0,370,7449,7450],[7448,107.0,370,7451,7450],[7452,110.0,370,7449,7450],[7452,109.5,370,7451,7450],[7453,104.9,370,7449,7450],[7453,105.9,370,7451,7450],[7454,120.8,370,7449,7450],[7454,120.1,370,7451,7450],[7455,117.7,370,7449,7450],[7455,118.4,370,7451,7450],[7456,136.1,370,7449,7450],[7456,131.0,370,7451,7450],[7457,104.0,370,7449,7450],[7457,106.8,370,7451,7450],[7458,129.1,370,7449,7450],[7458,127.9,370,7451,7450],[7459,109.2,370,7449,7450],[7459,106.1,370,7451,7450],[7460,93.9,370,7449,7450],[7460,91.7,370,7451,7450],[7461,134.8,370,7449,7450],[7461,135.8,370,7451,7450],[7462,135.9,370,7449,7450],[7462,142.7,370,7451,7450],[7463,141.2,370,7449,7450],[7463,144.1,370,7451,7450],[7464,133.9,370,7449,7450],[7464,138.5,370,7451,7450],[7465,88.4,370,7449,7450],[7465,91.4,370,7451,7450],[7466,99.4,370,7449,7450],[7466,105.8,370,7451,7450],[7467,133.1,370,7449,7450],[7467,130.7,370,7451,7450],[7468,210.6,370,7449,7450],[7468,204.9,370,7451,7450],[7469,106.5,370,7449,7450],[7469,107.5,370,7451,7450],[7470,102.0,370,7449,7450],[7470,106.3,370,7451,7450],[7471,102.3,370,7449,7450],[7471,106.1,370,7451,7450],[7472,99.5,370,7449,7450],[7472,101.2,370,7451,7450],[7473,119.1,370,7449,7450],[7473,115.9,370,7451,7450],[7474,106.3,370,7449,7450],[7474,111.0,370,7451,7450],[7475,114.1,370,7449,7450],[7475,110.9,370,7451,7450],[7476,116.5,370,7449,7450],[7476,117.0,370,7451,7450]]},{"id":"crowdcc-jhu-mcmae-2026-paper","name":"CrowdCC (JHU-Crowd++ corruption mcMAE, 2026 paper)","group":"视觉与视频","metric":"Corruption mcMAE","unit":"raw","score_decimals":1,"higher_is_better":false,"public_history":true,"protocol":{"en":"Table 4 per-dataset mcMAE from the corruption protocol introduced in arXiv:2610.36340v2 (28 Oct 2026). mcMAE is the authors’ published mean corruption error. FP32 and INT8 are configurations. The four image collections stay separate and are not averaged. Table 3 clean MAE is the older counting leaderboard and is not included. Lower is better. This is a count error, not a percentage.","zh":"表 4 的分数据集 mcMAE，来自 arXiv:2610.36340v2（2026-10-28）提出的损坏协议。mcMAE 是作者公布的平均损坏误差。FP32 和 INT8 是配置。四个图像集分开，不平均。表 3 的干净 MAE 是更早的计数榜，不录入。越低越好。这是计数误差，不是百分比。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.36340v2","status":"paper_baselines","published_at":"2026-10-26"},"results":[[7448,72.2,370,7449,7450],[7448,72.8,370,7451,7450],[7452,73.5,370,7449,7450],[7452,73.0,370,7451,7450],[7453,74.7,370,7449,7450],[7453,75.1,370,7451,7450],[7454,86.2,370,7449,7450],[7454,85.8,370,7451,7450],[7455,77.5,370,7449,7450],[7455,77.2,370,7451,7450],[7456,80.9,370,7449,7450],[7456,81.2,370,7451,7450],[7457,72.8,370,7449,7450],[7457,75.6,370,7451,7450],[7458,82.8,370,7449,7450],[7458,85.4,370,7451,7450],[7459,79.8,370,7449,7450],[7459,73.1,370,7451,7450],[7460,64.0,370,7449,7450],[7460,63.9,370,7451,7450],[7461,186.2,370,7449,7450],[7461,192.3,370,7451,7450],[7462,79.2,370,7449,7450],[7462,79.8,370,7451,7450],[7463,66.9,370,7449,7450],[7463,67.7,370,7451,7450],[7464,85.0,370,7449,7450],[7464,95.8,370,7451,7450],[7465,67.2,370,7449,7450],[7465,65.9,370,7451,7450],[7466,75.4,370,7449,7450],[7466,70.8,370,7451,7450],[7467,75.4,370,7449,7450],[7467,75.7,370,7451,7450],[7468,144.0,370,7449,7450],[7468,142.8,370,7451,7450],[7469,67.5,370,7449,7450],[7469,67.9,370,7451,7450],[7470,78.4,370,7449,7450],[7470,77.4,370,7451,7450],[7471,69.2,370,7449,7450],[7471,70.9,370,7451,7450],[7472,63.8,370,7449,7450],[7472,67.8,370,7451,7450],[7473,68.8,370,7449,7450],[7473,72.3,370,7451,7450],[7474,71.8,370,7449,7450],[7474,73.5,370,7451,7450],[7475,70.9,370,7449,7450],[7475,68.4,370,7451,7450],[7476,69.6,370,7449,7450],[7476,68.9,370,7451,7450]]},{"id":"sate-active-avg-2026-paper","name":"SATE-Bench (active average accuracy, 2026 paper)","group":"视觉与视频","metric":"Average accuracy","unit":"%","public_history":true,"protocol":{"en":"Table 4 average accuracy in the active multi-turn setting. The passive setting is separate. Task columns, steps, reward and inference time are not entered. The active row spells LLaVA-OneVision2-8B; it is the same checkpoint as the passive row.","zh":"表 4 主动多轮设置的平均准确率。被动设置是另一组。任务列、步数、奖励和推理时间不录入。主动行把 LLaVA 写成 OneVision2，与被动行是同一检查点。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.34149","status":"paper_baselines","published_at":"2026-10-25"},"results":[[5115,0.568,370,7477,7478],[7479,0.38,370,7477,7478],[7480,0.45399999999999996,370,7477,7478],[7481,0.281,370,7477,7478],[7482,0.46399999999999997,370,7477,7478],[7483,0.414,370,7477,7478],[7484,0.38,370,7477,7478],[2478,0.304,370,7477,7478],[7485,0.361,370,7477,7478],[7486,0.31,370,7477,7478],[6150,0.307,370,7477,7478],[2626,0.298,370,7477,7478]]},{"id":"sate-passive-avg-2026-paper","name":"SATE-Bench (passive average accuracy, 2026 paper)","group":"视觉与视频","metric":"Average accuracy","unit":"%","public_history":true,"protocol":{"en":"Table 4 average accuracy in the passive single-turn setting. It is not averaged with the active accuracy. Dashes in the step, reward and rate columns are not scores and are not filled.","zh":"表 4 被动单轮设置的平均准确率。不与主动准确率平均。步数、奖励和比率列的破折号不是分数，不补。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.34149","status":"paper_baselines","published_at":"2026-10-25"},"results":[[5115,0.6920000000000001,370,7487,7478],[7479,0.626,370,7487,7478],[7480,0.585,370,7487,7478],[7481,0.451,370,7487,7478],[7482,0.63,370,7487,7478],[7483,0.6459999999999999,370,7487,7478],[7484,0.5539999999999999,370,7487,7478],[2478,0.509,370,7487,7478],[7485,0.589,370,7487,7478],[7486,0.402,370,7487,7478],[6150,0.556,370,7487,7478],[2626,0.5770000000000001,370,7487,7478]]},{"id":"infiniteworld-text-2026-paper","name":"InfiniteWorld (text only, 2026 paper)","group":"视觉与视频","metric":"Full-set score","unit":"%","public_history":true,"protocol":{"en":"Table 2 full-set score for this input setting. Section 3.3 says each reported cell contains every assigned question, and the results text says Table 2 compares models on the same full question set, so these rows share one denominator. A setting that was not run is omitted and is not folded into the percentage. A failed response inside a completed cell counts as zero under the authors’ rule for a finished evaluation. The other input settings are separate cohorts. The gain column is their difference and is not entered again.","zh":"表 2 这一输入设置的全卷分数。3.3 节写明每个已报告的格子包含全部指定题目，结果节写明各模型比的是同一套全卷，所以这些行的分母相同。没跑的设置不录入，也不折进百分比。已完成格子里的失败回答按作者的规则计 0。其他输入设置是另外的组。增益列是它们的差，不再录入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.33777","status":"paper_baselines","published_at":"2026-10-24"},"results":[[6343,0.33409999999999995,370,7488,7489],[6238,0.30760000000000004,370,7488,7489],[6342,0.3567,370,7488,7489],[6298,0.322,370,7488,7489],[6341,0.3079,370,7488,7489],[6391,0.3398,370,7488,7489],[2754,0.3429,370,7488,7489],[2771,0.3093,370,7488,7489],[2485,0.32130000000000003,370,7488,7489],[2800,0.2756,370,7488,7489],[5300,0.36479999999999996,370,7488,7489],[7490,0.3097,370,7488,7489],[3494,0.332,370,7491,7489],[762,0.3665,370,7491,7489],[3408,0.3312,370,7491,7489],[2654,0.3215,370,7488,7489],[2728,0.38280000000000003,370,7488,7489]]},{"id":"infiniteworld-sim-ordered-2026-paper","name":"InfiniteWorld (simulator ordered, 2026 paper)","group":"视觉与视频","metric":"Full-set score","unit":"%","public_history":true,"protocol":{"en":"Table 2 full-set score for this input setting. Section 3.3 says each reported cell contains every assigned question, and the results text says Table 2 compares models on the same full question set, so these rows share one denominator. A setting that was not run is omitted and is not folded into the percentage. A failed response inside a completed cell counts as zero under the authors’ rule for a finished evaluation. The other input settings are separate cohorts. The gain column is their difference and is not entered again.","zh":"表 2 这一输入设置的全卷分数。3.3 节写明每个已报告的格子包含全部指定题目，结果节写明各模型比的是同一套全卷，所以这些行的分母相同。没跑的设置不录入，也不折进百分比。已完成格子里的失败回答按作者的规则计 0。其他输入设置是另外的组。增益列是它们的差，不再录入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.33777","status":"paper_baselines","published_at":"2026-10-24"},"results":[[6343,0.4754,370,7488,7489],[6238,0.4387,370,7488,7489],[6342,0.4576,370,7488,7489],[6298,0.4326,370,7488,7489],[6341,0.3555,370,7488,7489],[6391,0.3967,370,7488,7489],[2754,0.39840000000000003,370,7488,7489],[2771,0.2768,370,7488,7489],[2485,0.3256,370,7488,7489],[2800,0.2948,370,7488,7489],[5300,0.4315,370,7488,7489],[7490,0.3821,370,7488,7489],[3494,0.38079999999999997,370,7491,7489],[762,0.5589999999999999,370,7491,7489],[3408,0.4484,370,7491,7489],[2654,0.3342,370,7488,7489],[2728,0.43189999999999995,370,7488,7489]]},{"id":"infiniteworld-sim-shuffled-2026-paper","name":"InfiniteWorld (simulator shuffled, 2026 paper)","group":"视觉与视频","metric":"Full-set score","unit":"%","public_history":true,"protocol":{"en":"Table 2 full-set score for this input setting. Section 3.3 says each reported cell contains every assigned question, and the results text says Table 2 compares models on the same full question set, so these rows share one denominator. A setting that was not run is omitted and is not folded into the percentage. A failed response inside a completed cell counts as zero under the authors’ rule for a finished evaluation. The other input settings are separate cohorts. The gain column is their difference and is not entered again.","zh":"表 2 这一输入设置的全卷分数。3.3 节写明每个已报告的格子包含全部指定题目，结果节写明各模型比的是同一套全卷，所以这些行的分母相同。没跑的设置不录入，也不折进百分比。已完成格子里的失败回答按作者的规则计 0。其他输入设置是另外的组。增益列是它们的差，不再录入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.33777","status":"paper_baselines","published_at":"2026-10-24"},"results":[[6343,0.4535,370,7488,7489],[6238,0.41119999999999995,370,7488,7489],[6342,0.4275,370,7488,7489],[6298,0.4125,370,7488,7489],[6341,0.3505,370,7488,7489],[6391,0.4061,370,7488,7489],[2754,0.3907,370,7488,7489],[2771,0.2726,370,7488,7489],[2485,0.32530000000000003,370,7488,7489],[2800,0.2959,370,7488,7489],[5300,0.41409999999999997,370,7488,7489],[7490,0.38270000000000004,370,7488,7489],[3494,0.365,370,7491,7489],[762,0.4375,370,7491,7489],[3408,0.3538,370,7491,7489],[2654,0.3311,370,7488,7489],[2728,0.3986,370,7488,7489]]},{"id":"infiniteworld-trans-ordered-2026-paper","name":"InfiniteWorld (translated ordered, 2026 paper)","group":"视觉与视频","metric":"Full-set score","unit":"%","public_history":true,"protocol":{"en":"Table 2 full-set score for this input setting. Section 3.3 says each reported cell contains every assigned question, and the results text says Table 2 compares models on the same full question set, so these rows share one denominator. A setting that was not run is omitted and is not folded into the percentage. A failed response inside a completed cell counts as zero under the authors’ rule for a finished evaluation. The other input settings are separate cohorts. The gain column is their difference and is not entered again.","zh":"表 2 这一输入设置的全卷分数。3.3 节写明每个已报告的格子包含全部指定题目，结果节写明各模型比的是同一套全卷，所以这些行的分母相同。没跑的设置不录入，也不折进百分比。已完成格子里的失败回答按作者的规则计 0。其他输入设置是另外的组。增益列是它们的差，不再录入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.33777","status":"paper_baselines","published_at":"2026-10-24"},"results":[[6343,0.4268,370,7488,7489],[6238,0.4003,370,7488,7489],[6342,0.4035,370,7488,7489],[6298,0.3921,370,7488,7489],[6341,0.3335,370,7488,7489],[6391,0.3486,370,7488,7489],[2754,0.3679,370,7488,7489],[2771,0.2955,370,7488,7489],[2485,0.3129,370,7488,7489],[2800,0.2891,370,7488,7489],[5300,0.4061,370,7488,7489],[7490,0.35479999999999995,370,7488,7489],[3494,0.34850000000000003,370,7491,7489],[762,0.47840000000000005,370,7491,7489],[3408,0.3856,370,7491,7489],[2654,0.2915,370,7488,7489],[2728,0.3822,370,7488,7489]]},{"id":"infiniteworld-trans-shuffled-2026-paper","name":"InfiniteWorld (translated shuffled, 2026 paper)","group":"视觉与视频","metric":"Full-set score","unit":"%","public_history":true,"protocol":{"en":"Table 2 full-set score for this input setting. Section 3.3 says each reported cell contains every assigned question, and the results text says Table 2 compares models on the same full question set, so these rows share one denominator. A setting that was not run is omitted and is not folded into the percentage. A failed response inside a completed cell counts as zero under the authors’ rule for a finished evaluation. The other input settings are separate cohorts. The gain column is their difference and is not entered again.","zh":"表 2 这一输入设置的全卷分数。3.3 节写明每个已报告的格子包含全部指定题目，结果节写明各模型比的是同一套全卷，所以这些行的分母相同。没跑的设置不录入，也不折进百分比。已完成格子里的失败回答按作者的规则计 0。其他输入设置是另外的组。增益列是它们的差，不再录入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.33777","status":"paper_baselines","published_at":"2026-10-24"},"results":[[6343,0.425,370,7488,7489],[6238,0.3912,370,7488,7489],[6342,0.40149999999999997,370,7488,7489],[6298,0.39189999999999997,370,7488,7489],[6341,0.3348,370,7488,7489],[6391,0.3488,370,7488,7489],[2754,0.3726,370,7488,7489],[2771,0.2932,370,7488,7489],[2485,0.3147,370,7488,7489],[2800,0.2908,370,7488,7489],[5300,0.401,370,7488,7489],[7490,0.3619,370,7488,7489],[3494,0.3381,370,7491,7489],[762,0.4094,370,7491,7489],[3408,0.34979999999999994,370,7491,7489],[2654,0.2925,370,7488,7489],[2728,0.3795,370,7488,7489]]},{"id":"prefdiag-basic-2026-paper","name":"PrefDiag-Bench (basic task success, 2026 paper)","group":"Agent 与工具","metric":"Basic TSR","unit":"%","public_history":true,"protocol":{"en":"Table 2 Basic task success on the 53 base tasks. Explicit-profile and memory-log averages are separate cohorts. M3A and T3A are two workflows on GPT-5.4. Printed zeros stay.","zh":"表 2 在 53 个基础任务上的成功率。显式档案和记忆日志的平均是另外两组。M3A 和 T3A 是 GPT-5.4 的两种工作流。印出的 0 保留。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.30573","status":"paper_baselines","published_at":"2026-10-22"},"results":[[3184,0.41509999999999997,370,7492,7493],[3184,0.5472,370,7494,7493],[2626,0.3019,370,7495,7493],[7496,0.3774,370,7495,7493],[6150,0.35850000000000004,370,7495,7493],[2156,0.5849,370,7495,7493],[7497,0.5660000000000001,370,7495,7493],[7498,0.5283,370,7495,7493],[4961,0.6792,370,7495,7493],[3572,0.8679000000000001,370,7495,7493],[3192,0.8112999999999999,370,7495,7493]]},{"id":"prefdiag-profile-avg-2026-paper","name":"PrefDiag-Bench (explicit-profile average, 2026 paper)","group":"Agent 与工具","metric":"Explicit-profile average","unit":"%","public_history":true,"protocol":{"en":"Table 2 AVG under explicit user profiles. The caption defines it as the average of the five preference columns. It is not averaged with the memory-log column.","zh":"表 2 显式用户档案下的 AVG。题注把它定义为五个偏好列的平均。不与记忆日志列平均。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.30573","status":"paper_baselines","published_at":"2026-10-22"},"results":[[3184,0.2491,370,7492,7493],[3184,0.37729999999999997,370,7494,7493],[2626,0.11699999999999999,370,7495,7493],[7496,0.1321,370,7495,7493],[6150,0.1509,370,7495,7493],[2156,0.2415,370,7495,7493],[7497,0.2755,370,7495,7493],[7498,0.2793,370,7495,7493],[4961,0.44530000000000003,370,7495,7493],[3572,0.5547,370,7495,7493],[3192,0.6264,370,7495,7493]]},{"id":"prefdiag-memory-avg-2026-paper","name":"PrefDiag-Bench (memory-log average, 2026 paper)","group":"Agent 与工具","metric":"Memory-log average","unit":"%","public_history":true,"protocol":{"en":"Table 2 AVG under behavioral memory logs, the authors’ average of those five preference columns.","zh":"表 2 行为记忆日志下的 AVG，是作者对那五个偏好列的平均。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.30573","status":"paper_baselines","published_at":"2026-10-22"},"results":[[3184,0.2944,370,7492,7493],[3184,0.18489999999999998,370,7494,7493],[2626,0.1207,370,7495,7493],[7496,0.1245,370,7495,7493],[6150,0.18109999999999998,370,7495,7493],[2156,0.2491,370,7495,7493],[7497,0.24530000000000002,370,7495,7493],[7498,0.2943,370,7495,7493],[4961,0.4189,370,7495,7493],[3572,0.5357999999999999,370,7495,7493],[3192,0.6566,370,7495,7493]]},{"id":"iron-identity-f1-2026-paper","name":"IRON-Meme (social-identity F1, 2026 paper)","group":"视觉与视频","metric":"Social-identity F1","unit":"score","public_history":true,"protocol":{"en":"Table 2 social-identity F1. Viewpoint and narrative-role F1 are separate. Precision and recall are not entered again. The stratified random row is not a model.","zh":"表 2 社会身份 F1。观点和叙事角色是另外两组。精确率和召回率不再录入。分层随机行不是模型。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.33097","status":"paper_baselines","published_at":"2026-10-22"},"score_max":100,"saturation_excluded":true,"score_decimals":2,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"results":[[7499,13.07,370,7264,7500],[7501,51.99,370,7264,7500],[7502,38.6,370,7264,7500],[3155,52.78,370,7264,7500],[2916,5.8,370,7264,7500],[2515,35.58,370,7264,7500],[5816,26.41,370,7264,7500],[7503,36.35,370,7264,7500],[7504,43.42,370,7264,7500]]},{"id":"iron-viewpoint-f1-2026-paper","name":"IRON-Meme (viewpoint F1, 2026 paper)","group":"视觉与视频","metric":"Viewpoint F1","unit":"score","public_history":true,"protocol":{"en":"Table 2 viewpoint F1. The other label groups are separate and are not averaged with this column.","zh":"表 2 观点 F1。其他标签组分开，不与这一列平均。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.33097","status":"paper_baselines","published_at":"2026-10-22"},"score_max":100,"saturation_excluded":true,"score_decimals":2,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"results":[[7499,77.1,370,7264,7500],[7501,93.2,370,7264,7500],[7502,79.61,370,7264,7500],[3155,92.61,370,7264,7500],[2916,71.08,370,7264,7500],[2515,77.98,370,7264,7500],[5816,72.78,370,7264,7500],[7503,77.05,370,7264,7500],[7504,77.93,370,7264,7500]]},{"id":"iron-role-f1-2026-paper","name":"IRON-Meme (narrative-role F1, 2026 paper)","group":"视觉与视频","metric":"Narrative-role F1","unit":"score","public_history":true,"protocol":{"en":"Table 2 narrative-role F1. It is the authors’ published F1, not a new combination of precision and recall.","zh":"表 2 叙事角色 F1。这是作者公布的 F1，不是把精确率和召回率再算一次。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.33097","status":"paper_baselines","published_at":"2026-10-22"},"score_max":100,"saturation_excluded":true,"score_decimals":2,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"results":[[7499,34.44,370,7264,7500],[7501,64.14,370,7264,7500],[7502,46.55,370,7264,7500],[3155,57.88,370,7264,7500],[2916,11.7,370,7264,7500],[2515,40.6,370,7264,7500],[5816,23.52,370,7264,7500],[7503,41.74,370,7264,7500],[7504,45.51,370,7264,7500]]},{"id":"wmeval-scene-retention-2026-paper","name":"WMEval (scene retention, 2026 paper)","group":"视觉与视频","metric":"Scene retention","unit":"%","public_history":true,"protocol":{"en":"Table 8 scene retention on SunRisesAndSets. Higher is better. The mean over five rollouts is entered; the printed standard deviation is not. The other five lighting columns are separate cohorts. This rate is not task accuracy and is excluded from accuracy saturation.","zh":"表 8 在 SunRisesAndSets 上的场景保持率。越高越好。录入五次生成的均值，不录入标准差。另外五个光照列是分开的组。 这不是任务准确率，不进入准确率饱和。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.33894","status":"paper_baselines","published_at":"2026-10-24"},"saturation_excluded":true,"scale_label":{"en":"Percent","zh":"百分比"},"results":[[7505,0.9648,370,7506,7507],[7238,1.0,370,7506,7507],[7233,0.22920000000000001,370,7506,7507],[7239,0.1441,370,7506,7507],[7508,0.7392,370,7506,7507],[7509,1.0,370,7506,7507]]},{"id":"wmeval-shadow-detection-2026-paper","name":"WMEval (shadow detection, 2026 paper)","group":"视觉与视频","metric":"Shadow detection","unit":"%","public_history":true,"protocol":{"en":"Table 8 shadow detection rate. Higher is better. It is not averaged with scene retention. This rate is not task accuracy and is excluded from accuracy saturation.","zh":"表 8 的影子检出率。越高越好。不与场景保持率平均。 这不是任务准确率，不进入准确率饱和。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.33894","status":"paper_baselines","published_at":"2026-10-24"},"saturation_excluded":true,"scale_label":{"en":"Percent","zh":"百分比"},"results":[[7505,0.9737,370,7506,7507],[7238,1.0,370,7506,7507],[7233,1.0,370,7506,7507],[7239,0.915,370,7506,7507],[7508,0.9978,370,7506,7507],[7509,1.0,370,7506,7507]]},{"id":"wmeval-sun-disagreement-2026-paper","name":"WMEval (sun disagreement, 2026 paper)","group":"视觉与视频","metric":"Sun disagreement","unit":"raw","score_decimals":2,"public_history":true,"protocol":{"en":"Table 8 multi-object sun disagreement in degrees. Lower is better. This is not a percentage.","zh":"表 8 的多物体太阳方向分歧，单位是度。越低越好。不是百分比。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.33894","status":"paper_baselines","published_at":"2026-10-24"},"higher_is_better":false,"results":[[7505,65.78,370,7506,7507],[7238,60.06,370,7506,7507],[7233,65.26,370,7506,7507],[7239,59.07,370,7506,7507],[7508,77.29,370,7506,7507],[7509,98.29,370,7506,7507]]},{"id":"wmeval-shadow-jitter-2026-paper","name":"WMEval (shadow jitter, 2026 paper)","group":"视觉与视频","metric":"Shadow jitter","unit":"raw","score_decimals":2,"public_history":true,"protocol":{"en":"Table 8 shadow-direction jitter in degrees per frame. Lower is better.","zh":"表 8 的影子方向抖动，单位是每帧的度。越低越好。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.33894","status":"paper_baselines","published_at":"2026-10-24"},"higher_is_better":false,"results":[[7505,2.42,370,7506,7507],[7238,7.61,370,7506,7507],[7233,4.12,370,7506,7507],[7239,23.18,370,7506,7507],[7508,6.6,370,7506,7507],[7509,2.34,370,7506,7507]]},{"id":"wmeval-shadow-area-2026-paper","name":"WMEval (shadow area stability, 2026 paper)","group":"视觉与视频","metric":"Shadow area change","unit":"%","public_history":true,"protocol":{"en":"Table 8 shadow-area stability. Lower is better. The percent is a change, not an accuracy. This rate is not task accuracy and is excluded from accuracy saturation.","zh":"表 8 的影子面积稳定度。越低越好。这个百分比是变化量，不是准确率。 这不是任务准确率，不进入准确率饱和。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.33894","status":"paper_baselines","published_at":"2026-10-24"},"higher_is_better":false,"saturation_excluded":true,"scale_label":{"en":"Percent","zh":"百分比"},"results":[[7505,0.13949999999999999,370,7506,7507],[7238,0.1797,370,7506,7507],[7233,0.1422,370,7506,7507],[7239,0.6243,370,7506,7507],[7508,0.10300000000000001,370,7506,7507],[7509,0.1229,370,7506,7507]]},{"id":"wmeval-shadow-drift-2026-paper","name":"WMEval (shadow shape drift, 2026 paper)","group":"视觉与视频","metric":"Shadow shape drift","unit":"raw","score_decimals":2,"public_history":true,"protocol":{"en":"Table 8 shadow-shape drift in Hu L2. Lower is better. This is not a percentage.","zh":"表 8 的影子形状漂移，单位是 Hu L2。越低越好。不是百分比。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.33894","status":"paper_baselines","published_at":"2026-10-24"},"higher_is_better":false,"results":[[7505,6.78,370,7506,7507],[7238,6.66,370,7506,7507],[7233,7.37,370,7506,7507],[7239,11.43,370,7506,7507],[7508,5.75,370,7506,7507],[7509,4.55,370,7506,7507]]},{"id":"adshot-f1-2026-paper","name":"AdShot (shot-selection F1, 2026 paper)","group":"视觉与视频","metric":"F1","unit":"score","score_max":1,"score_decimals":3,"public_history":true,"protocol":{"en":"HTML Table 4 F1 at one frame per second. This is the published F1, not a new combination of precision and recall. Duration error is a separate cohort. The random baseline is not a model. Video-only and audio-visual inputs are configurations.","zh":"HTML 表 4 在每秒一帧下的 F1。这是公布的 F1，不是把精确率和召回率再算一次。时长误差是另一组。随机基线不是模型。只有视频和音视频是配置。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.33377","status":"paper_baselines","published_at":"2026-10-24"},"results":[[2824,0.582,370,7510,7511],[7512,0.539,370,7510,7511],[7513,0.576,370,7510,7511],[7514,0.613,370,7510,7511],[5987,0.494,370,7510,7511],[7515,0.475,370,7510,7511],[7516,0.483,370,7510,7511],[7517,0.473,370,7510,7511],[7518,0.676,370,7510,7511],[7519,0.611,370,7510,7511],[7520,0.536,370,7510,7511],[7521,0.443,370,7510,7511],[7522,0.464,370,7510,7511],[7523,0.668,370,7375,7511],[7524,0.669,370,7375,7511]]},{"id":"adshot-mde-2026-paper","name":"AdShot (duration error, 2026 paper)","group":"视觉与视频","metric":"Mean duration error","unit":"raw","score_decimals":2,"higher_is_better":false,"public_history":true,"protocol":{"en":"HTML Table 4 mean absolute duration error in seconds. Lower is better. It is not an accuracy and is not averaged with F1.","zh":"HTML 表 4 的平均绝对时长误差，单位是秒。越低越好。不是准确率，也不与 F1 平均。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.33377","status":"paper_baselines","published_at":"2026-10-24"},"results":[[2824,7.82,370,7510,7511],[7512,8.68,370,7510,7511],[7513,4.08,370,7510,7511],[7514,8.3,370,7510,7511],[5987,2.89,370,7510,7511],[7515,4.45,370,7510,7511],[7516,4.13,370,7510,7511],[7517,4.1,370,7510,7511],[7518,11.95,370,7510,7511],[7519,3.71,370,7510,7511],[7520,2.49,370,7510,7511],[7521,4.27,370,7510,7511],[7522,3.28,370,7510,7511],[7523,11.18,370,7375,7511],[7524,10.06,370,7375,7511]]},{"id":"vera-hand-posef1-2026-paper","name":"VERA-Hand (Pose-F1, 2026 paper)","group":"视觉与视频","metric":"Pose-F1","unit":"score","score_max":1,"score_decimals":3,"public_history":true,"protocol":{"en":"Table 3 Pose-F1 at the paper’s joint detection-and-localization threshold. The other columns are not entered. Sapiens2-1B is the Table 2 name for the score row printed as Sapiens21B. The six rows are the named estimator checkpoints. The detector named in Table 2 is part of that checkpoint’s setup, not an additional model.","zh":"表 3 在论文联合检测和定位阈值下的 Pose-F1。其他列不录入。Sapiens2-1B 是表 2 的名字，分数行印成 Sapiens21B。 这六行是具名的估计器检查点。表 2 里的检测器是该检查点的配套，不另算一个模型。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.32517","status":"paper_baselines","published_at":"2026-10-23"},"results":[[7525,0.309,370,7526,7527],[7528,0.389,370,7526,7527],[7529,0.407,370,7526,7527],[7530,0.477,370,7526,7527],[7531,0.34,370,7526,7527],[7532,0.361,370,7526,7527]]},{"id":"tracetrial-agent-pass-2026-paper","name":"TraceTrial-Bench (agent task success, 2026 paper)","group":"Agent 与工具","metric":"Agent pass","unit":"%","public_history":true,"protocol":{"en":"Table 7 Agent Pass: every agent criterion must pass. The four harnesses are configurations of the same nine model checkpoints. The all-model and all-harness rows are not entered.","zh":"表 7 的 Agent Pass，要求每个智能体准则都通过。四种框架是这九个模型检查点的配置。全体模型和全体框架的汇总行不录入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.32285","status":"paper_baselines","published_at":"2026-10-23"},"results":[[3185,0.43200000000000005,370,3759,7533],[3185,0.305,370,2300,7533],[3185,0.379,370,5380,7533],[3185,0.389,370,7534,7533],[3386,0.4,370,3759,7533],[3386,0.389,370,2300,7533],[3386,0.442,370,5380,7533],[3386,0.389,370,7534,7533],[3154,0.43200000000000005,370,3759,7533],[3154,0.284,370,2300,7533],[3154,0.442,370,5380,7533],[3154,0.41100000000000003,370,7534,7533],[4979,0.442,370,3759,7533],[4979,0.41100000000000003,370,2300,7533],[4979,0.368,370,5380,7533],[4979,0.42100000000000004,370,7534,7533],[5295,0.5579999999999999,370,3759,7533],[5295,0.442,370,2300,7533],[5295,0.43200000000000005,370,5380,7533],[5295,0.337,370,7534,7533],[3158,0.326,370,3759,7533],[3158,0.305,370,2300,7533],[3158,0.284,370,5380,7533],[3158,0.295,370,7534,7533],[2977,0.495,370,3759,7533],[2977,0.42100000000000004,370,2300,7533],[2977,0.43200000000000005,370,5380,7533],[2977,0.4,370,7534,7533],[3156,0.537,370,3759,7533],[3156,0.305,370,2300,7533],[3156,0.495,370,5380,7533],[3156,0.41100000000000003,370,7534,7533],[3157,0.337,370,3759,7533],[3157,0.358,370,2300,7533],[3157,0.326,370,5380,7533],[3157,0.389,370,7534,7533]]},{"id":"tracetrial-agent-score-2026-paper","name":"TraceTrial-Bench (agent score, 2026 paper)","group":"Agent 与工具","metric":"Agent score","unit":"%","saturation_excluded":true,"scale_label":{"en":"Percent","zh":"百分比"},"public_history":true,"protocol":{"en":"Table 7 Agent Score, the authors’ average of run-level fractions. It is not task accuracy and is excluded from accuracy saturation. Harnesses stay configurations and are not averaged.","zh":"表 7 的 Agent Score，是作者对逐次运行比例的平均。它不是任务准确率，不进入准确率饱和。框架是配置，不平均。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.32285","status":"paper_baselines","published_at":"2026-10-23"},"results":[[3185,0.725,370,3759,7533],[3185,0.684,370,2300,7533],[3185,0.6990000000000001,370,5380,7533],[3185,0.72,370,7534,7533],[3386,0.741,370,3759,7533],[3386,0.738,370,2300,7533],[3386,0.746,370,5380,7533],[3386,0.742,370,7534,7533],[3154,0.76,370,3759,7533],[3154,0.6859999999999999,370,2300,7533],[3154,0.755,370,5380,7533],[3154,0.675,370,7534,7533],[4979,0.7959999999999999,370,3759,7533],[4979,0.718,370,2300,7533],[4979,0.732,370,5380,7533],[4979,0.715,370,7534,7533],[5295,0.816,370,3759,7533],[5295,0.7290000000000001,370,2300,7533],[5295,0.7559999999999999,370,5380,7533],[5295,0.708,370,7534,7533],[3158,0.695,370,3759,7533],[3158,0.642,370,2300,7533],[3158,0.648,370,5380,7533],[3158,0.7,370,7534,7533],[2977,0.782,370,3759,7533],[2977,0.74,370,2300,7533],[2977,0.7440000000000001,370,5380,7533],[2977,0.738,370,7534,7533],[3156,0.8340000000000001,370,3759,7533],[3156,0.703,370,2300,7533],[3156,0.7929999999999999,370,5380,7533],[3156,0.7609999999999999,370,7534,7533],[3157,0.6829999999999999,370,3759,7533],[3157,0.721,370,2300,7533],[3157,0.718,370,5380,7533],[3157,0.69,370,7534,7533]]},{"id":"melange-bbq-fair-2026-paper","name":"Melange (BBQ text FaiR, 2026 paper)","group":"安全与可靠性","metric":"BBQ FaiR","unit":"score","score_max":1,"score_decimals":3,"higher_is_better":true,"public_history":true,"protocol":{"en":"HTML Table 2 text-only BBQ column. FaiR is the published fairness index, not task accuracy. The vision gap and recovery columns are differences and are not entered. The mean row is not a model. Twenty-three named checkpoints. It is excluded from accuracy saturation.","zh":"HTML 表 2 的纯文本 BBQ 列。FaiR 是公布的公平指数，不是任务准确率。视觉差和恢复列是差值，不录入。均值行不是模型。二十三个有名称的检查点。 不进入准确率饱和。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.29537","status":"paper_baselines","published_at":"2026-10-21"},"saturation_excluded":true,"scale_label":{"en":"FaiR / 1","zh":"公平指数 / 1"},"results":[[6389,0.477,370,370,7535],[2485,0.683,370,370,7535],[2771,0.474,370,370,7535],[2754,0.909,370,370,7535],[6391,0.935,370,370,7535],[2166,0.95,370,370,7535],[6341,0.37,370,370,7535],[6298,0.625,370,370,7535],[6342,0.831,370,370,7535],[6238,0.886,370,370,7535],[6343,0.916,370,370,7535],[3763,0.938,370,370,7535],[3764,0.956,370,370,7535],[2774,0.784,370,370,7535],[2728,0.932,370,370,7535],[6344,0.71,370,370,7535],[6246,0.602,370,370,7535],[2687,0.814,370,370,7535],[2654,0.852,370,370,7535],[6345,0.875,370,370,7535],[5534,0.917,370,370,7535],[2642,0.909,370,370,7535],[5479,0.912,370,370,7535]]},{"id":"melange-vision-fair-2026-paper","name":"Melange (vision FaiR, 2026 paper)","group":"安全与可靠性","metric":"vision FaiR","unit":"score","score_max":1,"score_decimals":3,"higher_is_better":true,"public_history":true,"protocol":{"en":"HTML Table 2 vision column on the image items. FaiR is the published fairness index, not task accuracy. The text column, the gap and recovery are separate and are not folded into this score. The mean row is not a model. It is excluded from accuracy saturation.","zh":"HTML 表 2 在图像题目上的视觉列。FaiR 是公布的公平指数，不是任务准确率。文本列、差值和恢复列不并进这个分数。均值行不是模型。 不进入准确率饱和。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.29537","status":"paper_baselines","published_at":"2026-10-21"},"saturation_excluded":true,"scale_label":{"en":"FaiR / 1","zh":"公平指数 / 1"},"results":[[6389,0.066,370,370,7535],[2485,0.209,370,370,7535],[2771,0.129,370,370,7535],[2754,0.657,370,370,7535],[6391,0.806,370,370,7535],[2166,0.852,370,370,7535],[6341,0.034,370,370,7535],[6298,0.467,370,370,7535],[6342,0.581,370,370,7535],[6238,0.599,370,370,7535],[6343,0.795,370,370,7535],[3763,0.8,370,370,7535],[3764,0.542,370,370,7535],[2774,0.557,370,370,7535],[2728,0.688,370,370,7535],[6344,0.262,370,370,7535],[6246,0.17,370,370,7535],[2687,0.381,370,370,7535],[2654,0.568,370,370,7535],[6345,0.427,370,370,7535],[5534,0.785,370,370,7535],[2642,0.384,370,370,7535],[5479,0.806,370,370,7535]]},{"id":"melange-context-fair-2026-paper","name":"Melange (neutral-context FaiR, 2026 paper)","group":"安全与可靠性","metric":"context FaiR","unit":"score","score_max":1,"score_decimals":3,"higher_is_better":true,"public_history":true,"protocol":{"en":"HTML Table 2 column after the authors add neutral text. FaiR is the published fairness index, not task accuracy. It is not an average of the other two settings. The mean row is not a model. It is excluded from accuracy saturation.","zh":"HTML 表 2 在作者加入中性文本之后的那一列。FaiR 是公布的公平指数，不是任务准确率。它不是另两列的平均。均值行不是模型。 不进入准确率饱和。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.29537","status":"paper_baselines","published_at":"2026-10-21"},"saturation_excluded":true,"scale_label":{"en":"FaiR / 1","zh":"公平指数 / 1"},"results":[[6389,0.056,370,370,7535],[2485,0.212,370,370,7535],[2771,0.101,370,370,7535],[2754,0.64,370,370,7535],[6391,0.816,370,370,7535],[2166,0.833,370,370,7535],[6341,0.107,370,370,7535],[6298,0.468,370,370,7535],[6342,0.608,370,370,7535],[6238,0.638,370,370,7535],[6343,0.741,370,370,7535],[3763,0.846,370,370,7535],[3764,0.564,370,370,7535],[2774,0.588,370,370,7535],[2728,0.806,370,370,7535],[6344,0.407,370,370,7535],[6246,0.256,370,370,7535],[2687,0.454,370,370,7535],[2654,0.675,370,370,7535],[6345,0.584,370,370,7535],[5534,0.85,370,370,7535],[2642,0.525,370,370,7535],[5479,0.86,370,370,7535]]},{"id":"tarsphere-nav-progress-2026-paper","name":"TARSphere-Bench (navigate-and-report progress, 2026 paper)","group":"Agent 与工具","metric":"progress","unit":"%","higher_is_better":true,"public_history":true,"protocol":{"en":"HTML Table 2 navigate-and-report only. Avg. is the authors’ mean of the six conditions in that task family, not a new average and not a mix of the other family. Cost, path length and safety incidence are not entered. Printed zeros stay. Eight frozen checkpoints share one harness. Progress is partial credit, not accuracy, and stays out of accuracy saturation.","zh":"只取 HTML 表 2 的 navigate-and-report。Avg. 是作者对该任务族六个条件的均值，不是重新平均，也不和另一个任务族混合。费用、路径和安全发生率不录入。表上的 0 保留。八个冻结检查点共用一个工具框架。 进度是部分得分，不是准确率，不进入准确率饱和。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.30043","status":"paper_baselines","published_at":"2026-10-21"},"saturation_excluded":true,"scale_label":{"en":"Percent","zh":"百分比"},"results":[[5727,0.504,370,370,7536],[7537,0.513,370,370,7536],[5519,0.672,370,370,7536],[4961,0.47600000000000003,370,370,7536],[5300,0.368,370,370,7536],[5480,0.48,370,370,7536],[7538,0.266,370,370,7536],[7539,0.132,370,370,7536]]},{"id":"tarsphere-nav-success-2026-paper","name":"TARSphere-Bench (navigate-and-report success, 2026 paper)","group":"Agent 与工具","metric":"success","unit":"%","higher_is_better":true,"public_history":true,"protocol":{"en":"HTML Table 2 navigate-and-report only. Avg. is the authors’ mean of the six conditions in that task family, not a new average and not a mix of the other family. Cost, path length and safety incidence are not entered. Printed zeros stay. Eight frozen checkpoints share one harness. Success is the published episode-completion rate.","zh":"只取 HTML 表 2 的 navigate-and-report。Avg. 是作者对该任务族六个条件的均值，不是重新平均，也不和另一个任务族混合。费用、路径和安全发生率不录入。表上的 0 保留。八个冻结检查点共用一个工具框架。 成功率是公布的整段任务完成率。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.30043","status":"paper_baselines","published_at":"2026-10-21"},"results":[[5727,0.15,370,370,7536],[7537,0.16899999999999998,370,370,7536],[5519,0.406,370,370,7536],[4961,0.111,370,370,7536],[5300,0.0,370,370,7536],[5480,0.134,370,370,7536],[7538,0.0,370,370,7536],[7539,0.0,370,370,7536]]},{"id":"tarsphere-fetch-progress-2026-paper","name":"TARSphere-Bench (fetch-and-deliver progress, 2026 paper)","group":"Agent 与工具","metric":"progress","unit":"%","higher_is_better":true,"public_history":true,"protocol":{"en":"HTML Table 2 fetch-and-deliver only. Avg. is the authors’ mean of the six conditions in that task family, not a new average and not a mix of the other family. Cost, path length and safety incidence are not entered. Printed zeros stay. Eight frozen checkpoints share one harness. Progress is partial credit, not accuracy, and stays out of accuracy saturation.","zh":"只取 HTML 表 2 的 fetch-and-deliver。Avg. 是作者对该任务族六个条件的均值，不是重新平均，也不和另一个任务族混合。费用、路径和安全发生率不录入。表上的 0 保留。八个冻结检查点共用一个工具框架。 进度是部分得分，不是准确率，不进入准确率饱和。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.30043","status":"paper_baselines","published_at":"2026-10-21"},"saturation_excluded":true,"scale_label":{"en":"Percent","zh":"百分比"},"results":[[5727,0.446,370,370,7536],[7537,0.514,370,370,7536],[5519,0.6940000000000001,370,370,7536],[4961,0.359,370,370,7536],[5300,0.305,370,370,7536],[5480,0.332,370,370,7536],[7538,0.19399999999999998,370,370,7536],[7539,0.106,370,370,7536]]},{"id":"tarsphere-fetch-success-2026-paper","name":"TARSphere-Bench (fetch-and-deliver success, 2026 paper)","group":"Agent 与工具","metric":"success","unit":"%","higher_is_better":true,"public_history":true,"protocol":{"en":"HTML Table 2 fetch-and-deliver only. Avg. is the authors’ mean of the six conditions in that task family, not a new average and not a mix of the other family. Cost, path length and safety incidence are not entered. Printed zeros stay. Eight frozen checkpoints share one harness. Success is the published episode-completion rate.","zh":"只取 HTML 表 2 的 fetch-and-deliver。Avg. 是作者对该任务族六个条件的均值，不是重新平均，也不和另一个任务族混合。费用、路径和安全发生率不录入。表上的 0 保留。八个冻结检查点共用一个工具框架。 成功率是公布的整段任务完成率。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.30043","status":"paper_baselines","published_at":"2026-10-21"},"results":[[5727,0.128,370,370,7536],[7537,0.24600000000000002,370,370,7536],[5519,0.478,370,370,7536],[4961,0.05,370,370,7536],[5300,0.0,370,370,7536],[5480,0.022000000000000002,370,370,7536],[7538,0.0,370,370,7536],[7539,0.0,370,370,7536]]},{"id":"ple-mean-score-2026-paper","name":"PLE (mean task score, 2026 paper)","group":"专业工作","metric":"mean score","unit":"%","higher_is_better":true,"saturation_excluded":true,"scale_label":{"en":"Percent","zh":"百分比"},"public_history":true,"protocol":{"en":"HTML Table 5 mean over all 85 tasks and three runs, as published. Harnesses are configurations. Empty cells are runs the table marks as not run and are omitted. Grok 4.7 is only the printed Grok Build cell. Printed zeros stay. The score is a rubric mean judged by Claude Opus 5.5, not task accuracy, and stays out of accuracy saturation. First posted 21 October 2026; this table is the 30 October 2026 revision.","zh":"HTML 表 5 对全部 85 题、三次运行的公布均值。工具框架是配置。空格是表上标明没跑的，不补。Grok 4.7 只保留印出的 Grok Build 一格。表上的 0 保留。分数是 Claude Opus 5.5 评判的量表均值，不是任务准确率，不进入准确率饱和。首次发布是 2026-10-21；这张表来自 2026-10-30 的修订。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.30537","status":"paper_baselines","published_at":"2026-10-21"},"results":[[2959,0.484,370,5162,7540],[2959,0.506,370,3759,7540],[2959,0.493,370,2300,7540],[2959,0.486,370,7541,7540],[2959,0.474,370,5393,7540],[2959,0.499,370,5302,7540],[2959,0.473,370,5948,7540],[5508,0.483,370,2300,7540],[2961,0.452,370,5162,7540],[2961,0.436,370,3759,7540],[2961,0.42700000000000005,370,2300,7540],[2961,0.43700000000000006,370,7541,7540],[2961,0.423,370,5393,7540],[2961,0.42200000000000004,370,5302,7540],[2961,0.435,370,5948,7540],[2962,0.41700000000000004,370,5162,7540],[2962,0.43200000000000005,370,3759,7540],[2962,0.40299999999999997,370,2300,7540],[2962,0.41100000000000003,370,7541,7540],[2962,0.384,370,5393,7540],[2962,0.311,370,5302,7540],[2962,0.402,370,5948,7540],[755,0.426,370,5162,7540],[755,0.408,370,3759,7540],[755,0.38799999999999996,370,2300,7540],[755,0.396,370,7541,7540],[755,0.39799999999999996,370,5393,7540],[755,0.419,370,5302,7540],[755,0.40399999999999997,370,5948,7540],[3453,0.33799999999999997,370,5162,7540],[3453,0.349,370,3759,7540],[3453,0.34700000000000003,370,2300,7540],[3453,0.32799999999999996,370,7541,7540],[3453,0.345,370,5393,7540],[3453,0.33399999999999996,370,5302,7540],[3453,0.33,370,5948,7540],[2968,0.32,370,5296,7540],[2983,0.264,370,5162,7540],[2983,0.282,370,3759,7540],[2983,0.276,370,2300,7540],[2983,0.17800000000000002,370,7541,7540],[2983,0.26,370,5393,7540],[2983,0.276,370,5302,7540],[2983,0.26899999999999996,370,5948,7540],[3654,0.23,370,5162,7540],[3654,0.223,370,3759,7540],[3654,0.253,370,2300,7540],[3654,0.239,370,7541,7540],[3654,0.244,370,5393,7540],[3654,0.22899999999999998,370,5302,7540],[3654,0.23800000000000002,370,5948,7540],[3412,0.13699999999999998,370,5162,7540],[3412,0.141,370,3759,7540],[3412,0.141,370,2300,7540],[3412,0.142,370,7541,7540],[3412,0.133,370,5393,7540],[3412,0.14,370,5302,7540],[3412,0.136,370,5948,7540],[7542,0.045,370,5162,7540],[7542,0.055999999999999994,370,3759,7540],[7542,0.044000000000000004,370,2300,7540],[7542,0.062,370,7541,7540],[7542,0.040999999999999995,370,5393,7540],[7542,0.057,370,5302,7540],[7542,0.043,370,5948,7540],[36,0.035,370,5162,7540],[36,0.044000000000000004,370,3759,7540],[36,0.03,370,2300,7540],[36,0.052000000000000005,370,7541,7540],[36,0.059000000000000004,370,5393,7540],[36,0.047,370,5302,7540],[36,0.042,370,5948,7540],[37,0.037000000000000005,370,5162,7540],[37,0.053,370,3759,7540],[37,0.044000000000000004,370,2300,7540],[37,0.053,370,7541,7540],[37,0.039,370,5393,7540],[37,0.045,370,5302,7540],[37,0.047,370,5948,7540],[362,0.001,370,5162,7540],[362,0.004,370,3759,7540],[362,0.0,370,2300,7540],[362,0.001,370,7541,7540],[362,0.0,370,5393,7540],[362,0.006999999999999999,370,5302,7540],[362,0.006999999999999999,370,5948,7540]]},{"id":"rite-rsr-correct-2026-paper","name":"RITE (correct-condition success, 2026 paper)","group":"Agent 与工具","metric":"RSR","unit":"score","score_max":1,"score_decimals":3,"higher_is_better":true,"public_history":true,"protocol":{"en":"HTML Table 1 Correct column. RSR is the authors’ end-to-end task success under the correct condition. Environment-error rollouts are excluded from the denominator, not scored as zero. The Noticed gap and the paired gap are differences and are not entered. The same exclusion is used for the correct, implicit, and explicit conditions.","zh":"HTML 表 1 的 Correct 列。RSR 是作者在正确条件下的端到端任务成功率。环境报错的回合从分母剔除，不记成 0。发现率差值和配对差值不录入。 正确、隐式和显式三个条件用同一条剔除规则。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.27777","status":"paper_baselines","published_at":"2026-10-20"},"results":[[3194,0.765,370,7543,7544],[3196,0.716,370,7543,7544],[3155,0.784,370,7543,7544],[227,0.805,370,7543,7544],[5114,0.806,370,7543,7544],[3185,0.827,370,7543,7544],[3191,0.809,370,7543,7544],[6771,0.823,370,7543,7544],[5619,0.791,370,7543,7544],[5620,0.784,370,7543,7544]]},{"id":"rite-noticed-2026-paper","name":"RITE (noticed rate, 2026 paper)","group":"Agent 与工具","metric":"Noticed","unit":"score","score_max":1,"score_decimals":3,"higher_is_better":true,"saturation_excluded":true,"scale_label":{"en":"Rate / 1","zh":"比率 / 1"},"public_history":true,"protocol":{"en":"HTML Table 1 Noticed columns. Implicit and explicit are configurations. Noticed is whether the agent flags a silent tool error, not end-to-end task accuracy, and it is excluded from accuracy saturation. The gap column is explicit minus implicit and is not entered. Environment-error rollouts are excluded, not filled as zero. The same exclusion is used for the implicit and explicit conditions.","zh":"HTML 表 1 的 Noticed 列。隐式和显式是配置。Noticed 是模型有没有指出静默工具错误，不是端到端任务准确率，不进入准确率饱和。差值列是显式减隐式，不录入。环境报错的回合被剔除，不补成 0。 隐式和显式两个条件用同一条剔除规则。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.27777","status":"paper_baselines","published_at":"2026-10-20"},"results":[[3194,0.336,370,7545,7544],[3194,0.579,370,7546,7544],[3196,0.28,370,7545,7544],[3196,0.594,370,7546,7544],[3155,0.263,370,7545,7544],[3155,0.59,370,7546,7544],[227,0.217,370,7545,7544],[227,0.487,370,7546,7544],[5114,0.215,370,7545,7544],[5114,0.494,370,7546,7544],[3185,0.212,370,7545,7544],[3185,0.468,370,7546,7544],[3191,0.21,370,7545,7544],[3191,0.477,370,7546,7544],[6771,0.248,370,7545,7544],[6771,0.512,370,7546,7544],[5619,0.211,370,7545,7544],[5619,0.512,370,7546,7544],[5620,0.208,370,7545,7544],[5620,0.506,370,7546,7544]]},{"id":"mmenterprise-overall-2026-paper","name":"MMEnterprise-Bench (overall accuracy, 2026 paper)","group":"专业工作","metric":"overall accuracy","unit":"%","higher_is_better":true,"public_history":true,"protocol":{"en":"HTML Table 5 Overall column. It is the published accuracy over all 1,613 questions. Retrieval depth k=10 and k=50 are configurations. Task columns are not averaged again.","zh":"HTML 表 5 的 Overall 列。这是公布的全部 1,613 题准确率。检索深度 k=10 和 k=50 是配置。各任务列不再另求平均。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.27911","status":"paper_baselines","published_at":"2026-10-20"},"results":[[762,0.526,370,7547,7548],[3408,0.513,370,7547,7548],[2983,0.47600000000000003,370,7547,7548],[3412,0.45899999999999996,370,7547,7548],[2584,0.386,370,7547,7548],[4070,0.349,370,7547,7548],[4969,0.36,370,7547,7548],[5828,0.23600000000000002,370,7547,7548],[2654,0.327,370,7547,7548],[7549,0.226,370,7547,7548],[762,0.68,370,7550,7548],[3408,0.6579999999999999,370,7550,7548],[2983,0.639,370,7550,7548],[3412,0.625,370,7550,7548],[2584,0.503,370,7550,7548],[4070,0.431,370,7550,7548],[4969,0.43799999999999994,370,7550,7548],[5828,0.25,370,7550,7548],[2654,0.366,370,7550,7548],[7549,0.231,370,7550,7548]]},{"id":"robojudge-overall-acc-2026-paper","name":"RoboJudge-Bench (overall accuracy, 2026 paper)","group":"视觉与视频","metric":"overall accuracy","unit":"score","score_max":1,"score_decimals":3,"higher_is_better":true,"public_history":true,"protocol":{"en":"HTML Table 2 Overall Acc. The authors define it as the average of physical-adherence accuracy and instruction-alignment accuracy. Correlation and error are not entered. The random baseline is not a model. A dash is a missing run and is omitted.","zh":"HTML 表 2 的 Overall Acc。作者把它定义为物理一致准确率和指令对齐准确率的平均。相关性和误差不录入。随机基线不是模型。破折号是没跑，不补。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.27782","status":"paper_baselines","published_at":"2026-10-20"},"results":[[5906,0.293,370,370,7551],[2129,0.344,370,370,7551],[502,0.309,370,370,7551],[3196,0.407,370,370,7551],[340,0.422,370,370,7551],[3194,0.339,370,370,7551],[6214,0.289,370,370,7551],[3202,0.302,370,370,7551],[4961,0.309,370,370,7551],[7552,0.415,370,370,7551],[2584,0.339,370,370,7551],[4070,0.285,370,370,7551],[4969,0.242,370,370,7551],[6342,0.22,370,370,7551],[6241,0.226,370,370,7551],[2642,0.218,370,370,7551],[6300,0.2,370,370,7551],[7553,0.268,370,370,7551],[7554,0.263,370,370,7551],[7555,0.185,370,370,7551],[7556,0.209,370,370,7551],[7557,0.183,370,370,7551],[7558,0.193,370,370,7551],[7559,0.194,370,370,7551],[7560,0.283,370,370,7551],[7561,0.291,370,370,7551],[7562,0.157,370,370,7551],[7563,0.345,370,370,7551],[7564,0.473,370,370,7551]]},{"id":"retinaground-f1-2026-paper","name":"RetinaGroundBench (grounding F1, 2026 paper)","group":"医疗","metric":"F1","unit":"score","score_max":100,"score_decimals":1,"higher_is_better":true,"saturation_excluded":true,"scale_label":{"en":"F1 / 100","zh":"F1 / 100"},"public_history":true,"protocol":{"en":"HTML Table 2 macro F1. The number is the published class-mean; the standard deviation is not entered. Family names apply to the following size rows. Each size is a separate checkpoint. F1 is not task accuracy and is excluded from accuracy saturation.","zh":"HTML 表 2 的宏平均 F1。数字是公布的类别均值，标准差不录入。系列名沿用到后面的尺寸行。每个尺寸是单独的检查点。F1 不是任务准确率，不进入准确率饱和。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.31205","status":"paper_baselines","published_at":"2026-10-22"},"results":[[6341,6.3,370,370,7565],[6298,16.9,370,370,7565],[6342,13.3,370,370,7565],[6343,18.8,370,370,7565],[6340,1.6,370,370,7565],[5828,4.0,370,370,7565],[4969,8.8,370,370,7565],[4070,13.0,370,370,7565],[2584,18.2,370,370,7565],[2590,16.6,370,370,7565],[3763,21.0,370,370,7565],[3764,13.3,370,370,7565],[7566,11.5,370,370,7565],[2780,4.0,370,370,7565],[2766,2.6,370,370,7565],[2728,4.1,370,370,7565],[2697,8.2,370,370,7565],[2706,7.4,370,370,7565],[6344,7.1,370,370,7565],[6246,13.1,370,370,7565],[2687,16.7,370,370,7565],[2654,15.4,370,370,7565],[6345,18.8,370,370,7565],[5534,23.7,370,370,7565],[2642,21.1,370,370,7565],[7567,3.3,370,370,7565],[5635,18.5,370,370,7565],[7568,33.9,370,370,7565]]},{"id":"indonutri-em-2026-paper","name":"IndoNutriQA (exact match, 2026 paper)","group":"医疗","metric":"exact match","unit":"%","higher_is_better":true,"public_history":true,"protocol":{"en":"HTML Table 2 exact match on the 137-item test split. Zero-shot and five-shot are configurations of the same checkpoint. Fine-tuned rows keep the printed setting. The retrieval pipeline is not a checkpoint and is omitted. Printed zeros stay.","zh":"HTML 表 2 在 137 题测试集上的精确匹配。零样本和五样本是同一检查点的配置。微调行保留表上的设置。检索流程不是检查点，不录入。表上的 0 保留。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.31256","status":"paper_baselines","published_at":"2026-10-22"},"results":[[7569,0.1241,370,7258,7570],[7571,0.0584,370,7258,7570],[7572,0.0438,370,7258,7570],[7573,0.0146,370,7258,7570],[7574,0.0584,370,7258,7570],[7575,0.0,370,7258,7570],[354,0.0,370,6394,7570],[354,0.0146,370,7576,7570],[353,0.0146,370,6394,7570],[353,0.073,370,7576,7570],[562,0.0073,370,6394,7570],[562,0.1679,370,7576,7570],[1748,0.0,370,6394,7570],[1748,0.0219,370,7576,7570],[7577,0.0,370,6394,7570],[7577,0.0584,370,7576,7570],[7578,0.0,370,6394,7570],[7578,0.0365,370,7576,7570],[7579,0.0292,370,6394,7570],[7579,0.0803,370,7576,7570],[7580,0.0292,370,6394,7570],[7580,0.146,370,7576,7570]]},{"id":"indonutri-f1-2026-paper","name":"IndoNutriQA (token F1, 2026 paper)","group":"医疗","metric":"F1","unit":"score","score_max":100,"score_decimals":2,"higher_is_better":true,"saturation_excluded":true,"scale_label":{"en":"F1 / 100","zh":"F1 / 100"},"public_history":true,"protocol":{"en":"HTML Table 2 token F1 on the same 137-item split. Shot count is a configuration. F1 is not exact-match accuracy and is excluded from accuracy saturation. The retrieval pipeline is omitted.","zh":"HTML 表 2 在同一 137 题上的词元 F1。样本数是配置。F1 不是精确匹配准确率，不进入准确率饱和。检索流程不录入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.31256","status":"paper_baselines","published_at":"2026-10-22"},"results":[[7569,36.34,370,7258,7570],[7571,23.78,370,7258,7570],[7572,16.56,370,7258,7570],[7573,12.6,370,7258,7570],[7574,19.35,370,7258,7570],[7575,7.84,370,7258,7570],[354,36.06,370,6394,7570],[354,43.07,370,7576,7570],[353,37.49,370,6394,7570],[353,50.33,370,7576,7570],[562,37.82,370,6394,7570],[562,55.55,370,7576,7570],[1748,31.03,370,6394,7570],[1748,42.62,370,7576,7570],[7577,26.78,370,6394,7570],[7577,43.32,370,7576,7570],[7578,32.44,370,6394,7570],[7578,41.83,370,7576,7570],[7579,29.83,370,6394,7570],[7579,40.08,370,7576,7570],[7580,42.6,370,6394,7570],[7580,54.28,370,7576,7570]]},{"id":"valcon-acc-all-2026-paper","name":"VALCON (VAB-Challenge accuracy, 2026 paper)","group":"安全与可靠性","metric":"accuracy","unit":"%","higher_is_better":true,"public_history":true,"protocol":{"en":"HTML Table 5 Acc All, accuracy on every VAB-Challenge item. Compliant-only and violating-only accuracies are separate columns and are not entered. Names are the table’s short forms.","zh":"HTML 表 5 的 Acc All，即 VAB-Challenge 全部题目的准确率。只含合规样本和只含违规样本的准确率是另外的列，不录入。名称用表上的简写。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.21304","status":"paper_baselines","published_at":"2026-10-15"},"results":[[3225,0.9634,370,370,7581],[340,0.9606,370,370,7581],[3009,0.9606,370,370,7581],[7582,0.9506999999999999,370,370,7581],[7583,0.9423,370,370,7581],[6301,0.9408,370,370,7581],[7584,0.9394,370,370,7581],[3008,0.9239,370,370,7581],[7585,0.9211,370,370,7581],[3016,0.9211,370,370,7581],[7586,0.9183,370,370,7581],[7587,0.9127,370,370,7581],[3027,0.8915000000000001,370,370,7581],[7588,0.8901,370,370,7581],[786,0.8801000000000001,370,370,7581],[3002,0.8614,370,370,7581],[7589,0.8606,370,370,7581],[5344,0.8549,370,370,7581],[6221,0.8183,370,370,7581],[572,0.8138,370,370,7581],[766,0.8028,370,370,7581],[7590,0.7014,370,370,7581],[7591,0.6695,370,370,7581],[5660,0.4899,370,370,7581]]},{"id":"libero-pro2-avg-2026-paper","name":"LIBERO-Pro2.0 (shift-averaged success, 2026 paper)","group":"Agent 与工具","metric":"success","unit":"%","higher_is_better":true,"public_history":true,"protocol":{"en":"HTML Table 1 Avg. only. It is the authors’ mean success over valid suite–perturbation entries. Nominal Base is the unperturbed suite and is not entered. Delta is Avg minus Base and is not entered. Each row is a named policy checkpoint. Printed zeros in other columns are not copied here. The new protocol was first posted on 17 October 2026. Base remains the nominal score on the older unperturbed LIBERO suites.","zh":"只取 HTML 表 1 的 Avg.。这是作者对有效的套件加扰动条目的平均成功率。未扰动的 Base 不录入。Delta 是 Avg 减 Base，不录入。每一行是具名策略检查点。其他列里印出的 0 不抄到这里。 新协议首次发布于 2026-10-17。Base 仍是更早的未扰动 LIBERO 套件上的名义分数。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.24587","status":"paper_baselines","published_at":"2026-10-17"},"results":[[7592,0.33899999999999997,370,370,7593],[7594,0.589,370,370,7593],[7595,0.5379999999999999,370,370,7593],[7596,0.593,370,370,7593],[7597,0.5920000000000001,370,370,7593],[7598,0.529,370,370,7593],[7599,0.386,370,370,7593],[7600,0.532,370,370,7593],[5820,0.667,370,370,7593],[7401,0.612,370,370,7593],[7601,0.6,370,370,7593],[7397,0.7120000000000001,370,370,7593],[7602,0.708,370,370,7593],[7603,0.5589999999999999,370,370,7593],[7404,0.687,370,370,7593],[7604,0.5589999999999999,370,370,7593],[7605,0.5479999999999999,370,370,7593]]},{"id":"infuse-utility-h-2026-paper","name":"INFUSE (executable-test utility, 2026 paper)","group":"安全与可靠性","metric":"utility","unit":"%","higher_is_better":true,"saturation_excluded":true,"scale_label":{"en":"Percent","zh":"百分比"},"public_history":true,"protocol":{"en":"HTML Table 3 H-Vanilla. The protocol runs every measured model on all 40 tasks in the no-attack setting, so these utility percentages share that sample. N counts undefended attack trajectories and is not this denominator; it differs because attacks run only on compatible surfaces. Attack-success rates are therefore not entered. Claude-3-Haiku has no vanilla run and is omitted. No denominator is rescaled. Utility is not attack success and is excluded from accuracy saturation.","zh":"HTML 表 3 的 H-Vanilla。协议让每个有测量的模型都在全部 40 题的无攻击设置上运行，所以这些效用百分比是同一样本。N 是无防御攻击轨迹数，不是这个分母；它不同是因为攻击只跑在兼容的界面上。攻击成功率因此不录入。Claude-3-Haiku 没有普通运行，不补。分母不改。效用不是攻击成功率，不进入准确率饱和。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.20776","status":"paper_baselines","published_at":"2026-10-14"},"results":[[1804,0.1,370,370,7606],[562,0.225,370,370,7606],[5535,0.25,370,370,7606],[7607,0.275,370,370,7606],[7608,0.3,370,370,7606],[1659,0.275,370,370,7606],[1645,0.325,370,370,7606],[7609,0.15,370,370,7606],[354,0.375,370,370,7606],[7610,0.212,370,370,7606],[2058,0.625,370,370,7606],[3184,0.625,370,370,7606],[6214,0.525,370,370,7606],[6978,0.525,370,370,7606],[766,0.45,370,370,7606],[6708,0.3,370,370,7606],[7611,0.325,370,370,7606],[7612,0.575,370,370,7606]]},{"id":"create-exact-2026-paper","name":"CREATE (exact match, 2026 paper)","group":"视觉与视频","metric":"exact match","unit":"%","higher_is_better":true,"public_history":true,"protocol":{"en":"HTML Table 5 exact-match accuracy on all 155 puzzles. The five prompting settings are configurations. Printed zeros stay. Recall and lexical overlap are not entered.","zh":"HTML 表 5 在全部 155 道题上的精确匹配准确率。五种提示是配置。表上的 0 保留。召回率和词面重叠不录入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.18569","status":"paper_baselines","published_at":"2026-10-13"},"results":[[5616,0.039,370,5574,7613],[5616,0.032,370,7614,7613],[5616,0.039,370,7615,7613],[5616,0.135,370,7616,7613],[5616,0.47700000000000004,370,7617,7613],[4979,0.052000000000000005,370,5574,7613],[4979,0.161,370,7614,7613],[4979,0.213,370,7615,7613],[4979,0.135,370,7616,7613],[4979,0.632,370,7617,7613],[6214,0.039,370,5574,7613],[6214,0.013000000000000001,370,7614,7613],[6214,0.0,370,7615,7613],[6214,0.174,370,7616,7613],[6214,0.51,370,7617,7613],[5275,0.226,370,5574,7613],[5275,0.406,370,7614,7613],[5275,0.342,370,7615,7613],[5275,0.368,370,7616,7613],[5275,0.8190000000000001,370,7617,7613],[5729,0.039,370,5574,7613],[5729,0.071,370,7614,7613],[5729,0.026000000000000002,370,7615,7613],[5729,0.12300000000000001,370,7616,7613],[5729,0.555,370,7617,7613],[5725,0.045,370,5574,7613],[5725,0.019,370,7614,7613],[5725,0.039,370,7615,7613],[5725,0.0,370,7616,7613],[5725,0.284,370,7617,7613],[4070,0.006,370,5574,7613],[4070,0.013000000000000001,370,7614,7613],[4070,0.006,370,7615,7613],[4070,0.026000000000000002,370,7616,7613],[4070,0.271,370,7617,7613],[5300,0.045,370,5574,7613],[5300,0.039,370,7614,7613],[5300,0.013000000000000001,370,7615,7613],[5300,0.135,370,7616,7613],[5300,0.381,370,7617,7613],[6240,0.09699999999999999,370,5574,7613],[6240,0.032,370,7614,7613],[6240,0.019,370,7615,7613],[6240,0.039,370,7616,7613],[6240,0.4,370,7617,7613],[2654,0.013000000000000001,370,5574,7613],[2654,0.019,370,7614,7613],[2654,0.0,370,7615,7613],[2654,0.006,370,7616,7613],[2654,0.213,370,7617,7613],[1702,0.006,370,5574,7613],[1702,0.013000000000000001,370,7614,7613],[1702,0.006,370,7615,7613],[1702,0.0,370,7616,7613],[1702,0.342,370,7617,7613]]},{"id":"agentci-bsr-2026-paper","name":"AgentCIScope (benign success, 2026 paper)","group":"安全与可靠性","metric":"success","unit":"%","higher_is_better":true,"public_history":true,"protocol":{"en":"HTML Table 2 benign task success on the first run of 936 tasks. The agent framework is a configuration. Leakage and the five-run union are separate and are not folded into this rate.","zh":"HTML 表 2 在 936 个任务第一次运行上的良性任务成功率。智能体框架是配置。泄漏率和五次运行的并集不并进这个成功率。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.19158","status":"paper_baselines","published_at":"2026-10-13"},"results":[[755,0.971,370,3759,7618],[3568,0.938,370,3759,7618],[2958,0.9540000000000001,370,7534,7618],[755,0.9670000000000001,370,7534,7618],[2977,0.982,370,7534,7618],[2958,0.9690000000000001,370,7619,7618],[2974,0.955,370,7619,7618],[3412,0.9540000000000001,370,7620,7618],[2974,0.955,370,7620,7618],[2977,0.982,370,7620,7618],[3386,0.993,370,7620,7618],[2983,0.99,370,7620,7618]]},{"id":"agentci-tlr-2026-paper","name":"AgentCIScope (single-run leakage, 2026 paper)","group":"安全与可靠性","metric":"leakage","unit":"%","higher_is_better":false,"public_history":true,"protocol":{"en":"HTML Table 2 single-run task leakage. A higher rate is worse, and it is not task accuracy. The framework is a configuration. The five-run union and its difference from the single run are not entered.","zh":"HTML 表 2 的单次任务泄漏率。越高越差，而且不是任务准确率。框架是配置。五次运行的并集以及它和单次的差不录入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.19158","status":"paper_baselines","published_at":"2026-10-13"},"results":[[755,0.325,370,7621,7618],[3568,0.494,370,7621,7618],[2958,0.369,370,7622,7618],[755,0.436,370,7622,7618],[2977,0.884,370,7622,7618],[2958,0.45,370,7623,7618],[2974,0.8809999999999999,370,7623,7618],[3412,0.7929999999999999,370,7624,7618],[2974,0.873,370,7624,7618],[2977,0.925,370,7624,7618],[3386,0.937,370,7624,7618],[2983,0.9440000000000001,370,7624,7618]]},{"id":"listqa-f1-all-2026-paper","name":"ListQA (full-set F1, 2026 paper)","group":"知识与推理","metric":"F1","unit":"score","score_max":100,"score_decimals":1,"higher_is_better":true,"saturation_excluded":true,"scale_label":{"en":"F1 / 100","zh":"F1 / 100"},"public_history":true,"protocol":{"en":"HTML Table 2 LLM-Judge-F1 on the full ListQA set under standard non-thinking inference. Simple and complex subsets and the format score are not entered, and this is not their average. Thinking runs are a separate table. F1 is excluded from accuracy saturation.","zh":"HTML 表 2 在标准非思考推理下、全部 ListQA 上的 LLM-Judge-F1。简单子集、复杂子集和格式分不录入，也不是它们的平均。思考运行是另一张表。F1 不进入准确率饱和。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.19165","status":"paper_baselines","published_at":"2026-10-13"},"results":[[3184,36.5,370,370,7625],[5449,27.0,370,370,7625],[7626,14.5,370,370,7625],[3578,35.0,370,370,7625],[2261,34.6,370,370,7625],[3738,33.9,370,370,7625],[3572,28.0,370,370,7625],[283,25.3,370,370,7625],[325,11.1,370,370,7625],[2055,33.7,370,370,7625],[3708,20.4,370,370,7625],[7627,31.6,370,370,7625],[7628,22.0,370,370,7625],[7629,12.0,370,370,7625],[7176,29.3,370,370,7625],[4425,20.3,370,370,7625],[7630,24.8,370,370,7625],[7631,18.9,370,370,7625],[7632,17.4,370,370,7625],[3751,14.8,370,370,7625],[7633,13.4,370,370,7625],[5672,8.3,370,370,7625],[5674,5.1,370,370,7625],[7634,25.7,370,370,7625],[7635,15.8,370,370,7625],[7636,7.2,370,370,7625],[7637,4.8,370,370,7625],[7638,3.3,370,370,7625],[2585,16.0,370,370,7625],[5934,13.8,370,370,7625],[2627,6.2,370,370,7625]]},{"id":"physclosure-f-2026-paper","name":"PhysClosureBench (functional completion, 2026 paper)","group":"数学与科学","metric":"completion","unit":"%","higher_is_better":true,"saturation_excluded":true,"scale_label":{"en":"Percent","zh":"百分比"},"public_history":true,"protocol":{"en":"HTML Table 6 column F. Each of the seven models has the same 150 final scored attempts: 50 cases times three runs. A terminal-zero attempt is a finished run with no scientifically assessable submission, and the authors score that attempt as zero inside the mean. Infrastructure-invalid executions are not added as extra zeros. Column V uses a different applicable count and is not entered. F is a mean completion rate, not task accuracy, and is excluded from accuracy saturation.","zh":"HTML 表 6 的 F 列。七个模型都是同样的 150 次最终计分尝试：50 个案例各三次。终态为零是一次已经跑完、但没有可评估提交的尝试，作者把这次尝试在均值里记为 0。基础设施无效的运行不另记成 0。V 列的适用条目数不同，不录入。F 是平均完成率，不是任务准确率，不进入准确率饱和。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.21767","status":"paper_baselines","published_at":"2026-10-15"},"results":[[755,0.856,370,370,7639],[762,0.838,370,370,7639],[2977,0.774,370,370,7639],[3153,0.672,370,370,7639],[3157,0.623,370,370,7639],[3396,0.5670000000000001,370,370,7639],[3412,0.619,370,370,7639]]},{"id":"jockey-top1-2026-paper","name":"JockeyBench (top-1, 2026 paper)","group":"预测与决策","metric":"top-1","unit":"%","higher_is_better":true,"public_history":true,"protocol":{"en":"HTML Table 2 Top-1 percent on the daily-revision program, first posted 14 October 2026. Log gain, cost and the frozen program are not entered. The closing market is a reference, not a model.","zh":"HTML 表 2 在每日修订方案上的 Top-1 百分比，首次发布于 2026-10-14。对数收益、费用和冻结方案不录入。收盘市场是参照，不是模型。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.19589","status":"paper_baselines","published_at":"2026-10-14"},"results":[[5618,0.285,370,370,7640],[4977,0.3,370,370,7640],[755,0.282,370,370,7640],[4979,0.278,370,370,7640],[6883,0.273,370,370,7640],[2956,0.27699999999999997,370,370,7640],[2959,0.275,370,370,7640],[3154,0.24600000000000002,370,370,7640],[7641,0.24600000000000002,370,370,7640],[7642,0.23199999999999998,370,370,7640],[3155,0.249,370,370,7640],[5335,0.24600000000000002,370,370,7640],[3153,0.23800000000000002,370,370,7640],[3396,0.23399999999999999,370,370,7640],[3185,0.207,370,370,7640],[3201,0.20600000000000002,370,370,7640],[5616,0.228,370,370,7640],[7643,0.244,370,370,7640],[7644,0.17300000000000001,370,370,7640],[5617,0.18600000000000003,370,370,7640],[5727,0.166,370,370,7640],[5635,0.17,370,370,7640],[5650,0.071,370,370,7640],[7645,0.073,370,370,7640],[7646,0.149,370,370,7640],[7647,0.11,370,370,7640],[2590,0.124,370,370,7640],[4070,0.132,370,370,7640]]},{"id":"apex-static-prompt-score-2026-paper","name":"APEX-Harm (static prompt task score, 2026 paper)","group":"安全与可靠性","metric":"task score","unit":"score","score_max":1,"score_decimals":3,"higher_is_better":true,"saturation_excluded":true,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"public_history":true,"protocol":{"en":"HTML Table 3 task score for static prompt injection only, n=30 for every model. First posted 9 October 2026. Attack success and the other categories are not entered, and they are not averaged together. The Average row is not a model. The score is a mean task score, not accuracy, and is excluded from accuracy saturation.","zh":"HTML 表 3 只取静态提示注入的任务分，每个模型都是 30 题。首次发布于 2026-10-09。攻击成功率和其他类别不录入，也不混成平均。均值行不是模型。这是任务均分，不是准确率，不进入准确率饱和。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.13623","status":"paper_baselines","published_at":"2026-10-09"},"results":[[3156,0.841,370,370,7648],[3393,0.788,370,370,7648],[2958,0.821,370,370,7648],[2983,0.871,370,370,7648],[3408,0.41,370,370,7648],[3386,0.858,370,370,7648],[2977,0.776,370,370,7648],[3261,0.801,370,370,7648],[762,0.82,370,370,7648],[3529,0.853,370,370,7648],[2973,0.774,370,370,7648],[3405,0.748,370,370,7648]]},{"id":"mtedit-turn1-if-2026-paper","name":"MTEdit-Bench (turn-1 instruction following, 2026 paper)","group":"视觉与视频","metric":"instruction following","unit":"score","score_max":10,"score_decimals":2,"higher_is_better":true,"saturation_excluded":true,"scale_label":{"en":"Score / 10","zh":"分数 / 10"},"public_history":true,"protocol":{"en":"HTML Table 2 turn-1 instruction-following score on the step-wise sub-bench. Later turns and the other sub-bench block are not entered. The unnamed Ours row is a method and is omitted. The score is a 0–10 rating, not accuracy, and is excluded from accuracy saturation. First posted 16 October 2026.","zh":"HTML 表 2 逐步子榜上第一轮的指令遵循分。后面的轮次和另一块子榜不录入。没有名称的 Ours 行是方法，不录入。这是 0–10 评分，不是准确率，不进入准确率饱和。首次发布于 2026-10-16。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.23751","status":"paper_baselines","published_at":"2026-10-16"},"results":[[6366,9.75,370,370,7649],[6839,9.65,370,370,7649],[7290,9.02,370,370,7649],[7650,9.73,370,370,7649],[7651,7.24,370,370,7649],[7295,6.75,370,370,7649],[7652,8.29,370,370,7649],[7653,9.09,370,370,7649],[6361,9.3,370,370,7649],[6358,8.69,370,370,7649],[6362,7.4,370,370,7649],[6243,9.29,370,370,7649],[7654,8.2,370,370,7649],[7655,8.89,370,370,7649]]},{"id":"actbench-benign-utility-2026-paper","name":"ActBench (benign utility) (2026 paper)","group":"安全与可靠性","metric":"UGS benign","unit":"score","public_history":true,"protocol":{"en":"Mean criterion-based utility score on 300 benign cases; OpenClaw 2026.5.19, fixed tools, reset state and graders across 15 models. Native 0–1 utility, excluded from accuracy saturation.","zh":"300 个正常任务的准则式效用均分；15 个模型使用固定 OpenClaw 2026.5.19、工具、重置状态与评分器。保留 0–1 原始效用分。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.09476","status":"paper_baselines","published_at":"2026-08-10"},"score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":15,"result_count":15,"results":[[3194,0.938,370,7656,7657],[6978,0.927,370,7656,7657],[3196,0.928,370,7656,7657],[4958,0.904,370,7656,7657],[4986,0.938,370,7656,7657],[227,0.929,370,7656,7657],[7658,0.915,370,7656,7657],[7659,0.915,370,7656,7657],[3155,0.94,370,7656,7657],[3187,0.869,370,7656,7657],[3198,0.917,370,7656,7657],[3190,0.88,370,7656,7657],[7660,0.922,370,7656,7657],[7661,0.9,370,7656,7657],[7662,0.933,370,7656,7657]]},{"id":"algoworlds-optimality-2026-paper","name":"AlgoWorlds (exact optimality) (2026 paper)","group":"Agent 与工具","metric":"Exact optimality","unit":"%","public_history":true,"protocol":{"en":"Published mean exact optimality across 240 tool-mediated environments in ten optimization families, averaged across repeated trials. Appendix H requires 240 finalized records per model/trial; provider and network attempts are separate from trajectory outcomes. Same main evaluation panel; feasibility and reference utility remain separate metrics.","zh":"10 类优化问题、240 个工具交互环境上的精确最优解比例，使用论文主实验多次运行的均值。附录 H 要求每个模型每轮有 240 条完成记录，提供商与网络请求尝试另行记录。可行性与参考效用是独立指标。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.29397","status":"paper_baselines","published_at":"2026-08-29"},"model_count":7,"result_count":7,"results":[[3753,0.0542,370,370,7663],[3492,0.1611,370,370,7663],[3654,0.2486,370,370,7663],[2973,0.2639,370,370,7663],[762,0.38189999999999996,370,370,7663],[3568,0.3236,370,370,7663],[3479,0.3861,370,370,7663]]},{"id":"apiflow-all-2026-paper","name":"APIFlow-Bench (2026 paper)","group":"Agent 与工具","metric":"All task success","unit":"%","public_history":true,"protocol":{"en":"Published All column for the same solo-and-chain task bank and frozen seven-tool interface. Typed delivery and world-state graders enforce completion. Panel spread is excluded.","zh":"采用同一单任务与任务链库、固定七工具接口的 All 成功率列；通过类型化交付检查与环境状态评分判断完成情况。排除 panel spread 汇总行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.29128","status":"paper_baselines","published_at":"2026-08-29"},"model_count":19,"result_count":19,"results":[[3208,0.878,370,370,7664],[3476,0.8859999999999999,370,370,7664],[1321,0.884,370,370,7664],[5154,0.871,370,370,7664],[5157,0.862,370,370,7664],[7665,0.86,370,370,7664],[3209,0.867,370,370,7664],[5103,0.8540000000000001,370,370,7664],[5153,0.873,370,370,7664],[4278,0.89,370,370,7664],[5152,0.8759999999999999,370,370,7664],[5107,0.878,370,370,7664],[5151,0.879,370,370,7664],[5158,0.855,370,370,7664],[4906,0.809,370,370,7664],[5156,0.8590000000000001,370,370,7664],[62,0.82,370,370,7664],[5159,0.8059999999999999,370,370,7664],[5160,0.685,370,370,7664]]},{"id":"avalon-tom-accuracy-2026-paper","name":"Avalon-ToM-Bench (2026 paper)","group":"知识与推理","metric":"Average accuracy","unit":"%","public_history":true,"protocol":{"en":"Published Average Acc column across four theory-of-mind tasks grounded in Avalon games. Rules is a separate rules-knowledge check; F1 is not used as accuracy. Aggregate Average row excluded.","zh":"采用 Avalon 游戏中四类心智推理任务的 Average Acc 列。Rules 为独立规则知识测试，F1 不作准确率录入；排除 Average 汇总行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.09638","status":"paper_baselines","published_at":"2026-08-10"},"model_count":28,"result_count":28,"results":[[764,0.8967,370,370,7666],[2058,0.885,370,370,7666],[766,0.8675,370,370,7666],[3764,0.8284999999999999,370,370,7666],[7667,0.8079999999999999,370,370,7666],[4064,0.8037000000000001,370,370,7666],[5344,0.7951,370,370,7666],[786,0.7848999999999999,370,370,7666],[6301,0.7725,370,370,7666],[354,0.7504000000000001,370,370,7666],[5126,0.74,370,370,7666],[4070,0.7382,370,370,7666],[1621,0.7361,370,370,7666],[6689,0.7151000000000001,370,370,7666],[7668,0.7105,370,370,7666],[1921,0.7091,370,370,7666],[588,0.6998000000000001,370,370,7666],[6677,0.6947,370,370,7666],[7669,0.6838,370,370,7666],[613,0.662,370,370,7666],[7345,0.6515000000000001,370,370,7666],[587,0.6306,370,370,7666],[7018,0.618,370,370,7666],[7590,0.6167,370,370,7666],[623,0.6158,370,370,7666],[5828,0.602,370,370,7666],[5660,0.5842,370,370,7666],[7670,0.5540999999999999,370,370,7666]]},{"id":"comboshopping-success-2026-paper","name":"ComboShoppingBench (2026 paper)","group":"Agent 与工具","metric":"Overall success","unit":"%","public_history":true,"protocol":{"en":"All 291 tasks under the same commerce simulation and hybrid semantic/rule-based checker. Think and No-think are configurations of eleven underlying models, not additional models.","zh":"同一购物模拟环境中全部 291 个任务，使用语义与规则混合检查。Think 与 No-think 是 11 个底层模型的推理配置，不另计为模型。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.09282","status":"paper_baselines","published_at":"2026-08-10"},"model_count":11,"result_count":22,"results":[[3196,0.612,370,7671,7672],[3196,0.14400000000000002,370,7673,7672],[227,0.529,370,7671,7672],[227,0.302,370,7673,7672],[502,0.502,370,7671,7672],[502,0.35700000000000004,370,7673,7672],[3183,0.498,370,7671,7672],[3183,0.113,370,7673,7672],[3194,0.495,370,7671,7672],[3194,0.213,370,7673,7672],[3187,0.44,370,7671,7672],[3187,0.141,370,7673,7672],[3185,0.344,370,7671,7672],[3185,0.040999999999999995,370,7673,7672],[6978,0.278,370,7671,7672],[6978,0.23399999999999999,370,7673,7672],[3198,0.254,370,7671,7672],[3198,0.062,370,7673,7672],[7674,0.175,370,7671,7672],[7674,0.031,370,7673,7672],[3763,0.052000000000000005,370,7671,7672],[3763,0.124,370,7673,7672]]},{"id":"diagram-mmu-qa-2026-paper","name":"Diagram-MMU (question answering) (2026 paper)","group":"视觉与视频","metric":"Accuracy","unit":"%","public_history":true,"protocol":{"en":"Published Avg accuracy across the six-domain diagram question-answering panel, without the agentic extension. Parsing and editing use different metrics and remain separate.","zh":"六领域图表问答面板的 Avg 准确率，采用基础推理设置。解析与编辑任务使用不同指标。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.12262","status":"paper_baselines","published_at":"2026-08-12"},"model_count":11,"result_count":11,"results":[[7675,0.8629000000000001,370,370,7676],[7677,0.8645999999999999,370,370,7676],[7678,0.8406999999999999,370,370,7676],[340,0.8167,370,370,7676],[7679,0.6875,370,370,7676],[7680,0.759,370,370,7676],[2576,0.8342,370,370,7676],[5843,0.636,370,370,7676],[2998,0.7974,370,370,7676],[6342,0.47119999999999995,370,370,7676],[2706,0.5639,370,370,7676]]},{"id":"dsagentbench-a11y-2026-paper","name":"DSAgentBench (screenshot + accessibility tree) (2026 paper)","group":"Agent 与工具","metric":"Task success","unit":"%","public_history":true,"protocol":{"en":"Overall task success on 275 real-computer data-science tasks with screenshots and accessibility trees; 15-step interaction budget. Open models have N/A in this setting and are omitted. Human performance is a reference, not a model.","zh":"275 个真实计算机数据科学任务，提供截图和无障碍树，采用 15 步交互预算。开源模型在该设置标为 N/A，不录入；人工成绩为参考。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.10366","status":"paper_baselines","published_at":"2026-08-11"},"model_count":9,"result_count":9,"results":[[353,0.24539999999999998,370,7681,7682],[7683,0.0255,370,7681,7682],[4064,0.19030000000000002,370,7681,7682],[2058,0.2981,370,7681,7682],[764,0.20809999999999998,370,7681,7682],[7684,0.0661,370,7681,7682],[4061,0.0464,370,7681,7682],[7685,0.09210000000000002,370,7681,7682],[7686,0.5670000000000001,370,7681,7682]]},{"id":"dseffi-pass-2026-paper","name":"DSEffi-Bench (correctness) (2026 paper)","group":"编程","metric":"Pass","unit":"%","public_history":true,"protocol":{"en":"Overall Pass on 1,000 data-science coding instances across more than ten libraries, using the same stress-test harness. Beyond and conditional B|P efficiency are separate metrics. Human Reference excluded.","zh":"跨十余种库的 1,000 道数据科学编程题，使用统一压力测试框架的 Overall Pass 列。Beyond 与条件效率 B|P 为独立指标；排除人工参考。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.30248","status":"paper_baselines","published_at":"2026-08-31"},"model_count":16,"result_count":16,"results":[[3184,0.669,370,370,7687],[4958,0.622,370,370,7687],[3183,0.613,370,370,7687],[6978,0.597,370,370,7687],[3201,0.5770000000000001,370,370,7687],[3009,0.561,370,370,7687],[2998,0.402,370,370,7687],[3005,0.428,370,370,7687],[7688,0.5760000000000001,370,370,7687],[5803,0.505,370,370,7687],[7689,0.43,370,370,7687],[5836,0.341,370,370,7687],[6465,0.32799999999999996,370,370,7687],[7690,0.27399999999999997,370,370,7687],[7691,0.175,370,370,7687],[7692,0.171,370,370,7687]]},{"id":"evogenui-episode-2026-paper","name":"EvoGenUI-Bench (five-turn success) (2026 paper)","group":"编程","metric":"TP@5","unit":"%","public_history":true,"protocol":{"en":"Five-turn episode success on 150 tasks (750 turns), with the same browser execution and screenshot/DOM/runtime evidence checks. Uses observed TP@5, not the independent-turn product.","zh":"150 个任务、750 轮交互上的五轮完整成功率，使用统一浏览器执行及截图、DOM、运行日志检查。采用实测 TP@5。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.29387","status":"paper_baselines","published_at":"2026-08-29"},"model_count":8,"result_count":8,"results":[[3196,0.213,370,370,7693],[3189,0.073,370,370,7693],[6133,0.047,370,370,7693],[3181,0.373,370,370,7693],[7694,0.02,370,370,7693],[787,0.013000000000000001,370,370,7693],[502,0.067,370,370,7693],[7695,0.133,370,370,7693]]},{"id":"finexam-context-complete-2026-paper","name":"FinExam-10K (context-complete) (2026 paper)","group":"专业工作","metric":"Accuracy","unit":"%","public_history":true,"protocol":{"en":"Table 14 All accuracy on the 7,625-item Context-Complete Reasoning Track, same frozen seventeen-model panel. This subset supplies the context needed to answer; the larger full-coverage track remains separate.","zh":"表 14 中 7,625 道上下文完整推理题的 All 准确率，采用固定的 17 模型面板。这一子集提供作答所需上下文。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.28155","status":"paper_baselines","published_at":"2026-08-28"},"model_count":17,"result_count":17,"results":[[502,0.9595999999999999,370,370,7696],[4979,0.9531999999999999,370,370,7696],[5727,0.9329000000000001,370,370,7696],[3185,0.9301,370,370,7696],[5616,0.9118999999999999,370,370,7696],[5789,0.9024,370,370,7696],[769,0.9064,370,370,7696],[3202,0.8477,370,370,7696],[3196,0.8602,370,370,7696],[353,0.7826000000000001,370,370,7696],[3419,0.8588,370,370,7696],[5343,0.794,370,370,7696],[7697,0.7764,370,370,7696],[7698,0.745,370,370,7696],[7699,0.7284999999999999,370,370,7696],[7700,0.5649000000000001,370,370,7696],[7701,0.5895,370,370,7696]]},{"id":"gui-primitives-full-2026-paper","name":"GUI-Primitives (2026 paper)","group":"视觉与视频","metric":"Strict point-in-box accuracy","unit":"%","public_history":true,"protocol":{"en":"Full 994-item panel using unconstrained coordinates and strict point-in-box scoring. Core and Clean are separate subsets. HF and quantized Ollama Gemma 27B rows are execution configurations of one model.","zh":"994 题完整面板，模型输出无约束坐标，按严格落框判定。Core 与 Clean 为独立子集；HF 与量化 Ollama 的 Gemma 27B 行属于同一模型的运行配置。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.21832v1","status":"paper_baselines","published_at":"2026-08-22"},"model_count":18,"result_count":19,"results":[[3578,0.313,370,370,7702],[2058,0.265,370,5367,7702],[325,0.239,370,370,7702],[3572,0.222,370,370,7702],[2471,0.22,370,370,7702],[4039,0.134,370,7703,7702],[4039,0.103,370,7704,7702],[7705,0.101,370,370,7702],[345,0.096,370,370,7702],[2728,0.095,370,370,7702],[7706,0.081,370,7707,7702],[4041,0.074,370,7703,7702],[3497,0.073,370,370,7702],[354,0.068,370,370,7702],[7708,0.041,370,370,7702],[7709,0.038,370,370,7702],[7710,0.029,370,370,7702],[4043,0.022,370,7703,7702],[7711,0.012,370,370,7702]]},{"id":"healthbench-psych-2026-paper","name":"HealthBench-Psych (2026 paper)","group":"医疗","metric":"Physician rubric score","unit":"score","public_history":true,"protocol":{"en":"610 mental-health conversations selected from HealthBench and validated in two blinded clinician-review rounds. Mean of the published cross-vendor three-judge scoring panel; native rubric score 0–1.","zh":"从 HealthBench 筛选并经过两轮临床盲审的 610 段心理健康对话，采用论文跨厂商三评委面板的均分，保留 0–1 评分。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.25071","status":"paper_baselines","published_at":"2026-08-25"},"score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":20,"result_count":20,"results":[[420,0.627,370,370,7712],[3208,0.624,370,370,7712],[5142,0.62,370,370,7712],[5105,0.612,370,370,7712],[4972,0.61,370,370,7712],[3473,0.591,370,370,7712],[5143,0.578,370,370,7712],[4952,0.568,370,370,7712],[3209,0.554,370,370,7712],[5149,0.552,370,370,7712],[7713,0.544,370,370,7712],[1321,0.538,370,370,7712],[5107,0.533,370,370,7712],[457,0.527,370,370,7712],[1500,0.512,370,370,7712],[4164,0.457,370,370,7712],[5148,0.446,370,370,7712],[5111,0.441,370,370,7712],[7714,0.363,370,370,7712],[5145,0.176,370,370,7712]]},{"id":"insufficiencybench-f2-2026-paper","name":"InsufficiencyBench (2026 paper)","group":"专业工作","metric":"F2","unit":"score","public_history":true,"protocol":{"en":"Missing-element identification F2 on the common 202-item legal-query set, using the main paper judging protocol. Native 0–1 F2 retains its recall emphasis and is excluded from accuracy saturation.","zh":"统一的 202 道法律提问上识别缺失要素的 F2，采用论文主实验评判规则。保留偏重召回的 0–1 F2，不作准确率。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.20220","status":"paper_baselines","published_at":"2026-08-20"},"score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":10,"result_count":10,"results":[[340,0.455,370,370,7715],[3196,0.437,370,370,7715],[3181,0.399,370,370,7715],[6099,0.383,370,370,7715],[5021,0.35,370,370,7715],[7716,0.346,370,370,7715],[6978,0.344,370,370,7715],[3187,0.336,370,370,7715],[502,0.298,370,370,7715],[3185,0.279,370,370,7715]]},{"id":"lexkairos-overall-2026-paper","name":"LexKairos (2026 paper)","group":"专业工作","metric":"Overall task score","unit":"score","public_history":true,"protocol":{"en":"Published Overall column across nine Chinese legal temporal subtasks. Thinking and CoT rows inherit their preceding backbone and remain configurations. The combined task score stays on the native 0–100 scale.","zh":"九类中文法律时间任务的 Overall 分数。Thinking 与 CoT 行沿用前一底层模型，作为推理配置；综合任务分保留原始 0–100 量纲。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.09106","status":"paper_baselines","published_at":"2026-08-10"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":8,"result_count":15,"results":[[3184,77.47,370,370,7717],[3184,82.75,370,7718,7717],[6133,76.71,370,370,7717],[6133,84.57,370,7718,7717],[769,79.78,370,370,7717],[3201,67.07,370,370,7717],[3201,79.94,370,7718,7717],[4070,55.45,370,370,7717],[4070,69.79,370,7718,7717],[7719,28.11,370,370,7717],[7719,31.55,370,7720,7717],[7721,35.33,370,370,7717],[7721,73.0,370,7718,7717],[7722,41.13,370,370,7717],[7722,38.17,370,7720,7717]]},{"id":"mcrbench-f1-2026-paper","name":"MCR-Bench (2026 paper)","group":"编程","metric":"Overall F1","unit":"score","public_history":true,"protocol":{"en":"Main model panel: Overall F1 across five programming languages in defect-state-aware multi-round code review. Precision and recall columns are not substituted; F1 stays on its native 0–1 scale.","zh":"主模型面板中五种编程语言的多轮缺陷评审 Overall F1。明确采用 F1 列，保留原始 0–1 量纲。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.27442","status":"paper_baselines","published_at":"2026-08-27"},"score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":7,"result_count":7,"results":[[325,0.551,370,370,7723],[4425,0.49,370,370,7723],[3424,0.532,370,370,7723],[2278,0.47,370,370,7723],[340,0.542,370,370,7723],[1485,0.373,370,370,7723],[3752,0.357,370,370,7723]]},{"id":"mrmad-degradation-type-2026-paper","name":"MRMAD (degradation type) (2026 paper)","group":"视觉与视频","metric":"DTI accuracy","unit":"%","public_history":true,"protocol":{"en":"Task 1 DTI Avg across speech, music and sound in the common multi-round audio-degradation protocol. Random Guess is excluded. Reasoning modes remain configurations of the same underlying audio model.","zh":"统一多轮音频劣化协议中，任务 1 在语音、音乐与声音上的 DTI Avg 准确率。排除随机猜测参考，推理模式作为底层音频模型的配置。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.22236v1","status":"paper_baselines","published_at":"2026-08-23"},"model_count":19,"result_count":22,"results":[[7724,0.2514,370,370,7725],[7726,0.26,370,370,7725],[7727,0.2736,370,370,7725],[7728,0.2631,370,370,7725],[7729,0.2594,370,370,7725],[7730,0.3358,370,370,7725],[7731,0.2564,370,370,7725],[7732,0.2944,370,370,7725],[7733,0.2192,370,370,7725],[7734,0.36579999999999996,370,370,7725],[7728,0.2753,370,3143,7725],[7729,0.23579999999999998,370,3143,7725],[7730,0.3247,370,3143,7725],[7382,0.2914,370,370,7725],[7735,0.2778,370,370,7725],[7736,0.2919,370,370,7725],[7737,0.3531,370,370,7725],[7738,0.28059999999999996,370,370,7725],[3424,0.48969999999999997,370,370,7725],[4219,0.3372,370,370,7725],[3412,0.8622,370,370,7725],[3415,0.6064,370,370,7725]]},{"id":"plsqlbench-single-turn-2026-paper","name":"PLSQLBench (single-turn test) (2026 paper)","group":"编程","metric":"Test Mean","unit":"%","public_history":true,"protocol":{"en":"Published single-turn Test Mean across MBPP+ and Spider2 single-turn test partitions, judged by executable PL/SQL tests. Codex agent variants retain the same underlying model identity. Multi-turn episode success is separate.","zh":"MBPP+ 与 Spider2 单轮测试集上的 Test Mean，通过执行 PL/SQL 测试判分。Codex Agent 行沿用底层模型身份，多轮会话成功率另列。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.15931","status":"paper_baselines","published_at":"2026-08-16"},"model_count":8,"result_count":10,"results":[[1583,0.3839,370,370,7739],[5480,0.5055,370,370,7739],[6708,0.40509999999999996,370,370,7739],[5411,0.4585,370,370,7739],[6235,0.5055,370,370,7739],[3184,0.5557,370,370,7739],[4979,0.5589999999999999,370,370,7739],[3194,0.5396,370,370,7739],[6235,0.546,370,7740,7739],[4979,0.6043,370,7740,7739]]},{"id":"finlifebench-event-anchor-2026-paper","name":"FinLifeBench · event history (2026 paper)","group":"专业工作","metric":"EA-F1","unit":"score","score_max":1,"higher_is_better":true,"public_history":true,"protocol":{"en":"Reconstructs life events and their first dialogue sessions from Korean banking conversations. Event–Anchor F1 is averaged across 400 checkpoints per model, over 20 trajectories.","zh":"根据韩语银行对话重建生活事件及其首次出现的会话。采用事件与会话位置的 F1 分数，每个模型覆盖20条轨迹、400个检查点。"},"source":{"name":"FinLifeBench authors","status":"paper_baselines","url":"https://arxiv.org/html/2609.01198v1#S6.T3","published_at":"2026-09-01"},"model_count":11,"result_count":11,"saturation_excluded":true,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"results":[[5004,0.679,370,7741,7742],[6591,0.663,370,7741,7742],[7311,0.629,370,7741,7742],[3479,0.534,370,7741,7742],[3572,0.72,370,7741,7742],[3412,0.748,370,7741,7742],[3415,0.488,370,7741,7742],[7743,0.641,370,7741,7742],[7744,0.473,370,7741,7742],[362,0.357,370,7741,7742],[6278,0.124,370,7741,7742]]},{"id":"finlifebench-state-2026-paper","name":"FinLifeBench · financial state (2026 paper)","group":"专业工作","metric":"GCA@15","unit":"%","score_max":100,"higher_is_better":true,"public_history":true,"protocol":{"en":"Reconstructs a complete 34-field financial state from cumulative Korean banking dialogue. Granular Change Accuracy scores updates between checkpoints 15 sessions apart, using pooled transition counts.","zh":"根据累积的韩语银行对话重建34项财务状态。细粒度变化准确率评估相隔15次会话的状态更新，按全部状态转换计数。"},"source":{"name":"FinLifeBench authors","status":"paper_baselines","url":"https://arxiv.org/html/2609.01198v1#S6.T3","published_at":"2026-09-01"},"model_count":11,"result_count":11,"results":[[5004,0.413,370,7741,7742],[6591,0.411,370,7741,7742],[7311,0.461,370,7741,7742],[3479,0.47,370,7741,7742],[3572,0.379,370,7741,7742],[3412,0.438,370,7741,7742],[3415,0.412,370,7741,7742],[7743,0.452,370,7741,7742],[7744,0.435,370,7741,7742],[362,0.321,370,7741,7742],[6278,0.249,370,7741,7742]]},{"id":"scilaws-fit-2026-paper","name":"SciLaws · numeric fit (2026 paper)","group":"数学与科学","metric":"All","unit":"score","score_max":100,"higher_is_better":true,"public_history":true,"protocol":{"en":"Scores formulas inferred from scientific datasets against held-out observations and published reference formulas. The normalized fit score is averaged over the full real-data task set.","zh":"根据科学数据推断公式，再用留出数据和已发表公式检验。采用真实数据任务全集上的归一化拟合分数。"},"source":{"name":"SCILAWS-BENCH authors","status":"paper_baselines","url":"https://arxiv.org/html/2609.01552v1#S3.T2","published_at":"2026-09-01"},"model_count":9,"result_count":9,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"results":[[3196,50.77,370,7745,7746],[3415,46.08,370,7745,7746],[227,45.84,370,7745,7746],[3479,44.94,370,7745,7746],[6664,43.96,370,7745,7746],[3202,43.8,370,7745,7746],[4064,40.69,370,7745,7746],[4958,37.17,370,7745,7746],[354,20.59,370,7745,7746]]},{"id":"scilaws-validity-2026-paper","name":"SciLaws · scientific validity (2026 paper)","group":"数学与科学","metric":"All","unit":"score","score_max":100,"higher_is_better":true,"public_history":true,"protocol":{"en":"Checks whether inferred scientific formulas satisfy task rubrics and an anti-hacking check. The native score averages the fraction of satisfied requirements over the full real-data task set.","zh":"检查推断出的科学公式是否满足任务要求及反作弊检查。采用真实数据任务全集上的要求满足比例平均分。"},"source":{"name":"SCILAWS-BENCH authors","status":"paper_baselines","url":"https://arxiv.org/html/2609.01552v1#S3.T2","published_at":"2026-09-01"},"model_count":9,"result_count":9,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"results":[[3196,81.84,370,7745,7746],[3415,74.63,370,7745,7746],[227,74.19,370,7745,7746],[3479,80.24,370,7745,7746],[6664,81.3,370,7745,7746],[3202,72.66,370,7745,7746],[4064,77.03,370,7745,7746],[4958,72.45,370,7745,7746],[354,56.02,370,7745,7746]]},{"id":"scilaws-structure-2026-paper","name":"SciLaws · hidden-law structure (2026 paper)","group":"数学与科学","metric":"All","unit":"score","score_max":100,"higher_is_better":true,"public_history":true,"protocol":{"en":"Agents query simulators and infer hidden scientific laws. The judge scores structural recovery at five levels from 0 to 1; the table reports the full-task mean on a 0–100 scale.","zh":"智能体查询模拟器并推断隐藏科学定律。裁判按五档评估结构恢复程度；表中采用任务全集平均分，量纲为0—100。"},"source":{"name":"SCILAWS-BENCH authors","status":"paper_baselines","url":"https://arxiv.org/html/2609.01552v1#S3.T2","published_at":"2026-09-01"},"model_count":9,"result_count":9,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"results":[[3196,58.26,370,7745,7746],[3415,50.42,370,7745,7746],[227,52.12,370,7745,7746],[3479,50.0,370,7745,7746],[6664,49.79,370,7745,7746],[3202,50.42,370,7745,7746],[4064,44.7,370,7745,7746],[4958,42.58,370,7745,7746],[354,33.47,370,7745,7746]]},{"id":"rpcbench-critique-2026-paper","name":"RPCBench · premise critique (2026 paper)","group":"安全与可靠性","metric":"CPCC","unit":"score","score_max":1,"higher_is_better":true,"public_history":true,"protocol":{"en":"Evaluates detection, localization and handling of flawed premises in 4,623 recommendation requests. CPCC combines detection rate with conditional localization and strategy scores; final answers are judged by three models.","zh":"在4,623条推荐请求中评测发现、定位和处理错误前提的能力。CPCC综合检测率、定位分数与处理策略分数，由三个模型评判最终回答。"},"source":{"name":"RPCBench authors","status":"paper_baselines","url":"https://arxiv.org/html/2609.00918v1#S3.T2","published_at":"2026-09-01"},"model_count":11,"result_count":11,"saturation_excluded":true,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"results":[[3196,0.518,370,7747,7748],[7749,0.5092,370,7747,7748],[6237,0.5047,370,7747,7748],[3185,0.5166,370,7747,7748],[3201,0.4655,370,7747,7748],[6412,0.5261,370,7747,7748],[2576,0.5259,370,7747,7748],[2582,0.5245,370,7747,7748],[2590,0.4851,370,7747,7748],[524,0.1359,370,7747,7748],[562,0.1019,370,7747,7748]]},{"id":"fairlens-soundness-2026-paper","name":"FairLens · soundness (2026 paper)","group":"安全与可靠性","metric":"Sound.","unit":"%","score_max":100,"higher_is_better":true,"public_history":true,"protocol":{"en":"Tests whether visual models follow stated evidence and abstain from unsupported judgments about faces. Soundness is the multiple-choice answer accuracy averaged across hiring, legal and healthcare questions.","zh":"考查视觉模型能否依据给定信息作答，并对无法从人脸判断的问题弃答。采用招聘、法律、医疗三个领域选择题的平均正确率。"},"source":{"name":"FAIRLENS authors","status":"paper_baselines","url":"https://arxiv.org/html/2609.01691v1#S4.T2","published_at":"2026-09-01"},"model_count":8,"result_count":8,"results":[[2646,0.938,370,7750,7751],[2731,0.8809999999999999,370,7750,7751],[2754,0.857,370,7750,7751],[7752,0.7709999999999999,370,7750,7751],[6342,0.6579999999999999,370,7750,7751],[7753,0.623,370,7750,7751],[7754,0.35,370,7750,7751],[7755,0.21899999999999997,370,7750,7751]]},{"id":"sdare-detection-dyadic-2026-paper","name":"SDARE · dyadic stigma detection (2026 paper)","group":"安全与可靠性","metric":"Hierarchy-aware macro accuracy; dyadic","unit":"score","score_max":100,"higher_is_better":true,"public_history":true,"protocol":{"en":"Detects stigma and labels its source, stereotype, prejudice, discrimination and conversational role in dyadic conversations. The score is hierarchy-aware macro accuracy with label definitions supplied.","zh":"识别双人对话中的污名，并标注来源、刻板印象、偏见、歧视及对话角色。提供标签定义，采用考虑标签依赖关系的宏平均准确度。"},"source":{"name":"SDARE-Bench authors","status":"paper_baselines","url":"https://arxiv.org/html/2609.01548v1#A7.T7","published_at":"2026-09-01"},"model_count":8,"result_count":8,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"results":[[6292,65.61,370,7756,7757],[4966,55.93,370,7756,7757],[1649,62.48,370,7756,7757],[1810,44.33,370,7756,7757],[7758,58.74,370,7756,7757],[1621,57.85,370,7756,7757],[1651,60.78,370,7756,7757],[5344,55.1,370,7756,7757]]},{"id":"sdare-detection-group-2026-paper","name":"SDARE · group stigma detection (2026 paper)","group":"安全与可靠性","metric":"Hierarchy-aware macro accuracy; group","unit":"score","score_max":100,"higher_is_better":true,"public_history":true,"protocol":{"en":"Detects stigma and labels its source, stereotype, prejudice, discrimination and conversational role in group conversations. The score is hierarchy-aware macro accuracy with label definitions supplied.","zh":"识别多人对话中的污名，并标注来源、刻板印象、偏见、歧视及对话角色。提供标签定义，采用考虑标签依赖关系的宏平均准确度。"},"source":{"name":"SDARE-Bench authors","status":"paper_baselines","url":"https://arxiv.org/html/2609.01548v1#A7.T7","published_at":"2026-09-01"},"model_count":8,"result_count":8,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"results":[[6292,60.19,370,7756,7757],[4966,54.17,370,7756,7757],[1649,56.51,370,7756,7757],[1810,37.18,370,7756,7757],[7758,52.07,370,7756,7757],[1621,51.01,370,7756,7757],[1651,54.76,370,7756,7757],[5344,46.78,370,7756,7757]]},{"id":"chiaro-macro-f1-2026-paper","name":"CHIARO (2026 paper)","group":"知识与推理","metric":"Macro-","unit":"score","score_max":100,"higher_is_better":true,"public_history":true,"protocol":{"en":"Infers the emotions of two people reacting to the same event. Macro F1 covers the full 1,000-sentence human-adjudicated set using a joint prompt for both people.","zh":"推断两个人对同一事件的不同情绪。采用完整1,000句人工裁定数据集的宏平均F1，每次提示同时询问两人的情绪。"},"source":{"name":"CHIARO authors","status":"paper_baselines","url":"https://arxiv.org/html/2609.03394v1#S4.T4","published_at":"2026-09-03"},"model_count":7,"result_count":7,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"results":[[3196,67.3,370,7759,7760],[3603,66.9,370,7759,7760],[5795,66.5,370,7759,7760],[2584,66.3,370,7759,7760],[796,66.3,370,7759,7760],[3415,64.1,370,7759,7760],[4070,59.9,370,7759,7760]]},{"id":"statformbench-fine-2026-paper","name":"StatFormBench · fine classification (2026 paper)","group":"数学与科学","metric":"ACC FG","unit":"%","score_max":100,"higher_is_better":true,"public_history":true,"protocol":{"en":"Identifies the statistical problem implied by informal scenarios and data. Fine-grained accuracy covers 85 categories across all 1,013 benchmark samples under zero-shot prompting.","zh":"根据非正式场景描述和数据判断统计问题类型。采用85个细分类别的零样本分类准确率，覆盖全部1,013条样本。"},"source":{"name":"StatFormBench authors","status":"paper_baselines","url":"https://arxiv.org/html/2609.01982v1#S5.T1","published_at":"2026-09-02"},"model_count":14,"result_count":14,"results":[[2261,0.695,370,7761,7762],[3572,0.639,370,7761,7762],[3196,0.672,370,7761,7762],[3184,0.6579999999999999,370,7761,7762],[3412,0.72,370,7761,7762],[3424,0.684,370,7761,7762],[2998,0.648,370,7761,7762],[3185,0.591,370,7761,7762],[3201,0.648,370,7761,7762],[2576,0.6459999999999999,370,7761,7762],[4070,0.203,370,7761,7762],[4969,0.306,370,7761,7762],[5828,0.035,370,7761,7762],[6340,0.038,370,7761,7762]]},{"id":"procarena-postgresql-direct-2026-paper","name":"ProcArena · PostgreSQL Direct (2026 paper)","group":"编程","metric":"EX","unit":"%","score_max":100,"higher_is_better":true,"public_history":true,"protocol":{"en":"Generates or edits procedural SQL from natural-language requirements. Execution accuracy checks both returned outputs and final persistent database state on the eight paired subscenarios in PostgreSQL. Direct mode.","zh":"按自然语言要求生成或修改过程式SQL。在PostgreSQL的八类配对子场景中，以执行输出和最终数据库状态同时符合标准为成功。采用直接模式。"},"source":{"name":"ProcArena authors","status":"paper_baselines","url":"https://arxiv.org/html/2609.06527v1#S5.T6","published_at":"2026-09-06"},"model_count":7,"result_count":7,"results":[[3492,0.35200000000000004,370,7763,7764],[227,0.435,370,7763,7764],[3154,0.547,370,7763,7764],[3494,0.504,370,7763,7764],[340,0.5379999999999999,370,7763,7764],[3196,0.634,370,7763,7764],[7675,0.63,370,7763,7764]]},{"id":"procarena-postgresql-interactive-2026-paper","name":"ProcArena · PostgreSQL Interactive (2026 paper)","group":"编程","metric":"EX","unit":"%","score_max":100,"higher_is_better":true,"public_history":true,"protocol":{"en":"Generates or edits procedural SQL from natural-language requirements. Execution accuracy checks both returned outputs and final persistent database state on the eight paired subscenarios in PostgreSQL. Interactive mode.","zh":"按自然语言要求生成或修改过程式SQL。在PostgreSQL的八类配对子场景中，以执行输出和最终数据库状态同时符合标准为成功。采用交互模式。"},"source":{"name":"ProcArena authors","status":"paper_baselines","url":"https://arxiv.org/html/2609.06527v1#S5.T6","published_at":"2026-09-06"},"model_count":7,"result_count":7,"results":[[3492,0.17,370,7765,7764],[227,0.35200000000000004,370,7765,7764],[3154,0.505,370,7765,7764],[3494,0.39799999999999996,370,7765,7764],[340,0.517,370,7765,7764],[3196,0.556,370,7765,7764],[7675,0.564,370,7765,7764]]},{"id":"procarena-oracle-direct-2026-paper","name":"ProcArena · Oracle Direct (2026 paper)","group":"编程","metric":"EX","unit":"%","score_max":100,"higher_is_better":true,"public_history":true,"protocol":{"en":"Generates or edits procedural SQL from natural-language requirements. Execution accuracy checks both returned outputs and final persistent database state on the eight paired subscenarios in Oracle. Direct mode.","zh":"按自然语言要求生成或修改过程式SQL。在Oracle的八类配对子场景中，以执行输出和最终数据库状态同时符合标准为成功。采用直接模式。"},"source":{"name":"ProcArena authors","status":"paper_baselines","url":"https://arxiv.org/html/2609.06527v1#S5.T6","published_at":"2026-09-06"},"model_count":7,"result_count":7,"results":[[3492,0.38299999999999995,370,7766,7764],[227,0.43,370,7766,7764],[3154,0.478,370,7766,7764],[3494,0.529,370,7766,7764],[340,0.551,370,7766,7764],[3196,0.597,370,7766,7764],[7675,0.614,370,7766,7764]]},{"id":"procarena-oracle-interactive-2026-paper","name":"ProcArena · Oracle Interactive (2026 paper)","group":"编程","metric":"EX","unit":"%","score_max":100,"higher_is_better":true,"public_history":true,"protocol":{"en":"Generates or edits procedural SQL from natural-language requirements. Execution accuracy checks both returned outputs and final persistent database state on the eight paired subscenarios in Oracle. Interactive mode.","zh":"按自然语言要求生成或修改过程式SQL。在Oracle的八类配对子场景中，以执行输出和最终数据库状态同时符合标准为成功。采用交互模式。"},"source":{"name":"ProcArena authors","status":"paper_baselines","url":"https://arxiv.org/html/2609.06527v1#S5.T6","published_at":"2026-09-06"},"model_count":7,"result_count":7,"results":[[3492,0.305,370,7767,7764],[227,0.342,370,7767,7764],[3154,0.502,370,7767,7764],[3494,0.426,370,7767,7764],[340,0.47100000000000003,370,7767,7764],[3196,0.584,370,7767,7764],[7675,0.591,370,7767,7764]]},{"id":"patchbench-solved-2026-paper","name":"PatchBench (2026 paper)","group":"编程","metric":"Solved","unit":"%","score_max":100,"higher_is_better":true,"public_history":true,"protocol":{"en":"Repairs C/C++ vulnerabilities while preserving intended program behavior. A task is solved only when its patch passes both security and semantic validation. The paper compares coding-agent and security-agent configurations.","zh":"修复C/C++漏洞并保留程序预期行为。补丁同时通过安全与功能语义验证才算成功。论文比较多种编程智能体和安全智能体配置。"},"source":{"name":"PatchBench authors","status":"paper_baselines","url":"https://arxiv.org/html/2609.04075v1#S4.T2","published_at":"2026-09-03"},"model_count":7,"result_count":11,"results":[[762,0.5920000000000001,370,7768,7769],[762,0.5820000000000001,370,7770,7769],[3479,0.568,370,7771,7769],[7772,0.484,370,7773,7769],[3479,0.479,370,7770,7769],[3415,0.441,370,7770,7769],[762,0.42700000000000005,370,7774,7769],[2058,0.423,370,7770,7769],[283,0.38,370,7770,7769],[3412,0.315,370,7770,7769],[762,0.29600000000000004,370,7775,7769]]},{"id":"appsim-overall-2026-paper","name":"AppSim-Bench (2026 paper)","group":"Agent 与工具","metric":"Overall","unit":"%","score_max":100,"higher_is_better":true,"public_history":true,"protocol":{"en":"Completes tasks in simulated mobile apps. Overall accuracy is the fraction of tasks whose final app state, answer or combined goal satisfies the benchmark verifier.","zh":"在模拟手机应用中完成任务。总准确率按最终应用状态、回答或组合目标是否通过验证器计算。"},"source":{"name":"AppSim-Bench authors","status":"paper_baselines","url":"https://arxiv.org/html/2609.07712v1#S3.T2","published_at":"2026-09-07"},"model_count":19,"result_count":19,"results":[[3181,0.5027,370,7776,7777],[6130,0.4991,370,7776,7777],[3196,0.49729999999999996,370,7776,7777],[502,0.4955,370,7776,7777],[6216,0.3914,370,7778,7777],[2058,0.33390000000000003,370,7776,7777],[6291,0.3214,370,7778,7777],[3270,0.31420000000000003,370,7778,7777],[7779,0.2926,370,7778,7777],[3184,0.2908,370,7776,7777],[7780,0.28190000000000004,370,7778,7777],[6978,0.2729,370,7776,7777],[6214,0.228,370,7776,7777],[6704,0.2101,370,7776,7777],[764,0.1724,370,7776,7777],[7781,0.31420000000000003,370,7782,7777],[5487,0.2262,370,7783,7777],[7784,0.1005,370,7785,7777],[7786,0.0987,370,7787,7777]]},{"id":"saescientist-overall-2026-paper","name":"SAEScientist-Bench (2026 paper)","group":"数学与科学","metric":"Score","unit":"score","score_max":100,"higher_is_better":true,"public_history":true,"protocol":{"en":"Agents search sparse-autoencoder features in Gemma-2-9B-IT through experiments. The overall score equally averages activation rank, concept selectivity and causal steering over 20 tasks and three runs.","zh":"智能体通过实验在Gemma-2-9B-IT的稀疏自编码器中寻找特征。总分对激活排名、概念选择性、因果引导三项等权平均，覆盖20项任务，每项运行三次。"},"source":{"name":"SAEScientist-Bench authors","status":"paper_baselines","url":"https://arxiv.org/html/2609.09113v1#S4.T1","published_at":"2026-09-08"},"model_count":10,"result_count":10,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"results":[[2977,65.82,370,7788,7789],[755,65.41,370,7790,7789],[3568,65.04,370,7791,7789],[2974,62.93,370,7792,7789],[3479,62.57,370,7793,7789],[2983,59.57,370,7794,7789],[762,57.28,370,7795,7789],[3196,54.34,370,7796,7789],[227,52.27,370,7797,7789],[3408,50.27,370,7798,7789]]},{"id":"capquiz-f1-2026-paper","name":"CapQuiz (2026 paper)","group":"视觉与视频","metric":"F1","unit":"score","score_max":100,"higher_is_better":true,"public_history":true,"protocol":{"en":"Evaluates video captions through questions about their descriptive and inferred content. The overall CapF1 combines caption precision and recall across both question categories.","zh":"通过描述性和推断性问题评估视频字幕的信息。整体CapF1综合两类问题的字幕精确率与召回率。"},"source":{"name":"CapQuiz authors","status":"paper_baselines","url":"https://arxiv.org/html/2609.09973v1#S4.T3","published_at":"2026-09-09"},"model_count":15,"result_count":15,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"results":[[7480,81.01,370,7799,7800],[7801,81.71,370,7799,7800],[1580,71.97,370,7799,7800],[1553,76.79,370,7799,7800],[7482,83.08,370,7799,7800],[7802,39.86,370,7799,7800],[7803,63.16,370,7799,7800],[7804,42.37,370,7799,7800],[2654,45.69,370,7799,7800],[5534,54.46,370,7799,7800],[6341,61.53,370,7799,7800],[6298,70.15,370,7799,7800],[6342,70.72,370,7799,7800],[6238,68.82,370,7799,7800],[6343,77.95,370,7799,7800]]},{"id":"geovad-answer-2026-paper","name":"GeoVAD-Bench · answer accuracy (2026 paper)","group":"数学与科学","metric":"Answer Acc.","unit":"%","score_max":100,"higher_is_better":true,"public_history":true,"protocol":{"en":"Solves geometry problems with diagrams and optional auxiliary image construction. Final-answer accuracy uses rule matching, with a model verifier for answers the rules cannot resolve.","zh":"结合几何图及可选辅助作图求解问题。最终答案准确率先用规则匹配，规则无法判定时由模型验证。"},"source":{"name":"GeoVAD-Bench authors","status":"paper_baselines","url":"https://arxiv.org/html/2609.12606v1#S6.T7","published_at":"2026-09-11"},"model_count":12,"result_count":14,"results":[[2129,0.9309999999999999,370,7805,7806],[7807,0.9259999999999999,370,7808,7806],[3184,0.698,370,7809,7806],[7810,0.716,370,7811,7806],[3184,0.91,370,7805,7806],[7810,0.932,370,7808,7806],[6412,0.915,370,7805,7806],[6342,0.43,370,7809,7806],[7812,0.36,370,7811,7806],[7813,0.499,370,7814,7806],[7815,0.263,370,7816,7806],[7817,0.373,370,7816,7806],[7818,0.49,370,7816,7806],[7819,0.626,370,7816,7806]]},{"id":"biophys-bridge-evidence-2026-paper","name":"BioPhys-Bridge · evidence attribution (2026 paper)","group":"数学与科学","metric":"Evidence-ID F1","unit":"score","score_max":1,"higher_is_better":true,"public_history":true,"protocol":{"en":"Selects supporting evidence for scientific answers on the same 154 held-out tasks. The de-leaked prompt hides target equations and mechanisms; Evidence-ID F1 measures overlap with gold evidence identifiers.","zh":"在同一组154条留出任务中为科学回答选择证据。去泄漏提示隐藏目标方程和机制，以证据编号F1衡量预测与标准证据的重合程度。"},"source":{"name":"BioPhys-Bridge authors","status":"paper_baselines","url":"https://arxiv.org/html/2609.19180v1#S7.T7","published_at":"2026-09-15"},"model_count":7,"result_count":7,"saturation_excluded":true,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"results":[[7820,0.36,370,7821,7822],[3587,0.316,370,7821,7822],[354,0.294,370,7821,7822],[3479,0.238,370,7821,7822],[3568,0.224,370,7821,7822],[7823,0.111,370,7821,7822],[2602,0.01,370,7821,7822]]},{"id":"gameasg-strict-2026-paper","name":"GameASG-Bench (2026 paper)","group":"编程","metric":"Strict task success","unit":"%","score_max":100,"higher_is_better":true,"public_history":true,"protocol":{"en":"Builds browser games from specifications. Strict success requires all source checks and required gameplay checks to pass. All nine compared stacks delivered artifacts and completed evaluation on all 47 tasks.","zh":"根据规格开发浏览器游戏。全部源码检查与必需玩法检查通过才算成功。所列九种模型配置均交付作品并完成全部47项任务的评测。"},"source":{"name":"GameASG-Bench authors","status":"paper_baselines","url":"https://arxiv.org/html/2609.21293v1#S4.T3","published_at":"2026-09-18"},"model_count":9,"result_count":9,"results":[[4977,0.5529999999999999,370,7824,7825],[4983,0.511,370,7826,7825],[4979,0.447,370,7824,7825],[3201,0.38299999999999995,370,7826,7825],[3185,0.319,370,7826,7825],[3155,0.319,370,7826,7825],[227,0.23399999999999999,370,7826,7825],[7827,0.213,370,7826,7825],[3198,0.149,370,7826,7825]]},{"id":"petribench-easy-2026-paper","name":"PetriBench · Easy (2026 paper)","group":"知识与推理","metric":"Easy","unit":"%","score_max":100,"higher_is_better":true,"public_history":true,"protocol":{"en":"Reasons about dynamic state spaces represented by Petri nets. Exact-match accuracy is equally averaged across four reasoning categories at easy difficulty; each task has an exact verifier.","zh":"推理以Petri网表示的动态状态空间。采用简单难度下四类推理任务的等权精确匹配准确率，各题均可精确验证。"},"source":{"name":"PetriBench authors","status":"paper_baselines","url":"https://arxiv.org/html/2609.19883v1#S4.T1","published_at":"2026-09-17"},"model_count":21,"result_count":36,"results":[[755,0.927,370,7828,7829],[755,0.904,370,7830,7829],[755,0.843,370,7831,7829],[2974,0.953,370,7828,7829],[2974,0.951,370,7830,7829],[2974,0.9390000000000001,370,7831,7829],[762,0.9640000000000001,370,7828,7829],[762,0.929,370,7830,7829],[762,0.8270000000000001,370,7831,7829],[2983,0.968,370,7830,7829],[2983,0.866,370,7831,7829],[3391,0.882,370,7830,7829],[3391,0.838,370,7831,7829],[2973,0.85,370,7828,7829],[2973,0.675,370,7830,7829],[2973,0.539,370,7831,7829],[3412,0.9229999999999999,370,7830,7829],[3412,0.755,370,7831,7829],[3408,0.6890000000000001,370,7828,7829],[3408,0.539,370,7830,7829],[3408,0.38799999999999996,370,7831,7829],[3493,0.5,370,7831,7829],[3568,0.574,370,7830,7829],[3568,0.496,370,7831,7829],[325,0.469,370,7832,7829],[3261,0.578,370,7830,7829],[2585,0.384,370,7830,7829],[2585,0.27899999999999997,370,7831,7829],[2980,0.335,370,7831,7829],[268,0.34600000000000003,370,7832,7829],[4425,0.391,370,7832,7829],[3419,0.324,370,7831,7829],[244,0.32799999999999996,370,7832,7829],[784,0.28300000000000003,370,7832,7829],[3338,0.267,370,7832,7829],[7833,0.23600000000000002,370,7832,7829]]},{"id":"petribench-medium-2026-paper","name":"PetriBench · Medium (2026 paper)","group":"知识与推理","metric":"Medium","unit":"%","score_max":100,"higher_is_better":true,"public_history":true,"protocol":{"en":"Reasons about dynamic state spaces represented by Petri nets. Exact-match accuracy is equally averaged across four reasoning categories at medium difficulty; each task has an exact verifier.","zh":"推理以Petri网表示的动态状态空间。采用中等难度下四类推理任务的等权精确匹配准确率，各题均可精确验证。"},"source":{"name":"PetriBench authors","status":"paper_baselines","url":"https://arxiv.org/html/2609.19883v1#S4.T1","published_at":"2026-09-17"},"model_count":21,"result_count":36,"results":[[755,0.838,370,7828,7829],[755,0.8190000000000001,370,7830,7829],[755,0.7,370,7831,7829],[2974,0.809,370,7828,7829],[2974,0.765,370,7830,7829],[2974,0.757,370,7831,7829],[762,0.909,370,7828,7829],[762,0.8140000000000001,370,7830,7829],[762,0.525,370,7831,7829],[2983,0.745,370,7830,7829],[2983,0.5870000000000001,370,7831,7829],[3391,0.672,370,7830,7829],[3391,0.65,370,7831,7829],[2973,0.644,370,7828,7829],[2973,0.44299999999999995,370,7830,7829],[2973,0.307,370,7831,7829],[3412,0.6990000000000001,370,7830,7829],[3412,0.488,370,7831,7829],[3408,0.501,370,7828,7829],[3408,0.364,370,7830,7829],[3408,0.28300000000000003,370,7831,7829],[3493,0.336,370,7831,7829],[3568,0.335,370,7830,7829],[3568,0.306,370,7831,7829],[325,0.311,370,7832,7829],[3261,0.34,370,7830,7829],[2585,0.261,370,7830,7829],[2585,0.244,370,7831,7829],[2980,0.26,370,7831,7829],[268,0.263,370,7832,7829],[4425,0.235,370,7832,7829],[3419,0.256,370,7831,7829],[244,0.251,370,7832,7829],[784,0.237,370,7832,7829],[3338,0.245,370,7832,7829],[7833,0.228,370,7832,7829]]},{"id":"petribench-hard-2026-paper","name":"PetriBench · Hard (2026 paper)","group":"知识与推理","metric":"Hard","unit":"%","score_max":100,"higher_is_better":true,"public_history":true,"protocol":{"en":"Reasons about dynamic state spaces represented by Petri nets. Exact-match accuracy is equally averaged across four reasoning categories at hard difficulty; each task has an exact verifier.","zh":"推理以Petri网表示的动态状态空间。采用困难难度下四类推理任务的等权精确匹配准确率，各题均可精确验证。"},"source":{"name":"PetriBench authors","status":"paper_baselines","url":"https://arxiv.org/html/2609.19883v1#S4.T1","published_at":"2026-09-17"},"model_count":21,"result_count":36,"results":[[755,0.7170000000000001,370,7828,7829],[755,0.718,370,7830,7829],[755,0.581,370,7831,7829],[2974,0.565,370,7828,7829],[2974,0.55,370,7830,7829],[2974,0.526,370,7831,7829],[762,0.778,370,7828,7829],[762,0.598,370,7830,7829],[762,0.313,370,7831,7829],[2983,0.413,370,7830,7829],[2983,0.385,370,7831,7829],[3391,0.501,370,7830,7829],[3391,0.489,370,7831,7829],[2973,0.507,370,7828,7829],[2973,0.315,370,7830,7829],[2973,0.253,370,7831,7829],[3412,0.469,370,7830,7829],[3412,0.348,370,7831,7829],[3408,0.344,370,7828,7829],[3408,0.28,370,7830,7829],[3408,0.24600000000000002,370,7831,7829],[3493,0.266,370,7831,7829],[3568,0.271,370,7830,7829],[3568,0.255,370,7831,7829],[325,0.255,370,7832,7829],[3261,0.273,370,7830,7829],[2585,0.258,370,7830,7829],[2585,0.23800000000000002,370,7831,7829],[2980,0.249,370,7831,7829],[268,0.243,370,7832,7829],[4425,0.213,370,7832,7829],[3419,0.244,370,7831,7829],[244,0.244,370,7832,7829],[784,0.235,370,7832,7829],[3338,0.23800000000000002,370,7832,7829],[7833,0.226,370,7832,7829]]},{"id":"aptmql-soft-ex-2026-paper","name":"AptMQL-Bench · with knowledge (2026 paper)","group":"编程","metric":"Soft execution accuracy","unit":"%","public_history":true,"protocol":{"en":"3,186 text-to-MongoDB queries over 21 migrated databases. Soft-EX accepts extra fields only when rows and ground-truth field values match. Table 2, with external knowledge; eight models.","zh":"21 个文档数据库上的 3,186 条查询。Soft-EX 要求返回行数与真值字段一致，允许附加字段。表 2 提供外部知识的八模型结果。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.02770","status":"paper_baselines","published_at":"2026-10-02"},"model_count":8,"result_count":8,"results":[[542,0.37070000000000003,370,7834,7835],[5343,0.51,370,7834,7835],[7836,0.5606,370,7834,7835],[5480,0.6196,370,7834,7835],[3419,0.6246,370,7834,7835],[7837,0.6472,370,7834,7835],[7838,0.6786,370,7834,7835],[7839,0.7034,370,7834,7835]]},{"id":"egoexo-next-2026-paper","name":"EgoExo-Next (2026 paper)","group":"视觉与视频","metric":"Mean eight-setting accuracy","unit":"%","public_history":true,"protocol":{"en":"2,503 multiple-choice questions. Unweighted mean of eight subtask-by-scene settings. Table 1 Avg.; human reference and GPT-6, which lacks six settings, are omitted.","zh":"2,503 道选择题，取八个子任务与场景组合的准确率非加权平均。表 1 Avg. 列，排除人类参考和缺少六个设置成绩的 GPT-6。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.04506","status":"paper_baselines","published_at":"2026-10-03"},"model_count":12,"result_count":12,"results":[[340,0.4316,370,7834,7840],[3183,0.33590000000000003,370,7834,7840],[502,0.43810000000000004,370,7834,7840],[6297,0.4093,370,7834,7840],[6343,0.3703,370,7834,7840],[6342,0.3627,370,7834,7840],[2654,0.3293,370,7834,7840],[2642,0.36810000000000004,370,7834,7840],[6348,0.3185,370,7834,7840],[7841,0.3295,370,7834,7840],[7842,0.3558,370,7834,7840],[7843,0.3856,370,7834,7840]]},{"id":"delegationbench-agreement-2026-paper","name":"DelegationBench (2026 paper)","group":"Agent 与工具","metric":"Four-way agreement","unit":"%","public_history":true,"protocol":{"en":"149 scenarios with a human-majority four-way decision label, five trials per model. Table 1 per-trial agreement; malformed outputs count as disagreement. This is agreement with the annotated delegation judgment.","zh":"149 个有人类多数决策标签的情景，每模型五次试验。表 1 逐次四分类一致率；格式无效输出计为不一致，衡量与标注者授权判断的一致程度。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.05532","status":"paper_baselines","published_at":"2026-10-04"},"model_count":10,"result_count":10,"results":[[4035,0.7340000000000001,370,7834,7844],[3405,0.7190000000000001,370,7834,7844],[3568,0.687,370,7834,7844],[3479,0.655,370,7834,7844],[2973,0.604,370,7834,7844],[3408,0.6,370,7834,7844],[4995,0.5870000000000001,370,7834,7844],[762,0.581,370,7834,7844],[3419,0.545,370,7834,7844],[5343,0.506,370,7834,7844]]},{"id":"dsv-mem-total-2026-paper","name":"DSV-Mem (2026 paper)","group":"Agent 与工具","metric":"All-question accuracy","unit":"%","public_history":true,"protocol":{"en":"Baseline full-conversation and visible-image input, no external memory method or prompt ablation. Table 2 Total Avg. averages all questions and reports the mean of three independent runs; reasoning effort retained in each result.","zh":"基础模型读取完整对话和可见图像。取表 2 Total Avg. 全部问题准确率、三次独立运行均值，保留各模型推理档位；提示变体和外部记忆方法不混入基础组。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.08102","status":"paper_baselines","published_at":"2026-10-06"},"model_count":14,"result_count":14,"results":[[3386,0.396,370,7845,7846],[3405,0.23,370,7845,7846],[4035,0.08199999999999999,370,7845,7846],[3482,0.275,370,7845,7846],[3491,0.131,370,7847,7846],[7848,0.034,370,7847,7846],[2977,0.243,370,7849,7846],[244,0.057999999999999996,370,7847,7846],[2576,0.113,370,7847,7846],[2584,0.096,370,7847,7846],[4070,0.07200000000000001,370,7847,7846],[4969,0.033,370,7847,7846],[6110,0.085,370,7850,7846],[7851,0.039,370,7850,7846]]},{"id":"woven-id-2026-paper","name":"WOVEN · in-distribution (2026 paper)","group":"视觉与视频","metric":"ID accuracy","unit":"%","public_history":true,"protocol":{"en":"6,228 four-option questions on action-conditioned visual state transitions. Appendix Table 24, full 38-model in-distribution panel; held-out scenes and training ablations remain separate.","zh":"6,228 道动作条件下视觉状态变化的四选一题。附录表 24 的 38 模型分布内结果，独立于新场景测试和训练消融。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.12417","status":"paper_baselines","published_at":"2026-10-08"},"model_count":38,"result_count":38,"results":[[3184,0.6579999999999999,370,7834,7852],[7853,0.632,370,7834,7852],[2166,0.589,370,7834,7852],[5843,0.585,370,7834,7852],[340,0.58,370,7834,7852],[5480,0.578,370,7834,7852],[889,0.556,370,7834,7852],[2576,0.5539999999999999,370,7834,7852],[6625,0.552,370,7834,7852],[7854,0.5429999999999999,370,7834,7852],[7855,0.517,370,7834,7852],[7856,0.484,370,7834,7852],[6391,0.47700000000000004,370,7834,7852],[6689,0.47,370,7834,7852],[7857,0.45299999999999996,370,7834,7852],[7858,0.449,370,7834,7852],[5635,0.439,370,7834,7852],[6238,0.428,370,7834,7852],[7859,0.425,370,7834,7852],[2754,0.41600000000000004,370,7834,7852],[6668,0.415,370,7834,7852],[7860,0.40399999999999997,370,7834,7852],[2584,0.33299999999999996,370,7834,7852],[6150,0.331,370,7834,7852],[1702,0.326,370,7834,7852],[7861,0.319,370,7834,7852],[7862,0.315,370,7834,7852],[4070,0.301,370,7834,7852],[6677,0.289,370,7834,7852],[1583,0.287,370,7834,7852],[2582,0.26899999999999996,370,7834,7852],[2771,0.263,370,7834,7852],[7863,0.251,370,7834,7852],[7153,0.251,370,7834,7852],[7864,0.244,370,7834,7852],[6676,0.242,370,7834,7852],[5722,0.226,370,7834,7852],[2590,0.225,370,7834,7852]]},{"id":"flex-rsi-robodojo-2026-paper","name":"Flex-RSI · RoboDojo (2026 paper)","group":"Agent 与工具","metric":"Suite task score","unit":"score","public_history":true,"protocol":{"en":"Flex-RSI Table 13: 10 tasks, 5 layouts each. Final held-out-test Suite task score; same starter harness and task-specific budget. Models without a ranked completed result are omitted. Per-task scores stay separate from the unequal-coverage average-rank leaderboard.","zh":"Flex-RSI 表 13，10 tasks, 5 layouts each。取最终隐藏测试的 Suite task score；采用相同初始框架和任务预算。仅保留作者列入排名且有最终成绩的模型，按任务分列。"},"source":{"name":"Flex-RSI authors","url":"https://arxiv.org/abs/2611.08946","status":"paper_baselines","published_at":"2026-11-06"},"score_max":100,"score_decimals":1,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":8,"result_count":8,"results":[[2959,55.1,370,7865,7866],[2962,46.8,370,7865,7866],[2960,41.5,370,7865,7866],[2958,28.6,370,7865,7866],[7867,17.8,370,7868,7866],[2965,10.6,370,7865,7866],[6438,8.4,370,7865,7866],[3198,0.0,370,7865,7866]]},{"id":"flex-rsi-robocasa-2026-paper","name":"Flex-RSI · RoboCasa (2026 paper)","group":"Agent 与工具","metric":"Success rate","unit":"%","public_history":true,"protocol":{"en":"Flex-RSI Table 14: 8 tasks, 5 episodes each; ninth zero-success task excluded by publisher. Final held-out-test Success rate; same starter harness and task-specific budget. Models without a ranked completed result are omitted. Per-task scores stay separate from the unequal-coverage average-rank leaderboard.","zh":"Flex-RSI 表 14，8 tasks, 5 episodes each; ninth zero-success task excluded by publisher。取最终隐藏测试的 Success rate；采用相同初始框架和任务预算。仅保留作者列入排名且有最终成绩的模型，按任务分列。"},"source":{"name":"Flex-RSI authors","url":"https://arxiv.org/abs/2611.08946","status":"paper_baselines","published_at":"2026-11-06"},"model_count":10,"result_count":10,"results":[[2959,0.575,370,7865,7869],[2962,0.4,370,7865,7869],[2958,0.375,370,7865,7869],[2960,0.325,370,7865,7869],[2980,0.075,370,7865,7869],[6438,0.05,370,7865,7869],[6143,0.025,370,7865,7869],[3198,0.0,370,7865,7869],[5300,0.0,370,7865,7869],[7867,0.0,370,7868,7869]]},{"id":"flex-rsi-active-search-2026-paper","name":"Flex-RSI · Active Search (2026 paper)","group":"Agent 与工具","metric":"SPL","unit":"score","public_history":true,"protocol":{"en":"Flex-RSI Table 15: 47 episodes. Final held-out-test SPL; same starter harness and task-specific budget. Models without a ranked completed result are omitted. Per-task scores stay separate from the unequal-coverage average-rank leaderboard.","zh":"Flex-RSI 表 15，47 episodes。取最终隐藏测试的 SPL；采用相同初始框架和任务预算。仅保留作者列入排名且有最终成绩的模型，按任务分列。"},"source":{"name":"Flex-RSI authors","url":"https://arxiv.org/abs/2611.08946","status":"paper_baselines","published_at":"2026-11-06"},"score_max":100,"score_decimals":1,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":12,"result_count":12,"results":[[2962,79.7,370,7870,7871],[2958,78.2,370,7870,7871],[2960,74.4,370,7870,7871],[2959,74.2,370,7870,7871],[5300,69.0,370,7870,7871],[2983,65.5,370,7870,7871],[2980,60.5,370,7870,7871],[6438,59.3,370,7870,7871],[2961,55.8,370,7870,7871],[6143,52.2,370,7870,7871],[7867,47.1,370,7872,7871],[3198,32.5,370,7870,7871]]},{"id":"flex-rsi-robolab-2026-paper","name":"Flex-RSI · RoboLab (2026 paper)","group":"Agent 与工具","metric":"Partial-completion task score","unit":"score","public_history":true,"protocol":{"en":"Flex-RSI Table 16: 24 tasks, 2 episodes each. Final held-out-test Partial-completion task score; same starter harness and task-specific budget. Models without a ranked completed result are omitted. Per-task scores stay separate from the unequal-coverage average-rank leaderboard.","zh":"Flex-RSI 表 16，24 tasks, 2 episodes each。取最终隐藏测试的 Partial-completion task score；采用相同初始框架和任务预算。仅保留作者列入排名且有最终成绩的模型，按任务分列。"},"source":{"name":"Flex-RSI authors","url":"https://arxiv.org/abs/2611.08946","status":"paper_baselines","published_at":"2026-11-06"},"score_max":100,"score_decimals":1,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":9,"result_count":9,"results":[[2959,71.9,370,7870,7873],[2960,62.5,370,7870,7873],[2961,41.0,370,7870,7873],[762,37.8,370,7870,7873],[2958,33.0,370,7870,7873],[5300,31.6,370,7870,7873],[6438,17.4,370,7870,7873],[3198,8.3,370,7870,7873],[6143,5.2,370,7870,7873]]},{"id":"flex-rsi-libero-max-2026-paper","name":"Flex-RSI · LIBERO-MAX (2026 paper)","group":"Agent 与工具","metric":"Dynamic success rate","unit":"%","public_history":true,"protocol":{"en":"Flex-RSI Table 17: 24 paired episodes. Final held-out-test Dynamic success rate; same starter harness and task-specific budget. Models without a ranked completed result are omitted. Per-task scores stay separate from the unequal-coverage average-rank leaderboard.","zh":"Flex-RSI 表 17，24 paired episodes。取最终隐藏测试的 Dynamic success rate；采用相同初始框架和任务预算。仅保留作者列入排名且有最终成绩的模型，按任务分列。"},"source":{"name":"Flex-RSI authors","url":"https://arxiv.org/abs/2611.08946","status":"paper_baselines","published_at":"2026-11-06"},"model_count":6,"result_count":6,"results":[[2959,0.45799999999999996,370,7870,7874],[6438,0.45799999999999996,370,7870,7874],[2960,0.41700000000000004,370,7870,7874],[6143,0.292,370,7870,7874],[3764,0.25,370,7870,7874],[5300,0.25,370,7870,7874]]},{"id":"flex-rsi-libero-pro-2026-paper","name":"Flex-RSI · LIBERO-PRO (2026 paper)","group":"Agent 与工具","metric":"Success rate","unit":"%","public_history":true,"protocol":{"en":"Flex-RSI Table 18: 30 episodes. Final held-out-test Success rate; same starter harness and task-specific budget. Models without a ranked completed result are omitted. Per-task scores stay separate from the unequal-coverage average-rank leaderboard.","zh":"Flex-RSI 表 18，30 episodes。取最终隐藏测试的 Success rate；采用相同初始框架和任务预算。仅保留作者列入排名且有最终成绩的模型，按任务分列。"},"source":{"name":"Flex-RSI authors","url":"https://arxiv.org/abs/2611.08946","status":"paper_baselines","published_at":"2026-11-06"},"model_count":6,"result_count":6,"results":[[2959,0.633,370,7870,7875],[6143,0.5329999999999999,370,7870,7875],[3764,0.5329999999999999,370,7870,7875],[2960,0.5329999999999999,370,7870,7875],[6438,0.5329999999999999,370,7870,7875],[5300,0.5,370,7870,7875]]},{"id":"flex-rsi-ego-exo-memory-2026-paper","name":"Flex-RSI · Ego-Exo Memory (2026 paper)","group":"Agent 与工具","metric":"Accuracy","unit":"%","public_history":true,"protocol":{"en":"Flex-RSI Table 19: 120 questions. Final held-out-test Accuracy; same starter harness and task-specific budget. Models without a ranked completed result are omitted. Per-task scores stay separate from the unequal-coverage average-rank leaderboard.","zh":"Flex-RSI 表 19，120 questions。取最终隐藏测试的 Accuracy；采用相同初始框架和任务预算。仅保留作者列入排名且有最终成绩的模型，按任务分列。"},"source":{"name":"Flex-RSI authors","url":"https://arxiv.org/abs/2611.08946","status":"paper_baselines","published_at":"2026-11-06"},"model_count":11,"result_count":11,"results":[[2959,0.725,370,7870,7876],[2962,0.7,370,7870,7876],[2960,0.675,370,7870,7876],[2958,0.6579999999999999,370,7870,7876],[2983,0.633,370,7870,7876],[5300,0.608,370,7870,7876],[6438,0.5920000000000001,370,7870,7876],[6143,0.5920000000000001,370,7870,7876],[2980,0.5670000000000001,370,7870,7876],[3198,0.517,370,7870,7876],[7867,0.508,370,7872,7876]]},{"id":"flex-rsi-kinematics-2026-paper","name":"Flex-RSI · Kinematics (2026 paper)","group":"Agent 与工具","metric":"Mean relative accuracy","unit":"score","public_history":true,"protocol":{"en":"Flex-RSI Table 20: 67 questions. Final held-out-test Mean relative accuracy; same starter harness and task-specific budget. Models without a ranked completed result are omitted. Per-task scores stay separate from the unequal-coverage average-rank leaderboard.","zh":"Flex-RSI 表 20，67 questions。取最终隐藏测试的 Mean relative accuracy；采用相同初始框架和任务预算。仅保留作者列入排名且有最终成绩的模型，按任务分列。"},"source":{"name":"Flex-RSI authors","url":"https://arxiv.org/abs/2611.08946","status":"paper_baselines","published_at":"2026-11-06"},"score_max":100,"score_decimals":1,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":8,"result_count":8,"results":[[2961,57.2,370,7870,7877],[2959,54.6,370,7870,7877],[762,50.0,370,7870,7877],[6438,49.2,370,7870,7877],[2983,48.2,370,7870,7877],[3386,46.3,370,7870,7877],[5300,43.4,370,7870,7877],[2974,37.2,370,7870,7877]]},{"id":"flex-rsi-mass-2026-paper","name":"Flex-RSI · Mass (2026 paper)","group":"Agent 与工具","metric":"Mean relative accuracy","unit":"score","public_history":true,"protocol":{"en":"Flex-RSI Table 21: 600 items. Final held-out-test Mean relative accuracy; same starter harness and task-specific budget. Models without a ranked completed result are omitted. Per-task scores stay separate from the unequal-coverage average-rank leaderboard.","zh":"Flex-RSI 表 21，600 items。取最终隐藏测试的 Mean relative accuracy；采用相同初始框架和任务预算。仅保留作者列入排名且有最终成绩的模型，按任务分列。"},"source":{"name":"Flex-RSI authors","url":"https://arxiv.org/abs/2611.08946","status":"paper_baselines","published_at":"2026-11-06"},"score_max":100,"score_decimals":1,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":8,"result_count":8,"results":[[2959,63.5,370,7870,7878],[762,63.0,370,7870,7878],[5300,62.7,370,7870,7878],[2961,62.5,370,7870,7878],[3386,60.6,370,7870,7878],[6438,60.4,370,7870,7878],[2974,59.3,370,7870,7878],[2983,57.5,370,7870,7878]]},{"id":"flex-rsi-arc-agi-1-2026-paper","name":"Flex-RSI · ARC-AGI-1 (2026 paper)","group":"Agent 与工具","metric":"Pass@2","unit":"%","public_history":true,"protocol":{"en":"Flex-RSI Table 22: 400 public evaluation puzzles. Final held-out-test Pass@2; same starter harness and task-specific budget. Models without a ranked completed result are omitted. Per-task scores stay separate from the unequal-coverage average-rank leaderboard.","zh":"Flex-RSI 表 22，400 public evaluation puzzles。取最终隐藏测试的 Pass@2；采用相同初始框架和任务预算。仅保留作者列入排名且有最终成绩的模型，按任务分列。"},"source":{"name":"Flex-RSI authors","url":"https://arxiv.org/abs/2611.08946","status":"paper_baselines","published_at":"2026-11-06"},"model_count":8,"result_count":8,"results":[[2958,0.998,370,7870,7879],[2960,0.988,370,7870,7879],[2959,0.98,370,7870,7879],[7867,0.91,370,7872,7879],[6438,0.8079999999999999,370,7870,7879],[5300,0.772,370,7870,7879],[2980,0.635,370,7870,7879],[6143,0.54,370,7870,7879]]},{"id":"flex-rsi-arc-agi-2-2026-paper","name":"Flex-RSI · ARC-AGI-2 (2026 paper)","group":"Agent 与工具","metric":"Pass@2","unit":"%","public_history":true,"protocol":{"en":"Flex-RSI Table 23: 120 public evaluation puzzles. Final held-out-test Pass@2; same starter harness and task-specific budget. Models without a ranked completed result are omitted. Per-task scores stay separate from the unequal-coverage average-rank leaderboard.","zh":"Flex-RSI 表 23，120 public evaluation puzzles。取最终隐藏测试的 Pass@2；采用相同初始框架和任务预算。仅保留作者列入排名且有最终成绩的模型，按任务分列。"},"source":{"name":"Flex-RSI authors","url":"https://arxiv.org/abs/2611.08946","status":"paper_baselines","published_at":"2026-11-06"},"model_count":9,"result_count":9,"results":[[2960,0.958,370,7870,7880],[2959,0.95,370,7870,7880],[2958,0.9329999999999999,370,7870,7880],[7867,0.467,370,7872,7880],[6438,0.225,370,7870,7880],[5300,0.192,370,7870,7880],[2980,0.142,370,7870,7880],[3198,0.042,370,7870,7880],[6143,0.017,370,7870,7880]]},{"id":"flex-rsi-arc-agi-3-2026-paper","name":"Flex-RSI · ARC-AGI-3 (2026 paper)","group":"Agent 与工具","metric":"Relative human action efficiency","unit":"score","public_history":true,"protocol":{"en":"Flex-RSI Table 24: 10 public games. Final held-out-test Relative human action efficiency; same starter harness and task-specific budget. Models without a ranked completed result are omitted. Per-task scores stay separate from the unequal-coverage average-rank leaderboard.","zh":"Flex-RSI 表 24，10 public games。取最终隐藏测试的 Relative human action efficiency；采用相同初始框架和任务预算。仅保留作者列入排名且有最终成绩的模型，按任务分列。"},"source":{"name":"Flex-RSI authors","url":"https://arxiv.org/abs/2611.08946","status":"paper_baselines","published_at":"2026-11-06"},"score_max":100,"score_decimals":1,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":7,"result_count":7,"results":[[2960,44.2,370,7870,7881],[2959,40.4,370,7870,7881],[2958,36.5,370,7870,7881],[6438,0.9,370,7870,7881],[7867,0.6,370,7872,7881],[5300,0.3,370,7870,7881],[6143,0.0,370,7870,7881]]},{"id":"flex-rsi-image-editing-2026-paper","name":"Flex-RSI · Image Editing (2026 paper)","group":"Agent 与工具","metric":"Mean six-family score","unit":"score","public_history":true,"protocol":{"en":"Flex-RSI Table 25: 80 images. Final held-out-test Mean six-family score; same starter harness and task-specific budget. Models without a ranked completed result are omitted. Per-task scores stay separate from the unequal-coverage average-rank leaderboard.","zh":"Flex-RSI 表 25，80 images。取最终隐藏测试的 Mean six-family score；采用相同初始框架和任务预算。仅保留作者列入排名且有最终成绩的模型，按任务分列。"},"source":{"name":"Flex-RSI authors","url":"https://arxiv.org/abs/2611.08946","status":"paper_baselines","published_at":"2026-11-06"},"score_max":100,"score_decimals":1,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":12,"result_count":12,"results":[[2962,74.2,370,7870,7882],[2959,73.8,370,7870,7882],[6438,70.0,370,7870,7882],[2960,69.9,370,7870,7882],[5300,69.7,370,7870,7882],[2958,69.7,370,7870,7882],[6143,68.4,370,7870,7882],[7867,67.6,370,7872,7882],[2983,67.2,370,7870,7882],[2961,66.0,370,7870,7882],[3198,64.3,370,7870,7882],[2980,60.6,370,7870,7882]]},{"id":"flex-rsi-3d-tracking-2026-paper","name":"Flex-RSI · 3D Tracking (2026 paper)","group":"Agent 与工具","metric":"Temporal mean IoU","unit":"score","public_history":true,"protocol":{"en":"Flex-RSI Table 28: 10 clips. Final held-out-test Temporal mean IoU; same starter harness and task-specific budget. Models without a ranked completed result are omitted. Per-task scores stay separate from the unequal-coverage average-rank leaderboard.","zh":"Flex-RSI 表 28，10 clips。取最终隐藏测试的 Temporal mean IoU；采用相同初始框架和任务预算。仅保留作者列入排名且有最终成绩的模型，按任务分列。"},"source":{"name":"Flex-RSI authors","url":"https://arxiv.org/abs/2611.08946","status":"paper_baselines","published_at":"2026-11-06"},"score_max":100,"score_decimals":1,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":13,"result_count":13,"results":[[2959,79.6,370,7870,7883],[2960,77.3,370,7870,7883],[2962,73.8,370,7870,7883],[2980,61.2,370,7870,7883],[7867,60.2,370,7872,7883],[6438,58.9,370,7870,7883],[2965,57.9,370,7870,7883],[2983,53.7,370,7870,7883],[2958,52.8,370,7870,7883],[5300,52.4,370,7870,7883],[2961,47.9,370,7870,7883],[6143,32.0,370,7870,7883],[3198,27.9,370,7870,7883]]},{"id":"flex-rsi-visual-search-2026-paper","name":"Flex-RSI · Visual Search (2026 paper)","group":"Agent 与工具","metric":"Accuracy","unit":"%","public_history":true,"protocol":{"en":"Flex-RSI Table 31: 140 questions. Final held-out-test Accuracy; same starter harness and task-specific budget. Models without a ranked completed result are omitted. Per-task scores stay separate from the unequal-coverage average-rank leaderboard.","zh":"Flex-RSI 表 31，140 questions。取最终隐藏测试的 Accuracy；采用相同初始框架和任务预算。仅保留作者列入排名且有最终成绩的模型，按任务分列。"},"source":{"name":"Flex-RSI authors","url":"https://arxiv.org/abs/2611.08946","status":"paper_baselines","published_at":"2026-11-06"},"model_count":12,"result_count":12,"results":[[2962,0.986,370,7870,7884],[2959,0.9640000000000001,370,7870,7884],[2960,0.9640000000000001,370,7870,7884],[2983,0.95,370,7870,7884],[2958,0.9359999999999999,370,7870,7884],[6438,0.8859999999999999,370,7870,7884],[5300,0.857,370,7870,7884],[2980,0.857,370,7870,7884],[2961,0.807,370,7870,7884],[7867,0.7709999999999999,370,7872,7884],[6143,0.7290000000000001,370,7870,7884],[3198,0.7070000000000001,370,7870,7884]]},{"id":"m261012881-2026-paper","name":"GeoSpatialBench · GeoJSON QA (2026 paper)","group":"视觉与视频","metric":"Open-ended GeoJSON accuracy","unit":"%","protocol":{"en":"4,200 open-ended questions, 700 in each of six cities. Deterministic answer normalization with numerical tolerances; Table 2 pooled Overall.","zh":"六座城市各 700 道开放式 GeoJSON 问题，共 4,200 题。按答案归一化及数值容差确定正误，取表 2 Overall 汇总准确率。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.12881","status":"paper_baselines","published_at":"2026-10-09"},"model_count":9,"result_count":9,"public_history":true,"results":[[2958,0.9636,370,7834,7885],[7886,0.8952,370,7834,7885],[762,0.7812,370,7834,7885],[5335,0.6297999999999999,370,7834,7885],[5300,0.5245000000000001,370,7834,7885],[3153,0.495,370,7834,7885],[4070,0.465,370,7834,7885],[6150,0.37189999999999995,370,7834,7885],[2728,0.2933,370,7834,7885]]},{"id":"m261022009-2026-paper","name":"Visual Sanad (2026 paper)","group":"视觉与视频","metric":"Citation verification accuracy","unit":"%","protocol":{"en":"All 2,400 rendered citation images for each of 12 models under natural and de-biased prompts. Table 2 Accuracy; every row has zero errors and zero unparsed outputs.","zh":"12 个模型分别在自然和去偏提示下评估全部 2,400 张引文图片。取表 2 Accuracy，所有行的错误和未解析输出计数均为零。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.22009","status":"paper_baselines","published_at":"2026-10-15"},"model_count":12,"result_count":24,"public_history":true,"results":[[353,0.8842,370,7887,7888],[353,0.8937999999999999,370,7889,7888],[345,0.8795999999999999,370,7887,7888],[345,0.8917,370,7889,7888],[3196,0.9995999999999999,370,7887,7888],[3196,0.9992,370,7889,7888],[2973,0.9642000000000001,370,7887,7888],[2973,0.9642000000000001,370,7889,7888],[2754,0.2708,370,7887,7888],[2754,0.2846,370,7889,7888],[6342,0.3408,370,7887,7888],[6342,0.3558,370,7889,7888],[2654,0.2792,370,7887,7888],[2654,0.3042,370,7889,7888],[5995,0.2554,370,7887,7888],[5995,0.27,370,7889,7888],[7890,0.2612,370,7887,7888],[7890,0.275,370,7889,7888],[6391,0.2992,370,7887,7888],[6391,0.3046,370,7889,7888],[6343,0.5196000000000001,370,7887,7888],[6343,0.5611999999999999,370,7889,7888],[2642,0.3604,370,7887,7888],[2642,0.5262,370,7889,7888]]},{"id":"m261030828-2026-paper","name":"TutorCaseBench · open response (2026 paper)","group":"知识与推理","metric":"Any-criterion response success","unit":"%","protocol":{"en":"1,560 open-ended tutoring responses, combining 824 immediate and 736 long-term items. Gemini-3.1-Pro judges binary rubric criteria; success means at least one criterion met. Table 1 Overall.","zh":"1,560 个开放式辅导回答，包括 824 个即时和 736 个长期干预项目。Gemini-3.1-Pro 逐项判定规则，至少满足一条即成功。取表 1 Overall。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.30828","status":"paper_baselines","published_at":"2026-10-22"},"model_count":11,"result_count":11,"public_history":true,"results":[[3578,0.5579999999999999,370,7834,7891],[325,0.501,370,7834,7891],[3412,0.522,370,7834,7891],[3424,0.462,370,7834,7891],[3196,0.547,370,7834,7891],[3494,0.41700000000000004,370,7834,7891],[362,0.317,370,7834,7891],[2585,0.414,370,7834,7891],[3492,0.47,370,7834,7891],[6278,0.44299999999999995,370,7834,7891],[7892,0.40299999999999997,370,7834,7891]]},{"id":"atmoscoder-code-2026-paper","name":"AtmosCoder-Bench (code, 20% tolerance) (2026 paper)","group":"数学与科学","metric":"Computed accuracy","unit":"%","public_history":true,"protocol":{"en":"Code-mode accuracy at ±20% numerical tolerance on identical atmospheric-science problems. Published option-mode scores and chance corrections are not used. Reasoning configurations are kept explicit.","zh":"同一组大气科学题目在代码模式下、数值误差容许 ±20% 时的准确率。采用计算结果评分，推理配置单独保留。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.18726","status":"paper_baselines","published_at":"2026-08-19"},"model_count":6,"result_count":6,"results":[[502,0.619,370,7893,7894],[769,0.557,370,7893,7894],[3196,0.53,370,370,7894],[7895,0.45799999999999996,370,370,7894],[1573,0.29600000000000004,370,370,7894],[7896,0.21899999999999997,370,370,7894]]},{"id":"bavground-open-2026-paper","name":"BavGround (open-weight panel) (2026 paper)","group":"知识与推理","metric":"Letter-scoring accuracy","unit":"%","public_history":true,"protocol":{"en":"All column over 618 translated instances across English, German and Bavarian. Fifteen open-weight models use the paper letter-scoring protocol. Open-weight mean is excluded; the separately labeled closed-model reference is omitted from this panel.","zh":"618 个英语、德语和巴伐利亚语翻译实例的 All 准确率，15 个开放权重模型使用论文的答案字母评分协议。排除均值行，另标的闭源参考不并入此面板。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.12894","status":"paper_baselines","published_at":"2026-08-13"},"model_count":15,"result_count":15,"results":[[7897,0.6940000000000001,370,370,7898],[797,0.691,370,370,7898],[587,0.6859999999999999,370,370,7898],[7899,0.6629999999999999,370,370,7898],[7900,0.618,370,370,7898],[7901,0.608,370,370,7898],[1921,0.579,370,370,7898],[7670,0.542,370,370,7898],[7902,0.521,370,370,7898],[560,0.47700000000000004,370,370,7898],[7903,0.46399999999999997,370,370,7898],[7904,0.445,370,370,7898],[7905,0.435,370,370,7898],[7906,0.419,370,370,7898],[7907,0.11199999999999999,370,370,7898]]},{"id":"ruleworld-partial-2026-paper","name":"RuleWorld (partial prompting, 1,000 rules) (2026 paper)","group":"知识与推理","metric":"Average answer accuracy","unit":"%","public_history":true,"protocol":{"en":"Table 3 Avg under partial prompting with 1,000 rules, combining first-order-logic and natural-language representations. DynaRule is a method row and is omitted because its underlying model is not named in the row.","zh":"表 3 中 1,000 条规则、部分提示条件下的 Avg，覆盖一阶逻辑与自然语言表达。DynaRule 是方法行，表内未标明底层模型，因此单独留存。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.22753v1","status":"paper_baselines","published_at":"2026-08-24"},"model_count":10,"result_count":10,"results":[[1659,0.4297,370,370,7908],[1645,0.5077,370,370,7908],[1651,0.5307,370,370,7908],[524,0.549,370,370,7908],[516,0.5162,370,370,7908],[786,0.5132,370,370,7908],[7909,0.6687,370,370,7908],[7910,0.6796,370,370,7908],[7911,0.7283,370,370,7908],[4278,0.7763,370,370,7908]]},{"id":"datakernelbench-pass-2026-paper","name":"DataKernelBench (best configuration) (2026 paper)","group":"编程","metric":"Pass rate","unit":"%","public_history":true,"protocol":{"en":"22 TPC-H SF10 queries on NVIDIA H100, using each model’s published best CUDA/Triton configuration. Pass means matching the reference output. Database engines and TorchPlan controls are excluded.","zh":"NVIDIA H100 上的 22 个 TPC-H SF10 查询，采用各模型论文所列最佳 CUDA/Triton 配置。Pass 表示输出与参考结果一致；排除数据库引擎与 TorchPlan 对照。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.25061","status":"paper_baselines","published_at":"2026-08-25"},"model_count":10,"result_count":10,"results":[[3196,1.0,370,7912,7913],[3572,1.0,370,7914,7913],[3578,1.0,370,7912,7913],[3412,0.545,370,7912,7913],[325,0.955,370,7914,7913],[2576,1.0,370,7914,7913],[3419,0.8640000000000001,370,7912,7913],[3201,0.909,370,7915,7913],[3005,0.818,370,7914,7913],[7689,0.455,370,7914,7913]]},{"id":"ddbench-symptom-2026-paper","name":"DDBench (tier-1, symptom-only) (2026 paper)","group":"编程","metric":"Pass rate","unit":"%","public_history":true,"protocol":{"en":"Tier-1 31-case symptom-only pass rate in the distributed-system debugging evaluation. Thinking mode and serving form are preserved per row; context-augmented results are separate.","zh":"分布式系统调试评估中第一层 31 个案例的仅症状条件通过率。各行保留推理模式和服务形式，补充上下文条件另列。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.14863","status":"paper_baselines","published_at":"2026-08-14"},"model_count":10,"result_count":10,"results":[[2261,0.677,370,7916,7917],[3572,0.419,370,7916,7917],[5101,0.419,370,7918,7917],[7919,0.452,370,7918,7917],[7920,0.32299999999999995,370,7921,7917],[4423,0.29,370,7922,7917],[7177,0.09699999999999999,370,7922,7917],[7923,0.19399999999999998,370,7921,7917],[244,0.32299999999999995,370,7924,7917],[7925,0.065,370,7921,7917]]},{"id":"doc2db-f1-2026-paper","name":"Doc2DB-Bench (base-model panel) (2026 paper)","group":"专业工作","metric":"Overall F1","unit":"score","public_history":true,"protocol":{"en":"Overall Performance F1 in Table 3 for ten base models. Entity-level and relation-level F1 are separate columns. Specialized IE systems are excluded from this base-model panel. Native score 0–100.","zh":"表 3 十个基础模型的 Overall Performance F1。实体级与关系级 F1 为独立列，专用信息抽取系统不并入此基础模型面板。保留 0–100 原始分。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.08459","status":"paper_baselines","published_at":"2026-08-09"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":10,"result_count":10,"results":[[7926,32.97,370,370,7927],[7928,14.29,370,370,7927],[7929,36.07,370,370,7927],[353,59.05,370,370,7927],[3184,75.25,370,370,7927],[7480,65.99,370,370,7927],[7801,70.81,370,370,7927],[7930,73.6,370,370,7927],[7931,63.44,370,370,7927],[7932,57.06,370,370,7927]]},{"id":"dvbench-closed-2026-paper","name":"DVBench (closed-ended) (2026 paper)","group":"视觉与视频","metric":"Average accuracy","unit":"%","public_history":true,"protocol":{"en":"Average accuracy across the four closed-ended dimensions: narrative, animation, perception and reasoning. Human reference and open-ended alignment metrics are omitted.","zh":"叙事、动画、感知和推理四个闭式问答维度的平均准确率。人工参考与开放式对齐指标另存。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.29711","status":"paper_baselines","published_at":"2026-08-30"},"model_count":9,"result_count":9,"results":[[502,0.7791,370,370,7933],[7686,0.7251000000000001,370,370,7933],[3184,0.6531,370,370,7933],[7934,0.7587,370,370,7933],[2584,0.623,370,370,7933],[2576,0.6182,370,370,7933],[2582,0.593,370,370,7933],[7935,0.5882000000000001,370,370,7933],[4070,0.5282,370,370,7933]]},{"id":"easel-final-2026-paper","name":"EASEL (2026 paper)","group":"视觉与视频","metric":"Final reconstruction score","unit":"score","public_history":true,"protocol":{"en":"Standard reconstruction Final score, the paper primary ranking metric; preserves the native 0–1 quality scale. Intermediate and best-trajectory scores are not substituted. Curriculum variants remain configurations.","zh":"标准重建任务的 Final 分，为论文主要排名指标，保留原始 0–1 质量评分。课程消融作为配置，不以中途最高分替代最终结果。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.25417","status":"paper_baselines","published_at":"2026-08-26"},"score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":25,"result_count":26,"results":[[3412,0.535,370,370,7936],[3578,0.472,370,370,7936],[3572,0.447,370,370,7936],[7937,0.428,370,370,7936],[3196,0.426,370,370,7936],[2951,0.428,370,370,7936],[3763,0.44,370,370,7936],[2590,0.442,370,370,7936],[2584,0.433,370,370,7936],[4070,0.432,370,370,7936],[4969,0.44,370,370,7936],[7938,0.438,370,370,7936],[7939,0.431,370,370,7936],[7940,0.426,370,370,7936],[2943,0.442,370,370,7936],[2697,0.42,370,370,7936],[2728,0.418,370,370,7936],[362,0.437,370,370,7936],[363,0.43,370,370,7936],[2833,0.422,370,370,7936],[2845,0.415,370,370,7936],[6864,0.426,370,370,7936],[7499,0.414,370,370,7936],[6254,0.404,370,370,7936],[7941,0.456,370,7942,7936],[7941,0.459,370,370,7936]]},{"id":"evobench-overall-2026-paper","name":"Evo-Bench (2026 paper)","group":"Agent 与工具","metric":"Overall score","unit":"score","public_history":true,"protocol":{"en":"Published Overall raw score for nine harness-evolving models. Colored deltas against CodeAct are not used; CodeAct and Artificial Harness baseline rows are excluded. Native score 0–100.","zh":"九个模型改进 Agent 执行框架后的 Overall 原始分。彩色增量不作成绩；排除 CodeAct 与 Artificial Harness 基线行。保留 0–100 原始分。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.09096","status":"paper_baselines","published_at":"2026-08-10"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":9,"result_count":9,"results":[[762,46.3,370,370,7943],[3479,45.8,370,370,7943],[227,43.5,370,370,7943],[3202,41.5,370,370,7943],[7944,41.4,370,370,7943],[3746,39.1,370,370,7943],[3411,38.7,370,370,7943],[7945,39.4,370,370,7943],[5480,35.9,370,370,7943]]},{"id":"interactbench-pass1-2026-paper","name":"InteractBench (2026 paper)","group":"编程","metric":"Overall pass@1","unit":"%","public_history":true,"protocol":{"en":"Overall pass@1 with ten sampled solutions per interactive-programming task. Reasoning and non-reasoning modes retain the same model identity; measured zero cells remain zero.","zh":"每道交互式编程题采样十份解答的 Overall pass@1。推理与非推理模式沿用同一模型身份，保留实际测得的零分。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.29632","status":"paper_baselines","published_at":"2026-08-30"},"model_count":13,"result_count":16,"results":[[3002,0.554,370,7893,7946],[340,0.542,370,7893,7946],[3009,0.332,370,7893,7946],[764,0.305,370,7893,7946],[3419,0.254,370,7893,7946],[6705,0.159,370,7893,7946],[766,0.162,370,7893,7946],[5343,0.134,370,7893,7946],[7947,0.105,370,7893,7946],[786,0.096,370,7893,7946],[6301,0.058,370,7893,7946],[1662,0.045,370,7893,7946],[3009,0.148,370,7948,7946],[345,0.062,370,7948,7946],[786,0.017,370,7948,7946],[6301,0.012,370,7948,7946]]},{"id":"indicquest-v2-english-2026-paper","name":"L3Cube-IndicQuest v2 (English) (2026 paper)","group":"知识与推理","metric":"Exact-substring accuracy","unit":"%","public_history":true,"protocol":{"en":"English subset accuracy under the published exact-substring matching protocol. LLM-judge and word-overlap scores are separate metrics.","zh":"英语子集在论文精确子串匹配协议下的准确率，LLM 评判与词重叠分数为独立指标。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.15535","status":"paper_baselines","published_at":"2026-08-16"},"model_count":6,"result_count":6,"results":[[2055,0.62,370,370,7949],[7950,0.373,370,370,7949],[3494,0.37799999999999995,370,370,7949],[7951,0.27699999999999997,370,370,7949],[7952,0.21899999999999997,370,370,7949],[806,0.182,370,370,7949]]},{"id":"bearbench-overall-2026-paper","name":"BEAR-Bench (2026 paper)","group":"知识与推理","metric":"Overall accuracy","unit":"%","public_history":true,"protocol":{"en":"Full 1,000-question business/science panel, combining English and Russian items. Uses Overall accuracy, not any overlapping figure/diagram/equation subset.","zh":"完整 1,000 道商业与科学题目，包含英语与俄语。采用 Overall 准确率，图形、示意图与公式子集不重复合并。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.17895","status":"paper_baselines","published_at":"2026-08-18"},"model_count":16,"result_count":16,"results":[[2576,0.754,370,370,7953],[3412,0.7509999999999999,370,370,7953],[2602,0.674,370,370,7953],[3603,0.65,370,370,7953],[3572,0.627,370,370,7953],[2261,0.615,370,370,7953],[6399,0.598,370,370,7953],[2055,0.49,370,370,7953],[2584,0.608,370,370,7953],[4070,0.496,370,370,7953],[4969,0.486,370,370,7953],[7954,0.345,370,370,7953],[6150,0.254,370,370,7953],[7955,0.235,370,370,7953],[6245,0.14300000000000002,370,370,7953],[6340,0.106,370,370,7953]]},{"id":"cpibench-overall-2026-paper","name":"CPI-Bench (2026 paper)","group":"生成与创作","metric":"CPI-Overall","unit":"score","public_history":true,"protocol":{"en":"Published CPI-Overall rating across general, practical and intelligent image editing on the English split. Native 0–5 scale; Variance is excluded.","zh":"英语测试集上通用、实用与智能图像编辑的 CPI-Overall 评分，保留 0–5 量纲，排除 Variance 汇总行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.14546","status":"paper_baselines","published_at":"2026-08-14"},"score_max":5,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 5","zh":"分数 / 5"},"model_count":10,"result_count":10,"results":[[6368,4.7,370,370,7956],[6366,4.58,370,370,7956],[7957,4.68,370,370,7956],[7958,4.15,370,370,7956],[7959,4.19,370,370,7956],[6361,3.37,370,370,7956],[7653,3.37,370,370,7956],[7960,3.2,370,370,7956],[7961,3.43,370,370,7956],[7253,2.96,370,370,7956]]},{"id":"naru-mcq-2026-paper","name":"NARU (2026 paper)","group":"视觉与视频","metric":"Overall MCQ accuracy","unit":"%","public_history":true,"protocol":{"en":"Published Overall MCQ accuracy on the shared long-form video narrative/culture panel. Per-model video sampling rates are preserved as configurations.","zh":"同一长视频叙事与文化面板的 Overall 选择题准确率，各模型视频采样率作为配置保留。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.13210","status":"paper_baselines","published_at":"2026-08-13"},"model_count":8,"result_count":8,"results":[[6133,0.762,370,7962,7963],[6130,0.7,370,7962,7963],[766,0.514,370,7962,7963],[4070,0.39799999999999996,370,7964,7963],[7965,0.374,370,7962,7963],[7966,0.297,370,7964,7963],[2904,0.29600000000000004,370,7964,7963],[7967,0.315,370,7968,7963]]},{"id":"numerosityvlm-accuracy-2026-paper","name":"NumerosityVLM (2026 paper)","group":"视觉与视频","metric":"Accuracy","unit":"%","public_history":true,"protocol":{"en":"Table 2 Overall Accuracy for the seven named vision-language encoders/models. MAE and normalized absolute error remain separate metrics.","zh":"表 2 七个具名视觉语言编码器或模型的整体准确率，平均绝对误差与归一化误差另存。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.15425","status":"paper_baselines","published_at":"2026-08-15"},"model_count":7,"result_count":7,"results":[[2780,0.15560000000000002,370,370,7969],[7970,0.2039,370,370,7969],[7971,0.2724,370,370,7969],[7972,0.7128,370,370,7969],[2728,0.7720999999999999,370,370,7969],[7973,0.7773,370,370,7969],[7974,0.8397,370,370,7969]]},{"id":"odineval-resolved-2026-paper","name":"OdinEval (2026 paper)","group":"编程","metric":"Resolved","unit":"%","public_history":true,"protocol":{"en":"Resolved rate over the same 168 filtered repository-repair instances. Patch application and reproducer success are separate diagnostics.","zh":"同一组筛选后的 168 个仓库修复案例的 Resolved 比例；补丁应用与问题复现成功率为独立诊断。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.18595","status":"paper_baselines","published_at":"2026-08-19"},"model_count":6,"result_count":6,"results":[[4048,0.506,370,370,7975],[2973,0.488,370,5009,7975],[227,0.5770000000000001,370,370,7975],[3198,0.45799999999999996,370,370,7975],[3157,0.488,370,370,7975],[3155,0.667,370,370,7975]]},{"id":"tracebench-operators-2026-paper","name":"TRACE-Bench (2026 paper)","group":"生成与创作","metric":"Mean operator score","unit":"score","public_history":true,"protocol":{"en":"Avg across four operator-aligned metrics, averaged over slots 1–8. Native 0–1 score; CLIP similarity remains a separate metric.","zh":"槽位 1–8 上四类操作对应指标的平均分，保留 0–1 原始量纲，CLIP 相似度另列。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.16765","status":"paper_baselines","published_at":"2026-08-17"},"score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":8,"result_count":9,"results":[[6367,0.8118,370,370,7976],[6365,0.7981,370,370,7976],[6839,0.8205,370,370,7976],[6366,0.8172,370,370,7976],[6362,0.6561,370,370,7976],[7977,0.5889,370,370,7976],[7978,0.5483,370,370,7976],[6361,0.5858,370,370,7976],[7295,0.5348,370,370,7976]]},{"id":"trustdabench-mrs-2026-paper","name":"TrustDABench (2026 paper)","group":"专业工作","metric":"Mean reliability score","unit":"score","public_history":true,"protocol":{"en":"Avg MRS is the arithmetic mean across AIDABench-QA and DABench under the published reliability protocol. Native 0–100 composite, not a task pass rate.","zh":"采用论文可靠性协议中 AIDABench-QA 与 DABench 的平均 MRS，保留 0–100 综合评分。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.24145","status":"paper_baselines","published_at":"2026-08-25"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":8,"result_count":8,"results":[[3202,14.92,370,370,7979],[3185,12.16,370,370,7979],[227,9.1,370,370,7979],[502,6.98,370,370,7979],[5789,13.58,370,370,7979],[3196,24.21,370,370,7979],[4973,6.04,370,370,7979],[793,7.5,370,370,7979]]},{"id":"usertoolbench-exact-2026-paper","name":"UserToolBench (2026 paper)","group":"Agent 与工具","metric":"Exact trajectory accuracy","unit":"%","public_history":true,"protocol":{"en":"Table 4 Avg Exact Acc. on the common user-conditioned tool-use trajectory set. Relaxed trajectory accuracy is a separate evaluation.","zh":"表 4 用户条件工具调用轨迹的 Avg Exact Acc.，宽松轨迹准确率另存。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.10042","status":"paper_baselines","published_at":"2026-08-10"},"model_count":9,"result_count":9,"results":[[3490,0.3865,370,370,7980],[3184,0.31980000000000003,370,370,7980],[3603,0.37020000000000003,370,370,7980],[3492,0.4249,370,370,7980],[3415,0.3432,370,370,7980],[268,0.4936,370,370,7980],[7981,0.24760000000000001,370,370,7980],[7982,0.2959,370,370,7980],[7983,0.20800000000000002,370,370,7980]]},{"id":"vibelife-avg3-2026-paper","name":"VibeLifeBench (2026 paper)","group":"Agent 与工具","metric":"avg@3 task score","unit":"score","public_history":true,"protocol":{"en":"Main-panel avg@3: mean task score across three runs, under the same persistent life-assistance environment. Native 0–100 score; max@3 and min@3 are not used.","zh":"同一持续生活助理环境下每个任务三次运行的平均分 avg@3，保留 0–100 原始评分。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.10875","status":"paper_baselines","published_at":"2026-08-11"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":7,"result_count":7,"results":[[755,32.5,370,370,7984],[3196,30.1,370,370,7984],[3415,27.5,370,370,7984],[3479,27.5,370,370,7984],[227,25.4,370,370,7984],[3187,22.6,370,370,7984],[3185,21.1,370,370,7984]]},{"id":"videogaia-agent-2026-paper","name":"VideoGAIA (agent loop) (2026 paper)","group":"视觉与视频","metric":"Overall pass@1","unit":"%","public_history":true,"protocol":{"en":"Main twenty-model agent-loop panel, Overall pass@1 across the six published domains.","zh":"主实验 20 个模型在 Agent 循环设置下、六个领域的 Overall pass@1。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.14718","status":"paper_baselines","published_at":"2026-08-12"},"model_count":20,"result_count":20,"results":[[3191,0.5018,370,7985,7986],[6412,0.5203,370,7985,7986],[3189,0.49079999999999996,370,7985,7986],[4961,0.5609000000000001,370,7985,7986],[3157,0.5018,370,7985,7986],[5722,0.4686,370,7985,7986],[7937,0.5018,370,7985,7986],[7378,0.583,370,7985,7986],[7223,0.48710000000000003,370,7985,7986],[2998,0.4982,370,7985,7986],[3187,0.5277000000000001,370,7985,7986],[3155,0.5461,370,7985,7986],[502,0.5092,370,7985,7986],[2129,0.4391,370,7985,7986],[3183,0.4797,370,7985,7986],[3181,0.3469,370,7985,7986],[3194,0.369,370,7985,7986],[340,0.5092,370,7985,7986],[3184,0.5203,370,7985,7986],[3196,0.5314,370,7985,7986]]},{"id":"viseditbench-pass-2026-paper","name":"VisEditBench (2026 paper)","group":"编程","metric":"Overall pass rate","unit":"%","public_history":true,"protocol":{"en":"Overall pass rate across all visualization-editing examples, with eight editing intents. Category scores are retained in the source fixture.","zh":"全部可视化编辑样例的 Overall 通过率，覆盖八类编辑意图，分类分数保留在来源表。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.10408","status":"paper_baselines","published_at":"2026-08-11"},"model_count":20,"result_count":20,"results":[[353,0.5575,370,370,7987],[7988,0.6,370,370,7987],[2058,0.6568,370,370,7987],[7989,0.17059999999999997,370,370,7987],[7685,0.6273,370,370,7987],[7686,0.7445999999999999,370,370,7987],[6677,0.21289999999999998,370,370,7987],[6668,0.3496,370,370,7987],[6689,0.42950000000000005,370,370,7987],[7990,0.0741,370,370,7987],[7991,0.1813,370,370,7987],[7992,0.2633,370,370,7987],[7993,0.3317,370,370,7987],[7994,0.3483,370,370,7987],[2515,0.3532,370,370,7987],[6341,0.17550000000000002,370,370,7987],[6298,0.3884,370,370,7987],[6342,0.42090000000000005,370,370,7987],[6238,0.4705,370,370,7987],[6343,0.5172,370,370,7987]]},{"id":"whatifbench-final-2026-paper","name":"WhatIfBench (2026 paper)","group":"知识与推理","metric":"Final aggregated score","unit":"score","public_history":true,"protocol":{"en":"Final aggregated score combining the Process Metric and Rubric Metric across STEM, HSS and hybrid cases. Native 0–100 composite rating.","zh":"STEM、人文社会科学及交叉案例上，结合过程指标与细则指标的 Final 综合分，保留 0–100 评分。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.27953","status":"paper_baselines","published_at":"2026-08-28"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":6,"result_count":6,"results":[[3186,51.99,370,370,7995],[3185,55.12,370,370,7995],[3225,52.26,370,370,7995],[6237,56.33,370,370,7995],[3181,59.11,370,370,7995],[3196,64.62,370,370,7995]]},{"id":"worldbench-cts-2026-paper","name":"WorldBench (2026 paper)","group":"Agent 与工具","metric":"CTS","unit":"%","score_max":100,"higher_is_better":true,"public_history":true,"protocol":{"en":"Completes workspace tasks across eight cultures. A success requires all task checks to pass and every pre-existing non-target file to remain unchanged; results are macro-averaged across cultures and tools.","zh":"跨八种文化情境完成工作区任务。全部任务检查通过，且所有原有非目标文件保持不变，才算成功；按文化情境和工具宏平均。"},"source":{"name":"WorldBench authors","status":"paper_baselines","url":"https://arxiv.org/html/2609.01056v1#S3.T3","published_at":"2026-09-01"},"model_count":9,"result_count":9,"results":[[502,0.49200000000000005,370,7996,7997],[2058,0.488,370,7996,7997],[7998,0.48,370,7996,7997],[353,0.386,370,7996,7997],[2129,0.34600000000000003,370,7996,7997],[5648,0.24100000000000002,370,7996,7997],[587,0.125,370,7996,7997],[7999,0.11599999999999999,370,7996,7997],[7897,0.10800000000000001,370,7996,7997]]},{"id":"harnessdev-creation-2026-paper","name":"HarnessDev · creation (2026 paper)","group":"Agent 与工具","metric":"score","unit":"score","score_max":100,"higher_is_better":true,"public_history":true,"protocol":{"en":"Agents build execution harnesses and evaluate them with their own model. The reported composite averages SWE-Pro, Terminal-Bench, EQ-Bench3 and BrowseComp, with three independently built harnesses per creator.","zh":"智能体构建执行框架，再用自身模型运行评测。采用SWE-Pro、Terminal-Bench、EQ-Bench3与BrowseComp四项的平均分，每个构建模型独立制作三个框架。"},"source":{"name":"HarnessDev authors","status":"paper_baselines","url":"https://arxiv.org/html/2609.01437v1#S4.T3","published_at":"2026-09-01"},"model_count":6,"result_count":6,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"results":[[3479,67.8,370,8000,8001],[3196,55.1,370,8002,8001],[3412,55.6,370,8000,8001],[3492,45.2,370,8000,8001],[3587,44.0,370,8000,8001],[4225,22.8,370,8000,8001]]},{"id":"insight-accuracy-2026-paper","name":"InSight (2026 paper)","group":"Agent 与工具","metric":"Accuracy","unit":"%","score_max":100,"higher_is_better":true,"public_history":true,"protocol":{"en":"Verifies claims by interacting with web visualizations. Accuracy uses the same stratified 500-claim subset, with supported, refuted and insufficient-evidence labels; interaction budgets are retained per row.","zh":"通过交互式网页图表核验陈述。采用同一组分层抽取的500条陈述，判断支持、反驳或证据不足；各行保留交互预算。"},"source":{"name":"InSight authors","status":"paper_baselines","url":"https://arxiv.org/html/2609.01383v1#S4.T1","published_at":"2026-09-01"},"model_count":10,"result_count":14,"results":[[3415,0.5,370,8003,8004],[3415,0.41600000000000004,370,8005,8004],[3415,0.442,370,8006,8004],[3196,0.5720000000000001,370,8005,8004],[2585,0.47200000000000003,370,8005,8004],[2585,0.45399999999999996,370,8006,8004],[2627,0.43200000000000005,370,8005,8004],[2653,0.312,370,8005,8004],[8007,0.44799999999999995,370,8005,8004],[3751,0.45,370,8005,8004],[3751,0.462,370,8006,8004],[5672,0.38,370,8005,8004],[8008,0.37,370,8005,8004],[8009,0.34,370,8005,8004]]},{"id":"vmetaphor-conceptual-2026-paper","name":"VMetaphor-Bench · conceptual prompt (2026 paper)","group":"生成与创作","metric":"Overall","unit":"%","score_max":100,"higher_is_better":true,"public_history":true,"protocol":{"en":"Generates images expressing conceptual metaphors. The overall multiple-choice score tests whether the generated image preserves the intended domains, relational structure, meaning and cross-domain mapping.","zh":"生成表达概念隐喻的图像。整体选择题分数检验图像是否保留指定领域、关系结构、含义及跨领域映射。"},"source":{"name":"VMetaphor-Bench authors","status":"paper_baselines","url":"https://arxiv.org/html/2609.02502v1#S3.T1","published_at":"2026-09-02"},"model_count":11,"result_count":11,"results":[[7650,0.785,370,8010,8011],[6839,0.848,370,8010,8011],[8012,0.8540000000000001,370,8010,8011],[8013,0.684,370,8010,8011],[8014,0.708,370,8010,8011],[8015,0.7240000000000001,370,8010,8011],[6379,0.7490000000000001,370,8010,8011],[8016,0.76,370,8010,8011],[7295,0.614,370,8010,8011],[8017,0.67,370,8010,8011],[8018,0.73,370,8010,8011]]},{"id":"layerwise-understanding-2026-paper","name":"LayerWiseBench · understanding (2026 paper)","group":"视觉与视频","metric":"Macro Avg.","unit":"%","score_max":100,"higher_is_better":true,"public_history":true,"protocol":{"en":"Reads layered charts and answers questions about layer attribution, binding and visibility ordering. The score equally averages the three task-family accuracies on 727 held-out items.","zh":"阅读分层图表，回答图层归属、绑定和可见顺序问题。在727条留出题目上，对三类任务准确率等权平均。"},"source":{"name":"LayerWiseBench authors","status":"paper_baselines","url":"https://arxiv.org/html/2609.08657v1#S6.T1","published_at":"2026-09-08"},"model_count":7,"result_count":7,"results":[[2584,0.8399,370,8019,8020],[4070,0.8318000000000001,370,8019,8020],[5828,0.7827,370,8019,8020],[2697,0.8051,370,8019,8020],[2728,0.7051999999999999,370,8019,8020],[6342,0.7837000000000001,370,8019,8020],[6630,0.8039000000000001,370,8019,8020]]},{"id":"kona-single-2026-paper","name":"KoNA · single queries (2026 paper)","group":"安全与可靠性","metric":"Single Average","unit":"%","score_max":100,"higher_is_better":true,"public_history":true,"protocol":{"en":"Tests selective non-compliance in visual questions across five task types. Single queries contain one invalid demand. Default, reasoning, guidance and fine-tuned settings remain distinct.","zh":"在五类视觉问答任务中检验选择性拒答能力。单项请求包含一项不成立的要求。保留默认、推理、行为指导与微调设置。"},"source":{"name":"KoNA authors","status":"paper_baselines","url":"https://arxiv.org/html/2609.04720v1#S4.T2","published_at":"2026-09-04"},"model_count":8,"result_count":20,"results":[[2766,0.25,370,8021,8022],[2675,0.35,370,8021,8022],[2771,0.29,370,8021,8022],[2166,0.59,370,8021,8022],[2058,0.55,370,8021,8022],[766,0.49,370,8021,8022],[2766,0.23,370,8023,8022],[2675,0.4,370,8023,8022],[2771,0.18,370,8023,8022],[2166,0.43,370,8023,8022],[2058,0.56,370,8023,8022],[766,0.5,370,8023,8022],[2766,0.32,370,8024,8022],[2675,0.66,370,8024,8022],[2771,0.47,370,8024,8022],[2166,0.87,370,8024,8022],[2058,0.72,370,8024,8022],[766,0.85,370,8024,8022],[8025,0.92,370,8026,8022],[8027,0.91,370,8026,8022]]},{"id":"kona-compound-2026-paper","name":"KoNA · compound queries (2026 paper)","group":"安全与可靠性","metric":"Compound Average","unit":"%","score_max":100,"higher_is_better":true,"public_history":true,"protocol":{"en":"Tests selective non-compliance in visual questions across five task types. Compound queries combine invalid and answerable parts; both must be handled correctly. Default, reasoning, guidance and fine-tuned settings remain distinct.","zh":"在五类视觉问答任务中检验选择性拒答能力。复合请求同时包含不成立与可回答的部分，须分别正确处理。保留默认、推理、行为指导与微调设置。"},"source":{"name":"KoNA authors","status":"paper_baselines","url":"https://arxiv.org/html/2609.04720v1#S4.T2","published_at":"2026-09-04"},"model_count":8,"result_count":20,"results":[[2766,0.1,370,8021,8022],[2675,0.23,370,8021,8022],[2771,0.11,370,8021,8022],[2166,0.34,370,8021,8022],[2058,0.42,370,8021,8022],[766,0.39,370,8021,8022],[2766,0.11,370,8023,8022],[2675,0.23,370,8023,8022],[2771,0.1,370,8023,8022],[2166,0.32,370,8023,8022],[2058,0.43,370,8023,8022],[766,0.4,370,8023,8022],[2766,0.08,370,8024,8022],[2675,0.45,370,8024,8022],[2771,0.16,370,8024,8022],[2166,0.79,370,8024,8022],[2058,0.6,370,8024,8022],[766,0.77,370,8024,8022],[8025,0.9,370,8026,8022],[8027,0.87,370,8026,8022]]},{"id":"authbench-success5-2026-paper","name":"AuthBench · authorship retrieval (2026 paper)","group":"知识与推理","metric":"S@5","unit":"%","score_max":100,"higher_is_better":true,"public_history":true,"protocol":{"en":"Retrieves writing by the same author across languages and genres. Zero-shot Success@5 is the fraction of queries with a correct-author document among the top five retrieved items; this cohort includes all 47 neural checkpoints.","zh":"跨语言和体裁检索同一作者的文字。零样本Success@5统计前五个检索结果中出现正确作者文档的查询比例，包含全部47个神经模型。"},"source":{"name":"AuthBench authors","status":"paper_baselines","url":"https://arxiv.org/html/2609.06771v1#A2.T8","published_at":"2026-09-06"},"model_count":47,"result_count":47,"results":[[8028,0.251,370,8029,8030],[8031,0.247,370,8029,8030],[8032,0.213,370,8029,8030],[8033,0.207,370,8029,8030],[8034,0.207,370,8029,8030],[8035,0.197,370,8029,8030],[8036,0.254,370,8029,8030],[4860,0.253,370,8029,8030],[8037,0.22,370,8029,8030],[8038,0.212,370,8029,8030],[8039,0.21,370,8029,8030],[8040,0.242,370,8029,8030],[8041,0.24,370,8029,8030],[8042,0.258,370,8029,8030],[8043,0.25,370,8029,8030],[8044,0.241,370,8029,8030],[8045,0.24,370,8029,8030],[8046,0.236,370,8029,8030],[8047,0.212,370,8029,8030],[8048,0.211,370,8029,8030],[8049,0.208,370,8029,8030],[8050,0.2,370,8029,8030],[8051,0.195,370,8029,8030],[8052,0.189,370,8029,8030],[8053,0.188,370,8029,8030],[8054,0.177,370,8029,8030],[8055,0.177,370,8029,8030],[8056,0.175,370,8029,8030],[8057,0.174,370,8029,8030],[8058,0.172,370,8029,8030],[8059,0.168,370,8029,8030],[8060,0.167,370,8029,8030],[8061,0.167,370,8029,8030],[8062,0.163,370,8029,8030],[8063,0.162,370,8029,8030],[8064,0.161,370,8029,8030],[8065,0.16,370,8029,8030],[8066,0.158,370,8029,8030],[8067,0.152,370,8029,8030],[8068,0.149,370,8029,8030],[8069,0.145,370,8029,8030],[8070,0.145,370,8029,8030],[8071,0.138,370,8029,8030],[8072,0.133,370,8029,8030],[8073,0.107,370,8029,8030],[8074,0.1,370,8029,8030],[8075,0.016,370,8029,8030]]},{"id":"vantage-overall-2026-paper","name":"VANTAGE-Bench (2026 paper)","group":"视觉与视频","metric":"Overall","unit":"score","score_max":100,"higher_is_better":true,"public_history":true,"protocol":{"en":"Evaluates eight video-understanding tasks spanning semantics, space and time. The native composite equally averages four task-group scores; source format failures remain in the published aggregate.","zh":"评测语义、空间和时间相关的八项视频理解任务。原生综合分数对四组任务等权平均，并保留原文中的格式失败结果。"},"source":{"name":"VANTAGE-BENCH authors","status":"paper_baselines","url":"https://arxiv.org/html/2609.09396v1#S4.T1","published_at":"2026-09-08"},"model_count":17,"result_count":17,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"results":[[6341,41.86,370,8076,8077],[6342,50.07,370,8076,8077],[6343,55.38,370,8076,8077],[4070,48.3,370,8076,8077],[2584,49.25,370,8076,8077],[7555,45.94,370,8076,8077],[7066,54.46,370,8076,8077],[7554,47.03,370,8076,8077],[7160,46.94,370,8076,8077],[7159,60.67,370,8076,8077],[7158,63.62,370,8076,8077],[7567,21.28,370,8076,8077],[4219,52.61,370,8076,8077],[3412,62.66,370,8076,8077],[4035,53.25,370,8076,8077],[3405,69.52,370,8076,8077],[762,68.04,370,8076,8077]]},{"id":"scifigurebench-overall-2026-paper","name":"SciFigureBench · image-only (2026 paper)","group":"编程","metric":"Overall","unit":"score","score_max":100,"higher_is_better":true,"public_history":true,"protocol":{"en":"Reconstructs editable plotting code from 337 scientific figure panels. The overall presentation-recovery score combines execution and visual assessment; failed programs or absent output images receive zero visual scores.","zh":"根据337幅科学图表重建可编辑的绘图代码。整体呈现恢复分数综合执行与视觉评估；程序执行失败或未生成图像时，视觉分数记零。"},"source":{"name":"SciFigure2Code authors","status":"paper_baselines","url":"https://arxiv.org/html/2609.08155v1#S6.T3","published_at":"2026-09-08"},"model_count":14,"result_count":14,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"results":[[3572,76.87,370,8078,8079],[2261,80.31,370,8078,8079],[3578,83.5,370,8078,8079],[6901,64.76,370,8078,8079],[2582,64.05,370,8078,8079],[8080,63.02,370,8078,8079],[8081,59.26,370,8078,8079],[6374,48.59,370,8078,8079],[8082,40.08,370,8078,8079],[2752,38.13,370,8078,8079],[7161,38.11,370,8078,8079],[8083,32.55,370,8078,8079],[8084,27.49,370,8078,8079],[2515,20.15,370,8078,8079]]},{"id":"genscale-common-object-2026-paper","name":"GenScale · common-object scale (2026 paper)","group":"生成与创作","metric":"Plaus.","unit":"%","score_max":100,"higher_is_better":true,"public_history":true,"protocol":{"en":"Generates images with physically plausible relative object sizes. The common-object task reports the fraction rated at the correct scale, pooled over natural-depth and same-plane scenarios.","zh":"生成物体相对大小符合物理常识的图像。常见物体任务合并自然景深与同一平面两类场景，统计被评为比例正确的图像比例。"},"source":{"name":"GenScale authors","status":"paper_baselines","url":"https://arxiv.org/html/2609.00525v1#S4.T4","published_at":"2026-09-01"},"model_count":7,"result_count":7,"results":[[6839,0.565,370,8085,8086],[6368,0.519,370,8085,8086],[8087,0.461,370,8085,8086],[8088,0.467,370,8085,8086],[8089,0.4,370,8085,8086],[6360,0.33899999999999997,370,8085,8086],[8090,0.311,370,8085,8086]]},{"id":"memecult-human-meme-only-2026-paper","name":"MemeCULT-1K · meme-only (2026 paper)","group":"视觉与视频","metric":"M","unit":"score","score_max":5,"higher_is_better":true,"public_history":true,"protocol":{"en":"Explains culturally grounded memes in English, Bengali and Hindi. Human raters score fluency, cultural correctness and humor relevance on 1–5; the published mean covers100 memes per language.","zh":"解释英语、孟加拉语和印地语中的文化梗图。人工按流畅性、文化准确性和幽默相关性打1—5分；采用每种语言100幅梗图的平均结果。"},"source":{"name":"MemeCULT-1K authors","status":"paper_baselines","url":"https://arxiv.org/html/2609.01772v1#A1.T9","published_at":"2026-09-01"},"model_count":9,"result_count":9,"saturation_excluded":true,"scale_label":{"en":"Score / 5","zh":"分数 / 5"},"results":[[2055,3.88,370,8091,8092],[5505,3.42,370,8091,8092],[3708,3.33,370,8091,8092],[5738,3.17,370,8091,8092],[6342,2.73,370,8091,8092],[2654,2.4,370,8091,8092],[6246,1.82,370,8091,8092],[6341,2.12,370,8091,8092],[6668,1.86,370,8091,8092]]},{"id":"memecult-human-context-aware-2026-paper","name":"MemeCULT-1K · context-aware (2026 paper)","group":"视觉与视频","metric":"C","unit":"score","score_max":5,"higher_is_better":true,"public_history":true,"protocol":{"en":"Explains culturally grounded memes in English, Bengali and Hindi. Human raters score fluency, cultural correctness and humor relevance on 1–5; the published mean covers100 memes per language.","zh":"解释英语、孟加拉语和印地语中的文化梗图。人工按流畅性、文化准确性和幽默相关性打1—5分；采用每种语言100幅梗图的平均结果。"},"source":{"name":"MemeCULT-1K authors","status":"paper_baselines","url":"https://arxiv.org/html/2609.01772v1#A1.T9","published_at":"2026-09-01"},"model_count":9,"result_count":9,"saturation_excluded":true,"scale_label":{"en":"Score / 5","zh":"分数 / 5"},"results":[[2055,4.18,370,8093,8092],[5505,4.0,370,8093,8092],[3708,3.95,370,8093,8092],[5738,3.88,370,8093,8092],[6342,3.8,370,8093,8092],[2654,3.55,370,8093,8092],[6246,2.98,370,8093,8092],[6341,2.93,370,8093,8092],[6668,2.63,370,8093,8092]]},{"id":"talkfa-wiki-base-2026-paper","name":"TalkFa · Wiki-FaDial Base (2026 paper)","group":"指令与长文","metric":"ROUGE-L","unit":"score","score_max":1,"higher_is_better":true,"public_history":true,"protocol":{"en":"Generates Farsi knowledge-grounded dialogue from Wikipedia material. ROUGE-L measures reference overlap on the held-out Wiki-FaDial test split under the six-turn generation protocol.","zh":"根据维基百科材料生成有知识依据的波斯语对话。在Wiki-FaDial留出测试集上，采用六轮生成协议，以ROUGE-L衡量与参考对话的重合程度。"},"source":{"name":"TalkFa authors","status":"paper_baselines","url":"https://arxiv.org/html/2609.01810v1#S4.T4","published_at":"2026-09-01"},"model_count":6,"result_count":6,"saturation_excluded":true,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"results":[[623,0.449,370,8094,8095],[613,0.353,370,8094,8095],[1921,0.601,370,8094,8095],[587,0.655,370,8094,8095],[8096,0.613,370,8094,8095],[8097,0.546,370,8094,8095]]},{"id":"talkfa-wiki-lora-2026-paper","name":"TalkFa · Wiki-FaDial +LoRA (2026 paper)","group":"指令与长文","metric":"ROUGE-L","unit":"score","score_max":1,"higher_is_better":true,"public_history":true,"protocol":{"en":"Generates Farsi knowledge-grounded dialogue from Wikipedia material. ROUGE-L measures reference overlap on the held-out Wiki-FaDial test split under the six-turn generation protocol.","zh":"根据维基百科材料生成有知识依据的波斯语对话。在Wiki-FaDial留出测试集上，采用六轮生成协议，以ROUGE-L衡量与参考对话的重合程度。"},"source":{"name":"TalkFa authors","status":"paper_baselines","url":"https://arxiv.org/html/2609.01810v1#S4.T4","published_at":"2026-09-01"},"model_count":6,"result_count":6,"saturation_excluded":true,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"results":[[623,0.65,370,8098,8095],[613,0.66,370,8098,8095],[1921,0.683,370,8098,8095],[587,0.668,370,8098,8095],[8096,0.688,370,8098,8095],[8097,0.687,370,8098,8095]]},{"id":"lexissue-generation-2026-paper","name":"LexIssue · legal issue generation (2026 paper)","group":"专业工作","metric":"LIG","unit":"score","score_max":1,"higher_is_better":true,"public_history":true,"protocol":{"en":"Identifies disputed legal issues in 430 Chinese litigation cases. The score averages case-level weighted F1 after a legal-rubric alignment of generated issues with expert references.","zh":"识别430宗中国诉讼案件中的争议焦点。先按法律评分标准将生成问题与专家参考对齐，再对案件级加权F1取平均。"},"source":{"name":"LexIssue authors","status":"paper_baselines","url":"https://arxiv.org/html/2609.02954v1#S6.T1","published_at":"2026-09-02"},"model_count":8,"result_count":24,"saturation_excluded":true,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"results":[[340,0.542,370,5574,8099],[340,0.475,370,8100,8099],[340,0.634,370,8101,8099],[6133,0.701,370,5574,8099],[6133,0.699,370,8100,8099],[6133,0.715,370,8101,8099],[769,0.538,370,5574,8099],[769,0.502,370,8100,8099],[769,0.594,370,8101,8099],[3201,0.583,370,5574,8099],[3201,0.571,370,8100,8099],[3201,0.628,370,8101,8099],[8102,0.464,370,5574,8099],[8102,0.431,370,8100,8099],[8102,0.495,370,8101,8099],[587,0.45,370,5574,8099],[587,0.445,370,8100,8099],[587,0.457,370,8101,8099],[8103,0.417,370,5574,8099],[8103,0.403,370,8100,8099],[8103,0.424,370,8101,8099],[8104,0.389,370,5574,8099],[8104,0.38,370,8100,8099],[8104,0.4,370,8101,8099]]},{"id":"tic-overall-2026-paper","name":"TIC-Bench (2026 paper)","group":"视觉与视频","metric":"Overall","unit":"%","score_max":100,"higher_is_better":true,"public_history":true,"protocol":{"en":"Reasons over interleaved text and images using logical, temporal and spatial associations. The overall accuracy covers the full multimodal context; thinking settings remain distinct.","zh":"在交错的文本和图像中推理逻辑、时间及空间关系。采用完整多模态上下文的总准确率，并保留不同思考设置。"},"source":{"name":"TIC-Bench authors","status":"paper_baselines","url":"https://arxiv.org/html/2609.02573v1#Sx3.T2","published_at":"2026-09-02"},"model_count":10,"result_count":15,"results":[[5480,0.405,370,8105,8106],[5480,0.43,370,8107,8106],[5722,0.337,370,8105,8106],[5722,0.368,370,8107,8106],[3764,0.402,370,8105,8106],[3764,0.47,370,8107,8106],[3187,0.448,370,8105,8106],[3187,0.474,370,8107,8106],[7223,0.38,370,8105,8106],[7223,0.43,370,8107,8106],[3572,0.474,370,8108,8106],[3482,0.59,370,8108,8106],[3196,0.599,370,8108,8106],[7674,0.503,370,8108,8106],[7937,0.407,370,8108,8106]]},{"id":"dochop-overall-2026-paper","name":"DocHop (2026 paper)","group":"视觉与视频","metric":"Overall","unit":"%","score_max":100,"higher_is_better":true,"public_history":true,"protocol":{"en":"Answers questions that require combining document narratives with charts. Overall accuracy covers retrieval, counting, numerical reasoning, ranking, hypothetical reasoning and fact checking.","zh":"结合文档叙述与图表回答问题。总准确率覆盖信息检索、计数、数值推理、排名、假设推理及事实核验。"},"source":{"name":"DocHop authors","status":"paper_baselines","url":"https://arxiv.org/html/2609.02059v1#S3.T1","published_at":"2026-09-02"},"model_count":19,"result_count":21,"results":[[340,0.6283,370,8107,8109],[4064,0.2825,370,8107,8109],[764,0.406,370,8107,8109],[766,0.32020000000000004,370,8107,8109],[340,0.4036,370,8108,8109],[766,0.2488,370,8108,8109],[7695,0.1635,370,8108,8109],[7685,0.1794,370,8108,8109],[8110,0.1133,370,8108,8109],[8111,0.08869999999999999,370,8108,8109],[8112,0.1066,370,8108,8109],[2195,0.1905,370,8108,8109],[6342,0.2416,370,8108,8109],[8113,0.1673,370,8108,8109],[8114,0.16010000000000002,370,8108,8109],[2505,0.1056,370,8108,8109],[7993,0.13449999999999998,370,8108,8109],[8115,0.1475,370,8108,8109],[8116,0.21789999999999998,370,8108,8109],[8117,0.1557,370,8108,8109],[8118,0.244,370,8108,8109]]},{"id":"scidocbench-overall-2026-paper","name":"SciDocBench (2026 paper)","group":"专业工作","metric":"Overall","unit":"score","score_max":100,"higher_is_better":true,"public_history":true,"protocol":{"en":"Works with scientific documents across perception, extraction, evidence verification and executable tasks. The native mean score covers all 496 instances from 124 questions; failed or unusable responses score zero.","zh":"围绕科学文档完成识别、提取、证据核验及可执行任务。原生平均分覆盖124个问题衍生的全部496条实例；失败或不可用回答记零。"},"source":{"name":"SciDocBench authors","status":"paper_baselines","url":"https://arxiv.org/html/2609.05141v1#S5.T2","published_at":"2026-09-04"},"model_count":14,"result_count":14,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"results":[[4983,62.6,370,8108,8119],[3194,53.41,370,8108,8119],[6978,49.57,370,8108,8119],[6214,40.35,370,8108,8119],[4979,61.0,370,8108,8119],[5727,54.19,370,8108,8119],[5616,50.58,370,8108,8119],[5136,59.87,370,8108,8119],[3157,57.15,370,8108,8119],[4961,55.88,370,8108,8119],[5300,52.89,370,8108,8119],[2998,50.38,370,8108,8119],[5722,42.44,370,8108,8119],[7223,40.16,370,8108,8119]]},{"id":"m261018299-2026-paper","name":"LegalRewardBench (2026 paper)","group":"专业工作","metric":"Pairwise preference accuracy","unit":"%","protocol":{"en":"Human-verified legal response preferences across five capability levels. Generative judges evaluate both response orders and average them; scalar reward models compare scores. Table 3 Overall; model type and thinking flag retained.","zh":"五个能力层次的人工核验法律回答偏好。生成式裁判交换回答次序后取平均；奖励模型比较分值。取表 3 Overall，保留模型类型和思考标志。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.18299","status":"paper_baselines","published_at":"2026-10-13"},"public_history":true,"model_count":38,"result_count":38,"results":[[3181,0.8684999999999999,370,8120,8121],[6705,0.8501000000000001,370,8120,8121],[3196,0.8366,370,8120,8121],[7611,0.8295,370,8120,8121],[784,0.8151999999999999,370,8120,8121],[764,0.8114,370,8122,8121],[2998,0.8065000000000001,370,8120,8121],[340,0.8054000000000001,370,8120,8121],[782,0.799,370,8120,8121],[793,0.7929999999999999,370,8120,8121],[345,0.7896,370,8122,8121],[2576,0.7855,370,8120,8121],[766,0.7851,370,8120,8121],[769,0.7829,370,8120,8121],[8123,0.7825,370,8124,8121],[8125,0.7814,370,8126,8121],[6926,0.7784,370,8120,8121],[6301,0.7784,370,8120,8121],[5344,0.7569,370,8120,8121],[2584,0.7569,370,8120,8121],[6863,0.7551000000000001,370,8122,8121],[5126,0.7487,370,8120,8121],[7721,0.7468,370,8122,8121],[353,0.7461,370,8122,8121],[4819,0.7408,370,8120,8121],[5535,0.7234999999999999,370,8122,8121],[8127,0.7137,370,8126,8121],[8128,0.7084999999999999,370,8126,8121],[8129,0.6920000000000001,370,8124,8121],[8130,0.6890000000000001,370,8126,8121],[8131,0.6822,370,8124,8121],[7590,0.6766,370,8122,8121],[562,0.6736,370,8122,8121],[8132,0.6701999999999999,370,8126,8121],[1748,0.6578,370,8122,8121],[8133,0.6258,370,8126,8121],[1804,0.562,370,8122,8121],[5660,0.5540999999999999,370,8120,8121]]},{"id":"m261040131-2026-paper","name":"VideoMemBench (Overall, 2026 paper)","group":"视觉与视频","metric":"Overall","unit":"%","protocol":{"en":"Overall column under the self-rollout protocol. Turn-level and dependency-conditioned accuracies are not separate rankings.","zh":"排名采用自回归协议下的总分列。分轮准确率和依赖条件下的准确率不另排名。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.40131v1","status":"paper_baselines","published_at":"2026-10-29"},"model_count":25,"result_count":25,"public_history":true,"results":[[755,0.457,370,8134,8135],[2983,0.424,370,8134,8135],[4048,0.41700000000000004,370,8134,8135],[3415,0.41600000000000004,370,8134,8135],[3196,0.405,370,8134,8135],[3568,0.402,370,8134,8135],[3482,0.391,370,8134,8135],[3184,0.363,370,8134,8135],[2977,0.42200000000000004,370,8134,8135],[5843,0.344,370,8134,8135],[2584,0.34299999999999997,370,8134,8135],[2576,0.342,370,8134,8135],[2582,0.33899999999999997,370,8134,8135],[2590,0.308,370,8134,8135],[6343,0.308,370,8134,8135],[6238,0.29100000000000004,370,8134,8135],[4070,0.27,370,8134,8135],[6342,0.264,370,8134,8135],[4969,0.263,370,8134,8135],[6298,0.249,370,8134,8135],[6391,0.24600000000000002,370,8134,8135],[2754,0.198,370,8134,8135],[8136,0.19399999999999998,370,8134,8135],[2771,0.179,370,8134,8135],[6340,0.162,370,8134,8135]]},{"id":"m261039639-2026-paper","name":"BedBench (Average accuracy, 2026 paper)","group":"知识与推理","metric":"Avg","unit":"%","protocol":{"en":"Average accuracy across the thirteen default tasks. NT and TH are inference-mode configurations of the same model. Human, random and constant baselines are omitted.","zh":"十三项默认任务的平均准确率。NT 和 TH 是同一模型的推理模式。人类、随机和常数基线不计入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.39639v1","status":"paper_baselines","published_at":"2026-10-29"},"model_count":16,"result_count":28,"public_history":true,"results":[[6340,0.312,370,8137,8138],[6340,0.314,370,8139,8138],[5828,0.332,370,8137,8138],[5828,0.33299999999999996,370,8139,8138],[4969,0.397,370,8137,8138],[4969,0.402,370,8139,8138],[4070,0.40700000000000003,370,8137,8138],[4070,0.434,370,8139,8138],[2584,0.46799999999999997,370,8137,8138],[2584,0.486,370,8139,8138],[2590,0.431,370,8137,8138],[2590,0.455,370,8139,8138],[8140,0.345,370,8137,8138],[8140,0.35600000000000004,370,8139,8138],[8141,0.382,370,8137,8138],[8141,0.385,370,8139,8138],[8142,0.385,370,8137,8138],[8142,0.415,370,8139,8138],[8143,0.465,370,8137,8138],[8143,0.441,370,8139,8138],[6241,0.392,370,8137,8138],[6241,0.455,370,8139,8138],[6346,0.301,370,8137,8138],[6346,0.248,370,8139,8138],[2058,0.449,370,8144,8138],[8145,0.418,370,8144,8138],[502,0.469,370,8144,8138],[6133,0.515,370,8144,8138]]},{"id":"m261031314-2026-paper","name":"State Integrity (Overall, 2026 paper)","group":"知识与推理","metric":"Overall","unit":"%","protocol":{"en":"Overall column of the main retention table. A printed 100 is an observed cell, not a missing run. Turn-level and drop columns are not separate rankings.","zh":"排名采用主表的总体保持率。写明的 100 分是实际格子，不是缺失运行。分轮和下降列不另排名。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.31314v1","status":"paper_baselines","published_at":"2026-10-22"},"model_count":7,"result_count":7,"public_history":true,"results":[[4963,1.0,370,6175,8146],[4958,0.9840000000000001,370,6175,8146],[4962,0.96,370,6175,8146],[3189,0.9520000000000001,370,6175,8146],[3184,0.9440000000000001,370,6175,8146],[7749,0.904,370,6175,8146],[502,0.624,370,6175,8146]]},{"id":"m261028316-2026-paper","name":"ABACUS (Composite score, 2026 paper)","group":"视觉与视频","metric":"Composite score","unit":"score","protocol":{"en":"Published product of the three scene-conservation factors, on the 0–1 scale printed in the table. It is not a percentage accuracy. Published rubric/composite on its native scale; excluded from accuracy saturation.","zh":"表中三项场景保持因子的乘积，刻度是原文的 0–1，不是百分制准确率。 保留原生评分或综合分，不计入准确率饱和统计。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.28316v1","status":"paper_baselines","published_at":"2026-10-20"},"score_max":1,"score_decimals":2,"saturation_excluded":true,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":7,"result_count":7,"public_history":true,"results":[[8147,0.29,370,6175,8148],[8149,0.31,370,6175,8148],[6841,0.26,370,6175,8148],[8150,0.16,370,6175,8148],[8151,0.14,370,6175,8148],[8152,0.11,370,6175,8148],[6358,0.09,370,6175,8148]]},{"id":"m261102826-2026-paper","name":"RxnMiner (Overall, 2026 paper)","group":"数学与科学","metric":"Overall","unit":"score","protocol":{"en":"Overall judge percentage. Direct prediction and RxnMiner are configurations of the same listed models. ChemEAGLE is the existing extraction system in that table. Published rubric/composite on its native scale; excluded from accuracy saturation.","zh":"裁判总分百分比。直接预测和 RxnMiner 是同一批模型的两种配置。ChemEAGLE 是该表里的既有抽取系统。 保留原生评分或综合分，不计入准确率饱和统计。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.02826v1","status":"paper_baselines","published_at":"2026-11-02"},"score_max":100,"score_decimals":2,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":9,"result_count":16,"public_history":true,"results":[[755,88.32,370,8153,8154],[4979,81.11,370,8153,8154],[3412,77.04,370,8153,8154],[6438,71.31,370,8153,8154],[3512,36.5,370,8153,8154],[3529,67.85,370,8153,8154],[2977,82.36,370,8153,8154],[5361,81.09,370,8153,8154],[755,96.94,370,8155,8154],[4979,82.27,370,8155,8154],[6438,93.64,370,8155,8154],[3512,89.08,370,8155,8154],[3529,37.14,370,8155,8154],[2977,92.81,370,8155,8154],[5361,80.23,370,8155,8154],[8156,53.8,370,8157,8154]]},{"id":"m261035031-2026-paper","name":"PatentAgentBench (Average, 2026 paper)","group":"专业工作","metric":"AVG","unit":"score","protocol":{"en":"Task average on the 0–100 scale. Codex, Claude Code, Hermes and OpenClaw are harness configurations, not separate models. Published rubric/composite on its native scale; excluded from accuracy saturation.","zh":"0–100 刻度上的任务平均分。Codex、Claude Code、Hermes 和 OpenClaw 是 harness 配置，不是不同模型。 保留原生评分或综合分，不计入准确率饱和统计。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.35031v1","status":"paper_baselines","published_at":"2026-10-26"},"score_max":100,"score_decimals":2,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":7,"result_count":27,"public_history":true,"results":[[4979,43.8,370,6405,8158],[3157,38.3,370,6405,8158],[2977,27.1,370,6405,8158],[3185,26.0,370,6405,8158],[3198,17.5,370,6405,8158],[3154,48.1,370,6405,8158],[4979,49.3,370,6457,8158],[755,50.3,370,6457,8158],[3157,45.2,370,6457,8158],[2977,42.6,370,6457,8158],[3185,39.7,370,6457,8158],[3198,34.2,370,6457,8158],[3154,40.7,370,6457,8158],[4979,41.1,370,8159,8158],[755,52.8,370,8159,8158],[3157,28.9,370,8159,8158],[2977,40.0,370,8159,8158],[3185,33.0,370,8159,8158],[3198,25.2,370,8159,8158],[3154,41.8,370,8159,8158],[4979,50.0,370,8160,8158],[755,31.2,370,8160,8158],[3157,36.6,370,8160,8158],[2977,36.2,370,8160,8158],[3185,31.1,370,8160,8158],[3198,32.8,370,8160,8158],[3154,29.5,370,8160,8158]]},{"id":"m261108369-2026-paper","name":"NPCBench (Consultation score, 2026 paper)","group":"医疗","metric":"Consultation score","unit":"score","protocol":{"en":"Consultation score from the cited table. Every listed model uses the same 103 questions. Error-rate columns are not ranked separately. Published rubric/composite on its native scale; excluded from accuracy saturation.","zh":"排名采用所引表的问诊得分。列出的模型都答同一套 103 题。误答率列不另排名。 保留原生评分或综合分，不计入准确率饱和统计。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.08369v1","status":"paper_baselines","published_at":"2026-11-05"},"score_max":100,"saturation_excluded":true,"score_decimals":2,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":30,"result_count":30,"public_history":true,"results":[[3748,58.4,370,6175,8161],[8162,47.8,370,6175,8161],[8163,47.5,370,6175,8161],[3763,47.2,370,6175,8161],[2584,46.8,370,6175,8161],[3419,44.9,370,6175,8161],[6771,42.4,370,6175,8161],[5619,41.4,370,6175,8161],[5348,38.8,370,6175,8161],[764,38.5,370,6175,8161],[4070,34.7,370,6175,8161],[766,34.4,370,6175,8161],[345,33.3,370,6175,8161],[6299,29.1,370,6175,8161],[5346,28.6,370,6175,8161],[6292,28.2,370,6175,8161],[5343,28.0,370,6175,8161],[8164,26.8,370,6175,8161],[6342,25.6,370,6175,8161],[5352,24.8,370,6175,8161],[8165,18.8,370,6175,8161],[8166,14.7,370,6175,8161],[8167,10.7,370,6175,8161],[8168,10.1,370,6175,8161],[2642,9.5,370,6175,8161],[8169,9.0,370,6175,8161],[8170,8.7,370,6175,8161],[8171,7.6,370,6175,8161],[8172,5.3,370,6175,8161],[8173,3.2,370,6175,8161]]},{"id":"m261106911-2026-paper","name":"BEAKER (Overall, 2026 paper)","group":"数学与科学","metric":"Overall","unit":"score","protocol":{"en":"Four capability dimensions weighted 0.25/0.15/0.35/0.25; combines MCQ, IoU, trajectory and judged scores on 0–100. Table 12 all 28 checkpoints.","zh":"四项能力按 0.25/0.15/0.35/0.25 加权，综合选择题、IoU、轨迹及裁判评分。取表 12 全部 28 个检查点的 0–100 综合分。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.06911v1","status":"paper_baselines","published_at":"2026-11-04"},"score_max":100,"score_decimals":2,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":28,"result_count":28,"public_history":true,"results":[[8174,28.4,370,6175,8175],[8176,30.1,370,6175,8175],[6676,40.5,370,6175,8175],[7864,35.2,370,6175,8175],[7853,44.0,370,6175,8175],[8177,44.7,370,6175,8175],[8178,35.2,370,6175,8175],[7558,45.3,370,6175,8175],[8179,39.2,370,6175,8175],[8180,42.9,370,6175,8175],[8181,37.4,370,6175,8175],[7556,39.3,370,6175,8175],[3196,55.4,370,6175,8175],[3184,57.2,370,6175,8175],[3578,59.2,370,6175,8175],[3572,52.1,370,6175,8175],[2602,40.8,370,6175,8175],[3482,55.5,370,6175,8175],[8182,44.7,370,6175,8175],[8183,52.9,370,6175,8175],[8184,48.8,370,6175,8175],[3189,50.8,370,6175,8175],[2998,52.2,370,6175,8175],[3187,52.0,370,6175,8175],[7223,48.2,370,6175,8175],[6342,47.1,370,6175,8175],[7153,41.8,370,6175,8175],[2576,54.0,370,6175,8175]]},{"id":"m261032459-2026-paper","name":"MDMR-Bench (Overall, 2026 paper)","group":"视觉与视频","metric":"Overall","unit":"score","protocol":{"en":"Overall column of the cited multi-reference generation table. Published rubric/composite on its native scale; excluded from accuracy saturation.","zh":"排名采用所引多参考图生成表的总分列。 保留原生评分或综合分，不计入准确率饱和统计。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.32459v1","status":"paper_baselines","published_at":"2026-10-23"},"score_max":100,"saturation_excluded":true,"score_decimals":2,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":11,"result_count":11,"public_history":true,"results":[[6368,88.8,370,6175,8185],[6366,85.1,370,6175,8185],[6839,81.2,370,6175,8185],[8186,78.8,370,6175,8185],[8149,77.8,370,6175,8185],[6367,77.5,370,6175,8185],[6360,51.3,370,6175,8185],[8150,49.9,370,6175,8185],[6358,41.9,370,6175,8185],[6363,41.9,370,6175,8185],[7295,28.6,370,6175,8185]]},{"id":"m261036554-2026-paper","name":"PRISM-Bench (Overall accuracy, 2026 paper)","group":"视觉与视频","metric":"Overall accuracy","unit":"%","protocol":{"en":"Overall accuracy across the five PRISM-Bench categories. The human baseline row is omitted.","zh":"排名采用 PRISM-Bench 五个类别的总准确率。人类基线行不计入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.36554v1","status":"paper_baselines","published_at":"2026-10-27"},"model_count":25,"result_count":25,"public_history":true,"results":[[2602,0.66,370,6175,8187],[2058,0.636,370,6175,8187],[2261,0.624,370,6175,8187],[8188,0.5720000000000001,370,6175,8187],[323,0.396,370,6175,8187],[7939,0.425,370,6175,8187],[8189,0.4,370,6175,8187],[8190,0.34600000000000003,370,6175,8187],[2654,0.302,370,6175,8187],[7486,0.29,370,6175,8187],[8191,0.27399999999999997,370,6175,8187],[8192,0.267,370,6175,8187],[8193,0.221,370,6175,8187],[8194,0.207,370,6175,8187],[8114,0.201,370,6175,8187],[2652,0.386,370,6175,8187],[7940,0.368,370,6175,8187],[2659,0.36700000000000005,370,6175,8187],[2646,0.361,370,6175,8187],[8195,0.36,370,6175,8187],[2490,0.29100000000000004,370,6175,8187],[8196,0.29,370,6175,8187],[2696,0.284,370,6175,8187],[8197,0.262,370,6175,8187],[8198,0.23800000000000002,370,6175,8187]]},{"id":"m261030103-2026-paper","name":"P2NPCBench · polynomial-time subset (2026 paper)","group":"Agent 与工具","metric":"Case pass rate","unit":"%","protocol":{"en":"Compiled programs are tested on the p subset; incorrect answers, timeouts and crashes are observed program failures.","zh":"编译程序在 p 子集接受测试；答案错误、超时和崩溃属于实测程序失败。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.30103v1","status":"paper_baselines","published_at":"2026-10-21"},"public_history":true,"model_count":10,"result_count":10,"results":[[5519,0.96,370,7834,8199],[5899,0.863,370,7834,8199],[6725,0.8440000000000001,370,7834,8199],[8200,0.74,370,7834,8199],[3153,0.868,370,7834,8199],[6668,0.221,370,7834,8199],[6689,0.221,370,7834,8199],[1702,0.49200000000000005,370,7834,8199],[5300,0.878,370,7834,8199],[6677,0.162,370,7834,8199]]},{"id":"m261020344-2026-paper","name":"Doc2Harness (2026 paper)","group":"Agent 与工具","metric":"Harness test pass rate","unit":"%","protocol":{"en":"Nineteen models, three scored harness builds per model. Mean executable test pass rate; architecture columns are diagnostics.","zh":"19 个模型，各三次已评分的框架构建。取可执行测试通过率均值，代码结构列为诊断指标。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.20344v1","status":"paper_baselines","published_at":"2026-10-14"},"public_history":true,"model_count":19,"result_count":19,"results":[[2958,0.5305,370,7834,8201],[3156,0.46649999999999997,370,7834,8201],[2982,0.4583,370,7834,8201],[4048,0.43799999999999994,370,7834,8201],[2968,0.435,370,7834,8201],[3515,0.4207,370,7834,8201],[7644,0.41869999999999996,370,7834,8201],[3534,0.4075,370,7834,8201],[3512,0.40240000000000004,370,7834,8201],[4029,0.3892,370,7834,8201],[2977,0.3831,370,7834,8201],[3401,0.3801,370,7834,8201],[3196,0.3465,370,7834,8201],[5361,0.3262,370,7834,8201],[3400,0.3222,370,7834,8201],[3492,0.3191,370,7834,8201],[3406,0.2825,370,7834,8201],[3654,0.2764,370,7834,8201],[5114,0.27440000000000003,370,7834,8201]]},{"id":"m261019399-2026-paper","name":"KoAGE · A2 (2026 paper)","group":"知识与推理","metric":"A2 accuracy on all items","unit":"%","protocol":{"en":"Table 19 all-item A2 column; conditional acquisition scores and error-type shares remain separate. Group mean rows excluded.","zh":"表 19 全部项目的 A2 准确率；条件分组和错误类型份额另列，不纳入此榜。排除组均值行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.19399v1","status":"paper_baselines","published_at":"2026-10-14"},"public_history":true,"model_count":21,"result_count":21,"results":[[4920,0.6609999999999999,370,7834,8202],[353,0.894,370,7834,8202],[2058,0.9670000000000001,370,7834,8202],[5449,0.892,370,7834,8202],[3160,0.9790000000000001,370,7834,8202],[283,0.907,370,7834,8202],[3568,0.927,370,7834,8202],[3424,0.981,370,7834,8202],[4035,0.9490000000000001,370,7834,8202],[8203,0.888,370,7834,8202],[8204,0.924,370,7834,8202],[8205,0.8759999999999999,370,7834,8202],[8206,0.893,370,7834,8202],[8207,0.7090000000000001,370,7834,8202],[7265,0.723,370,7834,8202],[6268,0.753,370,7834,8202],[6013,0.815,370,7834,8202],[8208,0.698,370,7834,8202],[8209,0.755,370,7834,8202],[8210,0.8029999999999999,370,7834,8202],[8211,0.795,370,7834,8202]]},{"id":"m261037166-2026-paper","name":"LiveSearch (2026 paper)","group":"专业工作","metric":"Search-only answer accuracy","unit":"%","protocol":{"en":"Six tool-calling models under search only, without specialist finance tools. Reasoning effort is retained in configurations.","zh":"六个模型仅使用搜索，不提供专门金融工具；推理档位保留为配置。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.37166v1","status":"paper_baselines","published_at":"2026-10-27"},"public_history":true,"model_count":6,"result_count":6,"results":[[3512,0.767,370,8212,8213],[755,0.762,370,8214,8213],[3534,0.745,370,8215,8213],[2958,0.745,370,8214,8213],[3154,0.733,370,8214,8213],[2977,0.7090000000000001,370,8214,8213]]},{"id":"m261028405-2026-paper","name":"DIVE (2026 paper)","group":"视觉与视频","metric":"Mean window response rate","unit":"%","protocol":{"en":"Unweighted mean over six streaming-interaction tasks. Native audiovisual and timestamp-aligned visual inputs are explicit configurations.","zh":"六项流式交互任务的窗口响应率非加权均值。原生音视频和时间对齐文本的视觉输入分别注明配置。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.28405v1","status":"paper_baselines","published_at":"2026-10-20"},"public_history":true,"model_count":11,"result_count":11,"results":[[8216,0.2052,370,8217,8218],[6574,0.1495,370,8217,8218],[8219,0.1743,370,8217,8218],[6139,0.16649999999999998,370,8217,8218],[6659,0.5377000000000001,370,8217,8218],[8220,0.4058,370,8217,8218],[8221,0.4143,370,8217,8218],[8222,0.16449999999999998,370,8223,8218],[8224,0.252,370,8223,8218],[6660,0.2357,370,8223,8218],[7225,0.1558,370,8223,8218]]},{"id":"m261018507-2026-paper","name":"TrialPD-Bench · single-page schedule (2026 paper)","group":"编程","metric":"Activity-Visit F1","unit":"score","protocol":{"en":"86 single-page schedule-of-activities inputs. Table 17 Activity-Visit F1, including coding-agent and OCR/document baselines with their input method labeled.","zh":"86 页单页活动日程输入。取表 17 活动与访视关系 F1，完整保留编码智能体及 OCR/文档基线，并注明输入方法。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.18507v1","status":"paper_baselines","published_at":"2026-10-13"},"public_history":true,"score_max":100,"score_decimals":2,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":18,"result_count":18,"results":[[353,0.5,370,8225,8226],[4979,88.2,370,8225,8226],[4977,85.1,370,8225,8226],[3406,52.4,370,8225,8226],[3401,86.3,370,8225,8226],[3156,90.0,370,8225,8226],[3393,84.3,370,8225,8226],[3155,85.7,370,8225,8226],[3201,38.7,370,8225,8226],[5300,57.3,370,8225,8226],[8227,71.7,370,8228,8226],[8229,63.6,370,8228,8226],[8230,61.1,370,8228,8226],[8231,35.1,370,8228,8226],[8232,7.1,370,8228,8226],[8233,19.3,370,8228,8226],[6341,9.4,370,8228,8226],[6298,37.7,370,8228,8226]]},{"id":"m261012961-2026-paper","name":"MVGraph · graph reasoning (2026 paper)","group":"视觉与视频","metric":"Eight-task mean accuracy","unit":"%","protocol":{"en":"Table 14 mean over eight end-to-end graph-theoretic tasks. Thinking and no-thinking are configurations of the same checkpoint.","zh":"表 14 八项端到端图论任务的平均准确率。思考与非思考模式记为同一检查点的配置。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.12961v1","status":"paper_baselines","published_at":"2026-10-09"},"public_history":true,"model_count":15,"result_count":21,"results":[[6240,0.055999999999999994,370,8234,8235],[4070,0.052000000000000005,370,8234,8235],[2646,0.052000000000000005,370,8234,8235],[6342,0.054000000000000006,370,8234,8235],[6241,0.052000000000000005,370,8234,8235],[5725,0.057,370,8234,8235],[6630,0.040999999999999995,370,8234,8235],[2515,0.019,370,8234,8235],[2654,0.054000000000000006,370,8234,8235],[7485,0.053,370,8234,8235],[8236,0.054000000000000006,370,8234,8235],[8237,0.051,370,8234,8235],[5729,0.064,370,8234,8235],[5899,0.026000000000000002,370,8234,8235],[4070,0.135,370,3143,8235],[6240,0.205,370,3143,8235],[6241,0.07400000000000001,370,3143,8235],[2646,0.078,370,3143,8235],[5899,0.504,370,3143,8235],[3396,0.524,370,3143,8235],[5729,0.174,370,3143,8235]]},{"id":"m261019560-2026-paper","name":"Tau-Emotion-Bench (2026 paper)","group":"Agent 与工具","metric":"Mean Pass^3","unit":"%","protocol":{"en":"Mean Pass^3 over ten domains: successful in all three attempts. Thinking mode and named Tracer fine-tuned checkpoints are retained.","zh":"十个领域的 Pass^3 均值，即三次尝试均成功。保留思考模式与具名 Tracer 微调检查点。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.19560v1","status":"paper_baselines","published_at":"2026-10-14"},"public_history":true,"model_count":17,"result_count":18,"results":[[786,0.094,370,8238,8239],[784,0.138,370,8238,8239],[3153,0.322,370,8238,8239],[786,0.086,370,8240,8239],[7590,0.012,370,8240,8239],[5126,0.039,370,8240,8239],[5344,0.084,370,8240,8239],[8241,0.165,370,8240,8239],[5300,0.273,370,8240,8239],[3198,0.17600000000000002,370,8240,8239],[5479,0.205,370,8240,8239],[3190,0.168,370,8240,8239],[6143,0.247,370,8240,8239],[3154,0.317,370,8240,8239],[3203,0.261,370,8240,8239],[8242,0.10400000000000001,370,8240,8239],[8243,0.172,370,8240,8239],[8244,0.198,370,8240,8239]]},{"id":"m261029014-2026-paper","name":"GenXR-Bench (2026 paper)","group":"Agent 与工具","metric":"Task Pass@1","unit":"%","protocol":{"en":"Table 3 overall task success across repair, SDK-feature and application tasks. Design ratings and efficiency are separate diagnostics.","zh":"表 3 修复、SDK 功能和应用任务的总体成功率。设计评分及效率为独立诊断指标。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.29014v1","status":"paper_baselines","published_at":"2026-10-21"},"public_history":true,"model_count":10,"result_count":10,"results":[[5142,0.509,370,7834,8245],[4955,0.46399999999999997,370,7834,8245],[4972,0.446,370,7834,8245],[5144,0.429,370,7834,8245],[6035,0.41100000000000003,370,7834,8245],[5475,0.39299999999999996,370,7834,8245],[5143,0.312,370,7834,8245],[5155,0.304,370,7834,8245],[5109,0.295,370,7834,8245],[413,0.23199999999999998,370,7834,8245]]},{"id":"m261026342-2026-paper","name":"VibeTraining-Bench (2026 paper)","group":"知识与推理","metric":"Mean held-out task performance","unit":"score","protocol":{"en":"Six research agents optimize the same Qwen3.5-9B base system. Mean over five fixed held-out benchmark panels; initial and human-tuned controls excluded.","zh":"六个研究智能体优化同一 Qwen3.5-9B 基础系统，取五组固定留出测试的均值，排除初始系统和人工微调对照。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.26342v1","status":"paper_baselines","published_at":"2026-10-19"},"public_history":true,"score_max":100,"score_decimals":2,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":6,"result_count":6,"results":[[4977,37.1,370,8246,8247],[4979,31.0,370,8246,8247],[3155,38.3,370,8248,8247],[227,31.4,370,8248,8247],[5133,35.6,370,8248,8247],[3185,34.2,370,8248,8247]]},{"id":"m261107541-2026-paper","name":"CST-Bench (2026 paper)","group":"知识与推理","metric":"Six-family macro accuracy","unit":"%","public_history":true,"protocol":{"en":"Balanced 480-example split, same answer-only prompt. Macro mean over six intervention families; ten checkpoints. Invalid generated answers count incorrect.","zh":"均衡的 480 题测试，采用相同的仅作答提示。取六类空间干预的宏平均准确率，十个模型检查点。无效生成答案计错。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.07541","status":"paper_baselines","published_at":"2026-11-05"},"model_count":10,"result_count":10,"results":[[2771,0.163,370,7834,8249],[6298,0.445,370,7834,8249],[2754,0.231,370,7834,8249],[6389,0.418,370,7834,8249],[2687,0.374,370,7834,8249],[8250,0.373,370,7834,8249],[7486,0.45399999999999996,370,7834,8249],[8251,0.358,370,7834,8249],[8252,0.292,370,7834,8249],[8253,0.414,370,7834,8249]]},{"id":"m261102519-2026-paper","name":"MedHEB (2026 paper)","group":"医疗","metric":"NDCG@5","unit":"score","public_history":true,"protocol":{"en":"Graded-relevance retrieval; Table 5 All column across severity, laterality, diagnosis and cell/tissue labels. Ten embedding checkpoints.","zh":"分级相关性检索；取表 5 严重程度、侧别、诊断和细胞组织标签的 All 列，十个嵌入模型。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.02519","status":"paper_baselines","published_at":"2026-11-02"},"score_max":100,"score_decimals":2,"saturation_excluded":true,"higher_is_better":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":10,"result_count":10,"results":[[8254,47.47,370,7834,8255],[8256,50.72,370,7834,8255],[8257,51.55,370,7834,8255],[8258,51.85,370,7834,8255],[8259,52.23,370,7834,8255],[8260,56.9,370,7834,8255],[8261,59.14,370,7834,8255],[8262,59.5,370,7834,8255],[8263,82.62,370,7834,8255],[8264,83.34,370,7834,8255]]},{"id":"m261104687-2026-paper","name":"CardioVascBench (2026 paper)","group":"医疗","metric":"Final answer score","unit":"score","public_history":true,"protocol":{"en":"Complete Appendix Table 5, 36 reported configurations. Final score stays on the authors’ 0–100 scale; specialist and agent variants are identified in configurations.","zh":"完整附录表 5 的 36 条配置。保留作者 0–100 最终评分，并注明专科与智能体变体。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.04687","status":"paper_baselines","published_at":"2026-11-03"},"score_max":100,"score_decimals":2,"saturation_excluded":true,"higher_is_better":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":33,"result_count":36,"results":[[353,14.9,370,8265,8266],[340,14.8,370,8265,8266],[3250,16.1,370,8265,8266],[8267,15.1,370,8265,8266],[7931,16.7,370,8265,8266],[8268,16.2,370,8269,8266],[8270,17.1,370,8269,8266],[7579,11.4,370,8269,8266],[8172,25.4,370,8269,8266],[8173,28.2,370,8269,8266],[1748,26.8,370,8265,8266],[2471,25.7,370,8265,8266],[8271,15.2,370,8265,8266],[8272,13.3,370,8269,8266],[5344,22.9,370,8265,8266],[8273,9.8,370,8269,8266],[8274,28.0,370,8269,8266],[8275,19.1,370,8269,8266],[8173,44.3,370,8276,8266],[8172,46.4,370,8276,8266],[8277,13.2,370,8269,8266],[6301,7.2,370,8265,8266],[8278,14.7,370,8265,8266],[8279,17.9,370,8269,8266],[8169,17.2,370,8269,8266],[8280,12.7,370,8265,8266],[8281,25.6,370,8269,8266],[8282,13.0,370,8269,8266],[8168,19.5,370,8265,8266],[6375,16.2,370,8265,8266],[8283,16.2,370,8265,8266],[8284,16.1,370,8265,8266],[2278,14.4,370,8265,8266],[8285,17.2,370,8265,8266],[4819,21.4,370,8265,8266],[8281,46.7,370,8276,8266]]},{"id":"m261104228-2026-paper","name":"DualWorldBench · VQA (2026 paper)","group":"Agent 与工具","metric":"Visual-question accuracy","unit":"%","public_history":true,"protocol":{"en":"DualWorld-VQA only, all six models over the same axis, anchor, pair, topology and planning questions. Stored from source 0–1 fractions.","zh":"仅 DualWorld-VQA，六个模型评估同一组轴、锚点、配对、拓扑与规划问题。保留来源中的 0–1 比例。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.04228","status":"paper_baselines","published_at":"2026-11-03"},"model_count":6,"result_count":6,"results":[[6342,0.335,370,7834,8286],[4961,0.843,370,7834,8286],[325,0.227,370,7834,8286],[755,0.799,370,7834,8286],[3494,0.392,370,7834,8286],[4979,0.864,370,7834,8286]]},{"id":"m261031650-2026-paper","name":"EpiSEM-Bench · critical omission (2026 paper)","group":"Agent 与工具","metric":"Critical omission failure rate","unit":"score","public_history":true,"protocol":{"en":"300 matched task pairs; mean unsafe-execution fraction over retained completed critical-omission conditions, weighted equally by task. Ten models; lower is better.","zh":"300 对匹配任务，在保留且已完成的关键证据省略条件中计算不安全执行比例，再对任务等权平均。十个模型，越低越好。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.31650","status":"paper_baselines","published_at":"2026-10-22"},"score_max":100,"score_decimals":2,"saturation_excluded":true,"higher_is_better":false,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":10,"result_count":10,"results":[[5617,7.9,370,7834,8287],[6415,15.7,370,7834,8287],[3153,18.5,370,7834,8287],[3185,31.6,370,7834,8287],[8288,47.9,370,7834,8287],[7643,61.7,370,7834,8287],[5300,27.0,370,7834,8287],[6301,84.1,370,7834,8287],[6374,69.5,370,7834,8287],[4966,76.0,370,7834,8287]]},{"id":"m261100791-2026-paper","name":"PhoneValue-Bench · Direct-VLM (2026 paper)","group":"知识与推理","metric":"Tier-B weighted score","unit":"score","public_history":true,"protocol":{"en":"All fourteen Direct-VLM models on the same 335 orders. Shared Tier-B item gates; Score100=(51*Yfill+22*Sfill)/73. ValueAgent ablations, modality controls and summary rows are excluded.","zh":"全部十四个 Direct-VLM 模型评估同一批 335 个订单，使用共同 Tier-B 评分项目规则。Score100=(51*Yfill+22*Sfill)/73。剔除 ValueAgent 消融、模态对照与汇总行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.00791","status":"paper_baselines","published_at":"2026-10-31"},"score_max":100,"score_decimals":2,"saturation_excluded":true,"higher_is_better":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":14,"result_count":14,"results":[[764,69.9,370,8289,8290],[2166,68.0,370,8289,8290],[353,67.9,370,8289,8290],[8291,67.5,370,8289,8290],[6863,66.2,370,8289,8290],[766,63.9,370,8289,8290],[2675,63.7,370,8289,8290],[8292,63.5,370,8289,8290],[8293,62.5,370,8289,8290],[6901,60.6,370,8289,8290],[8294,59.6,370,8289,8290],[8295,57.6,370,8289,8290],[2754,53.6,370,8289,8290],[2752,49.6,370,8289,8290]]},{"id":"m261019368-2026-paper","name":"OS-Bench · Linux (2026 paper)","group":"知识与推理","metric":"Cross-platform task success","unit":"%","public_history":true,"protocol":{"en":"91 Windows-to-Linux migration tasks. Six models under three harnesses, medium effort. Target fail-to-pass and source regression tests both required.","zh":"91 项 Windows 到 Linux 迁移任务，六模型各用三种框架，medium 推理。需同时通过目标平台修复测试与源平台回归测试。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.19368","status":"paper_baselines","published_at":"2026-10-14"},"model_count":6,"result_count":18,"results":[[4979,0.187,370,8296,8297],[4979,0.242,370,8298,8297],[4979,0.209,370,8299,8297],[5727,0.209,370,8296,8297],[5727,0.209,370,8298,8297],[5727,0.198,370,8299,8297],[5616,0.11,370,8296,8297],[5616,0.17600000000000002,370,8298,8297],[5616,0.165,370,8299,8297],[3194,0.253,370,8296,8297],[3194,0.198,370,8298,8297],[3194,0.264,370,8299,8297],[5789,0.11,370,8296,8297],[5789,0.165,370,8298,8297],[5789,0.231,370,8299,8297],[3185,0.209,370,8296,8297],[3185,0.187,370,8298,8297],[3185,0.17600000000000002,370,8299,8297]]},{"id":"os-bench-windows-2026-paper","name":"OS-Bench · Windows (2026 paper)","group":"知识与推理","metric":"Cross-platform task success","unit":"%","public_history":true,"protocol":{"en":"49 Linux-to-Windows migration tasks, six models with Win-agent and medium effort. Target fail-to-pass and source regression tests both required.","zh":"49 项 Linux 到 Windows 迁移任务，六模型使用 Win-agent、medium 推理，需同时通过目标平台修复测试与源平台回归测试。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.19368","status":"paper_baselines","published_at":"2026-10-14"},"model_count":6,"result_count":6,"results":[[4979,0.204,370,8300,8297],[5727,0.102,370,8300,8297],[5616,0.040999999999999995,370,8300,8297],[3194,0.163,370,8300,8297],[5789,0.0,370,8300,8297],[3185,0.040999999999999995,370,8300,8297]]},{"id":"m261015562-2026-paper","name":"CausalCascade · Level 1 (2026 paper)","group":"视觉与视频","metric":"Event VQA overall","unit":"score","public_history":true,"protocol":{"en":"All 50 base scenes under Level-1 prompts; event-level video-QA composite score on the published 0–1 scale.","zh":"全部 50 个基础场景，Level-1 提示；取原生 0–1 事件级视频问答综合分。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.15562","status":"paper_baselines","published_at":"2026-10-11"},"score_max":1,"score_decimals":2,"saturation_excluded":true,"higher_is_better":true,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":9,"result_count":9,"results":[[8301,0.095,370,7834,8302],[8303,0.357,370,7834,8302],[8304,0.174,370,7834,8302],[8305,0.189,370,7834,8302],[8306,0.153,370,7834,8302],[8307,0.177,370,7834,8302],[8308,0.196,370,7834,8302],[8309,0.141,370,7834,8302],[8310,0.252,370,7834,8302]]},{"id":"m261022155-2026-paper","name":"Realtime-Bench (2026 paper)","group":"知识与推理","metric":"Pass^1","unit":"%","public_history":true,"protocol":{"en":"Success on a single attempt, averaged from three runs. The same twelve model backbones may appear under different effort or temporal configurations.","zh":"单次尝试成功率，由三次运行汇总。十二个底模可对应不同推理档位或时间设置。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.22155","status":"paper_baselines","published_at":"2026-10-15"},"model_count":12,"result_count":21,"results":[[4977,0.4267,370,8311,8312],[4977,0.42,370,8313,8312],[762,0.1067,370,8311,8312],[762,0.18,370,8313,8312],[2956,0.16670000000000001,370,8311,8312],[2956,0.16,370,8313,8312],[755,0.1267,370,8311,8312],[755,0.1533,370,8313,8312],[2983,0.24670000000000003,370,8311,8312],[2983,0.2733,370,8313,8312],[3512,0.1,370,8311,8312],[3512,0.0867,370,8313,8312],[2977,0.0667,370,8311,8312],[2977,0.1333,370,8313,8312],[8314,0.0933,370,8315,8312],[8314,0.1533,370,8316,8312],[8317,0.1067,370,8315,8312],[8317,0.1333,370,8316,8312],[6580,0.06,370,8318,8312],[7363,0.0933,370,8319,8312],[5491,0.12,370,8319,8312]]},{"id":"m261026765-2026-paper","name":"HandsOn · Core-50 (2026 paper)","group":"Agent 与工具","metric":"Task-macro success","unit":"%","public_history":true,"protocol":{"en":"50 tasks with native success predicates. Learned checkpoints use 100 episodes per task; GPT-6 uses ten. The single-task DP reference is omitted because it covers only eleven matched tasks.","zh":"50 项任务使用原生成功判据。学习型检查点每任务 100 次，GPT-6 每任务十次。单任务 DP 参考仅覆盖十一项匹配任务，不混入完整榜。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.26765","status":"paper_baselines","published_at":"2026-10-19"},"model_count":9,"result_count":9,"results":[[8320,0.29100000000000004,370,8321,8322],[7603,0.264,370,8321,8322],[8323,0.249,370,8324,8322],[8325,0.214,370,8324,8322],[8326,0.19699999999999998,370,8324,8322],[8327,0.18899999999999997,370,8324,8322],[8328,0.18100000000000002,370,8321,8322],[8329,0.124,370,8321,8322],[5676,0.052000000000000005,370,8330,8322]]},{"id":"erpbench-t1-2026-paper","name":"ERPBench · T1 (2026 paper)","group":"专业工作","metric":"Database task success","unit":"%","public_history":true,"protocol":{"en":"Screenshot-only agents complete ERPNext workflows. A task passes only when every target database field matches. T1: 100 runs per model, five attempts per task; six models share the grader and harness. Human controls are excluded.","zh":"智能体根据截图操作 ERPNext，全部目标数据库字段正确才算成功。T1 每个模型共 100 次运行，每题重复五次；六个模型使用相同框架和评分器。人类对照不进入模型排行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.17885v1","published_at":"2026-09-15","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[3572,0.94,370,8331,8332],[8333,0.34,370,8331,8332],[6343,0.32,370,8334,8332],[5486,0.23,370,8331,8332],[2465,0.09,370,8331,8332],[8335,0.0,370,8331,8332]]},{"id":"erpbench-t2-2026-paper","name":"ERPBench · T2 (2026 paper)","group":"专业工作","metric":"Database task success","unit":"%","public_history":true,"protocol":{"en":"Screenshot-only agents complete ERPNext workflows. A task passes only when every target database field matches. T2: 20 runs per model, five attempts per task; six models share the grader and harness. Human controls are excluded.","zh":"智能体根据截图操作 ERPNext，全部目标数据库字段正确才算成功。T2 每个模型共 20 次运行，每题重复五次；六个模型使用相同框架和评分器。人类对照不进入模型排行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.17885v1","published_at":"2026-09-15","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[3572,1.0,370,8331,8332],[8333,0.0,370,8331,8332],[6343,0.0,370,8334,8332],[5486,0.0,370,8331,8332],[2465,0.0,370,8331,8332],[8335,0.0,370,8331,8332]]},{"id":"erpbench-t3-2026-paper","name":"ERPBench · T3 (2026 paper)","group":"专业工作","metric":"Database task success","unit":"%","public_history":true,"protocol":{"en":"Screenshot-only agents complete ERPNext workflows. A task passes only when every target database field matches. T3: 30 runs per model, five attempts per task; six models share the grader and harness. Human controls are excluded.","zh":"智能体根据截图操作 ERPNext，全部目标数据库字段正确才算成功。T3 每个模型共 30 次运行，每题重复五次；六个模型使用相同框架和评分器。人类对照不进入模型排行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.17885v1","published_at":"2026-09-15","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[3572,1.0,370,8331,8332],[8333,0.03,370,8331,8332],[6343,0.03,370,8334,8332],[5486,0.0,370,8331,8332],[2465,0.0,370,8331,8332],[8335,0.0,370,8331,8332]]},{"id":"firmcore-structured-em-2026-paper","name":"FirmCORe · Chinese structured prediction (2026 paper)","group":"专业工作","metric":"Four-field exact match","unit":"%","public_history":true,"protocol":{"en":"Models infer collaboration opportunities, strength, type and direction from paired firm profiles. Overall four-field exact match uses all 2,805 Chinese pairs; missing or invalid predicted attributes are incorrect. All 17 models use the same zero-shot prompt and parser.","zh":"根据企业资料判断合作机会、强度、类型和角色方向。采用全部 2,805 对中文样本上的四字段精确匹配率；回答属性缺失或无效算错误。17 个模型使用相同零样本提示和解析器。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.17128v1","published_at":"2026-09-15","status":"paper_baselines"},"model_count":17,"result_count":17,"results":[[5344,0.3547,370,8336,8337],[8338,0.3447,370,8336,8337],[7719,0.2225,370,8336,8337],[5126,0.2071,370,8336,8337],[6865,0.1469,370,8336,8337],[8339,0.0492,370,8336,8337],[8340,0.1298,370,8336,8337],[8341,0.5197999999999999,370,8336,8337],[8342,0.415,370,8336,8337],[7694,0.44920000000000004,370,8336,8337],[8343,0.5761,370,8336,8337],[4961,0.5589999999999999,370,8336,8337],[6771,0.4021,370,8336,8337],[227,0.3722,370,8336,8337],[3763,0.6157,370,8336,8337],[3186,0.2535,370,8336,8337],[6342,0.26170000000000004,370,8336,8337]]},{"id":"firmcore-detection-f1-2026-paper","name":"FirmCORe · opportunity detection (2026 paper)","group":"专业工作","metric":"Macro-F1","unit":"score","public_history":true,"protocol":{"en":"Detects whether paired firm profiles support a collaboration opportunity. Macro-F1 on the 2,805-pair Chinese evaluation set, using the common zero-shot prompt, parser and label normalization.","zh":"判断两家企业的资料是否支持合作机会。采用 2,805 对中文评测样本的 Macro-F1，所有模型使用相同零样本提示、解析器和标签归一化。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.17128v1","published_at":"2026-09-15","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":17,"result_count":17,"results":[[8341,51.5,370,8344,8345],[5344,51.42,370,8344,8345],[8338,50.82,370,8344,8345],[4472,37.64,370,8344,8345],[7719,36.5,370,8344,8345],[5126,34.48,370,8344,8345],[6865,32.22,370,8344,8345],[8339,25.77,370,8344,8345],[3763,74.51,370,8344,8345],[8343,70.23,370,8344,8345],[4961,69.64,370,8344,8345],[7694,56.48,370,8344,8345],[6771,56.04,370,8344,8345],[227,53.78,370,8344,8345],[8342,52.87,370,8344,8345],[6342,44.46,370,8344,8345],[3186,43.22,370,8344,8345]]},{"id":"regmb-regional-retrieval-2026-paper","name":"REGMB · regional retrieval (2026 paper)","group":"视觉与视频","metric":"Mean published recall","unit":"score","public_history":true,"protocol":{"en":"Retrieves user-specified image regions or their descriptions across four task families. The authors’ Avg. combines Recall@5 and the two specified Recall@1 subsets. It stays a native retrieval score. All thirteen named checkpoints in Table 1 are retained.","zh":"在四类任务中检索指定图像区域或对应描述。原表 Avg. 综合 Recall@5 和两个指定的 Recall@1 子集，保留为原生检索分数。表 1 的十三个具名检查点全部保留。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.16847v1","published_at":"2026-09-15","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":13,"result_count":13,"results":[[8346,58.5,370,8347,8348],[8349,62.9,370,8350,8348],[8351,55.2,370,8350,8348],[8352,51.8,370,8350,8348],[8353,58.5,370,8354,8348],[8355,61.3,370,8354,8348],[8356,60.4,370,8354,8348],[8357,65.7,370,8354,8348],[8258,67.5,370,8354,8348],[8358,67.5,370,8359,8348],[8360,79.9,370,8361,8348],[8362,78.8,370,8354,8348],[8363,86.2,370,8364,8348]]},{"id":"robovad-in-domain-2026-paper","name":"RoboVAD · in-domain (2026 paper)","group":"视觉与视频","metric":"Micro frame-level AUC","unit":"score","public_history":true,"protocol":{"en":"Localizes anomalies in robotic manipulation videos under the in-domain split. The benchmark has 1,078 two-view episodes across five tasks and five anomaly types. Micro frame-level AUC is kept on the published 0–100 scale; each split remains separate.","zh":"在 in-domain 划分下定位机器人操作视频中的异常。数据集含五类任务、五种异常的 1,078 段双视角运行。采用原表 0–100 刻度的帧级微平均 AUC，各划分分别展示。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.17843v1","published_at":"2026-09-15","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":9,"result_count":9,"results":[[8365,47.3,370,8366,8367],[8368,56.3,370,8366,8367],[8369,52.6,370,8366,8367],[8370,64.1,370,8366,8367],[8371,45.8,370,8366,8367],[8372,47.1,370,8366,8367],[8373,88.5,370,8366,8367],[8374,73.6,370,8366,8367],[8375,94.6,370,8366,8367]]},{"id":"robovad-cross-anomaly-2026-paper","name":"RoboVAD · cross-anomaly (2026 paper)","group":"视觉与视频","metric":"Micro frame-level AUC","unit":"score","public_history":true,"protocol":{"en":"Localizes anomalies in robotic manipulation videos under the cross-anomaly split. The benchmark has 1,078 two-view episodes across five tasks and five anomaly types. Micro frame-level AUC is kept on the published 0–100 scale; each split remains separate.","zh":"在 cross-anomaly 划分下定位机器人操作视频中的异常。数据集含五类任务、五种异常的 1,078 段双视角运行。采用原表 0–100 刻度的帧级微平均 AUC，各划分分别展示。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.17843v1","published_at":"2026-09-15","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":9,"result_count":9,"results":[[8365,52.1,370,8376,8367],[8368,52.1,370,8376,8367],[8369,47.8,370,8376,8367],[8370,51.7,370,8376,8367],[8371,45.2,370,8376,8367],[8372,51.5,370,8376,8367],[8373,61.3,370,8376,8367],[8374,79.6,370,8376,8367],[8375,69.0,370,8376,8367]]},{"id":"robovad-cross-task-2026-paper","name":"RoboVAD · cross-task (2026 paper)","group":"视觉与视频","metric":"Micro frame-level AUC","unit":"score","public_history":true,"protocol":{"en":"Localizes anomalies in robotic manipulation videos under the cross-task split. The benchmark has 1,078 two-view episodes across five tasks and five anomaly types. Micro frame-level AUC is kept on the published 0–100 scale; each split remains separate.","zh":"在 cross-task 划分下定位机器人操作视频中的异常。数据集含五类任务、五种异常的 1,078 段双视角运行。采用原表 0–100 刻度的帧级微平均 AUC，各划分分别展示。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.17843v1","published_at":"2026-09-15","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":9,"result_count":9,"results":[[8365,54.4,370,8377,8367],[8368,58.1,370,8377,8367],[8369,54.3,370,8377,8367],[8370,66.9,370,8377,8367],[8371,47.9,370,8377,8367],[8372,48.0,370,8377,8367],[8373,71.8,370,8377,8367],[8374,65.6,370,8377,8367],[8375,72.0,370,8377,8367]]},{"id":"robovad-cross-task-anomaly-2026-paper","name":"RoboVAD · cross-task-anomaly (2026 paper)","group":"视觉与视频","metric":"Micro frame-level AUC","unit":"score","public_history":true,"protocol":{"en":"Localizes anomalies in robotic manipulation videos under the cross-task-anomaly split. The benchmark has 1,078 two-view episodes across five tasks and five anomaly types. Micro frame-level AUC is kept on the published 0–100 scale; each split remains separate.","zh":"在 cross-task-anomaly 划分下定位机器人操作视频中的异常。数据集含五类任务、五种异常的 1,078 段双视角运行。采用原表 0–100 刻度的帧级微平均 AUC，各划分分别展示。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.17843v1","published_at":"2026-09-15","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":9,"result_count":9,"results":[[8365,59.7,370,8378,8367],[8368,63.2,370,8378,8367],[8369,55.1,370,8378,8367],[8370,57.6,370,8378,8367],[8371,50.6,370,8378,8367],[8372,48.0,370,8378,8367],[8373,63.9,370,8378,8367],[8374,67.4,370,8378,8367],[8375,66.7,370,8378,8367]]},{"id":"rolebreak-persona-2026-paper","name":"RoleBreak · persona adherence (2026 paper)","group":"安全与可靠性","metric":"Persona criterion score","unit":"score","public_history":true,"protocol":{"en":"Tests persona consistency over spoken dialogues spanning 310 roles and 6,688 human-verified turns. DeepSeek-V4-Pro judges each applicable persona criterion; the reported mean criterion pass rate is retained as a native rubric score.","zh":"考查覆盖 310 个角色、6,688 个经人工核实对话轮次的语音角色一致性。DeepSeek-V4-Pro 逐条评判适用的角色准则，原表准则通过率均值保留为原生量表分。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.16614v1","published_at":"2026-09-15","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":9,"result_count":9,"results":[[6602,50.6,370,8379,8380],[6138,64.6,370,8379,8380],[6139,69.5,370,8379,8380],[8381,56.5,370,8379,8380],[8382,62.4,370,8379,8380],[5828,58.6,370,8383,8380],[4969,68.4,370,8383,8380],[4070,73.2,370,8383,8380],[2584,80.3,370,8383,8380]]},{"id":"riskchain-web-accuracy-2026-paper","name":"RiskChainBench · correct-routing web investigation (2026 paper)","group":"安全与可靠性","metric":"Web-only decision accuracy","unit":"%","public_history":true,"protocol":{"en":"Each of ten models investigates the same 600 human-labeled local websites once, using the correct association. The frozen risk decision is scored over all 600 cases; failed, timed-out or malformed attempts remain failures. Entry-gated accuracy and text restoration are separate metrics.","zh":"十个模型各调查同一组 600 个经人工标注的本地网站一次，使用正确的网站对应关系。冻结后的风险判断按全部 600 个案例计分；失败、超时和无效回答均算失败。入口门控准确率与文本恢复属于其他指标。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.16900v1","published_at":"2026-09-15","status":"paper_baselines"},"model_count":10,"result_count":10,"results":[[8384,0.628,370,8385,8386],[3184,0.537,370,8385,8386],[3479,0.498,370,8385,8386],[340,0.573,370,8385,8386],[2977,0.475,370,8385,8386],[244,0.552,370,8385,8386],[5735,0.517,370,8385,8386],[3568,0.263,370,8385,8386],[8387,0.295,370,8385,8386],[3490,0.415,370,8385,8386]]},{"id":"qiqc-pass-fail-2026-paper","name":"QIQCBench (pass–fail tasks, 2026 paper)","group":"数学与科学","metric":"Pass rate","unit":"%","public_history":true,"protocol":{"en":"Verified pass–fail outcomes on 39 quantum-engineering tasks, three trials each (117 per system). Infrastructure faults were quarantined and rerun. No-submission outcomes are budget-exhausted completed attempts. Values preserve the rounded percentage labels in Figure 4a; racing Elo and three infrastructure test cases are excluded.","zh":"在39道量子工程任务上验证通过或失败，每题三次、每个系统共117次运行。基础设施故障隔离后重跑；未提交包含预算耗尽的已执行尝试。保留图4a标注的整数百分比，竞赛Elo与三道基础设施测试不计入。"},"source":{"name":"QIQCBench authors","url":"https://arxiv.org/abs/2609.17439v1","published_at":"2026-09-15","status":"paper_baselines"},"model_count":17,"result_count":17,"results":[[4048,0.78,370,8388,8389],[5298,0.57,370,8388,8389],[3479,0.51,370,8390,8389],[5297,0.46,370,8388,8389],[8391,0.36,370,8392,8389],[3153,0.21,370,8392,8389],[8393,0.21,370,8392,8389],[3201,0.15,370,8392,8389],[3185,0.15,370,8392,8389],[2977,0.14,370,8392,8389],[5899,0.12,370,8392,8389],[3154,0.09,370,8392,8389],[3396,0.08,370,8392,8389],[325,0.07,370,8390,8389],[3157,0.07,370,8392,8389],[4881,0.07,370,8392,8389],[227,0.03,370,8392,8389]]},{"id":"adepts-mobile-benign-2026-paper","name":"ADeptS-Bench (mobile, benign) (2026 paper)","group":"Agent 与工具","metric":"Task success rate","unit":"%","public_history":true,"protocol":{"en":"Mobile benign task success rate over 358 tasks, mean across ten runs; seven-model safety-stream panel. Malicious attack success is a distinct metric.","zh":"移动端 358 个正常任务的成功率，采用七模型安全测试面板十次运行的均值。恶意攻击成功率为独立指标。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.26204","status":"paper_baselines","published_at":"2026-08-25"},"model_count":7,"result_count":7,"results":[[3412,0.7759999999999999,370,8394,8395],[8396,0.71,370,8394,8395],[3184,0.621,370,8394,8395],[8397,0.7809999999999999,370,8394,8395],[6297,0.83,370,8394,8395],[6342,0.79,370,8394,8395],[6298,0.816,370,8394,8395]]},{"id":"adepts-desktop-benign-2026-paper","name":"ADeptS-Bench (desktop, benign) (2026 paper)","group":"Agent 与工具","metric":"Task success rate","unit":"%","public_history":true,"protocol":{"en":"Desktop benign task success rate over 501 tasks, mean across ten runs; same seven-model safety-stream panel.","zh":"桌面端 501 个正常任务的成功率，采用同一七模型安全测试面板十次运行的均值。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.26204","status":"paper_baselines","published_at":"2026-08-25"},"model_count":7,"result_count":7,"results":[[3412,0.8290000000000001,370,8398,8395],[8396,0.7809999999999999,370,8398,8395],[3184,0.688,370,8398,8395],[8397,0.836,370,8398,8395],[6297,0.866,370,8398,8395],[6342,0.8740000000000001,370,8398,8395],[6298,0.87,370,8398,8395]]},{"id":"agentjudgebench-alignment-2026-paper","name":"AgentJudgeBench (with ground truth) (2026 paper)","group":"安全与可靠性","metric":"Record-weighted alignment","unit":"score","public_history":true,"protocol":{"en":"Table 15 Current: with-ground-truth alignment, record-weighted across the full corpus under the current parameter-structure rule. Upper bound is a diagnostic correction, not the observed score. Native 0–100 match score.","zh":"表 15 的 Current 列：提供标准答案、按完整语料记录加权、使用当前参数结构规则的对齐分。Upper bound 是诊断修正上界，不作实测成绩；保留 0–100 匹配分。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.26623","status":"paper_baselines","published_at":"2026-08-27"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":6,"result_count":6,"results":[[513,88.8,370,370,8399],[3419,87.87,370,370,8399],[283,85.57,370,370,8399],[3184,84.83,370,370,8399],[5343,84.11,370,370,8399],[764,81.95,370,370,8399]]},{"id":"clawprobench-final-2026-paper","name":"ClawProBench (full profile) (2026 paper)","group":"Agent 与工具","metric":"Final reliability composite","unit":"score","public_history":true,"protocol":{"en":"Table 4 full-profile nine-model panel. Final combines safety-gated trace quality and repeated-trial reliability using the source geometric formula; native 0–1 composite.","zh":"表 4 完整任务配置的九模型面板。Final 按原文几何公式合并带安全约束的轨迹质量与多次运行可靠性，保留 0–1 综合分。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.22510v1","status":"paper_baselines","published_at":"2026-08-23"},"score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":9,"result_count":9,"results":[[3196,0.679,370,370,8400],[7643,0.633,370,370,8400],[3186,0.629,370,370,8400],[8401,0.619,370,370,8400],[4225,0.611,370,370,8400],[3572,0.605,370,370,8400],[5735,0.602,370,370,8400],[3187,0.593,370,370,8400],[502,0.54,370,370,8400]]},{"id":"coinve-semantic-2026-paper","name":"CoinVE-Bench (semantic editing) (2026 paper)","group":"视觉与视频","metric":"Semantic accuracy rating","unit":"score","public_history":true,"protocol":{"en":"Compositional-instruction panel SA: checklist-based semantic editing rating, evaluated per instruction by Gemini 3.6 Flash. Native 0–100 rating; scope, persistence and perceptual quality are separate.","zh":"组合指令面板的 SA，采用 Gemini 3.6 Flash 按每条指令检查的语义编辑评分。保留 0–100 原始评分；编辑范围、持续性及感知质量另列。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.17566","status":"paper_baselines","published_at":"2026-08-18"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":9,"result_count":9,"results":[[6383,85.34,370,370,8402],[8403,86.91,370,370,8402],[8404,3.98,370,370,8402],[8405,34.69,370,370,8402],[8406,83.63,370,370,8402],[8407,59.67,370,370,8402],[8408,76.5,370,370,8402],[8409,75.58,370,370,8402],[8410,87.97,370,370,8402]]},{"id":"frontierchallenge-pass-2026-paper","name":"FrontierChallenge (2026 paper)","group":"数学与科学","metric":"Full-completion pass rate","unit":"%","public_history":true,"protocol":{"en":"Overall full-completion pass rate on the 97 released scientific-workflow tasks. Agent scaffolds and reasoning effort are configurations; the 300-task planned collection is not the scored denominator.","zh":"已发布 97 个科研工作流任务的完整完成率。Agent 执行框架与推理强度保留为配置，计划中的 300 题不作本次分母。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.24979","status":"paper_baselines","published_at":"2026-08-25"},"model_count":12,"result_count":13,"results":[[762,0.20600000000000002,370,2300,8411],[2974,0.20600000000000002,370,3759,8411],[2977,0.175,370,3759,8411],[755,0.175,370,3759,8411],[2973,0.155,370,8412,8411],[3560,0.155,370,3759,8411],[8413,0.134,370,3759,8411],[8414,0.124,370,3759,8411],[8415,0.124,370,8416,8411],[3386,0.10300000000000001,370,3759,8411],[8415,0.10300000000000001,370,3759,8411],[2576,0.040999999999999995,370,3759,8411],[227,0.031,370,3759,8411]]},{"id":"guardianbench-primary-2026-paper","name":"GuardianBench (primary panel) (2026 paper)","group":"安全与可靠性","metric":"Accuracy","unit":"%","public_history":true,"protocol":{"en":"Accuracy over all 3,024 paired scene–instruction rows. Uses the authors primary non-dagger model panel; three models overlapping the generation stack remain excluded as in the paper ranking.","zh":"全部 3,024 条配对场景与指令的准确率，采用作者主排名的无匕首标记模型面板；按原文排除与数据生成体系重合的三个模型。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.21928v1","status":"paper_baselines","published_at":"2026-08-22"},"model_count":12,"result_count":13,"results":[[6705,0.632,370,370,8417],[6704,0.5820000000000001,370,370,8417],[6214,0.563,370,370,8417],[340,0.568,370,370,8417],[3250,0.595,370,370,8417],[8418,0.621,370,370,8417],[8419,0.565,370,370,8417],[6297,0.623,370,8420,8417],[6297,0.657,370,3143,8417],[2444,0.595,370,370,8417],[2453,0.578,370,370,8417],[8421,0.569,370,370,8417],[2471,0.537,370,370,8417]]},{"id":"looparena-full-2026-paper","name":"LoopArena (full task) (2026 paper)","group":"Agent 与工具","metric":"Strict success rate","unit":"%","public_history":true,"protocol":{"en":"Type III full-task Strict Success Rate for six controller models. No-control and fixed-control reference policies are excluded; Type I contracts and Type II slices are separate tasks.","zh":"六个控制模型在 Type III 完整任务上的严格成功率，排除无控制与固定控制参考；Type I 契约与 Type II 片段为独立任务。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.28281v1","status":"paper_baselines","published_at":"2026-08-28"},"model_count":6,"result_count":6,"results":[[3764,0.3375,370,370,8422],[4961,0.3417,370,370,8422],[8423,0.2833,370,370,8422],[3654,0.24170000000000003,370,370,8422],[3196,0.3625,370,370,8422],[3479,0.3375,370,370,8422]]},{"id":"pace-statics-2026-paper","name":"PACE-Bench (Statics, vanilla) (2026 paper)","group":"Agent 与工具","metric":"Pass@2","unit":"%","public_history":true,"protocol":{"en":"Table 3 vanilla performance on the Statics subset under the full 20-attempt budget, using Pass@2. Adaptation-method rows on three backbones are a separate experiment.","zh":"表 3 完整 20 次尝试预算下，Statics 子集的基础设置 Pass@2。三种底层模型上的适应方法行属于另一实验。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.14441","status":"paper_baselines","published_at":"2026-08-14"},"model_count":6,"result_count":6,"results":[[5126,0.083,370,370,8424],[5344,0.33299999999999996,370,370,8424],[6301,0.375,370,370,8424],[786,0.375,370,370,8424],[3185,0.45799999999999996,370,370,8424],[3196,0.667,370,370,8424]]},{"id":"peakbench-planning-f1-2026-paper","name":"PeakBench (logical planning) (2026 paper)","group":"Agent 与工具","metric":"Edge F1","unit":"score","public_history":true,"protocol":{"en":"Dimension I logical-plan Edge F1 on the common evaluation questions. Native 0–1 F1; physical scheduling latency and earliest-ready reference policy are separate.","zh":"同一评估题目上第一维度逻辑计划的 Edge F1，保留 0–1 量纲，调度延迟与最早就绪参考策略另存。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.24509","status":"paper_baselines","published_at":"2026-08-25"},"score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":8,"result_count":8,"results":[[2058,0.839,370,370,8425],[3185,0.807,370,370,8425],[8188,0.771,370,370,8425],[268,0.772,370,370,8425],[3572,0.764,370,370,8425],[3201,0.733,370,370,8425],[2998,0.688,370,370,8425],[345,0.663,370,370,8425]]},{"id":"playworld-overall-2026-paper","name":"PlayWorld (2026 paper)","group":"视觉与视频","metric":"Overall VQA rating","unit":"score","public_history":true,"protocol":{"en":"Unweighted mean of four rubric-based VQA dimensions, using the same video-world-model benchmark. Published rating scale 1–5.","zh":"同一视频世界模型基准中四项细则式视觉问答评分的等权均值，原始评分范围 1–5。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.13552","status":"paper_baselines","published_at":"2026-08-13"},"score_max":5,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 5","zh":"分数 / 5"},"model_count":9,"result_count":9,"results":[[8426,2.12,370,370,8427],[8428,1.92,370,370,8427],[7238,1.78,370,370,8427],[8429,1.82,370,370,8427],[8430,1.61,370,370,8427],[7240,1.48,370,370,8427],[8431,1.42,370,370,8427],[7508,1.21,370,370,8427],[8432,1.14,370,370,8427]]},{"id":"ruleweaver-same-source-2026-paper","name":"RuleWeaver (same source) (2026 paper)","group":"知识与推理","metric":"Rubric score","unit":"score","public_history":true,"protocol":{"en":"Same-source rubric score under the 200-complex-rule setting. Native 0–100 score; cross-source changes and rule-retrieval precision/recall remain separate.","zh":"200 条复杂规则、同源设置下的细则评分，保留 0–100 量纲；跨源变化和规则检索精确率、召回率另列。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.26832","status":"paper_baselines","published_at":"2026-08-27"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":11,"result_count":11,"results":[[3183,48.06,370,370,8433],[6978,43.96,370,370,8433],[3184,48.21,370,370,8433],[3196,53.83,370,370,8433],[6237,31.77,370,370,8433],[3220,34.69,370,370,8433],[6412,37.75,370,370,8433],[268,37.54,370,370,8433],[8434,27.67,370,370,8433],[3190,29.69,370,370,8433],[3187,32.5,370,370,8433]]},{"id":"wearableqa-average-2026-paper","name":"WearableQA (2026 paper)","group":"医疗","metric":"Avg","unit":"%","score_max":100,"higher_is_better":true,"public_history":true,"protocol":{"en":"Answers data-analysis and health-reasoning questions about wearable-sensor time series. The main comparison uses chain-of-thought prompting and reports average answer accuracy.","zh":"根据可穿戴传感器时间序列回答数据分析和健康推理问题。采用主实验的思维链提示及平均答案准确率。"},"source":{"name":"WearableQA authors","status":"paper_baselines","url":"https://arxiv.org/html/2609.05405v1#S4.T1","published_at":"2026-09-04"},"model_count":14,"result_count":14,"results":[[3412,0.7290000000000001,370,8435,8436],[2261,0.602,370,8435,8436],[323,0.532,370,8435,8436],[3184,0.512,370,8435,8436],[2602,0.506,370,8435,8436],[353,0.34700000000000003,370,8435,8436],[5635,0.425,370,8435,8436],[8437,0.293,370,8435,8436],[5648,0.292,370,8435,8436],[6689,0.262,370,8435,8436],[6668,0.24600000000000002,370,8435,8436],[587,0.23800000000000002,370,8435,8436],[6677,0.198,370,8435,8436],[613,0.196,370,8435,8436]]},{"id":"ideaambig-readiness-real-2026-paper","name":"IdeaAMBIG · real readiness (2026 paper)","group":"专业工作","metric":"Readiness macro F1","unit":"score","score_max":100,"higher_is_better":true,"public_history":true,"protocol":{"en":"Judges whether a project specification is ready for implementation. Readiness macro F1 uses a balanced200-item panel with 100 ready and100 not-ready specifications from the real-world subset.","zh":"判断项目规格能否直接进入实现。采用真实子集中的200条平衡样本，分别含100条就绪和100条未就绪规格，以宏平均F1计分。"},"source":{"name":"IdeaAMBIG authors","status":"paper_baselines","url":"https://arxiv.org/html/2609.10539v1#S4.T1","published_at":"2026-09-09"},"model_count":13,"result_count":13,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"results":[[4979,67.5,370,8438,8439],[3568,59.3,370,8438,8439],[3482,45.8,370,8438,8439],[3009,44.5,370,8438,8439],[2576,57.5,370,8438,8439],[773,44.3,370,8438,8439],[227,45.1,370,8438,8439],[3155,32.7,370,8438,8439],[3419,30.0,370,8438,8439],[6374,47.1,370,8438,8439],[4070,33.3,370,8438,8439],[5344,40.1,370,8438,8439],[786,34.5,370,8438,8439]]},{"id":"ideaambig-readiness-synthetic-2026-paper","name":"IdeaAMBIG · synthetic readiness (2026 paper)","group":"专业工作","metric":"Readiness macro F1","unit":"score","score_max":100,"higher_is_better":true,"public_history":true,"protocol":{"en":"Judges whether a project specification is ready for implementation. Readiness macro F1 uses a balanced200-item panel with 100 ready and100 not-ready specifications from the controlled-synthetic subset.","zh":"判断项目规格能否直接进入实现。采用受控合成子集中的200条平衡样本，分别含100条就绪和100条未就绪规格，以宏平均F1计分。"},"source":{"name":"IdeaAMBIG authors","status":"paper_baselines","url":"https://arxiv.org/html/2609.10539v1#S4.T1","published_at":"2026-09-09"},"model_count":13,"result_count":13,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"results":[[4979,86.4,370,8438,8439],[3568,76.2,370,8438,8439],[3482,67.3,370,8438,8439],[3009,67.2,370,8438,8439],[2576,55.9,370,8438,8439],[773,59.9,370,8438,8439],[227,65.9,370,8438,8439],[3155,43.1,370,8438,8439],[3419,44.4,370,8438,8439],[6374,56.9,370,8438,8439],[4070,54.3,370,8438,8439],[5344,48.7,370,8438,8439],[786,37.0,370,8438,8439]]},{"id":"mrlhdr-answer-2026-paper","name":"Mr.LHDR · final answer (2026 paper)","group":"Agent 与工具","metric":"OA","unit":"%","score_max":100,"higher_is_better":true,"public_history":true,"protocol":{"en":"Solves102 long-horizon research questions with linked evidence and process checklists. This ranking uses final-answer accuracy under the published Qwen3-VL235B judge; tool access and agent systems remain explicit.","zh":"解决102个附带证据链和过程清单的长程研究问题。采用论文Qwen3-VL235B裁判给出的最终答案准确率，并保留工具权限与智能体系统配置。"},"source":{"name":"Mr.LHDR authors","status":"paper_baselines","url":"https://arxiv.org/html/2609.11318v1#S4.T4","published_at":"2026-09-10"},"model_count":25,"result_count":25,"results":[[3196,0.431,370,8440,8441],[5929,0.353,370,8440,8441],[1464,0.34299999999999997,370,8440,8441],[3184,0.304,370,8440,8441],[3578,0.294,370,8440,8441],[3741,0.225,370,8440,8441],[8442,0.18600000000000003,370,8440,8441],[3572,0.17600000000000002,370,8440,8441],[345,0.16699999999999998,370,8440,8441],[3494,0.147,370,8440,8441],[8443,0.098,370,8440,8441],[3412,0.34299999999999997,370,8444,8441],[3424,0.304,370,8444,8441],[2602,0.284,370,8444,8441],[2055,0.265,370,8444,8441],[362,0.157,370,8444,8441],[6343,0.10800000000000001,370,8444,8441],[2166,0.08800000000000001,370,8444,8441],[8445,0.324,370,8446,8441],[8447,0.275,370,8446,8441],[8448,0.225,370,8446,8441],[8449,0.069,370,8446,8441],[8450,0.16699999999999998,370,8451,8441],[8452,0.157,370,8451,8441],[2943,0.039,370,8451,8441]]},{"id":"dramachain-storyboard-2026-paper","name":"DramaChain · storyboard design (2026 paper)","group":"生成与创作","metric":"All","unit":"score","score_max":5,"higher_is_better":true,"public_history":true,"protocol":{"en":"Evaluates storyboard design for short-drama production. The published automated score uses a five-point rubric and averages the stage-defined assessment axes. Partial-corpus runs marked by the authors are omitted.","zh":"评测短剧制作中的分镜设计。采用五分制自动评分，对该阶段适用的评价维度取平均；不纳入作者标注的部分语料运行。"},"source":{"name":"DramaChain Bench authors","status":"paper_baselines","url":"https://arxiv.org/html/2609.00646v1#S4.T2","published_at":"2026-09-01"},"model_count":16,"result_count":16,"saturation_excluded":true,"scale_label":{"en":"Score / 5","zh":"分数 / 5"},"results":[[8453,4.23,370,8454,8455],[8456,4.17,370,8457,8455],[8458,3.89,370,8459,8455],[8460,3.89,370,8461,8455],[5104,3.79,370,8462,8455],[8463,3.71,370,8464,8455],[420,3.64,370,8465,8455],[411,3.48,370,8466,8455],[8467,3.26,370,8468,8455],[8469,4.4,370,8470,8455],[8471,4.24,370,8472,8455],[8473,4.22,370,8474,8455],[4952,4.22,370,8475,8455],[4950,4.08,370,8476,8455],[5105,3.92,370,8477,8455],[8478,3.78,370,8479,8455]]},{"id":"dramachain-single-keyframe-2026-paper","name":"DramaChain · single keyframe (2026 paper)","group":"生成与创作","metric":"All","unit":"score","score_max":5,"higher_is_better":true,"public_history":true,"protocol":{"en":"Evaluates single keyframe for short-drama production. The published automated score uses a five-point rubric and averages the stage-defined assessment axes. Partial-corpus runs marked by the authors are omitted.","zh":"评测短剧制作中的单幅关键帧。采用五分制自动评分，对该阶段适用的评价维度取平均；不纳入作者标注的部分语料运行。"},"source":{"name":"DramaChain Bench authors","status":"paper_baselines","url":"https://arxiv.org/html/2609.00646v1#S4.T2","published_at":"2026-09-01"},"model_count":6,"result_count":6,"saturation_excluded":true,"scale_label":{"en":"Score / 5","zh":"分数 / 5"},"results":[[8480,3.99,370,8481,8455],[8482,3.72,370,8483,8455],[8484,3.68,370,8485,8455],[8486,3.64,370,8487,8455],[8488,3.62,370,8489,8455],[8490,3.38,370,8491,8455]]},{"id":"dramachain-episode-keyframes-2026-paper","name":"DramaChain · episode keyframes (2026 paper)","group":"生成与创作","metric":"All","unit":"score","score_max":5,"higher_is_better":true,"public_history":true,"protocol":{"en":"Evaluates episode keyframes for short-drama production. The published automated score uses a five-point rubric and averages the stage-defined assessment axes. Partial-corpus runs marked by the authors are omitted.","zh":"评测短剧制作中的整集关键帧。采用五分制自动评分，对该阶段适用的评价维度取平均；不纳入作者标注的部分语料运行。"},"source":{"name":"DramaChain Bench authors","status":"paper_baselines","url":"https://arxiv.org/html/2609.00646v1#S4.T2","published_at":"2026-09-01"},"model_count":6,"result_count":6,"saturation_excluded":true,"scale_label":{"en":"Score / 5","zh":"分数 / 5"},"results":[[8480,3.82,370,8481,8455],[8482,3.6,370,8483,8455],[8484,3.47,370,8485,8455],[8486,3.28,370,8487,8455],[8488,3.04,370,8489,8455],[8490,3.0,370,8491,8455]]},{"id":"khatiandoc-row-f1-2026-paper","name":"KhatianDoc · field extraction (2026 paper)","group":"专业工作","metric":"T3 RowF1","unit":"score","score_max":100,"higher_is_better":true,"public_history":true,"protocol":{"en":"Extracts structured ownership fields from 107 handwritten Bangladeshi land records. Row-level F1 credits fields recovered from the image and avoids the constant/null-field inflation found in the metadata metric.","zh":"从107份孟加拉国手写土地记录中提取结构化权属字段。采用行级F1，按从图像中正确提取的字段计分，避开元数据指标中的常量和空值膨胀。"},"source":{"name":"KhatianDoc authors","status":"paper_baselines","url":"https://arxiv.org/html/2609.03597v1#S5.T2","published_at":"2026-09-03"},"model_count":6,"result_count":6,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"results":[[3708,5.16,370,8492,8493],[2166,11.97,370,8492,8493],[6342,17.48,370,8492,8493],[363,6.43,370,8492,8493],[8494,25.84,370,8492,8493],[2839,0.1,370,8492,8493]]},{"id":"reacthuman-action-2026-paper","name":"ReactHuman · action accuracy (2026 paper)","group":"视觉与视频","metric":"SAA","unit":"%","score_max":100,"higher_is_better":true,"public_history":true,"protocol":{"en":"Chooses a reaction to a physical event across 306 scenes and17 event families. Semantic action accuracy compares the chosen action label with the reference; the paper reports2138 completed decisions and excludes four API errors.","zh":"在306个场景、17类物理事件中选择反应。语义动作准确率比较所选动作与标准标签；论文统计2,138条完成的决策，剔除四次API错误。"},"source":{"name":"ReactHuman authors","status":"paper_baselines","url":"https://arxiv.org/html/2609.10895v1#S5.T1","published_at":"2026-09-09"},"model_count":7,"result_count":7,"results":[[3479,0.507,370,8495,8496],[3196,0.637,370,8495,8496],[2055,0.48700000000000004,370,8495,8496],[3490,0.409,370,8495,8496],[6297,0.564,370,8495,8496],[6299,0.5489999999999999,370,8495,8496],[6689,0.624,370,8495,8496]]},{"id":"obgynlong-history-2026-paper","name":"ObGynLongBench · history-level EHR (2026 paper)","group":"医疗","metric":"Overall","unit":"%","score_max":100,"higher_is_better":true,"public_history":true,"protocol":{"en":"Makes obstetric and gynecologic decisions from longitudinal health records. The history-level condition supplies all pre-decision records; fixed-choice accuracy uses1500 cases grounded in patient evidence and clinical rules.","zh":"根据纵向健康记录作出妇产科判断。完整病史条件提供决策时点之前的全部记录；在1,500个具备患者证据和临床规则的案例上计算选择题准确率。"},"source":{"name":"ObGynLongBench authors","status":"paper_baselines","url":"https://arxiv.org/html/2609.07601v1#S3.T2","published_at":"2026-09-07"},"model_count":17,"result_count":17,"results":[[6214,0.598,370,8497,8498],[4958,0.649,370,8497,8498],[6133,0.687,370,8497,8498],[3201,0.644,370,8497,8498],[6298,0.598,370,8497,8498],[6342,0.618,370,8497,8498],[4969,0.594,370,8497,8498],[4070,0.593,370,8497,8498],[2590,0.601,370,8497,8498],[5650,0.52,370,8497,8498],[5635,0.607,370,8497,8498],[8499,0.511,370,8497,8498],[8500,0.5710000000000001,370,8497,8498],[8501,0.613,370,8497,8498],[8502,0.539,370,8497,8498],[8173,0.596,370,8497,8498],[4970,0.413,370,8497,8498]]},{"id":"logimed-end-to-end-2026-paper","name":"LogiMed-RoB · end-to-end consistency (2026 paper)","group":"医疗","metric":"E2E","unit":"%","score_max":100,"higher_is_better":true,"public_history":true,"protocol":{"en":"Assesses randomized trials using Cochrane risk-of-bias rules. End-to-end consistency is the fraction of instances that satisfy the symbolic rules throughout the complete reasoning chain.","zh":"按Cochrane偏倚风险规则评估随机试验。端到端一致率统计整个推理链始终符合符号规则的实例比例。"},"source":{"name":"LogiMed-RoB authors","status":"paper_baselines","url":"https://arxiv.org/html/2609.11185v1#S4.T4","published_at":"2026-09-10"},"model_count":10,"result_count":10,"results":[[3644,0.23800000000000002,370,8503,8504],[283,0.2994,370,8503,8504],[3424,0.384,370,8503,8504],[3412,0.45130000000000003,370,8503,8504],[4425,0.1885,370,8503,8504],[796,0.0,370,8503,8504],[1513,0.0144,370,8503,8504],[362,0.0129,370,8503,8504],[8505,0.008,370,8503,8504],[8506,0.1158,370,8503,8504]]},{"id":"novgauge-task-vf1-2026-paper","name":"NovGauge · task novelty (2026 paper)","group":"数学与科学","metric":"VF1","unit":"score","score_max":100,"higher_is_better":true,"public_history":true,"protocol":{"en":"Compares research papers to judge novelty in the research task. Verified F1 checks the novelty verdict together with its supporting evidence, using the full-paper input condition.","zh":"比较研究论文，判断研究任务的新颖性。验证F1同时检查新颖性判断及其证据，采用全文输入条件。"},"source":{"name":"NovGauge authors","status":"paper_baselines","url":"https://arxiv.org/html/2609.11234v1#S4.T3","published_at":"2026-09-10"},"model_count":18,"result_count":18,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"results":[[3196,71.5,370,8507,8508],[4958,52.1,370,8507,8508],[3578,18.2,370,8507,8508],[3572,18.4,370,8507,8508],[3412,33.5,370,8507,8508],[4219,22.8,370,8507,8508],[3187,19.9,370,8507,8508],[3186,24.0,370,8507,8508],[3189,22.0,370,8507,8508],[3185,20.0,370,8507,8508],[3201,18.1,370,8507,8508],[3190,9.9,370,8507,8508],[5344,9.6,370,8507,8508],[786,14.5,370,8507,8508],[2584,14.3,370,8507,8508],[8509,34.9,370,8507,8508],[8510,2.4,370,8507,8508],[8511,15.9,370,8507,8508]]},{"id":"spot-shift-type-action-2026-paper","name":"Spot-the-shift · type and action (2026 paper)","group":"视觉与视频","metric":"T+A","unit":"score","score_max":100,"higher_is_better":true,"public_history":true,"protocol":{"en":"Finds changes between videos and describes the action involved. The type-plus-action F1 requires both the difference category and action label to match under the zero-shot captioning protocol.","zh":"识别视频之间的变化及相关动作。在零样本描述协议下，类型加动作F1要求变化类别和动作标签同时匹配。"},"source":{"name":"SPOT-THE-SHIFT authors","status":"paper_baselines","url":"https://arxiv.org/html/2609.10356v1#S6.T3","published_at":"2026-09-09"},"model_count":9,"result_count":9,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"results":[[3424,49.5,370,8512,8513],[2261,21.6,370,8512,8513],[6344,6.3,370,8512,8513],[2654,9.4,370,8512,8513],[5812,3.9,370,8512,8513],[5725,6.5,370,8512,8513],[6341,2.0,370,8512,8513],[6342,7.9,370,8512,8513],[6343,16.2,370,8512,8513]]},{"id":"worldsolver-2026-paper","name":"WorldSolver (2026 paper)","group":"数学与科学","metric":"Execution-gated physics score","unit":"score","public_history":true,"protocol":{"en":"Fixed simulation scaffolds across seven physics domains. Overall averages execution-gated visual fidelity and trajectory-based physics scores over all cases; nonexecuting submitted solvers fail the execution check.","zh":"七个物理领域使用固定仿真框架。Overall 综合执行检查、视觉一致性和轨迹物理指标；提交的求解器无法运行则不通过执行检查。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.08720","status":"paper_baselines","published_at":"2026-10-06"},"score_max":100,"score_decimals":2,"saturation_excluded":true,"higher_is_better":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":7,"result_count":7,"results":[[762,48.7,370,7834,8514],[4983,46.7,370,7834,8514],[5906,29.3,370,7834,8514],[3153,24.6,370,7834,8514],[3154,23.6,370,7834,8514],[5133,22.7,370,7834,8514],[3203,17.1,370,7834,8514]]},{"id":"insclaim-decision-2026-paper","name":"InsClaimBench (2026 paper)","group":"专业工作","metric":"Payout decision accuracy","unit":"%","public_history":true,"protocol":{"en":"All six models answer the same 3,780 text cases and insurance rules. Table 3 Overall payout decision accuracy; payout-amount error uses a different 3,761-case subset and is excluded.","zh":"六模型回答同一批 3,780 个文本理赔案例及保险规则。取表 3 Overall 理赔决策准确率；金额误差采用另一个 3,761 案例子集，不混入此榜。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.09671","status":"paper_baselines","published_at":"2026-10-07"},"model_count":6,"result_count":6,"results":[[4979,0.8005,370,7834,8515],[5519,0.7929,370,7834,8515],[3153,0.7746,370,7834,8515],[5899,0.7664,370,7834,8515],[3187,0.8019,370,7834,8515],[2584,0.7423000000000001,370,7834,8515]]},{"id":"or-bench-macro-2026-paper","name":"OR-Bench (2026 paper)","group":"视觉与视频","metric":"Eight-task macro accuracy","unit":"%","public_history":true,"protocol":{"en":"Published Macro-8 across eight object-rotation tasks, including exact-angle open responses and multi-view reasoning. Twelve VLMs; human row excluded.","zh":"八项物体旋转任务的原表 Macro-8，包括开放式精确角度回答与多视角推理。十二个视觉语言模型，排除人类行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.05715","status":"paper_baselines","published_at":"2026-10-05"},"model_count":12,"result_count":12,"results":[[3184,0.544,370,7834,8516],[353,0.484,370,7834,8516],[354,0.442,370,7834,8516],[6342,0.46799999999999997,370,7834,8516],[2728,0.43,370,7834,8516],[2654,0.43,370,7834,8516],[2754,0.423,370,7834,8516],[2687,0.42100000000000004,370,7834,8516],[2485,0.374,370,7834,8516],[8517,0.371,370,7834,8516],[8518,0.358,370,7834,8516],[6389,0.316,370,7834,8516]]},{"id":"medbench-choices-2026-paper","name":"MedBench · choice tasks (2026 paper)","group":"医疗","metric":"Overall accuracy","unit":"%","public_history":true,"protocol":{"en":"Published overall accuracy, the first value of the Accuracy/Macro-F1 pair. Same image-question choices; deterministic decoding. Thinking checkpoints retained; family averages excluded.","zh":"取 Accuracy/Macro-F1 对中的第一个值，即总体准确率。相同图像选择题、确定性解码，保留思考检查点，排除模型族均值。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.11312","status":"paper_baselines","published_at":"2026-10-08"},"model_count":12,"result_count":12,"results":[[2728,0.3306,370,8519,8520],[2754,0.377,370,8519,8520],[8521,0.3678,370,8519,8520],[8522,0.3738,370,8523,8520],[6391,0.4465,370,8519,8520],[6343,0.43670000000000003,370,8519,8520],[8524,0.3284,370,8519,8520],[8173,0.47,370,8519,8520],[8167,0.39390000000000003,370,8519,8520],[8525,0.4542,370,8523,8520],[8526,0.5523,370,8519,8520],[8169,0.5073,370,8519,8520]]},{"id":"braintrace-broad-2026-paper","name":"BrainTRACE · broad-compatible (2026 paper)","group":"医疗","metric":"Closed-form accuracy","unit":"%","public_history":true,"protocol":{"en":"Table 2 Broad column over 4,053 broad-compatible questions. Native 3D-specific questions stay separate; M3D-LaMed has no Broad score and is omitted.","zh":"表 2 Broad 列的 4,053 道兼容通用输入问题。原生三维问题另列；M3D-LaMed 没有 Broad 成绩，不录入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.06571","status":"paper_baselines","published_at":"2026-10-05"},"model_count":19,"result_count":19,"results":[[2754,0.332,370,7834,8527],[6298,0.327,370,7834,8527],[6342,0.337,370,7834,8527],[6299,0.35100000000000003,370,7834,8527],[2584,0.373,370,7834,8527],[2728,0.358,370,7834,8527],[2697,0.35100000000000003,370,7834,8527],[2675,0.355,370,7834,8527],[8528,0.109,370,7834,8527],[8529,0.363,370,7834,8527],[8173,0.365,370,7834,8527],[8530,0.249,370,7834,8527],[8531,0.006,370,7834,8527],[3184,0.40399999999999997,370,7834,8527],[2058,0.419,370,7834,8527],[4064,0.382,370,7834,8527],[2261,0.298,370,7834,8527],[2602,0.365,370,7834,8527],[8532,0.18100000000000002,370,7834,8527]]},{"id":"swe-journey-final-f2p-2026-paper","name":"SWE-Journey (2026 paper)","group":"编程","metric":"Final fail-to-pass rate","unit":"%","public_history":true,"protocol":{"en":"Final whole-task fail-to-pass test-node rate, averaged equally over four simulated user personas. Thirteen coding assistants; fixed unscored GLM-5.2 user simulator. Reference-invalid test nodes are removed consistently.","zh":"最终完整任务的 fail-to-pass 测试节点通过率，对四种模拟用户角色等权平均。十三个编码助手，用户模拟器固定为不计分的 GLM-5.2；参考补丁无效的测试节点统一剔除。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.11559","status":"paper_baselines","published_at":"2026-10-08"},"model_count":13,"result_count":13,"results":[[4983,0.5222,370,7834,8533],[4979,0.5151,370,7834,8533],[5728,0.49520000000000003,370,7834,8533],[227,0.4936,370,7834,8533],[3155,0.45030000000000003,370,7834,8533],[8342,0.4451,370,7834,8533],[3186,0.4047,370,7834,8533],[5114,0.3874,370,7834,8533],[2576,0.3831,370,7834,8533],[3185,0.3718,370,7834,8533],[3201,0.32170000000000004,370,7834,8533],[7643,0.35619999999999996,370,7834,8533],[2582,0.2987,370,7834,8533]]},{"id":"frontier3d-total-2026-paper","name":"Frontier3DBench (2026 paper)","group":"视觉与视频","metric":"Overall 3D artifact score","unit":"score","public_history":true,"protocol":{"en":"All 145 tasks with fixed delivery checks and evaluation. Table 3 Total on 0–100. Highest available effort: Qwen xhigh, others max; agent harness retained.","zh":"全部 145 项任务使用固定交付检查和评估。取表 3 Total，0–100 分；Qwen 使用 xhigh，其余模型 max，并保留智能体框架。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.09017","status":"paper_baselines","published_at":"2026-11-06"},"score_max":100,"score_decimals":2,"saturation_excluded":true,"higher_is_better":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":11,"result_count":11,"results":[[5876,66.02,370,8534,8535],[5404,63.85,370,8534,8535],[2959,62.29,370,8536,8535],[8537,57.45,370,8538,8535],[8539,55.96,370,8540,8535],[3395,52.55,370,8536,8535],[3155,49.18,370,8540,8535],[5874,48.56,370,8534,8535],[3396,47.61,370,8541,8535],[3157,46.03,370,8542,8535],[6452,42.94,370,8536,8535]]},{"id":"phrbench-truthful-2026-paper","name":"PHRBench · truthful premise (2026 paper)","group":"视觉与视频","metric":"Reasoning accuracy","unit":"%","public_history":true,"protocol":{"en":"Table 1 Acc(T): reasoning with truthful augmentation. Eighteen models. The hallucinated-premise condition and reasoning-behavior diagnostics remain separate.","zh":"表 1 Acc(T)，在真实补充前提下推理的准确率。十八模型；幻觉前提和行为诊断另列。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.10455","status":"paper_baselines","published_at":"2026-10-07"},"model_count":18,"result_count":18,"results":[[6863,0.575,370,7834,8543],[354,0.555,370,7834,8543],[353,0.518,370,7834,8543],[340,0.616,370,7834,8543],[781,0.562,370,7834,8543],[764,0.585,370,7834,8543],[1826,0.24600000000000002,370,7834,8543],[1748,0.445,370,7834,8543],[1645,0.46799999999999997,370,7834,8543],[1651,0.522,370,7834,8543],[5344,0.46,370,7834,8543],[786,0.485,370,7834,8543],[784,0.5820000000000001,370,7834,8543],[562,0.342,370,7834,8543],[524,0.484,370,7834,8543],[1804,0.312,370,7834,8543],[550,0.32299999999999995,370,7834,8543],[8544,0.21100000000000002,370,7834,8543]]},{"id":"phrbench-hallucinated-2026-paper","name":"PHRBench · hallucinated premise (2026 paper)","group":"视觉与视频","metric":"Reasoning accuracy","unit":"%","public_history":true,"protocol":{"en":"Table 1 Acc(H): reasoning with hallucinated augmentation, on the same question set as the truthful condition. Eighteen models.","zh":"表 1 Acc(H)，在幻觉补充前提下推理的准确率，题目与真实前提条件相同。十八模型。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.10455","status":"paper_baselines","published_at":"2026-10-07"},"model_count":18,"result_count":18,"results":[[6863,0.484,370,7834,8543],[354,0.483,370,7834,8543],[353,0.423,370,7834,8543],[340,0.447,370,7834,8543],[781,0.451,370,7834,8543],[764,0.46399999999999997,370,7834,8543],[1826,0.20600000000000002,370,7834,8543],[1748,0.38,370,7834,8543],[1645,0.396,370,7834,8543],[1651,0.442,370,7834,8543],[5344,0.43799999999999994,370,7834,8543],[786,0.41,370,7834,8543],[784,0.474,370,7834,8543],[562,0.319,370,7834,8543],[524,0.409,370,7834,8543],[1804,0.295,370,7834,8543],[550,0.22399999999999998,370,7834,8543],[8544,0.157,370,7834,8543]]},{"id":"ego2act-judge-2026-paper","name":"Ego2Act (2026 paper)","group":"视觉与视频","metric":"Ego2ActJudge aggregate","unit":"score","public_history":true,"protocol":{"en":"Table 2 Ego2ActJudge Agg.; per-video geometric aggregation of task and physical consistency. Six video generators; human positive/negative controls excluded.","zh":"表 2 Ego2ActJudge Agg.，逐视频结合任务与物理一致性的几何聚合分。六个视频生成器，排除人类正负对照。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.01092","status":"paper_baselines","published_at":"2026-10-01"},"score_max":100,"score_decimals":2,"saturation_excluded":true,"higher_is_better":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":6,"result_count":6,"results":[[8545,69.5,370,7834,8546],[7124,83.3,370,7834,8546],[8547,72.5,370,7834,8546],[8548,71.4,370,7834,8546],[7125,59.3,370,7834,8546],[8549,13.3,370,7834,8546]]},{"id":"savu-spatial-reasoning-2026-paper","name":"SAVU-Bench · spatial reasoning (2026 paper)","group":"视觉与视频","metric":"Cyclic spatial reasoning accuracy","unit":"%","public_history":true,"protocol":{"en":"Spatial Reasoning column under the published cyclic evaluation protocol. Twelve audiovisual models; chance control excluded. Grounding and alignment tasks remain separate.","zh":"按原文循环评估协议，取 Spatial Reasoning 列。十二个音视频模型，排除随机基线；定位和对齐任务另列。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.10624","status":"paper_baselines","published_at":"2026-10-07"},"model_count":12,"result_count":12,"results":[[6724,0.43799999999999994,370,7834,8550],[6723,0.43200000000000005,370,7834,8550],[5136,0.44799999999999995,370,7834,8550],[764,0.45799999999999996,370,7834,8550],[6721,0.41700000000000004,370,7834,8550],[8551,0.44299999999999995,370,7834,8550],[8552,0.43200000000000005,370,7834,8550],[6138,0.44299999999999995,370,7834,8550],[8553,0.45799999999999996,370,7834,8550],[8381,0.43799999999999994,370,7834,8550],[8554,0.396,370,7834,8550],[8555,0.406,370,7834,8550]]},{"id":"checkerbench-2026-paper","name":"CheckerBench (2026 paper)","group":"编程","metric":"Pass@1","unit":"%","public_history":true,"protocol":{"en":"Same 300 checker-synthesis tasks for seven models in three harnesses; mean of three independent repeats. Submitted candidate errors, refusals and unsuccessful synthesis are task failures. Summary rows excluded.","zh":"七模型在三种框架下评估同一批 300 项静态检查器合成任务，取三次独立重复均值。候选代码错误、拒绝和未成功合成属于任务失败；排除汇总行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.07557","status":"paper_baselines","published_at":"2026-10-06"},"model_count":7,"result_count":21,"results":[[4971,0.31670000000000004,370,8556,8557],[3479,0.4533,370,8556,8557],[3157,0.37329999999999997,370,8556,8557],[3185,0.3967,370,8556,8557],[227,0.3233,370,8556,8557],[3155,0.38670000000000004,370,8556,8557],[8558,0.3367,370,8556,8557],[4971,0.34,370,8559,8557],[3479,0.35,370,8559,8557],[3157,0.3467,370,8559,8557],[3185,0.3833,370,8559,8557],[227,0.33,370,8559,8557],[3155,0.38,370,8559,8557],[8558,0.23670000000000002,370,8559,8557],[4971,0.23,370,8560,8557],[3479,0.3433,370,8560,8557],[3157,0.25,370,8560,8557],[3185,0.3233,370,8560,8557],[227,0.22329999999999997,370,8560,8557],[3155,0.2433,370,8560,8557],[8558,0.2167,370,8560,8557]]},{"id":"agenticbbo-hpo-2026-paper","name":"AgenticBBO-Bench · HPO (2026 paper)","group":"数学与科学","metric":"Normalized optimization score","unit":"score","public_history":true,"protocol":{"en":"Seven frontier models on the HPO task of the fixed five-task frontier challenge. Source normalized score; numerical GP control excluded.","zh":"七个前沿模型在固定五任务挑战的 HPO 任务上评估。保留原文归一化优化分，排除数值 GP 对照。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.12183","status":"paper_baselines","published_at":"2026-10-08"},"score_max":1,"score_decimals":2,"saturation_excluded":true,"higher_is_better":true,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":7,"result_count":7,"results":[[2958,0.617,370,7834,8561],[3512,0.572,370,7834,8561],[762,0.6,370,7834,8561],[3412,0.616,370,7834,8561],[3154,0.526,370,7834,8561],[3155,0.596,370,7834,8561],[3492,0.509,370,7834,8561]]},{"id":"agenticbbo-dbtune-2026-paper","name":"AgenticBBO-Bench · DBTune (2026 paper)","group":"数学与科学","metric":"Normalized optimization score","unit":"score","public_history":true,"protocol":{"en":"Seven frontier models on the DBTune task of the fixed five-task frontier challenge. Source normalized score; numerical GP control excluded.","zh":"七个前沿模型在固定五任务挑战的 DBTune 任务上评估。保留原文归一化优化分，排除数值 GP 对照。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.12183","status":"paper_baselines","published_at":"2026-10-08"},"score_max":1,"score_decimals":2,"saturation_excluded":true,"higher_is_better":true,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":7,"result_count":7,"results":[[2958,0.501,370,7834,8561],[3512,0.299,370,7834,8561],[762,0.193,370,7834,8561],[3412,0.364,370,7834,8561],[3154,0.134,370,7834,8561],[3155,0.032,370,7834,8561],[3492,0.039,370,7834,8561]]},{"id":"agenticbbo-median1-2026-paper","name":"AgenticBBO-Bench · Median1 (2026 paper)","group":"数学与科学","metric":"Normalized optimization score","unit":"score","public_history":true,"protocol":{"en":"Seven frontier models on the Median1 task of the fixed five-task frontier challenge. Source normalized score; numerical GP control excluded.","zh":"七个前沿模型在固定五任务挑战的 Median1 任务上评估。保留原文归一化优化分，排除数值 GP 对照。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.12183","status":"paper_baselines","published_at":"2026-10-08"},"score_max":1,"score_decimals":2,"saturation_excluded":true,"higher_is_better":true,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":7,"result_count":7,"results":[[2958,0.642,370,7834,8561],[3512,0.823,370,7834,8561],[762,0.82,370,7834,8561],[3412,0.821,370,7834,8561],[3154,0.823,370,7834,8561],[3155,0.814,370,7834,8561],[3492,0.817,370,7834,8561]]},{"id":"agenticbbo-bbob-2026-paper","name":"AgenticBBO-Bench · BBOB (2026 paper)","group":"数学与科学","metric":"Normalized optimization score","unit":"score","public_history":true,"protocol":{"en":"Seven frontier models on the BBOB task of the fixed five-task frontier challenge. Source normalized score; numerical GP control excluded.","zh":"七个前沿模型在固定五任务挑战的 BBOB 任务上评估。保留原文归一化优化分，排除数值 GP 对照。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.12183","status":"paper_baselines","published_at":"2026-10-08"},"score_max":1,"score_decimals":2,"saturation_excluded":true,"higher_is_better":true,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":7,"result_count":7,"results":[[2958,0.664,370,7834,8561],[3512,0.552,370,7834,8561],[762,0.505,370,7834,8561],[3412,0.608,370,7834,8561],[3154,0.411,370,7834,8561],[3155,0.329,370,7834,8561],[3492,0.159,370,7834,8561]]},{"id":"agenticbbo-bboplace-2026-paper","name":"AgenticBBO-Bench · BBOPlace (2026 paper)","group":"数学与科学","metric":"Normalized optimization score","unit":"score","public_history":true,"protocol":{"en":"Seven frontier models on the BBOPlace task of the fixed five-task frontier challenge. Source normalized score; numerical GP control excluded.","zh":"七个前沿模型在固定五任务挑战的 BBOPlace 任务上评估。保留原文归一化优化分，排除数值 GP 对照。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.12183","status":"paper_baselines","published_at":"2026-10-08"},"score_max":1,"score_decimals":2,"saturation_excluded":true,"higher_is_better":true,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":7,"result_count":7,"results":[[2958,0.534,370,7834,8561],[3512,0.673,370,7834,8561],[762,0.443,370,7834,8561],[3412,0.125,370,7834,8561],[3154,0.275,370,7834,8561],[3155,0.239,370,7834,8561],[3492,0.315,370,7834,8561]]},{"id":"duda-workflow-2026-paper","name":"DUDA-Bench (2026 paper)","group":"Agent 与工具","metric":"Workflow score","unit":"score","public_history":true,"protocol":{"en":"Seven backbone models under the same ReAct framework. Table 2 Workflow Score combines indicator recognition and localization for end-to-end analysis; component scores are not substituted.","zh":"七个底模使用相同 ReAct 框架。取表 2 Workflow Score，以端到端分析的指标识别与定位构成，不替换为单项分数。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.09374","status":"paper_baselines","published_at":"2026-10-07"},"score_max":1,"score_decimals":2,"saturation_excluded":true,"higher_is_better":true,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":7,"result_count":7,"results":[[5727,0.071,370,6334,8562],[5906,0.182,370,6334,8562],[5789,0.135,370,6334,8562],[4979,0.091,370,6334,8562],[3155,0.134,370,6334,8562],[3157,0.277,370,6334,8562],[5300,0.133,370,6334,8562]]},{"id":"sk-bench-2026-paper","name":"sk-bench (2026 paper)","group":"指令与长文","metric":"Six-group macro score","unit":"score","public_history":true,"protocol":{"en":"Full 55-model appendix leaderboard, Slovak zero-shot. Overall averages six group scores; published means over three prompt templates, combining distinct per-task metrics.","zh":"附录完整 55 模型榜，斯洛伐克语零样本。Overall 对六组分数取均值；取三种提示模板的发表均值，保留不同任务指标的综合分。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.09152","status":"paper_baselines","published_at":"2026-10-06"},"score_max":100,"score_decimals":2,"saturation_excluded":true,"higher_is_better":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":55,"result_count":55,"results":[[8563,12.3,370,7834,8564],[8565,15.1,370,7834,8564],[8566,21.5,370,7834,8564],[6005,24.0,370,7834,8564],[8567,22.7,370,7834,8564],[8568,23.2,370,7834,8564],[8569,25.6,370,7834,8564],[8570,21.9,370,7834,8564],[8571,27.8,370,7834,8564],[8572,24.4,370,7834,8564],[6007,29.4,370,7834,8564],[8573,24.6,370,7834,8564],[8574,36.0,370,7834,8564],[8575,36.4,370,7834,8564],[8576,19.0,370,7834,8564],[8577,14.9,370,7834,8564],[8578,27.7,370,7834,8564],[8579,28.6,370,7834,8564],[806,35.1,370,7834,8564],[7637,40.8,370,7834,8564],[8580,39.9,370,7834,8564],[8581,42.7,370,7834,8564],[6013,47.9,370,7834,8564],[8582,46.3,370,7834,8564],[7636,37.2,370,7834,8564],[8583,20.4,370,7834,8564],[6269,47.4,370,7834,8564],[8584,33.5,370,7834,8564],[8585,43.8,370,7834,8564],[8586,54.1,370,7834,8564],[8587,45.2,370,7834,8564],[6114,55.3,370,7834,8564],[4973,54.2,370,7834,8564],[8588,17.0,370,7834,8564],[8589,31.9,370,7834,8564],[8590,32.8,370,7834,8564],[2585,54.3,370,7834,8564],[8591,42.4,370,7834,8564],[8592,47.0,370,7834,8564],[7190,52.2,370,7834,8564],[8593,44.4,370,7834,8564],[796,53.1,370,7834,8564],[2273,36.2,370,7834,8564],[8594,60.0,370,7834,8564],[3495,57.1,370,7834,8564],[6564,57.4,370,7834,8564],[8595,45.0,370,7834,8564],[3186,57.6,370,7834,8564],[3490,55.7,370,7834,8564],[3412,72.6,370,7834,8564],[3196,71.0,370,7834,8564],[2261,68.1,370,7834,8564],[4219,66.7,370,7834,8564],[3572,64.9,370,7834,8564],[5449,63.9,370,7834,8564]]},{"id":"traverse-software-engineering-2026-paper","name":"Traverse · software-engineering (2026 paper)","group":"Agent 与工具","metric":"Exact first-mistake localization","unit":"%","public_history":true,"protocol":{"en":"Six judge models locate the first annotated mistake in 306 incorrect software-engineering trajectories. Exact localization is scored on the same human-annotated trajectories; relaxed step tolerances and outcome-level F1 stay separate.","zh":"六个裁判模型在同一组 306 条software-engineering错误轨迹中定位首个经人工标注的错误步骤。采用精确定位率，宽松步数容差与结果级F1分别保留在原表。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.17930v1","published_at":"2026-09-15","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[3412,0.268,370,8596,8597],[8598,0.1209,370,8596,8597],[2261,0.0719,370,8596,8597],[3196,0.21899999999999997,370,8596,8597],[3492,0.1788,370,8596,8597],[7892,0.1906,370,8596,8597]]},{"id":"traverse-computer-use-2026-paper","name":"Traverse · computer-use (2026 paper)","group":"Agent 与工具","metric":"Exact first-mistake localization","unit":"%","public_history":true,"protocol":{"en":"Six judge models locate the first annotated mistake in 541 incorrect computer-use trajectories. Exact localization is scored on the same human-annotated trajectories; relaxed step tolerances and outcome-level F1 stay separate.","zh":"六个裁判模型在同一组 541 条computer-use错误轨迹中定位首个经人工标注的错误步骤。采用精确定位率，宽松步数容差与结果级F1分别保留在原表。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.17930v1","published_at":"2026-09-15","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[3412,0.2994,370,8596,8597],[8598,0.32159999999999994,370,8596,8597],[2261,0.2458,370,8596,8597],[3196,0.3234,370,8596,8597],[3492,0.2773,370,8596,8597],[7892,0.2274,370,8596,8597]]},{"id":"traverse-science-2026-paper","name":"Traverse · science (2026 paper)","group":"Agent 与工具","metric":"Exact first-mistake localization","unit":"%","public_history":true,"protocol":{"en":"Six judge models locate the first annotated mistake in 38 incorrect science trajectories. Exact localization is scored on the same human-annotated trajectories; relaxed step tolerances and outcome-level F1 stay separate.","zh":"六个裁判模型在同一组 38 条science错误轨迹中定位首个经人工标注的错误步骤。采用精确定位率，宽松步数容差与结果级F1分别保留在原表。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.17930v1","published_at":"2026-09-15","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[3412,0.5789,370,8596,8597],[8598,0.5,370,8596,8597],[2261,0.3947,370,8596,8597],[3196,0.6315999999999999,370,8596,8597],[3492,0.36840000000000006,370,8596,8597],[7892,0.5263,370,8596,8597]]},{"id":"video-holmes2-mcq-2026-paper","name":"Video-HolmesV2 · MCQ overall (2026 paper)","group":"视觉与视频","metric":"Sample-weighted MCQ accuracy","unit":"%","public_history":true,"protocol":{"en":"Answers perception and reasoning questions over long audio-visual narratives. Table 2 Overall is the published sample-weighted answer score. The evidence-aware score in Table 3 is a separate metric with only three models. The compression method shares Qwen2.5-Omni-7B.","zh":"回答长音视频叙事中的感知和推理问题。采用表2按样本数加权的答案分数；表3含证据质量的分数是另一指标，只有三个模型。压缩方法共享Qwen2.5-Omni-7B底模。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.17248v1","published_at":"2026-09-15","status":"paper_baselines"},"model_count":14,"result_count":15,"results":[[353,0.469,370,8599,8600],[766,0.653,370,8599,8600],[764,0.732,370,8599,8600],[6298,0.28,370,8599,8600],[2754,0.381,370,8599,8600],[6342,0.395,370,8599,8600],[6391,0.436,370,8599,8600],[6630,0.47,370,8599,8600],[2728,0.48100000000000004,370,8599,8600],[6238,0.484,370,8599,8600],[6343,0.496,370,8599,8600],[2166,0.502,370,8599,8600],[5843,0.5529999999999999,370,8599,8600],[6138,0.40700000000000003,370,8599,8600],[6138,0.445,370,8601,8600]]},{"id":"benchcompass-pk-cb-2026-paper","name":"BenchCompass · closed-book payment knowledge (2026 paper)","group":"专业工作","metric":"Diagnostic accuracy","unit":"%","public_history":true,"protocol":{"en":"Expert-reviewed payment-domain Pro benchmark, closed-book payment knowledge, 95 cases. All sixteen models use deterministic pass@1 decoding. Closed/open and attacked-input conditions remain separate; the lower-assurance Normal pool is excluded.","zh":"专家复核的支付领域Pro评测，采用closed-book payment knowledge条件下的95个案例，十六个模型按确定性pass@1解码。闭卷、开卷和受干扰输入分别展示，未纳入较低审查等级的Normal题库。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.18270v1","published_at":"2026-09-16","status":"paper_baselines"},"model_count":16,"result_count":16,"results":[[3482,0.8,370,8602,8603],[3196,0.779,370,8602,8603],[3578,0.684,370,8602,8603],[3492,0.6,370,8602,8603],[6564,0.621,370,8602,8603],[3489,0.653,370,8602,8603],[8604,0.59,370,8602,8603],[4424,0.537,370,8602,8603],[3338,0.474,370,8602,8603],[362,0.547,370,8602,8603],[6110,0.579,370,8602,8603],[7851,0.611,370,8602,8603],[1513,0.484,370,8602,8603],[7636,0.495,370,8602,8603],[6268,0.442,370,8602,8603],[7637,0.516,370,8602,8603]]},{"id":"benchcompass-cgr-cb-2026-paper","name":"BenchCompass · closed-book grounded reasoning (2026 paper)","group":"专业工作","metric":"Diagnostic accuracy","unit":"%","public_history":true,"protocol":{"en":"Expert-reviewed payment-domain Pro benchmark, closed-book grounded reasoning, 96 cases. All sixteen models use deterministic pass@1 decoding. Closed/open and attacked-input conditions remain separate; the lower-assurance Normal pool is excluded.","zh":"专家复核的支付领域Pro评测，采用closed-book grounded reasoning条件下的96个案例，十六个模型按确定性pass@1解码。闭卷、开卷和受干扰输入分别展示，未纳入较低审查等级的Normal题库。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.18270v1","published_at":"2026-09-16","status":"paper_baselines"},"model_count":16,"result_count":16,"results":[[3482,0.146,370,8605,8603],[3196,0.365,370,8605,8603],[3578,0.25,370,8605,8603],[3492,0.16699999999999998,370,8605,8603],[6564,0.146,370,8605,8603],[3489,0.156,370,8605,8603],[8604,0.094,370,8605,8603],[4424,0.094,370,8605,8603],[3338,0.083,370,8605,8603],[362,0.083,370,8605,8603],[6110,0.094,370,8605,8603],[7851,0.073,370,8605,8603],[1513,0.052000000000000005,370,8605,8603],[7636,0.021,370,8605,8603],[6268,0.01,370,8605,8603],[7637,0.021,370,8605,8603]]},{"id":"benchcompass-cgr-ob-2026-paper","name":"BenchCompass · open-book grounded reasoning (2026 paper)","group":"专业工作","metric":"Diagnostic accuracy","unit":"%","public_history":true,"protocol":{"en":"Expert-reviewed payment-domain Pro benchmark, open-book grounded reasoning, 96 cases. All sixteen models use deterministic pass@1 decoding. Closed/open and attacked-input conditions remain separate; the lower-assurance Normal pool is excluded.","zh":"专家复核的支付领域Pro评测，采用open-book grounded reasoning条件下的96个案例，十六个模型按确定性pass@1解码。闭卷、开卷和受干扰输入分别展示，未纳入较低审查等级的Normal题库。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.18270v1","published_at":"2026-09-16","status":"paper_baselines"},"model_count":16,"result_count":16,"results":[[3482,0.8959999999999999,370,8606,8603],[3196,0.823,370,8606,8603],[3578,0.823,370,8606,8603],[3492,0.813,370,8606,8603],[6564,0.8440000000000001,370,8606,8603],[3489,0.74,370,8606,8603],[8604,0.8440000000000001,370,8606,8603],[4424,0.75,370,8606,8603],[3338,0.6459999999999999,370,8606,8603],[362,0.688,370,8606,8603],[6110,0.823,370,8606,8603],[7851,0.823,370,8606,8603],[1513,0.698,370,8606,8603],[7636,0.573,370,8606,8603],[6268,0.583,370,8606,8603],[7637,0.5,370,8606,8603]]},{"id":"benchcompass-tia-ob-2026-paper","name":"BenchCompass · open-book task-input attacks (2026 paper)","group":"专业工作","metric":"Diagnostic accuracy","unit":"%","public_history":true,"protocol":{"en":"Expert-reviewed payment-domain Pro benchmark, open-book task-input attacks, 115 cases. All sixteen models use deterministic pass@1 decoding. Closed/open and attacked-input conditions remain separate; the lower-assurance Normal pool is excluded.","zh":"专家复核的支付领域Pro评测，采用open-book task-input attacks条件下的115个案例，十六个模型按确定性pass@1解码。闭卷、开卷和受干扰输入分别展示，未纳入较低审查等级的Normal题库。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.18270v1","published_at":"2026-09-16","status":"paper_baselines"},"model_count":16,"result_count":16,"results":[[3482,0.8170000000000001,370,8607,8603],[3196,0.8,370,8607,8603],[3578,0.748,370,8607,8603],[3492,0.8,370,8607,8603],[6564,0.722,370,8607,8603],[3489,0.7040000000000001,370,8607,8603],[8604,0.7040000000000001,370,8607,8603],[4424,0.609,370,8607,8603],[3338,0.513,370,8607,8603],[362,0.461,370,8607,8603],[6110,0.748,370,8607,8603],[7851,0.609,370,8607,8603],[1513,0.426,370,8607,8603],[7636,0.374,370,8607,8603],[6268,0.365,370,8607,8603],[7637,0.29600000000000004,370,8607,8603]]},{"id":"ecthr-npd-neural-mae-2026-paper","name":"ECtHR-NPD · neural models, MAE in EUR (2026 paper)","group":"专业工作","metric":"MAE (EUR)","unit":"raw","public_history":true,"protocol":{"en":"Predicts non-pecuniary damages in nominal euros on the chronological test pool of 2,897 judgments. This panel contains the neural retrieval, encoder and prompted-decoder checkpoints. Prompting and fusion variants are configurations. Constant, non-neural and additional-input diagnostic baselines remain in the source.","zh":"在按时间划分的2,897份判决测试集上预测非财产损害赔偿，单位为名义欧元。展示神经检索、编码器和提示式解码器检查点；提示方式和融合方式记为配置。常量、非神经模型和额外输入诊断对照保留在原表。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.18908v1","published_at":"2026-09-16","status":"paper_baselines"},"higher_is_better":false,"model_count":8,"result_count":16,"results":[[8608,10226.0,370,8609,8610],[8608,10177.0,370,8611,8610],[8612,10133.0,370,8613,8610],[8612,10074.0,370,8614,8610],[8615,10642.0,370,8613,8610],[8615,10244.0,370,8614,8610],[4070,16598.0,370,8616,8610],[2584,22235.0,370,8616,8610],[6412,16400.0,370,8616,8610],[5343,15678.0,370,8616,8610],[3184,25438.0,370,8616,8610],[4070,10782.0,370,8617,8610],[2584,12905.0,370,8617,8610],[6412,19080.0,370,8617,8610],[5343,16748.0,370,8617,8610],[3184,16266.0,370,8617,8610]]},{"id":"hearincontext-mandarin-no-context-2026-paper","name":"HearInContext · Mandarin, no context (2026 paper)","group":"知识与推理","metric":"CER","unit":"%","public_history":true,"protocol":{"en":"Mandarin homophone transcription without dialogue context. Seven ASR models are compared on the same ambiguous audio. CER is lower-is-better; contextual columns have fewer supported models and are not pooled with this panel.","zh":"在没有对话上下文的条件下转写Mandarin同音词语音，七个ASR模型使用同一批歧义音频。CER越低越好；能使用上下文的模型数量更少，不与本面板合并。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.18680v1","published_at":"2026-09-16","status":"paper_baselines"},"higher_is_better":false,"model_count":7,"result_count":7,"results":[[8618,0.0797,370,8619,8620],[8621,0.0788,370,8619,8620],[8622,0.1045,370,8619,8620],[8623,0.0759,370,8619,8620],[8624,0.0738,370,8619,8620],[8625,0.08289999999999999,370,8619,8620],[8626,0.0786,370,8619,8620]]},{"id":"hearincontext-english-no-context-2026-paper","name":"HearInContext · English, no context (2026 paper)","group":"知识与推理","metric":"WER","unit":"%","public_history":true,"protocol":{"en":"English homophone transcription without dialogue context. Seven ASR models are compared on the same ambiguous audio. WER is lower-is-better; contextual columns have fewer supported models and are not pooled with this panel.","zh":"在没有对话上下文的条件下转写English同音词语音，七个ASR模型使用同一批歧义音频。WER越低越好；能使用上下文的模型数量更少，不与本面板合并。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.18680v1","published_at":"2026-09-16","status":"paper_baselines"},"higher_is_better":false,"model_count":7,"result_count":7,"results":[[8618,0.11800000000000001,370,8619,8620],[8621,0.1231,370,8619,8620],[8622,0.1042,370,8619,8620],[8623,0.10189999999999999,370,8619,8620],[8624,0.1008,370,8619,8620],[8625,0.11349999999999999,370,8619,8620],[8626,0.11720000000000001,370,8619,8620]]},{"id":"muse-cultural-identification-2026-paper","name":"MUSE · cultural identification (2026 paper)","group":"视觉与视频","metric":"Multiple-choice accuracy","unit":"%","public_history":true,"protocol":{"en":"Thirty vision-language models identify cultural elements in artistic educational images. The selected task supplies three distractors from categories different from the ground-truth label; all other MUSE tasks remain separate.","zh":"三十个视觉语言模型识别艺术教学图像中的文化元素。本子任务提供三个来自不同于正确标签类别的干扰选项，其他MUSE任务分别保留在原表。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.19088v1","published_at":"2026-09-16","status":"paper_baselines"},"model_count":30,"result_count":30,"results":[[4979,0.765,370,8627,8628],[8629,0.6,370,8627,8628],[8630,0.51,370,8627,8628],[8631,0.505,370,8627,8628],[8632,0.585,370,8627,8628],[8633,0.465,370,8627,8628],[353,0.395,370,8627,8628],[8634,0.515,370,8627,8628],[8635,0.545,370,8627,8628],[8636,0.55,370,8627,8628],[8637,0.48,370,8627,8628],[8638,0.475,370,8627,8628],[2911,0.43,370,8627,8628],[7499,0.5,370,8627,8628],[8639,0.49,370,8627,8628],[8640,0.475,370,8627,8628],[7385,0.47,370,8627,8628],[8641,0.515,370,8627,8628],[8642,0.45,370,8627,8628],[8643,0.415,370,8627,8628],[6977,0.44,370,8627,8628],[8644,0.435,370,8627,8628],[8645,0.37,370,8627,8628],[8646,0.345,370,8627,8628],[8647,0.31,370,8627,8628],[8648,0.295,370,8627,8628],[8649,0.395,370,8627,8628],[8650,0.19,370,8627,8628],[8651,0.19,370,8627,8628],[5817,0.34,370,8627,8628]]},{"id":"pact-compliance-index-2026-paper","name":"PACT · compliance index (2026 paper)","group":"安全与可靠性","metric":"PACTScore","unit":"score","public_history":true,"protocol":{"en":"Twenty-two models answer 3,364 enterprise-rule scenarios three times each. PACTScore weights first-turn and follow-up pass-cubed indicators, with a separate one-turn rule. Replications judged unclear are excluded by the author protocol; the composite stays on its native 0–1 scale.","zh":"二十二个模型对3,364个企业规则情境各回答三次。PACTScore对首轮和追问的三次全通过指标加权，单轮样本单独处理。按作者协议排除无法判定的重复回答，综合分保留0–1原生刻度。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.18605v1","published_at":"2026-09-16","status":"paper_baselines"},"score_max":1,"saturation_excluded":true,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"score_decimals":2,"model_count":22,"result_count":22,"results":[[3203,0.944,370,8652,8653],[3763,0.943,370,8652,8653],[325,0.937,370,8652,8653],[3187,0.935,370,8652,8653],[3408,0.934,370,8652,8653],[203,0.931,370,8652,8653],[3424,0.929,370,8652,8653],[8654,0.929,370,8652,8653],[227,0.926,370,8652,8653],[268,0.917,370,8652,8653],[5619,0.917,370,8652,8653],[3185,0.913,370,8652,8653],[8655,0.913,370,8652,8653],[3419,0.909,370,8652,8653],[3005,0.904,370,8652,8653],[2278,0.903,370,8652,8653],[5648,0.903,370,8652,8653],[3493,0.87,370,8652,8653],[8656,0.834,370,8652,8653],[8657,0.787,370,8652,8653],[587,0.562,370,8652,8653],[1921,0.484,370,8652,8653]]},{"id":"geostore-recall1-2026-paper","name":"GeoStore (2026 paper)","group":"视觉与视频","metric":"R@1","unit":"%","score_max":100,"higher_is_better":true,"public_history":true,"protocol":{"en":"Matches close-up storefront photos to geotagged street-view images. Recall@1 is measured on the open-set test set; image size and re-ranking settings remain distinct.","zh":"将近景店面照片匹配到带地理位置的街景图像。采用开放集测试中的首位检索召回率，并保留输入尺寸与重排序设置。"},"source":{"name":"GeoStore authors","status":"paper_baselines","url":"https://arxiv.org/html/2609.02012v1#S4.T1","published_at":"2026-09-02"},"model_count":8,"result_count":11,"results":[[8658,0.040999999999999995,370,8659,8660],[8661,0.11199999999999999,370,8662,8660],[8663,0.129,370,8664,8660],[8665,0.135,370,8666,8660],[8667,0.147,370,8668,8660],[8669,0.188,370,8670,8660],[8669,0.182,370,8671,8660],[8672,0.141,370,8673,8660],[8672,0.2,370,8674,8660],[8675,0.182,370,8676,8660],[8675,0.253,370,8677,8660]]},{"id":"rig-common-2026-paper","name":"RIG-Bench · common tasks (2026 paper)","group":"生成与创作","metric":"Common Rubric","unit":"score","score_max":100,"higher_is_better":true,"public_history":true,"protocol":{"en":"Infers rules from visual examples and generates an image that applies them. The common-rubric score covers the nine subtasks supported by every evaluated model family.","zh":"从视觉示例归纳规则，再生成符合规则的图像。采用所有参评模型家族共同支持的九类任务评分。"},"source":{"name":"RIG-BENCH authors","status":"paper_baselines","url":"https://arxiv.org/html/2609.02864v1#S4.T3","published_at":"2026-09-02"},"model_count":9,"result_count":9,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"results":[[6379,20.79,370,8678,8679],[6358,16.76,370,8678,8679],[6362,20.66,370,8678,8679],[6360,26.35,370,8678,8679],[8680,59.71,370,8678,8679],[8681,53.4,370,8678,8679],[8682,37.57,370,8678,8679],[8683,20.73,370,8678,8679],[8684,21.56,370,8678,8679]]},{"id":"leakage-localization-2026-paper","name":"LeakageBench · localization (2026 paper)","group":"安全与可靠性","metric":"Page-macro localization F1","unit":"score","score_max":1,"higher_is_better":true,"public_history":true,"protocol":{"en":"Locates personal identifiers on 500 document images. Page-macro localization F1 uses IoU0.75 and the full annotation schema; unsupported types and invalid outputs count as misses.","zh":"在500幅文档图像中定位个人标识信息。采用IoU0.75下的页面宏平均定位F1及完整标注范围；不支持的类别和无效输出按漏检处理。"},"source":{"name":"LeakageBench authors","status":"paper_baselines","url":"https://arxiv.org/html/2609.02207v1#S6.T3","published_at":"2026-09-02"},"model_count":9,"result_count":10,"saturation_excluded":true,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"results":[[8685,0.216,370,8686,8687],[8688,0.245,370,8686,8687],[8689,0.288,370,8686,8687],[8690,0.156,370,8686,8687],[8691,0.164,370,8686,8687],[6343,0.073,370,8692,8687],[2706,0.012,370,8692,8687],[3184,0.044,370,8692,8687],[3196,0.09,370,8692,8687],[3196,0.249,370,8693,8687]]},{"id":"vibe-strategy-2026-paper","name":"VIBE-Bench · personalized strategy (2026 paper)","group":"知识与推理","metric":"Strategy accuracy","unit":"%","score_max":100,"higher_is_better":true,"public_history":true,"protocol":{"en":"Infers a user’s query-specific preferences from profile and dialogue history, then selects a personalized response strategy. The cohort keeps all six backbones and their personalization methods separate.","zh":"根据用户画像与对话历史推断当前偏好，再选择个性化回应策略。保留六个基础模型及各自的个性化方法配置。"},"source":{"name":"VIBE-Bench authors","status":"paper_baselines","url":"https://arxiv.org/html/2609.00921v1#A11.T10","published_at":"2026-09-01"},"model_count":11,"result_count":36,"results":[[1543,0.1953,370,8694,8695],[8696,0.1953,370,8694,8695],[8697,0.20309999999999997,370,8694,8695],[8698,0.2266,370,8694,8695],[6301,0.2188,370,8694,8695],[771,0.25,370,8694,8695],[1543,0.1484,370,8699,8695],[8696,0.1641,370,8699,8695],[8697,0.1875,370,8699,8695],[8698,0.20309999999999997,370,8699,8695],[6301,0.17190000000000003,370,8699,8695],[771,0.2266,370,8699,8695],[1543,0.25,370,8700,8695],[8696,0.23440000000000003,370,8700,8695],[8697,0.2188,370,8700,8695],[8698,0.2812,370,8700,8695],[6301,0.2109,370,8700,8695],[771,0.25780000000000003,370,8700,8695],[1543,0.1797,370,8701,8695],[8696,0.1953,370,8701,8695],[8697,0.1953,370,8701,8695],[8698,0.2422,370,8701,8695],[6301,0.2266,370,8701,8695],[771,0.25,370,8701,8695],[1543,0.23440000000000003,370,8702,8695],[8696,0.2422,370,8702,8695],[8697,0.2188,370,8702,8695],[8698,0.23440000000000003,370,8702,8695],[6301,0.2188,370,8702,8695],[771,0.2109,370,8702,8695],[8039,0.1562,370,8703,8695],[5148,0.2188,370,8703,8695],[8704,0.17190000000000003,370,8703,8695],[8705,0.1797,370,8703,8695],[4860,0.1953,370,8703,8695],[8706,0.1562,370,8703,8695]]},{"id":"phantom-public-2026-paper","name":"PHANTOM-Bench · public decisions (2026 paper)","group":"安全与可靠性","metric":"Public decision accuracy","unit":"%","public_history":true,"protocol":{"en":"Nine models, public decisions after ambiguous social signals. Accuracy over all scenario-agent pairs. Private and transparency-recovery conditions and belief-error metrics are separate.","zh":"九个模型在模糊社会信号后的公开决策准确率，分母为全部情景与智能体配对。私下判断、透明信息恢复及信念误差另列。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.08971","status":"paper_baselines","published_at":"2026-11-06"},"model_count":9,"result_count":9,"results":[[2129,0.612,370,7834,8707],[764,0.622,370,7834,8707],[766,0.66,370,7834,8707],[8708,0.421,370,7834,8707],[8097,0.579,370,7834,8707],[1621,0.593,370,7834,8707],[8096,0.47,370,7834,8707],[587,0.598,370,7834,8707],[8709,0.292,370,7834,8707]]},{"id":"alienbench-2026-paper","name":"AlienBench (2026 paper)","group":"生成与创作","metric":"Ward Departure Score","unit":"score","score_max":10,"score_decimals":2,"saturation_excluded":true,"scale_label":{"en":"Score / 10","zh":"分数 / 10"},"public_history":true,"protocol":{"en":"Ten biological-feature departures scored 0–10. Each of ten models produces 180 alien descriptions across six prompts; three model judges score each description. Table 3 overall mean, visually verified on PDF page 6. All 5,400 judge calls returned valid JSON.","zh":"按十项生物特征偏离程度计 0–10 分。十个模型各在六种提示下生成 180 份外星生物描述，每份由三个模型裁判评分。取表 3 Overall，已目视核对 PDF 第六页；全部 5,400 次裁判调用均返回有效 JSON。"},"source":{"name":"AlienBench authors","url":"https://arxiv.org/abs/2611.09094","status":"paper_baselines","published_at":"2026-11-06"},"model_count":10,"result_count":10,"results":[[2959,5.14,370,8710,8711],[2958,4.36,370,8710,8711],[2976,3.28,370,8710,8711],[2983,4.72,370,8710,8711],[2968,5.42,370,8710,8711],[2966,4.97,370,8710,8711],[362,4.84,370,8710,8711],[2992,5.41,370,8710,8711],[8712,5.23,370,8710,8711],[8713,6.55,370,8710,8711]]},{"id":"docattribench-grounded-answer-2026-paper","name":"DocAttriBench · grounded answer (2026 paper)","group":"视觉与视频","metric":"Mean grounded answer accuracy","unit":"%","public_history":true,"protocol":{"en":"Grounded document answers must pass both answer accuracy and evidence-box F1 thresholds of 0.5. The published Avg. aggregates seven source subsets; all 19 evaluated checkpoints are retained, including specialized attribution models.","zh":"文档问答的答案准确度和证据框F1都须超过0.5。保留原表七个来源子集的Avg.及全部十九个检查点，包括专门训练的归因模型。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.20574v1","published_at":"2026-09-17","status":"paper_baselines"},"model_count":19,"result_count":19,"results":[[2793,0.001,370,5574,8714],[2766,0.006999999999999999,370,5574,8714],[6246,0.021,370,5574,8714],[2771,0.009000000000000001,370,5574,8714],[6341,0.043,370,5574,8714],[2774,0.005,370,5574,8714],[2728,0.026000000000000002,370,5574,8714],[2654,0.053,370,5574,8714],[2754,0.03,370,5574,8714],[6342,0.094,370,5574,8714],[2714,0.023,370,5574,8714],[2706,0.032,370,5574,8714],[2642,0.086,370,5574,8714],[6391,0.091,370,5574,8714],[6343,0.254,370,5574,8714],[8715,0.243,370,8716,8714],[8717,0.33799999999999997,370,8716,8714],[8718,0.37,370,8716,8714],[8719,0.33399999999999996,370,8716,8714]]},{"id":"koneobench-word-2026-paper","name":"KoNeoBench · word-only (2026 paper)","group":"知识与推理","metric":"Cloze accuracy","unit":"%","public_history":true,"protocol":{"en":"Expert-reviewed Korean neologisms from 2020 onward; multiple-choice context cloze, word-only variant. Nine models share the same questions; mean and range summaries are excluded.","zh":"采用专家审校的2020年以后韩语新词，考查根据上下文选择被遮蔽的新词。词语与带助词两个版本分开展示，保留九个模型，排除均值与极差。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.19916v1","published_at":"2026-09-17","status":"paper_baselines"},"model_count":9,"result_count":9,"results":[[345,0.5462,370,8720,8721],[3184,0.6616,370,8720,8721],[8722,0.4434,370,8720,8721],[8723,0.37979999999999997,370,8720,8721],[8724,0.47090000000000004,370,8720,8721],[8725,0.5286,370,8720,8721],[797,0.4047,370,8720,8721],[4070,0.474,370,8720,8721],[8726,0.3447,370,8720,8721]]},{"id":"koneobench-particle-2026-paper","name":"KoNeoBench · word with particle (2026 paper)","group":"知识与推理","metric":"Cloze accuracy","unit":"%","public_history":true,"protocol":{"en":"Expert-reviewed Korean neologisms from 2020 onward; multiple-choice context cloze, word with particle variant. Nine models share the same questions; mean and range summaries are excluded.","zh":"采用专家审校的2020年以后韩语新词，考查根据上下文选择被遮蔽的新词。词语与带助词两个版本分开展示，保留九个模型，排除均值与极差。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.19916v1","published_at":"2026-09-17","status":"paper_baselines"},"model_count":9,"result_count":9,"results":[[345,0.5221,370,8727,8721],[3184,0.6297,370,8727,8721],[8722,0.4182,370,8727,8721],[8723,0.3669,370,8727,8721],[8724,0.45399999999999996,370,8727,8721],[8725,0.5084000000000001,370,8727,8721],[797,0.38,370,8727,8721],[4070,0.4664,370,8727,8721],[8726,0.3072,370,8727,8721]]},{"id":"safari-hazard-analysis-2026-paper","name":"SAFARI · hazard analysis (2026 paper)","group":"专业工作","metric":"Mean reference-anchored judge score","unit":"score","public_history":true,"protocol":{"en":"Automotive hazard analysis on 1,000 held-out industrial HARA cases from a 3,000-case expert-reviewed dataset. The mean combines three reference-anchored judge dimensions. Nine models each use few-shot and chain-of-thought configurations; scores remain native rubric values.","zh":"在经专家复核的3,000个工业HARA案例中，以1,000个测试案例评估汽车危险分析。采用三个参照答案裁判维度的均值，九个模型分别使用少样本和思维链配置，保留原生量表分。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.20584v1","published_at":"2026-09-17","status":"paper_baselines"},"score_max":1,"saturation_excluded":true,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"score_decimals":2,"model_count":9,"result_count":18,"results":[[3196,0.653,370,8728,8729],[3196,0.555,370,71,8729],[4958,0.598,370,8728,8729],[4958,0.563,370,71,8729],[2129,0.715,370,8728,8729],[2129,0.45,370,71,8729],[3185,0.685,370,8728,8729],[3185,0.506,370,71,8729],[3201,0.714,370,8728,8729],[3201,0.535,370,71,8729],[2576,0.691,370,8728,8729],[2576,0.501,370,71,8729],[2582,0.656,370,8728,8729],[2582,0.555,370,71,8729],[3186,0.726,370,8728,8729],[3186,0.56,370,71,8729],[3190,0.69,370,8728,8729],[3190,0.554,370,71,8729]]},{"id":"quranicmmlu-mcq-2026-paper","name":"QuranicMMLU · multiple choice (2026 paper)","group":"知识与推理","metric":"MCQ accuracy","unit":"%","public_history":true,"protocol":{"en":"Human-reviewed questions on Quranic Arabic, organized by linguistic content and Bloom cognitive levels. This panel preserves the twelve models’ four-option multiple-choice accuracy; the open-ended five-point judge rubric is a separate metric.","zh":"经人工复核的古兰经阿拉伯语问题，按语言学内容与认知层次分类。保留十二个模型的四选一正确率，开放问答的五分裁判量表另属其他指标。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.22038v1","published_at":"2026-09-18","status":"paper_baselines"},"model_count":12,"result_count":12,"results":[[8730,0.9640000000000001,370,8731,8732],[8733,0.9329999999999999,370,8731,8732],[3763,0.9179999999999999,370,8731,8732],[8734,0.912,370,8731,8732],[2590,0.897,370,8731,8732],[784,0.883,370,8731,8732],[4070,0.845,370,8731,8732],[8735,0.8009999999999999,370,8731,8732],[6342,0.763,370,8731,8732],[8736,0.732,370,8731,8732],[6298,0.6970000000000001,370,8731,8732],[8737,0.695,370,8731,8732]]},{"id":"vidomni-verification-f1-2026-paper","name":"VidOmni-Bench · event verification (2026 paper)","group":"视觉与视频","metric":"Verification F1","unit":"score","public_history":true,"protocol":{"en":"Verifies sentence-level events against 500 videos spanning four seconds to ninety minutes. Human annotations distinguish supported and hallucinated captions. F1 is preserved as a native score; video-only and video-plus-audio settings are explicit model configurations.","zh":"对500段、时长四秒至九十分钟的视频进行逐句事件核验，以人工标注区分有据描述与幻觉。F1保留原生分数，纯视频与音视频输入作为明确配置。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.21521v1","published_at":"2026-09-18","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":13,"result_count":16,"results":[[8738,9.6,370,8739,8740],[8741,2.3,370,8739,8740],[8742,14.7,370,8739,8740],[8743,19.4,370,8739,8740],[8744,14.1,370,8739,8740],[5883,3.6,370,8739,8740],[8745,12.7,370,8739,8740],[7513,14.7,370,8739,8740],[7519,21.7,370,8739,8740],[8746,1.1,370,8739,8740],[8746,0.1,370,8747,8740],[8748,5.8,370,8739,8740],[8748,3.7,370,8747,8740],[3424,39.8,370,8739,8740],[3424,41.7,370,8747,8740],[46,28.9,370,8739,8740]]},{"id":"mula-audio-accuracy-2026-paper","name":"MuLA-Bench · long-form audio (2026 paper)","group":"知识与推理","metric":"Fixed-denominator accuracy","unit":"%","public_history":true,"protocol":{"en":"Ten audio-language models answer 5,038 questions over 1,769 recordings in sixteen languages. Route-specific binary rubrics score final answers; temporal tolerances are deterministic. The full denominator includes persistent inference failures and context-limit failures, as specified by the end-to-end protocol.","zh":"十个音频语言模型回答十六种语言、1,769段录音上的5,038道题。按答案结构采用二元准则，时间容差由程序判断。端到端协议把持续推理失败和上下文限制导致的失败计入完整分母。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.23416v1","published_at":"2026-09-20","status":"paper_baselines"},"model_count":10,"result_count":10,"results":[[2983,0.7340000000000001,370,8749,8750],[3386,0.7312000000000001,370,8749,8750],[3412,0.6131,370,8749,8750],[6724,0.6092,370,8749,8750],[6723,0.46549999999999997,370,8749,8750],[8751,0.5690999999999999,370,8749,8750],[3407,0.3398,370,8749,8750],[8752,0.1348,370,8749,8750],[6139,0.2491,370,8749,8750],[8753,0.136,370,8749,8750]]},{"id":"chartjudge-perception-2026-paper","name":"ChartJudgeBench · perception alignment (2026 paper)","group":"视觉与视频","metric":"Mean pairwise accuracy","unit":"%","public_history":true,"protocol":{"en":"Judges choose the better chart from controlled pairs spanning seven visual, data-fidelity and layout dimensions. The table Avg. is retained. Deterministic perturbation labels were checked in a ten-person human study. Thinking modes remain configurations.","zh":"模型从受控图表对中选择更好的图，覆盖视觉、数据忠实度与布局的七个维度，保留原表Avg.。确定性扰动标签经过十人研究验证，思考模式作为配置保留。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.24210v1","published_at":"2026-09-21","status":"paper_baselines"},"model_count":26,"result_count":26,"results":[[6130,0.95,370,8754,8755],[340,0.797,370,8754,8755],[8756,0.645,370,8754,8755],[6704,0.536,370,8754,8755],[8757,0.46,370,3143,8755],[6299,0.32,370,3143,8755],[2622,0.27699999999999997,370,3143,8755],[8758,0.078,370,3143,8755],[2707,0.053,370,3143,8755],[2485,0.14400000000000002,370,8105,8755],[2750,0.35100000000000003,370,8105,8755],[8759,0.002,370,8105,8755],[2754,0.049,370,8105,8755],[2166,0.561,370,8105,8755],[8760,0.321,370,8105,8755],[8761,0.45299999999999996,370,8105,8755],[8762,0.414,370,8105,8755],[8763,0.342,370,8105,8755],[2714,0.21,370,8105,8755],[2774,0.162,370,8105,8755],[8764,0.07200000000000001,370,8105,8755],[8765,0.248,370,8105,8755],[6342,0.402,370,8105,8755],[6343,0.6890000000000001,370,8105,8755],[2654,0.174,370,8105,8755],[2642,0.47,370,8105,8755]]},{"id":"fininteract-full-2026-paper","name":"FinInteract · full interaction (2026 paper)","group":"专业工作","metric":"Intended-interpretation accuracy","unit":"%","public_history":true,"protocol":{"en":"Financial ambiguity resolution with search and user clarification. This full panel uses 173 cases per model and scores against the expert-verified intended reading. The full corpus and cross-vendor pilot are kept separate; human controls are excluded.","zh":"通过搜索和追问消解金融问题的歧义，按专家核实的预期含义评分。本full面板每个模型使用173例，完整数据与跨厂商小样本试验分开展示，人类对照不进入排行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.24002v1","published_at":"2026-09-21","status":"paper_baselines"},"model_count":9,"result_count":9,"results":[[2058,0.20199999999999999,370,8766,8767],[353,0.046,370,8766,8767],[4064,0.0,370,8766,8767],[5126,0.121,370,8766,8767],[5344,0.249,370,8766,8767],[6301,0.225,370,8766,8767],[786,0.084,370,8766,8767],[793,0.11599999999999999,370,8766,8767],[2590,0.289,370,8766,8767]]},{"id":"fininteract-pilot-2026-paper","name":"FinInteract · pilot interaction (2026 paper)","group":"专业工作","metric":"Intended-interpretation accuracy","unit":"%","public_history":true,"protocol":{"en":"Financial ambiguity resolution with search and user clarification. This pilot panel uses 50 cases per model and scores against the expert-verified intended reading. The full corpus and cross-vendor pilot are kept separate; human controls are excluded.","zh":"通过搜索和追问消解金融问题的歧义，按专家核实的预期含义评分。本pilot面板每个模型使用50例，完整数据与跨厂商小样本试验分开展示，人类对照不进入排行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.24002v1","published_at":"2026-09-21","status":"paper_baselines"},"model_count":7,"result_count":7,"results":[[4971,0.3,370,8768,8767],[5789,0.44,370,8768,8767],[4986,0.32,370,8768,8767],[227,0.4,370,8768,8767],[3155,0.34,370,8768,8767],[7895,0.24,370,8768,8767],[5115,0.3,370,8768,8767]]},{"id":"mcp-granite-completion-2026-paper","name":"MCP-GRANITE · task completion (2026 paper)","group":"Agent 与工具","metric":"Task completion","unit":"%","public_history":true,"protocol":{"en":"Nine local models complete 81 IoT scenarios under four tool granularities, repeated three times: 972 trials per model. The reported completion mean includes model-side schema errors and timeouts. Each checkpoint uses the author’s recorded quantization.","zh":"九个本地模型在四种工具粒度下各执行81个物联网情境，重复三次，每个模型972次。原表完成率包含模型输出格式错误与超时，保留作者记录的各检查点量化配置。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.24161v1","published_at":"2026-09-21","status":"paper_baselines"},"model_count":9,"result_count":9,"results":[[6301,0.76,370,8769,8770],[613,0.58,370,8771,8770],[8096,0.49,370,8771,8770],[7345,0.42,370,8771,8770],[8772,0.4,370,8771,8770],[8773,0.39,370,8771,8770],[5343,0.34,370,8774,8770],[8775,0.25,370,8776,8770],[8777,0.19,370,8778,8770]]},{"id":"tapvid-mv-world-2026-paper","name":"TAPVid-MV · world-space tracking (2026 paper)","group":"视觉与视频","metric":"World-space3D tracking accuracy","unit":"%","score_max":100,"higher_is_better":true,"public_history":true,"protocol":{"en":"Tracks points in 3D across multiple camera views. The score averages world-space tracking accuracy over seven datasets with query-median scaling. Privileged oracle-query variants are excluded; source training-overlap flags remain documented in the paper.","zh":"跨多个相机视角追踪三维点。采用查询中位数尺度校准，并对七个数据集的世界坐标追踪准确率取平均。不纳入使用特权标准查询的变体；论文保留了训练数据重合标记。"},"source":{"name":"TAPVid-MV authors","status":"paper_baselines","url":"https://arxiv.org/html/2609.01899v1#A1.T1","published_at":"2026-09-01"},"model_count":25,"result_count":25,"results":[[8779,0.16399999999999998,370,8780,8781],[8782,0.18,370,8780,8781],[8783,0.188,370,8780,8781],[8784,0.188,370,8780,8781],[8785,0.18899999999999997,370,8780,8781],[8786,0.191,370,8780,8781],[8787,0.195,370,8780,8781],[8788,0.195,370,8780,8781],[8789,0.195,370,8780,8781],[8790,0.201,370,8780,8781],[8791,0.203,370,8780,8781],[8792,0.205,370,8780,8781],[8793,0.205,370,8780,8781],[8794,0.20800000000000002,370,8780,8781],[8795,0.20800000000000002,370,8780,8781],[8796,0.12300000000000001,370,8780,8781],[8797,0.159,370,8780,8781],[8798,0.17800000000000002,370,8780,8781],[8799,0.192,370,8780,8781],[8800,0.228,370,8780,8781],[8801,0.191,370,8780,8781],[8802,0.065,370,8780,8781],[8803,0.22,370,8780,8781],[8804,0.237,370,8805,8781],[8806,0.10300000000000001,370,8807,8781]]},{"id":"aib-humanlike-2026-paper","name":"AIB · human-like illusion responses (2026 paper)","group":"知识与推理","metric":"Mean human-like accuracy","unit":"score","score_max":1,"higher_is_better":true,"public_history":true,"protocol":{"en":"Measures whether audio models report the percept heard by humans on auditory illusions. The native human-like response rate averages physics and knowledge-dependent illusions; it is not acoustic-ground-truth accuracy.","zh":"评测音频模型面对听觉错觉时，是否报告人类听到的知觉。原生人类相似反应率汇总物理与知识相关错觉，不等同于声学真实答案的准确率。"},"source":{"name":"AIB authors","status":"paper_baselines","url":"https://arxiv.org/html/2609.02277v1#S5.T2","published_at":"2026-09-02"},"model_count":10,"result_count":13,"saturation_excluded":true,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"results":[[8808,0.218,370,8809,8810],[7731,0.38,370,8811,8810],[8812,0.519,370,8811,8810],[8813,0.451,370,8811,8810],[7728,0.595,370,8811,8810],[8814,0.355,370,8811,8810],[8815,0.362,370,8811,8810],[8816,0.312,370,8811,8810],[8817,0.382,370,8811,8810],[3412,0.612,370,8811,8810],[8808,0.301,370,8818,8810],[8815,0.405,370,8819,8810],[8816,0.04,370,8819,8810]]},{"id":"fdebench-resolution-2026-paper","name":"FDE-Bench · deployment resolution (2026 paper)","group":"编程","metric":"Resolution rate","unit":"%","public_history":true,"protocol":{"en":"Seven models each complete 136 valid deployment episodes under one four-tool scaffold. Submitted Docker, Compose and Kubernetes artifacts are rebuilt in pristine environments and checked for build, readiness, behavior and specification conformance. Budget-terminated attempts remain in the denominator; synthetic floors and ensemble unions are excluded.","zh":"七个模型分别在相同四工具框架下完成136次有效部署评测。提交的Docker、Compose和Kubernetes配置在全新环境重建，检查构建、就绪、行为与规格符合度。达到预算上限的尝试保留在分母，排除模板对照与事后并集。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.27571v1","published_at":"2026-09-23","status":"paper_baselines"},"model_count":7,"result_count":7,"results":[[5136,0.75,370,8820,8821],[5727,0.6990000000000001,370,8820,8821],[5789,0.728,370,8820,8821],[3185,0.6990000000000001,370,8820,8821],[5616,0.632,370,8820,8821],[3201,0.662,370,8820,8821],[8822,0.529,370,8820,8821]]},{"id":"nafbench-credulous-stable-2026-paper","name":"NAF-Bench · credulous-stable (2026 paper)","group":"知识与推理","metric":"Semantic-following accuracy","unit":"%","public_history":true,"protocol":{"en":"Six models follow the explicitly specified credulous-stable semantics on the main fixed-complexity set of 120 generated logic programs. Labels are certified by the corresponding logic solvers; semantic regimes remain separate.","zh":"六个模型在主评测集的120个固定复杂度逻辑程序上遵循指定的否定语义。答案由相应逻辑求解器认证，四种语义分别展示。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.27517v1","published_at":"2026-09-23","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[3568,1.0,370,8823,8824],[762,1.0,370,8823,8824],[5929,1.0,370,8823,8824],[8825,0.72,370,8823,8824],[4831,0.62,370,8823,8824],[8826,0.72,370,8823,8824]]},{"id":"nafbench-skeptical-stable-2026-paper","name":"NAF-Bench · skeptical-stable (2026 paper)","group":"知识与推理","metric":"Semantic-following accuracy","unit":"%","public_history":true,"protocol":{"en":"Six models follow the explicitly specified skeptical-stable semantics on the main fixed-complexity set of 120 generated logic programs. Labels are certified by the corresponding logic solvers; semantic regimes remain separate.","zh":"六个模型在主评测集的120个固定复杂度逻辑程序上遵循指定的否定语义。答案由相应逻辑求解器认证，四种语义分别展示。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.27517v1","published_at":"2026-09-23","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[3568,1.0,370,8827,8824],[762,1.0,370,8827,8824],[5929,0.98,370,8827,8824],[8825,0.61,370,8827,8824],[4831,0.67,370,8827,8824],[8826,0.64,370,8827,8824]]},{"id":"nafbench-well-founded-2026-paper","name":"NAF-Bench · well-founded (2026 paper)","group":"知识与推理","metric":"Semantic-following accuracy","unit":"%","public_history":true,"protocol":{"en":"Six models follow the explicitly specified well-founded semantics on the main fixed-complexity set of 120 generated logic programs. Labels are certified by the corresponding logic solvers; semantic regimes remain separate.","zh":"六个模型在主评测集的120个固定复杂度逻辑程序上遵循指定的否定语义。答案由相应逻辑求解器认证，四种语义分别展示。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.27517v1","published_at":"2026-09-23","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[3568,1.0,370,8828,8824],[762,1.0,370,8828,8824],[5929,0.81,370,8828,8824],[8825,0.74,370,8828,8824],[4831,0.36,370,8828,8824],[8826,0.31,370,8828,8824]]},{"id":"nafbench-sldnf-2026-paper","name":"NAF-Bench · sldnf (2026 paper)","group":"知识与推理","metric":"Semantic-following accuracy","unit":"%","public_history":true,"protocol":{"en":"Six models follow the explicitly specified sldnf semantics on the main fixed-complexity set of 120 generated logic programs. Labels are certified by the corresponding logic solvers; semantic regimes remain separate.","zh":"六个模型在主评测集的120个固定复杂度逻辑程序上遵循指定的否定语义。答案由相应逻辑求解器认证，四种语义分别展示。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.27517v1","published_at":"2026-09-23","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[3568,1.0,370,8829,8824],[762,1.0,370,8829,8824],[5929,1.0,370,8829,8824],[8825,0.59,370,8829,8824],[4831,0.31,370,8829,8824],[8826,0.43,370,8829,8824]]},{"id":"moldesign-generation-2026-paper","name":"MolDesignBench · generation (2026 paper)","group":"数学与科学","metric":"Success rate","unit":"%","public_history":true,"protocol":{"en":"Scenario-grounded molecular generation with explicit and implicit constraints and infeasible cases. Six model backbones are evaluated with and without the released chemistry tools. The selected column is task success; constraint-satisfaction, validity and inference metrics stay separate.","zh":"在包含显式、隐式约束及不可行情境的分子设计任务中，六个底模分别使用和不使用公开化学工具。采用任务成功率，不与约束满足度、有效性或推断指标混合。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.27349v1","published_at":"2026-09-23","status":"paper_baselines"},"model_count":6,"result_count":12,"results":[[3184,0.11,370,8830,8831],[3184,0.38,370,8832,8831],[3183,0.04,370,8830,8831],[3183,0.2,370,8832,8831],[8833,0.06,370,8830,8831],[8833,0.13,370,8832,8831],[4823,0.08,370,8830,8831],[4823,0.19,370,8832,8831],[8834,0.0,370,8830,8831],[8834,0.0,370,8832,8831],[8835,0.08,370,8830,8831],[8835,0.09,370,8832,8831]]},{"id":"moldesign-optimization-2026-paper","name":"MolDesignBench · optimization (2026 paper)","group":"数学与科学","metric":"Success rate","unit":"%","public_history":true,"protocol":{"en":"Scenario-grounded molecular optimization with explicit and implicit constraints and infeasible cases. Six model backbones are evaluated with and without the released chemistry tools. The selected column is task success; constraint-satisfaction, validity and inference metrics stay separate.","zh":"在包含显式、隐式约束及不可行情境的分子设计任务中，六个底模分别使用和不使用公开化学工具。采用任务成功率，不与约束满足度、有效性或推断指标混合。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.27349v1","published_at":"2026-09-23","status":"paper_baselines"},"model_count":6,"result_count":12,"results":[[3184,0.17,370,8830,8831],[3184,0.43,370,8832,8831],[3183,0.17,370,8830,8831],[3183,0.41,370,8832,8831],[8833,0.11,370,8830,8831],[8833,0.23,370,8832,8831],[4823,0.14,370,8830,8831],[4823,0.25,370,8832,8831],[8834,0.02,370,8830,8831],[8834,0.0,370,8832,8831],[8835,0.13,370,8830,8831],[8835,0.1,370,8832,8831]]},{"id":"vabench-base-success-2026-paper","name":"VA-Bench · base tasks (2026 paper)","group":"Agent 与工具","metric":"Macro-average task success","unit":"%","public_history":true,"protocol":{"en":"Twelve model conditions perform all fourteen base robot tasks. Each condition uses three independent runs of twenty physically verified seeds per task, with terminal success checked by the simulator. Values preserve the published All 14 mean; smaller transfer and shared-summary experiments are separate.","zh":"十二个模型条件执行全部十四项基础机器人任务，每项使用二十个经过物理验证的种子，独立运行三次，由仿真器检查最终成功。保留原表All 14均值，迁移与共享摘要小规模实验另行区分。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.19554v1","published_at":"2026-09-17","status":"paper_baselines"},"model_count":12,"result_count":12,"results":[[8836,0.5393,370,8837,8838],[755,0.5286,370,8837,8838],[3160,0.5155,370,8837,8838],[5387,0.231,370,8837,8838],[8839,0.15,370,8837,8838],[8840,0.1417,370,8837,8838],[5386,0.1393,370,8837,8838],[3568,0.09050000000000001,370,8837,8838],[8841,0.081,370,8837,8838],[8842,0.07139999999999999,370,8837,8838],[3491,0.0405,370,8837,8838],[8843,0.0333,370,8837,8838]]},{"id":"vakqa-oracle-telugu-2026-paper","name":"VākQA · oracle-telugu (2026 paper)","group":"知识与推理","metric":"Mean correctness rubric","unit":"score","public_history":true,"protocol":{"en":"Eight models answer the same 2,001 human-verified Telugu factoid questions under the oracle-telugu text-input setting. Gemini grades correctness on the published 1–5 rubric, calibrated against human ratings. Speech-only and ASR-only panels have fewer supported models and remain separate.","zh":"八个模型在oracle-telugu文本输入条件下回答同一组2,001道经人工核实的泰卢固语事实题。Gemini按原文1–5分准则评分，并与人工评分校准。纯语音及纯ASR面板支持的模型更少，另行区分。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.19879v1","published_at":"2026-09-17","status":"paper_baselines"},"score_max":5,"saturation_excluded":true,"scale_label":{"en":"Score / 5","zh":"分数 / 5"},"score_decimals":2,"model_count":8,"result_count":8,"results":[[766,3.63,370,8844,8845],[6689,2.55,370,8844,8845],[6668,2.01,370,8844,8845],[6677,1.43,370,8844,8845],[587,1.41,370,8844,8845],[8846,1.33,370,8844,8845],[5126,1.17,370,8844,8845],[8847,1.98,370,8844,8845]]},{"id":"vakqa-seamless-translation-2026-paper","name":"VākQA · seamless-translation (2026 paper)","group":"知识与推理","metric":"Mean correctness rubric","unit":"score","public_history":true,"protocol":{"en":"Eight models answer the same 2,001 human-verified Telugu factoid questions under the seamless-translation text-input setting. Gemini grades correctness on the published 1–5 rubric, calibrated against human ratings. Speech-only and ASR-only panels have fewer supported models and remain separate.","zh":"八个模型在seamless-translation文本输入条件下回答同一组2,001道经人工核实的泰卢固语事实题。Gemini按原文1–5分准则评分，并与人工评分校准。纯语音及纯ASR面板支持的模型更少，另行区分。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.19879v1","published_at":"2026-09-17","status":"paper_baselines"},"score_max":5,"saturation_excluded":true,"scale_label":{"en":"Score / 5","zh":"分数 / 5"},"score_decimals":2,"model_count":8,"result_count":8,"results":[[766,2.74,370,8848,8845],[6689,2.45,370,8848,8845],[6668,2.28,370,8848,8845],[6677,1.84,370,8848,8845],[587,2.08,370,8848,8845],[8846,1.96,370,8848,8845],[5126,1.74,370,8848,8845],[8847,2.29,370,8848,8845]]},{"id":"vakqa-indic-translation-2026-paper","name":"VākQA · indic-translation (2026 paper)","group":"知识与推理","metric":"Mean correctness rubric","unit":"score","public_history":true,"protocol":{"en":"Eight models answer the same 2,001 human-verified Telugu factoid questions under the indic-translation text-input setting. Gemini grades correctness on the published 1–5 rubric, calibrated against human ratings. Speech-only and ASR-only panels have fewer supported models and remain separate.","zh":"八个模型在indic-translation文本输入条件下回答同一组2,001道经人工核实的泰卢固语事实题。Gemini按原文1–5分准则评分，并与人工评分校准。纯语音及纯ASR面板支持的模型更少，另行区分。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.19879v1","published_at":"2026-09-17","status":"paper_baselines"},"score_max":5,"saturation_excluded":true,"scale_label":{"en":"Score / 5","zh":"分数 / 5"},"score_decimals":2,"model_count":8,"result_count":8,"results":[[766,3.05,370,8849,8845],[6689,2.67,370,8849,8845],[6668,2.38,370,8849,8845],[6677,1.96,370,8849,8845],[587,2.21,370,8849,8845],[8846,2.06,370,8849,8845],[5126,1.84,370,8849,8845],[8847,2.39,370,8849,8845]]},{"id":"agentvid-single-turn-2026-paper","name":"AgentVidBench · single-turn (2026 paper)","group":"视觉与视频","metric":"Multiple-choice accuracy","unit":"%","public_history":true,"protocol":{"en":"One hundred human-cross-checked multihop questions over 71 videos, with 25 distractors per question. This panel uses the single-turn workflow; twelve backbones and supported thinking configurations are all preserved. Accuracy is distinct from the trajectory rubric.","zh":"在71段视频上回答100道经多人交叉核实的多步问题，每题25个干扰项。本面板采用single-turn流程，保留十二个底模及支持的思考配置，正确率与轨迹量表分开。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.21386v1","published_at":"2026-09-18","status":"paper_baselines"},"model_count":12,"result_count":20,"results":[[2058,0.29,370,8850,8851],[3578,0.3,370,8850,8851],[2055,0.46,370,8850,8851],[2602,0.51,370,8850,8851],[6298,0.16,370,8850,8851],[6298,0.18,370,8852,8851],[6342,0.18,370,8850,8851],[6342,0.18,370,8852,8851],[4070,0.19,370,8850,8851],[4070,0.22,370,8852,8851],[2584,0.22,370,8850,8851],[2584,0.25,370,8852,8851],[7567,0.11,370,8850,8851],[7567,0.07,370,8852,8851],[5650,0.16,370,8850,8851],[5650,0.14,370,8852,8851],[5635,0.21,370,8850,8851],[5635,0.22,370,8852,8851],[5480,0.38,370,8850,8851],[5480,0.37,370,8852,8851]]},{"id":"agentvid-star-2026-paper","name":"AgentVidBench · STAR (2026 paper)","group":"视觉与视频","metric":"Multiple-choice accuracy","unit":"%","public_history":true,"protocol":{"en":"One hundred human-cross-checked multihop questions over 71 videos, with 25 distractors per question. This panel uses the STAR workflow; twelve backbones and supported thinking configurations are all preserved. Accuracy is distinct from the trajectory rubric.","zh":"在71段视频上回答100道经多人交叉核实的多步问题，每题25个干扰项。本面板采用STAR流程，保留十二个底模及支持的思考配置，正确率与轨迹量表分开。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.21386v1","published_at":"2026-09-18","status":"paper_baselines"},"model_count":12,"result_count":20,"results":[[2058,0.14,370,8853,8851],[3578,0.19,370,8853,8851],[2055,0.1,370,8853,8851],[2602,0.13,370,8853,8851],[6298,0.07,370,8853,8851],[6298,0.07,370,8854,8851],[6342,0.04,370,8853,8851],[6342,0.08,370,8854,8851],[4070,0.08,370,8853,8851],[4070,0.06,370,8854,8851],[2584,0.09,370,8853,8851],[2584,0.09,370,8854,8851],[7567,0.07,370,8853,8851],[7567,0.08,370,8854,8851],[5650,0.05,370,8853,8851],[5650,0.07,370,8854,8851],[5635,0.09,370,8853,8851],[5635,0.1,370,8854,8851],[5480,0.08,370,8853,8851],[5480,0.1,370,8854,8851]]},{"id":"agentvid-dvd-2026-paper","name":"AgentVidBench · DVD (2026 paper)","group":"视觉与视频","metric":"Multiple-choice accuracy","unit":"%","public_history":true,"protocol":{"en":"One hundred human-cross-checked multihop questions over 71 videos, with 25 distractors per question. This panel uses the DVD workflow; twelve backbones and supported thinking configurations are all preserved. Accuracy is distinct from the trajectory rubric.","zh":"在71段视频上回答100道经多人交叉核实的多步问题，每题25个干扰项。本面板采用DVD流程，保留十二个底模及支持的思考配置，正确率与轨迹量表分开。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.21386v1","published_at":"2026-09-18","status":"paper_baselines"},"model_count":12,"result_count":20,"results":[[2058,0.34,370,8855,8851],[3578,0.43,370,8855,8851],[2055,0.27,370,8855,8851],[2602,0.3,370,8855,8851],[6298,0.08,370,8855,8851],[6298,0.23,370,8856,8851],[6342,0.1,370,8855,8851],[6342,0.16,370,8856,8851],[4070,0.32,370,8855,8851],[4070,0.33,370,8856,8851],[2584,0.37,370,8855,8851],[2584,0.38,370,8856,8851],[7567,0.22,370,8855,8851],[7567,0.3,370,8856,8851],[5650,0.33,370,8855,8851],[5650,0.24,370,8856,8851],[5635,0.31,370,8855,8851],[5635,0.37,370,8856,8851],[5480,0.31,370,8855,8851],[5480,0.4,370,8856,8851]]},{"id":"agentvid-avp-2026-paper","name":"AgentVidBench · AVP (2026 paper)","group":"视觉与视频","metric":"Multiple-choice accuracy","unit":"%","public_history":true,"protocol":{"en":"One hundred human-cross-checked multihop questions over 71 videos, with 25 distractors per question. This panel uses the AVP workflow; twelve backbones and supported thinking configurations are all preserved. Accuracy is distinct from the trajectory rubric.","zh":"在71段视频上回答100道经多人交叉核实的多步问题，每题25个干扰项。本面板采用AVP流程，保留十二个底模及支持的思考配置，正确率与轨迹量表分开。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.21386v1","published_at":"2026-09-18","status":"paper_baselines"},"model_count":12,"result_count":20,"results":[[2058,0.41,370,8857,8851],[3578,0.38,370,8857,8851],[2055,0.46,370,8857,8851],[2602,0.56,370,8857,8851],[6298,0.38,370,8857,8851],[6298,0.49,370,8858,8851],[6342,0.38,370,8857,8851],[6342,0.41,370,8858,8851],[4070,0.34,370,8857,8851],[4070,0.47,370,8858,8851],[2584,0.44,370,8857,8851],[2584,0.46,370,8858,8851],[7567,0.38,370,8857,8851],[7567,0.53,370,8858,8851],[5650,0.44,370,8857,8851],[5650,0.46,370,8858,8851],[5635,0.51,370,8857,8851],[5635,0.52,370,8858,8851],[5480,0.48,370,8857,8851],[5480,0.49,370,8858,8851]]},{"id":"agentvid-video-react-2026-paper","name":"AgentVidBench · Video-ReAct (2026 paper)","group":"视觉与视频","metric":"Multiple-choice accuracy","unit":"%","public_history":true,"protocol":{"en":"One hundred human-cross-checked multihop questions over 71 videos, with 25 distractors per question. This panel uses the Video-ReAct workflow; twelve backbones and supported thinking configurations are all preserved. Accuracy is distinct from the trajectory rubric.","zh":"在71段视频上回答100道经多人交叉核实的多步问题，每题25个干扰项。本面板采用Video-ReAct流程，保留十二个底模及支持的思考配置，正确率与轨迹量表分开。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.21386v1","published_at":"2026-09-18","status":"paper_baselines"},"model_count":12,"result_count":20,"results":[[2058,0.49,370,8859,8851],[3578,0.54,370,8859,8851],[2055,0.45,370,8859,8851],[2602,0.58,370,8859,8851],[6298,0.37,370,8859,8851],[6298,0.38,370,8860,8851],[6342,0.4,370,8859,8851],[6342,0.4,370,8860,8851],[4070,0.4,370,8859,8851],[4070,0.41,370,8860,8851],[2584,0.49,370,8859,8851],[2584,0.5,370,8860,8851],[7567,0.4,370,8859,8851],[7567,0.38,370,8860,8851],[5650,0.51,370,8859,8851],[5650,0.4,370,8860,8851],[5635,0.49,370,8859,8851],[5635,0.53,370,8860,8851],[5480,0.51,370,8859,8851],[5480,0.48,370,8860,8851]]},{"id":"emem-bench-success-2026-paper","name":"EMem-Bench · embodied memory (2026 paper)","group":"Agent 与工具","metric":"Mean task-family success","unit":"%","public_history":true,"protocol":{"en":"Interactive embodied memory across 2,554 episodes and four task families. The published success mean preserves full-context policies and each memory-system configuration. Robotics-ER 1.5 ran only a ten-percent subset because of API instability and is excluded; the repeated full_context row is not duplicated.","zh":"在四类任务、2,554次交互中评估具身记忆，保留原表成功率均值、完整上下文策略及各记忆系统配置。Robotics-ER 1.5因API不稳定仅运行10%子集，未与完整面板混排；重复full_context行只保留一次。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.28236v1","published_at":"2026-09-23","status":"paper_baselines"},"model_count":17,"result_count":25,"results":[[8861,0.301,370,8862,8863],[8864,0.254,370,8862,8863],[8865,0.439,370,8862,8863],[2728,0.155,370,8862,8863],[2706,0.318,370,8862,8863],[2725,0.29,370,8862,8863],[3763,0.5820000000000001,370,8862,8863],[8866,0.409,370,8862,8863],[8179,0.036000000000000004,370,8862,8863],[8867,0.008,370,8862,8863],[8868,0.2,370,8862,8863],[7556,0.38799999999999996,370,8862,8863],[8864,0.36200000000000004,370,8869,8863],[8865,0.58,370,8869,8863],[8866,0.573,370,8869,8863],[8870,0.46,370,8869,8863],[4958,0.353,370,8862,8863],[3184,0.568,370,8862,8863],[764,0.608,370,8862,8863],[6133,0.642,370,8862,8863],[4958,0.282,370,8871,8863],[4958,0.289,370,8872,8863],[4958,0.37799999999999995,370,8873,8863],[4958,0.23399999999999999,370,8874,8863],[4958,0.589,370,8869,8863]]},{"id":"jarvis-full-task-2026-paper","name":"JARVIS · full tasks (2026 paper)","group":"Agent 与工具","metric":"Overall assembly soft score","unit":"score","public_history":true,"protocol":{"en":"All 50 JARVIS-v1 tasks; nine models under generic and task-oriented tool libraries. Overall combines geometry, completion and final-state validity with bounded excess-time penalty.","zh":"全部 50 项 JARVIS-v1 任务，九个模型分别使用通用和任务专用工具。Overall 综合几何、完成度与最终状态有效性，并包含有界超时惩罚；此榜使用完整任务分数。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.27618","status":"paper_baselines","published_at":"2026-10-20"},"score_max":100,"score_decimals":2,"saturation_excluded":true,"higher_is_better":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":9,"result_count":18,"results":[[6130,8.19,370,8875,8876],[6130,17.13,370,8877,8876],[6705,13.83,370,8875,8876],[6705,17.04,370,8877,8876],[6214,5.93,370,8875,8876],[6214,11.68,370,8877,8876],[8878,8.49,370,8875,8876],[8878,17.39,370,8877,8876],[6297,8.27,370,8875,8876],[6297,15.54,370,8877,8876],[2998,5.58,370,8875,8876],[2998,11.21,370,8877,8876],[353,4.62,370,8875,8876],[353,11.55,370,8877,8876],[3748,7.63,370,8875,8876],[3748,14.99,370,8877,8876],[3419,8.38,370,8875,8876],[3419,15.9,370,8877,8876]]},{"id":"realcad-part-2026-paper","name":"RealCADBench · part track (2026 paper)","group":"专业工作","metric":"Regime- balanced PA","unit":"score","score_max":1,"higher_is_better":true,"public_history":true,"protocol":{"en":"Generates executable CAD artifacts. The published PA composite combines executability with geometry and visual-identity scores averaged over evaluator-available artifacts; part results balance four input regimes.","zh":"生成可执行CAD作品。论文PA综合分数结合可执行率，以及在可评估作品上平均的几何和视觉身份分数；零件结果对四种输入方式等权。"},"source":{"name":"RealCADBench authors","status":"paper_baselines","url":"https://arxiv.org/html/2609.03773v1#S4.T4","published_at":"2026-09-03"},"model_count":9,"result_count":9,"saturation_excluded":true,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"results":[[3479,0.5316,370,8879,8880],[3482,0.5507,370,8879,8880],[3184,0.4814,370,8879,8880],[3196,0.5478,370,8879,8880],[2977,0.4195,370,8879,8880],[4225,0.3842,370,8879,8880],[6342,0.2309,370,8879,8880],[6343,0.2902,370,8879,8880],[5300,0.4575,370,8879,8880]]},{"id":"realcad-assembly-2026-paper","name":"RealCADBench · assembly track (2026 paper)","group":"专业工作","metric":"PA","unit":"score","score_max":1,"higher_is_better":true,"public_history":true,"protocol":{"en":"Generates executable CAD artifacts. The published PA composite combines executability with geometry and visual-identity scores averaged over evaluator-available artifacts; assembly results use the fixed25-task RCB-Assm25 study.","zh":"生成可执行CAD作品。论文PA综合分数结合可执行率，以及在可评估作品上平均的几何和视觉身份分数；装配结果采用固定25题的RCB-Assm25研究。"},"source":{"name":"RealCADBench authors","status":"paper_baselines","url":"https://arxiv.org/html/2609.03773v1#S4.T5","published_at":"2026-09-03"},"model_count":6,"result_count":8,"saturation_excluded":true,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"results":[[3196,0.5228,370,8881,8882],[3479,0.4644,370,8883,8882],[3479,0.4638,370,8884,8882],[3482,0.4724,370,8884,8882],[3184,0.5028,370,8884,8882],[3196,0.4777,370,8884,8882],[2977,0.4338,370,8884,8882],[4225,0.3389,370,8884,8882]]},{"id":"principia-generators-2026-paper","name":"Principia · video generation (2026 paper)","group":"数学与科学","metric":"Overall","unit":"score","score_max":1,"higher_is_better":true,"public_history":true,"protocol":{"en":"Generates videos of physical phenomena. The continuous relational-consistency score measures agreement with physical invariants and averages scenario scores after repeated generation seeds.","zh":"生成物理现象视频。连续关系一致性分数衡量视频是否符合物理不变量，在多随机种子生成后，对场景分数取平均。"},"source":{"name":"Principia authors","status":"paper_baselines","url":"https://arxiv.org/html/2609.04200v1#S4.T3","published_at":"2026-09-03"},"model_count":6,"result_count":6,"saturation_excluded":true,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"results":[[8885,0.379,370,8886,8887],[8888,0.148,370,8886,8887],[8889,0.199,370,8886,8887],[8890,0.205,370,8886,8887],[8683,0.419,370,8886,8887],[8891,0.409,370,8886,8887]]},{"id":"fpco-corrtp10-2026-paper","name":"FPCO-Dialog · correction through turn10 (2026 paper)","group":"安全与可靠性","metric":"CorrTP@10","unit":"%","score_max":100,"higher_is_better":true,"public_history":true,"protocol":{"en":"Corrects repeated false premises in visual dialogue. CorrTP@10 averages correction labels across false-premise turns4–10 from two independently validated detectors, over 1080 images.","zh":"在视觉对话中纠正反复出现的错误前提。CorrTP@10覆盖1,080幅图像，对第4—10轮错误前提回答的两名裁判纠正标签取平均。"},"source":{"name":"FPCO-Dialog authors","status":"paper_baselines","url":"https://arxiv.org/html/2609.03331v1#A9.T5","published_at":"2026-09-03"},"model_count":20,"result_count":20,"results":[[2055,0.654,370,8892,8893],[2602,0.649,370,8892,8893],[3482,0.588,370,8892,8893],[345,0.587,370,8892,8893],[353,0.482,370,8892,8893],[2058,0.079,370,8892,8893],[3184,0.294,370,8892,8893],[2793,0.052,370,8892,8893],[7065,0.074,370,8892,8893],[2774,0.11,370,8892,8893],[8894,0.092,370,8892,8893],[8895,0.022,370,8892,8893],[6389,0.01,370,8892,8893],[2453,0.571,370,8892,8893],[2478,0.169,370,8892,8893],[2444,0.629,370,8892,8893],[2471,0.488,370,8892,8893],[8896,0.617,370,8892,8893],[8897,0.633,370,8892,8893],[889,0.688,370,8892,8893]]},{"id":"vips-pseudo-2026-paper","name":"VIPS · pseudo-simulation (2026 paper)","group":"预测与决策","metric":"Integrated EPDMS","unit":"score","score_max":100,"higher_is_better":true,"public_history":true,"protocol":{"en":"Evaluates driving plans in vehicle-to-infrastructure pseudo-simulation. The integrated EPDMS score combines both evaluation stages under the published safety, progress and comfort criteria.","zh":"在车路协同伪仿真中评测驾驶规划。采用两个评测阶段的综合EPDMS分数，按论文的安全、行进与舒适性标准计分。"},"source":{"name":"VIPS authors","status":"paper_baselines","url":"https://arxiv.org/html/2609.02462v1#S5.T3","published_at":"2026-09-02"},"model_count":6,"result_count":6,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"results":[[8898,37.28,370,8899,8900],[8901,43.26,370,8899,8900],[8902,42.03,370,8899,8900],[8903,45.38,370,8899,8900],[8904,43.79,370,8899,8900],[8905,50.88,370,8899,8900]]},{"id":"cwe-bench-v2-publisher","name":"CWE-bench v2 (2026-11-02)","group":"编程","metric":"Pass@1","unit":"%","score_max":100,"protocol":{"en":"Coding agents audit and patch250 tasks spanning185 weakness classes:100 public tasks and150 held-out tasks. Deterministic verifiers require the exploit to be closed while legitimate behavior still works. Pass@1 is averaged over four rollouts per task, at high reasoning effort.","zh":"编程智能体审计并修复250项任务，覆盖185类弱点，其中100项公开、150项留出。确定性验证器要求漏洞失效且正常功能仍然有效。采用高思考设置，每题四次运行，报告平均Pass@1。"},"source":{"name":"Collinear AI / CWE-bench","status":"current_snapshot","url":"https://cwe-bench.com/#leaderboard","published_at":"2026-11-02"},"model_count":11,"result_count":11,"results":[[2959,0.6709999999999999,370,8906,8907],[3732,0.6609999999999999,370,8908,8907],[2962,0.65,370,8909,8907],[2958,0.642,370,8910,8907],[8911,0.5760000000000001,370,8908,8907],[2968,0.547,370,8908,8907],[3524,0.473,370,8908,8907],[2992,0.456,370,8908,8907],[8912,0.45399999999999996,370,8908,8907],[2967,0.45299999999999996,370,8908,8907],[3154,0.42,370,8908,8907]]},{"id":"cesbench-composite-2026-paper","name":"CESBench · cryptographic engineering (2026 paper)","group":"安全与可靠性","metric":"Composite rubric score","unit":"score","public_history":true,"protocol":{"en":"Eleven models answer every one of 380 expert-written cryptographic-engineering items. The composite combines MCQ, gated judgment, scenario rubrics and executable code tests; it remains a native 0–100 score.","zh":"十一个模型完整回答380道专家编写的密码工程问题。综合分包含选择题、带结论门控的判断题、情境准则与可执行代码测试，保留0–100原生量表。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.21344v1","published_at":"2026-09-18","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":11,"result_count":11,"results":[[227,83.6,370,8913,8914],[3187,82.5,370,8913,8914],[5906,81.4,370,8913,8914],[5616,80.8,370,8913,8914],[3185,80.2,370,8913,8914],[3005,73.4,370,8913,8914],[1573,68.4,370,8913,8914],[8915,61.3,370,8913,8914],[8916,60.0,370,8913,8914],[1583,59.7,370,8913,8914],[8917,54.4,370,8913,8914]]},{"id":"cibuzz-english-2026-paper","name":"CIBuzzBench · English prompts (2026 paper)","group":"知识与推理","metric":"Equivalent-selection macro-F1","unit":"score","public_history":true,"protocol":{"en":"Six models select English equivalents for a benchmark of 3,001 annotated Chinese internet buzzwords, using English prompts. The selected metric is macro-F1; meaning-generation rubrics and harmfulness labels remain separate.","zh":"六个模型为包含3,001个标注中文网络热词的评测选择英文对应表达。中英文提示分别展示，采用Macro-F1，不与释义生成量表或危害分类混合。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.21722v1","published_at":"2026-09-18","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":6,"result_count":6,"results":[[3196,87.12,370,8918,8919],[3479,79.94,370,8918,8919],[3412,88.03,370,8918,8919],[3185,79.66,370,8918,8919],[3202,82.38,370,8918,8919],[227,82.27,370,8918,8919]]},{"id":"cibuzz-chinese-2026-paper","name":"CIBuzzBench · Chinese prompts (2026 paper)","group":"知识与推理","metric":"Equivalent-selection macro-F1","unit":"score","public_history":true,"protocol":{"en":"Six models select English equivalents for a benchmark of 3,001 annotated Chinese internet buzzwords, using Chinese prompts. The selected metric is macro-F1; meaning-generation rubrics and harmfulness labels remain separate.","zh":"六个模型为包含3,001个标注中文网络热词的评测选择英文对应表达。中英文提示分别展示，采用Macro-F1，不与释义生成量表或危害分类混合。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.21722v1","published_at":"2026-09-18","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":6,"result_count":6,"results":[[3196,87.83,370,8920,8919],[3479,83.17,370,8920,8919],[3412,88.41,370,8920,8919],[3185,78.91,370,8920,8919],[3202,84.8,370,8920,8919],[227,84.4,370,8920,8919]]},{"id":"gamelogic-pass1-2026-paper","name":"GameLogicBench · game rules (2026 paper)","group":"编程","metric":"Task solve rate","unit":"%","public_history":true,"protocol":{"en":"Human-screened game mechanics are tested with deterministic tick-level assertions and multiple hidden scenarios. All twenty model–scaffold configurations use sealed egress and one bounded attempt per task; the published Average is retained.","zh":"经人工筛选的游戏机制通过逐帧确定性断言与多个隐藏情境检查。全部二十个模型与框架配置禁止外网访问，每题一次有界尝试，保留原表Average。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.21562v1","published_at":"2026-09-18","status":"paper_baselines"},"model_count":11,"result_count":20,"results":[[4983,0.5278,370,8921,8922],[5133,0.44439999999999996,370,8921,8922],[4979,0.4167,370,8921,8922],[3404,0.4167,370,8921,8922],[227,0.375,370,8921,8922],[3155,0.3611,370,8921,8922],[8423,0.3194,370,8921,8922],[5789,0.25,370,8921,8922],[3185,0.25,370,8921,8922],[3198,0.19440000000000002,370,8921,8922],[7643,0.16670000000000001,370,8921,8922],[3201,0.16670000000000001,370,8921,8922],[4979,0.3472,370,8923,8922],[5727,0.2917,370,8923,8922],[5133,0.2639,370,8923,8922],[3155,0.2639,370,8923,8922],[4979,0.3472,370,8924,8922],[227,0.3194,370,8924,8922],[5133,0.2917,370,8924,8922],[3155,0.22219999999999998,370,8924,8922]]},{"id":"omnivbench-overall-2026-paper","name":"OmniVBench · reference-to-video (2026 paper)","group":"生成与创作","metric":"Overall factor-grounded score","unit":"score","public_history":true,"protocol":{"en":"Eleven video generators are assessed across seven reference-control task families. Manually checked cases use reference-grounded checklists for fidelity, instruction realization and visual quality. Their published Overall stays a native composite score.","zh":"十一个视频生成器在七类参考控制任务上评测。案例经过人工核验，按参考忠实度、指令实现和视觉质量的细分清单评分，保留原表Overall综合分。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.22069v1","published_at":"2026-09-18","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":11,"result_count":11,"results":[[8925,49.69,370,8926,8927],[8407,48.88,370,8926,8927],[8928,51.25,370,8926,8927],[8929,56.95,370,8926,8927],[8930,72.41,370,8926,8927],[8931,59.45,370,8926,8927],[8932,68.59,370,8926,8927],[8933,70.89,370,8926,8927],[6384,70.76,370,8926,8927],[6383,70.91,370,8926,8927],[8934,72.68,370,8926,8927]]},{"id":"chemclir-google-patents-2026-paper","name":"ChemCLIR-Bench · Google-Patents (2026 paper)","group":"专业工作","metric":"Language-balanced cross-lingual Recall@10","unit":"score","public_history":true,"protocol":{"en":"Eight embedding models retrieve chemical patents across five languages in the Google-Patents corpus. Recall@10 is balanced across cross-language routes and stays on its published 0–1 retrieval scale. Same-language results and censored retrieval depths are separate.","zh":"八个嵌入模型在Google-Patents专利库进行五种语言的化学专利检索。跨语言Recall@10按语言路线平衡，保留0–1检索刻度，不与同语言成绩或截断的检索深度混合。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.23231v1","published_at":"2026-09-19","status":"paper_baselines"},"score_max":1,"saturation_excluded":true,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"score_decimals":2,"model_count":8,"result_count":8,"results":[[8935,0.54,370,8936,8937],[8053,0.47,370,8938,8937],[8939,0.45,370,8940,8937],[8941,0.41,370,8942,8937],[8943,0.39,370,8944,8937],[8945,0.27,370,8946,8937],[8947,0.09,370,8948,8937],[8949,0.2,370,8950,8937]]},{"id":"chemclir-epo-2026-paper","name":"ChemCLIR-Bench · EPO (2026 paper)","group":"专业工作","metric":"Language-balanced cross-lingual Recall@10","unit":"score","public_history":true,"protocol":{"en":"Eight embedding models retrieve chemical patents across five languages in the EPO corpus. Recall@10 is balanced across cross-language routes and stays on its published 0–1 retrieval scale. Same-language results and censored retrieval depths are separate.","zh":"八个嵌入模型在EPO专利库进行五种语言的化学专利检索。跨语言Recall@10按语言路线平衡，保留0–1检索刻度，不与同语言成绩或截断的检索深度混合。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.23231v1","published_at":"2026-09-19","status":"paper_baselines"},"score_max":1,"saturation_excluded":true,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"score_decimals":2,"model_count":8,"result_count":8,"results":[[8935,0.52,370,8936,8937],[8053,0.47,370,8938,8937],[8939,0.39,370,8940,8937],[8941,0.43,370,8942,8937],[8943,0.36,370,8944,8937],[8945,0.25,370,8946,8937],[8947,0.12,370,8948,8937],[8949,0.17,370,8950,8937]]},{"id":"physai-heldout-2026-paper","name":"PhysAI-Bench · held-out decisions (2026 paper)","group":"Agent 与工具","metric":"Held-out decision accuracy","unit":"%","public_history":true,"protocol":{"en":"Models choose trace-derived reference actions on 500 episode-disjoint held-out UAV decision instances. Prompt and temperature are selected on development data, then frozen for three runs. All thirty numerical model rows in Table IV are retained; the abstract still states 29. This is action-selection agreement, not executed mission success.","zh":"模型在500个与开发集按运行片段隔离的无人机决策样本中选择轨迹参考动作。提示与温度在开发集选择后冻结，评测三次。保留表IV全部三十个数值模型行，摘要仍写29个；成绩表示动作选择一致率，不是实机任务成功率。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.23695v1","published_at":"2026-09-20","status":"paper_baselines"},"model_count":30,"result_count":30,"results":[[8951,0.52,370,8952,8953],[8954,0.494,370,8955,8953],[8956,0.4907,370,8957,8953],[4961,0.47729999999999995,370,8958,8953],[3202,0.46399999999999997,370,8952,8953],[4425,0.4607,370,8959,8953],[8960,0.4587,370,8955,8953],[8961,0.45270000000000005,370,8962,8953],[325,0.44799999999999995,370,8955,8953],[6110,0.44670000000000004,370,8963,8953],[8964,0.4387,370,8952,8953],[8965,0.43799999999999994,370,8966,8953],[8967,0.43670000000000003,370,8955,8953],[5596,0.42469999999999997,370,8957,8953],[244,0.41869999999999996,370,8952,8953],[8968,0.414,370,8955,8953],[8969,0.408,370,8955,8953],[4823,0.406,370,8958,8953],[8970,0.38670000000000004,370,8966,8953],[8971,0.3833,370,8972,8953],[4824,0.3753,370,8962,8953],[8973,0.37329999999999997,370,8955,8953],[6058,0.37329999999999997,370,8955,8953],[3453,0.3607,370,8952,8953],[8974,0.344,370,8955,8953],[8975,0.3367,370,8963,8953],[8976,0.292,370,8958,8953],[8977,0.2787,370,8963,8953],[8978,0.27399999999999997,370,8958,8953],[8979,0.254,370,8980,8953]]},{"id":"finfirst-complete-strict-2026-paper","name":"FinFIRST · complete-run strict pass (2026 paper)","group":"专业工作","metric":"All-criteria task pass rate","unit":"%","public_history":true,"protocol":{"en":"Financial research on 123 expert-authored tasks with 701 atomic criteria and a shared search, webpage and Python toolset. A task passes only when every criterion passes. This panel contains the ten models with all 123 valid outputs; five incomplete-output rows are held separately.","zh":"使用统一搜索、网页与Python工具完成123道专家编写的金融研究题，含701条原子准则。每题全部准则通过才算成功。本面板仅含123题输出完整的十个模型，五个输出不完整的模型暂存。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.25192v1","published_at":"2026-09-21","status":"paper_baselines"},"model_count":10,"result_count":10,"results":[[5906,0.4472,370,8981,8982],[3157,0.5447,370,8981,8982],[5899,0.6179,370,8981,8982],[3185,0.5122,370,8981,8982],[3396,0.5690999999999999,370,8981,8982],[227,0.43090000000000006,370,8981,8982],[3155,0.5935,370,8981,8982],[3201,0.4878,370,8981,8982],[3198,0.374,370,8981,8982],[8983,0.4065,370,8981,8982]]},{"id":"indicbank-single-run-2026-paper","name":"IndicBankBench · mean single run (2026 paper)","group":"专业工作","metric":"Mean single-run success","unit":"%","public_history":true,"protocol":{"en":"Eleven banking assistants each run all 799 cases three times. Success combines safety, tool actions, response adequacy and advisory checks, with mostly deterministic checks and narrow semantic judges. The mean single-run score is distinct from strict three-run reliability.","zh":"十一个银行助手分别对全部799例运行三次，结合安全、工具动作、回答充分性及建议检查评分，主要使用确定性检查和范围明确的语义裁判。采用单次平均成功率，与三次全通过的可靠性指标区分。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.29167v1","published_at":"2026-09-24","status":"paper_baselines"},"model_count":11,"result_count":11,"results":[[3404,0.6559999999999999,370,8984,8985],[2974,0.644,370,8984,8985],[7641,0.664,370,8984,8985],[755,0.647,370,8984,8985],[3396,0.659,370,8984,8985],[8423,0.642,370,8984,8985],[6110,0.601,370,8984,8985],[2983,0.589,370,8984,8985],[3491,0.594,370,8984,8985],[8964,0.57,370,8984,8985],[8986,0.521,370,8984,8985]]},{"id":"ccrv-constrained-2026-paper","name":"CCRV-Bench · constrained reasoning (2026 paper)","group":"视觉与视频","metric":"Cascaded DCR rubric","unit":"score","public_history":true,"protocol":{"en":"Fifteen models perform human-checked visual causal reasoning under four constraint families. The constrained DCR averages cascaded grounding, mechanism and conclusion credit; it permits partial credit and remains a native rubric score.","zh":"十五个模型在四类约束下回答经人工检查的视觉因果推理问题。约束DCR对实体定位、机制和结论逐级门控后取均值，允许部分得分，保留原生量表。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.30979v1","published_at":"2026-09-25","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":15,"result_count":15,"results":[[3196,79.0,370,8987,8988],[3181,77.4,370,8987,8988],[3009,50.0,370,8987,8988],[8989,62.9,370,8987,8988],[6237,60.0,370,8987,8988],[2129,60.1,370,8987,8988],[7937,62.5,370,8987,8988],[2515,52.5,370,8987,8988],[2754,52.3,370,8987,8988],[3764,51.8,370,8987,8988],[2706,55.5,370,8987,8988],[7607,50.8,370,8987,8988],[7711,46.6,370,8987,8988],[8990,58.9,370,8987,8988],[3198,74.7,370,8987,8988]]},{"id":"eebench-v1-publisher","name":"EEBench V1 (2026)","group":"数学与科学","metric":"Technical and cost-efficiency composite","unit":"score","score_max":100,"score_decimals":1,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"saturation_excluded":true,"protocol":{"en":"Electrical-engineering agents submit atopile designs for 13 held-out tasks. Deterministic build, simulation and bill-of-materials checks weight technical requirements at 65% and cost efficiency at 35%. Cost credit requires a working design. Every result covers all 13 tasks; measured attempt counts remain explicit.","zh":"电气工程智能体为13个留出任务提交atopile设计。构建、仿真和物料清单检查按技术要求65%、成本效率35%计分，设计可工作才获得成本分。每项结果覆盖全部13题，并保留实际运行次数。"},"source":{"name":"atopile / EEBench","status":"current_snapshot","url":"https://www.eebench.org/","published_at":"2026-09-04"},"model_count":26,"result_count":28,"results":[[2959,75.0,1,8991,8992],[2958,69.3,15,8993,8992],[2959,68.9,1,8994,8992],[2961,67.2,1,8991,8992],[2970,67.1,1,8994,8992],[2962,67.1,83,8995,8992],[2968,64.0,143,8996,8992],[2960,63.6,15,8997,8992],[2965,62.7,66,8998,8992],[755,61.6,1,8999,8992],[2974,57.1,143,8996,8992],[2956,56.4,1,8994,8992],[3394,56.3,15,9000,8992],[2983,55.4,83,8995,8992],[3453,54.3,1,8999,8992],[3479,51.4,1,9001,8992],[3391,50.9,143,8996,8992],[2979,49.4,66,8998,8992],[3412,46.3,83,9002,8992],[3568,40.3,1,9003,8992],[3405,37.4,83,9004,8992],[3479,35.2,1,9005,8992],[3415,34.3,83,9006,8992],[2976,31.1,15,8997,8992],[3572,20.4,1,9007,8992],[3424,19.5,83,9008,8992],[5742,18.0,143,9009,8992],[325,3.5,1,9010,8992]]},{"id":"autonomous-science-bench-2026-paper","name":"Autonomous Science Bench (2026 paper)","group":"数学与科学","metric":"Campaign resolution rate","unit":"%","public_history":true,"protocol":{"en":"Version 0.1: six campaigns, three trials each for eight configurations. Resolution requires both discovery and held-out learning gates. All 144 trials completed every round and passed submission-format validation. Table 6 published rounded rate.","zh":"v0.1 六项研究任务，八种模型配置每任务三次试验。须同时通过发现与留出学习两道判据。全部 144 次试验完成所有轮次并通过提交格式检查；取表 6 发表的舍入成功率。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.18449","status":"paper_baselines","published_at":"2026-10-13"},"model_count":8,"result_count":8,"results":[[2959,0.67,370,9011,9012],[2973,0.44,370,9013,9012],[3408,0.28,370,9013,9012],[2961,0.11,370,9014,9012],[2980,0.11,370,9015,9012],[2986,0.06,370,9015,9012],[3491,0.06,370,9015,9012],[325,0.0,370,9016,9012]]},{"id":"helpmeout-llm-auprc-2026-paper","name":"HelpMeOut · LLM subset (2026 paper)","group":"Agent 与工具","metric":"Step-level AUPRC","unit":"score","public_history":true,"protocol":{"en":"All thirteen LLM configurations on the same held-out 60 episodes. M2 is AUPRC for the error-onset step; later consequences are negative. Non-LLM and privileged symbolic references remain separate.","zh":"全部十三种 LLM 配置评估同一组留出 60 个情景。M2 为错误起始步骤的 AUPRC，后续影响步骤记为负例。非 LLM 和拥有额外符号知识的参考方法另列。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.17694","status":"paper_baselines","published_at":"2026-10-13"},"score_max":1,"score_decimals":2,"saturation_excluded":true,"higher_is_better":true,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":12,"result_count":13,"results":[[3490,0.14,370,9017,9018],[3572,0.325,370,9019,9018],[3492,0.286,370,5017,9018],[3196,0.238,370,5015,9018],[3196,0.231,370,5017,9018],[3493,0.086,370,9017,9018],[5449,0.055,370,9017,9018],[3186,0.077,370,9017,9018],[3412,0.119,370,5017,9018],[3497,0.061,370,9017,9018],[3190,0.079,370,5015,9018],[3764,0.057,370,9017,9018],[3261,0.027,370,9017,9018]]},{"id":"ladder-state-tracking-2026-paper","name":"LADDER · offline state tracking (2026 paper)","group":"Agent 与工具","metric":"All-state-correct accuracy","unit":"%","public_history":true,"protocol":{"en":"Table 4 offline orchestrator evaluation, ten VLMs. State tracking counts frames with every queried state correct; this panel is separate from closed-loop robot task success.","zh":"表 4 离线协调器评估，十个视觉语言模型。状态跟踪要求一帧中的全部待询状态正确；本组与闭环机器人任务成功率分开。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.18053","status":"paper_baselines","published_at":"2026-10-13"},"model_count":10,"result_count":10,"results":[[6341,0.09300000000000001,370,7834,9020],[6342,0.479,370,7834,9020],[4070,0.11699999999999999,370,7834,9020],[5300,0.5820000000000001,370,7834,9020],[3412,0.687,370,7834,9020],[3386,0.708,370,7834,9020],[9021,0.696,370,7834,9020],[762,0.564,370,7834,9020],[2958,0.532,370,7834,9020],[755,0.61,370,7834,9020]]},{"id":"tastebench-decision-2026-paper","name":"Taste-Bench · decision forks (2026 paper)","group":"预测与决策","metric":"Both-order macro accuracy","unit":"%","public_history":true,"protocol":{"en":"Fourteen models choose better continuations at 502 engineering and research decision forks. A question passes only when both candidate orders are answered correctly. Average gives equal weight to the engineering and research subsets; unparseable attempts count as incorrect.","zh":"十四个模型在502个工程与研究决策分叉处选择较好后续。两个候选顺序都答对才算通过，Average对工程和研究两部分等权；无法解析的已执行回答算错误。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.25804v1","published_at":"2026-09-22","status":"paper_baselines"},"model_count":14,"result_count":14,"results":[[762,0.597,370,9022,9023],[3196,0.595,370,9022,9023],[755,0.555,370,9022,9023],[3391,0.546,370,9022,9023],[2973,0.54,370,9022,9023],[227,0.539,370,9022,9023],[3568,0.516,370,9022,9023],[3408,0.49,370,9022,9023],[3491,0.45299999999999996,370,9022,9023],[3261,0.433,370,9022,9023],[3494,0.401,370,9022,9023],[3613,0.377,370,9022,9023],[6058,0.366,370,9022,9023],[9024,0.157,370,9022,9023]]},{"id":"clashbench-preemption-2026-paper","name":"ClashBench · destructive preemption (2026 paper)","group":"安全与可靠性","metric":"Successful preemption rate","unit":"%","public_history":true,"protocol":{"en":"Seventeen models in thirty model–harness settings face 268 validated shared-resource conflicts. Successful preemption requires the requested task to succeed while the incumbent fails its independent health check. Only runs with valid environments, traces and graders enter the author denominator; lower is safer.","zh":"十七个模型、三十种模型与框架配置面对268个经验证的共享资源冲突。新任务成功且原有任务未通过独立健康检查才记为成功抢占。按作者协议仅纳入环境、轨迹和评分器有效的运行，数值越低越安全。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.19892v1","published_at":"2026-09-17","status":"paper_baselines"},"higher_is_better":false,"model_count":17,"result_count":30,"results":[[8423,0.547,370,9025,9026],[3196,0.5379999999999999,370,9025,9026],[5616,0.442,370,9025,9026],[4979,0.45899999999999996,370,9025,9026],[5727,0.345,370,9025,9026],[5789,0.252,370,9027,9026],[8423,0.47100000000000003,370,9027,9026],[2278,0.33,370,9027,9026],[227,0.38299999999999995,370,9027,9026],[3154,0.473,370,9027,9026],[3198,0.272,370,9027,9026],[4070,0.41,370,9027,9026],[3763,0.49,370,9027,9026],[3764,0.485,370,9027,9026],[6923,0.46799999999999997,370,9027,9026],[4961,0.556,370,9027,9026],[5300,0.508,370,9027,9026],[3157,0.612,370,9027,9026],[8423,0.455,370,9028,9026],[2278,0.337,370,9028,9026],[227,0.47200000000000003,370,9028,9026],[3154,0.511,370,9028,9026],[3198,0.371,370,9028,9026],[4070,0.321,370,9028,9026],[3763,0.45299999999999996,370,9028,9026],[3764,0.371,370,9028,9026],[6923,0.49200000000000005,370,9028,9026],[4961,0.508,370,9028,9026],[5300,0.5429999999999999,370,9028,9026],[3157,0.5429999999999999,370,9028,9026]]},{"id":"crsbench-complete-panel-2026-paper","name":"CRS-Bench (three-domain panel) (2026 paper)","group":"医疗","metric":"Aggregate CRS","unit":"score","public_history":true,"protocol":{"en":"Published aggregate CRS for thirteen encoders measured on all three domains (ISIC, APTOS and CheXpert), with matched splits, probes, tuning budgets and seeds. BioViL and CheXzero have no APTOS measurement and are excluded from this complete-domain panel. Native reference-relative 0–1 reliability composite.","zh":"采用 ISIC、APTOS、CheXpert 三个领域均有实测结果的十三个编码器的 Aggregate CRS。数据划分、探针、调参预算和随机种子统一。BioViL 与 CheXzero 缺少 APTOS 结果，未并入完整三领域面板。保留参照集合下的 0–1 可靠性综合分。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.22059v1","status":"paper_baselines","published_at":"2026-08-22"},"score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":13,"result_count":13,"results":[[9029,0.703,370,370,9030],[8256,0.701,370,370,9030],[9031,0.645,370,370,9030],[9032,0.424,370,370,9030],[9033,0.32,370,370,9030],[8257,0.298,370,370,9030],[7324,0.284,370,370,9030],[9034,0.266,370,370,9030],[9035,0.251,370,370,9030],[9036,0.241,370,370,9030],[9037,0.187,370,370,9030],[9038,0.184,370,370,9030],[9039,0.072,370,370,9030]]},{"id":"mmarch-accuracy-2026-paper","name":"MMArch (2026 paper)","group":"专业工作","metric":"Average answer accuracy","unit":"%","public_history":true,"protocol":{"en":"Published Avg. answer accuracy over 1,212 short-answer items across ten architecture and civil-engineering domains. All eighteen models use temperature 0 and a 2,048-token output cap. Final answers are matched with units, synonyms, numeric tolerance and accepted answer sets normalized. Human reference excluded.","zh":"1,212 道建筑与土木工程短答题、十个领域的 Avg. 答案准确率。十八个模型均采用温度 0、最多 2,048 输出 token；匹配最终答案时统一处理单位、同义词、数值容差与可接受答案集。排除人工参考。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.09281","status":"paper_baselines","published_at":"2026-08-10"},"model_count":18,"result_count":18,"results":[[3763,0.20329999999999998,370,370,9040],[3764,0.2193,370,370,9040],[4070,0.1564,370,370,9040],[2584,0.1905,370,370,9040],[2590,0.19820000000000002,370,370,9040],[5480,0.2585,370,370,9040],[2471,0.1335,370,370,9040],[2728,0.14529999999999998,370,370,9040],[9041,0.1497,370,370,9040],[6408,0.1928,370,370,9040],[9042,0.2114,370,370,9040],[3491,0.2994,370,370,9040],[3196,0.5173,370,370,9040],[3479,0.5112,370,370,9040],[283,0.4694,370,370,9040],[3482,0.4167,370,370,9040],[3415,0.4068,370,370,9040],[353,0.3427,370,370,9040]]},{"id":"mmpcbench-pcq-2026-paper","name":"MMPCBench (2026 paper)","group":"安全与可靠性","metric":"Proactive critique quality","unit":"score","public_history":true,"protocol":{"en":"Main fourteen-model PCQ panel on the same 3,146 erroneous-input cases. The score combines error detection with conditional diagnostic and remedy ratings. Uses the published 0–100 composite, not detection accuracy alone; Average row and the seven-model paired-validity experiment are excluded.","zh":"同一组 3,146 个错误输入案例上的十四模型主 PCQ 面板。分数结合错误检测与检出后的诊断、补救评分，保留论文 0–100 综合分；排除 Average 行，不与七模型有效输入配对实验混合。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.29286","status":"paper_baselines","published_at":"2026-08-29"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":14,"result_count":14,"results":[[6704,35.4,370,3143,9043],[6130,30.4,370,370,9043],[6133,26.5,370,370,9043],[3748,29.6,370,5010,9043],[340,24.9,370,370,9043],[4064,22.9,370,370,9043],[9044,10.4,370,370,9043],[2613,47.4,370,370,9043],[5485,40.4,370,370,9043],[6150,38.7,370,370,9043],[8190,18.1,370,370,9043],[3029,17.6,370,370,9043],[7486,10.0,370,370,9043],[5722,12.8,370,370,9043]]},{"id":"formstruct-schema-2026-paper","name":"FormStruct-Bench (2026 paper)","group":"专业工作","metric":"Schema-nTED","unit":"score","public_history":true,"protocol":{"en":"Main table document-level Schema-nTED over the same page images, with fixed parsing, repair and conversion rules and a common JSON schema. Fourteen systems plus two SFT configurations. Schema tree similarity retains field names, container types and hierarchy while discarding values. Native 0–100 similarity, not accuracy.","zh":"主表同一组页面上的文档级 Schema-nTED，采用固定解析、修复与转换规则和统一 JSON 结构。十四个系统及两个 SFT 配置。树结构相似度保留字段名、容器类型和层级，忽略字段值；保留 0–100 原始相似度。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.10396","status":"paper_baselines","published_at":"2026-08-11"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":14,"result_count":16,"results":[[3196,54.15,370,370,9045],[3568,52.22,370,370,9045],[3415,48.59,370,370,9045],[4961,34.01,370,370,9045],[9046,48.2,370,370,9045],[4881,52.81,370,370,9045],[3764,48.46,370,370,9045],[3764,57.45,370,9047,9045],[9048,5.89,370,370,9045],[4070,45.58,370,370,9045],[4070,56.98,370,9047,9045],[6241,30.41,370,370,9045],[6239,26.2,370,370,9045],[2654,30.34,370,370,9045],[9049,4.53,370,370,9045],[9050,2.97,370,370,9045]]},{"id":"mmds-stance-f1-2026-paper","name":"MMDS-Bench (Task 1) (2026 paper)","group":"视觉与视频","metric":"Macro-F1","unit":"score","public_history":true,"protocol":{"en":"Task 1 primary Macro-F1 on all 3,482 multimodal parent–reply instances and seven stance labels. Twelve configurations represent ten underlying models; Qwen Thinking/Instruct variants retain a common model identity. Task 2 uses a separate 800-instance diagnostic subset and is not mixed into this panel. Native 0–100 F1.","zh":"任务 1 在全部 3,482 个多模态原帖与回复实例、七类立场上的主指标 Macro-F1。十二种配置对应十个底模，Qwen 的 Thinking/Instruct 作为配置保留。任务 2 的 800 条诊断子集另列，不混入本面板；保留 0–100 F1。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.30903","status":"paper_baselines","published_at":"2026-08-31"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":10,"result_count":12,"results":[[2602,42.22,370,370,9051],[3572,37.29,370,370,9051],[3748,41.54,370,370,9051],[2998,37.85,370,370,9051],[5843,33.46,370,3143,9051],[5843,28.81,370,8420,9051],[5722,25.93,370,370,9051],[9052,25.36,370,370,9051],[5535,23.22,370,370,9051],[6342,27.19,370,3143,9051],[6342,25.51,370,8420,9051],[9053,19.21,370,370,9051]]},{"id":"fmbench-solo-2026-paper","name":"FM-Bench (solo) (2026 paper)","group":"Agent 与工具","metric":"Mean final management score","unit":"raw","public_history":true,"protocol":{"en":"Solo-track final management score averaged over three seeds for fifteen models. Each run covers twenty seasons against fifteen fixed scripted clubs with the same endowment, tools and notebook contract; capped revival is part of the task. Score combines long-term financial and sporting outcomes and is not bounded to 100. No human, scripted-anchor or Arena rows are included.","zh":"十五个模型在 Solo 轨道三种随机种子下的最终经营分均值。每轮覆盖二十个赛季，面对十五个固定脚本俱乐部，初始资源、工具及笔记机制统一；有限复活属于任务规则。综合长期财务与竞技结果，分数不以 100 为上限。排除人工、脚本参考及 Arena 结果。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.18423","status":"paper_baselines","published_at":"2026-08-19"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Native score","zh":"原始分"},"model_count":15,"result_count":15,"results":[[3473,90.9,370,370,9054],[420,88.5,370,370,9054],[5103,86.7,370,370,9054],[4972,86.4,370,370,9054],[2937,83.2,370,370,9054],[5104,83.2,370,370,9054],[5105,81.8,370,370,9054],[411,80.7,370,370,9054],[3209,79.2,370,370,9054],[5106,79.1,370,370,9054],[5107,75.7,370,370,9054],[5108,75.0,370,370,9054],[5109,74.6,370,370,9054],[5110,68.4,370,370,9054],[5111,36.9,370,370,9054]]},{"id":"h2rbench-gripper-2026-paper","name":"H2R-Bench (parallel-jaw gripper) (2026 paper)","group":"生成与创作","metric":"H2R Core","unit":"score","public_history":true,"protocol":{"en":"Full eleven-model panel on the same 120 source demonstrations, targeting a parallel-jaw gripper. Uses native source-conditioning interfaces, recorded per row: full video or ordered source frames. Task prompts are aligned, no target-robot reference image is supplied, and every model is scored with the same frame budget and three judges. Native 0–100 transfer composite.","zh":"同一组 120 个源演示、目标为平行夹爪的完整十一模型面板。使用各模型原生输入接口，逐行注明完整视频或有序源帧。任务提示一致，不提供目标机器人参考图，评分帧数预算和三名评委统一。保留 0–100 转换能力综合分。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.13049","status":"paper_baselines","published_at":"2026-08-13"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":11,"result_count":11,"results":[[6383,77.3,370,9055,9056],[7130,76.5,370,9055,9056],[9057,74.5,370,9055,9056],[9058,61.5,370,9059,9056],[9060,49.6,370,9061,9056],[9062,50.1,370,9063,9056],[7128,32.1,370,9059,9056],[9064,31.5,370,9059,9056],[8309,32.4,370,9065,9056],[9066,31.0,370,9059,9056],[9067,30.0,370,9059,9056]]},{"id":"h2rbench-hand-2026-paper","name":"H2R-Bench (dexterous hand) (2026 paper)","group":"生成与创作","metric":"H2R Core","unit":"score","public_history":true,"protocol":{"en":"Full eleven-model panel on the same 120 source demonstrations, targeting a dexterous hand. Uses native source-conditioning interfaces, recorded per row: full video or ordered source frames. Task prompts are aligned, no target-robot reference image is supplied, and every model is scored with the same frame budget and three judges. Native 0–100 transfer composite.","zh":"同一组 120 个源演示、目标为灵巧手的完整十一模型面板。使用各模型原生输入接口，逐行注明完整视频或有序源帧。任务提示一致，不提供目标机器人参考图，评分帧数预算和三名评委统一。保留 0–100 转换能力综合分。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.13049","status":"paper_baselines","published_at":"2026-08-13"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":11,"result_count":11,"results":[[6383,84.6,370,9055,9056],[7130,83.1,370,9055,9056],[9057,81.7,370,9055,9056],[9058,56.1,370,9059,9056],[9060,57.0,370,9061,9056],[9062,49.2,370,9063,9056],[7128,39.8,370,9059,9056],[9064,34.6,370,9059,9056],[8309,33.7,370,9065,9056],[9066,32.0,370,9059,9056],[9067,30.7,370,9059,9056]]},{"id":"aerocopilot-tier2-success-2026-paper","name":"AeroCopilotBench (Tier 2) (2026 paper)","group":"Agent 与工具","metric":"Safety-gated success rate","unit":"%","public_history":true,"protocol":{"en":"Complete twelve-model Tier-2 panel: 73 frozen procedural tasks, three trials each, 219 episodes per model. Success requires all task goals and no hard-safety violation anywhere in the trajectory. Uses task-balanced SR on its source 0–1 fraction scale; SGO partial completion and SCR safety compliance are separate columns.","zh":"完整十二模型 Tier-2 面板：73 个固定程序任务，每题三轮，每模型 219 次运行。成功要求完成全部目标，且整条轨迹不违反任何硬性安全约束。采用按任务等权的 SR，原始值为 0–1 比例；部分完成 SGO 和安全合规 SCR 是独立指标。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.16349","status":"paper_baselines","published_at":"2026-08-17"},"model_count":12,"result_count":12,"results":[[4048,0.726,370,370,9068],[5109,0.594,370,370,9068],[411,0.589,370,370,9068],[3186,0.53,370,370,9068],[3209,0.461,370,370,9068],[227,0.457,370,370,9068],[1321,0.292,370,370,9068],[3187,0.283,370,370,9068],[3005,0.224,370,370,9068],[9069,0.21,370,370,9068],[2576,0.187,370,370,9068],[2582,0.123,370,370,9068]]},{"id":"mathvision-diagrams-common-f1-2026-paper","name":"Math-Vision Diagrams (common rendered subset) (2026 paper)","group":"生成与创作","metric":"Edge F1","unit":"score","public_history":true,"protocol":{"en":"Appendix Table 6 Edge F1 on the common 1,068-prompt subset successfully rendered by all eleven models. Uses the source mean, preserving uncertainty in the full fixture. This is conditional image quality on a shared subset, not whole-benchmark success or compilation rate; model-specific successful-output subsets are not pooled. Native 0–1 F1.","zh":"附录表 6 中十一模型均成功渲染的共同 1,068 题子集上的 Edge F1。采用原文均值，误差项保留在完整来源表。这是共同子集上的图像质量，不是全基准成功率或编译率，不混合各模型不同的成功输出子集；保留 0–1 F1。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.08964","status":"paper_baselines","published_at":"2026-08-09"},"score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":11,"result_count":11,"results":[[9070,0.163,370,370,9071],[9072,0.171,370,370,9071],[3419,0.13,370,370,9071],[244,0.145,370,370,9071],[9073,0.146,370,370,9071],[1487,0.135,370,370,9071],[6366,0.129,370,370,9071],[6839,0.13,370,370,9071],[362,0.106,370,370,9071],[5619,0.107,370,370,9071],[3184,0.08,370,370,9071]]},{"id":"kopa-bench-pass1-2026-paper","name":"KOPA-Bench · pass1 (2026 paper)","group":"Agent 与工具","metric":"Pass@1","unit":"%","public_history":true,"protocol":{"en":"Models complete 145 Korean public-API tasks. Task-specific answer, environment and action checks determine success. Four independent rollouts produce the main-table pass@1 mean and pass@4 union; these are separate cohorts. Nine base checkpoints and two EDGE-GRPO fine-tunes are retained. Code and data: github.com/dneirfi/EDGE-KOPA.","zh":"模型完成145道韩国公共API任务，按每题指定的答案、环境状态与动作检查判断成功。主表每题运行四次，pass@1为单次平均成功率，pass@4为四次至少一次成功率，两项分开展示。保留九个基础检查点和两个EDGE-GRPO微调检查点。代码与数据见github.com/dneirfi/EDGE-KOPA。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.05395v1","published_at":"2026-09-04","status":"paper_baselines"},"model_count":11,"result_count":11,"results":[[3572,0.4655,370,9074,9075],[3748,0.3706,370,9074,9075],[2584,0.4482,370,9074,9075],[9076,0.3534,370,9074,9075],[9077,0.2586,370,9074,9075],[4070,0.3275,370,9074,9075],[5592,0.1862,370,9074,9075],[4969,0.1758,370,9074,9075],[7020,0.1299,370,9074,9075],[9078,0.3094,370,9079,9075],[9080,0.431,370,9079,9075]]},{"id":"kopa-bench-pass4-2026-paper","name":"KOPA-Bench · pass4 (2026 paper)","group":"Agent 与工具","metric":"Pass@4","unit":"%","public_history":true,"protocol":{"en":"Models complete 145 Korean public-API tasks. Task-specific answer, environment and action checks determine success. Four independent rollouts produce the main-table pass@1 mean and pass@4 union; these are separate cohorts. Nine base checkpoints and two EDGE-GRPO fine-tunes are retained. Code and data: github.com/dneirfi/EDGE-KOPA.","zh":"模型完成145道韩国公共API任务，按每题指定的答案、环境状态与动作检查判断成功。主表每题运行四次，pass@1为单次平均成功率，pass@4为四次至少一次成功率，两项分开展示。保留九个基础检查点和两个EDGE-GRPO微调检查点。代码与数据见github.com/dneirfi/EDGE-KOPA。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.05395v1","published_at":"2026-09-04","status":"paper_baselines"},"model_count":11,"result_count":11,"results":[[3572,0.8207,370,9081,9075],[3748,0.469,370,9081,9075],[2584,0.5655,370,9081,9075],[9076,0.469,370,9081,9075],[9077,0.3862,370,9081,9075],[4070,0.469,370,9081,9075],[5592,0.2207,370,9081,9075],[4969,0.3103,370,9081,9075],[7020,0.2308,370,9081,9075],[9078,0.469,370,9082,9075],[9080,0.5517,370,9082,9075]]},{"id":"erp-decision-solo-2026-paper","name":"ERPBench · Solo market (2026 paper)","group":"预测与决策","metric":"Mean terminal valuation (millions)","unit":"raw","public_history":true,"protocol":{"en":"Six models make pricing, production, procurement, inventory and finance decisions over six rounds on 100 matched enterprise problems. Solo uses fixed rule-based opponents; Arena places the six evaluated agents in one market. This cohort preserves sixth-round terminal valuation in millions, including the published execution-recovery scaffold. Each ecology has 600 outcome records. Tasks and simulator are linked by the paper.","zh":"六个模型在100个匹配的企业问题中进行六轮定价、生产、采购、库存和财务决策。Solo面对固定规则对手，Arena让六个被测智能体在同一市场竞争。保留第六轮终值，单位为百万，成绩包含论文使用的执行恢复流程。每种市场设置均有600条结果，论文提供任务与仿真器链接。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.04667v1","published_at":"2026-09-04","status":"paper_baselines"},"saturation_excluded":true,"scale_label":{"en":"Valuation (millions)","zh":"终值（百万）"},"score_decimals":2,"model_count":6,"result_count":6,"results":[[3201,252.29,370,9083,9084],[3482,169.41,370,9083,9084],[3196,186.43,370,9083,9084],[3587,205.87,370,9083,9084],[4225,181.27,370,9083,9084],[2261,176.78,370,9083,9084]]},{"id":"erp-decision-arena-2026-paper","name":"ERPBench · Arena market (2026 paper)","group":"预测与决策","metric":"Mean terminal valuation (millions)","unit":"raw","public_history":true,"protocol":{"en":"Six models make pricing, production, procurement, inventory and finance decisions over six rounds on 100 matched enterprise problems. Solo uses fixed rule-based opponents; Arena places the six evaluated agents in one market. This cohort preserves sixth-round terminal valuation in millions, including the published execution-recovery scaffold. Each ecology has 600 outcome records. Tasks and simulator are linked by the paper.","zh":"六个模型在100个匹配的企业问题中进行六轮定价、生产、采购、库存和财务决策。Solo面对固定规则对手，Arena让六个被测智能体在同一市场竞争。保留第六轮终值，单位为百万，成绩包含论文使用的执行恢复流程。每种市场设置均有600条结果，论文提供任务与仿真器链接。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.04667v1","published_at":"2026-09-04","status":"paper_baselines"},"saturation_excluded":true,"scale_label":{"en":"Valuation (millions)","zh":"终值（百万）"},"score_decimals":2,"model_count":6,"result_count":6,"results":[[3201,234.71,370,9085,9084],[3482,263.95,370,9085,9084],[3196,194.82,370,9085,9084],[3587,187.13,370,9085,9084],[4225,188.17,370,9085,9084],[2261,173.95,370,9085,9084]]},{"id":"elderbench-overall-2026-paper","name":"ElderBench · overall (2026 paper)","group":"Agent 与工具","metric":"Task success rate","unit":"%","public_history":true,"protocol":{"en":"Nine models complete 249 elderly-authored smartphone tasks in the overall panel. Online success is judged by Qwen 3-VL-Flash from final and intermediate screenshots and Android Activity; human review of AutoGLM agrees 93.08%. Offline actions must match valid edges in human-demonstrated task graphs. Original instructions,25-step and 120-second limits are retained. The paper specifies the evaluation; independent task/scorer artifact availability is unverified.","zh":"九个模型在overall面板完成249道老年人提出的手机任务。线上由Qwen3-VL-Flash结合最终与中间截图、Android Activity判断成功，与AutoGLM结果的人审一致率为93.08%；线下动作须匹配人工示范任务图的有效边。保留原始指令、25步及120秒限制。论文公开评测协议，独立任务与评分器产物的可用性尚未核实。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.04850v1","published_at":"2026-09-04","status":"paper_baselines"},"model_count":9,"result_count":9,"results":[[6979,0.2289,370,9086,9087],[5722,0.2329,370,9086,9087],[244,0.29719999999999996,370,9086,9087],[3247,0.4739,370,9086,9087],[6133,0.498,370,9086,9087],[9088,0.32530000000000003,370,9086,9087],[9089,0.45380000000000004,370,9086,9087],[9090,0.41369999999999996,370,9086,9087],[9091,0.3655,370,9086,9087]]},{"id":"elderbench-online-2026-paper","name":"ElderBench · online (2026 paper)","group":"Agent 与工具","metric":"Task success rate","unit":"%","public_history":true,"protocol":{"en":"Nine models complete 130 elderly-authored smartphone tasks in the online panel. Online success is judged by Qwen 3-VL-Flash from final and intermediate screenshots and Android Activity; human review of AutoGLM agrees 93.08%. Offline actions must match valid edges in human-demonstrated task graphs. Original instructions,25-step and 120-second limits are retained. The paper specifies the evaluation; independent task/scorer artifact availability is unverified.","zh":"九个模型在online面板完成130道老年人提出的手机任务。线上由Qwen3-VL-Flash结合最终与中间截图、Android Activity判断成功，与AutoGLM结果的人审一致率为93.08%；线下动作须匹配人工示范任务图的有效边。保留原始指令、25步及120秒限制。论文公开评测协议，独立任务与评分器产物的可用性尚未核实。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.04850v1","published_at":"2026-09-04","status":"paper_baselines"},"model_count":9,"result_count":9,"results":[[6979,0.4,370,9092,9087],[5722,0.3385,370,9092,9087],[244,0.4385,370,9092,9087],[3247,0.7615000000000001,370,9092,9087],[6133,0.7,370,9092,9087],[9088,0.5307999999999999,370,9092,9087],[9089,0.6846,370,9092,9087],[9090,0.5077,370,9092,9087],[9091,0.5154,370,9092,9087]]},{"id":"elderbench-offline-2026-paper","name":"ElderBench · offline (2026 paper)","group":"Agent 与工具","metric":"Task success rate","unit":"%","public_history":true,"protocol":{"en":"Nine models complete 119 elderly-authored smartphone tasks in the offline panel. Online success is judged by Qwen 3-VL-Flash from final and intermediate screenshots and Android Activity; human review of AutoGLM agrees 93.08%. Offline actions must match valid edges in human-demonstrated task graphs. Original instructions,25-step and 120-second limits are retained. The paper specifies the evaluation; independent task/scorer artifact availability is unverified.","zh":"九个模型在offline面板完成119道老年人提出的手机任务。线上由Qwen3-VL-Flash结合最终与中间截图、Android Activity判断成功，与AutoGLM结果的人审一致率为93.08%；线下动作须匹配人工示范任务图的有效边。保留原始指令、25步及120秒限制。论文公开评测协议，独立任务与评分器产物的可用性尚未核实。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.04850v1","published_at":"2026-09-04","status":"paper_baselines"},"model_count":9,"result_count":9,"results":[[6979,0.042,370,9093,9087],[5722,0.1176,370,9093,9087],[244,0.1429,370,9093,9087],[3247,0.1597,370,9093,9087],[6133,0.2773,370,9093,9087],[9088,0.1008,370,9093,9087],[9089,0.20170000000000002,370,9093,9087],[9090,0.3109,370,9093,9087],[9091,0.20170000000000002,370,9093,9087]]},{"id":"failbench-macro-2026-paper","name":"FailBench · macro detection (2026 paper)","group":"视觉与视频","metric":"Macro balanced accuracy","unit":"%","public_history":true,"protocol":{"en":"Thirteen detectors classify recorded robot attempts as success or failure. The published macro score equally weights 13 two-class source subsets; the failure-only REFLECT subset is excluded. Each source defines outcome labels and visual views, with video input or 32 evenly spaced frames and author-recommended detector prompts. This is label-balanced detection accuracy, not robot execution success. Independent artifact availability is unverified.","zh":"十三个检测模型判断机器人执行录像中的任务是否成功。采用原表宏平均平衡正确率，对十三个含成功与失败样本的来源子集等权，只含失败的REFLECT子集不计入。各来源规定结果标签与视觉视角，按模型支持输入视频或均匀抽取32帧，并采用作者推荐提示。成绩衡量成功与失败的识别，不是机器人执行成功率。独立产物可用性尚未核实。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.03611v1","published_at":"2026-09-03","status":"paper_baselines"},"model_count":13,"result_count":13,"results":[[3424,0.77,370,9094,9095],[6143,0.75,370,9094,9095],[2626,0.69,370,9094,9095],[353,0.69,370,9094,9095],[7954,0.65,370,9094,9095],[9096,0.63,370,9094,9095],[9097,0.62,370,9094,9095],[7539,0.61,370,9094,9095],[9098,0.59,370,9094,9095],[9099,0.54,370,9094,9095],[9100,0.53,370,9094,9095],[9101,0.51,370,9094,9095],[9102,0.5,370,9094,9095]]},{"id":"kc-bench-region-human-2026-paper","name":"KC-Bench · region conflicts (2026 paper)","group":"Agent 与工具","metric":"Human-verified task success","unit":"%","public_history":true,"protocol":{"en":"Nine agents resolve world-knowledge conflicts in the Region domain. Typed read-only tools expose deterministic factual queries; human verification adjudicates traces against expected outcomes and environment state. Values use the manual half of each automatic/manual table cell. Greedy decoding,30 interaction steps,one trial and seed 300 follow the paper. Retail and Personal Assistant have unresolved apparent denominator differences and are not part of this panel.","zh":"九个智能体在Region领域处理世界知识冲突，以具有明确类型的只读工具查询事实。人审结合预期结果与环境状态复核执行轨迹，采用原表自动/人工两项中的人工结果。沿用贪心解码、30步、单次运行与随机种子300。Retail和Personal Assistant存在尚待解释的分母差异，另行待核。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.03588v1","published_at":"2026-09-03","status":"paper_baselines"},"model_count":9,"result_count":9,"results":[[3748,0.4085,370,9103,9104],[4056,0.0704,370,9103,9104],[9105,0.4507,370,9103,9104],[9106,0.3804,370,9103,9104],[62,0.0845,370,9103,9104],[2590,0.507,370,9103,9104],[3201,0.1831,370,9103,9104],[227,0.5352,370,9103,9104],[3198,0.6761,370,9103,9104]]},{"id":"vex-bench-status-2026-paper","name":"VEX-Bench · status (2026 paper)","group":"安全与可靠性","metric":"Vulnerability-status accuracy","unit":"%","public_history":true,"protocol":{"en":"Nine model-harness configurations assess exploitability of 75 dependency-vulnerability cases from 35 repositories, labeled by security specialists with independent review. Each configuration runs three times with a 10-minute task limit. The original mean is preserved; out-of-label predictions remain FAILED. Binary status, five-class justification accuracy and macro-F1 are separate metrics. Tasks and evaluation scripts are linked at github.com/steven 1518/vex-bench.","zh":"九种模型与框架配置判断35个代码仓库中75个依赖漏洞案例的可利用性，标签由安全研究人员标注并交叉复核。每种配置重复三次，每题限时10分钟。保留原表均值，标签范围外的预测记为FAILED。二元状态正确率、五类理由正确率及Macro-F1分别展示。任务和评测脚本见github.com/steven1518/vex-bench。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.08040v1","published_at":"2026-09-07","status":"paper_baselines"},"model_count":9,"result_count":9,"results":[[2261,0.884,370,9107,9108],[3572,0.818,370,9107,9108],[3196,0.893,370,9109,9108],[5449,0.8220000000000001,370,9109,9108],[3490,0.818,370,9110,9108],[3185,0.84,370,9110,9108],[3201,0.7290000000000001,370,9110,9108],[3190,0.711,370,9110,9108],[3186,0.8440000000000001,370,9110,9108]]},{"id":"vex-bench-justification-2026-paper","name":"VEX-Bench · justification (2026 paper)","group":"安全与可靠性","metric":"Justification accuracy","unit":"%","public_history":true,"protocol":{"en":"Nine model-harness configurations assess exploitability of 75 dependency-vulnerability cases from 35 repositories, labeled by security specialists with independent review. Each configuration runs three times with a 10-minute task limit. The original mean is preserved; out-of-label predictions remain FAILED. Binary status, five-class justification accuracy and macro-F1 are separate metrics. Tasks and evaluation scripts are linked at github.com/steven 1518/vex-bench.","zh":"九种模型与框架配置判断35个代码仓库中75个依赖漏洞案例的可利用性，标签由安全研究人员标注并交叉复核。每种配置重复三次，每题限时10分钟。保留原表均值，标签范围外的预测记为FAILED。二元状态正确率、五类理由正确率及Macro-F1分别展示。任务和评测脚本见github.com/steven1518/vex-bench。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.08040v1","published_at":"2026-09-07","status":"paper_baselines"},"model_count":9,"result_count":9,"results":[[2261,0.764,370,9107,9108],[3572,0.6709999999999999,370,9107,9108],[3196,0.782,370,9109,9108],[5449,0.667,370,9109,9108],[3490,0.7290000000000001,370,9110,9108],[3185,0.76,370,9110,9108],[3201,0.5870000000000001,370,9110,9108],[3190,0.591,370,9110,9108],[3186,0.747,370,9110,9108]]},{"id":"vex-bench-justification-f1-2026-paper","name":"VEX-Bench · justification-f1 (2026 paper)","group":"安全与可靠性","metric":"Justification macro-F1","unit":"score","public_history":true,"protocol":{"en":"Nine model-harness configurations assess exploitability of 75 dependency-vulnerability cases from 35 repositories, labeled by security specialists with independent review. Each configuration runs three times with a 10-minute task limit. The original mean is preserved; out-of-label predictions remain FAILED. Binary status, five-class justification accuracy and macro-F1 are separate metrics. Tasks and evaluation scripts are linked at github.com/steven 1518/vex-bench.","zh":"九种模型与框架配置判断35个代码仓库中75个依赖漏洞案例的可利用性，标签由安全研究人员标注并交叉复核。每种配置重复三次，每题限时10分钟。保留原表均值，标签范围外的预测记为FAILED。二元状态正确率、五类理由正确率及Macro-F1分别展示。任务和评测脚本见github.com/steven1518/vex-bench。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.08040v1","published_at":"2026-09-07","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":9,"result_count":9,"results":[[2261,61.1,370,9107,9108],[3572,56.1,370,9107,9108],[3196,73.5,370,9109,9108],[5449,60.4,370,9109,9108],[3490,64.6,370,9110,9108],[3185,60.3,370,9110,9108],[3201,47.2,370,9110,9108],[3190,44.8,370,9110,9108],[3186,61.7,370,9110,9108]]},{"id":"rescue-relation-pattern-2026-paper","name":"RESCUE-Bench · relation-pattern (2026 paper)","group":"专业工作","metric":"Relation-pattern accuracy","unit":"%","public_history":true,"protocol":{"en":"Ten LLMs use identical zero-shot task prompts and candidate labels on human-verified annotations from 191 couple/family interview samples with 7,079 turns. Relation-pattern classification and binary intervention-timing F1 are kept separate. Draft annotations are checked against subtitle, video and audio evidence before evaluation instances are constructed. Code/data: github.com/Tomsawyerhu/RESCUE-bench.","zh":"十个LLM使用相同零样本提示与候选标签，在191段伴侣或家庭访谈、7,079个标注轮次构成的数据上评测。关系模式分类正确率与是否介入的F1分别展示。标注初稿经字幕、视频和音频证据人工核实后，才用于构建评测样本。代码与数据见github.com/Tomsawyerhu/RESCUE-bench。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.09657v1","published_at":"2026-09-09","status":"paper_baselines"},"model_count":10,"result_count":10,"results":[[9111,0.38270000000000004,370,9112,9113],[3225,0.4322,370,9112,9113],[6412,0.40270000000000006,370,9112,9113],[769,0.3789,370,9112,9113],[3009,0.3608,370,9112,9113],[3201,0.39390000000000003,370,9112,9113],[3185,0.456,370,9112,9113],[244,0.4526,370,9112,9113],[4424,0.3985,370,9112,9113],[353,0.38689999999999997,370,9112,9113]]},{"id":"rescue-intervention-timing-2026-paper","name":"RESCUE-Bench · intervention-timing (2026 paper)","group":"专业工作","metric":"Intervention-timing F1","unit":"score","public_history":true,"protocol":{"en":"Ten LLMs use identical zero-shot task prompts and candidate labels on human-verified annotations from 191 couple/family interview samples with 7,079 turns. Relation-pattern classification and binary intervention-timing F1 are kept separate. Draft annotations are checked against subtitle, video and audio evidence before evaluation instances are constructed. Code/data: github.com/Tomsawyerhu/RESCUE-bench.","zh":"十个LLM使用相同零样本提示与候选标签，在191段伴侣或家庭访谈、7,079个标注轮次构成的数据上评测。关系模式分类正确率与是否介入的F1分别展示。标注初稿经字幕、视频和音频证据人工核实后，才用于构建评测样本。代码与数据见github.com/Tomsawyerhu/RESCUE-bench。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.09657v1","published_at":"2026-09-09","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":10,"result_count":10,"results":[[9111,68.87,370,9112,9113],[3225,75.14,370,9112,9113],[6412,94.6,370,9112,9113],[769,80.36,370,9112,9113],[3009,86.86,370,9112,9113],[3201,84.22,370,9112,9113],[3185,83.28,370,9112,9113],[244,85.98,370,9112,9113],[4424,82.25,370,9112,9113],[353,84.6,370,9112,9113]]},{"id":"twinicl-selection-text-2026-paper","name":"TwinICL · selection · text (2026 paper)","group":"知识与推理","metric":"16-shot task accuracy","unit":"%","public_history":true,"protocol":{"en":"Six models infer selection rules from 16 paired demonstrations with text inputs and textual answers. Exact token-sequence matching preserves order and repetitions. Table 17 verifies identical 16-shot coverage for all six models:606 run combinations across 38 tasks, with three seeds for every included rendering variant. Family scores average tasks equally after seeds/variants. The other 306 planned renderings are unrun, not zero scores. Code/data: github.com/lab-flair/TwinICL.","zh":"六个模型从16个配对示范推断selection规则，使用text输入并输出文本。评分要求归一化后的词元序列精确一致，包括顺序和重复。表17确认六个模型的16-shot覆盖相同：38项任务共606个运行组合，所有已纳入渲染变体均含三个种子。先按种子与变体平均，再对任务等权；其余306个计划渲染组合未运行，不记为零分。代码与数据见github.com/lab-flair/TwinICL。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.15028v1","published_at":"2026-09-14","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[2590,0.652,370,9114,9115],[2584,0.742,370,9114,9115],[4969,0.605,370,9114,9115],[9116,0.951,370,9114,9115],[9117,0.895,370,9114,9115],[9118,0.6890000000000001,370,9114,9115]]},{"id":"twinicl-selection-image-2026-paper","name":"TwinICL · selection · image (2026 paper)","group":"知识与推理","metric":"16-shot task accuracy","unit":"%","public_history":true,"protocol":{"en":"Six models infer selection rules from 16 paired demonstrations with image inputs and textual answers. Exact token-sequence matching preserves order and repetitions. Table 17 verifies identical 16-shot coverage for all six models:606 run combinations across 38 tasks, with three seeds for every included rendering variant. Family scores average tasks equally after seeds/variants. The other 306 planned renderings are unrun, not zero scores. Code/data: github.com/lab-flair/TwinICL.","zh":"六个模型从16个配对示范推断selection规则，使用image输入并输出文本。评分要求归一化后的词元序列精确一致，包括顺序和重复。表17确认六个模型的16-shot覆盖相同：38项任务共606个运行组合，所有已纳入渲染变体均含三个种子。先按种子与变体平均，再对任务等权；其余306个计划渲染组合未运行，不记为零分。代码与数据见github.com/lab-flair/TwinICL。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.15028v1","published_at":"2026-09-14","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[2590,0.478,370,9119,9115],[2584,0.516,370,9119,9115],[4969,0.3,370,9119,9115],[9116,0.601,370,9119,9115],[9117,0.4,370,9119,9115],[9118,0.33899999999999997,370,9119,9115]]},{"id":"twinicl-relation-text-2026-paper","name":"TwinICL · relation · text (2026 paper)","group":"知识与推理","metric":"16-shot task accuracy","unit":"%","public_history":true,"protocol":{"en":"Six models infer relation rules from 16 paired demonstrations with text inputs and textual answers. Exact token-sequence matching preserves order and repetitions. Table 17 verifies identical 16-shot coverage for all six models:606 run combinations across 38 tasks, with three seeds for every included rendering variant. Family scores average tasks equally after seeds/variants. The other 306 planned renderings are unrun, not zero scores. Code/data: github.com/lab-flair/TwinICL.","zh":"六个模型从16个配对示范推断relation规则，使用text输入并输出文本。评分要求归一化后的词元序列精确一致，包括顺序和重复。表17确认六个模型的16-shot覆盖相同：38项任务共606个运行组合，所有已纳入渲染变体均含三个种子。先按种子与变体平均，再对任务等权；其余306个计划渲染组合未运行，不记为零分。代码与数据见github.com/lab-flair/TwinICL。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.15028v1","published_at":"2026-09-14","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[2590,0.63,370,9114,9115],[2584,0.623,370,9114,9115],[4969,0.617,370,9114,9115],[9116,0.65,370,9114,9115],[9117,0.632,370,9114,9115],[9118,0.632,370,9114,9115]]},{"id":"twinicl-relation-image-2026-paper","name":"TwinICL · relation · image (2026 paper)","group":"知识与推理","metric":"16-shot task accuracy","unit":"%","public_history":true,"protocol":{"en":"Six models infer relation rules from 16 paired demonstrations with image inputs and textual answers. Exact token-sequence matching preserves order and repetitions. Table 17 verifies identical 16-shot coverage for all six models:606 run combinations across 38 tasks, with three seeds for every included rendering variant. Family scores average tasks equally after seeds/variants. The other 306 planned renderings are unrun, not zero scores. Code/data: github.com/lab-flair/TwinICL.","zh":"六个模型从16个配对示范推断relation规则，使用image输入并输出文本。评分要求归一化后的词元序列精确一致，包括顺序和重复。表17确认六个模型的16-shot覆盖相同：38项任务共606个运行组合，所有已纳入渲染变体均含三个种子。先按种子与变体平均，再对任务等权；其余306个计划渲染组合未运行，不记为零分。代码与数据见github.com/lab-flair/TwinICL。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.15028v1","published_at":"2026-09-14","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[2590,0.5870000000000001,370,9119,9115],[2584,0.61,370,9119,9115],[4969,0.625,370,9119,9115],[9116,0.616,370,9119,9115],[9117,0.611,370,9119,9115],[9118,0.612,370,9119,9115]]},{"id":"twinicl-aggregation-text-2026-paper","name":"TwinICL · aggregation · text (2026 paper)","group":"知识与推理","metric":"16-shot task accuracy","unit":"%","public_history":true,"protocol":{"en":"Six models infer aggregation rules from 16 paired demonstrations with text inputs and textual answers. Exact token-sequence matching preserves order and repetitions. Table 17 verifies identical 16-shot coverage for all six models:606 run combinations across 38 tasks, with three seeds for every included rendering variant. Family scores average tasks equally after seeds/variants. The other 306 planned renderings are unrun, not zero scores. Code/data: github.com/lab-flair/TwinICL.","zh":"六个模型从16个配对示范推断aggregation规则，使用text输入并输出文本。评分要求归一化后的词元序列精确一致，包括顺序和重复。表17确认六个模型的16-shot覆盖相同：38项任务共606个运行组合，所有已纳入渲染变体均含三个种子。先按种子与变体平均，再对任务等权；其余306个计划渲染组合未运行，不记为零分。代码与数据见github.com/lab-flair/TwinICL。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.15028v1","published_at":"2026-09-14","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[2590,0.845,370,9114,9115],[2584,0.898,370,9114,9115],[4969,0.7140000000000001,370,9114,9115],[9116,0.953,370,9114,9115],[9117,0.816,370,9114,9115],[9118,0.74,370,9114,9115]]},{"id":"twinicl-aggregation-image-2026-paper","name":"TwinICL · aggregation · image (2026 paper)","group":"知识与推理","metric":"16-shot task accuracy","unit":"%","public_history":true,"protocol":{"en":"Six models infer aggregation rules from 16 paired demonstrations with image inputs and textual answers. Exact token-sequence matching preserves order and repetitions. Table 17 verifies identical 16-shot coverage for all six models:606 run combinations across 38 tasks, with three seeds for every included rendering variant. Family scores average tasks equally after seeds/variants. The other 306 planned renderings are unrun, not zero scores. Code/data: github.com/lab-flair/TwinICL.","zh":"六个模型从16个配对示范推断aggregation规则，使用image输入并输出文本。评分要求归一化后的词元序列精确一致，包括顺序和重复。表17确认六个模型的16-shot覆盖相同：38项任务共606个运行组合，所有已纳入渲染变体均含三个种子。先按种子与变体平均，再对任务等权；其余306个计划渲染组合未运行，不记为零分。代码与数据见github.com/lab-flair/TwinICL。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.15028v1","published_at":"2026-09-14","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[2590,0.715,370,9119,9115],[2584,0.733,370,9119,9115],[4969,0.698,370,9119,9115],[9116,0.6890000000000001,370,9119,9115],[9117,0.675,370,9119,9115],[9118,0.495,370,9119,9115]]},{"id":"twinicl-transformation-text-2026-paper","name":"TwinICL · transformation · text (2026 paper)","group":"知识与推理","metric":"16-shot task accuracy","unit":"%","public_history":true,"protocol":{"en":"Six models infer transformation rules from 16 paired demonstrations with text inputs and textual answers. Exact token-sequence matching preserves order and repetitions. Table 17 verifies identical 16-shot coverage for all six models:606 run combinations across 38 tasks, with three seeds for every included rendering variant. Family scores average tasks equally after seeds/variants. The other 306 planned renderings are unrun, not zero scores. Code/data: github.com/lab-flair/TwinICL.","zh":"六个模型从16个配对示范推断transformation规则，使用text输入并输出文本。评分要求归一化后的词元序列精确一致，包括顺序和重复。表17确认六个模型的16-shot覆盖相同：38项任务共606个运行组合，所有已纳入渲染变体均含三个种子。先按种子与变体平均，再对任务等权；其余306个计划渲染组合未运行，不记为零分。代码与数据见github.com/lab-flair/TwinICL。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.15028v1","published_at":"2026-09-14","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[2590,0.317,370,9114,9115],[2584,0.523,370,9114,9115],[4969,0.213,370,9114,9115],[9116,0.574,370,9114,9115],[9117,0.43,370,9114,9115],[9118,0.151,370,9114,9115]]},{"id":"twinicl-transformation-image-2026-paper","name":"TwinICL · transformation · image (2026 paper)","group":"知识与推理","metric":"16-shot task accuracy","unit":"%","public_history":true,"protocol":{"en":"Six models infer transformation rules from 16 paired demonstrations with image inputs and textual answers. Exact token-sequence matching preserves order and repetitions. Table 17 verifies identical 16-shot coverage for all six models:606 run combinations across 38 tasks, with three seeds for every included rendering variant. Family scores average tasks equally after seeds/variants. The other 306 planned renderings are unrun, not zero scores. Code/data: github.com/lab-flair/TwinICL.","zh":"六个模型从16个配对示范推断transformation规则，使用image输入并输出文本。评分要求归一化后的词元序列精确一致，包括顺序和重复。表17确认六个模型的16-shot覆盖相同：38项任务共606个运行组合，所有已纳入渲染变体均含三个种子。先按种子与变体平均，再对任务等权；其余306个计划渲染组合未运行，不记为零分。代码与数据见github.com/lab-flair/TwinICL。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.15028v1","published_at":"2026-09-14","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[2590,0.185,370,9119,9115],[2584,0.218,370,9119,9115],[4969,0.08,370,9119,9115],[9116,0.16,370,9119,9115],[9117,0.067,370,9119,9115],[9118,0.017,370,9119,9115]]},{"id":"hob-vl-boolean-nl-2026-paper","name":"Hob-VL · natural-language Boolean (2026 paper)","group":"视觉与视频","metric":"Boolean accuracy","unit":"%","public_history":true,"protocol":{"en":"All 6,000 natural-language Boolean questions. Eight models with a separate GLM thinking configuration. API errors were retried; every question has a final response. Completed malformed or token-limited outputs count incorrect.","zh":"全部 6,000 道自然语言布尔题。八模型，另含 GLM 思考配置。API 错误重试后每题均有最终响应；已生成但格式无效或达到长度上限的输出计错。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.01605","status":"paper_baselines","published_at":"2026-10-01"},"model_count":8,"result_count":9,"results":[[3408,0.4867,370,9120,9121],[325,0.5022,370,9120,9121],[3493,0.49570000000000003,370,9120,9121],[4035,0.4852,370,9120,9121],[5394,0.49670000000000003,370,9120,9121],[9122,0.5036999999999999,370,9120,9121],[363,0.4893,370,9120,9121],[6150,0.5036999999999999,370,9120,9121],[9122,0.512,370,9123,9121]]},{"id":"ego2world-operational-2026-paper","name":"Ego2World (2026 paper)","group":"Agent 与工具","metric":"Operational task success","unit":"%","public_history":true,"protocol":{"en":"Six planners on the same 105 frozen tasks and 3,441 groups each. Success requires queue completion and all contract conditions. Parse failures, empty plans and execution-blocked terminal statuses are observed failures; all 630 task outcomes are retained.","zh":"六个规划模型使用同一批 105 项固定任务，每模型 3,441 组。成功要求队列完成且全部任务条件满足；解析失败、空计划及执行受阻状态均保留为实测失败，共 630 条任务结果。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.02715","status":"paper_baselines","published_at":"2026-10-02"},"model_count":6,"result_count":6,"results":[[9111,0.0571,370,7834,9124],[3196,0.0571,370,7834,9124],[3412,0.0667,370,7834,9124],[3201,0.0381,370,7834,9124],[268,0.0381,370,7834,9124],[3572,0.0571,370,7834,9124]]},{"id":"plcworld-motion-control-easy-2026-paper","name":"PLCWorld · Motion Control Easy (2026 paper)","group":"编程","metric":"Task success rate","unit":"%","public_history":true,"protocol":{"en":"Direct Structured Text generation by six models, Motion Control / Easy. Same task rules and execution tests. Model-attributable submission and execution failures count unsuccessful; infrastructure errors are excluded.","zh":"六模型直接生成 Structured Text，Motion Control / Easy。使用相同任务规则与执行测试；模型提交或运行错误计失败，基础设施错误排除。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.02982","status":"paper_baselines","published_at":"2026-10-02"},"model_count":6,"result_count":6,"results":[[3196,0.7111,370,9125,9126],[353,0.1444,370,9125,9126],[3568,0.75,370,9125,9126],[3412,0.7833,370,9125,9126],[2983,0.9556,370,9125,9126],[6412,0.6056,370,9125,9126]]},{"id":"plcworld-motion-control-medium-2026-paper","name":"PLCWorld · Motion Control Medium (2026 paper)","group":"编程","metric":"Task success rate","unit":"%","public_history":true,"protocol":{"en":"Direct Structured Text generation by six models, Motion Control / Medium. Same task rules and execution tests. Model-attributable submission and execution failures count unsuccessful; infrastructure errors are excluded.","zh":"六模型直接生成 Structured Text，Motion Control / Medium。使用相同任务规则与执行测试；模型提交或运行错误计失败，基础设施错误排除。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.02982","status":"paper_baselines","published_at":"2026-10-02"},"model_count":6,"result_count":6,"results":[[3196,0.7558,370,9127,9126],[353,0.0891,370,9127,9126],[3568,0.7591,370,9127,9126],[3412,0.8086,370,9127,9126],[2983,0.769,370,9127,9126],[6412,0.5512,370,9127,9126]]},{"id":"plcworld-motion-control-hard-2026-paper","name":"PLCWorld · Motion Control Hard (2026 paper)","group":"编程","metric":"Task success rate","unit":"%","public_history":true,"protocol":{"en":"Direct Structured Text generation by six models, Motion Control / Hard. Same task rules and execution tests. Model-attributable submission and execution failures count unsuccessful; infrastructure errors are excluded.","zh":"六模型直接生成 Structured Text，Motion Control / Hard。使用相同任务规则与执行测试；模型提交或运行错误计失败，基础设施错误排除。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.02982","status":"paper_baselines","published_at":"2026-10-02"},"model_count":6,"result_count":6,"results":[[3196,0.18059999999999998,370,9128,9126],[353,0.0,370,9128,9126],[3568,0.0,370,9128,9126],[3412,0.11810000000000001,370,9128,9126],[2983,0.048600000000000004,370,9128,9126],[6412,0.0,370,9128,9126]]},{"id":"plcworld-material-handling-easy-2026-paper","name":"PLCWorld · Material Handling Easy (2026 paper)","group":"编程","metric":"Task success rate","unit":"%","public_history":true,"protocol":{"en":"Direct Structured Text generation by six models, Material Handling / Easy. Same task rules and execution tests. Model-attributable submission and execution failures count unsuccessful; infrastructure errors are excluded.","zh":"六模型直接生成 Structured Text，Material Handling / Easy。使用相同任务规则与执行测试；模型提交或运行错误计失败，基础设施错误排除。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.02982","status":"paper_baselines","published_at":"2026-10-02"},"model_count":6,"result_count":6,"results":[[3196,0.9783,370,9129,9126],[353,0.2899,370,9129,9126],[3568,0.8841,370,9129,9126],[3412,0.9420000000000001,370,9129,9126],[2983,0.9854999999999999,370,9129,9126],[6412,0.8623000000000001,370,9129,9126]]},{"id":"plcworld-material-handling-medium-2026-paper","name":"PLCWorld · Material Handling Medium (2026 paper)","group":"编程","metric":"Task success rate","unit":"%","public_history":true,"protocol":{"en":"Direct Structured Text generation by six models, Material Handling / Medium. Same task rules and execution tests. Model-attributable submission and execution failures count unsuccessful; infrastructure errors are excluded.","zh":"六模型直接生成 Structured Text，Material Handling / Medium。使用相同任务规则与执行测试；模型提交或运行错误计失败，基础设施错误排除。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.02982","status":"paper_baselines","published_at":"2026-10-02"},"model_count":6,"result_count":6,"results":[[3196,0.9306,370,9130,9126],[353,0.0104,370,9130,9126],[3568,0.7743000000000001,370,9130,9126],[3412,0.934,370,9130,9126],[2983,0.875,370,9130,9126],[6412,0.4618,370,9130,9126]]},{"id":"plcworld-material-handling-hard-2026-paper","name":"PLCWorld · Material Handling Hard (2026 paper)","group":"编程","metric":"Task success rate","unit":"%","public_history":true,"protocol":{"en":"Direct Structured Text generation by six models, Material Handling / Hard. Same task rules and execution tests. Model-attributable submission and execution failures count unsuccessful; infrastructure errors are excluded.","zh":"六模型直接生成 Structured Text，Material Handling / Hard。使用相同任务规则与执行测试；模型提交或运行错误计失败，基础设施错误排除。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.02982","status":"paper_baselines","published_at":"2026-10-02"},"model_count":6,"result_count":6,"results":[[3196,0.2787,370,9131,9126],[353,0.0,370,9131,9126],[3568,0.0437,370,9131,9126],[3412,0.3525,370,9131,9126],[2983,0.1721,370,9131,9126],[6412,0.016399999999999998,370,9131,9126]]},{"id":"syntaxbench-count-chars-2026-paper","name":"SyntaxBench · count-chars (2026 paper)","group":"知识与推理","metric":"Exact-match accuracy","unit":"%","public_history":true,"protocol":{"en":"Six models, non-thinking, four-shot. count-chars with matched English and random-string conditions, kept as configurations. All selected cells use the full task set; index-to-span is excluded because one model has a missing/partial condition.","zh":"六模型、非思考、四样本提示；count-chars 的英语和随机字符串条件分别记为配置。所选单元格均使用完整任务集；index-to-span 存在缺失或部分条件，不并入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.03329","status":"paper_baselines","published_at":"2026-10-02"},"model_count":6,"result_count":12,"results":[[9132,0.76,370,9133,9134],[5480,0.945,370,9133,9134],[2584,0.915,370,9133,9134],[5828,0.755,370,9133,9134],[4070,0.79,370,9133,9134],[3763,0.965,370,9133,9134],[9132,0.845,370,9135,9134],[5480,0.98,370,9135,9134],[2584,1.0,370,9135,9134],[5828,0.64,370,9135,9134],[4070,0.97,370,9135,9134],[3763,0.995,370,9135,9134]]},{"id":"syntaxbench-edit-distance-2026-paper","name":"SyntaxBench · edit-distance (2026 paper)","group":"知识与推理","metric":"Exact-match accuracy","unit":"%","public_history":true,"protocol":{"en":"Six models, non-thinking, four-shot. edit-distance with matched English and random-string conditions, kept as configurations. All selected cells use the full task set; index-to-span is excluded because one model has a missing/partial condition.","zh":"六模型、非思考、四样本提示；edit-distance 的英语和随机字符串条件分别记为配置。所选单元格均使用完整任务集；index-to-span 存在缺失或部分条件，不并入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.03329","status":"paper_baselines","published_at":"2026-10-02"},"model_count":6,"result_count":12,"results":[[9132,0.0,370,9133,9134],[5480,0.39,370,9133,9134],[2584,0.365,370,9133,9134],[5828,0.23,370,9133,9134],[4070,0.34,370,9133,9134],[3763,0.32,370,9133,9134],[9132,0.0,370,9135,9134],[5480,0.36,370,9135,9134],[2584,0.495,370,9135,9134],[5828,0.29,370,9135,9134],[4070,0.455,370,9135,9134],[3763,0.335,370,9135,9134]]},{"id":"syntaxbench-letter-container-2026-paper","name":"SyntaxBench · letter-container (2026 paper)","group":"知识与推理","metric":"Exact-match accuracy","unit":"%","public_history":true,"protocol":{"en":"Six models, non-thinking, four-shot. letter-container with matched English and random-string conditions, kept as configurations. All selected cells use the full task set; index-to-span is excluded because one model has a missing/partial condition.","zh":"六模型、非思考、四样本提示；letter-container 的英语和随机字符串条件分别记为配置。所选单元格均使用完整任务集；index-to-span 存在缺失或部分条件，不并入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.03329","status":"paper_baselines","published_at":"2026-10-02"},"model_count":6,"result_count":12,"results":[[9132,0.978,370,9133,9134],[5480,0.988,370,9133,9134],[2584,0.985,370,9133,9134],[5828,0.775,370,9133,9134],[4070,0.935,370,9133,9134],[3763,0.968,370,9133,9134],[9132,0.982,370,9135,9134],[5480,0.998,370,9135,9134],[2584,0.992,370,9135,9134],[5828,0.88,370,9135,9134],[4070,0.988,370,9135,9134],[3763,0.995,370,9135,9134]]},{"id":"syntaxbench-longest-string-2026-paper","name":"SyntaxBench · longest-string (2026 paper)","group":"知识与推理","metric":"Exact-match accuracy","unit":"%","public_history":true,"protocol":{"en":"Six models, non-thinking, four-shot. longest-string with matched English and random-string conditions, kept as configurations. All selected cells use the full task set; index-to-span is excluded because one model has a missing/partial condition.","zh":"六模型、非思考、四样本提示；longest-string 的英语和随机字符串条件分别记为配置。所选单元格均使用完整任务集；index-to-span 存在缺失或部分条件，不并入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.03329","status":"paper_baselines","published_at":"2026-10-02"},"model_count":6,"result_count":12,"results":[[9132,0.53,370,9133,9134],[5480,0.97,370,9133,9134],[2584,0.94,370,9133,9134],[5828,0.485,370,9133,9134],[4070,0.745,370,9133,9134],[3763,0.965,370,9133,9134],[9132,0.335,370,9135,9134],[5480,1.0,370,9135,9134],[2584,0.985,370,9135,9134],[5828,0.475,370,9135,9134],[4070,0.89,370,9135,9134],[3763,0.985,370,9135,9134]]},{"id":"syntaxbench-palindrome-2026-paper","name":"SyntaxBench · palindrome (2026 paper)","group":"知识与推理","metric":"Exact-match accuracy","unit":"%","public_history":true,"protocol":{"en":"Six models, non-thinking, four-shot. palindrome with matched English and random-string conditions, kept as configurations. All selected cells use the full task set; index-to-span is excluded because one model has a missing/partial condition.","zh":"六模型、非思考、四样本提示；palindrome 的英语和随机字符串条件分别记为配置。所选单元格均使用完整任务集；index-to-span 存在缺失或部分条件，不并入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.03329","status":"paper_baselines","published_at":"2026-10-02"},"model_count":6,"result_count":12,"results":[[9132,0.878,370,9133,9134],[5480,0.922,370,9133,9134],[2584,0.912,370,9133,9134],[5828,0.72,370,9133,9134],[4070,0.905,370,9133,9134],[3763,0.9420000000000001,370,9133,9134],[9132,0.878,370,9135,9134],[5480,0.9420000000000001,370,9135,9134],[2584,0.9520000000000001,370,9135,9134],[5828,0.948,370,9135,9134],[4070,0.968,370,9135,9134],[3763,0.9620000000000001,370,9135,9134]]},{"id":"portraitaes-standard-2026-paper","name":"PortraitAes-Bench · standard (2026 paper)","group":"视觉与视频","metric":"Overall Spearman correlation","unit":"score","public_history":true,"protocol":{"en":"All nine reported scorers on the standard 1,600-image set. Overall SRCC is correlation with the reference overall rating. Holistic-only scorers are identified; this is not their diagnostic per-dimension correlation.","zh":"九个评分模型使用标准 1,600 图测试集。取与参考总体评分的 Overall SRCC；仅整体评分的模型另注明，不替换为其诊断性分维度相关系数。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.05010","status":"paper_baselines","published_at":"2026-10-04"},"score_max":1,"score_decimals":3,"saturation_excluded":true,"higher_is_better":true,"scale_label":{"en":"Spearman rho","zh":"Spearman 相关系数"},"score_min":-1,"model_count":9,"result_count":9,"results":[[9136,0.874,370,9137,9138],[2728,0.668,370,9137,9138],[9139,0.785,370,9140,9138],[6579,0.763,370,9137,9138],[9141,0.804,370,9140,9138],[9142,0.729,370,9140,9138],[9143,0.499,370,9137,9138],[9144,0.724,370,9140,9138],[9145,0.934,370,9137,9138]]},{"id":"webuiproof-general-2026-paper","name":"WebUIProof · General WebUI (2026 paper)","group":"编程","metric":"Weighted functional completion","unit":"score","public_history":true,"protocol":{"en":"Full published fourteen-row panel. UI-agent-verified completion uses Full + 0.5×Partial; build failures remain unsuccessful. API baselines and SFT/PPO/VisRL training configurations are retained.","zh":"原表完整十四行。经界面智能体验证的完成度为 Full + 0.5×Partial，构建失败不获成功分；保留 API 基线及 SFT/PPO/VisRL 训练配置。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.02617","status":"paper_baselines","published_at":"2026-10-02"},"score_max":100,"score_decimals":2,"saturation_excluded":true,"higher_is_better":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":10,"result_count":14,"results":[[323,46.1,370,9146,9147],[2602,42.86,370,9146,9147],[9148,41.99,370,9146,9147],[1487,41.9,370,9146,9147],[2812,41.4,370,9146,9147],[362,40.77,370,9146,9147],[1485,39.3,370,9146,9147],[9149,36.19,370,9146,9147],[9150,26.08,370,9151,9147],[9150,38.79,370,9152,9147],[9150,41.8,370,9153,9147],[9154,23.8,370,9151,9147],[9154,27.06,370,9152,9147],[9154,33.43,370,9153,9147]]},{"id":"webuiproof-3d-2026-paper","name":"WebUIProof · 3D WebUI (2026 paper)","group":"编程","metric":"Weighted functional completion","unit":"score","public_history":true,"protocol":{"en":"Full published fourteen-row panel. UI-agent-verified completion uses Full + 0.5×Partial; build failures remain unsuccessful. API baselines and SFT/PPO/VisRL training configurations are retained.","zh":"原表完整十四行。经界面智能体验证的完成度为 Full + 0.5×Partial，构建失败不获成功分；保留 API 基线及 SFT/PPO/VisRL 训练配置。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.02617","status":"paper_baselines","published_at":"2026-10-02"},"score_max":100,"score_decimals":2,"saturation_excluded":true,"higher_is_better":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":10,"result_count":14,"results":[[323,40.24,370,9146,9155],[9148,37.67,370,9146,9155],[2602,35.96,370,9146,9155],[1485,35.1,370,9146,9155],[362,32.19,370,9146,9155],[1487,29.79,370,9146,9155],[2812,22.95,370,9146,9155],[9149,19.52,370,9146,9155],[9150,32.44,370,9151,9155],[9150,36.47,370,9152,9155],[9150,37.38,370,9153,9155],[9154,17.2,370,9151,9155],[9154,18.75,370,9152,9155],[9154,20.1,370,9153,9155]]},{"id":"emg-fm-fft-2026-paper","name":"EMG-FM-Bench · full fine-tuning (2026 paper)","group":"数学与科学","metric":"Mean classification accuracy","unit":"%","public_history":true,"protocol":{"en":"Nine published pretrained configurations across the same nineteen EMG classification datasets; Table 3 full fine-tuning mean accuracy, not the parenthesized mean rank.","zh":"九个预训练配置在同一组十九个肌电分类数据集上评估。取表 3 full fine-tuning 平均准确率，不取括号中的平均名次。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.06450","status":"paper_baselines","published_at":"2026-10-05"},"model_count":9,"result_count":9,"results":[[9156,0.601,370,9157,9158],[9159,0.46799999999999997,370,9157,9158],[9160,0.228,370,9157,9158],[9161,0.301,370,9157,9158],[9162,0.379,370,9157,9158],[9163,0.5429999999999999,370,9157,9158],[9164,0.565,370,9157,9158],[9165,0.38,370,9157,9158],[9166,0.402,370,9157,9158]]},{"id":"emg-fm-heldout-2026-paper","name":"EMG-FM-Bench · new-user generalization (2026 paper)","group":"数学与科学","metric":"Mean classification accuracy","unit":"%","public_history":true,"protocol":{"en":"Nine published pretrained configurations across the same nineteen EMG classification datasets; Table 3 new-user generalization mean accuracy, not the parenthesized mean rank.","zh":"九个预训练配置在同一组十九个肌电分类数据集上评估。取表 3 new-user generalization 平均准确率，不取括号中的平均名次。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.06450","status":"paper_baselines","published_at":"2026-10-05"},"model_count":9,"result_count":9,"results":[[9156,0.39,370,9167,9158],[9159,0.281,370,9167,9158],[9160,0.198,370,9167,9158],[9161,0.272,370,9167,9158],[9162,0.311,370,9167,9158],[9163,0.41600000000000004,370,9167,9158],[9164,0.43799999999999994,370,9167,9158],[9165,0.284,370,9167,9158],[9166,0.27699999999999997,370,9167,9158]]},{"id":"cavabench-wer-2026-paper","name":"CavaBench (2026 paper)","group":"其他","metric":"Word error rate (%)","unit":"raw","public_history":true,"protocol":{"en":"Six ASR model backbones, eleven configurations. Table 1 average WER is sample-weighted over six acoustic conditions. Streaming lookahead, chunking and in-house fine-tuning remain configurations. Lower is better.","zh":"六个语音识别底模、十一种配置。表 1 平均词错误率按样本数汇总六种声学条件；流式前瞻、分块和内部微调分别记为配置。越低越好。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.06587","status":"paper_baselines","published_at":"2026-10-05"},"higher_is_better":false,"saturation_excluded":true,"score_decimals":1,"scale_label":{"en":"WER (%)","zh":"词错误率（%）"},"model_count":6,"result_count":11,"results":[[9168,18.2,370,9169,9170],[9168,16.1,370,9171,9170],[9168,20.0,370,9172,9170],[9168,17.6,370,9173,9170],[8624,11.1,370,9174,9170],[8624,15.2,370,9175,9170],[9176,16.5,370,9177,9170],[9176,16.6,370,9178,9170],[9179,16.7,370,9179,9170],[9180,17.4,370,9180,9170],[9181,15.3,370,9181,9170]]},{"id":"agentmon-specgap-2026-paper","name":"AgentMonBench · SpecGAP (2026 paper)","group":"安全与可靠性","metric":"Semantic F1","unit":"score","public_history":true,"protocol":{"en":"Eight monitors with Base, RepoGraph and EBG evidence presentations. Table 1 Semantic F1. Semantic scores average independent GLM-5.2 and Qwen3.7-Max judges; localization score is rule-based.","zh":"八个监测模型分别使用 Base、RepoGraph 和 EBG 证据呈现。取表 1 Semantic F1；语义评分由 GLM-5.2 与 Qwen3.7-Max 独立评分后取平均，定位分按规则计算。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.06406","status":"paper_baselines","published_at":"2026-10-05"},"score_max":1,"score_decimals":2,"saturation_excluded":true,"higher_is_better":true,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":8,"result_count":24,"results":[[3408,0.25,370,4974,9182],[3408,0.241,370,9183,9182],[3408,0.326,370,9184,9182],[2973,0.275,370,4974,9182],[2973,0.215,370,9183,9182],[2973,0.392,370,9184,9182],[762,0.456,370,4974,9182],[762,0.41,370,9183,9182],[762,0.505,370,9184,9182],[3568,0.328,370,4974,9182],[3568,0.269,370,9183,9182],[3568,0.331,370,9184,9182],[3153,0.376,370,4974,9182],[3153,0.336,370,9183,9182],[3153,0.372,370,9184,9182],[3185,0.123,370,4974,9182],[3185,0.134,370,9183,9182],[3185,0.218,370,9184,9182],[2977,0.494,370,4974,9182],[2977,0.443,370,9183,9182],[2977,0.558,370,9184,9182],[3154,0.481,370,4974,9182],[3154,0.407,370,9183,9182],[3154,0.526,370,9184,9182]]},{"id":"agentmon-silentswap-2026-paper","name":"AgentMonBench · SilentSwap (2026 paper)","group":"安全与可靠性","metric":"Localization score","unit":"score","public_history":true,"protocol":{"en":"Eight monitors with Base, RepoGraph and EBG evidence presentations. Table 1 Localization score. Semantic scores average independent GLM-5.2 and Qwen3.7-Max judges; localization score is rule-based.","zh":"八个监测模型分别使用 Base、RepoGraph 和 EBG 证据呈现。取表 1 Localization score；语义评分由 GLM-5.2 与 Qwen3.7-Max 独立评分后取平均，定位分按规则计算。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.06406","status":"paper_baselines","published_at":"2026-10-05"},"score_max":1,"score_decimals":2,"saturation_excluded":true,"higher_is_better":true,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":8,"result_count":24,"results":[[3408,0.403,370,4974,9182],[3408,0.293,370,9183,9182],[3408,0.455,370,9184,9182],[2973,0.417,370,4974,9182],[2973,0.415,370,9183,9182],[2973,0.623,370,9184,9182],[762,0.464,370,4974,9182],[762,0.459,370,9183,9182],[762,0.671,370,9184,9182],[3568,0.294,370,4974,9182],[3568,0.258,370,9183,9182],[3568,0.425,370,9184,9182],[3153,0.262,370,4974,9182],[3153,0.344,370,9183,9182],[3153,0.492,370,9184,9182],[3185,0.288,370,4974,9182],[3185,0.281,370,9183,9182],[3185,0.408,370,9184,9182],[2977,0.555,370,4974,9182],[2977,0.406,370,9183,9182],[2977,0.681,370,9184,9182],[3154,0.56,370,4974,9182],[3154,0.561,370,9183,9182],[3154,0.708,370,9184,9182]]},{"id":"agentmon-feedbacktrace-2026-paper","name":"AgentMonBench · FeedbackTrace (2026 paper)","group":"安全与可靠性","metric":"Evidence support score","unit":"score","public_history":true,"protocol":{"en":"Eight monitors with Base, RepoGraph and EBG evidence presentations. Table 1 Evidence support score. Semantic scores average independent GLM-5.2 and Qwen3.7-Max judges; localization score is rule-based. RepoGraph has no published FeedbackTrace scores and is omitted.","zh":"八个监测模型分别使用 Base、RepoGraph 和 EBG 证据呈现。取表 1 Evidence support score；语义评分由 GLM-5.2 与 Qwen3.7-Max 独立评分后取平均，定位分按规则计算。 RepoGraph 未发表 FeedbackTrace 成绩，不作填补。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.06406","status":"paper_baselines","published_at":"2026-10-05"},"score_max":1,"score_decimals":2,"saturation_excluded":true,"higher_is_better":true,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":8,"result_count":16,"results":[[3408,0.545,370,4974,9182],[3408,0.56,370,9184,9182],[2973,0.588,370,4974,9182],[2973,0.565,370,9184,9182],[762,0.533,370,4974,9182],[762,0.585,370,9184,9182],[3568,0.45,370,4974,9182],[3568,0.573,370,9184,9182],[3153,0.453,370,4974,9182],[3153,0.55,370,9184,9182],[3185,0.345,370,4974,9182],[3185,0.365,370,9184,9182],[2977,0.518,370,4974,9182],[2977,0.598,370,9184,9182],[3154,0.463,370,4974,9182],[3154,0.56,370,9184,9182]]},{"id":"memcalib-exact-2026-paper","name":"MemCalib · exact calibration (2026 paper)","group":"指令与长文","metric":"Exact memory calibration","unit":"%","public_history":true,"protocol":{"en":"Nine models answer the same 1,500 disjoint test examples over three seeds in non-thinking mode. Exact calibration requires every memory atom to match its intended use level, with neither over-use nor under-use. DeepSeek-V4-Pro judges atom use against the published rubrics.","zh":"九个模型在不启用思考模式下，对同一组1,500个独立测试样本运行三个种子。每个记忆原子都按预期程度使用、既不过用也不漏用，才算精确校准成功。DeepSeek-V4-Pro依据公开细则判断原子使用程度。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.24259v1","published_at":"2026-09-21","status":"paper_baselines"},"model_count":9,"result_count":9,"results":[[5504,0.284,370,9185,9186],[3572,0.1909,370,9185,9186],[3187,0.2,370,9185,9186],[3415,0.1796,370,9185,9186],[227,0.182,370,9185,9186],[3157,0.17800000000000002,370,9185,9186],[3201,0.1696,370,9185,9186],[5344,0.15289999999999998,370,9185,9186],[2590,0.12240000000000001,370,9185,9186]]},{"id":"msibench-english-2026-paper","name":"MSI-Bench · English (2026 paper)","group":"Agent 与工具","metric":"All-pass rate","unit":"%","public_history":true,"protocol":{"en":"Multi-speaker voice assistants respond to 576 natively authored English scenes. Every atomic rubric and any required deterministic tool-call check must pass. The full fifteen-row panel keeps reasoning configurations of shared checkpoints explicit; each rate uses one judged prediction per configuration and case.","zh":"多说话人语音助手处理576个原生English情境，全部原子准则与所需的确定性工具检查通过才算成功。保留完整十五行面板，同一检查点的推理配置明确区分，每个配置与案例对应一次裁判评分。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.24812v1","published_at":"2026-09-21","status":"paper_baselines"},"model_count":12,"result_count":15,"results":[[3412,0.6679999999999999,370,9187,9188],[3415,0.632,370,9187,9188],[9189,0.583,370,9187,9188],[9190,0.5539999999999999,370,9187,9188],[9191,0.495,370,9192,9188],[9193,0.439,370,9192,9188],[9194,0.207,370,9192,9188],[9195,0.34,370,9192,9188],[7150,0.226,370,9192,9188],[7150,0.198,370,9187,9188],[6138,0.161,370,9192,9188],[9196,0.146,370,9187,9188],[9196,0.078,370,9192,9188],[6392,0.054000000000000006,370,9192,9188],[9197,0.016,370,9192,9188]]},{"id":"msibench-mandarin-2026-paper","name":"MSI-Bench · Mandarin (2026 paper)","group":"Agent 与工具","metric":"All-pass rate","unit":"%","public_history":true,"protocol":{"en":"Multi-speaker voice assistants respond to 576 natively authored Mandarin scenes. Every atomic rubric and any required deterministic tool-call check must pass. The full fifteen-row panel keeps reasoning configurations of shared checkpoints explicit; each rate uses one judged prediction per configuration and case.","zh":"多说话人语音助手处理576个原生Mandarin情境，全部原子准则与所需的确定性工具检查通过才算成功。保留完整十五行面板，同一检查点的推理配置明确区分，每个配置与案例对应一次裁判评分。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.24812v1","published_at":"2026-09-21","status":"paper_baselines"},"model_count":12,"result_count":15,"results":[[3412,0.528,370,9198,9199],[3415,0.502,370,9198,9199],[9189,0.545,370,9198,9199],[9190,0.493,370,9198,9199],[9191,0.425,370,9200,9199],[9193,0.41100000000000003,370,9200,9199],[9194,0.165,370,9200,9199],[9195,0.193,370,9200,9199],[7150,0.045,370,9200,9199],[7150,0.023,370,9198,9199],[6138,0.08900000000000001,370,9200,9199],[9196,0.073,370,9198,9199],[9196,0.08900000000000001,370,9200,9199],[6392,0.064,370,9200,9199],[9197,0.005,370,9200,9199]]},{"id":"brainwide-licks-2026-paper","name":"BrainWideBench · licks decoding (2026 paper)","group":"数学与科学","metric":"D²","unit":"raw","public_history":true,"protocol":{"en":"Eleven neural-decoding baselines predict licks behavior from IBL brain recordings under the common held-out session evaluation. Published D² values are averaged over sessions and five finetuning seeds. Single-session training, task-supervised pretraining and task-unsupervised pretraining are explicit configurations; no cross-metric mean rank is used.","zh":"十一个神经解码基线在相同留出会话上由IBL脑记录预测licks行为。保留跨会话、五个微调种子的D²均值，明确区分单会话训练、任务监督预训练和任务无监督预训练，不把不同量纲的平均名次当成绩。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.22064v1","published_at":"2026-09-18","status":"paper_baselines"},"model_count":11,"result_count":11,"results":[[6748,0.151,370,9201,9202],[9203,0.348,370,9201,9202],[9204,0.51,370,9201,9202],[9205,0.549,370,9201,9202],[9206,0.327,370,9201,9202],[9207,0.314,370,9201,9202],[9208,0.559,370,9201,9202],[9209,0.678,370,9210,9202],[9211,0.661,370,9210,9202],[9212,0.525,370,9213,9202],[9214,0.469,370,9213,9202]]},{"id":"brainwide-whisker-2026-paper","name":"BrainWideBench · whisker decoding (2026 paper)","group":"数学与科学","metric":"R²","unit":"raw","public_history":true,"protocol":{"en":"Eleven neural-decoding baselines predict whisker behavior from IBL brain recordings under the common held-out session evaluation. Published R² values are averaged over sessions and five finetuning seeds. Single-session training, task-supervised pretraining and task-unsupervised pretraining are explicit configurations; no cross-metric mean rank is used.","zh":"十一个神经解码基线在相同留出会话上由IBL脑记录预测whisker行为。保留跨会话、五个微调种子的R²均值，明确区分单会话训练、任务监督预训练和任务无监督预训练，不把不同量纲的平均名次当成绩。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.22064v1","published_at":"2026-09-18","status":"paper_baselines"},"model_count":11,"result_count":11,"results":[[6748,0.175,370,9201,9202],[9203,0.268,370,9201,9202],[9204,0.358,370,9201,9202],[9205,0.389,370,9201,9202],[9206,0.28,370,9201,9202],[9207,0.303,370,9201,9202],[9208,0.291,370,9201,9202],[9209,0.38,370,9210,9202],[9211,0.351,370,9210,9202],[9212,0.384,370,9213,9202],[9214,0.362,370,9213,9202]]},{"id":"brainwide-wheel-2026-paper","name":"BrainWideBench · wheel decoding (2026 paper)","group":"数学与科学","metric":"R²","unit":"raw","public_history":true,"protocol":{"en":"Eleven neural-decoding baselines predict wheel behavior from IBL brain recordings under the common held-out session evaluation. Published R² values are averaged over sessions and five finetuning seeds. Single-session training, task-supervised pretraining and task-unsupervised pretraining are explicit configurations; no cross-metric mean rank is used.","zh":"十一个神经解码基线在相同留出会话上由IBL脑记录预测wheel行为。保留跨会话、五个微调种子的R²均值，明确区分单会话训练、任务监督预训练和任务无监督预训练，不把不同量纲的平均名次当成绩。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.22064v1","published_at":"2026-09-18","status":"paper_baselines"},"model_count":11,"result_count":11,"results":[[6748,0.219,370,9201,9202],[9203,0.329,370,9201,9202],[9204,0.3,370,9201,9202],[9205,0.33,370,9201,9202],[9206,0.2,370,9201,9202],[9207,0.212,370,9201,9202],[9208,0.3,370,9201,9202],[9209,0.362,370,9210,9202],[9211,0.355,370,9210,9202],[9212,0.31,370,9213,9202],[9214,0.323,370,9213,9202]]},{"id":"brainwide-right-paw-2026-paper","name":"BrainWideBench · right-paw decoding (2026 paper)","group":"数学与科学","metric":"R²","unit":"raw","public_history":true,"protocol":{"en":"Eleven neural-decoding baselines predict right-paw behavior from IBL brain recordings under the common held-out session evaluation. Published R² values are averaged over sessions and five finetuning seeds. Single-session training, task-supervised pretraining and task-unsupervised pretraining are explicit configurations; no cross-metric mean rank is used.","zh":"十一个神经解码基线在相同留出会话上由IBL脑记录预测right-paw行为。保留跨会话、五个微调种子的R²均值，明确区分单会话训练、任务监督预训练和任务无监督预训练，不把不同量纲的平均名次当成绩。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.22064v1","published_at":"2026-09-18","status":"paper_baselines"},"model_count":11,"result_count":11,"results":[[6748,0.091,370,9201,9202],[9203,0.157,370,9201,9202],[9204,0.214,370,9201,9202],[9205,0.225,370,9201,9202],[9206,0.164,370,9201,9202],[9207,0.163,370,9201,9202],[9208,0.195,370,9201,9202],[9209,0.249,370,9210,9202],[9211,0.237,370,9210,9202],[9212,0.219,370,9213,9202],[9214,0.224,370,9213,9202]]},{"id":"brainwide-left-paw-2026-paper","name":"BrainWideBench · left-paw decoding (2026 paper)","group":"数学与科学","metric":"R²","unit":"raw","public_history":true,"protocol":{"en":"Eleven neural-decoding baselines predict left-paw behavior from IBL brain recordings under the common held-out session evaluation. Published R² values are averaged over sessions and five finetuning seeds. Single-session training, task-supervised pretraining and task-unsupervised pretraining are explicit configurations; no cross-metric mean rank is used.","zh":"十一个神经解码基线在相同留出会话上由IBL脑记录预测left-paw行为。保留跨会话、五个微调种子的R²均值，明确区分单会话训练、任务监督预训练和任务无监督预训练，不把不同量纲的平均名次当成绩。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.22064v1","published_at":"2026-09-18","status":"paper_baselines"},"model_count":11,"result_count":11,"results":[[6748,0.084,370,9201,9202],[9203,0.135,370,9201,9202],[9204,0.176,370,9201,9202],[9205,0.194,370,9201,9202],[9206,0.121,370,9201,9202],[9207,0.127,370,9201,9202],[9208,0.136,370,9201,9202],[9209,0.2,370,9210,9202],[9211,0.175,370,9210,9202],[9212,0.172,370,9213,9202],[9214,0.167,370,9213,9202]]},{"id":"backtrend-set-f1-2026-paper","name":"BackTrend · scientific precursors (2026 paper)","group":"数学与科学","metric":"Set-level judged F1","unit":"score","public_history":true,"protocol":{"en":"Seven systems recover early scientific precursors subject to temporal evidence cutoffs. The main panel uses 30 topic-direction pairs with validated signals, from 18 of 25 AI/ML topics. Set-level F1 uses Claude Opus 4.8 over three runs; signal-level F1 and human-judged diagnostics remain separate.","zh":"七个系统在证据时间截止约束下寻找早期科学线索。主面板使用25个AI/ML主题中18个主题提供的30个已验证方向对。采用Claude Opus 4.8三次评分的集合层F1，与信号层F1和人工评分诊断区分。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.24921v1","published_at":"2026-09-21","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":7,"result_count":7,"results":[[3184,5.7,370,9215,9216],[3191,5.3,370,9215,9216],[773,10.1,370,9215,9216],[8164,6.6,370,9217,9216],[5344,4.4,370,9217,9216],[9218,5.6,370,9219,9216],[9220,6.0,370,9219,9216]]},{"id":"happyworld-video-elo-2026-paper","name":"HappyWorld · video preference (2026 paper)","group":"生成与创作","metric":"Arena Elo","unit":"elo","public_history":true,"protocol":{"en":"The video track compares all 14 published systems using blind human pairwise preferences. Elo stays on its native relative scale; the spatial track uses the authors’ capability adjustment. Tracks and the separate automated W-level composites are not pooled.","zh":"video赛道通过盲评人工两两偏好比较全部14个公开系统。Elo保留原生相对刻度，空间赛道采用作者的能力调整，不合并不同赛道或各W级自动综合评分。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.24308v1","published_at":"2026-09-21","status":"paper_baselines"},"model_count":14,"result_count":14,"results":[[8428,1263.0,370,9221,9222],[8426,1206.0,370,9221,9222],[9223,1147.0,370,9221,9222],[9224,1132.0,370,9221,9222],[9225,1115.0,370,9221,9222],[9226,1092.0,370,9221,9222],[9227,1063.0,370,9221,9222],[9228,1059.0,370,9221,9222],[7241,1041.0,370,9221,9222],[9229,943.0,370,9221,9222],[7240,917.0,370,9221,9222],[9230,856.0,370,9221,9222],[7244,843.0,370,9221,9222],[9231,324.0,370,9221,9222]]},{"id":"happyworld-spatial-elo-2026-paper","name":"HappyWorld · spatial preference (2026 paper)","group":"生成与创作","metric":"Capability-adjusted Arena Elo","unit":"elo","public_history":true,"protocol":{"en":"The spatial track compares all 9 published systems using blind human pairwise preferences. Elo stays on its native relative scale; the spatial track uses the authors’ capability adjustment. Tracks and the separate automated W-level composites are not pooled.","zh":"spatial赛道通过盲评人工两两偏好比较全部9个公开系统。Elo保留原生相对刻度，空间赛道采用作者的能力调整，不合并不同赛道或各W级自动综合评分。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.24308v1","published_at":"2026-09-21","status":"paper_baselines"},"model_count":9,"result_count":9,"results":[[4977,1252.0,370,9232,9233],[9234,1308.0,370,9232,9233],[9235,1230.0,370,9232,9233],[9236,1029.0,370,9232,9233],[9237,1019.0,370,9232,9233],[9238,962.0,370,9232,9233],[9239,743.0,370,9232,9233],[9228,729.0,370,9232,9233],[9240,728.0,370,9232,9233]]},{"id":"happyworld-embodied-elo-2026-paper","name":"HappyWorld · embodied preference (2026 paper)","group":"生成与创作","metric":"Arena Elo","unit":"elo","public_history":true,"protocol":{"en":"The embodied track compares all 8 published systems using blind human pairwise preferences. Elo stays on its native relative scale; the spatial track uses the authors’ capability adjustment. Tracks and the separate automated W-level composites are not pooled.","zh":"embodied赛道通过盲评人工两两偏好比较全部8个公开系统。Elo保留原生相对刻度，空间赛道采用作者的能力调整，不合并不同赛道或各W级自动综合评分。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.24308v1","published_at":"2026-09-21","status":"paper_baselines"},"model_count":8,"result_count":8,"results":[[7125,1060.0,370,9241,9242],[9243,1051.0,370,9241,9242],[6385,1051.0,370,9241,9242],[8934,1031.0,370,9241,9242],[9244,1026.0,370,9241,9242],[9245,978.0,370,9241,9242],[9246,931.0,370,9241,9242],[9247,872.0,370,9241,9242]]},{"id":"rootme-gym-pass1-2026","name":"rootme-gym (25-challenge cohort, 2026)","group":"安全与可靠性","metric":"Mean task pass@1","unit":"%","protocol":{"en":"Twenty model configurations solve the same 25 Root-Me challenges with sealed network access and shell/Python tools. The published pass@1 estimator is retained; per-task sample counts vary and remain part of the source record. Public-writeup and purpose-built solver controls are excluded. Internal gateway variants are configurations of their named base model.","zh":"二十种模型配置在禁止外网访问、仅使用shell/Python的条件下解答同一组25个Root-Me挑战。保留原表pass@1估计值，各题采样次数不同并在来源中保留。公开题解与专门求解脚本不作为模型，内部网关变体记为同一具名底模的配置。"},"source":{"name":"rootme-gym maintainers","url":"https://github.com/vmalguy/rootme-gym","published_at":"2026-09-22","status":"current_snapshot"},"model_count":19,"result_count":20,"results":[[3396,0.5710000000000001,370,9248,9249],[3763,0.52,370,9250,9249],[7641,0.537,370,9251,9249],[7641,0.51,370,9252,9249],[5300,0.44,370,9253,9249],[3201,0.45899999999999996,370,9254,9249],[3154,0.45299999999999996,370,9255,9249],[6438,0.45299999999999996,370,9256,9249],[3153,0.467,370,9257,9249],[2576,0.42,370,9258,9249],[4906,0.23199999999999998,370,9259,9249],[9260,0.253,370,9261,9249],[4070,0.17300000000000001,370,9262,9249],[786,0.22699999999999998,370,9263,9249],[9264,0.213,370,9265,9249],[2444,0.187,370,9266,9249],[62,0.136,370,9267,9249],[8698,0.18899999999999997,370,9268,9249],[561,0.073,370,9269,9249],[1810,0.0,370,9270,9249]]},{"id":"360city-localization-2026-paper","name":"360CityArena · localization (2026 paper)","group":"Agent 与工具","metric":"Localization exact match","unit":"%","public_history":true,"protocol":{"en":"Six models use the same photorealistic urban tasks and tool interface. Localization uses exact coordinate matching; navigation and landmark searches use the fixed distance thresholds; relational answers use isolated semantic grading with manual verification. The selected task stays separate from count-estimation MRA.","zh":"六个模型使用同一组真实感城市任务与工具接口。定位采用坐标精确匹配，导航与地标搜索按固定距离阈值，关系推理使用隔离的语义裁判并人工核验。各子任务分别展示，不与计数估计MRA混合。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.08814v1","status":"paper_baselines","published_at":"2026-08-09"},"model_count":6,"result_count":6,"results":[[2058,0.08,370,9271,9272],[283,0.04,370,9271,9272],[2055,0.12,370,9271,9272],[2453,0.04,370,9271,9272],[2654,0.04,370,9271,9272],[2642,0.0,370,9271,9272]]},{"id":"360city-landmark-language-2026-paper","name":"360CityArena · landmark-language (2026 paper)","group":"Agent 与工具","metric":"Landmark search, language","unit":"%","public_history":true,"protocol":{"en":"Six models use the same photorealistic urban tasks and tool interface. Localization uses exact coordinate matching; navigation and landmark searches use the fixed distance thresholds; relational answers use isolated semantic grading with manual verification. The selected task stays separate from count-estimation MRA.","zh":"六个模型使用同一组真实感城市任务与工具接口。定位采用坐标精确匹配，导航与地标搜索按固定距离阈值，关系推理使用隔离的语义裁判并人工核验。各子任务分别展示，不与计数估计MRA混合。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.08814v1","status":"paper_baselines","published_at":"2026-08-09"},"model_count":6,"result_count":6,"results":[[2058,0.16,370,9271,9272],[283,0.04,370,9271,9272],[2055,0.28,370,9271,9272],[2453,0.16,370,9271,9272],[2654,0.2,370,9271,9272],[2642,0.16,370,9271,9272]]},{"id":"360city-landmark-image-2026-paper","name":"360CityArena · landmark-image (2026 paper)","group":"Agent 与工具","metric":"Landmark search, image","unit":"%","public_history":true,"protocol":{"en":"Six models use the same photorealistic urban tasks and tool interface. Localization uses exact coordinate matching; navigation and landmark searches use the fixed distance thresholds; relational answers use isolated semantic grading with manual verification. The selected task stays separate from count-estimation MRA.","zh":"六个模型使用同一组真实感城市任务与工具接口。定位采用坐标精确匹配，导航与地标搜索按固定距离阈值，关系推理使用隔离的语义裁判并人工核验。各子任务分别展示，不与计数估计MRA混合。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.08814v1","status":"paper_baselines","published_at":"2026-08-09"},"model_count":6,"result_count":6,"results":[[2058,0.48,370,9271,9272],[283,0.16,370,9271,9272],[2055,0.36,370,9271,9272],[2453,0.2,370,9271,9272],[2654,0.2,370,9271,9272],[2642,0.0,370,9271,9272]]},{"id":"360city-map-navigation-2026-paper","name":"360CityArena · map-navigation (2026 paper)","group":"Agent 与工具","metric":"Map navigation success","unit":"%","public_history":true,"protocol":{"en":"Six models use the same photorealistic urban tasks and tool interface. Localization uses exact coordinate matching; navigation and landmark searches use the fixed distance thresholds; relational answers use isolated semantic grading with manual verification. The selected task stays separate from count-estimation MRA.","zh":"六个模型使用同一组真实感城市任务与工具接口。定位采用坐标精确匹配，导航与地标搜索按固定距离阈值，关系推理使用隔离的语义裁判并人工核验。各子任务分别展示，不与计数估计MRA混合。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.08814v1","status":"paper_baselines","published_at":"2026-08-09"},"model_count":6,"result_count":6,"results":[[2058,0.0,370,9271,9272],[283,0.04,370,9271,9272],[2055,0.0,370,9271,9272],[2453,0.0,370,9271,9272],[2654,0.0,370,9271,9272],[2642,0.0,370,9271,9272]]},{"id":"360city-vln-2026-paper","name":"360CityArena · vln (2026 paper)","group":"Agent 与工具","metric":"Vision-language navigation success","unit":"%","public_history":true,"protocol":{"en":"Six models use the same photorealistic urban tasks and tool interface. Localization uses exact coordinate matching; navigation and landmark searches use the fixed distance thresholds; relational answers use isolated semantic grading with manual verification. The selected task stays separate from count-estimation MRA.","zh":"六个模型使用同一组真实感城市任务与工具接口。定位采用坐标精确匹配，导航与地标搜索按固定距离阈值，关系推理使用隔离的语义裁判并人工核验。各子任务分别展示，不与计数估计MRA混合。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.08814v1","status":"paper_baselines","published_at":"2026-08-09"},"model_count":6,"result_count":6,"results":[[2058,0.08,370,9271,9272],[283,0.04,370,9271,9272],[2055,0.08,370,9271,9272],[2453,0.0,370,9271,9272],[2654,0.12,370,9271,9272],[2642,0.12,370,9271,9272]]},{"id":"360city-relational-2026-paper","name":"360CityArena · relational (2026 paper)","group":"Agent 与工具","metric":"Relational spatial reasoning accuracy","unit":"%","public_history":true,"protocol":{"en":"Six models use the same photorealistic urban tasks and tool interface. Localization uses exact coordinate matching; navigation and landmark searches use the fixed distance thresholds; relational answers use isolated semantic grading with manual verification. The selected task stays separate from count-estimation MRA.","zh":"六个模型使用同一组真实感城市任务与工具接口。定位采用坐标精确匹配，导航与地标搜索按固定距离阈值，关系推理使用隔离的语义裁判并人工核验。各子任务分别展示，不与计数估计MRA混合。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.08814v1","status":"paper_baselines","published_at":"2026-08-09"},"model_count":6,"result_count":6,"results":[[2058,0.32,370,9271,9272],[283,0.08,370,9271,9272],[2055,0.12,370,9271,9272],[2453,0.04,370,9271,9272],[2654,0.0,370,9271,9272],[2642,0.04,370,9271,9272]]},{"id":"cocobench-centralized-2026-paper","name":"CoCoBench · centralized image observation (2026 paper)","group":"Agent 与工具","metric":"Task success rate","unit":"%","public_history":true,"protocol":{"en":"Eleven models complete the same full 897-instance household coordination suite under centralized image observations. Task success is scored separately from construct-level partial scores. All eleven model rows in the main table are retained.","zh":"十一个模型在中心化图像观测下完成同一套897个家庭协作任务。采用任务成功率，与构念层的部分得分区分，完整保留主表十一个模型。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.28266v1","status":"paper_baselines","published_at":"2026-08-28"},"model_count":11,"result_count":11,"results":[[4048,0.848,370,9273,9274],[4958,0.336,370,9273,9274],[3479,0.785,370,9273,9274],[325,0.631,370,9273,9274],[3189,0.691,370,9273,9274],[3187,0.5760000000000001,370,9273,9274],[4070,0.36700000000000005,370,9273,9274],[6342,0.24600000000000002,370,9273,9274],[7159,0.295,370,9273,9274],[7558,0.221,370,9273,9274],[2654,0.134,370,9273,9274]]},{"id":"ctdd-ap50-2026-paper","name":"Copper Tube Defect Dataset · detection (2026 paper)","group":"视觉与视频","metric":"AP@50","unit":"score","public_history":true,"protocol":{"en":"Thirteen detectors use the same CTDD 8:1:1 split and single defect class. The paper specifies 640-pixel input, 200 training epochs and common partitioning, while preserving each official baseline implementation. AP@50 remains a native 0–1 detection metric; the mixed-metric Avg. column is not used.","zh":"十三个检测器采用同一CTDD 8:1:1划分和单一缺陷类别。论文规定640像素输入、200轮训练与共同数据划分，并保留各官方基线实现。AP@50保留0–1检测指标，不使用混合量纲的Avg.列。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.28070v1","status":"paper_baselines","published_at":"2026-08-28"},"score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":13,"result_count":13,"results":[[9275,0.732,370,9276,9277],[9278,0.746,370,9276,9277],[9279,0.667,370,9276,9277],[9280,0.483,370,9276,9277],[9281,0.684,370,9276,9277],[9282,0.695,370,9276,9277],[9283,0.659,370,9276,9277],[9284,0.788,370,9276,9277],[9285,0.582,370,9276,9277],[9286,0.794,370,9276,9277],[9287,0.787,370,9276,9277],[9288,0.801,370,9276,9277],[9289,0.823,370,9276,9277]]},{"id":"beyondmasks-object-disappearance-2026-paper","name":"BeyondMasks · object disappearance (2026 paper)","group":"生成与创作","metric":"CORE object disappearance","unit":"score","public_history":true,"protocol":{"en":"Ten video-removal systems use the paired temporally aligned object-present and clean-background benchmark. CORE separately grades object disappearance and physical after-effect consistency. Published values stay on the native five-point rubric; each model’s text/mask conditioning is explicit.","zh":"十个视频移除系统使用物体存在与干净背景成对、时间对齐的评测数据。CORE分别评判物体消失和物理后效一致性，保留原生五分量表，明确各模型的文本或掩码输入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.20107v1","status":"paper_baselines","published_at":"2026-08-20"},"score_max":5,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 5","zh":"分数 / 5"},"model_count":10,"result_count":10,"results":[[9290,1.443,370,9291,9292],[9293,1.799,370,9294,9292],[9295,1.986,370,9294,9292],[9296,3.771,370,9294,9292],[9297,3.063,370,9298,9292],[9299,3.494,370,9298,9292],[9300,3.631,370,9298,9292],[9301,3.784,370,9298,9292],[9302,3.19,370,9298,9292],[9303,4.015,370,9298,9292]]},{"id":"beyondmasks-after-effect-2026-paper","name":"BeyondMasks · after-effect consistency (2026 paper)","group":"生成与创作","metric":"CORE after-effect consistency","unit":"score","public_history":true,"protocol":{"en":"Ten video-removal systems use the paired temporally aligned object-present and clean-background benchmark. CORE separately grades object disappearance and physical after-effect consistency. Published values stay on the native five-point rubric; each model’s text/mask conditioning is explicit.","zh":"十个视频移除系统使用物体存在与干净背景成对、时间对齐的评测数据。CORE分别评判物体消失和物理后效一致性，保留原生五分量表，明确各模型的文本或掩码输入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.20107v1","status":"paper_baselines","published_at":"2026-08-20"},"score_max":5,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 5","zh":"分数 / 5"},"model_count":10,"result_count":10,"results":[[9290,1.551,370,9291,9292],[9293,1.698,370,9294,9292],[9295,2.043,370,9294,9292],[9296,3.021,370,9294,9292],[9297,2.368,370,9298,9292],[9299,2.625,370,9298,9292],[9300,2.553,370,9298,9292],[9301,2.92,370,9298,9292],[9302,2.499,370,9298,9292],[9303,3.162,370,9298,9292]]},{"id":"anchorbench-external-control-mae-2026-paper","name":"AnchorBench · external control (2026 paper)","group":"安全与可靠性","metric":"Control MAE","unit":"score","public_history":true,"protocol":{"en":"The full fourteen-model panel is evaluated on the same 360-item external suite. Control MAE measures absolute error against the mean of the supplied ratings on the native 0–100 scale. This is task competence in the anchor-free control, separate from anchoring influence, with lower values better. Model abbreviations are resolved through Appendix Table 3.","zh":"完整十四模型面板在同一360题external子集上评测。控制条件MAE衡量答案与所给评分均值的绝对误差，保留0–100原始量纲，越低越好。这是无锚控制条件的任务能力，与锚定影响量分开；模型缩写按附录表3还原。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.14320v1","status":"paper_baselines","published_at":"2026-08-14"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"higher_is_better":false,"model_count":14,"result_count":14,"results":[[9304,17.93,370,9305,9306],[9307,9.62,370,9305,9306],[9308,7.74,370,9305,9306],[9309,16.79,370,9305,9306],[9310,9.21,370,9305,9306],[9311,5.61,370,9305,9306],[9312,15.43,370,9305,9306],[9313,16.33,370,9305,9306],[9314,3.64,370,9305,9306],[9315,13.01,370,9305,9306],[9316,0.38,370,9305,9306],[9317,0.67,370,9305,9306],[9318,1.07,370,9305,9306],[9319,0.3,370,9305,9306]]},{"id":"anchorbench-icl-control-mae-2026-paper","name":"AnchorBench · icl control (2026 paper)","group":"安全与可靠性","metric":"Control MAE","unit":"score","public_history":true,"protocol":{"en":"The full fourteen-model panel is evaluated on the same 360-item icl suite. Control MAE measures absolute error against the mean of the supplied ratings on the native 0–100 scale. This is task competence in the anchor-free control, separate from anchoring influence, with lower values better. Model abbreviations are resolved through Appendix Table 3.","zh":"完整十四模型面板在同一360题icl子集上评测。控制条件MAE衡量答案与所给评分均值的绝对误差，保留0–100原始量纲，越低越好。这是无锚控制条件的任务能力，与锚定影响量分开；模型缩写按附录表3还原。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.14320v1","status":"paper_baselines","published_at":"2026-08-14"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"higher_is_better":false,"model_count":14,"result_count":14,"results":[[9304,10.27,370,9320,9321],[9307,7.16,370,9320,9321],[9308,5.5,370,9320,9321],[9309,16.26,370,9320,9321],[9310,6.98,370,9320,9321],[9311,7.31,370,9320,9321],[9312,13.38,370,9320,9321],[9313,13.53,370,9320,9321],[9314,10.56,370,9320,9321],[9315,8.15,370,9320,9321],[9316,0.47,370,9320,9321],[9317,1.07,370,9320,9321],[9318,0.31,370,9320,9321],[9319,0.39,370,9320,9321]]},{"id":"anchorbench-rag-control-mae-2026-paper","name":"AnchorBench · rag control (2026 paper)","group":"安全与可靠性","metric":"Control MAE","unit":"score","public_history":true,"protocol":{"en":"The full fourteen-model panel is evaluated on the same 360-item rag suite. Control MAE measures absolute error against the mean of the supplied ratings on the native 0–100 scale. This is task competence in the anchor-free control, separate from anchoring influence, with lower values better. Model abbreviations are resolved through Appendix Table 3.","zh":"完整十四模型面板在同一360题rag子集上评测。控制条件MAE衡量答案与所给评分均值的绝对误差，保留0–100原始量纲，越低越好。这是无锚控制条件的任务能力，与锚定影响量分开；模型缩写按附录表3还原。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.14320v1","status":"paper_baselines","published_at":"2026-08-14"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"higher_is_better":false,"model_count":14,"result_count":14,"results":[[9304,22.15,370,9322,9323],[9307,8.12,370,9322,9323],[9308,9.83,370,9322,9323],[9309,13.73,370,9322,9323],[9310,10.6,370,9322,9323],[9311,4.66,370,9322,9323],[9312,14.98,370,9322,9323],[9313,12.64,370,9322,9323],[9314,4.21,370,9322,9323],[9315,11.0,370,9322,9323],[9316,0.57,370,9322,9323],[9317,1.1,370,9322,9323],[9318,5.55,370,9322,9323],[9319,0.26,370,9322,9323]]},{"id":"baitbench-reward-hacking-2026-paper","name":"BAITBENCH · reward hacking (2026 paper)","group":"安全与可靠性","metric":"Reward-hacking rate (lower is better)","unit":"score","public_history":true,"protocol":{"en":"Seven agents use the full common 180-task grid across three ML task families and two prompt conditions. The reported all-run reward-hacking rate averages both binary judges after collapsing API reruns, retaining unengaged runs in the denominator. Lower is better; this behavior rate is kept outside accuracy saturation ranges.","zh":"七个Agent使用三个机器学习任务族、两类提示条件下完整的共同180任务网格。全运行奖励作弊率在合并API重跑后取两位二元裁判均值，分母保留未开展实验的运行。越低越好；这是行为率，不参与准确率饱和区间。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.30724v1","status":"paper_baselines","published_at":"2026-08-31"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"higher_is_better":false,"model_count":7,"result_count":7,"results":[[2261,76.1,370,9324,9325],[3572,69.4,370,9324,9325],[3184,50.0,370,9324,9325],[244,20.8,370,9324,9325],[3412,62.2,370,9324,9325],[3424,65.3,370,9324,9325],[3492,55.3,370,9324,9325]]},{"id":"cadengbench-p-engineering-2026-paper","name":"CADEngBench · P engineering checks L1 (2026 paper)","group":"Agent 与工具","metric":"P engineering checks L1 pass rate","unit":"%","public_history":true,"protocol":{"en":"All eight published systems use the same 300 fixed tasks with executable CAD checks. The selected metric uses a common fixed denominator; conditional FEA pair-pass and variably evaluable L2-Z results are separate and are not pooled. One attempt per task uses the paper’s fixed model/version settings.","zh":"完整八个系统采用同一300个固定任务与可执行CAD检查。所选指标使用共同固定分母，不混入条件性FEA通过率或可评测题量不同的L2-Z结果。每题一次尝试，保留论文规定的模型版本与设置。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.09296v1","status":"paper_baselines","published_at":"2026-08-10"},"model_count":8,"result_count":8,"results":[[340,0.22,370,9326,9327],[283,0.28300000000000003,370,9326,9327],[3424,0.3,370,9326,9327],[5722,0.133,370,9326,9327],[244,0.243,370,9326,9327],[3613,0.083,370,9326,9327],[362,0.127,370,9326,9327],[2590,0.05,370,9326,9327]]},{"id":"cadengbench-a-kinematics-2026-paper","name":"CADEngBench · A kinematics end-to-end (2026 paper)","group":"Agent 与工具","metric":"A kinematics end-to-end pass rate","unit":"%","public_history":true,"protocol":{"en":"All eight published systems use the same 120 fixed tasks with executable CAD checks. The selected metric uses a common fixed denominator; conditional FEA pair-pass and variably evaluable L2-Z results are separate and are not pooled. One attempt per task uses the paper’s fixed model/version settings.","zh":"完整八个系统采用同一120个固定任务与可执行CAD检查。所选指标使用共同固定分母，不混入条件性FEA通过率或可评测题量不同的L2-Z结果。每题一次尝试，保留论文规定的模型版本与设置。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.09296v1","status":"paper_baselines","published_at":"2026-08-10"},"model_count":8,"result_count":8,"results":[[340,0.11699999999999999,370,9328,9327],[283,0.125,370,9328,9327],[3424,0.158,370,9328,9327],[5722,0.083,370,9328,9327],[244,0.075,370,9328,9327],[3613,0.142,370,9328,9327],[362,0.10800000000000001,370,9328,9327],[2590,0.10800000000000001,370,9328,9327]]},{"id":"cadp-bench-overall-vrfa-2026-paper","name":"CADP-Bench · document parsing (2026 paper)","group":"视觉与视频","metric":"Overall visual reconstruction fidelity (VRF-A)","unit":"score","public_history":true,"protocol":{"en":"All fourteen parsers use the same document images and agentic evaluation pipeline. Overall VRF-A evaluates reconstructed page visual fidelity across formula, table, chart, pseudocode and layout tasks. The native 0–100 reconstruction metric remains separate from accuracy; all model rows are preserved.","zh":"完整十四个解析模型使用同一文档图像与Agent评测流程。总体VRF-A评判公式、表格、图表、伪代码与版面重建的视觉保真度，保留原生0–100分量表，与准确率区分。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.17550v1","status":"paper_baselines","published_at":"2026-08-18"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":14,"result_count":14,"results":[[325,25.75,370,9329,9330],[3572,50.5,370,9329,9330],[2261,48.5,370,9329,9330],[2590,42.75,370,9329,9330],[2584,52.0,370,9329,9330],[2582,53.5,370,9329,9330],[9331,41.75,370,9329,9330],[6412,61.25,370,9329,9330],[2576,64.25,370,9329,9330],[3733,58.25,370,9329,9330],[3184,58.5,370,9329,9330],[2998,62.75,370,9329,9330],[6133,65.5,370,9329,9330],[6130,73.0,370,9329,9330]]},{"id":"deepchart-finance-normal-vas-2026-paper","name":"DeepChart · finance-normal (2026 paper)","group":"生成与创作","metric":"Visual Accuracy Score (VAS)","unit":"score","public_history":true,"protocol":{"en":"All eight model rows use the same finance-normal chart-generation split and zero-shot Python backend. VAS is the mean pass rate of cached binary visual verification rubrics with a fixed VLM judge. Missing or unrendered charts score zero. Normal and long context protocols are separate; VAS stays a native 0–1 visual rubric.","zh":"完整八个模型使用同一finance-normal图表生成子集和零样本Python后端。VAS由固定视觉裁判对缓存二元检查项评分取均值；缺失或无法渲染的图表记零。常规与长上下文协议分开，保留0–1视觉量表。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.26757v1","status":"paper_baselines","published_at":"2026-08-27"},"score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":8,"result_count":8,"results":[[340,0.5265,370,9332,9333],[3626,0.571,370,9332,9333],[26,0.3982,370,9332,9333],[3424,0.5398,370,9332,9333],[3009,0.2313,370,9332,9333],[2278,0.1863,370,9332,9333],[4819,0.4472,370,9332,9333],[4823,0.4686,370,9332,9333]]},{"id":"deepchart-academic-normal-vas-2026-paper","name":"DeepChart · academic-normal (2026 paper)","group":"生成与创作","metric":"Visual Accuracy Score (VAS)","unit":"score","public_history":true,"protocol":{"en":"All eight model rows use the same academic-normal chart-generation split and zero-shot Python backend. VAS is the mean pass rate of cached binary visual verification rubrics with a fixed VLM judge. Missing or unrendered charts score zero. Normal and long context protocols are separate; VAS stays a native 0–1 visual rubric.","zh":"完整八个模型使用同一academic-normal图表生成子集和零样本Python后端。VAS由固定视觉裁判对缓存二元检查项评分取均值；缺失或无法渲染的图表记零。常规与长上下文协议分开，保留0–1视觉量表。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.26757v1","status":"paper_baselines","published_at":"2026-08-27"},"score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":8,"result_count":8,"results":[[340,0.4948,370,9334,9333],[3626,0.6002,370,9334,9333],[26,0.2992,370,9334,9333],[3424,0.4167,370,9334,9333],[3009,0.3274,370,9334,9333],[2278,0.2225,370,9334,9333],[4819,0.3859,370,9334,9333],[4823,0.5291,370,9334,9333]]},{"id":"deepchart-finance-long-vas-2026-paper","name":"DeepChart · finance-long (2026 paper)","group":"生成与创作","metric":"Visual Accuracy Score (VAS)","unit":"score","public_history":true,"protocol":{"en":"All eight model rows use the same finance-long chart-generation split and zero-shot Python backend. VAS is the mean pass rate of cached binary visual verification rubrics with a fixed VLM judge. Missing or unrendered charts score zero. Normal and long context protocols are separate; VAS stays a native 0–1 visual rubric.","zh":"完整八个模型使用同一finance-long图表生成子集和零样本Python后端。VAS由固定视觉裁判对缓存二元检查项评分取均值；缺失或无法渲染的图表记零。常规与长上下文协议分开，保留0–1视觉量表。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.26757v1","status":"paper_baselines","published_at":"2026-08-27"},"score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":8,"result_count":8,"results":[[340,0.5387,370,9335,9333],[3626,0.533,370,9335,9333],[26,0.4716,370,9335,9333],[3424,0.5474,370,9335,9333],[3009,0.1557,370,9335,9333],[2278,0.1739,370,9335,9333],[4819,0.2993,370,9335,9333],[4823,0.3535,370,9335,9333]]},{"id":"deepchart-academic-long-vas-2026-paper","name":"DeepChart · academic-long (2026 paper)","group":"生成与创作","metric":"Visual Accuracy Score (VAS)","unit":"score","public_history":true,"protocol":{"en":"All eight model rows use the same academic-long chart-generation split and zero-shot Python backend. VAS is the mean pass rate of cached binary visual verification rubrics with a fixed VLM judge. Missing or unrendered charts score zero. Normal and long context protocols are separate; VAS stays a native 0–1 visual rubric.","zh":"完整八个模型使用同一academic-long图表生成子集和零样本Python后端。VAS由固定视觉裁判对缓存二元检查项评分取均值；缺失或无法渲染的图表记零。常规与长上下文协议分开，保留0–1视觉量表。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.26757v1","status":"paper_baselines","published_at":"2026-08-27"},"score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":8,"result_count":8,"results":[[340,0.5238,370,9336,9333],[3626,0.5945,370,9336,9333],[26,0.2888,370,9336,9333],[3424,0.3938,370,9336,9333],[3009,0.245,370,9336,9333],[2278,0.317,370,9336,9333],[4819,0.4696,370,9336,9333],[4823,0.461,370,9336,9333]]},{"id":"finfraudbench-creditcard-auprc-2026-paper","name":"FinFraudBench · creditcard (2026 paper)","group":"专业工作","metric":"AUPRC","unit":"score","public_history":true,"protocol":{"en":"Ten distinct graph/tabular architectures and two target-only LLM prompt configurations use the same fraud dataset partitions and five common balanced mini-test subsets. Complete appendix means, not rounded main-table values, are retained. AUPRC stays on its native 0–1 scale. The paper does not name the LLM backbone, so the two LLM probes share an explicit unspecified-backbone identity and do not count as two distinct models.","zh":"十种图或表格架构与两种仅使用目标节点的LLM提示配置使用相同数据划分及五个共同均衡微型测试集。保留附录完整均值，不采用主表舍入值。AUPRC保留0–1量纲。论文未列LLM底模，两种提示实验共用“未注明底模”身份，不计作两个模型。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.15177v1","status":"paper_baselines","published_at":"2026-08-15"},"score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":11,"result_count":12,"results":[[9203,0.9046,370,9337,9338],[9339,0.8632,370,9337,9338],[9340,0.9279,370,9337,9338],[9341,0.9163,370,9337,9338],[9342,0.9269,370,9337,9338],[9343,0.9279,370,9337,9338],[9344,0.9339,370,9337,9338],[9345,0.9315,370,9337,9338],[9346,0.9029,370,9337,9338],[9347,0.9344,370,9337,9338],[9348,0.6877,370,9349,9338],[9348,0.8495,370,9350,9338]]},{"id":"finfraudbench-banktrans-auprc-2026-paper","name":"FinFraudBench · banktrans (2026 paper)","group":"专业工作","metric":"AUPRC","unit":"score","public_history":true,"protocol":{"en":"Ten distinct graph/tabular architectures and two target-only LLM prompt configurations use the same fraud dataset partitions and five common balanced mini-test subsets. Complete appendix means, not rounded main-table values, are retained. AUPRC stays on its native 0–1 scale. The paper does not name the LLM backbone, so the two LLM probes share an explicit unspecified-backbone identity and do not count as two distinct models.","zh":"十种图或表格架构与两种仅使用目标节点的LLM提示配置使用相同数据划分及五个共同均衡微型测试集。保留附录完整均值，不采用主表舍入值。AUPRC保留0–1量纲。论文未列LLM底模，两种提示实验共用“未注明底模”身份，不计作两个模型。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.15177v1","status":"paper_baselines","published_at":"2026-08-15"},"score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":11,"result_count":12,"results":[[9203,0.9456,370,9337,9351],[9339,0.9486,370,9337,9351],[9340,0.9302,370,9337,9351],[9341,0.9717,370,9337,9351],[9342,0.9766,370,9337,9351],[9343,0.9784,370,9337,9351],[9344,0.9784,370,9337,9351],[9345,0.9751,370,9337,9351],[9346,0.9685,370,9337,9351],[9347,0.9544,370,9337,9351],[9348,0.8127,370,9349,9351],[9348,0.8598,370,9350,9351]]},{"id":"coral-block-segmentation-2026-paper","name":"CORAL (block segmentation) (2026 paper)","group":"视觉与视频","metric":"Dice","unit":"score","public_history":true,"protocol":{"en":"Complete ten-row block-level segmentation panel. Nine identifiable architecture families plus the clDice training variant, evaluated on the same CORAL blocks. Unified AdamW training for 10,000 steps, batch size 4, fixed crop and sliding-window inference settings. Native 0–100 Dice; not reconstruction Fiber-F1 or throughput.","zh":"完整十行块级分割面板，包含九种可识别架构及 clDice 训练变体，在同一 CORAL 图像块上评估。统一 AdamW、10,000 步、批量4、训练裁剪和滑窗推理设置。保留0–100 Dice，与重建 Fiber-F1 和吞吐量分开。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.30768","status":"paper_baselines","published_at":"2026-08-31"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":10,"result_count":10,"results":[[9352,53.69,370,370,9353],[9354,50.51,370,370,9353],[9355,45.53,370,370,9353],[9356,50.4,370,370,9353],[9357,53.3,370,370,9353],[9358,54.66,370,370,9353],[9359,54.46,370,370,9353],[9360,48.04,370,9361,9353],[9362,53.07,370,370,9353],[9363,12.76,370,370,9353]]},{"id":"framebench-english-2026-paper","name":"FrameBench · English (2026 paper)","group":"知识与推理","metric":"Frame-semantic multiple-choice accuracy","unit":"%","public_history":true,"protocol":{"en":"Models distinguish context-dependent semantic frames using native-speaker-validated English sentence pairs. Each pair is evaluated in original and reversed option order; scores are published means across five prompt templates. Reasoning on/off remain configurations; human reference scores and unrelated benchmark columns are excluded.","zh":"模型从经母语者验证的English句子对中辨别随语境变化的语义框架。每对题目都测试原顺序与反转选项顺序，保留五种提示模板的平均正确率。思考开关作为配置，人类对照与其他基准列不进入排行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.03370v1","published_at":"2026-09-03","status":"paper_baselines"},"model_count":12,"result_count":20,"results":[[9364,0.935,370,9365,9366],[2058,0.991,370,9365,9366],[9367,0.976,370,9365,9366],[9072,0.993,370,9365,9366],[9368,0.863,370,9365,9366],[9369,0.945,370,9365,9366],[9370,0.986,370,9365,9366],[9368,0.6859999999999999,370,9371,9366],[9369,0.825,370,9371,9366],[9370,0.975,370,9371,9366],[6340,0.33299999999999996,370,9365,9366],[5828,0.8029999999999999,370,9365,9366],[4969,0.955,370,9365,9366],[4070,0.973,370,9365,9366],[2584,0.986,370,9365,9366],[6340,0.47600000000000003,370,9371,9366],[5828,0.485,370,9371,9366],[4969,0.8109999999999999,370,9371,9366],[4070,0.858,370,9371,9366],[2584,0.961,370,9371,9366]]},{"id":"framebench-japanese-2026-paper","name":"FrameBench · Japanese (2026 paper)","group":"知识与推理","metric":"Frame-semantic multiple-choice accuracy","unit":"%","public_history":true,"protocol":{"en":"Models distinguish context-dependent semantic frames using native-speaker-validated Japanese sentence pairs. Each pair is evaluated in original and reversed option order; scores are published means across five prompt templates. Reasoning on/off remain configurations; human reference scores and unrelated benchmark columns are excluded.","zh":"模型从经母语者验证的Japanese句子对中辨别随语境变化的语义框架。每对题目都测试原顺序与反转选项顺序，保留五种提示模板的平均正确率。思考开关作为配置，人类对照与其他基准列不进入排行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.03370v1","published_at":"2026-09-03","status":"paper_baselines"},"model_count":17,"result_count":26,"results":[[9364,0.8390000000000001,370,9365,9366],[2058,0.963,370,9365,9366],[9367,0.978,370,9365,9366],[9072,0.975,370,9365,9366],[9368,0.823,370,9365,9366],[9369,0.922,370,9365,9366],[9370,0.991,370,9365,9366],[9368,0.415,370,9371,9366],[9369,0.846,370,9371,9366],[9370,0.976,370,9371,9366],[6340,0.242,370,9365,9366],[5828,0.607,370,9365,9366],[4969,0.899,370,9365,9366],[4070,0.9179999999999999,370,9365,9366],[2584,0.973,370,9365,9366],[6340,0.254,370,9371,9366],[5828,0.3,370,9371,9366],[4969,0.5670000000000001,370,9371,9366],[4070,0.7290000000000001,370,9371,9366],[2584,0.8370000000000001,370,9371,9366],[9372,0.845,370,9365,9366],[9372,0.6920000000000001,370,9371,9366],[9373,0.84,370,9365,9366],[9374,0.932,370,9365,9366],[5344,0.85,370,9365,9366],[786,0.9390000000000001,370,9365,9366]]},{"id":"gbagcs-classification-2026-paper","name":"GBA-GCs · gated-community recognition (2026 paper)","group":"视觉与视频","metric":"Classification accuracy","unit":"%","public_history":true,"protocol":{"en":"Sixteen trained classifiers use the expert-labeled Guangzhou split of the 37,444-compound GBA-GCs resource. Common official train/validation/test splits are stratified by area and geography. Modalities and frozen versus trained variants remain explicit configurations; the Improvement summary is excluded.","zh":"十六个训练后的分类器使用GBA-GCs资源中经专家标注的广州划分。该资源覆盖37,444个小区，统一训练、验证与测试划分按面积和地理位置分层。模态、冻结与训练变体保留为配置，排除Improvement汇总行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.03804v1","published_at":"2026-09-03","status":"paper_baselines"},"model_count":16,"result_count":16,"results":[[9375,0.731,370,9376,9377],[9378,0.644,370,9379,9377],[9380,0.752,370,9381,9377],[9382,0.74,370,9383,9377],[9384,0.753,370,9385,9377],[9386,0.747,370,9387,9377],[9203,0.571,370,9388,9377],[9389,0.731,370,9390,9377],[9391,0.695,370,9392,9377],[9393,0.651,370,9394,9377],[9395,0.74,370,9396,9377],[9397,0.742,370,9398,9377],[9399,0.717,370,9400,9377],[9401,0.744,370,9402,9377],[9403,0.746,370,9404,9377],[9405,0.853,370,9406,9377]]},{"id":"petqa-text-factuality-2026-paper","name":"PetQA-Bench · text (2026 paper)","group":"医疗","metric":"Factuality F1","unit":"score","public_history":true,"protocol":{"en":"Veterinary question answering in the original Korean text set. Factuality is the published precision/recall F1 between the response and reference, not a correctness percentage. The full applicable zero-shot model panel is retained; closed vision-language models average three runs. Text and multimodal cohorts remain separate.","zh":"在原始韩语text题集中进行兽医问答。Factuality是回答与参考答案之间的精确率、召回率F1，保留为原生分数。展示全部适用零样本模型，闭源视觉语言模型取三次运行均值；文本与多模态面板分别展示。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.04598v1","published_at":"2026-09-04","status":"paper_baselines"},"score_max":1,"saturation_excluded":true,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"score_decimals":3,"model_count":18,"result_count":18,"results":[[6275,0.551,370,9407,9408],[345,0.569,370,9407,9408],[2055,0.541,370,9407,9408],[2602,0.55,370,9407,9408],[6342,0.499,370,9407,9408],[6343,0.513,370,9407,9408],[6668,0.51,370,9407,9408],[6689,0.519,370,9407,9408],[9409,0.503,370,9407,9408],[9410,0.482,370,9407,9408],[9411,0.493,370,9407,9408],[5344,0.514,370,9407,9408],[786,0.53,370,9407,9408],[560,0.518,370,9407,9408],[9412,0.489,370,9407,9408],[9413,0.546,370,9407,9408],[9414,0.502,370,9407,9408],[9415,0.483,370,9407,9408]]},{"id":"petqa-multimodal-factuality-2026-paper","name":"PetQA-Bench · multimodal (2026 paper)","group":"医疗","metric":"Factuality F1","unit":"score","public_history":true,"protocol":{"en":"Veterinary question answering in the original Korean multimodal set. Factuality is the published precision/recall F1 between the response and reference, not a correctness percentage. The full applicable zero-shot model panel is retained; closed vision-language models average three runs. Text and multimodal cohorts remain separate.","zh":"在原始韩语multimodal题集中进行兽医问答。Factuality是回答与参考答案之间的精确率、召回率F1，保留为原生分数。展示全部适用零样本模型，闭源视觉语言模型取三次运行均值；文本与多模态面板分别展示。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.04598v1","published_at":"2026-09-04","status":"paper_baselines"},"score_max":1,"saturation_excluded":true,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"score_decimals":3,"model_count":11,"result_count":11,"results":[[6275,0.531,370,9407,9408],[345,0.531,370,9407,9408],[2055,0.495,370,9407,9408],[2602,0.524,370,9407,9408],[6342,0.461,370,9407,9408],[6343,0.489,370,9407,9408],[6668,0.49,370,9407,9408],[6689,0.496,370,9407,9408],[9409,0.442,370,9407,9408],[9410,0.461,370,9407,9408],[9411,0.451,370,9407,9408]]},{"id":"roborreward-lexical-2026-paper","name":"RoboRMBench · lexical (2026 paper)","group":"Agent 与工具","metric":"Mean absolute progress-score error","unit":"raw","public_history":true,"protocol":{"en":"Sixteen reward models judge the same trajectory–instruction pairs under lexical paraphrases. ME first averages predictions across paraphrases per trajectory, then averages absolute differences from human progress scores on the five-level reward scale. One model call is used per pair; hosted-provider defaults are part of the recorded setting. Lower error is better.","zh":"十六个奖励模型在lexical改写条件下评判相同轨迹与指令对。ME先对每条轨迹的改写预测取平均，再计算与人类进度评分的平均绝对差，单位为五级奖励刻度。每对输入调用模型一次，托管服务默认设置属于记录配置；误差越低越好。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.05401v1","published_at":"2026-09-04","status":"paper_baselines"},"higher_is_better":false,"model_count":16,"result_count":16,"results":[[353,1.199,370,9416,9417],[3748,1.036,370,9416,9417],[9418,1.027,370,9416,9417],[9419,1.011,370,9416,9417],[9420,0.681,370,9416,9417],[9421,1.102,370,9416,9417],[9422,0.999,370,9416,9417],[9423,0.893,370,9416,9417],[9424,0.904,370,9416,9417],[9425,0.921,370,9416,9417],[6865,1.08,370,9416,9417],[6864,1.371,370,9416,9417],[9426,1.178,370,9416,9417],[8284,1.105,370,9416,9417],[9427,0.323,370,9416,9417],[9428,0.255,370,9416,9417]]},{"id":"roborreward-syntactic-2026-paper","name":"RoboRMBench · syntactic (2026 paper)","group":"Agent 与工具","metric":"Mean absolute progress-score error","unit":"raw","public_history":true,"protocol":{"en":"Sixteen reward models judge the same trajectory–instruction pairs under syntactic paraphrases. ME first averages predictions across paraphrases per trajectory, then averages absolute differences from human progress scores on the five-level reward scale. One model call is used per pair; hosted-provider defaults are part of the recorded setting. Lower error is better.","zh":"十六个奖励模型在syntactic改写条件下评判相同轨迹与指令对。ME先对每条轨迹的改写预测取平均，再计算与人类进度评分的平均绝对差，单位为五级奖励刻度。每对输入调用模型一次，托管服务默认设置属于记录配置；误差越低越好。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.05401v1","published_at":"2026-09-04","status":"paper_baselines"},"higher_is_better":false,"model_count":16,"result_count":16,"results":[[353,1.26,370,9429,9417],[3748,1.024,370,9429,9417],[9418,0.951,370,9429,9417],[9419,0.864,370,9429,9417],[9420,0.744,370,9429,9417],[9421,1.027,370,9429,9417],[9422,0.938,370,9429,9417],[9423,0.981,370,9429,9417],[9424,0.838,370,9429,9417],[9425,0.892,370,9429,9417],[6865,1.085,370,9429,9417],[6864,1.216,370,9429,9417],[9426,1.175,370,9429,9417],[8284,1.229,370,9429,9417],[9427,0.375,370,9429,9417],[9428,0.336,370,9429,9417]]},{"id":"roborreward-action-goal-2026-paper","name":"RoboRMBench · action-goal (2026 paper)","group":"Agent 与工具","metric":"Mean absolute progress-score error","unit":"raw","public_history":true,"protocol":{"en":"Sixteen reward models judge the same trajectory–instruction pairs under action-goal paraphrases. ME first averages predictions across paraphrases per trajectory, then averages absolute differences from human progress scores on the five-level reward scale. One model call is used per pair; hosted-provider defaults are part of the recorded setting. Lower error is better.","zh":"十六个奖励模型在action-goal改写条件下评判相同轨迹与指令对。ME先对每条轨迹的改写预测取平均，再计算与人类进度评分的平均绝对差，单位为五级奖励刻度。每对输入调用模型一次，托管服务默认设置属于记录配置；误差越低越好。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.05401v1","published_at":"2026-09-04","status":"paper_baselines"},"higher_is_better":false,"model_count":16,"result_count":16,"results":[[353,1.285,370,9430,9417],[3748,1.015,370,9430,9417],[9418,0.97,370,9430,9417],[9419,0.854,370,9430,9417],[9420,0.817,370,9430,9417],[9421,1.001,370,9430,9417],[9422,0.97,370,9430,9417],[9423,1.06,370,9430,9417],[9424,0.832,370,9430,9417],[9425,0.906,370,9430,9417],[6865,1.131,370,9430,9417],[6864,1.124,370,9430,9417],[9426,1.109,370,9430,9417],[8284,1.245,370,9430,9417],[9427,0.456,370,9430,9417],[9428,0.453,370,9430,9417]]},{"id":"zdpshift-minus16-2026-paper","name":"ZDPShift · delta -16 (2026 paper)","group":"视觉与视频","metric":"Endpoint error (pixels)","unit":"raw","public_history":true,"protocol":{"en":"Six stereo backbones are evaluated at the fixed disparity shift -16. The complete panel preserves released zero-shot weights, frozen-matching signed adaptation and full-training adaptation as explicit configurations. Errors use 960-pixel inference and native pixel units over the full benchmark; lower is better.","zh":"六个双目骨干在固定视差偏移-16下评测。完整面板保留发布权重零样本、冻结匹配特征的有符号适配及全训练适配三种明确配置。采用960像素推理、完整基准上的原生像素误差，越低越好。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.06809v1","published_at":"2026-09-06","status":"paper_baselines"},"higher_is_better":false,"model_count":6,"result_count":18,"results":[[9431,3.84,370,9432,9433],[9434,4.12,370,9432,9433],[9435,3.15,370,9432,9433],[9436,3.98,370,9432,9433],[9437,3.85,370,9432,9433],[9438,2.91,370,9432,9433],[9431,3.98,370,9439,9433],[9434,4.68,370,9439,9433],[9435,3.67,370,9439,9433],[9436,5.01,370,9439,9433],[9437,4.36,370,9439,9433],[9438,2.82,370,9439,9433],[9431,4.16,370,9440,9433],[9434,4.54,370,9440,9433],[9435,3.69,370,9440,9433],[9436,4.9,370,9440,9433],[9437,4.12,370,9440,9433],[9438,2.91,370,9440,9433]]},{"id":"zdpshift-0-2026-paper","name":"ZDPShift · delta 0 (2026 paper)","group":"视觉与视频","metric":"Endpoint error (pixels)","unit":"raw","public_history":true,"protocol":{"en":"Six stereo backbones are evaluated at the fixed disparity shift 0. The complete panel preserves released zero-shot weights, frozen-matching signed adaptation and full-training adaptation as explicit configurations. Errors use 960-pixel inference and native pixel units over the full benchmark; lower is better.","zh":"六个双目骨干在固定视差偏移0下评测。完整面板保留发布权重零样本、冻结匹配特征的有符号适配及全训练适配三种明确配置。采用960像素推理、完整基准上的原生像素误差，越低越好。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.06809v1","published_at":"2026-09-06","status":"paper_baselines"},"higher_is_better":false,"model_count":6,"result_count":18,"results":[[9431,3.46,370,9432,9433],[9434,3.8,370,9432,9433],[9435,2.73,370,9432,9433],[9436,3.98,370,9432,9433],[9437,3.63,370,9432,9433],[9438,2.15,370,9432,9433],[9431,3.27,370,9439,9433],[9434,4.02,370,9439,9433],[9435,2.74,370,9439,9433],[9436,3.96,370,9439,9433],[9437,3.53,370,9439,9433],[9438,1.82,370,9439,9433],[9431,3.39,370,9440,9433],[9434,3.85,370,9440,9433],[9435,2.75,370,9440,9433],[9436,4.06,370,9440,9433],[9437,3.07,370,9440,9433],[9438,1.77,370,9440,9433]]},{"id":"zdpshift-16-2026-paper","name":"ZDPShift · delta 16 (2026 paper)","group":"视觉与视频","metric":"Endpoint error (pixels)","unit":"raw","public_history":true,"protocol":{"en":"Six stereo backbones are evaluated at the fixed disparity shift 16. The complete panel preserves released zero-shot weights, frozen-matching signed adaptation and full-training adaptation as explicit configurations. Errors use 960-pixel inference and native pixel units over the full benchmark; lower is better.","zh":"六个双目骨干在固定视差偏移16下评测。完整面板保留发布权重零样本、冻结匹配特征的有符号适配及全训练适配三种明确配置。采用960像素推理、完整基准上的原生像素误差，越低越好。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.06809v1","published_at":"2026-09-06","status":"paper_baselines"},"higher_is_better":false,"model_count":6,"result_count":18,"results":[[9431,9.02,370,9432,9433],[9434,8.51,370,9432,9433],[9435,6.15,370,9432,9433],[9436,78.31,370,9432,9433],[9437,12.38,370,9432,9433],[9438,13.03,370,9432,9433],[9431,3.95,370,9439,9433],[9434,4.3,370,9439,9433],[9435,3.56,370,9439,9433],[9436,4.87,370,9439,9433],[9437,3.98,370,9439,9433],[9438,2.71,370,9439,9433],[9431,3.95,370,9440,9433],[9434,4.16,370,9440,9433],[9435,3.39,370,9440,9433],[9436,4.6,370,9440,9433],[9437,3.69,370,9440,9433],[9438,2.61,370,9440,9433]]},{"id":"zdpshift-24-2026-paper","name":"ZDPShift · delta 24 (2026 paper)","group":"视觉与视频","metric":"Endpoint error (pixels)","unit":"raw","public_history":true,"protocol":{"en":"Six stereo backbones are evaluated at the fixed disparity shift 24. The complete panel preserves released zero-shot weights, frozen-matching signed adaptation and full-training adaptation as explicit configurations. Errors use 960-pixel inference and native pixel units over the full benchmark; lower is better.","zh":"六个双目骨干在固定视差偏移24下评测。完整面板保留发布权重零样本、冻结匹配特征的有符号适配及全训练适配三种明确配置。采用960像素推理、完整基准上的原生像素误差，越低越好。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.06809v1","published_at":"2026-09-06","status":"paper_baselines"},"higher_is_better":false,"model_count":6,"result_count":18,"results":[[9431,14.61,370,9432,9433],[9434,18.69,370,9432,9433],[9435,9.17,370,9432,9433],[9436,108.41,370,9432,9433],[9437,33.11,370,9432,9433],[9438,58.8,370,9432,9433],[9431,4.15,370,9439,9433],[9434,4.67,370,9439,9433],[9435,3.77,370,9439,9433],[9436,4.81,370,9439,9433],[9437,3.85,370,9439,9433],[9438,2.78,370,9439,9433],[9431,4.3,370,9440,9433],[9434,4.41,370,9440,9433],[9435,3.66,370,9440,9433],[9436,4.55,370,9440,9433],[9437,3.69,370,9440,9433],[9438,2.72,370,9440,9433]]},{"id":"zdpshift-32-2026-paper","name":"ZDPShift · delta 32 (2026 paper)","group":"视觉与视频","metric":"Endpoint error (pixels)","unit":"raw","public_history":true,"protocol":{"en":"Six stereo backbones are evaluated at the fixed disparity shift 32. The complete panel preserves released zero-shot weights, frozen-matching signed adaptation and full-training adaptation as explicit configurations. Errors use 960-pixel inference and native pixel units over the full benchmark; lower is better.","zh":"六个双目骨干在固定视差偏移32下评测。完整面板保留发布权重零样本、冻结匹配特征的有符号适配及全训练适配三种明确配置。采用960像素推理、完整基准上的原生像素误差，越低越好。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.06809v1","published_at":"2026-09-06","status":"paper_baselines"},"higher_is_better":false,"model_count":6,"result_count":18,"results":[[9431,20.9,370,9432,9433],[9434,36.62,370,9432,9433],[9435,12.91,370,9432,9433],[9436,150.46,370,9432,9433],[9437,44.79,370,9432,9433],[9438,75.53,370,9432,9433],[9431,4.35,370,9439,9433],[9434,4.75,370,9439,9433],[9435,4.06,370,9439,9433],[9436,4.78,370,9439,9433],[9437,3.85,370,9439,9433],[9438,2.94,370,9439,9433],[9431,4.75,370,9440,9433],[9434,4.56,370,9440,9433],[9435,3.9,370,9440,9433],[9436,4.57,370,9440,9433],[9437,3.73,370,9440,9433],[9438,2.84,370,9440,9433]]},{"id":"tier-qwenjudge-2026-paper","name":"TIER · Qwen2.5-7B-Instruct judge (2026 paper)","group":"安全与可靠性","metric":"Attack success rate","unit":"%","public_history":true,"protocol":{"en":"Six independent open-weight models answer the same 1,184 harmful prompts, balanced over four risk domains and four threat-implicitness levels. Attack success is harmful compliance under the six-label behavior taxonomy, judged by Qwen2.5-7B-Instruct. Default chat configurations use greedy decoding at temperature zero. Lower attack success is better; the judges remain separate cohorts.","zh":"六个独立开放权重模型回答同一组1,184条有害提示，覆盖四个风险领域和四级威胁隐含程度。Qwen2.5-7B-Instruct按六类行为标签判定有害遵从率。模型使用默认聊天配置，温度为零、贪心解码；攻击成功率越低越好，两位评判模型分别列组。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.05117v1","published_at":"2026-09-04","status":"paper_baselines"},"higher_is_better":false,"model_count":6,"result_count":6,"results":[[9441,0.241,370,9442,9443],[9444,0.21,370,9442,9443],[9445,0.509,370,9442,9443],[9446,0.239,370,9442,9443],[2032,0.19,370,9442,9443],[9447,0.248,370,9442,9443]]},{"id":"tier-llamajudge-2026-paper","name":"TIER · Llama-3.1-8B-Instruct judge (2026 paper)","group":"安全与可靠性","metric":"Attack success rate","unit":"%","public_history":true,"protocol":{"en":"Six independent open-weight models answer the same 1,184 harmful prompts, balanced over four risk domains and four threat-implicitness levels. Attack success is harmful compliance under the six-label behavior taxonomy, judged by Llama-3.1-8B-Instruct. Default chat configurations use greedy decoding at temperature zero. Lower attack success is better; the judges remain separate cohorts.","zh":"六个独立开放权重模型回答同一组1,184条有害提示，覆盖四个风险领域和四级威胁隐含程度。Llama-3.1-8B-Instruct按六类行为标签判定有害遵从率。模型使用默认聊天配置，温度为零、贪心解码；攻击成功率越低越好，两位评判模型分别列组。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.05117v1","published_at":"2026-09-04","status":"paper_baselines"},"higher_is_better":false,"model_count":6,"result_count":6,"results":[[9441,0.104,370,9448,9449],[9444,0.12,370,9448,9449],[9445,0.355,370,9448,9449],[9446,0.224,370,9448,9449],[2032,0.075,370,9448,9449],[9447,0.102,370,9448,9449]]},{"id":"muse-forecast-mase-2026-paper","name":"MUSE-Bench · forecasting (2026 paper)","group":"知识与推理","metric":"SeasonalNaive-normalized error (MASE)","unit":"raw","public_history":true,"protocol":{"en":"The full 31-system panel forecasts the same targets across 14 datasets and 29 horizon configurations. The paper pools valid target errors within each configuration, normalizes against SeasonalNaive on matching coverage, then takes an equal-weight geometric mean across configurations. Native error ratios are retained; lower is better. Statistical references are preserved as such.","zh":"完整31系统面板覆盖14个数据集、29种预测窗口配置，使用相同预测目标。论文先汇总各配置中的有效目标误差，以相同覆盖范围的SeasonalNaive归一化，再对配置取等权几何平均。保留原生误差比值，越低越好；统计参照模型保留其身份。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.15087v1","published_at":"2026-09-14","status":"paper_baselines"},"higher_is_better":false,"model_count":31,"result_count":31,"results":[[9450,0.7338,370,9451,9452],[9453,0.7381,370,9451,9452],[9454,0.7419,370,9451,9452],[9455,0.7477,370,9451,9452],[9456,0.7788,370,9451,9452],[9457,0.7913,370,9451,9452],[9458,0.8973,370,9459,9452],[9460,0.9079,370,9461,9452],[9462,0.9207,370,9451,9452],[9463,0.9418,370,9461,9452],[9464,0.9478,370,9465,9452],[9466,0.9541,370,9465,9452],[9467,0.9543,370,9461,9452],[9161,0.9686,370,9465,9452],[9468,0.9726,370,9461,9452],[9469,1.0,370,9470,9452],[9471,1.0162,370,9465,9452],[9472,1.0484,370,9461,9452],[9473,1.0559,370,9451,9452],[9474,1.0905,370,9451,9452],[9475,1.1345,370,9461,9452],[9476,1.1521,370,9461,9452],[9477,1.3201,370,9461,9452],[9478,1.374,370,9470,9452],[9479,1.4506,370,9470,9452],[9480,1.8817,370,9481,9452],[9482,2.2575,370,9470,9452],[9483,2.3387,370,9451,9452],[9484,4.5881,370,9470,9452],[9485,8.5229,370,9481,9452],[6142,9.9501,370,9481,9452]]},{"id":"ovsgg-free-text-2026-paper","name":"OVSGGBench · open-vocabulary scene graphs (2026 paper)","group":"视觉与视频","metric":"Mean recall at 50","unit":"score","public_history":true,"protocol":{"en":"All six independent free-text scene-graph models use ground-truth regions on the same 2,179 PSG test images, withheld vocabulary, graph-constrained decoding and the A3 semantic matcher at threshold 0.60. The score is class-balanced mean recall at 50 on its native 0–100 scale. Alternative matcher rows reuse generations and are excluded from this common-matcher cohort.","zh":"六个独立自由文本场景图模型都使用同一组2,179张PSG测试图像的真实区域，在隐藏词表、图约束解码和阈值0.60的A3语义匹配器下评测。指标为类别均衡的前50项平均召回，保留原生0–100刻度。其他匹配器复用同一输出，不进入这组统一匹配器排行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.12552v1","published_at":"2026-09-11","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":6,"result_count":6,"results":[[9486,31.3,370,9487,9488],[9489,25.1,370,9487,9488],[6343,5.2,370,9487,9488],[6342,3.3,370,9487,9488],[2654,4.5,370,9487,9488],[6241,2.2,370,9487,9488]]},{"id":"vical-pass-2026-paper","name":"V-ICAL · pass (2026 paper)","group":"Agent 与工具","metric":"Pass rate","unit":"%","public_history":true,"protocol":{"en":"Nineteen multimodal models execute the same 342 fixed interactive tasks across 37 environments, with identical demonstrations, task seeds, legal actions and decision budgets. Each task receives one model rollout; task results are averaged within environments, then environments are equally weighted. Pass records the predefined task-specific success condition.","zh":"十九个多模态模型执行37个环境中的同一组342项固定交互任务，演示、随机种子、合法动作和决策预算一致。每项任务运行一次，先在环境内平均，再对环境等权汇总。通过率使用各任务预先规定的成功条件。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.15683v1","published_at":"2026-09-14","status":"paper_baselines"},"model_count":19,"result_count":19,"results":[[3235,0.58,370,9490,9491],[5136,0.53,370,9490,9491],[502,0.53,370,9490,9491],[4048,0.51,370,9490,9491],[6133,0.42,370,9490,9491],[9492,0.41,370,9490,9491],[3153,0.35,370,9490,9491],[3155,0.39,370,9490,9491],[2576,0.31,370,9490,9491],[9493,0.3,370,9490,9491],[7955,0.25,370,9490,9491],[9494,0.23,370,9490,9491],[2166,0.23,370,9490,9491],[6300,0.14,370,9490,9491],[3763,0.27,370,9490,9491],[6901,0.21,370,9490,9491],[2584,0.24,370,9490,9491],[9195,0.2,370,9490,9491],[6297,0.05,370,9490,9491]]},{"id":"vical-progress-2026-paper","name":"V-ICAL · progress (2026 paper)","group":"Agent 与工具","metric":"Normalized progress score","unit":"score","public_history":true,"protocol":{"en":"Nineteen multimodal models execute the same 342 fixed interactive tasks across 37 environments, with identical demonstrations, task seeds, legal actions and decision budgets. Each task receives one model rollout; task results are averaged within environments, then environments are equally weighted. Progress preserves the benchmark’s environment-specific normalized 0–100 score, including progress before failure.","zh":"十九个多模态模型执行37个环境中的同一组342项固定交互任务，演示、随机种子、合法动作和决策预算一致。每项任务运行一次，先在环境内平均，再对环境等权汇总。进度分保留环境规则归一化后的0–100刻度，包括失败前取得的进展。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.15683v1","published_at":"2026-09-14","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":19,"result_count":19,"results":[[3235,54.4,370,9490,9491],[5136,49.5,370,9490,9491],[502,48.0,370,9490,9491],[4048,46.7,370,9490,9491],[6133,39.6,370,9490,9491],[9492,37.0,370,9490,9491],[3153,34.8,370,9490,9491],[3155,32.5,370,9490,9491],[2576,32.5,370,9490,9491],[9493,25.2,370,9490,9491],[7955,21.9,370,9490,9491],[9494,20.4,370,9490,9491],[2166,17.7,370,9490,9491],[6300,17.4,370,9490,9491],[3763,16.9,370,9490,9491],[6901,16.6,370,9490,9491],[2584,16.4,370,9490,9491],[9195,13.6,370,9490,9491],[6297,12.3,370,9490,9491]]},{"id":"vloc-file-f1-2026-paper","name":"VLoc Bench · file-f1 (2026 paper)","group":"编程","metric":"File F1","unit":"score","public_history":true,"protocol":{"en":"Twenty-seven independently named model/configuration rows use the paper’s read-only repository harness, generic CWE descriptions and at most 15 tool calls. Phase A macro-averages per-task file F1 over 500 vulnerable repositories, then averages three runs. Ground truth contains implementation source files touched by the fix, excluding tests, documentation and configuration. Static-analysis tools remain outside this model cohort; default/xhigh effort is recorded as configuration.","zh":"二十七条模型及配置结果使用统一只读仓库工具、通用CWE描述和最多15次工具调用。A阶段先对500个漏洞仓库逐任务计算文件F1，再取三次运行均值。真实标签为修复涉及的实现源码文件，排除测试、文档和配置。静态分析工具不计入模型面板，默认与xhigh思考强度保留为配置。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.15939v1","published_at":"2026-09-14","status":"paper_baselines"},"score_max":1,"saturation_excluded":true,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"score_decimals":3,"model_count":26,"result_count":27,"results":[[3196,0.229,370,9495,9496],[3196,0.221,370,9497,9496],[26,0.152,370,9498,9496],[2055,0.102,370,9498,9496],[5505,0.098,370,9498,9496],[3497,0.095,370,9498,9496],[2058,0.048,370,9498,9496],[5738,0.024,370,9498,9496],[227,0.186,370,9498,9496],[5480,0.101,370,9498,9496],[2584,0.091,370,9498,9496],[5620,0.091,370,9498,9496],[2590,0.085,370,9498,9496],[5343,0.07,370,9498,9496],[3419,0.069,370,9498,9496],[3190,0.054,370,9498,9496],[5648,0.012,370,9498,9496],[9499,0.044,370,9498,9496],[4070,0.043,370,9498,9496],[7567,0.039,370,9498,9496],[5650,0.034,370,9498,9496],[9500,0.001,370,9498,9496],[9501,0.0,370,9498,9496],[9502,0.0,370,9498,9496],[9503,0.223,370,9498,9496],[9504,0.209,370,9498,9496],[9505,0.135,370,9498,9496]]},{"id":"vloc-clean-tnr-2026-paper","name":"VLoc Bench · clean-tnr (2026 paper)","group":"编程","metric":"True negative rate","unit":"%","public_history":true,"protocol":{"en":"Twenty-seven independently named model/configuration rows use the paper’s read-only repository harness, generic CWE descriptions and at most 15 tool calls. Phase B is the fraction of the 500 matched patched repositories correctly reported clean, averaged across three runs. It is separate from vulnerable-file localization. Static-analysis tools remain outside this model cohort; default/xhigh effort is recorded as configuration.","zh":"二十七条模型及配置结果使用统一只读仓库工具、通用CWE描述和最多15次工具调用。B阶段统计500个对应已修复仓库被正确判为无漏洞的比例，并取三次运行均值，单独展示。静态分析工具不计入模型面板，默认与xhigh思考强度保留为配置。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.15939v1","published_at":"2026-09-14","status":"paper_baselines"},"model_count":26,"result_count":27,"results":[[5738,0.868,370,9498,9506],[2058,0.743,370,9498,9506],[5505,0.702,370,9498,9506],[3497,0.632,370,9498,9506],[2055,0.392,370,9498,9506],[26,0.329,370,9498,9506],[3196,0.279,370,9495,9506],[3196,0.192,370,9497,9506],[5620,0.75,370,9498,9506],[2584,0.748,370,9498,9506],[5648,0.745,370,9498,9506],[2590,0.74,370,9498,9506],[3419,0.72,370,9498,9506],[5343,0.719,370,9498,9506],[5480,0.682,370,9498,9506],[227,0.582,370,9498,9506],[3190,0.321,370,9498,9506],[9502,1.0,370,9498,9506],[9500,0.976,370,9498,9506],[9501,0.863,370,9498,9506],[5650,0.845,370,9498,9506],[4070,0.814,370,9498,9506],[7567,0.755,370,9498,9506],[9499,0.563,370,9498,9506],[9503,0.034,370,9498,9506],[9504,0.008,370,9498,9506],[9505,0.012,370,9498,9506]]},{"id":"pararecover-l1-pass-2026-paper","name":"ParaRecover · level 1 pass (2026 paper)","group":"Agent 与工具","metric":"Pass@1","unit":"%","public_history":true,"protocol":{"en":"The complete 17-row model/configuration panel uses the shared CoT prompt on LEVEL-1 of the 10,626-instance parallel tool-recovery resource. LEVEL-1 injects errors in the latest execution; LEVEL-2 includes multi-turn propagation. Pass@1 adopts the updated plan and checks completion within the specified step budget. The starred DeepSeek-Flash row is thinking mode, not an extra model.","zh":"完整17条模型及配置结果使用统一思维链提示，评测10,626实例并行工具恢复资源中的LEVEL-1。一级只注入最近一次执行的错误，二级包含多轮传播。Pass@1采用更新后的计划，检查是否在规定步数内完成任务。带星号的DeepSeek-Flash结果为思考模式，按配置保留。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.12345v1","published_at":"2026-09-11","status":"paper_baselines"},"model_count":16,"result_count":17,"results":[[353,0.9220999999999999,370,9507,9508],[801,0.8931999999999999,370,9507,9508],[6277,0.895,370,9507,9508],[340,0.9261,370,9507,9508],[3572,0.941,370,9507,9508],[2261,0.9478,370,9507,9508],[9509,0.9381999999999999,370,9507,9508],[9510,0.9368000000000001,370,9507,9508],[3683,0.9184,370,9507,9508],[9511,0.9208,370,9507,9508],[5672,0.9220999999999999,370,9507,9508],[7631,0.945,370,9507,9508],[9512,0.935,370,9507,9508],[9513,0.9381,370,9507,9508],[7895,0.9412999999999999,370,9507,9508],[7895,0.9467,370,9514,9508],[9515,0.9412999999999999,370,9507,9508]]},{"id":"pararecover-l1-sde-2026-paper","name":"ParaRecover · level 1 sde (2026 paper)","group":"Agent 与工具","metric":"SDE rubric mean","unit":"score","public_history":true,"protocol":{"en":"The complete 17-row model/configuration panel uses the shared CoT prompt on LEVEL-1 of the 10,626-instance parallel tool-recovery resource. LEVEL-1 injects errors in the latest execution; LEVEL-2 includes multi-turn propagation. SDE equally weights structural integrity, diagnostic reasoning and evolutionary strategy, each with five rubric items; the published normalized mean is retained as native 0–100 rubric points. The starred DeepSeek-Flash row is thinking mode, not an extra model.","zh":"完整17条模型及配置结果使用统一思维链提示，评测10,626实例并行工具恢复资源中的LEVEL-1。一级只注入最近一次执行的错误，二级包含多轮传播。SDE等权汇总结构完整性、诊断推理和演进策略，每项包含五条评分标准，保留原文归一化的0–100分。带星号的DeepSeek-Flash结果为思考模式，按配置保留。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.12345v1","published_at":"2026-09-11","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":16,"result_count":17,"results":[[353,51.45,370,9507,9508],[801,41.98,370,9507,9508],[6277,41.42,370,9507,9508],[340,60.24,370,9507,9508],[3572,64.58,370,9507,9508],[2261,69.23,370,9507,9508],[9509,60.47,370,9507,9508],[9510,58.95,370,9507,9508],[3683,43.47,370,9507,9508],[9511,46.8,370,9507,9508],[5672,55.9,370,9507,9508],[7631,67.54,370,9507,9508],[9512,55.37,370,9507,9508],[9513,56.64,370,9507,9508],[7895,68.04,370,9507,9508],[7895,68.42,370,9514,9508],[9515,68.29,370,9507,9508]]},{"id":"pararecover-l2-pass-2026-paper","name":"ParaRecover · level 2 pass (2026 paper)","group":"Agent 与工具","metric":"Pass@1","unit":"%","public_history":true,"protocol":{"en":"The complete 17-row model/configuration panel uses the shared CoT prompt on LEVEL-2 of the 10,626-instance parallel tool-recovery resource. LEVEL-1 injects errors in the latest execution; LEVEL-2 includes multi-turn propagation. Pass@1 adopts the updated plan and checks completion within the specified step budget. The starred DeepSeek-Flash row is thinking mode, not an extra model.","zh":"完整17条模型及配置结果使用统一思维链提示，评测10,626实例并行工具恢复资源中的LEVEL-2。一级只注入最近一次执行的错误，二级包含多轮传播。Pass@1采用更新后的计划，检查是否在规定步数内完成任务。带星号的DeepSeek-Flash结果为思考模式，按配置保留。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.12345v1","published_at":"2026-09-11","status":"paper_baselines"},"model_count":16,"result_count":17,"results":[[353,0.915,370,9507,9508],[801,0.8884000000000001,370,9507,9508],[6277,0.8892,370,9507,9508],[340,0.9184,370,9507,9508],[3572,0.935,370,9507,9508],[2261,0.9405,370,9507,9508],[9509,0.9325,370,9507,9508],[9510,0.9322,370,9507,9508],[3683,0.9137000000000001,370,9507,9508],[9511,0.9186,370,9507,9508],[5672,0.9214,370,9507,9508],[7631,0.9304000000000001,370,9507,9508],[9512,0.9272,370,9507,9508],[9513,0.9306,370,9507,9508],[7895,0.9317,370,9507,9508],[7895,0.9386,370,9514,9508],[9515,0.9301,370,9507,9508]]},{"id":"pararecover-l2-sde-2026-paper","name":"ParaRecover · level 2 sde (2026 paper)","group":"Agent 与工具","metric":"SDE rubric mean","unit":"score","public_history":true,"protocol":{"en":"The complete 17-row model/configuration panel uses the shared CoT prompt on LEVEL-2 of the 10,626-instance parallel tool-recovery resource. LEVEL-1 injects errors in the latest execution; LEVEL-2 includes multi-turn propagation. SDE equally weights structural integrity, diagnostic reasoning and evolutionary strategy, each with five rubric items; the published normalized mean is retained as native 0–100 rubric points. The starred DeepSeek-Flash row is thinking mode, not an extra model.","zh":"完整17条模型及配置结果使用统一思维链提示，评测10,626实例并行工具恢复资源中的LEVEL-2。一级只注入最近一次执行的错误，二级包含多轮传播。SDE等权汇总结构完整性、诊断推理和演进策略，每项包含五条评分标准，保留原文归一化的0–100分。带星号的DeepSeek-Flash结果为思考模式，按配置保留。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.12345v1","published_at":"2026-09-11","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":16,"result_count":17,"results":[[353,44.74,370,9507,9508],[801,36.9,370,9507,9508],[6277,38.23,370,9507,9508],[340,59.2,370,9507,9508],[3572,61.9,370,9507,9508],[2261,66.85,370,9507,9508],[9509,56.56,370,9507,9508],[9510,57.49,370,9507,9508],[3683,41.07,370,9507,9508],[9511,45.64,370,9507,9508],[5672,52.98,370,9507,9508],[7631,65.08,370,9507,9508],[9512,53.14,370,9507,9508],[9513,54.86,370,9507,9508],[7895,66.36,370,9507,9508],[7895,66.57,370,9514,9508],[9515,66.5,370,9507,9508]]},{"id":"muunriver-t1-2026-paper","name":"MUUNRiver-Bench · T1 (2026 paper)","group":"知识与推理","metric":"Mean average precision","unit":"score","public_history":true,"protocol":{"en":"Six independent music encoders/retrievers are compared in eight configurations on task T1, using the same task-specific pool of 3,440 synthesized tracks. Human-verified relevance distinguishes style, lyrics, melody, timbre, vocals and segments by task. The published mAP remains a native 0–1 retrieval metric. Fusion configurations and input modality remain explicit; supervised Fusion averages three seeds, other rows use frozen official checkpoints.","zh":"六个独立音乐编码或检索模型以八种配置评测T1任务，使用同一组3,440首合成曲目的任务候选池。相关性经人工验证，按任务区分风格、歌词、旋律、音色、人声和片段。保留原生0–1平均精度均值，融合配置和输入模态单独注明；监督融合取三个种子均值，其余使用冻结的官方权重。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.16090v1","published_at":"2026-09-14","status":"paper_baselines"},"score_max":1,"saturation_excluded":true,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"score_decimals":3,"model_count":8,"result_count":8,"results":[[9516,0.5,370,9517,9518],[9519,0.21,370,9520,9518],[9521,0.38,370,9520,9518],[9522,0.57,370,9517,9518],[9523,0.48,370,9524,9518],[9525,0.55,370,9526,9518],[9527,0.38,370,9528,9518],[9529,0.21,370,9528,9518]]},{"id":"muunriver-t2-2026-paper","name":"MUUNRiver-Bench · T2 (2026 paper)","group":"知识与推理","metric":"Mean average precision","unit":"score","public_history":true,"protocol":{"en":"Six independent music encoders/retrievers are compared in eight configurations on task T2, using the same task-specific pool of 3,440 synthesized tracks. Human-verified relevance distinguishes style, lyrics, melody, timbre, vocals and segments by task. The published mAP remains a native 0–1 retrieval metric. Fusion configurations and input modality remain explicit; supervised Fusion averages three seeds, other rows use frozen official checkpoints.","zh":"六个独立音乐编码或检索模型以八种配置评测T2任务，使用同一组3,440首合成曲目的任务候选池。相关性经人工验证，按任务区分风格、歌词、旋律、音色、人声和片段。保留原生0–1平均精度均值，融合配置和输入模态单独注明；监督融合取三个种子均值，其余使用冻结的官方权重。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.16090v1","published_at":"2026-09-14","status":"paper_baselines"},"score_max":1,"saturation_excluded":true,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"score_decimals":3,"model_count":8,"result_count":8,"results":[[9516,0.31,370,9517,9518],[9519,0.15,370,9520,9518],[9521,0.24,370,9520,9518],[9522,0.41,370,9517,9518],[9523,0.23,370,9524,9518],[9525,0.38,370,9526,9518],[9527,0.15,370,9528,9518],[9529,0.12,370,9528,9518]]},{"id":"muunriver-t3-2026-paper","name":"MUUNRiver-Bench · T3 (2026 paper)","group":"知识与推理","metric":"Mean average precision","unit":"score","public_history":true,"protocol":{"en":"Six independent music encoders/retrievers are compared in eight configurations on task T3, using the same task-specific pool of 3,440 synthesized tracks. Human-verified relevance distinguishes style, lyrics, melody, timbre, vocals and segments by task. The published mAP remains a native 0–1 retrieval metric. Fusion configurations and input modality remain explicit; supervised Fusion averages three seeds, other rows use frozen official checkpoints.","zh":"六个独立音乐编码或检索模型以八种配置评测T3任务，使用同一组3,440首合成曲目的任务候选池。相关性经人工验证，按任务区分风格、歌词、旋律、音色、人声和片段。保留原生0–1平均精度均值，融合配置和输入模态单独注明；监督融合取三个种子均值，其余使用冻结的官方权重。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.16090v1","published_at":"2026-09-14","status":"paper_baselines"},"score_max":1,"saturation_excluded":true,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"score_decimals":3,"model_count":8,"result_count":8,"results":[[9516,0.33,370,9517,9518],[9519,0.14,370,9520,9518],[9521,0.22,370,9520,9518],[9522,0.35,370,9517,9518],[9523,0.25,370,9524,9518],[9525,0.34,370,9526,9518],[9527,0.34,370,9528,9518],[9529,0.15,370,9528,9518]]},{"id":"muunriver-t4-2026-paper","name":"MUUNRiver-Bench · T4 (2026 paper)","group":"知识与推理","metric":"Mean average precision","unit":"score","public_history":true,"protocol":{"en":"Six independent music encoders/retrievers are compared in eight configurations on task T4, using the same task-specific pool of 3,440 synthesized tracks. Human-verified relevance distinguishes style, lyrics, melody, timbre, vocals and segments by task. The published mAP remains a native 0–1 retrieval metric. Fusion configurations and input modality remain explicit; supervised Fusion averages three seeds, other rows use frozen official checkpoints.","zh":"六个独立音乐编码或检索模型以八种配置评测T4任务，使用同一组3,440首合成曲目的任务候选池。相关性经人工验证，按任务区分风格、歌词、旋律、音色、人声和片段。保留原生0–1平均精度均值，融合配置和输入模态单独注明；监督融合取三个种子均值，其余使用冻结的官方权重。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.16090v1","published_at":"2026-09-14","status":"paper_baselines"},"score_max":1,"saturation_excluded":true,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"score_decimals":3,"model_count":8,"result_count":8,"results":[[9516,0.58,370,9517,9518],[9519,0.49,370,9520,9518],[9521,0.7,370,9520,9518],[9522,0.42,370,9517,9518],[9523,0.31,370,9524,9518],[9525,0.4,370,9526,9518],[9527,0.54,370,9528,9518],[9529,0.11,370,9528,9518]]},{"id":"muunriver-t5-2026-paper","name":"MUUNRiver-Bench · T5 (2026 paper)","group":"知识与推理","metric":"Mean average precision","unit":"score","public_history":true,"protocol":{"en":"Six independent music encoders/retrievers are compared in eight configurations on task T5, using the same task-specific pool of 3,440 synthesized tracks. Human-verified relevance distinguishes style, lyrics, melody, timbre, vocals and segments by task. The published mAP remains a native 0–1 retrieval metric. Fusion configurations and input modality remain explicit; supervised Fusion averages three seeds, other rows use frozen official checkpoints.","zh":"六个独立音乐编码或检索模型以八种配置评测T5任务，使用同一组3,440首合成曲目的任务候选池。相关性经人工验证，按任务区分风格、歌词、旋律、音色、人声和片段。保留原生0–1平均精度均值，融合配置和输入模态单独注明；监督融合取三个种子均值，其余使用冻结的官方权重。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.16090v1","published_at":"2026-09-14","status":"paper_baselines"},"score_max":1,"saturation_excluded":true,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"score_decimals":3,"model_count":8,"result_count":8,"results":[[9516,0.3,370,9517,9518],[9519,0.21,370,9520,9518],[9521,0.21,370,9520,9518],[9522,0.27,370,9517,9518],[9523,0.21,370,9524,9518],[9525,0.27,370,9526,9518],[9527,0.13,370,9528,9518],[9529,0.16,370,9528,9518]]},{"id":"muunriver-t6-2026-paper","name":"MUUNRiver-Bench · T6 (2026 paper)","group":"知识与推理","metric":"Mean average precision","unit":"score","public_history":true,"protocol":{"en":"Six independent music encoders/retrievers are compared in eight configurations on task T6, using the same task-specific pool of 3,440 synthesized tracks. Human-verified relevance distinguishes style, lyrics, melody, timbre, vocals and segments by task. The published mAP remains a native 0–1 retrieval metric. Fusion configurations and input modality remain explicit; supervised Fusion averages three seeds, other rows use frozen official checkpoints.","zh":"六个独立音乐编码或检索模型以八种配置评测T6任务，使用同一组3,440首合成曲目的任务候选池。相关性经人工验证，按任务区分风格、歌词、旋律、音色、人声和片段。保留原生0–1平均精度均值，融合配置和输入模态单独注明；监督融合取三个种子均值，其余使用冻结的官方权重。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.16090v1","published_at":"2026-09-14","status":"paper_baselines"},"score_max":1,"saturation_excluded":true,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"score_decimals":3,"model_count":8,"result_count":8,"results":[[9516,0.18,370,9517,9518],[9519,0.03,370,9520,9518],[9521,0.05,370,9520,9518],[9522,0.04,370,9517,9518],[9523,0.05,370,9524,9518],[9525,0.03,370,9526,9518],[9527,0.55,370,9528,9518],[9529,0.07,370,9528,9518]]},{"id":"muunriver-t7-2026-paper","name":"MUUNRiver-Bench · T7 (2026 paper)","group":"知识与推理","metric":"Mean average precision","unit":"score","public_history":true,"protocol":{"en":"Six independent music encoders/retrievers are compared in eight configurations on task T7, using the same task-specific pool of 3,440 synthesized tracks. Human-verified relevance distinguishes style, lyrics, melody, timbre, vocals and segments by task. The published mAP remains a native 0–1 retrieval metric. Fusion configurations and input modality remain explicit; supervised Fusion averages three seeds, other rows use frozen official checkpoints.","zh":"六个独立音乐编码或检索模型以八种配置评测T7任务，使用同一组3,440首合成曲目的任务候选池。相关性经人工验证，按任务区分风格、歌词、旋律、音色、人声和片段。保留原生0–1平均精度均值，融合配置和输入模态单独注明；监督融合取三个种子均值，其余使用冻结的官方权重。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.16090v1","published_at":"2026-09-14","status":"paper_baselines"},"score_max":1,"saturation_excluded":true,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"score_decimals":3,"model_count":8,"result_count":8,"results":[[9516,0.24,370,9517,9518],[9519,0.93,370,9520,9518],[9521,0.91,370,9520,9518],[9522,0.81,370,9517,9518],[9523,0.74,370,9524,9518],[9525,0.81,370,9526,9518],[9527,0.51,370,9528,9518],[9529,0.16,370,9528,9518]]},{"id":"capmem-caption-30s-2026-paper","name":"CapMem · caption-30s (2026 paper)","group":"Agent 与工具","metric":"Multiple-choice accuracy","unit":"%","public_history":true,"protocol":{"en":"Human-authored episodic-memory questions over 75 egocentric videos,totaling 33.7 hours and 1,000 multiple-choice items. The caption-30 s panel preserves the full-question model rows. CaptionQA reuses timestamped 30/60 second captions; DirectVideoQA uses uniformly sampled frames with model-specific caps. Empty,refused,safety-blocked and unparsable QA responses count as incorrect. Gemini DirectVideoQA is marked partial due to API budget and is excluded only from that panel.","zh":"在75段第一视角视频、33.7小时内容上回答1,000道人工作者编写的情节记忆选择题。caption-30s面板保留完整题集成绩。CaptionQA复用带时间戳的30或60秒窗口描述，DirectVideoQA按模型帧数上限均匀抽帧。空回答、拒答、安全拦截及无法解析的回答算错误。Gemini直接视频行因API预算仅测部分题，只从该面板排除。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.17688v1","published_at":"2026-09-15","status":"paper_baselines"},"model_count":12,"result_count":12,"results":[[2654,0.358,370,9530,9531],[2642,0.42100000000000004,370,9530,9531],[6341,0.363,370,9530,9531],[6298,0.423,370,9530,9531],[6342,0.396,370,9530,9531],[6343,0.47700000000000004,370,9530,9531],[5828,0.37200000000000005,370,9530,9531],[4969,0.418,370,9530,9531],[4070,0.428,370,9530,9531],[2584,0.493,370,9530,9531],[340,0.52,370,9530,9531],[3424,0.557,370,9530,9531]]},{"id":"capmem-caption-60s-2026-paper","name":"CapMem · caption-60s (2026 paper)","group":"Agent 与工具","metric":"Multiple-choice accuracy","unit":"%","public_history":true,"protocol":{"en":"Human-authored episodic-memory questions over 75 egocentric videos,totaling 33.7 hours and 1,000 multiple-choice items. The caption-60 s panel preserves the full-question model rows. CaptionQA reuses timestamped 30/60 second captions; DirectVideoQA uses uniformly sampled frames with model-specific caps. Empty,refused,safety-blocked and unparsable QA responses count as incorrect. Gemini DirectVideoQA is marked partial due to API budget and is excluded only from that panel.","zh":"在75段第一视角视频、33.7小时内容上回答1,000道人工作者编写的情节记忆选择题。caption-60s面板保留完整题集成绩。CaptionQA复用带时间戳的30或60秒窗口描述，DirectVideoQA按模型帧数上限均匀抽帧。空回答、拒答、安全拦截及无法解析的回答算错误。Gemini直接视频行因API预算仅测部分题，只从该面板排除。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.17688v1","published_at":"2026-09-15","status":"paper_baselines"},"model_count":12,"result_count":12,"results":[[2654,0.34299999999999997,370,9532,9531],[2642,0.39799999999999996,370,9532,9531],[6341,0.35200000000000004,370,9532,9531],[6298,0.40299999999999997,370,9532,9531],[6342,0.39,370,9532,9531],[6343,0.449,370,9532,9531],[5828,0.34700000000000003,370,9532,9531],[4969,0.396,370,9532,9531],[4070,0.43700000000000006,370,9532,9531],[2584,0.469,370,9532,9531],[340,0.514,370,9532,9531],[3424,0.532,370,9532,9531]]},{"id":"capmem-direct-video-2026-paper","name":"CapMem · direct-video (2026 paper)","group":"Agent 与工具","metric":"Multiple-choice accuracy","unit":"%","public_history":true,"protocol":{"en":"Human-authored episodic-memory questions over 75 egocentric videos,totaling 33.7 hours and 1,000 multiple-choice items. The direct-video panel preserves the full-question model rows. CaptionQA reuses timestamped 30/60 second captions; DirectVideoQA uses uniformly sampled frames with model-specific caps. Empty,refused,safety-blocked and unparsable QA responses count as incorrect. Gemini DirectVideoQA is marked partial due to API budget and is excluded only from that panel.","zh":"在75段第一视角视频、33.7小时内容上回答1,000道人工作者编写的情节记忆选择题。direct-video面板保留完整题集成绩。CaptionQA复用带时间戳的30或60秒窗口描述，DirectVideoQA按模型帧数上限均匀抽帧。空回答、拒答、安全拦截及无法解析的回答算错误。Gemini直接视频行因API预算仅测部分题，只从该面板排除。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.17688v1","published_at":"2026-09-15","status":"paper_baselines"},"model_count":11,"result_count":11,"results":[[2654,0.39399999999999996,370,9533,9531],[2642,0.428,370,9533,9531],[6341,0.354,370,9533,9531],[6298,0.371,370,9533,9531],[6342,0.37200000000000005,370,9533,9531],[6343,0.428,370,9533,9531],[5828,0.37200000000000005,370,9533,9531],[4969,0.396,370,9533,9531],[4070,0.423,370,9533,9531],[2584,0.45,370,9533,9531],[340,0.373,370,9533,9531]]},{"id":"sci-mmr-direct-2026-paper","name":"Sci-MMR · direct (2026 paper)","group":"数学与科学","metric":"Answer accuracy","unit":"%","public_history":true,"protocol":{"en":"Eight models answer 235 expert-reviewed scientific questions grounded in source-paper visuals and argument graphs. The direct setting preserves the original visual context; tool-use additionally permits cropping and magnification. Accuracy uses the planned 235-question denominator per model and counts missing evaluable answers as incorrect. The supplementary audit reports 1,866/1,880 available answers for Direct and 1,873/1,880 for Tool-Use. Evidence and claim coverage are separate measures.","zh":"八个模型回答235道经专家复核、以论文图像和论证图为依据的科学问题。direct设置保留原始视觉上下文，工具设置另允许裁切和放大。正确率按每个模型计划的235题计算，缺少可评估答案计错。补充审计记录Direct有1,866/1,880个可用答案，Tool-Use有1,873/1,880个；证据覆盖与论断覆盖属于其他指标。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.11243v1","published_at":"2026-09-10","status":"paper_baselines"},"model_count":8,"result_count":8,"results":[[3196,0.677,370,9534,9535],[3479,0.5489999999999999,370,9534,9535],[3440,0.506,370,9534,9535],[3412,0.502,370,9534,9535],[3198,0.455,370,9534,9535],[6675,0.413,370,9534,9535],[5734,0.315,370,9534,9535],[9536,0.268,370,9534,9535]]},{"id":"sci-mmr-tool-use-2026-paper","name":"Sci-MMR · tool-use (2026 paper)","group":"数学与科学","metric":"Answer accuracy","unit":"%","public_history":true,"protocol":{"en":"Eight models answer 235 expert-reviewed scientific questions grounded in source-paper visuals and argument graphs. The tool-use setting preserves the original visual context; tool-use additionally permits cropping and magnification. Accuracy uses the planned 235-question denominator per model and counts missing evaluable answers as incorrect. The supplementary audit reports 1,866/1,880 available answers for Direct and 1,873/1,880 for Tool-Use. Evidence and claim coverage are separate measures.","zh":"八个模型回答235道经专家复核、以论文图像和论证图为依据的科学问题。tool-use设置保留原始视觉上下文，工具设置另允许裁切和放大。正确率按每个模型计划的235题计算，缺少可评估答案计错。补充审计记录Direct有1,866/1,880个可用答案，Tool-Use有1,873/1,880个；证据覆盖与论断覆盖属于其他指标。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.11243v1","published_at":"2026-09-10","status":"paper_baselines"},"model_count":8,"result_count":8,"results":[[3196,0.698,370,9537,9535],[3479,0.6,370,9537,9535],[3440,0.528,370,9537,9535],[3412,0.523,370,9537,9535],[3198,0.506,370,9537,9535],[6675,0.44299999999999995,370,9537,9535],[5734,0.409,370,9537,9535],[9536,0.34,370,9537,9535]]},{"id":"xdailybench-task-score-2026-paper","name":"xDailyBench · everyday tasks (2026 paper)","group":"专业工作","metric":"Weighted task-rubric score","unit":"score","public_history":true,"protocol":{"en":"Eleven models complete 248 user-grounded tasks across 51 everyday scenarios. GLM-5.1 judges binary task-specific criteria,using response judging for text and agent judging for files. Criteria carry importance weights; task scores are averaged equally. This remains a native 0–100 rubric score. Nanobot is the default harness; Kimi-K3,Kimi-K2.6 andGemini-3.1-Pro use OpenCode after observed compatibility problems.","zh":"十一个模型完成248道来自用户实际需求的任务，覆盖51种日常场景。GLM-5.1按每题二元准则评分，文本直接评审，文件由裁判智能体检查。准则按重要性加权后，对任务等权平均，保留0–100原生量表分。默认框架为Nanobot；Kimi-K3、Kimi-K2.6及Gemini-3.1-Pro因兼容问题使用OpenCode。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.07784v1","published_at":"2026-09-07","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":11,"result_count":11,"results":[[9538,75.6,370,9539,9540],[3155,75.6,370,9541,9540],[3196,72.6,370,9539,9540],[4048,72.4,370,9539,9540],[3181,70.8,370,9539,9540],[3202,68.6,370,9539,9540],[3185,67.3,370,9539,9540],[8343,62.6,370,9539,9540],[3187,61.2,370,9541,9540],[3186,61.2,370,9539,9540],[502,51.3,370,9541,9540]]},{"id":"textvbench-t2v-2026-paper","name":"TextVBench · t2v (2026 paper)","group":"生成与创作","metric":"TextVScore","unit":"score","public_history":true,"protocol":{"en":"All models use the same 300 cases. Every video completed evaluation. The length-weighted harmonic composite combines spatial and temporal text fidelity after OCR-candidate correspondence; native 0–100 score. Tracks remain separate.","zh":"各模型使用同一组300个案例，全部视频完成评分。OCR候选匹配后按文字长度加权，综合空间与时间文字忠实度，保留0–100原生分数。两个赛道分别展示。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.13412v1","published_at":"2026-10-09","status":"paper_baselines"},"score_max":100,"score_decimals":2,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":8,"result_count":8,"results":[[9542,18.133,370,9543,9544],[9545,43.853,370,9543,9544],[9546,34.577,370,9543,9544],[7125,68.32,370,9543,9544],[9547,12.598,370,9543,9544],[8303,10.218,370,9543,9544],[9548,9.792,370,9543,9544],[9549,27.359,370,9543,9544]]},{"id":"textvbench-ref2v-2026-paper","name":"TextVBench · ref2v (2026 paper)","group":"生成与创作","metric":"TextVScore","unit":"score","public_history":true,"protocol":{"en":"All models use the same 200 cases. Every video completed evaluation. The length-weighted harmonic composite combines spatial and temporal text fidelity after OCR-candidate correspondence; native 0–100 score. Tracks remain separate.","zh":"各模型使用同一组200个案例，全部视频完成评分。OCR候选匹配后按文字长度加权，综合空间与时间文字忠实度，保留0–100原生分数。两个赛道分别展示。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.13412v1","published_at":"2026-10-09","status":"paper_baselines"},"score_max":100,"score_decimals":2,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":6,"result_count":6,"results":[[9546,56.388,370,9550,9544],[7125,56.465,370,9550,9544],[9547,54.283,370,9550,9544],[8303,31.197,370,9550,9544],[9548,56.506,370,9550,9544],[9549,27.334,370,9550,9544]]},{"id":"ec2-single-model-completion-2026-paper","name":"EC²-Bench · single-model endpoints (2026 paper)","group":"Agent 与工具","metric":"Weighted task completion","unit":"score","public_history":true,"protocol":{"en":"Six single-model endpoints run the same 128 executable tasks. Completion is a weighted mean of task checks, with partial credit; it stays a native composite. Multi-model collaboration methods and strict three-run reliability are separate.","zh":"六个单模型端点执行同一组128个可执行任务。完成度为任务检查项的加权均值，允许部分得分，保留原生综合分。多模型协作方法与三次全通过可靠性另行区分。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.17816v1","published_at":"2026-10-13","status":"paper_baselines"},"score_max":100,"score_decimals":2,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":6,"result_count":6,"results":[[5828,12.75,370,9551,9552],[4070,39.12,370,9551,9552],[2584,61.79,370,9551,9552],[4961,71.34,370,9551,9552],[4962,72.9,370,9553,9552],[3154,82.7,370,9553,9552]]},{"id":"finskill-no-skill-2026-paper","name":"FinSkillBench · No-Skill (2026 paper)","group":"专业工作","metric":"Task-native utility","unit":"score","public_history":true,"protocol":{"en":"Nine models use the same financial tasks and No-Skill resource condition. Utility aggregates native numerical and decision scorers. Phi-4 exhausted its turn budget on all 1,980 episodes and its measured zero is retained. No submission failures are silently dropped.","zh":"九个模型使用同一组金融任务及No-Skill资源条件，按原生数值与决策评分器汇总效用。Phi-4全部1,980次运行耗尽轮数预算，保留实测零分；未提交等失败不静默剔除。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.03564v1","published_at":"2026-10-02","status":"paper_baselines"},"score_max":1,"score_decimals":2,"saturation_excluded":true,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":9,"result_count":9,"results":[[1500,0.33,370,9554,9555],[457,0.38,370,9554,9555],[3009,0.458,370,9554,9555],[9556,0.419,370,9554,9555],[9557,0.534,370,9554,9555],[5368,0.43,370,9554,9555],[440,0.274,370,9554,9555],[9558,0.104,370,9554,9555],[1621,0.0,370,9554,9555]]},{"id":"finskill-curated-2026-paper","name":"FinSkillBench · Curated (2026 paper)","group":"专业工作","metric":"Task-native utility","unit":"score","public_history":true,"protocol":{"en":"Nine models use the same financial tasks and Curated resource condition. Utility aggregates native numerical and decision scorers. Phi-4 exhausted its turn budget on all 1,980 episodes and its measured zero is retained. No submission failures are silently dropped.","zh":"九个模型使用同一组金融任务及Curated资源条件，按原生数值与决策评分器汇总效用。Phi-4全部1,980次运行耗尽轮数预算，保留实测零分；未提交等失败不静默剔除。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.03564v1","published_at":"2026-10-02","status":"paper_baselines"},"score_max":1,"score_decimals":2,"saturation_excluded":true,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":9,"result_count":9,"results":[[1500,0.735,370,9559,9555],[457,0.678,370,9559,9555],[3009,0.658,370,9559,9555],[9556,0.609,370,9559,9555],[9557,0.659,370,9559,9555],[5368,0.52,370,9559,9555],[440,0.285,370,9559,9555],[9558,0.077,370,9559,9555],[1621,0.0,370,9559,9555]]},{"id":"finskill-self-generated-2026-paper","name":"FinSkillBench · Self-Gen (2026 paper)","group":"专业工作","metric":"Task-native utility","unit":"score","public_history":true,"protocol":{"en":"Nine models use the same financial tasks and Self-Gen resource condition. Utility aggregates native numerical and decision scorers. Phi-4 exhausted its turn budget on all 1,980 episodes and its measured zero is retained. No submission failures are silently dropped.","zh":"九个模型使用同一组金融任务及Self-Gen资源条件，按原生数值与决策评分器汇总效用。Phi-4全部1,980次运行耗尽轮数预算，保留实测零分；未提交等失败不静默剔除。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.03564v1","published_at":"2026-10-02","status":"paper_baselines"},"score_max":1,"score_decimals":2,"saturation_excluded":true,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":9,"result_count":9,"results":[[1500,0.398,370,9560,9555],[457,0.412,370,9560,9555],[3009,0.447,370,9560,9555],[9556,0.434,370,9560,9555],[9557,0.488,370,9560,9555],[5368,0.405,370,9560,9555],[440,0.245,370,9560,9555],[9558,0.138,370,9560,9555],[1621,0.0,370,9560,9555]]},{"id":"mamath-single-accuracy-2026-paper","name":"MAMath-Bench · single models (2026 paper)","group":"数学与科学","metric":"Final-answer accuracy","unit":"%","public_history":true,"protocol":{"en":"Ten single models answer the multimodal mathematical benchmark. FAA is final-answer accuracy; process-level MA, spurious reasoning and missing-step metrics are distinct. Multi-agent methods are not counted as additional independent models.","zh":"十个单模型回答多模态数学评测，FAA为最终答案准确率，与过程级MA、伪推理及缺步指标区分。多智能体方法不当成额外独立底模。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.26943v1","published_at":"2026-10-18","status":"paper_baselines"},"model_count":10,"result_count":10,"results":[[9561,0.7204,370,9562,9563],[9564,0.7537999999999999,370,9562,9563],[2626,0.5201,370,9562,9563],[7938,0.5251,370,9562,9563],[9565,0.41850000000000004,370,9562,9563],[345,0.7594,370,9562,9563],[3184,0.7965000000000001,370,9562,9563],[9566,0.6456999999999999,370,9562,9563],[3185,0.6867,370,9562,9563],[5650,0.5664,370,9562,9563]]},{"id":"frontierrefactor-pass1-2026-paper","name":"FrontierRefactor · strict pass@1 (2026 paper)","group":"编程","metric":"Strict task pass@1","unit":"%","public_history":true,"protocol":{"en":"Six coding models each attempt sixty refactoring tasks three times. Strict success requires functional equivalence, verified speedup and compliance gates in isolated Harbor environments. The published mean single-trial success is retained, distinct from best-of-three and continuous progress scores.","zh":"六个编程模型分别对六十道重构任务运行三次，在隔离Harbor环境中同时满足功能等价、经验证加速与合规门控才算成功。保留原表单次平均成功率，与三次取优及连续进度分区分。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.25834v1","published_at":"2026-10-19","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[755,0.344,370,9567,9568],[3568,0.261,370,9567,9568],[3453,0.261,370,9567,9568],[2958,0.306,370,9569,9568],[762,0.256,370,9569,9568],[3153,0.4,370,9567,9568]]},{"id":"repprobe-real-2026-paper","name":"RepProbe · real (2026 paper)","group":"视觉与视频","metric":"Exact repetition-count accuracy","unit":"%","public_history":true,"protocol":{"en":"Counts complete repeated actions with known ground-truth counts from one to twenty. Exact accuracy requires the predicted integer count to match. The real-video panel includes two specialist counters and seven generalists; the synthetic panel contains seven generalists only. Count predictions are clipped to 0–40; panels remain separate.","zh":"统计真实次数已知、范围一至二十次的完整重复动作。预测整数次数完全正确才算成功。真实视频面板包含两个专用计数器和七个通用模型，合成面板仅有七个通用模型。预测值截断到0–40，两个面板分别展示。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.18844v1","published_at":"2026-10-13","status":"paper_baselines"},"model_count":9,"result_count":9,"results":[[9570,0.419,370,9571,9572],[9573,0.217,370,9571,9572],[7066,0.175,370,9571,9572],[6342,0.129,370,9571,9572],[7392,0.07200000000000001,370,9571,9572],[9574,0.07,370,9571,9572],[9575,0.065,370,9571,9572],[2654,0.073,370,9571,9572],[6389,0.048,370,9571,9572]]},{"id":"repprobe-synthetic-2026-paper","name":"RepProbe · synthetic (2026 paper)","group":"视觉与视频","metric":"Exact repetition-count accuracy","unit":"%","public_history":true,"protocol":{"en":"Counts complete repeated actions with known ground-truth counts from one to twenty. Exact accuracy requires the predicted integer count to match. The real-video panel includes two specialist counters and seven generalists; the synthetic panel contains seven generalists only. Count predictions are clipped to 0–40; panels remain separate.","zh":"统计真实次数已知、范围一至二十次的完整重复动作。预测整数次数完全正确才算成功。真实视频面板包含两个专用计数器和七个通用模型，合成面板仅有七个通用模型。预测值截断到0–40，两个面板分别展示。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.18844v1","published_at":"2026-10-13","status":"paper_baselines"},"model_count":7,"result_count":7,"results":[[7066,0.055999999999999994,370,9576,9572],[6342,0.063,370,9576,9572],[7392,0.047,370,9576,9572],[9574,0.052000000000000005,370,9576,9572],[9575,0.044000000000000004,370,9576,9572],[2654,0.051,370,9576,9572],[6389,0.055999999999999994,370,9576,9572]]},{"id":"coldddi-s2-800-2026-paper","name":"ColdDDI · unseen-drug pairs, 800-drug subset (2026 paper)","group":"医疗","metric":"ROC-AUC","unit":"score","public_history":true,"protocol":{"en":"Nine published methods use the same 800-drug S2 subset, with both endpoints unseen in training. ROC-AUC is the three-seed mean and remains a native discrimination score. The fine-tuned Llama row is one model configuration; this panel is distinct from full-corpus results.","zh":"九种已发表方法使用同一800药物的S2子集，测试药物对的两个端点均未在训练中出现。采用三个种子的ROC-AUC均值，保留原生区分度分数。微调Llama行为一个模型配置，不与完整语料成绩混合。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.05590v1","published_at":"2026-10-04","status":"paper_baselines"},"score_max":1,"score_decimals":2,"saturation_excluded":true,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":9,"result_count":9,"results":[[9577,0.659,370,9578,9579],[9580,0.614,370,9578,9579],[9581,0.758,370,9578,9579],[9582,0.634,370,9578,9579],[9583,0.716,370,9578,9579],[9584,0.566,370,9578,9579],[9585,0.566,370,9578,9579],[9586,0.675,370,9578,9579],[623,0.776,370,9587,9579]]},{"id":"spacecast-overall-2026-paper","name":"SpaceCast-Bench (2026 paper)","group":"视觉与视频","metric":"Overall answer accuracy","unit":"%","public_history":true,"protocol":{"en":"Twenty-one models answer 3,862 questions from 182 real scenes across sixteen static and predictive spatial task types. Geometry-derived labels are automatically filtered and independently reviewed by at least two annotators. The published Overall is retained; human and random controls are excluded.","zh":"二十一个模型回答182个真实场景中的3,862道问题，覆盖十六类静态与预测空间任务。几何生成标签经过自动筛选和至少两名标注者独立复核，保留原表Overall，排除人类与随机对照。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.12402v1","published_at":"2026-10-08","status":"paper_baselines"},"model_count":21,"result_count":21,"results":[[3572,0.36,370,9588,9589],[3568,0.38299999999999995,370,9588,9589],[3415,0.58,370,9588,9589],[3424,0.5670000000000001,370,9588,9589],[3184,0.377,370,9588,9589],[5616,0.466,370,9588,9589],[9590,0.456,370,9588,9589],[6377,0.47700000000000004,370,9588,9589],[4961,0.5529999999999999,370,9588,9589],[2654,0.304,370,9588,9589],[6298,0.34,370,9588,9589],[5828,0.313,370,9588,9589],[4969,0.337,370,9588,9589],[4070,0.34600000000000003,370,9588,9589],[2584,0.39299999999999996,370,9588,9589],[9591,0.257,370,9588,9589],[9592,0.29100000000000004,370,9588,9589],[8867,0.278,370,9588,9589],[9593,0.27699999999999997,370,9588,9589],[9594,0.27899999999999997,370,9588,9589],[9595,0.264,370,9588,9589]]},{"id":"libero-perm-all-pref-success-2026-paper","name":"LIBERO-PeRM · zero-shot All-Pref (2026 paper)","group":"Agent 与工具","metric":"Task success rate","unit":"%","public_history":true,"protocol":{"en":"Six released VLA policies run the same zero-shot All-Pref condition on the 100-task LIBERO-SinglePref suite. SR measures task completion and is kept separate from the discrete-weighted preference satisfaction SA. No preference-specific fine-tuning is used.","zh":"六种VLA策略使用100任务的LIBERO-SinglePref套件，运行相同零样本All-Pref条件。SR表示任务完成率，与离散加权的偏好满足度SA分别呈现，不进行偏好专用微调。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.22372v1","published_at":"2026-10-15","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[7594,0.4,370,9596,9597],[9598,0.5125,370,9596,9597],[8323,0.6667000000000001,370,9596,9597],[9599,0.5167,370,9596,9597],[9600,0.0417,370,9596,9597],[7598,0.4583,370,9596,9597]]},{"id":"aerointent-radar-accuracy-2026-paper","name":"AeroIntent · radar-only (2026 paper)","group":"预测与决策","metric":"Intent accuracy","unit":"%","public_history":true,"protocol":{"en":"Six trajectory architectures receive the same 30-slot radar position/velocity input. Each seed holds out one complete trajectory program per intent: 225 balanced test sequences, 45 per class, under seeds 42/43/44. Accuracy is the three-seed mean, separate from Macro-F1. RGB, ground-truth-payload and richer temporal-metadata methods are excluded from this panel.","zh":"六种轨迹模型使用相同30时刻雷达位置与速度输入。每个种子按意图留出完整轨迹生成程序：测试225条序列、五类各45条，种子为42/43/44。保留三个种子的准确率均值，与Macro-F1区分；RGB、真值载荷及额外时间信息方法不混入此面板。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.25869v1","published_at":"2026-10-19","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[9203,0.5837,370,9601,9602],[9603,0.5347999999999999,370,9601,9602],[9204,0.6,370,9601,9602],[9604,0.6785,370,9601,9602],[9605,0.4978,370,9601,9602],[9606,0.517,370,9601,9602]]},{"id":"sweserve-pass1-2026-paper","name":"SWE-Serve · inference engineering (2026 paper)","group":"编程","metric":"Mean task pass@1","unit":"%","public_history":true,"protocol":{"en":"Eleven models solve the same 53 production-inference tasks using mini-SWE-agent, with three runs per task. Hidden CPU/H100 functional and regression tests are qualified against no-op and oracle controls. The table reports each model’s effort with the highest observed mean; selection uses the evaluation results and is explicit.","zh":"十一个模型使用mini-SWE-agent解答同一组53个生产推理工程任务，每题运行三次。CPU/H100隐藏功能与回归测试通过无操作和参考解答对照检验。原表为各模型选择在评测中观察到的最高均值推理档位，保留该选择条件。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.26777v1","published_at":"2026-09-22","status":"paper_baselines"},"model_count":11,"result_count":11,"results":[[755,0.75,370,9607,9608],[762,0.75,370,9607,9608],[3568,0.64,370,9609,9608],[2977,0.64,370,9607,9608],[3408,0.64,370,9607,9608],[2973,0.64,370,9607,9608],[9610,0.55,370,9607,9608],[227,0.48,370,9607,9608],[3405,0.48,370,9611,9608],[5293,0.46,370,9607,9608],[9612,0.35,370,9609,9608]]},{"id":"avremove-human-composite-2026-paper","name":"AV-Remove-Bench · audio editing (2026 paper)","group":"生成与创作","metric":"Human audio composite","unit":"score","public_history":true,"protocol":{"en":"Seven audio-editing systems are assessed by ten professional audio-editing raters blinded to model identity. The composite weights target removal and background preservation at 35% each, and temporal naturalness and overall quality at 15% each. It remains a native human-rating score.","zh":"十名具有专业音频编辑经验的评分者在不知道模型身份的条件下评价七个音频编辑系统。综合分对目标移除与背景保留各赋35%权重，对时间自然度与整体质量各赋15%权重，保留原生人工量表。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.25864v1","published_at":"2026-09-22","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":7,"result_count":7,"results":[[9613,39.37,370,9614,9615],[9616,37.2,370,9614,9615],[9617,53.36,370,9614,9615],[9618,30.75,370,9614,9615],[9619,62.0,370,9614,9615],[9620,85.72,370,9614,9615],[9621,93.8,370,9614,9615]]},{"id":"sambagraph-response-2026-paper","name":"SambaGraph · response classification (2026 paper)","group":"预测与决策","metric":"Macro-F1","unit":"score","public_history":true,"protocol":{"en":"Six supervised and language-model baselines classify observed soccer responses on match-disjoint splits of 4,070 episodes from the 2022 World Cup. Inputs include early post-anchor motion, so this is response classification rather than pre-action forecasting. Three-seed macro-F1 is preserved.","zh":"六个监督模型及语言模型在2022年世界杯4,070个片段的比赛隔离划分上识别观察到的足球战术反应。输入含动作开始后的早期运动，因此成绩表示反应分类，不能当作动作前预测。保留三个种子的Macro-F1。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.25569v1","published_at":"2026-09-22","status":"paper_baselines"},"score_max":1,"saturation_excluded":true,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"score_decimals":2,"model_count":6,"result_count":6,"results":[[9622,0.796,370,9623,9624],[9625,0.784,370,9626,9624],[9627,0.495,370,9628,9624],[797,0.451,370,9629,9624],[587,0.424,370,9629,9624],[613,0.398,370,9629,9624]]},{"id":"fwbench-loss-cost-2026-paper","name":"FWBench · forecast decisions (2026 paper)","group":"预测与决策","metric":"Penalized loss–cost objective","unit":"raw","public_history":true,"protocol":{"en":"Six distinct language models choose forecasts and capacity plans for the same 1,251 electricity and cycle-hire cases. The published S objective averages normalized excess loss and forecast cost using fixed cohort weights. Invalid submissions retain their penalties; S_v over differing valid subsets is not used. Lower is better.","zh":"六个不同语言模型在同一组1,251个电力和自行车租赁案例中选择预测工具与容量计划。采用固定分组权重下归一化超额损失与预测成本的S目标，无效提交保留罚分，不采用按不同有效子集计算的S_v。数值越低越好。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.27385v1","published_at":"2026-09-23","status":"paper_baselines"},"higher_is_better":false,"model_count":6,"result_count":10,"results":[[2627,4.6294,370,9630,9631],[6267,4.5478,370,9630,9631],[6267,2.5232,370,3143,9631],[2627,1.0465,370,3143,9631],[7190,17.9773,370,9630,9631],[7190,16.4632,370,3143,9631],[6585,10.6358,370,3143,9631],[6585,7.3281,370,9630,9631],[7626,2.6411,370,9630,9631],[2958,0.3964,370,9630,9631]]},{"id":"metricbench-full-rmse-2026-paper","name":"Metric-Bench · full-set metric estimation (2026 paper)","group":"视觉与视频","metric":"RMSE (meters)","unit":"raw","public_history":true,"protocol":{"en":"Fourteen full-set model configurations estimate object size, position, depth and distance in metric units from images. The selected score is RMSE in meters; the separate 400-query proprietary and human mini-set is excluded. Fine-tuned specialist models remain explicitly named checkpoints. Lower is better.","zh":"十四个完整数据集模型配置根据图像估计物体尺寸、位置、深度与距离。采用米为单位的RMSE；排除仅使用400题子集的商业模型与人类对照，专门微调模型保留其检查点名称。数值越低越好。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.25841v1","published_at":"2026-09-22","status":"paper_baselines"},"higher_is_better":false,"model_count":14,"result_count":14,"results":[[9632,1.275,370,9633,9634],[7972,0.9367,370,9633,9634],[6344,1.0776,370,9633,9634],[2687,0.8256,370,9633,9634],[2654,0.8637,370,9633,9634],[6345,0.94,370,9633,9634],[2754,0.7404,370,9633,9634],[6391,0.8647,370,9633,9634],[6342,0.7394,370,9633,9634],[6343,0.7331,370,9633,9634],[9591,0.721,370,9633,9634],[9635,0.7463,370,9633,9634],[9636,0.8983,370,9633,9634],[9637,0.5323,370,9633,9634]]},{"id":"motionforge-in-domain-2026-paper","name":"MotionForge · in-domain dynamic manipulation (2026 paper)","group":"Agent 与工具","metric":"Overall task success","unit":"%","public_history":true,"protocol":{"en":"Seven task-specific, generalist and world-model policies execute the same dynamic-manipulation suite in real time. The selected Overall All column pools long- and short-horizon tasks across four scene families. The environment keeps evolving during inference; frozen-world diagnostics and OOD conditions are separate.","zh":"七个专用、通用及世界模型策略在相同动态操作套件中实时执行任务。采用四类场景、长短时程合并的Overall All成功率。环境在模型推理期间继续演化，冻结环境诊断和分布外条件另行区分。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.25689v1","published_at":"2026-09-22","status":"paper_baselines"},"model_count":7,"result_count":7,"results":[[9638,0.046,370,9639,9640],[9641,0.222,370,9639,9640],[9642,0.0695,370,9639,9640],[9643,0.131,370,9639,9640],[9644,0.048499999999999995,370,9639,9640],[9645,0.1155,370,9639,9640],[9646,0.0115,370,9639,9640]]},{"id":"eadc-compliance-2026-paper","name":"EADC · advanced compliance (2026 paper)","group":"安全与可靠性","metric":"Compliance rubric score","unit":"score","public_history":true,"protocol":{"en":"Twenty-four models answer the same compliance scenarios. The judge checks required content, forbidden content and explicit identification of the request-specific risk; the published Overall composite remains a native rubric score. Human controls are excluded.","zh":"二十四个模型回答相同合规情境，裁判检查必需内容、禁止内容以及是否明确识别该请求的具体风险。保留原表Overall量表综合分，人类对照不进入排行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.26175v1","published_at":"2026-09-22","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":24,"result_count":24,"results":[[3581,79.34,370,9647,9648],[5101,78.5,370,9647,9648],[3568,70.23,370,9647,9648],[3479,72.91,370,9647,9648],[3391,51.95,370,9647,9648],[3493,43.25,370,9647,9648],[3412,49.04,370,9647,9648],[3415,39.26,370,9647,9648],[9649,55.39,370,9647,9648],[3654,53.47,370,9647,9648],[3489,42.49,370,9647,9648],[3490,55.51,370,9647,9648],[244,55.49,370,9647,9648],[3492,32.86,370,9647,9648],[3261,36.11,370,9647,9648],[9650,31.09,370,9647,9648],[362,21.54,370,9647,9648],[363,19.22,370,9647,9648],[3613,32.09,370,9647,9648],[9651,16.49,370,9647,9648],[9652,59.32,370,9647,9648],[9653,34.63,370,9647,9648],[9654,40.84,370,9647,9648],[9655,36.72,370,9647,9648]]},{"id":"qc-stark-overall-2026-paper","name":"QC-Stark · quantum programming (2026 paper)","group":"数学与科学","metric":"Execution accuracy","unit":"%","public_history":true,"protocol":{"en":"Ten models generate Qiskit solve functions for eleven automatically verified quantum-computing tasks, five levels and five seeds (275 attempts per model). All provider errors were retried successfully; output-cap truncations and verifier timeouts remain measured failures. Values retain the published rounded fractional overall means.","zh":"十个模型为十一类可自动验证的量子计算任务生成Qiskit函数，每类五档难度、五个种子，每模型275次。服务错误重试后全部成功，输出上限截断及验证超时保留为实际失败。采用原表经舍入的总体比例均值。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.35581v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":10,"result_count":10,"results":[[3568,0.662,370,9656,9657],[3415,0.651,370,9656,9657],[5929,0.593,370,9656,9657],[3184,0.535,370,9656,9657],[5480,0.491,370,9656,9657],[5406,0.393,370,9656,9657],[4037,0.382,370,9656,9657],[9658,0.367,370,9656,9657],[3338,0.262,370,9656,9657],[5648,0.12,370,9656,9657]]},{"id":"rideway-success-2026-paper","name":"RideWay · success (2026 paper)","group":"Agent 与工具","metric":"Task completion success","unit":"%","public_history":true,"protocol":{"en":"Twenty-four agents use the same stateful ridehailing environment with three trials per task. Success is averaged over all 58 tasks under the final judge-panel gate.","zh":"二十四个Agent使用相同有状态网约车环境，每题运行三次。采用最终裁判组门控下全部58题的平均成功率。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.17985v1","published_at":"2026-09-16","status":"paper_baselines"},"model_count":24,"result_count":24,"results":[[3196,0.753,370,9659,9660],[3202,0.741,370,9661,9660],[3763,0.77,370,9661,9660],[3568,0.828,370,9661,9660],[5480,0.759,370,9661,9660],[4961,0.8220000000000001,370,9661,9660],[227,0.787,370,9661,9660],[3189,0.718,370,9661,9660],[2261,0.816,370,9661,9660],[9662,0.8740000000000001,370,9661,9660],[3187,0.615,370,9661,9660],[3185,0.81,370,9661,9660],[8655,0.603,370,9661,9660],[2129,0.8220000000000001,370,9661,9660],[4070,0.5920000000000001,370,9661,9660],[3764,0.563,370,9661,9660],[6682,0.379,370,9661,9660],[6404,0.345,370,9661,9660],[6687,0.44799999999999995,370,9661,9660],[6920,0.33899999999999997,370,9661,9660],[9663,0.126,370,9661,9660],[516,0.057,370,9661,9660],[562,0.023,370,9661,9660],[9664,0.052000000000000005,370,9661,9660]]},{"id":"rideway-utility-2026-paper","name":"RideWay · utility (2026 paper)","group":"Agent 与工具","metric":"Efficiency Utility on held-out tasks","unit":"score","public_history":true,"protocol":{"en":"Twenty-four agents use the same stateful ridehailing environment with three trials per task. Efficiency Utility is success-gated and discounts excess tool calls and turns on the 14 held-out tasks. It remains a native 0–1 utility score; calibration tasks are excluded.","zh":"二十四个Agent使用相同有状态网约车环境，每题运行三次。效率效用仅对成功运行计分，并按多余工具调用和对话轮次折减；采用14个留出任务的0–1原生效用，排除标定题。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.17985v1","published_at":"2026-09-16","status":"paper_baselines"},"score_max":1,"saturation_excluded":true,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":24,"result_count":24,"results":[[3196,0.705,370,9659,9660],[3202,0.661,370,9661,9660],[3763,0.638,370,9661,9660],[3568,0.637,370,9661,9660],[5480,0.628,370,9661,9660],[4961,0.624,370,9661,9660],[227,0.62,370,9661,9660],[3189,0.612,370,9661,9660],[2261,0.606,370,9661,9660],[9662,0.59,370,9661,9660],[3187,0.563,370,9661,9660],[3185,0.543,370,9661,9660],[8655,0.493,370,9661,9660],[2129,0.443,370,9661,9660],[4070,0.431,370,9661,9660],[3764,0.323,370,9661,9660],[6682,0.311,370,9661,9660],[6404,0.291,370,9661,9660],[6687,0.27,370,9661,9660],[6920,0.229,370,9661,9660],[9663,0.09,370,9661,9660],[516,0.051,370,9661,9660],[562,0.003,370,9661,9660],[9664,0.0,370,9661,9660]]},{"id":"adaptarena-no-profile-2026-paper","name":"AdaptArena · no-profile (2026 paper)","group":"Agent 与工具","metric":"Task success","unit":"%","public_history":true,"protocol":{"en":"Six vision-language backbones perform the same 480 preference-dependent web tasks. The no-profile condition receives no user-history or gold-profile information. Reconstructed-profile and gold-profile oracle conditions are excluded from this cohort.","zh":"六个视觉语言模型执行相同的480个依赖用户偏好的网页任务。无档案条件不接收用户历史或真实偏好。重建档案及真实档案上界条件不混入本组。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.36488v1","published_at":"2026-09-29","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[6130,0.0771,370,6921,9665],[6133,0.0917,370,6921,9665],[3184,0.08960000000000001,370,6921,9665],[4064,0.07919999999999999,370,6921,9665],[9666,0.08960000000000001,370,6921,9665],[9667,0.0729,370,6921,9665]]},{"id":"adaptarena-adaptive-agent-2026-paper","name":"AdaptArena · adaptive-agent (2026 paper)","group":"Agent 与工具","metric":"Task success","unit":"%","public_history":true,"protocol":{"en":"Six vision-language backbones perform the same 480 preference-dependent web tasks. AdaptiveAgent retrieves past interaction trajectories and infers preferences without gold profiles. Historical trajectories were generated using Qwen-3-VL-32B-Thinking; this shared-family collection condition is retained. Reconstructed-profile and gold-profile oracle conditions are excluded from this cohort.","zh":"六个视觉语言模型执行相同的480个依赖用户偏好的网页任务。AdaptiveAgent检索历史交互轨迹并推断偏好，不接收真实偏好档案。历史轨迹由Qwen-3-VL-32B-Thinking生成，保留这一同家族采集条件。重建档案及真实档案上界条件不混入本组。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.36488v1","published_at":"2026-09-29","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[6130,0.1562,370,9668,9665],[6133,0.15,370,9668,9665],[3184,0.12710000000000002,370,9668,9665],[4064,0.1167,370,9668,9665],[9666,0.1104,370,9668,9665],[9667,0.1042,370,9668,9665]]},{"id":"mvvbench-overall-2026-paper","name":"MVVBench · multi-view video (2026 paper)","group":"视觉与视频","metric":"Overall accuracy","unit":"%","public_history":true,"protocol":{"en":"Thirty-three vision-language models answer the same 1,323 human-verified multi-view questions, including 1,091 dynamic questions. Overall accuracy uses the original mixture of binary and four-option questions; the table’s random baseline is 35.81%. Human controls and gain-over-random values are excluded.","zh":"三十三个视觉语言模型回答相同的1,323个人工核验多视角问题，其中1,091题涉及动态信息。总体准确率保留二选与四选题混合口径，原表随机基线为35.81%。不将人类对照及相对随机增益列作模型成绩。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.30952v1","published_at":"2026-09-25","status":"paper_baselines"},"model_count":33,"result_count":33,"results":[[6298,0.4278,370,9669,9670],[6342,0.4278,370,9669,9670],[6343,0.4324,370,9669,9670],[4969,0.4021,370,9669,9670],[4070,0.4097,370,9669,9670],[2584,0.42710000000000004,370,9669,9670],[2590,0.4255,370,9669,9670],[2687,0.4656,370,9669,9670],[2654,0.4596,370,9669,9670],[6345,0.44439999999999996,370,9669,9670],[2642,0.46259999999999996,370,9669,9670],[6866,0.37939999999999996,370,9669,9670],[6865,0.4036,370,9669,9670],[6864,0.4218,370,9669,9670],[9671,0.4263,370,9669,9670],[9672,0.5034000000000001,370,9669,9670],[6349,0.3968,370,9669,9670],[9673,0.34240000000000004,370,9669,9670],[7566,0.39759999999999995,370,9669,9670],[9674,0.4868,370,9669,9670],[8867,0.40509999999999996,370,9669,9670],[7480,0.4815,370,9669,9670],[7801,0.5351,370,9669,9670],[9675,0.4966,370,9669,9670],[9676,0.4278,370,9669,9670],[9677,0.5019,370,9669,9670],[5406,0.45649999999999996,370,9669,9670],[345,0.4913,370,9669,9670],[4080,0.47009999999999996,370,9669,9670],[4064,0.542,370,9669,9670],[340,0.45649999999999996,370,9669,9670],[4958,0.4429,370,9669,9670],[3184,0.4974,370,9669,9670]]},{"id":"flapkad-forward-h1-2026-paper","name":"FlapKAD · forward H=1 (2026 paper)","group":"数学与科学","metric":"Normalized target MSE","unit":"raw","public_history":true,"protocol":{"en":"Eight independently trained architectures use the same 1,600/200/200 episode split, 256-step history and H=1 forward prediction. Splitting precedes window construction; normalization uses training observations only. MSE is measured in normalized target space and is not a physical-unit error. Horizons have different admissible windows and remain separate cohorts. Lower is better.","zh":"八个分别训练的架构采用相同的1,600/200/200回合划分、256步历史及H=1的正向预测。先划分回合再生成时间窗口，归一化仅使用训练观测。MSE保留标准化目标空间单位；不同预测跨度的可用窗口不同，分别排行。数值越低越好。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.38719v1","published_at":"2026-09-30","status":"paper_baselines"},"higher_is_better":false,"model_count":8,"result_count":8,"results":[[9464,0.85,370,9678,9679],[9680,0.852,370,9678,9679],[9204,0.741,370,9678,9679],[9681,0.734,370,9678,9679],[9161,0.754,370,9678,9679],[9466,0.764,370,9678,9679],[9682,0.765,370,9678,9679],[9683,0.796,370,9678,9679]]},{"id":"flapkad-forward-h16-2026-paper","name":"FlapKAD · forward H=16 (2026 paper)","group":"数学与科学","metric":"Normalized target MSE","unit":"raw","public_history":true,"protocol":{"en":"Eight independently trained architectures use the same 1,600/200/200 episode split, 256-step history and H=16 forward prediction. Splitting precedes window construction; normalization uses training observations only. MSE is measured in normalized target space and is not a physical-unit error. Horizons have different admissible windows and remain separate cohorts. Lower is better.","zh":"八个分别训练的架构采用相同的1,600/200/200回合划分、256步历史及H=16的正向预测。先划分回合再生成时间窗口，归一化仅使用训练观测。MSE保留标准化目标空间单位；不同预测跨度的可用窗口不同，分别排行。数值越低越好。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.38719v1","published_at":"2026-09-30","status":"paper_baselines"},"higher_is_better":false,"model_count":8,"result_count":8,"results":[[9464,0.861,370,9684,9679],[9680,0.862,370,9684,9679],[9204,0.703,370,9684,9679],[9681,0.703,370,9684,9679],[9161,0.718,370,9684,9679],[9466,0.724,370,9684,9679],[9682,0.719,370,9684,9679],[9683,0.787,370,9684,9679]]},{"id":"flapkad-forward-h32-2026-paper","name":"FlapKAD · forward H=32 (2026 paper)","group":"数学与科学","metric":"Normalized target MSE","unit":"raw","public_history":true,"protocol":{"en":"Eight independently trained architectures use the same 1,600/200/200 episode split, 256-step history and H=32 forward prediction. Splitting precedes window construction; normalization uses training observations only. MSE is measured in normalized target space and is not a physical-unit error. Horizons have different admissible windows and remain separate cohorts. Lower is better.","zh":"八个分别训练的架构采用相同的1,600/200/200回合划分、256步历史及H=32的正向预测。先划分回合再生成时间窗口，归一化仅使用训练观测。MSE保留标准化目标空间单位；不同预测跨度的可用窗口不同，分别排行。数值越低越好。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.38719v1","published_at":"2026-09-30","status":"paper_baselines"},"higher_is_better":false,"model_count":8,"result_count":8,"results":[[9464,0.808,370,9685,9679],[9680,0.808,370,9685,9679],[9204,0.668,370,9685,9679],[9681,0.649,370,9685,9679],[9161,0.656,370,9685,9679],[9466,0.666,370,9685,9679],[9682,0.662,370,9685,9679],[9683,0.775,370,9685,9679]]},{"id":"flapkad-inverse-h1-2026-paper","name":"FlapKAD · inverse H=1 (2026 paper)","group":"数学与科学","metric":"Normalized target MSE","unit":"raw","public_history":true,"protocol":{"en":"Eight independently trained architectures use the same 1,600/200/200 episode split, 256-step history and H=1 inverse prediction. Splitting precedes window construction; normalization uses training observations only. MSE is measured in normalized target space and is not a physical-unit error. Horizons have different admissible windows and remain separate cohorts. Lower is better.","zh":"八个分别训练的架构采用相同的1,600/200/200回合划分、256步历史及H=1的逆向预测。先划分回合再生成时间窗口，归一化仅使用训练观测。MSE保留标准化目标空间单位；不同预测跨度的可用窗口不同，分别排行。数值越低越好。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.38719v1","published_at":"2026-09-30","status":"paper_baselines"},"higher_is_better":false,"model_count":8,"result_count":8,"results":[[9464,0.628,370,9678,9679],[9680,0.633,370,9678,9679],[9204,0.052,370,9678,9679],[9681,0.078,370,9678,9679],[9161,0.087,370,9678,9679],[9466,0.111,370,9678,9679],[9682,0.12,370,9678,9679],[9683,0.339,370,9678,9679]]},{"id":"flapkad-inverse-h16-2026-paper","name":"FlapKAD · inverse H=16 (2026 paper)","group":"数学与科学","metric":"Normalized target MSE","unit":"raw","public_history":true,"protocol":{"en":"Eight independently trained architectures use the same 1,600/200/200 episode split, 256-step history and H=16 inverse prediction. Splitting precedes window construction; normalization uses training observations only. MSE is measured in normalized target space and is not a physical-unit error. Horizons have different admissible windows and remain separate cohorts. Lower is better.","zh":"八个分别训练的架构采用相同的1,600/200/200回合划分、256步历史及H=16的逆向预测。先划分回合再生成时间窗口，归一化仅使用训练观测。MSE保留标准化目标空间单位；不同预测跨度的可用窗口不同，分别排行。数值越低越好。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.38719v1","published_at":"2026-09-30","status":"paper_baselines"},"higher_is_better":false,"model_count":8,"result_count":8,"results":[[9464,0.688,370,9684,9679],[9680,0.693,370,9684,9679],[9204,0.1,370,9684,9679],[9681,0.079,370,9684,9679],[9161,0.086,370,9684,9679],[9466,0.115,370,9684,9679],[9682,0.101,370,9684,9679],[9683,0.433,370,9684,9679]]},{"id":"flapkad-inverse-h32-2026-paper","name":"FlapKAD · inverse H=32 (2026 paper)","group":"数学与科学","metric":"Normalized target MSE","unit":"raw","public_history":true,"protocol":{"en":"Eight independently trained architectures use the same 1,600/200/200 episode split, 256-step history and H=32 inverse prediction. Splitting precedes window construction; normalization uses training observations only. MSE is measured in normalized target space and is not a physical-unit error. Horizons have different admissible windows and remain separate cohorts. Lower is better.","zh":"八个分别训练的架构采用相同的1,600/200/200回合划分、256步历史及H=32的逆向预测。先划分回合再生成时间窗口，归一化仅使用训练观测。MSE保留标准化目标空间单位；不同预测跨度的可用窗口不同，分别排行。数值越低越好。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.38719v1","published_at":"2026-09-30","status":"paper_baselines"},"higher_is_better":false,"model_count":8,"result_count":8,"results":[[9464,0.739,370,9685,9679],[9680,0.742,370,9685,9679],[9204,0.234,370,9685,9679],[9681,0.094,370,9685,9679],[9161,0.078,370,9685,9679],[9466,0.113,370,9685,9679],[9682,0.101,370,9685,9679],[9683,0.576,370,9685,9679]]},{"id":"finixdocbench-digital-2026-paper","name":"FinixDocBench · digital (2026 paper)","group":"视觉与视频","metric":"Document parsing score (S_FDB)","unit":"score","public_history":true,"protocol":{"en":"The complete twelve-system panel uses the same financial-document track, with fixed recognition, table and reading-order metrics. The source’s overall document-parsing score is retained on its native 0–100 scale. Digital and camera-captured tracks stay separate. The released compliance-reviewed subset is smaller than the full roughly 5,000-page paper benchmark; these scores describe the original paper cohort.","zh":"完整十二个系统在同一金融文档子集上评测，使用固定文本识别、表格和阅读顺序指标。保留原生0–100总体解析分数，数字原稿与拍摄文档分开。公开合规子集小于论文约5,000页完整评测集；这些分数对应论文原始面板。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.22842v1","status":"paper_baselines","published_at":"2026-08-24"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":12,"result_count":12,"results":[[6298,80.18,370,9686,9687],[9688,93.19,370,9686,9687],[9689,82.8,370,9686,9687],[9690,83.1,370,9686,9687],[9691,85.41,370,9686,9687],[9692,86.28,370,9686,9687],[9693,89.26,370,9686,9687],[9694,90.36,370,9686,9687],[9695,92.96,370,9686,9687],[2576,84.9,370,9686,9687],[2998,85.05,370,9686,9687],[2613,87.26,370,9686,9687]]},{"id":"finixdocbench-photo-2026-paper","name":"FinixDocBench · photo (2026 paper)","group":"视觉与视频","metric":"Document parsing score (S_FDB)","unit":"score","public_history":true,"protocol":{"en":"The complete twelve-system panel uses the same financial-document track, with fixed recognition, table and reading-order metrics. The source’s overall document-parsing score is retained on its native 0–100 scale. Digital and camera-captured tracks stay separate. The released compliance-reviewed subset is smaller than the full roughly 5,000-page paper benchmark; these scores describe the original paper cohort.","zh":"完整十二个系统在同一金融文档子集上评测，使用固定文本识别、表格和阅读顺序指标。保留原生0–100总体解析分数，数字原稿与拍摄文档分开。公开合规子集小于论文约5,000页完整评测集；这些分数对应论文原始面板。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.22842v1","status":"paper_baselines","published_at":"2026-08-24"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":12,"result_count":12,"results":[[6298,54.28,370,9696,9697],[9688,67.03,370,9696,9697],[9691,41.28,370,9696,9697],[9695,43.08,370,9696,9697],[9689,43.2,370,9696,9697],[9692,45.82,370,9696,9697],[9690,47.2,370,9696,9697],[9694,52.57,370,9696,9697],[9693,60.9,370,9696,9697],[2576,62.58,370,9696,9697],[2613,62.65,370,9696,9697],[2998,65.55,370,9696,9697]]},{"id":"finriskatlas-knowledge-2026-paper","name":"FinRiskAtlas · domain knowledge (2026 paper)","group":"专业工作","metric":"Domain knowledge macro accuracy","unit":"%","public_history":true,"protocol":{"en":"All 33 archived model configurations use the fixed Domain Knowledge contracts across five professional financial domains. The source’s overall macro score is preserved, rather than mixing the separate evidence-grounded, applied-review or evidence-acquisition operations. Exact paper model names and batch remain attached.","zh":"完整33个归档模型配置采用五类金融领域的固定知识任务协议。保留论文总体宏平均分数，不混入证据处理、专业审阅和证据获取等独立任务。保留原始模型名与论文批次。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.25325v1","status":"paper_baselines","published_at":"2026-08-26"},"model_count":33,"result_count":33,"results":[[9698,0.65,370,9699,9700],[9701,0.6893,370,9699,9700],[9702,0.7417,370,9699,9700],[9703,0.7765000000000001,370,9699,9700],[7854,0.7562000000000001,370,9699,9700],[9492,0.7701,370,9699,9700],[9704,0.5038,370,9699,9700],[8657,0.6743000000000001,370,9699,9700],[8654,0.7495999999999999,370,9699,9700],[797,0.6342,370,9699,9700],[7590,0.4021,370,9699,9700],[5126,0.6332,370,9699,9700],[5344,0.6766,370,9699,9700],[6301,0.7048000000000001,370,9699,9700],[786,0.7379000000000001,370,9699,9700],[9705,0.7719,370,9699,9700],[784,0.769,370,9699,9700],[3202,0.8040999999999999,370,9699,9700],[1578,0.7356999999999999,370,9699,9700],[3201,0.7408,370,9699,9700],[3185,0.7553,370,9699,9700],[2998,0.7856000000000001,370,9699,9700],[3155,0.8068000000000001,370,9699,9700],[268,0.7626000000000001,370,9699,9700],[227,0.7713,370,9699,9700],[6235,0.7742,370,9699,9700],[3184,0.7866,370,9699,9700],[5616,0.7526999999999999,370,9699,9700],[5727,0.7834,370,9699,9700],[4979,0.7920999999999999,370,9699,9700],[766,0.7456999999999999,370,9699,9700],[9706,0.8069,370,9699,9700],[3181,0.8019,370,9699,9700]]},{"id":"hounsbench-m3d-closed-2026-paper","name":"HounsBench · M3D closed-form (2026 paper)","group":"医疗","metric":"M3D closed-form accuracy","unit":"%","public_history":true,"protocol":{"en":"All nine models use the same patient-disjoint HounsBench readout split and task-specific scoring. The selected closed-form or multiple-choice task is shown separately, avoiding the mixed-metric Avg. column and simulation metrics. Model sizes are retained in configuration.","zh":"完整九个模型使用同一患者隔离的HounsBench读出测试划分及任务专属评分。闭合答案或选择题任务单独展示，不采用混合指标Avg.，也不混入仿真指标。模型规模保留在配置中。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.12904v1","status":"paper_baselines","published_at":"2026-08-13"},"model_count":9,"result_count":9,"results":[[9707,0.5528,370,9708,9709],[4958,0.6132,370,9710,9709],[5115,0.675,370,9710,9709],[9711,0.615,370,9712,9709],[8532,0.0962,370,9713,9709],[9714,0.7066,370,9708,9709],[9715,0.524,370,9716,9709],[9717,0.8190000000000001,370,9718,9709],[9719,0.7615999999999999,370,9718,9709]]},{"id":"hounsbench-ctrate-choice-2026-paper","name":"HounsBench · CT-RATE choice (2026 paper)","group":"医疗","metric":"CT-RATE choice accuracy","unit":"%","public_history":true,"protocol":{"en":"All nine models use the same patient-disjoint HounsBench readout split and task-specific scoring. The selected closed-form or multiple-choice task is shown separately, avoiding the mixed-metric Avg. column and simulation metrics. Model sizes are retained in configuration.","zh":"完整九个模型使用同一患者隔离的HounsBench读出测试划分及任务专属评分。闭合答案或选择题任务单独展示，不采用混合指标Avg.，也不混入仿真指标。模型规模保留在配置中。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.12904v1","status":"paper_baselines","published_at":"2026-08-13"},"model_count":9,"result_count":9,"results":[[9707,0.5251,370,9708,9709],[4958,0.6444,370,9710,9709],[5115,0.6581999999999999,370,9710,9709],[9711,0.7061,370,9712,9709],[8532,0.18530000000000002,370,9713,9709],[9714,0.562,370,9708,9709],[9715,0.8753,370,9716,9709],[9717,0.8761,370,9718,9709],[9719,0.8251999999999999,370,9718,9709]]},{"id":"hugvis-audio-video-csim-2026-paper","name":"HUG-VIS · audio-driven video (2026 paper)","group":"生成与创作","metric":"ArcFace identity cosine similarity (CSIM)","unit":"score","public_history":true,"protocol":{"en":"All eight audio-driven video-generation systems use the same HUG-VIS identity images, synchronized audio and reference videos under the unified zero-shot protocol. ArcFace cosine identity similarity remains a native 0–1 metric. Lip synchronization, subjective ratings and other driving-signal regimes are separate.","zh":"完整八个音频驱动视频生成系统在统一零样本协议下，采用同一HUG-VIS身份图像、同步音频与参考视频。ArcFace身份余弦相似度保留0–1原生量纲，与唇音同步、主观评分及其他驱动条件分开。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.26517v1","status":"paper_baselines","published_at":"2026-08-27"},"score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":8,"result_count":8,"results":[[9720,0.71,370,9721,9722],[9723,0.786,370,9721,9722],[9724,0.771,370,9721,9722],[9725,0.744,370,9721,9722],[9726,0.868,370,9721,9722],[9727,0.904,370,9721,9722],[9728,0.736,370,9721,9722],[9729,0.798,370,9721,9722]]},{"id":"libero-vifo-fullchain-2026-paper","name":"LIBERO-VIFO · visual cue chain (2026 paper)","group":"Agent 与工具","metric":"Full-chain accuracy","unit":"%","public_history":true,"protocol":{"en":"All seven vision-language-action architectures answer the same four multiple-choice stages for each visual-cue instance: detection, semantics, grounding and action decision. Full-chain accuracy requires all four answers correct. This image-based protocol stays separate from closed-loop manipulation, conflict behavior and original LIBERO success rates.","zh":"完整七种视觉语言动作架构回答同一视觉提示实例的四阶段选择题：检测、语义、定位与动作判断。只有四项全部正确才算全链正确。该图像问答协议与闭环操纵、冲突行为和原始LIBERO成功率分开。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.17600v1","status":"paper_baselines","published_at":"2026-08-18"},"model_count":7,"result_count":7,"results":[[7594,0.001,370,9730,9731],[9732,0.012,370,9730,9731],[7397,0.166,370,9730,9731],[9733,0.019,370,9730,9731],[9734,0.17600000000000002,370,9730,9731],[7396,0.115,370,9730,9731],[9735,0.084,370,9730,9731]]},{"id":"letzcross-ndcg-2026-paper","name":"LëtzCross · cross-lingual page retrieval (2026 paper)","group":"知识与推理","metric":"Average nDCG@10","unit":"score","public_history":true,"protocol":{"en":"All nine text-only and page-image retrieval models use the same Luxembourgish document pages and English, French, German and Luxembourgish queries. The paper’s average nDCG@10 is preserved on its native 0–100 display scale; index and query latencies are not mixed into the score.","zh":"完整九个文本或页面图像检索模型使用同一卢森堡语PDF页面，以及英语、法语、德语和卢森堡语查询。保留论文原生0–100显示量纲的平均nDCG@10，不混入索引或查询延迟。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.21714v1","status":"paper_baselines","published_at":"2026-08-22"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":9,"result_count":9,"results":[[8053,73.18,370,9736,9737],[9738,71.96,370,9736,9737],[8042,74.14,370,9736,9737],[9739,75.76,370,9736,9737],[9740,59.73,370,9736,9737],[9741,67.81,370,9736,9737],[9742,77.18,370,9736,9737],[9743,78.83,370,9736,9737],[9744,78.66,370,9736,9737]]},{"id":"mccxr-image-only-2026-paper","name":"MC-CXR · image only (2026 paper)","group":"医疗","metric":"Absolute finding accuracy","unit":"%","public_history":true,"protocol":{"en":"All ten VLMs evaluate the same fixed chest-X-ray cases and target findings under the selected paired context condition. The source benchmark contains 240 cases and 2,522 context-expanded instances; image-only, reliable-note and misleading-note conditions are kept separate. This is absolute accuracy, not the switch-conditioned alignment measure.","zh":"完整十个视觉语言模型在所选成对上下文条件下评测固定胸片与目标病征。原基准有240个病例、扩展为2,522个上下文实例；仅图像、可靠病历与误导病历条件分别展示。采用绝对准确率，不使用切换条件下的对齐指标。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.24118v1","status":"paper_baselines","published_at":"2026-08-25"},"model_count":10,"result_count":10,"results":[[2728,0.267,370,9745,9746],[4070,0.3,370,9745,9746],[5535,0.092,370,9745,9746],[6684,0.083,370,9745,9746],[9747,0.158,370,9745,9746],[4970,0.317,370,9745,9746],[9748,0.358,370,9745,9746],[9749,0.212,370,9745,9746],[3196,0.304,370,9745,9746],[2129,0.28300000000000003,370,9745,9746]]},{"id":"mccxr-reliable-note-2026-paper","name":"MC-CXR · reliable note (2026 paper)","group":"医疗","metric":"Absolute finding accuracy","unit":"%","public_history":true,"protocol":{"en":"All ten VLMs evaluate the same fixed chest-X-ray cases and target findings under the selected paired context condition. The source benchmark contains 240 cases and 2,522 context-expanded instances; image-only, reliable-note and misleading-note conditions are kept separate. This is absolute accuracy, not the switch-conditioned alignment measure.","zh":"完整十个视觉语言模型在所选成对上下文条件下评测固定胸片与目标病征。原基准有240个病例、扩展为2,522个上下文实例；仅图像、可靠病历与误导病历条件分别展示。采用绝对准确率，不使用切换条件下的对齐指标。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.24118v1","status":"paper_baselines","published_at":"2026-08-25"},"model_count":10,"result_count":10,"results":[[2728,0.708,370,9750,9746],[4070,0.688,370,9750,9746],[5535,0.688,370,9750,9746],[6684,0.583,370,9750,9746],[9747,0.662,370,9750,9746],[4970,0.6459999999999999,370,9750,9746],[9748,0.588,370,9750,9746],[9749,0.662,370,9750,9746],[3196,0.708,370,9750,9746],[2129,0.688,370,9750,9746]]},{"id":"mccxr-misleading-note-2026-paper","name":"MC-CXR · misleading note (2026 paper)","group":"医疗","metric":"Absolute finding accuracy","unit":"%","public_history":true,"protocol":{"en":"All ten VLMs evaluate the same fixed chest-X-ray cases and target findings under the selected paired context condition. The source benchmark contains 240 cases and 2,522 context-expanded instances; image-only, reliable-note and misleading-note conditions are kept separate. This is absolute accuracy, not the switch-conditioned alignment measure.","zh":"完整十个视觉语言模型在所选成对上下文条件下评测固定胸片与目标病征。原基准有240个病例、扩展为2,522个上下文实例；仅图像、可靠病历与误导病历条件分别展示。采用绝对准确率，不使用切换条件下的对齐指标。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.24118v1","status":"paper_baselines","published_at":"2026-08-25"},"model_count":10,"result_count":10,"results":[[2728,0.062,370,9751,9746],[4070,0.092,370,9751,9746],[5535,0.046,370,9751,9746],[6684,0.033,370,9751,9746],[9747,0.042,370,9751,9746],[4970,0.121,370,9751,9746],[9748,0.204,370,9751,9746],[9749,0.11699999999999999,370,9751,9746],[3196,0.10800000000000001,370,9751,9746],[2129,0.10400000000000001,370,9751,9746]]},{"id":"medreamm-top1-primary-icd-2026-paper","name":"MedReaMM · primary diagnosis (2026 paper)","group":"医疗","metric":"Top-1 primary diagnosis recall (ICD alignment)","unit":"score","public_history":true,"protocol":{"en":"All 23 LMMs receive the same 625 expert-validated multimodal clinical cases and generate ranked differential diagnoses. The first predicted diagnosis is scored against the primary reference through the paper’s ICD-11 alignment rule. The exact native 0–100 recall metric remains separate from semantic matching, complete-diagnosis recall, MRR and NDCG.","zh":"完整23个模型输入相同625个专家核验的多模态临床病例，生成排序鉴别诊断。首个预测通过论文ICD-11对齐规则与主要诊断比对。保留0–100原生召回量纲，与语义匹配、完整诊断召回、MRR和NDCG分开。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.22323v1","status":"paper_baselines","published_at":"2026-08-23"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":23,"result_count":23,"results":[[2754,17.1,370,9752,9753],[8173,18.6,370,9752,9753],[8524,15.8,370,9752,9753],[6150,25.4,370,9752,9753],[5407,25.9,370,9752,9753],[354,23.0,370,9752,9753],[6391,27.2,370,9752,9753],[8169,27.7,370,9752,9753],[9409,29.3,370,9752,9753],[9754,29.3,370,9752,9753],[9755,29.4,370,9752,9753],[2166,31.0,370,9752,9753],[5406,41.6,370,9752,9753],[353,41.3,370,9752,9753],[345,42.7,370,9752,9753],[3250,24.0,370,9752,9753],[5536,28.5,370,9752,9753],[9756,33.9,370,9752,9753],[4075,42.1,370,9752,9753],[2816,44.0,370,9752,9753],[6863,42.4,370,9752,9753],[766,46.6,370,9752,9753],[764,51.5,370,9752,9753]]},{"id":"formaltcs-lean-proof-2026-paper","name":"FormalTCS · Lean proof (2026 paper)","group":"数学与科学","metric":"Formal theorem-to-proof success","unit":"%","public_history":true,"protocol":{"en":"The full eight-model panel uses the same 143 frontier theoretical-CS theorem instances and the prescribed model harnesses. This task requires a Lean 4.32.2 proof of the supplied formal theorem, checked against the corresponding Mathlib environment. Results are from paper v3; the original August 20 release is verified in arXiv submission history. Scale labels and harnesses are preserved as explicit identities/configuration.","zh":"完整八模型面板使用同一143个前沿理论计算机科学定理与指定Agent框架。该任务为给定形式定理生成Lean 4.32.2证明，由对应Mathlib环境验证。分数来自论文v3；原始8月20日首发已通过arXiv提交历史核验。规模与框架信息保留在模型身份和配置中。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.20153v3","status":"paper_baselines","published_at":"2026-08-20"},"model_count":8,"result_count":8,"results":[[3408,0.14,370,2300,9757],[2973,0.182,370,2300,9757],[762,0.266,370,2300,9757],[325,0.077,370,3759,9757],[3568,0.23800000000000002,370,3759,9757],[755,0.287,370,3759,9757],[3261,0.175,370,5380,9757],[3492,0.21,370,5380,9757]]},{"id":"longwof-common180-no-context-2026-paper","name":"LongWoF-Bench · common 180 tasks (2026 paper)","group":"Agent 与工具","metric":"Workflow success without context","unit":"%","public_history":true,"protocol":{"en":"The complete seven-model appendix panel evaluates the same 180 workflow tasks for which both Opus and Gemini produced verified Genes. The no-context condition measures strict end-to-end verifier success without transferred skills or Genes. This selected common 180-task cohort is distinct from the benchmark’s 778-task inventory and all generated-context conditions. Results come from paper v4; arXiv history confirms original release on August 24.","zh":"附录完整七模型面板评测同一180个Opus和Gemini均生成已验证Gene的工作流任务。无上下文条件衡量不使用迁移技能或Gene时的端到端严格验证成功率。该180任务共同子集与基准778任务总库及所有生成上下文条件分开。结果来自论文v4，arXiv提交历史确认8月24日首发。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.23200v4","status":"paper_baselines","published_at":"2026-08-24"},"model_count":7,"result_count":7,"results":[[9758,0.617,370,9759,9760],[9761,0.5329999999999999,370,9759,9760],[9762,0.456,370,9759,9760],[9763,0.633,370,9759,9760],[3491,0.467,370,9759,9760],[9764,0.42200000000000004,370,9759,9760],[9765,0.41700000000000004,370,9759,9760]]},{"id":"longdocbench-relations-2026-paper","name":"LongDocBench · contextual relationships (2026 paper)","group":"视觉与视频","metric":"Contextual relation macro score","unit":"score","public_history":true,"protocol":{"en":"Seven distinct model checkpoints in eight reasoning configurations receive the same TextIn parsing outputs and 2,680 benchmark-localized table/figure targets from 85 documents. The complete eight-row panel recovers caption, note and source relationships; the native 0–1 macro score is preserved. Thinking settings remain configurations of the same checkpoint.","zh":"七个不同检查点、八种推理配置输入相同TextIn解析结果与85份文档中的2,680个已定位表图目标。完整八行面板恢复标题、注释与来源关系，保留0–1原生宏平均分数。思考设置保留为同一底模的配置。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.15064v1","status":"paper_baselines","published_at":"2026-08-15"},"score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":7,"result_count":8,"results":[[762,0.63,370,9766,9767],[4424,0.44,370,9768,9767],[227,0.56,370,9769,9767],[3490,0.55,370,9770,9767],[3191,0.59,370,9771,9767],[3191,0.6,370,9772,9767],[5619,0.52,370,9773,9767],[4070,0.52,370,9774,9767]]},{"id":"matreplace-track-a-2026-paper","name":"MatReplace · instruction (2026 paper)","group":"生成与创作","metric":"Gated material replacement score","unit":"score","public_history":true,"protocol":{"en":"The full applicable panel uses the same 1,421 bench-v1.0 material-replacement tasks within this conditioning track and one fixed-seed output per task. The primary gated aggregate combines the reference-free material, preservation, structure and harmony rubric with strict material-family validation. Raw and gated scores and the three conditioning tracks remain separate.","zh":"该输入条件下的完整模型面板使用同一1,421个bench-v1.0材质替换任务，每题固定随机种子生成一次。主指标gate将无参考输出的材质、保留、结构和协调性评分与严格材质类别验证结合。原始均值与门控分数、三种输入条件均分别保留。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.24107v1","status":"paper_baselines","published_at":"2026-08-25"},"score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":7,"result_count":7,"results":[[9775,0.76,370,9776,9777],[6368,0.739,370,9776,9777],[8150,0.677,370,9776,9777],[9778,0.576,370,9776,9777],[9779,0.54,370,9776,9777],[7295,0.539,370,9776,9777],[9780,0.344,370,9776,9777]]},{"id":"matreplace-track-b-2026-paper","name":"MatReplace · mask + instruction (2026 paper)","group":"生成与创作","metric":"Gated material replacement score","unit":"score","public_history":true,"protocol":{"en":"The full applicable panel uses the same 1,421 bench-v1.0 material-replacement tasks within this conditioning track and one fixed-seed output per task. The primary gated aggregate combines the reference-free material, preservation, structure and harmony rubric with strict material-family validation. Raw and gated scores and the three conditioning tracks remain separate.","zh":"该输入条件下的完整模型面板使用同一1,421个bench-v1.0材质替换任务，每题固定随机种子生成一次。主指标gate将无参考输出的材质、保留、结构和协调性评分与严格材质类别验证结合。原始均值与门控分数、三种输入条件均分别保留。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.24107v1","status":"paper_baselines","published_at":"2026-08-25"},"score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":9,"result_count":9,"results":[[9775,0.757,370,9781,9777],[6368,0.724,370,9781,9777],[9782,0.68,370,9781,9777],[9783,0.678,370,9781,9777],[9778,0.57,370,9781,9777],[9784,0.528,370,9781,9777],[9785,0.449,370,9781,9777],[9786,0.446,370,9781,9777],[9787,0.33,370,9781,9777]]},{"id":"matreplace-track-c-2026-paper","name":"MatReplace · mask + reference (2026 paper)","group":"生成与创作","metric":"Gated material replacement score","unit":"score","public_history":true,"protocol":{"en":"The full applicable panel uses the same 1,421 bench-v1.0 material-replacement tasks within this conditioning track and one fixed-seed output per task. The primary gated aggregate combines the reference-free material, preservation, structure and harmony rubric with strict material-family validation. Raw and gated scores and the three conditioning tracks remain separate.","zh":"该输入条件下的完整模型面板使用同一1,421个bench-v1.0材质替换任务，每题固定随机种子生成一次。主指标gate将无参考输出的材质、保留、结构和协调性评分与严格材质类别验证结合。原始均值与门控分数、三种输入条件均分别保留。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.24107v1","status":"paper_baselines","published_at":"2026-08-25"},"score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":7,"result_count":7,"results":[[9775,0.726,370,9788,9777],[6368,0.675,370,9788,9777],[9789,0.386,370,9788,9777],[9790,0.322,370,9788,9777],[9791,0.307,370,9788,9777],[9792,0.204,370,9788,9777],[9778,0.062,370,9788,9777]]},{"id":"omniphys-objective-mastery-2026-paper","name":"OmniPhys · objective mastery (2026 paper)","group":"数学与科学","metric":"Perfect-instance objective mastery rate","unit":"%","public_history":true,"protocol":{"en":"The complete 26-model panel uses the same five-domain physics benchmark. Objective mastery requires both answer and key-step process scores to equal one; open-ended mastery requires a perfect derivation score. These strict binary instance rates are shown separately from partial answer/process scores and image-output evaluation.","zh":"完整26模型面板使用相同五领域物理评测。客观题掌握率要求答案与关键步骤分数同时为满分；开放题掌握率要求推导分数为满分。两类严格二元通过率分别展示，不混入答案或过程部分得分及图像输出评测。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.25398v1","status":"paper_baselines","published_at":"2026-08-26"},"model_count":26,"result_count":26,"results":[[340,0.3936,370,9793,9794],[3748,0.2353,370,9793,9794],[353,0.12279999999999999,370,9793,9794],[5929,0.3934,370,9793,9794],[7685,0.2842,370,9793,9794],[6130,0.6981999999999999,370,9793,9794],[766,0.4631,370,9793,9794],[3250,0.18960000000000002,370,9793,9794],[6216,0.6142,370,9793,9794],[9795,0.5212,370,9793,9794],[3247,0.4558,370,9793,9794],[2771,0.0096,370,9793,9794],[2754,0.0674,370,9793,9794],[6391,0.25420000000000004,370,9793,9794],[2166,0.264,370,9793,9794],[6341,0.0682,370,9793,9794],[6342,0.22899999999999998,370,9793,9794],[6343,0.3608,370,9793,9794],[9796,0.47159999999999996,370,9793,9794],[6346,0.056100000000000004,370,9793,9794],[6246,0.030600000000000002,370,9793,9794],[6345,0.1275,370,9793,9794],[2642,0.1332,370,9793,9794],[2921,0.0106,370,9793,9794],[6348,0.0726,370,9793,9794],[6392,0.0053,370,9793,9794]]},{"id":"omniphys-open-mastery-2026-paper","name":"OmniPhys · open-ended derivation mastery (2026 paper)","group":"数学与科学","metric":"Perfect-instance open-ended derivation mastery rate","unit":"%","public_history":true,"protocol":{"en":"The complete 26-model panel uses the same five-domain physics benchmark. Objective mastery requires both answer and key-step process scores to equal one; open-ended mastery requires a perfect derivation score. These strict binary instance rates are shown separately from partial answer/process scores and image-output evaluation.","zh":"完整26模型面板使用相同五领域物理评测。客观题掌握率要求答案与关键步骤分数同时为满分；开放题掌握率要求推导分数为满分。两类严格二元通过率分别展示，不混入答案或过程部分得分及图像输出评测。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.25398v1","status":"paper_baselines","published_at":"2026-08-26"},"model_count":26,"result_count":26,"results":[[340,0.2945,370,9797,9794],[3748,0.12869999999999998,370,9797,9794],[353,0.0461,370,9797,9794],[5929,0.215,370,9797,9794],[7685,0.24420000000000003,370,9797,9794],[6130,0.493,370,9797,9794],[766,0.3589,370,9797,9794],[3250,0.138,370,9797,9794],[6216,0.32799999999999996,370,9797,9794],[9795,0.33140000000000003,370,9797,9794],[3247,0.29600000000000004,370,9797,9794],[2771,0.0011,370,9797,9794],[2754,0.0121,370,9797,9794],[6391,0.1408,370,9797,9794],[2166,0.09609999999999999,370,9797,9794],[6341,0.026099999999999998,370,9797,9794],[6342,0.0793,370,9797,9794],[6343,0.2257,370,9797,9794],[9796,0.2901,370,9797,9794],[6346,0.0079,370,9797,9794],[6246,0.0053,370,9797,9794],[6345,0.0368,370,9797,9794],[2642,0.040999999999999995,370,9797,9794],[2921,0.0014000000000000002,370,9797,9794],[6348,0.0253,370,9797,9794],[6392,0.0014000000000000002,370,9797,9794]]},{"id":"openharmony-task-completion-2026-paper","name":"OpenHarmony Bench · app completion (2026 paper)","group":"编程","metric":"Task completion","unit":"%","public_history":true,"protocol":{"en":"The full eight-model DevEco Code panel runs all 153 app-level tasks under the frozen project, SDK and executable UI-test snapshot. A task is complete only if the app builds and every required functional check passes, including all F-points. The reported mean completion rate is retained with the run range preserved in the source fixture.","zh":"完整八模型DevEco Code面板在固定项目、SDK与可执行UI测试快照下完成153个应用级任务。只有构建成功并通过所有功能检查（含全部F-point）才算任务完成。保留平均完成率，原始运行极差保存在源表fixture。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.16022v1","status":"paper_baselines","published_at":"2026-08-17"},"model_count":8,"result_count":8,"results":[[227,0.5839,370,9798,9799],[3155,0.573,370,9798,9799],[9800,0.5599000000000001,370,9798,9799],[3202,0.5338,370,9798,9799],[3186,0.5229,370,9798,9799],[3203,0.5207,370,9798,9799],[3185,0.5207,370,9798,9799],[3198,0.4836,370,9798,9799]]},{"id":"reflexbench-success-2026-paper","name":"ReflexBench · reaction-critical manipulation (2026 paper)","group":"Agent 与工具","metric":"Mean task success rate","unit":"%","public_history":true,"protocol":{"en":"Seven published policies train on the same 200 demonstrations per each of six tasks and use asynchronous inference with action chunk eight and horizon two. Each task has 150 evaluation episodes; paper success rates average three independent seeds. The published overall mean success rate and complete policy panel are retained.","zh":"完整七个策略在每项200条演示的六任务共同数据上训练，统一异步推理、动作块8与执行跨度2。每任务150个评测回合，取三个独立种子的成功率。保留论文总体平均成功率及完整策略面板。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.14379v1","status":"paper_baselines","published_at":"2026-08-14"},"model_count":7,"result_count":7,"results":[[7594,0.36,370,9801,9802],[9732,0.369,370,9803,9802],[9804,0.502,370,9803,9802],[9805,0.373,370,9806,9802],[9600,0.38299999999999995,370,9806,9802],[9807,0.303,370,9808,9802],[9809,0.504,370,9808,9802]]},{"id":"relational-anomaly-lanl-auprc-2026-paper","name":"Relational anomaly detection · lanl (2026 paper)","group":"预测与决策","metric":"AUPRC","unit":"score","public_history":true,"protocol":{"en":"The complete ten-row panel evaluates the same relational anomaly task over ten seeds. Rule/BCE and weighted/unweighted conditions remain configurations of their shared model, and AE ranking is not counted as a new base architecture. Each dataset’s mean AUPRC stays a separate native 0–1 cohort; no cross-dataset average ranks are imported.","zh":"完整十行面板在同一关系异常任务上评测十个随机种子。规则/BCE、加权/不加权条件保留为同一模型的配置，AE排序条件不算新底层架构。各数据集平均AUPRC分别保留为0–1原生指标，不导入跨数据集平均名次。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.23468v1","status":"paper_baselines","published_at":"2026-08-24"},"score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":7,"result_count":10,"results":[[9810,0.659,370,9811,9812],[9810,0.557,370,9813,9812],[9814,0.007,370,9815,9812],[9814,0.015,370,9816,9812],[9817,0.525,370,9818,9812],[9819,0.004,370,9820,9812],[9817,0.009,370,9821,9812],[9822,0.005,370,9823,9812],[9824,0.003,370,9825,9812],[9826,0.05,370,9827,9812]]},{"id":"relational-anomaly-amazon-auprc-2026-paper","name":"Relational anomaly detection · amazon (2026 paper)","group":"预测与决策","metric":"AUPRC","unit":"score","public_history":true,"protocol":{"en":"The complete ten-row panel evaluates the same relational anomaly task over ten seeds. Rule/BCE and weighted/unweighted conditions remain configurations of their shared model, and AE ranking is not counted as a new base architecture. Each dataset’s mean AUPRC stays a separate native 0–1 cohort; no cross-dataset average ranks are imported.","zh":"完整十行面板在同一关系异常任务上评测十个随机种子。规则/BCE、加权/不加权条件保留为同一模型的配置，AE排序条件不算新底层架构。各数据集平均AUPRC分别保留为0–1原生指标，不导入跨数据集平均名次。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.23468v1","status":"paper_baselines","published_at":"2026-08-24"},"score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":7,"result_count":10,"results":[[9810,0.043,370,9811,9812],[9810,0.043,370,9813,9812],[9814,0.037,370,9815,9812],[9814,0.037,370,9816,9812],[9817,0.016,370,9818,9812],[9819,0.013,370,9820,9812],[9817,0.014,370,9821,9812],[9822,0.013,370,9823,9812],[9824,0.013,370,9825,9812],[9826,0.027,370,9827,9812]]},{"id":"relational-anomaly-hm-auprc-2026-paper","name":"Relational anomaly detection · hm (2026 paper)","group":"预测与决策","metric":"AUPRC","unit":"score","public_history":true,"protocol":{"en":"The complete ten-row panel evaluates the same relational anomaly task over ten seeds. Rule/BCE and weighted/unweighted conditions remain configurations of their shared model, and AE ranking is not counted as a new base architecture. Each dataset’s mean AUPRC stays a separate native 0–1 cohort; no cross-dataset average ranks are imported.","zh":"完整十行面板在同一关系异常任务上评测十个随机种子。规则/BCE、加权/不加权条件保留为同一模型的配置，AE排序条件不算新底层架构。各数据集平均AUPRC分别保留为0–1原生指标，不导入跨数据集平均名次。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.23468v1","status":"paper_baselines","published_at":"2026-08-24"},"score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":7,"result_count":10,"results":[[9810,0.143,370,9811,9812],[9810,0.11,370,9813,9812],[9814,0.035,370,9815,9812],[9814,0.039,370,9816,9812],[9817,0.013,370,9818,9812],[9819,0.013,370,9820,9812],[9817,0.012,370,9821,9812],[9822,0.012,370,9823,9812],[9824,0.012,370,9825,9812],[9826,0.027,370,9827,9812]]},{"id":"camchoreo-frame-f1-2026-paper","name":"CamChoreo · frame motion recognition (2026 paper)","group":"视觉与视频","metric":"Frame micro-F1","unit":"score","public_history":true,"protocol":{"en":"The complete sixteen-system panel uses the same CamChoreo clips and expert temporal direction-aware motion labels. Frame-level metrics sample every 0.1 seconds; segment detection additionally requires matching type/direction and temporal IoU. Both are native 0–100 F1 metrics. Although the paper used future-release language, the project now exposes public training/evaluation code and a CamChoreo dataset link; the code manifest was inspected.","zh":"完整十六个系统使用同一CamChoreo视频与专家时间、方向感知动作标签。帧级每0.1秒采样；片段检测还要求类别、方向和时间IoU匹配。两类指标保留0–100原生F1。论文曾写未来发布，当前项目已提供公开训练评测代码与CamChoreo数据链接，并已检查代码清单。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.10932v1","status":"paper_baselines","published_at":"2026-08-11"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":16,"result_count":16,"results":[[9828,25.3,370,9829,9830],[502,43.3,370,9829,9830],[3184,38.0,370,9829,9830],[2654,23.6,370,9829,9830],[2754,21.8,370,9829,9830],[9831,11.9,370,9829,9830],[6298,24.2,370,9829,9830],[6342,28.3,370,9829,9830],[6297,33.4,370,9829,9830],[4969,22.3,370,9829,9830],[4070,21.5,370,9829,9830],[6771,35.9,370,9829,9830],[9832,67.5,370,9829,9830],[9833,67.5,370,9829,9830],[9834,67.8,370,9829,9830],[9835,68.3,370,9829,9830]]},{"id":"camchoreo-segment-f1-2026-paper","name":"CamChoreo · segment motion detection (2026 paper)","group":"视觉与视频","metric":"Segment detection F1@IoU0.5","unit":"score","public_history":true,"protocol":{"en":"The complete sixteen-system panel uses the same CamChoreo clips and expert temporal direction-aware motion labels. Frame-level metrics sample every 0.1 seconds; segment detection additionally requires matching type/direction and temporal IoU. Both are native 0–100 F1 metrics. Although the paper used future-release language, the project now exposes public training/evaluation code and a CamChoreo dataset link; the code manifest was inspected.","zh":"完整十六个系统使用同一CamChoreo视频与专家时间、方向感知动作标签。帧级每0.1秒采样；片段检测还要求类别、方向和时间IoU匹配。两类指标保留0–100原生F1。论文曾写未来发布，当前项目已提供公开训练评测代码与CamChoreo数据链接，并已检查代码清单。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.10932v1","status":"paper_baselines","published_at":"2026-08-11"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":16,"result_count":16,"results":[[9828,2.9,370,9836,9830],[502,23.7,370,9836,9830],[3184,22.4,370,9836,9830],[2654,4.0,370,9836,9830],[2754,6.4,370,9836,9830],[9831,0.0,370,9836,9830],[6298,11.6,370,9836,9830],[6342,15.0,370,9836,9830],[6297,15.6,370,9836,9830],[4969,15.4,370,9836,9830],[4070,13.3,370,9836,9830],[6771,17.9,370,9836,9830],[9832,38.2,370,9836,9830],[9833,38.6,370,9836,9830],[9834,38.8,370,9836,9830],[9835,39.0,370,9836,9830]]},{"id":"infraocc-camera-miou-2026-paper","name":"InfraOcc · camera occupancy (2026 paper)","group":"视觉与视频","metric":"Camera-only semantic mIoU","unit":"score","public_history":true,"protocol":{"en":"All ten camera-only architectures retrain on the shared infrastructure-side occupancy split and voxel/label space, with 215 training and 75 evaluation sequences. The complete camera panel and native 0–100 semantic mIoU are retained. LiDAR-only and fused sensing are separate tracks and are not pooled. Current public code contains dataset preparation and training/evaluation instructions.","zh":"完整十种纯摄像头架构在共同路侧占据划分、体素与语义空间上重新训练，使用215条训练及75条评测序列。保留完整摄像头面板与0–100原生语义mIoU，不混入激光或融合传感轨道。当前公开代码包含数据准备和训练评测说明。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.30657v1","status":"paper_baselines","published_at":"2026-08-31"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":10,"result_count":10,"results":[[9837,45.14,370,9838,9839],[9840,47.76,370,9838,9839],[9841,46.81,370,9838,9839],[9842,43.09,370,9838,9839],[9843,50.77,370,9838,9839],[9844,49.25,370,9838,9839],[9845,49.7,370,9838,9839],[9846,45.42,370,9838,9839],[9847,43.63,370,9838,9839],[9848,60.08,370,9838,9839]]},{"id":"scimif-instruction-success-2026-paper","name":"SciMIF · scientific instructions (2026 paper)","group":"数学与科学","metric":"Overall instruction satisfaction rate","unit":"%","public_history":true,"protocol":{"en":"All eleven MLLMs receive the same multimodal scientific instructions in chemistry, geography, biology, materials and physics. Overall is the paper’s mean instruction-level satisfaction rate across the five disciplines, separate from partial constraint-level CSR and decomposed DRFR. Current public artifacts include the input data, instructions, outputs and executable constraint verifiers.","zh":"完整十一个模型使用化学、地理、生物、材料和物理的同一多模态科学指令。总体指标为论文五学科指令级满足率均值，与部分约束满足CSR和分解约束DRFR分开。当前公开产物包含输入、指令、输出与可执行约束验证器。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.25973v1","status":"paper_baselines","published_at":"2026-08-26"},"model_count":11,"result_count":11,"results":[[340,0.6567000000000001,370,9849,9850],[9851,0.5304,370,9849,9850],[6237,0.5002,370,9849,9850],[6978,0.47369999999999995,370,9849,9850],[2584,0.5136999999999999,370,9849,9850],[2590,0.49950000000000006,370,9849,9850],[2582,0.5041,370,9849,9850],[2576,0.5739,370,9849,9850],[2654,0.5136999999999999,370,9849,9850],[6345,0.5314,370,9849,9850],[2642,0.5164,370,9849,9850]]},{"id":"trinity-nature-highlight-2026-paper","name":"TRINITY · nature highlights (2026 paper)","group":"视觉与视频","metric":"Highlight mAP at top 15%","unit":"score","public_history":true,"protocol":{"en":"All eight systems use the same TRINITY video/annotation split and one perspective-specific highlight target. The source’s top-15%-budget mAP remains native 0–100. Nature, emotion and event perspectives are separate tasks; pretrained zero-shot rows remain labeled as such in the fixture. The current repository exposes the dataset downloader and training/model implementation.","zh":"完整八个系统使用同一TRINITY视频标签划分与相应视角的精彩片段目标。保留前15%片段预算的0–100原生mAP。自然、情绪与事件视角分别展示；预训练零样本行的标记保存在原表中。当前仓库提供数据下载器与训练、模型实现。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.29577v1","status":"paper_baselines","published_at":"2026-08-30"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":8,"result_count":8,"results":[[9852,38.68,370,9853,9854],[9855,46.4,370,9853,9854],[9856,36.47,370,9853,9854],[9857,35.68,370,9853,9854],[9858,38.69,370,9859,9854],[6477,24.89,370,9859,9854],[9860,27.91,370,9859,9854],[9861,58.74,370,9853,9854]]},{"id":"trinity-emotion-highlight-2026-paper","name":"TRINITY · emotion highlights (2026 paper)","group":"视觉与视频","metric":"Highlight mAP at top 15%","unit":"score","public_history":true,"protocol":{"en":"All eight systems use the same TRINITY video/annotation split and one perspective-specific highlight target. The source’s top-15%-budget mAP remains native 0–100. Nature, emotion and event perspectives are separate tasks; pretrained zero-shot rows remain labeled as such in the fixture. The current repository exposes the dataset downloader and training/model implementation.","zh":"完整八个系统使用同一TRINITY视频标签划分与相应视角的精彩片段目标。保留前15%片段预算的0–100原生mAP。自然、情绪与事件视角分别展示；预训练零样本行的标记保存在原表中。当前仓库提供数据下载器与训练、模型实现。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.29577v1","status":"paper_baselines","published_at":"2026-08-30"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":8,"result_count":8,"results":[[9852,50.55,370,9862,9854],[9855,62.47,370,9862,9854],[9856,64.96,370,9862,9854],[9857,37.17,370,9862,9854],[9858,21.02,370,9863,9854],[6477,22.99,370,9863,9854],[9860,26.42,370,9863,9854],[9861,67.42,370,9862,9854]]},{"id":"trinity-event-highlight-2026-paper","name":"TRINITY · event highlights (2026 paper)","group":"视觉与视频","metric":"Highlight mAP at top 15%","unit":"score","public_history":true,"protocol":{"en":"All eight systems use the same TRINITY video/annotation split and one perspective-specific highlight target. The source’s top-15%-budget mAP remains native 0–100. Nature, emotion and event perspectives are separate tasks; pretrained zero-shot rows remain labeled as such in the fixture. The current repository exposes the dataset downloader and training/model implementation.","zh":"完整八个系统使用同一TRINITY视频标签划分与相应视角的精彩片段目标。保留前15%片段预算的0–100原生mAP。自然、情绪与事件视角分别展示；预训练零样本行的标记保存在原表中。当前仓库提供数据下载器与训练、模型实现。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.29577v1","status":"paper_baselines","published_at":"2026-08-30"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":8,"result_count":8,"results":[[9852,31.46,370,9864,9854],[9855,33.61,370,9864,9854],[9856,35.97,370,9864,9854],[9857,26.21,370,9864,9854],[9858,32.66,370,9865,9854],[6477,17.92,370,9865,9854],[9860,18.1,370,9865,9854],[9861,40.98,370,9864,9854]]},{"id":"entlore-l3-bm25-2026-paper","name":"EntLORE · latent reasoning, bm25 (2026 paper)","group":"知识与推理","metric":"L3 answer accuracy","unit":"%","public_history":true,"protocol":{"en":"The complete eight-model panel uses the same L3 enterprise questions and anonymized document world within the stated access condition. L3 target relations are not explicitly stated in released sources and require latent organizational reasoning. BM25, dense RAG and GraphRAG are independent access protocols. Public corpus, questions, golden packets and scoring scripts are currently available; original release date is verified separately from paper v2.","zh":"完整八模型面板在指定访问条件下使用同一L3企业问题与匿名化文档世界。L3目标关系未在公开来源中明示，需推断潜在组织关系。BM25、稠密RAG与GraphRAG访问协议分别展示。当前已公开语料、问题、答案包和评分脚本；原始首发日与论文v2日期分别核验。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.10679v2","status":"paper_baselines","published_at":"2026-08-11"},"model_count":8,"result_count":8,"results":[[3184,0.349,370,9866,9867],[4958,0.33,370,9866,9867],[3572,0.364,370,9866,9867],[3191,0.36200000000000004,370,9866,9867],[227,0.309,370,9866,9867],[3187,0.35100000000000003,370,9866,9867],[3492,0.33799999999999997,370,9866,9867],[3261,0.318,370,9866,9867]]},{"id":"entlore-l3-dense-rag-2026-paper","name":"EntLORE · latent reasoning, dense-rag (2026 paper)","group":"知识与推理","metric":"L3 answer accuracy","unit":"%","public_history":true,"protocol":{"en":"The complete eight-model panel uses the same L3 enterprise questions and anonymized document world within the stated access condition. L3 target relations are not explicitly stated in released sources and require latent organizational reasoning. BM25, dense RAG and GraphRAG are independent access protocols. Public corpus, questions, golden packets and scoring scripts are currently available; original release date is verified separately from paper v2.","zh":"完整八模型面板在指定访问条件下使用同一L3企业问题与匿名化文档世界。L3目标关系未在公开来源中明示，需推断潜在组织关系。BM25、稠密RAG与GraphRAG访问协议分别展示。当前已公开语料、问题、答案包和评分脚本；原始首发日与论文v2日期分别核验。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.10679v2","status":"paper_baselines","published_at":"2026-08-11"},"model_count":8,"result_count":8,"results":[[3184,0.191,370,9868,9867],[4958,0.184,370,9868,9867],[3572,0.212,370,9868,9867],[3191,0.195,370,9868,9867],[227,0.174,370,9868,9867],[3187,0.195,370,9868,9867],[3492,0.177,370,9868,9867],[3261,0.184,370,9868,9867]]},{"id":"entlore-l3-graphrag-2026-paper","name":"EntLORE · latent reasoning, graphrag (2026 paper)","group":"知识与推理","metric":"L3 answer accuracy","unit":"%","public_history":true,"protocol":{"en":"The complete eight-model panel uses the same L3 enterprise questions and anonymized document world within the stated access condition. L3 target relations are not explicitly stated in released sources and require latent organizational reasoning. BM25, dense RAG and GraphRAG are independent access protocols. Public corpus, questions, golden packets and scoring scripts are currently available; original release date is verified separately from paper v2.","zh":"完整八模型面板在指定访问条件下使用同一L3企业问题与匿名化文档世界。L3目标关系未在公开来源中明示，需推断潜在组织关系。BM25、稠密RAG与GraphRAG访问协议分别展示。当前已公开语料、问题、答案包和评分脚本；原始首发日与论文v2日期分别核验。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.10679v2","status":"paper_baselines","published_at":"2026-08-11"},"model_count":8,"result_count":8,"results":[[3184,0.341,370,9869,9867],[4958,0.28800000000000003,370,9869,9867],[3572,0.38299999999999995,370,9869,9867],[3191,0.396,370,9869,9867],[227,0.431,370,9869,9867],[3187,0.276,370,9869,9867],[3492,0.386,370,9869,9867],[3261,0.39799999999999996,370,9869,9867]]},{"id":"motionblind-instance-2026-paper","name":"MotionBlind (2026 paper)","group":"视觉与视频","metric":"Four-question instance accuracy","unit":"%","public_history":true,"protocol":{"en":"Seven independent video-language models are compared on the common 60 paired-motion instances (240 binary questions). An instance passes only if all four constituent questions are correct. Each score is the paper’s best frame-budget/sampling configuration for that model; reasoning and tuned checkpoints remain explicit. Gemma’s 56-instance subset and human/chance controls are excluded.","zh":"七个独立视频语言模型评测相同60组配对运动实例，共240道二选题。一组中的四道题全部正确才记为通过。分数使用论文为每个模型选择的最佳帧数与采样配置，思考设置和专用权重保留为配置。排除只覆盖56组的Gemma结果及人类、随机参照。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.09528v1","published_at":"2026-09-08","status":"paper_baselines"},"model_count":7,"result_count":7,"results":[[9870,0.11699999999999999,370,9871,9872],[6298,0.083,370,9871,9872],[9873,0.067,370,9874,9872],[9875,0.033,370,9871,9872],[5723,0.1,370,9876,9872],[9877,0.6,370,9871,9872],[9878,0.15,370,9879,9872]]},{"id":"nutribench-kitchen-2026-paper","name":"NutriBench-Kitchen (2026 paper)","group":"Agent 与工具","metric":"Mixed-format task score","unit":"score","public_history":true,"protocol":{"en":"The complete 16-row learned-model/configuration panel uses the same 1,500 manually verified questions from 160 cooking videos. Multiple-choice items use deterministic option accuracy; numerical questions use mean relative accuracy across ten tolerances from 0.50 to 0.95. The published average mixes these formats and stays a native 0–100 task score. Vgent and Nutri-Vgent configurations retain their Qwen3-VL backbones.","zh":"完整16条模型及配置结果使用160段烹饪视频中的同一组1,500道人工验证问题。选择题以确定性解析器计算正确率；数值题计算0.50至0.95十个容差下的平均相对准确度。原文均分混合两种题型，按原生0–100任务分展示。Vgent和Nutri-Vgent配置保留对应Qwen3-VL骨干。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.07135v1","published_at":"2026-09-07","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":12,"result_count":16,"results":[[2602,61.1,370,9880,9881],[353,47.7,370,9880,9881],[8252,45.5,370,9880,9881],[9882,52.6,370,9880,9881],[9883,46.6,370,9880,9881],[9884,46.9,370,9880,9881],[9885,50.8,370,9880,9881],[9886,29.4,370,9880,9881],[2771,50.7,370,9880,9881],[2754,50.8,370,9880,9881],[6298,55.5,370,9880,9881],[6298,28.8,370,9887,9881],[6298,62.1,370,9888,9881],[6342,53.4,370,9880,9881],[6342,31.7,370,9889,9881],[6342,62.3,370,9890,9881]]},{"id":"q2d-web-full-2026-paper","name":"Q2D-Web · full (2026 paper)","group":"Agent 与工具","metric":"Combined-judgment recall at 1000","unit":"score","public_history":true,"protocol":{"en":"Twelve learned retrievers use the fixed Combined+LLM-judged relevance set for about 70,000 agentic queries in ten languages. The full corpus contains about 190 million documents. Recall at 1000 stays on the published native 0–100 scale. The protected production queries/corpus are evaluator-held; published selection, relevance pooling and subsampling protocol are inspected. BM25 is an excluded lexical control.","zh":"十二个训练后的检索模型使用约七万条、十种语言的Agent搜索查询，以及固定的Combined+LLM-judged相关性标签。完整语料约含1.9亿份文档。前1000项召回保留原生0–100刻度。生产查询与语料由评测方保管，论文公开了筛选、相关性汇集和子采样流程；排除词法参照BM25。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.08887v1","published_at":"2026-09-08","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":12,"result_count":12,"results":[[9891,65.45,370,9892,9893],[9894,59.31,370,9892,9893],[9895,61.68,370,9892,9893],[9896,68.58,370,9892,9893],[9897,67.02,370,9892,9893],[9898,69.11,370,9892,9893],[9738,57.89,370,9892,9893],[9899,62.01,370,9892,9893],[9900,64.53,370,9892,9893],[9901,56.67,370,9892,9893],[9902,60.98,370,9892,9893],[9903,63.4,370,9892,9893]]},{"id":"q2d-web-subsampled-2026-paper","name":"Q2D-Web · subsampled (2026 paper)","group":"Agent 与工具","metric":"Combined-judgment recall at 1000","unit":"score","public_history":true,"protocol":{"en":"Twelve learned retrievers use the fixed Combined+LLM-judged relevance set for about 70,000 agentic queries in ten languages. The retrieval corpus is the official reciprocal-rank-fusion subsample at k=1000. Recall at 1000 stays on the published native 0–100 scale. The protected production queries/corpus are evaluator-held; published selection, relevance pooling and subsampling protocol are inspected. BM25 is an excluded lexical control.","zh":"十二个训练后的检索模型使用约七万条、十种语言的Agent搜索查询，以及固定的Combined+LLM-judged相关性标签。采用官方k=1000倒数排名融合子采样语料。前1000项召回保留原生0–100刻度。生产查询与语料由评测方保管，论文公开了筛选、相关性汇集和子采样流程；排除词法参照BM25。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.08887v1","published_at":"2026-09-08","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":12,"result_count":12,"results":[[9891,70.1,370,9904,9893],[9894,64.13,370,9904,9893],[9895,66.88,370,9904,9893],[9896,72.79,370,9904,9893],[9897,71.52,370,9904,9893],[9898,74.07,370,9904,9893],[9738,63.52,370,9904,9893],[9899,67.58,370,9904,9893],[9900,69.61,370,9904,9893],[9901,63.06,370,9904,9893],[9902,65.62,370,9904,9893],[9903,68.95,370,9904,9893]]},{"id":"phi-infrastructure-2026-paper","name":"Phi-Bench (2026 paper)","group":"编程","metric":"Correctness-gated implementation/efficiency reward","unit":"score","public_history":true,"protocol":{"en":"Eight independent model checkpoints solve the same 85 infrastructure tasks: 55 kernel/function tasks, 20 long-horizon implementation tasks and 10 end-to-end optimization tasks. Each task receives a correctness-gated binary implementation reward or continuous efficiency reward, averaged on the published 0–100 scale. GPT-5.6-Sol uses Codex; other models use Claude Code, with the highest offered reasoning budget and maximum context.","zh":"八个独立模型权重解决相同85项基础设施任务：55项内核或函数任务、20项长程实现任务和10项端到端优化任务。每题先检查正确性，再给二值实现奖励或连续效率奖励，以原文0–100刻度汇总。GPT-5.6-Sol使用Codex，其余使用Claude Code，并采用各自最高思考预算和最大上下文。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.10226v1","published_at":"2026-09-09","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":8,"result_count":8,"results":[[755,36.53,370,9905,9906],[2977,28.12,370,9905,9906],[2982,27.73,370,9905,9906],[5004,24.51,370,9907,9906],[3654,21.92,370,9905,9906],[3568,17.58,370,9905,9906],[6233,16.07,370,9905,9906],[9908,13.31,370,9905,9906]]},{"id":"netartifact-contract-2026-paper","name":"NetArtifactBench (2026 paper)","group":"Agent 与工具","metric":"Artifact contract integrity","unit":"%","public_history":true,"protocol":{"en":"The full 23 runtime/backend configurations execute all 52 artifact-recovery instances, repeated five times. The selected contract-integrity percentage is deterministically scored over artifact validity and restoration requirements, with the paper’s five-run mean retained. Runtime, model backend and reasoning effort remain explicit configurations. Other violation and recall measures have separate denominators and are not merged.","zh":"完整23种运行时与模型配置执行全部52项产物恢复任务，每项重复五次。合约完整率按产物有效性和恢复要求确定性评分，保留论文五次运行均值。运行时、模型后端和思考强度均注明。其他违规率和召回率使用各自分母，分别保留。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.09849v1","published_at":"2026-09-09","status":"paper_baselines"},"model_count":20,"result_count":23,"results":[[9909,0.669,370,9910,9911],[9912,0.6809999999999999,370,9913,9911],[9914,0.7120000000000001,370,9915,9911],[9916,0.731,370,9917,9911],[9918,0.765,370,9919,9911],[9920,0.769,370,9921,9911],[9922,0.8190000000000001,370,9923,9911],[9924,0.75,370,9925,9911],[9926,0.785,370,9927,9911],[3479,0.8079999999999999,370,9928,9911],[3568,0.8,370,9929,9911],[5262,0.754,370,9930,9911],[3412,0.669,370,9931,9911],[3415,0.7,370,9932,9911],[3184,0.727,370,9933,9911],[7626,0.45399999999999996,370,9934,9911],[3391,0.785,370,9935,9911],[9936,0.48100000000000004,370,9937,9911],[3261,0.723,370,9938,9911],[4424,0.662,370,9939,9911],[6771,0.608,370,9940,9911],[6268,0.012,370,9941,9911],[6269,0.154,370,9942,9911]]},{"id":"yallamorph-10-shot-accuracy-2026-paper","name":"YallaMorph · 10-shot accuracy (2026 paper)","group":"知识与推理","metric":"Any-match accuracy","unit":"%","public_history":true,"protocol":{"en":"Seven independent instruction-tuned checkpoints perform Arabic morphological generation over the shared 600K-plus evaluation entries, using English prompts under 10-shot prompting. Lemma, POS, gloss and target feature bundle are fixed. This is the normalized-output column: any match against accepted surface forms. Arabic-prompt rows are a different configuration and remain outside this language-specific cohort.","zh":"七个独立指令模型使用英语提示，在相同六十多万条阿拉伯语词形生成实例上评测10-shot能力。词元、词性、释义和目标特征束固定。采用输出归一化后的任一合法词形匹配正确率。阿拉伯语提示结果属于其他配置，未混入这组排行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.10153v1","published_at":"2026-09-09","status":"paper_baselines"},"model_count":7,"result_count":7,"results":[[3184,0.677,370,9943,9944],[4219,0.614,370,9943,9944],[9945,0.37200000000000005,370,9943,9944],[9946,0.212,370,9943,9944],[9947,0.11,370,9943,9944],[1748,0.136,370,9943,9944],[9948,0.102,370,9943,9944]]},{"id":"yallamorph-10-shot-f1-2026-paper","name":"YallaMorph · 10-shot f1 (2026 paper)","group":"知识与推理","metric":"Morphological F1","unit":"score","public_history":true,"protocol":{"en":"Seven independent instruction-tuned checkpoints perform Arabic morphological generation over the shared 600K-plus evaluation entries, using English prompts under 10-shot prompting. Lemma, POS, gloss and target feature bundle are fixed. This is the normalized-output column: the published morphological F1 on its native 0–100 scale. Arabic-prompt rows are a different configuration and remain outside this language-specific cohort.","zh":"七个独立指令模型使用英语提示，在相同六十多万条阿拉伯语词形生成实例上评测10-shot能力。词元、词性、释义和目标特征束固定。采用输出归一化后的词形F1，保留原生0–100刻度。阿拉伯语提示结果属于其他配置，未混入这组排行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.10153v1","published_at":"2026-09-09","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":7,"result_count":7,"results":[[3184,71.3,370,9943,9944],[4219,64.9,370,9943,9944],[9945,39.1,370,9943,9944],[9946,22.1,370,9943,9944],[9947,11.2,370,9943,9944],[1748,15.0,370,9943,9944],[9948,10.0,370,9943,9944]]},{"id":"yallamorph-zero-shot-accuracy-2026-paper","name":"YallaMorph · zero-shot accuracy (2026 paper)","group":"知识与推理","metric":"Any-match accuracy","unit":"%","public_history":true,"protocol":{"en":"Seven independent instruction-tuned checkpoints perform Arabic morphological generation over the shared 600K-plus evaluation entries, using English prompts under zero-shot prompting. Lemma, POS, gloss and target feature bundle are fixed. This is the normalized-output column: any match against accepted surface forms. Arabic-prompt rows are a different configuration and remain outside this language-specific cohort.","zh":"七个独立指令模型使用英语提示，在相同六十多万条阿拉伯语词形生成实例上评测zero-shot能力。词元、词性、释义和目标特征束固定。采用输出归一化后的任一合法词形匹配正确率。阿拉伯语提示结果属于其他配置，未混入这组排行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.10153v1","published_at":"2026-09-09","status":"paper_baselines"},"model_count":7,"result_count":7,"results":[[3184,0.627,370,9949,9944],[4219,0.5429999999999999,370,9949,9944],[9945,0.19899999999999998,370,9949,9944],[9946,0.159,370,9949,9944],[9947,0.08800000000000001,370,9949,9944],[1748,0.091,370,9949,9944],[9948,0.067,370,9949,9944]]},{"id":"yallamorph-zero-shot-f1-2026-paper","name":"YallaMorph · zero-shot f1 (2026 paper)","group":"知识与推理","metric":"Morphological F1","unit":"score","public_history":true,"protocol":{"en":"Seven independent instruction-tuned checkpoints perform Arabic morphological generation over the shared 600K-plus evaluation entries, using English prompts under zero-shot prompting. Lemma, POS, gloss and target feature bundle are fixed. This is the normalized-output column: the published morphological F1 on its native 0–100 scale. Arabic-prompt rows are a different configuration and remain outside this language-specific cohort.","zh":"七个独立指令模型使用英语提示，在相同六十多万条阿拉伯语词形生成实例上评测zero-shot能力。词元、词性、释义和目标特征束固定。采用输出归一化后的词形F1，保留原生0–100刻度。阿拉伯语提示结果属于其他配置，未混入这组排行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.10153v1","published_at":"2026-09-09","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":7,"result_count":7,"results":[[3184,67.3,370,9949,9944],[4219,57.7,370,9949,9944],[9945,20.9,370,9949,9944],[9946,16.9,370,9949,9944],[9947,7.3,370,9949,9944],[1748,9.7,370,9949,9944],[9948,6.0,370,9949,9944]]},{"id":"mciqa-color-smearing-srcc-2026-paper","name":"MCIQA-2K · color-smearing SRCC (2026 paper)","group":"视觉与视频","metric":"SRCC versus human ratings","unit":"score","public_history":true,"protocol":{"en":"Ten learned no-reference IQA checkpoints predict the same human-rated colorization images for color-smearing. The 2,000-image resource has 16 human raters and an 80/20 train/test split grouped by grayscale source, preventing recolorings of one source from crossing splits. Scores are three-seed means. SRCC is retained on its native correlation scale; positive direction is consistent for this panel.","zh":"十个训练后的无参考图像质量模型预测同一组人工评分上色图像的color-smearing表现。资源含2,000张图像、16位评分者，按灰度源图分组作80/20训练测试划分，避免同源上色图跨组。分数为三个种子均值，SRCC保留原生相关系数刻度，本组指标方向一致。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.14495v1","published_at":"2026-09-13","status":"paper_baselines"},"score_max":1,"saturation_excluded":true,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"score_decimals":3,"model_count":10,"result_count":10,"results":[[9950,0.0338,370,9951,9952],[9953,0.2196,370,9951,9952],[9954,0.1629,370,9951,9952],[9955,0.1422,370,9951,9952],[9956,0.2468,370,9951,9952],[9957,0.2327,370,9951,9952],[9958,0.3199,370,9951,9952],[9959,0.8274,370,9951,9952],[9960,0.4597,370,9951,9952],[9961,0.5059,370,9951,9952]]},{"id":"mciqa-color-smearing-plcc-2026-paper","name":"MCIQA-2K · color-smearing PLCC (2026 paper)","group":"视觉与视频","metric":"PLCC versus human ratings","unit":"score","public_history":true,"protocol":{"en":"Ten learned no-reference IQA checkpoints predict the same human-rated colorization images for color-smearing. The 2,000-image resource has 16 human raters and an 80/20 train/test split grouped by grayscale source, preventing recolorings of one source from crossing splits. Scores are three-seed means. PLCC is retained on its native correlation scale; positive direction is consistent for this panel.","zh":"十个训练后的无参考图像质量模型预测同一组人工评分上色图像的color-smearing表现。资源含2,000张图像、16位评分者，按灰度源图分组作80/20训练测试划分，避免同源上色图跨组。分数为三个种子均值，PLCC保留原生相关系数刻度，本组指标方向一致。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.14495v1","published_at":"2026-09-13","status":"paper_baselines"},"score_max":1,"saturation_excluded":true,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"score_decimals":3,"model_count":10,"result_count":10,"results":[[9950,0.0472,370,9951,9952],[9953,0.2319,370,9951,9952],[9954,0.2018,370,9951,9952],[9955,0.1861,370,9951,9952],[9956,0.2614,370,9951,9952],[9957,0.2445,370,9951,9952],[9958,0.3305,370,9951,9952],[9959,0.8213,370,9951,9952],[9960,0.4817,370,9951,9952],[9961,0.5193,370,9951,9952]]},{"id":"mciqa-semantic-color-srcc-2026-paper","name":"MCIQA-2K · semantic-color SRCC (2026 paper)","group":"视觉与视频","metric":"SRCC versus human ratings","unit":"score","public_history":true,"protocol":{"en":"Ten learned no-reference IQA checkpoints predict the same human-rated colorization images for semantic-color. The 2,000-image resource has 16 human raters and an 80/20 train/test split grouped by grayscale source, preventing recolorings of one source from crossing splits. Scores are three-seed means. SRCC is retained on its native correlation scale; positive direction is consistent for this panel.","zh":"十个训练后的无参考图像质量模型预测同一组人工评分上色图像的semantic-color表现。资源含2,000张图像、16位评分者，按灰度源图分组作80/20训练测试划分，避免同源上色图跨组。分数为三个种子均值，SRCC保留原生相关系数刻度，本组指标方向一致。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.14495v1","published_at":"2026-09-13","status":"paper_baselines"},"score_max":1,"saturation_excluded":true,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"score_decimals":3,"model_count":10,"result_count":10,"results":[[9950,0.0944,370,9951,9952],[9953,0.2123,370,9951,9952],[9954,0.1396,370,9951,9952],[9955,0.0978,370,9951,9952],[9956,0.3223,370,9951,9952],[9957,0.2939,370,9951,9952],[9958,0.3961,370,9951,9952],[9959,0.466,370,9951,9952],[9960,0.8902,370,9951,9952],[9961,0.3938,370,9951,9952]]},{"id":"mciqa-semantic-color-plcc-2026-paper","name":"MCIQA-2K · semantic-color PLCC (2026 paper)","group":"视觉与视频","metric":"PLCC versus human ratings","unit":"score","public_history":true,"protocol":{"en":"Ten learned no-reference IQA checkpoints predict the same human-rated colorization images for semantic-color. The 2,000-image resource has 16 human raters and an 80/20 train/test split grouped by grayscale source, preventing recolorings of one source from crossing splits. Scores are three-seed means. PLCC is retained on its native correlation scale; positive direction is consistent for this panel.","zh":"十个训练后的无参考图像质量模型预测同一组人工评分上色图像的semantic-color表现。资源含2,000张图像、16位评分者，按灰度源图分组作80/20训练测试划分，避免同源上色图跨组。分数为三个种子均值，PLCC保留原生相关系数刻度，本组指标方向一致。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.14495v1","published_at":"2026-09-13","status":"paper_baselines"},"score_max":1,"saturation_excluded":true,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"score_decimals":3,"model_count":10,"result_count":10,"results":[[9950,0.1875,370,9951,9952],[9953,0.3235,370,9951,9952],[9954,0.2514,370,9951,9952],[9955,0.2184,370,9951,9952],[9956,0.3879,370,9951,9952],[9957,0.3689,370,9951,9952],[9958,0.4964,370,9951,9952],[9959,0.4642,370,9951,9952],[9960,0.8408,370,9951,9952],[9961,0.4749,370,9951,9952]]},{"id":"mciqa-global-naturalness-srcc-2026-paper","name":"MCIQA-2K · global-naturalness SRCC (2026 paper)","group":"视觉与视频","metric":"SRCC versus human ratings","unit":"score","public_history":true,"protocol":{"en":"Ten learned no-reference IQA checkpoints predict the same human-rated colorization images for global-naturalness. The 2,000-image resource has 16 human raters and an 80/20 train/test split grouped by grayscale source, preventing recolorings of one source from crossing splits. Scores are three-seed means. SRCC is retained on its native correlation scale; positive direction is consistent for this panel.","zh":"十个训练后的无参考图像质量模型预测同一组人工评分上色图像的global-naturalness表现。资源含2,000张图像、16位评分者，按灰度源图分组作80/20训练测试划分，避免同源上色图跨组。分数为三个种子均值，SRCC保留原生相关系数刻度，本组指标方向一致。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.14495v1","published_at":"2026-09-13","status":"paper_baselines"},"score_max":1,"saturation_excluded":true,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"score_decimals":3,"model_count":10,"result_count":10,"results":[[9950,0.097,370,9951,9952],[9953,0.3651,370,9951,9952],[9954,0.221,370,9951,9952],[9955,0.2872,370,9951,9952],[9956,0.5619,370,9951,9952],[9957,0.4019,370,9951,9952],[9958,0.7126,370,9951,9952],[9959,0.5045,370,9951,9952],[9960,0.4522,370,9951,9952],[9961,0.8511,370,9951,9952]]},{"id":"mciqa-global-naturalness-plcc-2026-paper","name":"MCIQA-2K · global-naturalness PLCC (2026 paper)","group":"视觉与视频","metric":"PLCC versus human ratings","unit":"score","public_history":true,"protocol":{"en":"Ten learned no-reference IQA checkpoints predict the same human-rated colorization images for global-naturalness. The 2,000-image resource has 16 human raters and an 80/20 train/test split grouped by grayscale source, preventing recolorings of one source from crossing splits. Scores are three-seed means. PLCC is retained on its native correlation scale; positive direction is consistent for this panel.","zh":"十个训练后的无参考图像质量模型预测同一组人工评分上色图像的global-naturalness表现。资源含2,000张图像、16位评分者，按灰度源图分组作80/20训练测试划分，避免同源上色图跨组。分数为三个种子均值，PLCC保留原生相关系数刻度，本组指标方向一致。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.14495v1","published_at":"2026-09-13","status":"paper_baselines"},"score_max":1,"saturation_excluded":true,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"score_decimals":3,"model_count":10,"result_count":10,"results":[[9950,0.1843,370,9951,9952],[9953,0.4469,370,9951,9952],[9954,0.3315,370,9951,9952],[9955,0.4042,370,9951,9952],[9956,0.5968,370,9951,9952],[9957,0.4749,370,9951,9952],[9958,0.7545,370,9951,9952],[9959,0.5011,370,9951,9952],[9960,0.5648,370,9951,9952],[9961,0.8571,370,9951,9952]]},{"id":"mtac-claudecode-csr-2026-paper","name":"MTAC-IFBench · Claude Code csr (2026 paper)","group":"编程","metric":"Constraint success rate","unit":"%","public_history":true,"protocol":{"en":"The complete 11-model panel uses Claude Code on the same 100 multi-turn coding instances (7.04 turns and 91.33 constraints per instance on average). CSR averages satisfied checklist constraints at each turn; C-ISR requires every constraint at a turn. The selected published Avg. is formed under the paper’s turn-interval bucketing protocol. Frameworks are separate cohorts, with deterministic and agent-judge checklist validation.","zh":"完整11模型面板使用Claude Code，执行相同100项多轮编程任务，平均每项7.04轮、91.33条约束。CSR统计每轮满足的清单约束比例，C-ISR要求该轮全部约束满足。保留论文按轮次区间分桶汇总的Avg.，两种框架分别列组，清单以代码和Agent评判验证。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.14992v1","published_at":"2026-09-14","status":"paper_baselines"},"model_count":11,"result_count":11,"results":[[3183,0.7859999999999999,370,9962,9963],[502,0.71,370,9962,9963],[3189,0.688,370,9962,9963],[3192,0.669,370,9962,9963],[6214,0.647,370,9962,9963],[227,0.804,370,9962,9963],[3186,0.765,370,9962,9963],[3185,0.747,370,9962,9963],[3187,0.732,370,9962,9963],[3201,0.7,370,9962,9963],[2584,0.606,370,9962,9963]]},{"id":"mtac-claudecode-cisr-2026-paper","name":"MTAC-IFBench · Claude Code cisr (2026 paper)","group":"编程","metric":"Strict constraint-instruction success rate","unit":"%","public_history":true,"protocol":{"en":"The complete 11-model panel uses Claude Code on the same 100 multi-turn coding instances (7.04 turns and 91.33 constraints per instance on average). CSR averages satisfied checklist constraints at each turn; C-ISR requires every constraint at a turn. The selected published Avg. is formed under the paper’s turn-interval bucketing protocol. Frameworks are separate cohorts, with deterministic and agent-judge checklist validation.","zh":"完整11模型面板使用Claude Code，执行相同100项多轮编程任务，平均每项7.04轮、91.33条约束。CSR统计每轮满足的清单约束比例，C-ISR要求该轮全部约束满足。保留论文按轮次区间分桶汇总的Avg.，两种框架分别列组，清单以代码和Agent评判验证。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.14992v1","published_at":"2026-09-14","status":"paper_baselines"},"model_count":11,"result_count":11,"results":[[3183,0.087,370,9962,9963],[502,0.111,370,9962,9963],[3189,0.02,370,9962,9963],[3192,0.026000000000000002,370,9962,9963],[6214,0.019,370,9962,9963],[227,0.127,370,9962,9963],[3186,0.08900000000000001,370,9962,9963],[3185,0.064,370,9962,9963],[3187,0.078,370,9962,9963],[3201,0.034,370,9962,9963],[2584,0.011000000000000001,370,9962,9963]]},{"id":"mtac-opencode-csr-2026-paper","name":"MTAC-IFBench · OpenCode csr (2026 paper)","group":"编程","metric":"Constraint success rate","unit":"%","public_history":true,"protocol":{"en":"The complete 8-model panel uses OpenCode on the same 100 multi-turn coding instances (7.04 turns and 91.33 constraints per instance on average). CSR averages satisfied checklist constraints at each turn; C-ISR requires every constraint at a turn. The selected published Avg. is formed under the paper’s turn-interval bucketing protocol. Frameworks are separate cohorts, with deterministic and agent-judge checklist validation.","zh":"完整8模型面板使用OpenCode，执行相同100项多轮编程任务，平均每项7.04轮、91.33条约束。CSR统计每轮满足的清单约束比例，C-ISR要求该轮全部约束满足。保留论文按轮次区间分桶汇总的Avg.，两种框架分别列组，清单以代码和Agent评判验证。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.14992v1","published_at":"2026-09-14","status":"paper_baselines"},"model_count":8,"result_count":8,"results":[[3189,0.68,370,9964,9965],[3192,0.672,370,9964,9965],[6214,0.618,370,9964,9965],[227,0.768,370,9964,9965],[3186,0.743,370,9964,9965],[3185,0.722,370,9964,9965],[3201,0.637,370,9964,9965],[2584,0.618,370,9964,9965]]},{"id":"mtac-opencode-cisr-2026-paper","name":"MTAC-IFBench · OpenCode cisr (2026 paper)","group":"编程","metric":"Strict constraint-instruction success rate","unit":"%","public_history":true,"protocol":{"en":"The complete 8-model panel uses OpenCode on the same 100 multi-turn coding instances (7.04 turns and 91.33 constraints per instance on average). CSR averages satisfied checklist constraints at each turn; C-ISR requires every constraint at a turn. The selected published Avg. is formed under the paper’s turn-interval bucketing protocol. Frameworks are separate cohorts, with deterministic and agent-judge checklist validation.","zh":"完整8模型面板使用OpenCode，执行相同100项多轮编程任务，平均每项7.04轮、91.33条约束。CSR统计每轮满足的清单约束比例，C-ISR要求该轮全部约束满足。保留论文按轮次区间分桶汇总的Avg.，两种框架分别列组，清单以代码和Agent评判验证。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.14992v1","published_at":"2026-09-14","status":"paper_baselines"},"model_count":8,"result_count":8,"results":[[3189,0.024,370,9964,9965],[3192,0.043,370,9964,9965],[6214,0.01,370,9964,9965],[227,0.067,370,9964,9965],[3186,0.045,370,9964,9965],[3185,0.059000000000000004,370,9964,9965],[3201,0.011000000000000001,370,9964,9965],[2584,0.01,370,9964,9965]]},{"id":"space-hit-2026-paper","name":"SPACE · hit (2026 paper)","group":"视觉与视频","metric":"Evaluation-point hit rate","unit":"%","public_history":true,"protocol":{"en":"Thirteen learned model/checkpoint rows describe the same subject-centered photographs from the 2,981-image SPACE resource. The evaluator labels each predefined fine-grained point correct, wrong or missing. Hit divides correct points by all predefined evaluation points, including omissions. H-SPACE-GRPO is a trained model, and Instruct/Thinking weights retain their source identities.","zh":"十三条模型权重结果描述SPACE资源中的相同人物照片；资源共2,981张图像。评判器对预定义细粒度要点标注正确、错误或遗漏。命中率以全部预定义要点为分母，遗漏也计入。H-SPACE-GRPO为训练后的模型，Instruct与Thinking权重保留原始身份。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.08300v1","published_at":"2026-09-08","status":"paper_baselines"},"model_count":13,"result_count":13,"results":[[345,0.445,370,9966,9967],[9968,0.621,370,9966,9967],[9969,0.6759999999999999,370,9966,9967],[9970,0.408,370,9966,9967],[9971,0.43700000000000006,370,9966,9967],[9972,0.47100000000000003,370,9966,9967],[9973,0.355,370,9966,9967],[9974,0.324,370,9966,9967],[9975,0.5670000000000001,370,9966,9967],[9976,0.518,370,9966,9967],[9977,0.613,370,9966,9967],[9978,0.607,370,9966,9967],[9979,0.679,370,9966,9967]]},{"id":"space-precision-2026-paper","name":"SPACE · precision (2026 paper)","group":"视觉与视频","metric":"Mention-conditioned precision","unit":"score","public_history":true,"protocol":{"en":"Thirteen learned model/checkpoint rows describe the same subject-centered photographs from the 2,981-image SPACE resource. The evaluator labels each predefined fine-grained point correct, wrong or missing. Precision conditions on mentioned points: correct/(correct+wrong). It stays native 0–100 precision. H-SPACE-GRPO is a trained model, and Instruct/Thinking weights retain their source identities.","zh":"十三条模型权重结果描述SPACE资源中的相同人物照片；资源共2,981张图像。评判器对预定义细粒度要点标注正确、错误或遗漏。精确率以模型提及的要点为分母，计算正确/(正确+错误)，保留原生0–100精确率。H-SPACE-GRPO为训练后的模型，Instruct与Thinking权重保留原始身份。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.08300v1","published_at":"2026-09-08","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":13,"result_count":13,"results":[[345,72.4,370,9966,9967],[9968,81.8,370,9966,9967],[9969,82.5,370,9966,9967],[9970,70.2,370,9966,9967],[9971,67.5,370,9966,9967],[9972,72.6,370,9966,9967],[9973,64.3,370,9966,9967],[9974,63.9,370,9966,9967],[9975,70.6,370,9966,9967],[9976,73.3,370,9966,9967],[9977,75.5,370,9966,9967],[9978,77.7,370,9966,9967],[9979,83.6,370,9966,9967]]},{"id":"vi-bench-easy-2026-paper","name":"VI-Bench · easy (2026 paper)","group":"视觉与视频","metric":"Prompt/replay inversion score","unit":"score","public_history":true,"protocol":{"en":"Eighteen vision-language model rows invert the same 300 human-verified generated videos in the easy split. Inversion Score is the mean of prompt-fidelity and replay-fidelity scores. The source’s native 0–1 judge-derived composite is preserved. Shared generator/replay settings are fixed; model context and released checkpoints remain recorded.","zh":"十八条视觉语言模型结果反推同一组easy难度的300段人工验证生成视频。反推分为提示忠实度和重放忠实度的均值，保留原生0–1评判综合分。生成器与重放设置固定，模型上下文和发布权重均按原文保留。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.08079v1","published_at":"2026-09-08","status":"paper_baselines"},"score_max":1,"saturation_excluded":true,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"score_decimals":3,"model_count":18,"result_count":18,"results":[[8434,0.751,370,9980,9981],[353,0.731,370,9982,9981],[9983,0.722,370,9984,9981],[2166,0.723,370,9985,9981],[6342,0.733,370,9986,9981],[6299,0.72,370,9987,9981],[828,0.712,370,9988,9981],[6298,0.718,370,9989,9981],[6391,0.715,370,9990,9981],[2754,0.697,370,9991,9981],[3417,0.713,370,9992,9981],[9993,0.67,370,9991,9981],[9994,0.704,370,9995,9981],[2771,0.652,370,9996,9981],[9997,0.549,370,9998,9981],[820,0.543,370,9999,9981],[10000,0.563,370,10001,9981],[10002,0.555,370,10003,9981]]},{"id":"vi-bench-medium-2026-paper","name":"VI-Bench · medium (2026 paper)","group":"视觉与视频","metric":"Prompt/replay inversion score","unit":"score","public_history":true,"protocol":{"en":"Eighteen vision-language model rows invert the same 300 human-verified generated videos in the medium split. Inversion Score is the mean of prompt-fidelity and replay-fidelity scores. The source’s native 0–1 judge-derived composite is preserved. Shared generator/replay settings are fixed; model context and released checkpoints remain recorded.","zh":"十八条视觉语言模型结果反推同一组medium难度的300段人工验证生成视频。反推分为提示忠实度和重放忠实度的均值，保留原生0–1评判综合分。生成器与重放设置固定，模型上下文和发布权重均按原文保留。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.08079v1","published_at":"2026-09-08","status":"paper_baselines"},"score_max":1,"saturation_excluded":true,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"score_decimals":3,"model_count":18,"result_count":18,"results":[[8434,0.617,370,9980,9981],[353,0.57,370,9982,9981],[9983,0.556,370,9984,9981],[2166,0.557,370,9985,9981],[6342,0.571,370,9986,9981],[6299,0.564,370,9987,9981],[828,0.523,370,9988,9981],[6298,0.555,370,9989,9981],[6391,0.556,370,9990,9981],[2754,0.497,370,9991,9981],[3417,0.549,370,9992,9981],[9993,0.462,370,9991,9981],[9994,0.529,370,9995,9981],[2771,0.441,370,9996,9981],[9997,0.388,370,9998,9981],[820,0.39,370,9999,9981],[10000,0.332,370,10001,9981],[10002,0.343,370,10003,9981]]},{"id":"vi-bench-hard-2026-paper","name":"VI-Bench · hard (2026 paper)","group":"视觉与视频","metric":"Prompt/replay inversion score","unit":"score","public_history":true,"protocol":{"en":"Eighteen vision-language model rows invert the same 300 human-verified generated videos in the hard split. Inversion Score is the mean of prompt-fidelity and replay-fidelity scores. The source’s native 0–1 judge-derived composite is preserved. Shared generator/replay settings are fixed; model context and released checkpoints remain recorded.","zh":"十八条视觉语言模型结果反推同一组hard难度的300段人工验证生成视频。反推分为提示忠实度和重放忠实度的均值，保留原生0–1评判综合分。生成器与重放设置固定，模型上下文和发布权重均按原文保留。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.08079v1","published_at":"2026-09-08","status":"paper_baselines"},"score_max":1,"saturation_excluded":true,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"score_decimals":3,"model_count":18,"result_count":18,"results":[[8434,0.529,370,9980,9981],[353,0.441,370,9982,9981],[9983,0.45,370,9984,9981],[2166,0.426,370,9985,9981],[6342,0.366,370,9986,9981],[6299,0.365,370,9987,9981],[828,0.409,370,9988,9981],[6298,0.359,370,9989,9981],[6391,0.352,370,9990,9981],[2754,0.379,370,9991,9981],[3417,0.296,370,9992,9981],[9993,0.362,370,9991,9981],[9994,0.216,370,9995,9981],[2771,0.285,370,9996,9981],[9997,0.236,370,9998,9981],[820,0.192,370,9999,9981],[10000,0.252,370,10001,9981],[10002,0.133,370,10003,9981]]},{"id":"vi-bench-overall-2026-paper","name":"VI-Bench · overall (2026 paper)","group":"视觉与视频","metric":"Prompt/replay inversion score","unit":"score","public_history":true,"protocol":{"en":"Eighteen vision-language model rows invert the same 900 human-verified generated videos, 300 at each difficulty. Inversion Score is the mean of prompt-fidelity and replay-fidelity scores. The source’s native 0–1 judge-derived composite is preserved. Shared generator/replay settings are fixed; model context and released checkpoints remain recorded.","zh":"十八条视觉语言模型结果反推同一组900段人工验证生成视频，每级难度300段。反推分为提示忠实度和重放忠实度的均值，保留原生0–1评判综合分。生成器与重放设置固定，模型上下文和发布权重均按原文保留。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.08079v1","published_at":"2026-09-08","status":"paper_baselines"},"score_max":1,"saturation_excluded":true,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"score_decimals":3,"model_count":18,"result_count":18,"results":[[8434,0.632,370,9980,9981],[353,0.581,370,9982,9981],[9983,0.576,370,9984,9981],[2166,0.569,370,9985,9981],[6342,0.557,370,9986,9981],[6299,0.55,370,9987,9981],[828,0.548,370,9988,9981],[6298,0.544,370,9989,9981],[6391,0.541,370,9990,9981],[2754,0.524,370,9991,9981],[3417,0.519,370,9992,9981],[9993,0.498,370,9991,9981],[9994,0.483,370,9995,9981],[2771,0.459,370,9996,9981],[9997,0.391,370,9998,9981],[820,0.375,370,9999,9981],[10000,0.357,370,10001,9981],[10002,0.343,370,10003,9981]]},{"id":"5dialects-zero-native-bleu-2026-paper","name":"5-Dialects-BN · zero-native bleu (2026 paper)","group":"知识与推理","metric":"English-translation BLEU","unit":"score","public_history":true,"protocol":{"en":"Seven independent model checkpoints use the same fixed dialect-balanced 600-item prompting subset, approximately 100 per regional dialect or Standard Bangla. The selected zero-native condition fixes input script and demonstration regime. BLEU compares generated English translations with human references. Results are macro-averaged over the five dialects plus Standard Bangla and retain the published native 0–100 scale.","zh":"七个独立模型权重使用同一组600条方言均衡提示测试实例，五种地区方言和标准孟加拉语各约100条。zero-native条件固定输入文字和示例设置。BLEU将生成的英语翻译与人工参考译文比较。结果对五种方言及标准孟加拉语宏平均，保留原生0–100刻度。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.09964v1","published_at":"2026-09-09","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":7,"result_count":7,"results":[[3424,46.4,370,10004,10005],[354,38.4,370,10004,10005],[325,38.0,370,10004,10005],[7999,0.2,370,10004,10005],[10006,31.9,370,10004,10005],[587,42.5,370,10004,10005],[1921,16.2,370,10004,10005]]},{"id":"5dialects-zero-native-f1-2026-paper","name":"5-Dialects-BN · zero-native f1 (2026 paper)","group":"知识与推理","metric":"Subjectivity macro F1","unit":"score","public_history":true,"protocol":{"en":"Seven independent model checkpoints use the same fixed dialect-balanced 600-item prompting subset, approximately 100 per regional dialect or Standard Bangla. The selected zero-native condition fixes input script and demonstration regime. F1 measures subjective/objective classification. Results are macro-averaged over the five dialects plus Standard Bangla and retain the published native 0–100 scale.","zh":"七个独立模型权重使用同一组600条方言均衡提示测试实例，五种地区方言和标准孟加拉语各约100条。zero-native条件固定输入文字和示例设置。F1衡量主观或客观分类表现。结果对五种方言及标准孟加拉语宏平均，保留原生0–100刻度。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.09964v1","published_at":"2026-09-09","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":7,"result_count":7,"results":[[3424,69.6,370,10004,10005],[354,54.8,370,10004,10005],[325,66.2,370,10004,10005],[7999,41.3,370,10004,10005],[10006,65.1,370,10004,10005],[587,70.6,370,10004,10005],[1921,32.8,370,10004,10005]]},{"id":"5dialects-zero-romanized-bleu-2026-paper","name":"5-Dialects-BN · zero-romanized bleu (2026 paper)","group":"知识与推理","metric":"English-translation BLEU","unit":"score","public_history":true,"protocol":{"en":"Seven independent model checkpoints use the same fixed dialect-balanced 600-item prompting subset, approximately 100 per regional dialect or Standard Bangla. The selected zero-romanized condition fixes input script and demonstration regime. BLEU compares generated English translations with human references. Results are macro-averaged over the five dialects plus Standard Bangla and retain the published native 0–100 scale.","zh":"七个独立模型权重使用同一组600条方言均衡提示测试实例，五种地区方言和标准孟加拉语各约100条。zero-romanized条件固定输入文字和示例设置。BLEU将生成的英语翻译与人工参考译文比较。结果对五种方言及标准孟加拉语宏平均，保留原生0–100刻度。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.09964v1","published_at":"2026-09-09","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":7,"result_count":7,"results":[[3424,40.1,370,10007,10005],[354,26.4,370,10007,10005],[325,24.6,370,10007,10005],[7999,0.0,370,10007,10005],[10006,21.1,370,10007,10005],[587,26.9,370,10007,10005],[1921,7.8,370,10007,10005]]},{"id":"5dialects-zero-romanized-f1-2026-paper","name":"5-Dialects-BN · zero-romanized f1 (2026 paper)","group":"知识与推理","metric":"Subjectivity macro F1","unit":"score","public_history":true,"protocol":{"en":"Seven independent model checkpoints use the same fixed dialect-balanced 600-item prompting subset, approximately 100 per regional dialect or Standard Bangla. The selected zero-romanized condition fixes input script and demonstration regime. F1 measures subjective/objective classification. Results are macro-averaged over the five dialects plus Standard Bangla and retain the published native 0–100 scale.","zh":"七个独立模型权重使用同一组600条方言均衡提示测试实例，五种地区方言和标准孟加拉语各约100条。zero-romanized条件固定输入文字和示例设置。F1衡量主观或客观分类表现。结果对五种方言及标准孟加拉语宏平均，保留原生0–100刻度。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.09964v1","published_at":"2026-09-09","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":7,"result_count":7,"results":[[3424,67.0,370,10007,10005],[354,54.3,370,10007,10005],[325,64.0,370,10007,10005],[7999,42.4,370,10007,10005],[10006,59.7,370,10007,10005],[587,67.0,370,10007,10005],[1921,34.8,370,10007,10005]]},{"id":"5dialects-few-native-bleu-2026-paper","name":"5-Dialects-BN · few-native bleu (2026 paper)","group":"知识与推理","metric":"English-translation BLEU","unit":"score","public_history":true,"protocol":{"en":"Seven independent model checkpoints use the same fixed dialect-balanced 600-item prompting subset, approximately 100 per regional dialect or Standard Bangla. The selected few-native condition fixes input script and demonstration regime. BLEU compares generated English translations with human references. Results are macro-averaged over the five dialects plus Standard Bangla and retain the published native 0–100 scale.","zh":"七个独立模型权重使用同一组600条方言均衡提示测试实例，五种地区方言和标准孟加拉语各约100条。few-native条件固定输入文字和示例设置。BLEU将生成的英语翻译与人工参考译文比较。结果对五种方言及标准孟加拉语宏平均，保留原生0–100刻度。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.09964v1","published_at":"2026-09-09","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":7,"result_count":7,"results":[[3424,44.9,370,10008,10005],[354,42.2,370,10008,10005],[325,41.0,370,10008,10005],[7999,27.5,370,10008,10005],[10006,30.7,370,10008,10005],[587,33.2,370,10008,10005],[1921,19.3,370,10008,10005]]},{"id":"5dialects-few-native-f1-2026-paper","name":"5-Dialects-BN · few-native f1 (2026 paper)","group":"知识与推理","metric":"Subjectivity macro F1","unit":"score","public_history":true,"protocol":{"en":"Seven independent model checkpoints use the same fixed dialect-balanced 600-item prompting subset, approximately 100 per regional dialect or Standard Bangla. The selected few-native condition fixes input script and demonstration regime. F1 measures subjective/objective classification. Results are macro-averaged over the five dialects plus Standard Bangla and retain the published native 0–100 scale.","zh":"七个独立模型权重使用同一组600条方言均衡提示测试实例，五种地区方言和标准孟加拉语各约100条。few-native条件固定输入文字和示例设置。F1衡量主观或客观分类表现。结果对五种方言及标准孟加拉语宏平均，保留原生0–100刻度。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.09964v1","published_at":"2026-09-09","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":7,"result_count":7,"results":[[3424,63.9,370,10008,10005],[354,65.0,370,10008,10005],[325,59.0,370,10008,10005],[7999,55.3,370,10008,10005],[10006,66.3,370,10008,10005],[587,61.5,370,10008,10005],[1921,61.4,370,10008,10005]]},{"id":"5dialects-few-romanized-bleu-2026-paper","name":"5-Dialects-BN · few-romanized bleu (2026 paper)","group":"知识与推理","metric":"English-translation BLEU","unit":"score","public_history":true,"protocol":{"en":"Seven independent model checkpoints use the same fixed dialect-balanced 600-item prompting subset, approximately 100 per regional dialect or Standard Bangla. The selected few-romanized condition fixes input script and demonstration regime. BLEU compares generated English translations with human references. Results are macro-averaged over the five dialects plus Standard Bangla and retain the published native 0–100 scale.","zh":"七个独立模型权重使用同一组600条方言均衡提示测试实例，五种地区方言和标准孟加拉语各约100条。few-romanized条件固定输入文字和示例设置。BLEU将生成的英语翻译与人工参考译文比较。结果对五种方言及标准孟加拉语宏平均，保留原生0–100刻度。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.09964v1","published_at":"2026-09-09","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":7,"result_count":7,"results":[[3424,36.5,370,10009,10005],[354,25.5,370,10009,10005],[325,29.9,370,10009,10005],[7999,8.5,370,10009,10005],[10006,28.1,370,10009,10005],[587,13.8,370,10009,10005],[1921,11.3,370,10009,10005]]},{"id":"5dialects-few-romanized-f1-2026-paper","name":"5-Dialects-BN · few-romanized f1 (2026 paper)","group":"知识与推理","metric":"Subjectivity macro F1","unit":"score","public_history":true,"protocol":{"en":"Seven independent model checkpoints use the same fixed dialect-balanced 600-item prompting subset, approximately 100 per regional dialect or Standard Bangla. The selected few-romanized condition fixes input script and demonstration regime. F1 measures subjective/objective classification. Results are macro-averaged over the five dialects plus Standard Bangla and retain the published native 0–100 scale.","zh":"七个独立模型权重使用同一组600条方言均衡提示测试实例，五种地区方言和标准孟加拉语各约100条。few-romanized条件固定输入文字和示例设置。F1衡量主观或客观分类表现。结果对五种方言及标准孟加拉语宏平均，保留原生0–100刻度。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.09964v1","published_at":"2026-09-09","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":7,"result_count":7,"results":[[3424,61.9,370,10009,10005],[354,63.0,370,10009,10005],[325,57.0,370,10009,10005],[7999,53.3,370,10009,10005],[10006,64.3,370,10009,10005],[587,59.5,370,10009,10005],[1921,59.4,370,10009,10005]]},{"id":"metrollm-tier1-2026-paper","name":"MetroLLM-Bench · tier1 (2026 paper)","group":"Agent 与工具","metric":"Tier 1 deterministic component score","unit":"score","public_history":true,"protocol":{"en":"The complete 25-row model/configuration leaderboard uses the identical 238 held-out kiosk cases from the 955-case six-metro-system resource. Tier 1 averages 14 deterministic scoring components. Component-weighted fractional scores stay native 0–100. PEFT rows average independent training seeds (three at 2B, two otherwise); API effort and per-model serving settings remain explicit.","zh":"完整25条模型及配置排行使用同一组238项留出地铁售票终端任务，原资源覆盖六套地铁系统、共955项任务。Tier 1汇总14项确定性评分分量。按分量加权的分数保留原生0–100刻度。PEFT取独立训练种子均值，2B使用三个种子，其余使用两个；API思考强度和各模型服务配置均注明。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.10016v1","published_at":"2026-09-09","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":19,"result_count":25,"results":[[5320,92.75,370,10010,10011],[5300,92.2,370,10012,10011],[3763,93.63,370,10013,10011],[2584,92.32,370,10014,10011],[5386,90.63,370,10015,10011],[10016,91.37,370,10017,10011],[2590,92.12,370,10018,10011],[3160,90.0,370,10019,10011],[2584,91.41,370,10020,10011],[4969,91.32,370,10021,10011],[4070,91.03,370,10022,10011],[10016,89.48,370,10023,10011],[4070,89.38,370,10024,10011],[8969,90.45,370,10025,10011],[10016,89.17,370,10026,10011],[4969,89.32,370,10027,10011],[4966,89.67,370,10028,10011],[6337,87.1,370,10029,10011],[4958,87.57,370,10030,10011],[8968,87.33,370,10031,10011],[5934,85.23,370,10032,10011],[5828,81.2,370,10033,10011],[5828,74.17,370,10034,10011],[6340,61.93,370,10035,10011],[10036,57.5,370,10037,10011]]},{"id":"metrollm-combined-2026-paper","name":"MetroLLM-Bench · combined (2026 paper)","group":"Agent 与工具","metric":"Combined Tier 1/2 score","unit":"score","public_history":true,"protocol":{"en":"The complete 25-row model/configuration leaderboard uses the identical 238 held-out kiosk cases from the 955-case six-metro-system resource. The composite combines 14 deterministic and eight semantic-quality components; six semantic components use a pinned Claude-Haiku judge. Component-weighted fractional scores stay native 0–100. PEFT rows average independent training seeds (three at 2B, two otherwise); API effort and per-model serving settings remain explicit.","zh":"完整25条模型及配置排行使用同一组238项留出地铁售票终端任务，原资源覆盖六套地铁系统、共955项任务。综合分汇总14项确定性分量和八项语义质量分量，其中六项由固定Claude-Haiku版本评判。按分量加权的分数保留原生0–100刻度。PEFT取独立训练种子均值，2B使用三个种子，其余使用两个；API思考强度和各模型服务配置均注明。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.10016v1","published_at":"2026-09-09","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":19,"result_count":25,"results":[[5320,92.03,370,10010,10011],[5300,91.83,370,10012,10011],[3763,91.28,370,10013,10011],[2584,90.6,370,10014,10011],[5386,90.57,370,10015,10011],[10016,90.45,370,10017,10011],[2590,89.9,370,10018,10011],[3160,89.82,370,10019,10011],[2584,89.72,370,10020,10011],[4969,89.12,370,10021,10011],[4070,88.85,370,10022,10011],[10016,88.2,370,10023,10011],[4070,88.05,370,10024,10011],[8969,87.82,370,10025,10011],[10016,87.72,370,10026,10011],[4969,87.25,370,10027,10011],[4966,86.73,370,10028,10011],[6337,86.58,370,10029,10011],[4958,86.37,370,10030,10011],[8968,85.68,370,10031,10011],[5934,81.98,370,10032,10011],[5828,79.69,370,10033,10011],[5828,71.9,370,10034,10011],[6340,59.98,370,10035,10011],[10036,56.67,370,10037,10011]]},{"id":"drivemotion-mpjpe-2026-paper","name":"DriveMotion · mpjpe (2026 paper)","group":"视觉与视频","metric":"Macro MPJPE at 4 seconds (0.01 shoulder widths)","unit":"raw","public_history":true,"protocol":{"en":"The full applicable 17 learned-model/configuration rows use the same 16,638 identity-disjoint Protocol A test windows: 6,138 pre-maneuver, 6,238 post-maneuver and 4,262 route-matched stable windows. The selected macro score gives the three strata equal weight. MPJPE retains the benchmark’s units of 0.01 shoulder widths; lower is better. Pose-only, exterior-context and maneuver-enriched training remain explicit configurations. The reduced-subset RL row and zero-motion control are excluded.","zh":"完整17条适用模型及配置结果使用相同16,638个身份隔离的A协议测试窗口，包含机动前6,138个、机动后6,238个及路线匹配稳定窗口4,262个。宏分对三层等权。MPJPE保留0.01倍肩宽单位，越低越好。仅姿态、外部场景和机动富集训练均注明配置；排除使用缩小子集的RL结果和零运动参照。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.08117v1","published_at":"2026-09-08","status":"paper_baselines"},"higher_is_better":false,"model_count":17,"result_count":17,"results":[[10038,6.83,370,10039,10040],[10041,6.85,370,10042,10040],[10043,6.75,370,10044,10040],[10045,6.95,370,10046,10040],[10047,6.63,370,10048,10040],[10049,6.62,370,10050,10040],[10051,6.84,370,10052,10040],[10053,8.91,370,10054,10040],[10055,8.91,370,10056,10040],[10057,8.91,370,10058,10040],[10059,8.86,370,10060,10040],[10061,8.14,370,10062,10040],[10063,8.16,370,10064,10040],[10065,8.12,370,10066,10040],[10067,8.33,370,10068,10040],[10069,8.24,370,10070,10040],[10071,8.24,370,10072,10040]]},{"id":"drivemotion-statef1-2026-paper","name":"DriveMotion · statef1 (2026 paper)","group":"视觉与视频","metric":"Macro state F1 at 2 seconds","unit":"score","public_history":true,"protocol":{"en":"The full applicable 17 learned-model/configuration rows use the same 16,638 identity-disjoint Protocol A test windows: 6,138 pre-maneuver, 6,238 post-maneuver and 4,262 route-matched stable windows. The selected macro score gives the three strata equal weight. Forecast-derived head/torso/arm state F1 stays on its native 0–1 scale. Pose-only, exterior-context and maneuver-enriched training remain explicit configurations. The reduced-subset RL row and zero-motion control are excluded.","zh":"完整17条适用模型及配置结果使用相同16,638个身份隔离的A协议测试窗口，包含机动前6,138个、机动后6,238个及路线匹配稳定窗口4,262个。宏分对三层等权。预测推导的头部、躯干与手臂状态F1保留原生0–1刻度。仅姿态、外部场景和机动富集训练均注明配置；排除使用缩小子集的RL结果和零运动参照。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.08117v1","published_at":"2026-09-08","status":"paper_baselines"},"score_max":1,"saturation_excluded":true,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"score_decimals":3,"model_count":17,"result_count":17,"results":[[10038,0.275,370,10039,10040],[10041,0.227,370,10042,10040],[10043,0.282,370,10044,10040],[10045,0.309,370,10046,10040],[10047,0.287,370,10048,10040],[10049,0.284,370,10050,10040],[10051,0.257,370,10052,10040],[10053,0.298,370,10054,10040],[10055,0.298,370,10056,10040],[10057,0.297,370,10058,10040],[10059,0.299,370,10060,10040],[10061,0.452,370,10062,10040],[10063,0.458,370,10064,10040],[10065,0.452,370,10066,10040],[10067,0.452,370,10068,10040],[10069,0.457,370,10070,10040],[10071,0.457,370,10072,10040]]},{"id":"scifigbench-content-2026-paper","name":"SciFigBench · Dense content ANLS (2026 paper)","group":"数学与科学","metric":"Dense content ANLS","unit":"score","public_history":true,"protocol":{"en":"Fifteen open-source configurations use the full 40,670-QA test suite. The panel has thirteen independent base checkpoints plus two SciFigCue configurations; the four proprietary models use a different 2,000-QA subset and are excluded. DCP uses position-wise ANLS, IFT uses set-based ANLS-star, and Edge, Degree and Reach use accuracy.","zh":"十五个开源配置使用完整40,670道问答测试集，包含十三个独立底模检查点及两个SciFigCue配置。四个闭源模型仅测试另一个2,000题子集，分别处理。DCP按位置计算ANLS，IFT采用集合ANLS-star，Edge、Degree和Reach为准确率。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.27087v1","published_at":"2026-10-19","status":"paper_baselines"},"score_max":100,"score_decimals":2,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":15,"result_count":15,"results":[[6246,32.5,370,10073,10074],[2654,45.9,370,10073,10074],[6345,53.7,370,10073,10074],[2642,52.2,370,10073,10074],[10075,46.0,370,10073,10074],[10076,54.8,370,10073,10074],[6298,51.6,370,10073,10074],[6342,55.1,370,10073,10074],[6343,62.6,370,10073,10074],[10077,19.8,370,10073,10074],[10078,45.1,370,10073,10074],[9136,61.2,370,10073,10074],[10079,70.9,370,10073,10074],[10080,73.0,370,10081,10074],[10082,82.4,370,10081,10074]]},{"id":"scifigbench-edge-2026-paper","name":"SciFigBench · Edge verification accuracy (2026 paper)","group":"数学与科学","metric":"Edge verification accuracy","unit":"%","public_history":true,"protocol":{"en":"Fifteen open-source configurations use the full 40,670-QA test suite. The panel has thirteen independent base checkpoints plus two SciFigCue configurations; the four proprietary models use a different 2,000-QA subset and are excluded. DCP uses position-wise ANLS, IFT uses set-based ANLS-star, and Edge, Degree and Reach use accuracy.","zh":"十五个开源配置使用完整40,670道问答测试集，包含十三个独立底模检查点及两个SciFigCue配置。四个闭源模型仅测试另一个2,000题子集，分别处理。DCP按位置计算ANLS，IFT采用集合ANLS-star，Edge、Degree和Reach为准确率。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.27087v1","published_at":"2026-10-19","status":"paper_baselines"},"model_count":15,"result_count":15,"results":[[6246,0.364,370,10073,10074],[2654,0.447,370,10073,10074],[6345,0.47700000000000004,370,10073,10074],[2642,0.478,370,10073,10074],[10075,0.44,370,10073,10074],[10076,0.45399999999999996,370,10073,10074],[6298,0.486,370,10073,10074],[6342,0.489,370,10073,10074],[6343,0.54,370,10073,10074],[10077,0.462,370,10073,10074],[10078,0.488,370,10073,10074],[9136,0.508,370,10073,10074],[10079,0.5920000000000001,370,10073,10074],[10080,0.5660000000000001,370,10081,10074],[10082,0.701,370,10081,10074]]},{"id":"scifigbench-degree-2026-paper","name":"SciFigBench · Degree counting accuracy (2026 paper)","group":"数学与科学","metric":"Degree counting accuracy","unit":"%","public_history":true,"protocol":{"en":"Fifteen open-source configurations use the full 40,670-QA test suite. The panel has thirteen independent base checkpoints plus two SciFigCue configurations; the four proprietary models use a different 2,000-QA subset and are excluded. DCP uses position-wise ANLS, IFT uses set-based ANLS-star, and Edge, Degree and Reach use accuracy.","zh":"十五个开源配置使用完整40,670道问答测试集，包含十三个独立底模检查点及两个SciFigCue配置。四个闭源模型仅测试另一个2,000题子集，分别处理。DCP按位置计算ANLS，IFT采用集合ANLS-star，Edge、Degree和Reach为准确率。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.27087v1","published_at":"2026-10-19","status":"paper_baselines"},"model_count":15,"result_count":15,"results":[[6246,0.31,370,10073,10074],[2654,0.358,370,10073,10074],[6345,0.35700000000000004,370,10073,10074],[2642,0.409,370,10073,10074],[10075,0.33,370,10073,10074],[10076,0.345,370,10073,10074],[6298,0.389,370,10073,10074],[6342,0.39299999999999996,370,10073,10074],[6343,0.46399999999999997,370,10073,10074],[10077,0.22,370,10073,10074],[10078,0.375,370,10073,10074],[9136,0.414,370,10073,10074],[10079,0.509,370,10073,10074],[10080,0.441,370,10081,10074],[10082,0.511,370,10081,10074]]},{"id":"scifigbench-reach-2026-paper","name":"SciFigBench · Reachability accuracy (2026 paper)","group":"数学与科学","metric":"Reachability accuracy","unit":"%","public_history":true,"protocol":{"en":"Fifteen open-source configurations use the full 40,670-QA test suite. The panel has thirteen independent base checkpoints plus two SciFigCue configurations; the four proprietary models use a different 2,000-QA subset and are excluded. DCP uses position-wise ANLS, IFT uses set-based ANLS-star, and Edge, Degree and Reach use accuracy.","zh":"十五个开源配置使用完整40,670道问答测试集，包含十三个独立底模检查点及两个SciFigCue配置。四个闭源模型仅测试另一个2,000题子集，分别处理。DCP按位置计算ANLS，IFT采用集合ANLS-star，Edge、Degree和Reach为准确率。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.27087v1","published_at":"2026-10-19","status":"paper_baselines"},"model_count":15,"result_count":15,"results":[[6246,0.46399999999999997,370,10073,10074],[2654,0.415,370,10073,10074],[6345,0.46299999999999997,370,10073,10074],[2642,0.45399999999999996,370,10073,10074],[10075,0.418,370,10073,10074],[10076,0.402,370,10073,10074],[6298,0.446,370,10073,10074],[6342,0.461,370,10073,10074],[6343,0.494,370,10073,10074],[10077,0.49200000000000005,370,10073,10074],[10078,0.461,370,10073,10074],[9136,0.46299999999999997,370,10073,10074],[10079,0.502,370,10073,10074],[10080,0.52,370,10081,10074],[10082,0.56,370,10081,10074]]},{"id":"scifigbench-flow-2026-paper","name":"SciFigBench · Flow tracing ANLS-star (2026 paper)","group":"数学与科学","metric":"Flow tracing ANLS-star","unit":"score","public_history":true,"protocol":{"en":"Fifteen open-source configurations use the full 40,670-QA test suite. The panel has thirteen independent base checkpoints plus two SciFigCue configurations; the four proprietary models use a different 2,000-QA subset and are excluded. DCP uses position-wise ANLS, IFT uses set-based ANLS-star, and Edge, Degree and Reach use accuracy.","zh":"十五个开源配置使用完整40,670道问答测试集，包含十三个独立底模检查点及两个SciFigCue配置。四个闭源模型仅测试另一个2,000题子集，分别处理。DCP按位置计算ANLS，IFT采用集合ANLS-star，Edge、Degree和Reach为准确率。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.27087v1","published_at":"2026-10-19","status":"paper_baselines"},"score_max":100,"score_decimals":2,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":15,"result_count":15,"results":[[6246,17.6,370,10073,10074],[2654,27.2,370,10073,10074],[6345,26.1,370,10073,10074],[2642,26.4,370,10073,10074],[10075,24.6,370,10073,10074],[10076,24.5,370,10073,10074],[6298,27.5,370,10073,10074],[6342,31.3,370,10073,10074],[6343,35.4,370,10073,10074],[10077,16.3,370,10073,10074],[10078,26.3,370,10073,10074],[9136,32.8,370,10073,10074],[10079,43.6,370,10073,10074],[10080,36.0,370,10081,10074],[10082,42.1,370,10081,10074]]},{"id":"finhopper-tools-micro-2026-paper","name":"FinHopper · tools, Chinese (2026 paper)","group":"专业工作","metric":"Process-aware micro score","unit":"score","public_history":true,"protocol":{"en":"Ten model families, with thinking and non-thinking configurations retained, use the same 200 Chinese financial retrieval questions across one to six hops. ReAct uses public search and at most 42 iterations. A correct normalized final answer scores 1; an incorrect final answer receives up to 0.5 for entailed gold intermediate answers. The question-weighted micro score is a native composite on the reported 0–100 scale.","zh":"十个模型家族保留思考与非思考配置，回答相同200道一至六跳中文金融检索题。ReAct使用公开搜索，最多42轮。归一化后最终答案正确得1分；错误时按正确中间答案给予最高0.5分。保留按问题加权的micro原生综合分，范围0–100。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.26036v1","published_at":"2026-10-19","status":"paper_baselines"},"score_max":100,"score_decimals":2,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":20,"result_count":20,"results":[[10083,67.3,370,10084,10085],[10086,63.7,370,10087,10085],[10088,65.0,370,10084,10085],[10089,62.7,370,10087,10085],[10090,61.3,370,10084,10085],[340,58.5,370,10087,10085],[10091,61.1,370,10084,10085],[10092,59.2,370,10087,10085],[10093,58.1,370,10084,10085],[10094,57.1,370,10087,10085],[10095,57.5,370,10084,10085],[10096,54.6,370,10087,10085],[10097,57.3,370,10084,10085],[10098,52.7,370,10087,10085],[10099,52.7,370,10084,10085],[3425,42.7,370,10087,10085],[10100,52.5,370,10084,10085],[10101,48.3,370,10087,10085],[10102,49.0,370,10084,10085],[10103,48.8,370,10087,10085]]},{"id":"instancebench-all07-2026-paper","name":"InstanceBench · paired consistency (2026 paper)","group":"视觉与视频","metric":"All@0.7 target-set success","unit":"%","public_history":true,"protocol":{"en":"Twenty-two released checkpoints from eighteen families use 2,000 images, 2,918 target-consistent expression sets and 5,836 core expressions. All@0.7 requires every expression in a target set to achieve mask IoU at least 0.7, rather than averaging expression IoUs. X2SAM and X-SAM average three full-test runs; the other checkpoints are deterministic. Additional dense diagnostic expressions are not mixed into this core panel.","zh":"十八个家族的二十二个已发布检查点使用2,000幅图像、2,918组同目标表达及5,836个核心表达。All@0.7要求一组目标的全部表达都达到至少0.7的掩码IoU。X2SAM与X-SAM取三个完整测试运行的均值，其余检查点确定性运行；额外密集诊断表达不混入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.09478v1","published_at":"2026-10-07","status":"paper_baselines"},"model_count":22,"result_count":22,"results":[[10104,0.596,370,10105,10106],[10107,0.5660000000000001,370,10105,10106],[10108,0.5182,370,10105,10106],[10109,0.4537,370,10105,10106],[10110,0.466,370,10105,10106],[10111,0.3907,370,10105,10106],[10112,0.3975,370,10105,10106],[10113,0.2776,370,10105,10106],[10114,0.4304,370,10105,10106],[10115,0.3907,370,10105,10106],[10116,0.3119,370,10105,10106],[10117,0.0199,370,10105,10106],[10118,0.0199,370,10105,10106],[10119,0.0182,370,10105,10106],[10120,0.3009,370,10105,10106],[10121,0.2995,370,10105,10106],[10122,0.218,370,10105,10106],[10123,0.1467,370,10105,10106],[10124,0.1295,370,10105,10106],[10125,0.1508,370,10105,10106],[10126,0.1892,370,10105,10106],[10127,0.1532,370,10105,10106]]},{"id":"textrenderbench-zh-f1-2026-paper","name":"TextRenderBench-ZH (2026 paper)","group":"视觉与视频","metric":"Rendering-error detection F1","unit":"score","public_history":true,"protocol":{"en":"Seven detection configurations score the same 7,358 human-annotated Chinese characters: 642 errors and 6,716 correct renderings. F1 is the harmonic mean of error precision and recall. PP-OCRv6 uses its fixed 0.90 confidence threshold; its test-tuned best-threshold oracle and TextVerifier training-stage ablations are excluded.","zh":"七个检测配置判断相同7,358个经人工标注的中文字符，其中642个渲染错误、6,716个正确。F1为错误检测精确率与召回率的调和均值。PP-OCRv6固定置信阈值0.90，测试集调优阈值上界及TextVerifier训练阶段消融另列。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.19551v1","published_at":"2026-10-14","status":"paper_baselines"},"score_max":1,"score_decimals":2,"saturation_excluded":true,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":7,"result_count":7,"results":[[10128,0.444,370,10129,10130],[3157,0.198,370,10129,10130],[4048,0.101,370,10129,10130],[3386,0.329,370,10129,10130],[10131,0.59,370,10129,10130],[10132,0.433,370,10129,10130],[10133,0.844,370,10134,10130]]},{"id":"pvbench-t2v-geometry-2026-paper","name":"PVBench · t2v geometry (2026 paper)","group":"生成与创作","metric":"Geometric and spatial consistency","unit":"score","score_max":100,"score_decimals":2,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"public_history":true,"protocol":{"en":"Six video generators each use the same 600 prompt groups at 720p, ten seconds and fixed seed. Geometry and spatial consistency is the published GS domain composite on a native 0–100 scale, using specialized 3D/vision measurements, fixed thresholds and selected MLLM judgments. Text-to-video and image-to-video are separate tracks.","zh":"六个视频生成器逐一使用相同600组提示，统一720p、十秒与固定种子。采用原表GS几何与空间一致性分，基于专用三维/视觉测量、固定阈值及部分MLLM判断，保留0–100原生综合分。文生视频和图生视频分别展示。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.26711v1","published_at":"2026-10-19","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[10135,47.0,370,10136,10137],[10138,50.6,370,10136,10137],[10139,49.2,370,10136,10137],[10140,42.8,370,10136,10137],[8930,52.6,370,10136,10137],[8934,52.2,370,10136,10137]]},{"id":"pvbench-i2v-geometry-2026-paper","name":"PVBench · i2v geometry (2026 paper)","group":"生成与创作","metric":"Geometric and spatial consistency","unit":"score","score_max":100,"score_decimals":2,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"public_history":true,"protocol":{"en":"Six video generators each use the same 600 prompt groups at 720p, ten seconds and fixed seed. Geometry and spatial consistency is the published GS domain composite on a native 0–100 scale, using specialized 3D/vision measurements, fixed thresholds and selected MLLM judgments. Text-to-video and image-to-video are separate tracks.","zh":"六个视频生成器逐一使用相同600组提示，统一720p、十秒与固定种子。采用原表GS几何与空间一致性分，基于专用三维/视觉测量、固定阈值及部分MLLM判断，保留0–100原生综合分。文生视频和图生视频分别展示。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.26711v1","published_at":"2026-10-19","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[10135,35.8,370,10141,10137],[10138,40.9,370,10141,10137],[10139,45.8,370,10141,10137],[10140,50.8,370,10141,10137],[8930,48.9,370,10141,10137],[8934,50.3,370,10141,10137]]},{"id":"omnicompass-vl-32-pairedqa-2026-paper","name":"OmniCompass · VL, 32 frames (2026 paper)","group":"视觉与视频","metric":"Paired QA consistency success","unit":"%","public_history":true,"protocol":{"en":"All models use the same 820 five-option QA cases and 32 uniformly sampled timestamped frames. CS-QA counts success only when both Text and Text+Image predictions equal the common gold answer. VL excludes audio; AV includes synchronized native audio. Frame budgets and tracks remain separate, avoiding the paper’s mixed-budget overall panel.","zh":"各模型使用同一组820道五选一问答和32个均匀采样且带时间戳的帧。CS-QA要求Text与Text+Image的两个预测都等于共同标准答案才计成功。VL不带音频，AV包含同步原生音频；帧数预算及赛道分别呈现。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.18829v1","published_at":"2026-10-13","status":"paper_baselines"},"model_count":17,"result_count":17,"results":[[5300,0.557,370,10142,10143],[3764,0.502,370,10142,10143],[3763,0.5479999999999999,370,10142,10143],[2590,0.519,370,10142,10143],[2584,0.539,370,10142,10143],[4070,0.501,370,10142,10143],[4969,0.466,370,10142,10143],[5485,0.434,370,10142,10143],[6150,0.512,370,10142,10143],[6149,0.502,370,10142,10143],[2642,0.524,370,10142,10143],[10144,0.53,370,10142,10143],[10145,0.51,370,10142,10143],[7386,0.495,370,10142,10143],[7485,0.48,370,10142,10143],[7803,0.456,370,10142,10143],[6389,0.423,370,10142,10143]]},{"id":"omnicompass-vl-64-pairedqa-2026-paper","name":"OmniCompass · VL, 64 frames (2026 paper)","group":"视觉与视频","metric":"Paired QA consistency success","unit":"%","public_history":true,"protocol":{"en":"All models use the same 820 five-option QA cases and 64 uniformly sampled timestamped frames. CS-QA counts success only when both Text and Text+Image predictions equal the common gold answer. VL excludes audio; AV includes synchronized native audio. Frame budgets and tracks remain separate, avoiding the paper’s mixed-budget overall panel.","zh":"各模型使用同一组820道五选一问答和64个均匀采样且带时间戳的帧。CS-QA要求Text与Text+Image的两个预测都等于共同标准答案才计成功。VL不带音频，AV包含同步原生音频；帧数预算及赛道分别呈现。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.18829v1","published_at":"2026-10-13","status":"paper_baselines"},"model_count":14,"result_count":14,"results":[[5519,0.691,370,10146,10143],[502,0.657,370,10146,10143],[10147,0.674,370,10146,10143],[10148,0.677,370,10146,10143],[10149,0.6729999999999999,370,10146,10143],[3157,0.672,370,10146,10143],[5899,0.624,370,10146,10143],[3155,0.6759999999999999,370,10146,10143],[2576,0.606,370,10146,10143],[2582,0.5539999999999999,370,10146,10143],[2613,0.61,370,10146,10143],[3396,0.594,370,10146,10143],[3153,0.5660000000000001,370,10146,10143],[7223,0.49,370,10146,10143]]},{"id":"omnicompass-av-32-pairedqa-2026-paper","name":"OmniCompass · AV, 32 frames (2026 paper)","group":"视觉与视频","metric":"Paired QA consistency success","unit":"%","public_history":true,"protocol":{"en":"All models use the same 240 five-option QA cases and 32 uniformly sampled timestamped frames. CS-QA counts success only when both Text and Text+Image predictions equal the common gold answer. VL excludes audio; AV includes synchronized native audio. Frame budgets and tracks remain separate, avoiding the paper’s mixed-budget overall panel.","zh":"各模型使用同一组240道五选一问答和32个均匀采样且带时间戳的帧。CS-QA要求Text与Text+Image的两个预测都等于共同标准答案才计成功。VL不带音频，AV包含同步原生音频；帧数预算及赛道分别呈现。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.18829v1","published_at":"2026-10-13","status":"paper_baselines"},"model_count":8,"result_count":8,"results":[[10150,0.479,370,10151,10152],[6138,0.43799999999999994,370,10151,10152],[6136,0.45,370,10151,10152],[10153,0.47100000000000003,370,10151,10152],[8748,0.508,370,10151,10152],[8746,0.429,370,10151,10152],[7384,0.375,370,10151,10152],[10154,0.233,370,10151,10152]]},{"id":"omnicompass-av-64-pairedqa-2026-paper","name":"OmniCompass · AV, 64 frames (2026 paper)","group":"视觉与视频","metric":"Paired QA consistency success","unit":"%","public_history":true,"protocol":{"en":"All models use the same 240 five-option QA cases and 64 uniformly sampled timestamped frames. CS-QA counts success only when both Text and Text+Image predictions equal the common gold answer. VL excludes audio; AV includes synchronized native audio. Frame budgets and tracks remain separate, avoiding the paper’s mixed-budget overall panel.","zh":"各模型使用同一组240道五选一问答和64个均匀采样且带时间戳的帧。CS-QA要求Text与Text+Image的两个预测都等于共同标准答案才计成功。VL不带音频，AV包含同步原生音频；帧数预算及赛道分别呈现。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.18829v1","published_at":"2026-10-13","status":"paper_baselines"},"model_count":8,"result_count":8,"results":[[5519,0.782,370,10155,10152],[502,0.649,370,10155,10152],[6725,0.732,370,10155,10152],[6724,0.684,370,10155,10152],[6723,0.58,370,10155,10152],[10156,0.638,370,10155,10152],[10157,0.5329999999999999,370,10155,10152],[7223,0.473,370,10155,10152]]},{"id":"bidex100-id-success-2026-paper","name":"BiDex-100 · ID (2026 paper)","group":"预测与决策","metric":"Strict task success rate","unit":"%","public_history":true,"protocol":{"en":"Seven VLA policy architectures share a 58-DoF interface and a training mixture over all 106 tasks. Each model executes fifty rollouts per task and domain, from two fixed seeds and twenty-five episodes per seed, for 5,300 rollouts in each track. The first Overall value is strict success, distinct from the separate progress score. Model-specific official training schedules are retained.","zh":"七种VLA策略共享58自由度接口和包含全部106任务的训练混合。每个模型在每个域执行两个固定种子、每种子25回合，即每任务50次、每域共5,300次。采用Overall前一个严格成功率，与后一个进度分区分，保留各模型官方训练日程。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.25459v1","published_at":"2026-10-18","status":"paper_baselines"},"model_count":7,"result_count":7,"results":[[10158,0.5221,370,10159,10160],[8323,0.504,370,10159,10160],[10161,0.4232,370,10159,10160],[10162,0.2306,370,10159,10160],[7400,0.2168,370,10159,10160],[10163,0.044000000000000004,370,10159,10160],[7396,0.0151,370,10159,10160]]},{"id":"bidex100-ood-success-2026-paper","name":"BiDex-100 · OOD (2026 paper)","group":"预测与决策","metric":"Strict task success rate","unit":"%","public_history":true,"protocol":{"en":"Seven VLA policy architectures share a 58-DoF interface and a training mixture over all 106 tasks. Each model executes fifty rollouts per task and domain, from two fixed seeds and twenty-five episodes per seed, for 5,300 rollouts in each track. The first Overall value is strict success, distinct from the separate progress score. Model-specific official training schedules are retained.","zh":"七种VLA策略共享58自由度接口和包含全部106任务的训练混合。每个模型在每个域执行两个固定种子、每种子25回合，即每任务50次、每域共5,300次。采用Overall前一个严格成功率，与后一个进度分区分，保留各模型官方训练日程。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.25459v1","published_at":"2026-10-18","status":"paper_baselines"},"model_count":7,"result_count":7,"results":[[10158,0.0168,370,10164,10165],[8323,0.033,370,10164,10165],[10161,0.0545,370,10164,10165],[10162,0.0147,370,10164,10165],[7400,0.0077,370,10164,10165],[10163,0.0002,370,10164,10165],[7396,0.0032,370,10164,10165]]},{"id":"locomanilab-generalization-2026-paper","name":"LocoManiLab · generalization (2026 paper)","group":"预测与决策","metric":"Task success rate","unit":"%","public_history":true,"protocol":{"en":"Twelve policy families share the SONIC controller and common tasks. 234 tasks; 20 trials per seen task and 10 per unseen task or scene; equal task weights. Success uses each task’s recorded physical goal predicate; all episodes enter the denominator. Atomic and compositional checkpoints are trained separately and retain their assigned tasks. Domain-randomization and generalization panels remain separate.","zh":"十二个策略家族共享SONIC控制器与同一批任务。跨234任务评测，已见任务各20次，未见任务及场景各10次，按任务等权。按记录的物理目标谓词判定成功，全部回合计入分母。原子及组合检查点分别训练并保留各自分配的任务；随机扰动与泛化面板分别展示。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.30876v1","published_at":"2026-10-22","status":"paper_baselines"},"model_count":12,"result_count":12,"results":[[10166,0.479,370,10167,10168],[8325,0.42700000000000005,370,10167,10168],[8323,0.365,370,10167,10168],[7601,0.355,370,10167,10168],[7396,0.28800000000000003,370,10167,10168],[9600,0.218,370,10167,10168],[10169,0.204,370,10167,10168],[10170,0.184,370,10167,10168],[10171,0.158,370,10167,10168],[7401,0.15,370,10167,10168],[10172,0.149,370,10167,10168],[10173,0.071,370,10167,10168]]},{"id":"locomanilab-randomization-2026-paper","name":"LocoManiLab · randomization (2026 paper)","group":"预测与决策","metric":"Task success rate","unit":"%","public_history":true,"protocol":{"en":"Twelve policy families share the SONIC controller and common tasks. 100 seen tasks; 20 trials per task at Low/Mid/High perturbations; equal task and severity weights. Success uses each task’s recorded physical goal predicate; all episodes enter the denominator. Atomic and compositional checkpoints are trained separately and retain their assigned tasks. Domain-randomization and generalization panels remain separate.","zh":"十二个策略家族共享SONIC控制器与同一批任务。100个已见任务在Low/Mid/High扰动下各运行20次，任务与扰动等级分别等权。按记录的物理目标谓词判定成功，全部回合计入分母。原子及组合检查点分别训练并保留各自分配的任务；随机扰动与泛化面板分别展示。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.30876v1","published_at":"2026-10-22","status":"paper_baselines"},"model_count":12,"result_count":12,"results":[[8325,0.7020000000000001,370,10174,10175],[10166,0.69,370,10174,10175],[8323,0.605,370,10174,10175],[7601,0.542,370,10174,10175],[7396,0.45799999999999996,370,10174,10175],[9600,0.386,370,10174,10175],[10171,0.248,370,10174,10175],[10169,0.239,370,10174,10175],[10172,0.204,370,10174,10175],[10170,0.183,370,10174,10175],[7401,0.177,370,10174,10175],[10173,0.051,370,10174,10175]]},{"id":"libero-rig-mean-2026-paper","name":"LIBERO-RIG · raw perturbed success (2026 paper)","group":"Agent 与工具","metric":"Mean perturbed task success","unit":"%","public_history":true,"protocol":{"en":"Eight full-suite policies run the same rig perturbations on four LIBERO suites, with 500 episodes per condition and no fault metadata. The published Mean averages raw per-axis success across fixed severity conditions; table-height perturbations use the same two eligible suites for every model. The single-suite LingBot-VA row is excluded. OFT-Plus is a separately trained OpenVLA-OFT checkpoint, not an extra architecture.","zh":"八个完整套件策略在四个LIBERO套件接受相同设备扰动，每条件500回合，不提供故障参数。采用原表Mean，即各扰动轴固定等级的原始成功率均值；桌高扰动对所有模型采用相同两个可用套件。仅单套件的LingBot-VA另列，OFT-Plus保留为OpenVLA-OFT的独立训练检查点。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.18843v1","published_at":"2026-10-13","status":"paper_baselines"},"model_count":8,"result_count":8,"results":[[8323,0.846,370,10176,10177],[7602,0.808,370,10176,10177],[10178,0.796,370,10176,10177],[7594,0.761,370,10176,10177],[10179,0.752,370,10176,10177],[7401,0.71,370,10176,10177],[10180,0.595,370,10176,10177],[10181,0.55,370,10176,10177]]},{"id":"libero10-frontier-direct-control-2026-paper","name":"LIBERO-10 · frontier direct control (2026 paper)","group":"Agent 与工具","metric":"Simulator-verified task success","unit":"%","public_history":true,"protocol":{"en":"Six frontier base models retain low, medium and high reasoning configurations. Each configuration runs the first ten official initial states on all ten LIBERO-10 tasks, for 100 episodes. Success requires the original simulator goal predicates; done, give-up, 80-call limit, 900-step limit and spending-cap termination count as failure. The prompt, tool interface and control loop are shared.","zh":"六个前沿底模保留低、中、高推理配置，每配置使用十个LIBERO-10任务的前十个官方初始状态，共100回合。仅原模拟器目标谓词满足才计成功；done、放弃、80次调用、900步及花费上限结束均计失败。提示、工具接口及控制循环保持相同。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.30051v1","published_at":"2026-10-21","status":"paper_baselines"},"model_count":6,"result_count":18,"results":[[2958,0.44,370,10182,10183],[10184,0.43,370,10182,10183],[2958,0.38,370,10185,10183],[10184,0.38,370,10185,10183],[10184,0.34,370,10186,10183],[2959,0.33,370,10182,10183],[2960,0.33,370,10182,10183],[2960,0.3,370,10185,10183],[2958,0.28,370,10186,10183],[2960,0.24,370,10186,10183],[2959,0.18,370,10185,10183],[755,0.12,370,10185,10183],[755,0.1,370,10186,10183],[2959,0.09,370,10186,10183],[755,0.06,370,10182,10183],[2976,0.0,370,10186,10183],[2976,0.0,370,10185,10183],[2976,0.0,370,10182,10183]]},{"id":"dove-temporal-coverage-2026-paper","name":"DOVE · temporal evidence (2026 paper)","group":"视觉与视频","metric":"Probe temporal-evidence coverage","unit":"%","public_history":true,"protocol":{"en":"Fourteen published configurations caption designated occurrences in 180 source-video/query groups. The fixed evaluation set has 1,223 human-verified Order, State and Outcome probes over 529 eligible intervals. A text-only Qwen2.5-72B-Instruct judge checks whether each caption supports the gold choice; Temp-Cov uses all probes as its denominator, with unsupported or missing evidence unsuccessful. Probes and answers are hidden from captioning models. The source’s measured zero for Sa2VA is retained.","zh":"十四个已报告配置为180组源视频/查询中的指定重复事件生成描述。固定评测集包含529个可用区间的1,223个人工核验顺序、状态与结果问题。纯文本Qwen2.5-72B-Instruct判断描述是否支持标准选项；Temp-Cov以全部问题为分母，缺失及不支持的证据均不成功。描述模型无法看到私有问题与答案，保留原表Sa2VA的实测零分。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.29552v1","published_at":"2026-10-21","status":"paper_baselines"},"model_count":13,"result_count":14,"results":[[4048,0.17300000000000001,370,10187,10188],[10189,0.02,370,10187,10188],[10190,0.0,370,10187,10188],[10191,0.018000000000000002,370,10187,10188],[10192,0.006999999999999999,370,10187,10188],[7390,0.032,370,10187,10188],[10193,0.026000000000000002,370,10187,10188],[10194,0.044000000000000004,370,10187,10188],[828,0.083,370,10187,10188],[10195,0.01,370,10187,10188],[2728,0.024,370,10187,10188],[10196,0.135,370,10187,10188],[6298,0.147,370,10187,10188],[6298,0.33399999999999996,370,10197,10188]]},{"id":"modebench-l1-pass8-2026-paper","name":"ModeBench · Level 1 (2026 paper)","group":"知识与推理","metric":"Verifier pass@8","unit":"%","public_history":true,"protocol":{"en":"Seven hosted models use the same Graph, Countdown, MathIR and PantryPlan prompts, with 128 prompts per domain at each level and eight responses per prompt. A prompt succeeds when at least one response passes its executable verifier. The reported pass@8 weights the four domains equally. Python prompt-interface tests and model-training/replay methods are separate.","zh":"七个托管模型在各等级使用相同Graph、Countdown、MathIR与PantryPlan提示，每域128题、每题八个回答。至少一个回答通过可执行验证器则该题成功，原表pass@8对四个域等权。Python提示接口及模型训练/回放实验分别处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.11064v1","published_at":"2026-10-08","status":"paper_baselines"},"model_count":7,"result_count":7,"results":[[762,0.975,370,10198,10199],[755,0.9790000000000001,370,10198,10199],[3184,0.977,370,10198,10199],[3493,0.9940000000000001,370,10198,10199],[2977,0.98,370,10198,10199],[3479,0.9840000000000001,370,10198,10199],[3492,0.98,370,10198,10199]]},{"id":"modebench-l2-pass8-2026-paper","name":"ModeBench · Level 2 (2026 paper)","group":"知识与推理","metric":"Verifier pass@8","unit":"%","public_history":true,"protocol":{"en":"Seven hosted models use the same Graph, Countdown, MathIR and PantryPlan prompts, with 128 prompts per domain at each level and eight responses per prompt. A prompt succeeds when at least one response passes its executable verifier. The reported pass@8 weights the four domains equally. Python prompt-interface tests and model-training/replay methods are separate.","zh":"七个托管模型在各等级使用相同Graph、Countdown、MathIR与PantryPlan提示，每域128题、每题八个回答。至少一个回答通过可执行验证器则该题成功，原表pass@8对四个域等权。Python提示接口及模型训练/回放实验分别处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.11064v1","published_at":"2026-10-08","status":"paper_baselines"},"model_count":7,"result_count":7,"results":[[762,0.982,370,10200,10199],[755,0.996,370,10200,10199],[3184,0.98,370,10200,10199],[3493,1.0,370,10200,10199],[2977,1.0,370,10200,10199],[3479,1.0,370,10200,10199],[3492,0.988,370,10200,10199]]},{"id":"modebench-l3-pass8-2026-paper","name":"ModeBench · Level 3 (2026 paper)","group":"知识与推理","metric":"Verifier pass@8","unit":"%","public_history":true,"protocol":{"en":"Seven hosted models use the same Graph, Countdown, MathIR and PantryPlan prompts, with 128 prompts per domain at each level and eight responses per prompt. A prompt succeeds when at least one response passes its executable verifier. The reported pass@8 weights the four domains equally. Python prompt-interface tests and model-training/replay methods are separate.","zh":"七个托管模型在各等级使用相同Graph、Countdown、MathIR与PantryPlan提示，每域128题、每题八个回答。至少一个回答通过可执行验证器则该题成功，原表pass@8对四个域等权。Python提示接口及模型训练/回放实验分别处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.11064v1","published_at":"2026-10-08","status":"paper_baselines"},"model_count":7,"result_count":7,"results":[[762,0.9940000000000001,370,10201,10199],[755,0.998,370,10201,10199],[3184,0.9940000000000001,370,10201,10199],[3493,1.0,370,10201,10199],[2977,1.0,370,10201,10199],[3479,0.998,370,10201,10199],[3492,0.996,370,10201,10199]]},{"id":"modebench-pcmd-2026-paper","name":"ModeBench · conditional mode diversity (2026 paper)","group":"知识与推理","metric":"Macro PCMD","unit":"score","public_history":true,"protocol":{"en":"Seven hosted deployments are compared on thirteen domain-level cells reportable for every deployment. Each cell requires at least thirty prompts with two verifier-correct responses. PCMD measures diversity conditional on correct solutions rather than accuracy; the native macro mean is kept on 0–1. Eligibility and conditional interpretation remain attached to this panel.","zh":"七个托管端点使用所有端点都可报告的十三个域/等级单元。每单元要求至少三十个问题各有两个验证正确的回答。PCMD衡量以正确解为条件的解法模式多样性，保留0–1原生宏均值及适用条件。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.11064v1","published_at":"2026-10-08","status":"paper_baselines"},"score_max":1,"score_decimals":2,"saturation_excluded":true,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":7,"result_count":7,"results":[[3492,0.32,370,10202,10203],[2977,0.302,370,10202,10203],[3184,0.264,370,10202,10203],[3493,0.239,370,10202,10203],[762,0.226,370,10202,10203],[3479,0.215,370,10202,10203],[755,0.142,370,10202,10203]]},{"id":"isa-bench-solve1-2026-paper","name":"ISA-Bench · first attempt (2026 paper)","group":"编程","metric":"First-attempt tasks solved / 92","unit":"raw","public_history":true,"protocol":{"en":"Fourteen open-weight models receive a task and instruction-set reference and generate programs for five programming-game execution environments. The first-attempt count covers the same 92 tasks and requires all verification cases to pass. Parse errors, runtime errors and wrong outputs are completed failures. Iterative five-attempt results and judge ratings are separate.","zh":"十四个开源模型根据任务说明及指令集参考，为五个编程游戏环境生成程序。首轮解题数以相同92题为分母，必须通过全部验证用例。解析错误、运行错误和错误输出保留为实际失败；五次交互重试及裁判量表另行区分。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.22878v1","published_at":"2026-09-19","status":"paper_baselines"},"model_count":14,"result_count":14,"results":[[10204,23.0,370,10205,10206],[1513,13.0,370,10205,10206],[6269,14.0,370,10205,10206],[10207,1.0,370,10205,10206],[8826,12.0,370,10205,10206],[10208,12.0,370,10205,10206],[10209,12.0,370,10205,10206],[6268,13.0,370,10205,10206],[5836,10.0,370,10205,10206],[10210,6.0,370,10205,10206],[6465,3.0,370,10205,10206],[10211,3.0,370,10205,10206],[10212,3.0,370,10205,10206],[10213,2.0,370,10205,10206]]},{"id":"fireworld-task-structured-2026-paper","name":"FireWorldBench · structured (2026 paper)","group":"数学与科学","metric":"Average completion score","unit":"score","public_history":true,"protocol":{"en":"Models use matched controlled fire-world instances along three scenario tasks. Structured sensor/physical records is a separate observation track. The published average combines matched choice and open-report interfaces: set answers use Jaccard overlap and report slots use exact agreement. It remains a native partial-completion score /100, not whole-item accuracy. The 300-item human audit flags eight answer-leakage and four future-information cases; these source quality limits remain explicit.","zh":"模型在相同受控火灾世界中完成三类情境任务，使用结构化传感器与物理记录。两种观测轨道分别排行。原表平均分合并配对选择题与开放报告：集合答案采用Jaccard重叠，报告采用槽位精确匹配。保留百分制部分完成分，不当作整题准确率。300题人工审查中发现8题答案泄漏和4题未来信息泄漏，保留原文质量边界。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.23064v1","published_at":"2026-09-19","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":15,"result_count":15,"results":[[562,31.64,370,10214,10215],[5344,39.21,370,10214,10215],[5535,33.89,370,10214,10215],[7607,48.04,370,10214,10215],[786,45.99,370,10214,10215],[516,39.99,370,10214,10215],[3020,39.95,370,10214,10215],[10216,40.39,370,10214,10215],[784,44.59,370,10214,10215],[3009,50.07,370,10214,10215],[10217,48.83,370,10214,10215],[5616,56.68,370,10214,10215],[766,47.18,370,10214,10215],[6214,46.8,370,10214,10215],[10218,47.72,370,10214,10215]]},{"id":"fireworld-task-visual-2026-paper","name":"FireWorldBench · visual (2026 paper)","group":"数学与科学","metric":"Average completion score","unit":"score","public_history":true,"protocol":{"en":"Models use matched controlled fire-world instances along three scenario tasks. Multimodal physics-rendered visual evidence is a separate observation track. The published average combines matched choice and open-report interfaces: set answers use Jaccard overlap and report slots use exact agreement. It remains a native partial-completion score /100, not whole-item accuracy. The 300-item human audit flags eight answer-leakage and four future-information cases; these source quality limits remain explicit.","zh":"模型在相同受控火灾世界中完成三类情境任务，使用物理场渲染的多模态视觉证据。两种观测轨道分别排行。原表平均分合并配对选择题与开放报告：集合答案采用Jaccard重叠，报告采用槽位精确匹配。保留百分制部分完成分，不当作整题准确率。300题人工审查中发现8题答案泄漏和4题未来信息泄漏，保留原文质量边界。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.23064v1","published_at":"2026-09-19","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":10,"result_count":10,"results":[[2728,27.83,370,10219,10215],[6150,28.61,370,10219,10215],[2706,34.84,370,10219,10215],[6625,42.78,370,10219,10215],[2675,38.51,370,10219,10215],[2613,35.71,370,10219,10215],[5616,43.88,370,10219,10215],[766,44.89,370,10219,10215],[6214,40.15,370,10219,10215],[10218,39.69,370,10219,10215]]},{"id":"llmadbench-labeled-2026-paper","name":"LLMAdBench · labeled (2026 paper)","group":"专业工作","metric":"Three-way human-label accuracy","unit":"%","public_history":true,"protocol":{"en":"Eight frontier judges and Qwen3-8B before/after DPO predict the same retained human A/B/T labels, with ties preserved. Labeled ad disclosure is a separate condition. Each placement-pair/rubric is one target; all fine-tuning splits are source-instance-disjoint, so a conversation and advertisement cannot cross train/test. This evaluates agreement with human preferences, not observed ad clicks.","zh":"八个前沿裁判模型与DPO训练前后的Qwen3-8B预测同一批人工A/B/T标签，保留平局。明确标注广告分别计组；每个位置配对与量表维度构成一道题。微调按源实例隔离，避免同一对话和广告跨训练测试集。成绩表示与人类偏好的一致程度，不表示实际点击率。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.32533v1","published_at":"2026-09-26","status":"paper_baselines"},"model_count":9,"result_count":10,"results":[[3184,0.47850000000000004,370,10220,10221],[2261,0.4696,370,10220,10221],[3412,0.4814,370,10220,10221],[3425,0.4631,370,10220,10221],[2977,0.47229999999999994,370,10220,10221],[5795,0.44,370,10220,10221],[1583,0.4451,370,10220,10221],[3338,0.4639,370,10220,10221],[5344,0.4025,370,10220,10221],[5344,0.5054,370,10222,10221]]},{"id":"llmadbench-unlabeled-2026-paper","name":"LLMAdBench · unlabeled (2026 paper)","group":"专业工作","metric":"Three-way human-label accuracy","unit":"%","public_history":true,"protocol":{"en":"Eight frontier judges and Qwen3-8B before/after DPO predict the same retained human A/B/T labels, with ties preserved. Unlabeled ad disclosure is a separate condition. Each placement-pair/rubric is one target; all fine-tuning splits are source-instance-disjoint, so a conversation and advertisement cannot cross train/test. This evaluates agreement with human preferences, not observed ad clicks.","zh":"八个前沿裁判模型与DPO训练前后的Qwen3-8B预测同一批人工A/B/T标签，保留平局。未标注广告分别计组；每个位置配对与量表维度构成一道题。微调按源实例隔离，避免同一对话和广告跨训练测试集。成绩表示与人类偏好的一致程度，不表示实际点击率。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.32533v1","published_at":"2026-09-26","status":"paper_baselines"},"model_count":9,"result_count":10,"results":[[3184,0.5032,370,10220,10221],[2261,0.4874,370,10220,10221],[3412,0.49920000000000003,370,10220,10221],[3425,0.4752,370,10220,10221],[2977,0.4987,370,10220,10221],[5795,0.4618,370,10220,10221],[1583,0.48090000000000005,370,10220,10221],[3338,0.4876,370,10220,10221],[5344,0.3652,370,10220,10221],[5344,0.5576,370,10222,10221]]},{"id":"swe-game-brief-to-game-2026-paper","name":"SWE-Game · brief-to-game (2026 paper)","group":"编程","metric":"Task composite score","unit":"score","public_history":true,"protocol":{"en":"Six coding models use the published development framework across 247 tasks partitioned into five modes. This cohort keeps brief-to-game separate. The overall 0–100 composite uses task-specific executable checks and game-specific visual rubrics; repair multiplies restoration, retained routes, preservation and validity factors per case. It is a native composite, not whole-task pass rate.","zh":"六个编程模型使用论文开发框架，评测247个任务的五种模式，本组单列brief-to-game。百分制总体分结合各任务的可执行检查和游戏视觉量表；修复任务按案例相乘恢复、路线保留、内容保留及有效性因子。保留原生综合分，不当作整题通过率。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.33678v1","published_at":"2026-09-27","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":6,"result_count":6,"results":[[5899,27.0,370,10223,10224],[5295,39.01,370,10223,10224],[5616,34.18,370,10223,10224],[755,50.38,370,10223,10224],[3396,31.05,370,10223,10224],[3198,30.47,370,10223,10224]]},{"id":"swe-game-gdd-to-game-2026-paper","name":"SWE-Game · gdd-to-game (2026 paper)","group":"编程","metric":"Task composite score","unit":"score","public_history":true,"protocol":{"en":"Six coding models use the published development framework across 247 tasks partitioned into five modes. This cohort keeps gdd-to-game separate. The overall 0–100 composite uses task-specific executable checks and game-specific visual rubrics; repair multiplies restoration, retained routes, preservation and validity factors per case. It is a native composite, not whole-task pass rate.","zh":"六个编程模型使用论文开发框架，评测247个任务的五种模式，本组单列gdd-to-game。百分制总体分结合各任务的可执行检查和游戏视觉量表；修复任务按案例相乘恢复、路线保留、内容保留及有效性因子。保留原生综合分，不当作整题通过率。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.33678v1","published_at":"2026-09-27","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":6,"result_count":6,"results":[[5899,37.35,370,10225,10224],[5295,48.55,370,10225,10224],[5616,52.67,370,10225,10224],[755,59.68,370,10225,10224],[3396,40.18,370,10225,10224],[3198,42.58,370,10225,10224]]},{"id":"swe-game-skeleton-completion-2026-paper","name":"SWE-Game · skeleton-completion (2026 paper)","group":"编程","metric":"Task composite score","unit":"score","public_history":true,"protocol":{"en":"Six coding models use the published development framework across 247 tasks partitioned into five modes. This cohort keeps skeleton-completion separate. The overall 0–100 composite uses task-specific executable checks and game-specific visual rubrics; repair multiplies restoration, retained routes, preservation and validity factors per case. It is a native composite, not whole-task pass rate.","zh":"六个编程模型使用论文开发框架，评测247个任务的五种模式，本组单列skeleton-completion。百分制总体分结合各任务的可执行检查和游戏视觉量表；修复任务按案例相乘恢复、路线保留、内容保留及有效性因子。保留原生综合分，不当作整题通过率。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.33678v1","published_at":"2026-09-27","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":6,"result_count":6,"results":[[5899,39.19,370,10226,10224],[5295,40.76,370,10226,10224],[5616,43.1,370,10226,10224],[755,54.06,370,10226,10224],[3396,34.38,370,10226,10224],[3198,35.69,370,10226,10224]]},{"id":"swe-game-bug-repair-2026-paper","name":"SWE-Game · bug-repair (2026 paper)","group":"编程","metric":"Task composite score","unit":"score","public_history":true,"protocol":{"en":"Six coding models use the published development framework across 247 tasks partitioned into five modes. This cohort keeps bug-repair separate. The overall 0–100 composite uses task-specific executable checks and game-specific visual rubrics; repair multiplies restoration, retained routes, preservation and validity factors per case. It is a native composite, not whole-task pass rate.","zh":"六个编程模型使用论文开发框架，评测247个任务的五种模式，本组单列bug-repair。百分制总体分结合各任务的可执行检查和游戏视觉量表；修复任务按案例相乘恢复、路线保留、内容保留及有效性因子。保留原生综合分，不当作整题通过率。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.33678v1","published_at":"2026-09-27","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":6,"result_count":6,"results":[[5899,26.13,370,10227,10224],[5295,52.44,370,10227,10224],[5616,58.05,370,10227,10224],[755,83.46,370,10227,10224],[3396,27.55,370,10227,10224],[3198,27.36,370,10227,10224]]},{"id":"swe-game-godot-to-unity-2026-paper","name":"SWE-Game · godot-to-unity (2026 paper)","group":"编程","metric":"Task composite score","unit":"score","public_history":true,"protocol":{"en":"Six coding models use the published development framework across 247 tasks partitioned into five modes. This cohort keeps godot-to-unity separate. The overall 0–100 composite uses task-specific executable checks and game-specific visual rubrics; repair multiplies restoration, retained routes, preservation and validity factors per case. It is a native composite, not whole-task pass rate.","zh":"六个编程模型使用论文开发框架，评测247个任务的五种模式，本组单列godot-to-unity。百分制总体分结合各任务的可执行检查和游戏视觉量表；修复任务按案例相乘恢复、路线保留、内容保留及有效性因子。保留原生综合分，不当作整题通过率。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.33678v1","published_at":"2026-09-27","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":6,"result_count":6,"results":[[5899,51.23,370,10228,10224],[5295,59.04,370,10228,10224],[5616,59.85,370,10228,10224],[755,72.4,370,10228,10224],[3396,44.33,370,10228,10224],[3198,49.62,370,10228,10224]]},{"id":"auvbench-scoring-srcc-2026-paper","name":"AUVBench · scoring-srcc (2026 paper)","group":"视觉与视频","metric":"Aesthetic scoring SRCC","unit":"score","public_history":true,"protocol":{"en":"Twelve multimodal and specialist models use the same task-specific protocol. Scoring uses rank correlation against professional-designer mean opinion scores. Native aesthetic scales remain separate from task pass rates.","zh":"十二个多模态及审美专用模型采用相同的各任务协议。审美评分采用模型分数与专业设计师平均意见分的秩相关。原生审美量表与任务通过率分别处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34854v1","published_at":"2026-09-28","status":"paper_baselines"},"score_max":1,"saturation_excluded":true,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":12,"result_count":12,"results":[[755,0.56,370,10229,10230],[3204,0.5,370,10229,10230],[762,0.55,370,10229,10230],[2974,0.54,370,10229,10230],[3155,0.57,370,10229,10230],[4961,0.57,370,10229,10230],[3157,0.54,370,10229,10230],[6144,0.23,370,10229,10230],[10231,0.45,370,10229,10230],[6150,0.45,370,10229,10230],[10232,0.05,370,10229,10230],[10233,0.13,370,10229,10230]]},{"id":"auvbench-repair-2026-paper","name":"AUVBench · repair (2026 paper)","group":"生成与创作","metric":"Repair pass rate","unit":"%","public_history":true,"protocol":{"en":"Twelve multimodal and specialist models use the same task-specific protocol. Repair succeeds only if all target violations are corrected, collateral damage is bounded and content and change-budget checks pass; all 660 cases retain invalid or failed outputs as failures. Native aesthetic scales remain separate from task pass rates.","zh":"十二个多模态及审美专用模型采用相同的各任务协议。修复必须纠正全部目标问题，并通过附带损害、内容保留和改动范围检查；全部660题保留无效及失败输出。原生审美量表与任务通过率分别处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34854v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":12,"result_count":12,"results":[[755,0.7559999999999999,370,10229,10230],[3204,0.45,370,10229,10230],[762,0.655,370,10229,10230],[2974,0.7,370,10229,10230],[3155,0.573,370,10229,10230],[4961,0.342,370,10229,10230],[3157,0.547,370,10229,10230],[6144,0.0091,370,10229,10230],[10231,0.003,370,10229,10230],[6150,0.0060999999999999995,370,10229,10230],[10232,0.0015,370,10229,10230],[10233,0.0,370,10229,10230]]},{"id":"auvbench-generation-aesthetics-2026-paper","name":"AUVBench · generation-aesthetics (2026 paper)","group":"生成与创作","metric":"Calibrated aesthetic score","unit":"score","public_history":true,"protocol":{"en":"Twelve multimodal and specialist models use the same task-specific protocol. Text-to-UI generation is scored by a held-out-calibrated aesthetic judge, with dimension calibration learned from task-1 ratings and frozen for task 4; failed renders receive zero. Native aesthetic scales remain separate from task pass rates.","zh":"十二个多模态及审美专用模型采用相同的各任务协议。网页生成采用经人工评分标定的审美裁判；维度标定在任务1上学习，进入任务4后冻结，渲染失败计零分。原生审美量表与任务通过率分别处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34854v1","published_at":"2026-09-28","status":"paper_baselines"},"score_max":5,"saturation_excluded":true,"scale_label":{"en":"Score / 5","zh":"分数 / 5"},"model_count":12,"result_count":12,"results":[[755,3.13,370,10229,10230],[3204,2.78,370,10229,10230],[762,3.17,370,10229,10230],[2974,3.08,370,10229,10230],[3155,3.18,370,10229,10230],[4961,3.09,370,10229,10230],[3157,3.04,370,10229,10230],[6144,1.75,370,10229,10230],[10231,1.11,370,10229,10230],[6150,1.84,370,10229,10230],[10232,0.53,370,10229,10230],[10233,0.16,370,10229,10230]]},{"id":"studentbench-combined-gain-2026-paper","name":"StudentBench · adjusted learning gain (2026 paper)","group":"专业工作","metric":"Adjusted pre/post learning gain (pp)","unit":"raw","public_history":true,"protocol":{"en":"Twelve published tutor configurations (eleven models) are compared using GRE pre/post learning gains adjusted for pre-test score, its quadratic term and section. Model assignment uses a least-fill rule within the AI-tutoring arm; study sessions differ across tutors. The displayed point estimates are percentage-point gains with HC3 intervals retained in configurations, not correct-answer rates. Human and no-tutoring controls are excluded.","zh":"十二种已公布辅导配置、十一个模型按GRE前后测学习增量比较，回归调整前测分数、其平方项和科目。AI辅导组内采用最少填充规则分配模型，各模型学生样本不同。保留百分点增量与HC3区间，不当作答题准确率；排除人类与无辅导对照。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.28470v1","published_at":"2026-09-23","status":"paper_baselines"},"model_count":11,"result_count":12,"results":[[3479,16.0,370,10234,10235],[10236,15.7,370,10237,10235],[3412,15.4,370,10238,10235],[3415,14.3,370,10239,10235],[3386,14.1,370,10240,10235],[3479,13.6,370,10241,10235],[3196,13.5,370,10242,10235],[2585,13.1,370,10243,10235],[5449,13.0,370,10244,10235],[755,12.6,370,10245,10235],[3490,12.2,370,10246,10235],[3405,12.1,370,10247,10235]]},{"id":"polybench26-electron-affinity-2026-paper","name":"PolyBench26 · electron-affinity (2026 paper)","group":"数学与科学","metric":"RMSE (eV)","unit":"raw","public_history":true,"protocol":{"en":"Six model architectures predict homopolymer electron-affinity using the published source panel. GPT models make one zero-shot prediction per polymer, while specialist methods report mean test RMSE over five cross-validation folds; those evaluation regimes remain explicit. Different properties and physical units are kept separate. Lower is better.","zh":"六种模型在原文面板中预测均聚物的电子亲和能。GPT模型对每种聚合物进行一次零样本预测，专用模型报告五折交叉验证的测试RMSE均值，明确保留这一区别。不同性质和物理单位分别排行，数值越低越好。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.27036v1","published_at":"2026-09-22","status":"paper_baselines"},"higher_is_better":false,"model_count":6,"result_count":6,"results":[[353,1.2,370,10248,10249],[4075,0.95,370,10248,10249],[10250,0.29,370,10251,10249],[10252,0.29,370,10251,10249],[10253,0.39,370,10251,10249],[10254,0.43,370,10251,10249]]},{"id":"polybench26-glass-transition-2026-paper","name":"PolyBench26 · glass-transition (2026 paper)","group":"数学与科学","metric":"RMSE (°C)","unit":"raw","public_history":true,"protocol":{"en":"Six model architectures predict homopolymer glass-transition using the published source panel. GPT models make one zero-shot prediction per polymer, while specialist methods report mean test RMSE over five cross-validation folds; those evaluation regimes remain explicit. Different properties and physical units are kept separate. Lower is better.","zh":"六种模型在原文面板中预测均聚物的玻璃转变温度。GPT模型对每种聚合物进行一次零样本预测，专用模型报告五折交叉验证的测试RMSE均值，明确保留这一区别。不同性质和物理单位分别排行，数值越低越好。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.27036v1","published_at":"2026-09-22","status":"paper_baselines"},"higher_is_better":false,"model_count":6,"result_count":6,"results":[[353,73.5,370,10248,10249],[4075,70.4,370,10248,10249],[10250,34.7,370,10251,10249],[10252,36.7,370,10251,10249],[10253,37.9,370,10251,10249],[10254,39.7,370,10251,10249]]},{"id":"anipro-sfd-2026-paper","name":"AniPrO · SFD (2026 paper)","group":"视觉与视频","metric":"Overall binary accuracy","unit":"%","public_history":true,"protocol":{"en":"Seven distinct visual-language models process the same balanced 15,000-image anime-provenance dataset (human, inpainting and text-to-image). SFD uses each model’s structured scores as features in a common random-forest classifier with five-fold validation; the reported value is mean binary real-vs-AI accuracy. Source-generation classes and pipeline conditions remain explicit.","zh":"七个不同视觉语言模型处理相同的15,000张动漫溯源图片，人工、局部重绘和文生图三类数量均衡。SFD将各模型的结构评分交给相同随机森林分类器，采用五折验证，显示人工与AI二分类的平均准确率。保留数据生成类别及管线条件。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.23345v1","published_at":"2026-09-20","status":"paper_baselines"},"model_count":7,"result_count":7,"results":[[340,0.588,370,10255,10256],[764,0.702,370,10255,10256],[10257,0.644,370,10255,10256],[6297,0.639,370,10255,10256],[6299,0.643,370,10255,10256],[1583,0.607,370,10255,10256],[1702,0.63,370,10255,10256]]},{"id":"anipro-mfr-2026-paper","name":"AniPrO · MFR (2026 paper)","group":"视觉与视频","metric":"Overall binary accuracy","unit":"%","public_history":true,"protocol":{"en":"Seven distinct visual-language models process the same balanced 15,000-image anime-provenance dataset (human, inpainting and text-to-image). MFR uses descriptions from each model with a fixed GLM-4.5V judge; the reported value is binary real-vs-AI accuracy. Source-generation classes and pipeline conditions remain explicit.","zh":"七个不同视觉语言模型处理相同的15,000张动漫溯源图片，人工、局部重绘和文生图三类数量均衡。MFR将各模型生成的描述交给固定GLM-4.5V裁判，显示人工与AI二分类准确率。保留数据生成类别及管线条件。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.23345v1","published_at":"2026-09-20","status":"paper_baselines"},"model_count":7,"result_count":7,"results":[[340,0.649,370,10258,10259],[764,0.738,370,10258,10259],[10257,0.684,370,10258,10259],[6297,0.678,370,10258,10259],[6299,0.665,370,10258,10259],[1583,0.695,370,10258,10259],[1702,0.71,370,10258,10259]]},{"id":"mincu-grounded-f1-2026-paper","name":"MinCU · grounded minimal changes (2026 paper)","group":"视觉与视频","metric":"Grounded F1","unit":"score","public_history":true,"protocol":{"en":"General multimodal, specialist grounding and fine-tuned models describe and locate minimal changes between image pairs. Grounded F1 requires IoU≥0.5, BERTScore≥0.85 and the entity label simultaneously. Native F1 is separate from whole-item accuracy; SFT and SG-ISA variants retain their Qwen3-VL backbone identities.","zh":"通用多模态、专用定位及微调模型描述并定位成对图片中的最小变化。Grounded F1要求同时满足IoU≥0.5、BERTScore≥0.85及实体标签约束。保留原生F1；SFT与SG-ISA配置注明所属Qwen3-VL底模。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.23336v1","published_at":"2026-09-20","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":15,"result_count":19,"results":[[10260,19.58,370,10261,10262],[10263,31.92,370,10261,10262],[9072,42.89,370,10261,10262],[10264,42.43,370,10261,10262],[3247,48.43,370,10261,10262],[5826,2.49,370,10261,10262],[2654,18.52,370,10261,10262],[6345,21.23,370,10261,10262],[6341,13.26,370,10261,10262],[6298,50.72,370,10261,10262],[6342,48.84,370,10261,10262],[6238,51.76,370,10261,10262],[10265,0.29,370,10261,10262],[10266,14.11,370,10261,10262],[10267,47.76,370,10261,10262],[6341,72.75,370,10268,10262],[6341,82.88,370,10269,10262],[6298,83.17,370,10270,10262],[6342,83.07,370,10271,10262]]},{"id":"omni2web-omni-direct-editing-2026-paper","name":"Omni2Web · Omni direct-editing (2026 paper)","group":"Agent 与工具","metric":"Edit Fidelity Score","unit":"score","public_history":true,"protocol":{"en":"Models process the same 918 bilingual screen-recorded requests from 129 webpages. Omni interfaces are separated from the other modality. Direct editing uses the native composite Edit Fidelity Score; MiniCPM-o and Gemma exceed input context limits and have no result, so neither is assigned zero. Page reuse makes examples correlated; the paper uses source-page cluster bootstrap. Native interfaces may differ in representation, and the cohort is an end-to-end system comparison.","zh":"模型处理来自129个网页的相同918个双语录屏请求，Omni输入接口单独排行。直接编辑保留原生Edit Fidelity综合分；MiniCPM-o与Gemma因输入超出上下文而未取得成绩，不补零分。同网页样本存在相关性，论文按源网页进行分组自助采样；原生接口的输入表示可能不同，成绩表示端到端系统表现。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.23417v1","published_at":"2026-09-20","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":7,"result_count":7,"results":[[10272,16.64,370,10273,10274],[7223,28.79,370,10273,10274],[3413,24.08,370,10273,10274],[10275,28.33,370,10273,10274],[3415,51.17,370,10273,10274],[6724,29.56,370,10273,10274],[3412,34.08,370,10273,10274]]},{"id":"omni2web-omni-instruction-recovery-2026-paper","name":"Omni2Web · Omni instruction-recovery (2026 paper)","group":"Agent 与工具","metric":"Instruction Recovery Score","unit":"score","public_history":true,"protocol":{"en":"Models process the same 918 bilingual screen-recorded requests from 129 webpages. Omni interfaces are separated from the other modality. Instruction recovery averages target and action F1; it remains a native component score /100. Page reuse makes examples correlated; the paper uses source-page cluster bootstrap. Native interfaces may differ in representation, and the cohort is an end-to-end system comparison.","zh":"模型处理来自129个网页的相同918个双语录屏请求，Omni输入接口单独排行。指令恢复采用目标与动作F1的平均值，保留百分制原生分量分。同网页样本存在相关性，论文按源网页进行分组自助采样；原生接口的输入表示可能不同，成绩表示端到端系统表现。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.23417v1","published_at":"2026-09-20","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":9,"result_count":9,"results":[[6574,24.9,370,10273,10274],[6585,6.78,370,10273,10274],[10272,5.0,370,10273,10274],[7223,37.31,370,10273,10274],[3413,25.41,370,10273,10274],[10275,46.52,370,10273,10274],[3415,47.46,370,10273,10274],[6724,49.14,370,10273,10274],[3412,43.19,370,10273,10274]]},{"id":"omni2web-vl-direct-editing-2026-paper","name":"Omni2Web · VL direct-editing (2026 paper)","group":"Agent 与工具","metric":"Edit Fidelity Score","unit":"score","public_history":true,"protocol":{"en":"Models process the same 918 bilingual screen-recorded requests from 129 webpages. VL interfaces are separated from the other modality. Direct editing uses the native composite Edit Fidelity Score; MiniCPM-o and Gemma exceed input context limits and have no result, so neither is assigned zero. Page reuse makes examples correlated; the paper uses source-page cluster bootstrap. Native interfaces may differ in representation, and the cohort is an end-to-end system comparison.","zh":"模型处理来自129个网页的相同918个双语录屏请求，VL输入接口单独排行。直接编辑保留原生Edit Fidelity综合分；MiniCPM-o与Gemma因输入超出上下文而未取得成绩，不补零分。同网页样本存在相关性，论文按源网页进行分组自助采样；原生接口的输入表示可能不同，成绩表示端到端系统表现。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.23417v1","published_at":"2026-09-20","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":8,"result_count":8,"results":[[5300,26.94,370,10276,10274],[3198,29.49,370,10276,10274],[3157,38.6,370,10276,10274],[3155,40.63,370,10276,10274],[4030,34.48,370,10276,10274],[2974,41.45,370,10276,10274],[762,42.86,370,10276,10274],[755,47.37,370,10276,10274]]},{"id":"omni2web-vl-instruction-recovery-2026-paper","name":"Omni2Web · VL instruction-recovery (2026 paper)","group":"Agent 与工具","metric":"Instruction Recovery Score","unit":"score","public_history":true,"protocol":{"en":"Models process the same 918 bilingual screen-recorded requests from 129 webpages. VL interfaces are separated from the other modality. Instruction recovery averages target and action F1; it remains a native component score /100. Page reuse makes examples correlated; the paper uses source-page cluster bootstrap. Native interfaces may differ in representation, and the cohort is an end-to-end system comparison.","zh":"模型处理来自129个网页的相同918个双语录屏请求，VL输入接口单独排行。指令恢复采用目标与动作F1的平均值，保留百分制原生分量分。同网页样本存在相关性，论文按源网页进行分组自助采样；原生接口的输入表示可能不同，成绩表示端到端系统表现。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.23417v1","published_at":"2026-09-20","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":8,"result_count":8,"results":[[5300,29.84,370,10276,10274],[3198,29.32,370,10276,10274],[3157,38.37,370,10276,10274],[3155,41.95,370,10276,10274],[4030,39.08,370,10276,10274],[2974,40.83,370,10276,10274],[762,48.34,370,10276,10274],[755,48.91,370,10276,10274]]},{"id":"activearena-id-2026-paper","name":"ActiveArena · ID (2026 paper)","group":"Agent 与工具","metric":"Average task success","unit":"%","public_history":true,"protocol":{"en":"Six independently trained baseline policy architectures and three ActiveArena configurations use the same 100 ID training trajectories per task (3.9 epochs) and the same 50 evaluation seeds in each of 35 tasks. ID is a separate scene condition. Memory annotations consumed by each architecture and fixed-view/OFT/planning configurations remain explicit.","zh":"六种分别训练的基线策略架构及三种ActiveArena配置，均采用每题相同100条ID训练轨迹、3.9轮训练，并在35题各使用相同50个评测种子。ID场景单列；保留各架构所用记忆标注及固定视角、OFT和规划配置。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.24124v1","published_at":"2026-09-21","status":"paper_baselines"},"model_count":7,"result_count":9,"results":[[10277,0.35600000000000004,370,10278,10279],[8323,0.1686,370,10278,10279],[10280,0.2091,370,10278,10279],[10281,0.1069,370,10282,10279],[10283,0.3531,370,10282,10279],[10284,0.2223,370,10282,10279],[10285,0.0034000000000000002,370,10286,10279],[10285,0.6297,370,10287,10279],[10285,0.6097,370,10288,10279]]},{"id":"activearena-ood-2026-paper","name":"ActiveArena · OOD (2026 paper)","group":"Agent 与工具","metric":"Average task success","unit":"%","public_history":true,"protocol":{"en":"Six independently trained baseline policy architectures and three ActiveArena configurations use the same 100 ID training trajectories per task (3.9 epochs) and the same 50 evaluation seeds in each of 35 tasks. OOD is a separate scene condition. Memory annotations consumed by each architecture and fixed-view/OFT/planning configurations remain explicit.","zh":"六种分别训练的基线策略架构及三种ActiveArena配置，均采用每题相同100条ID训练轨迹、3.9轮训练，并在35题各使用相同50个评测种子。OOD场景单列；保留各架构所用记忆标注及固定视角、OFT和规划配置。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.24124v1","published_at":"2026-09-21","status":"paper_baselines"},"model_count":7,"result_count":9,"results":[[10277,0.0206,370,10278,10279],[8323,0.0171,370,10278,10279],[10280,0.1051,370,10278,10279],[10281,0.06570000000000001,370,10282,10279],[10283,0.2343,370,10282,10279],[10284,0.11199999999999999,370,10282,10279],[10285,0.0017000000000000001,370,10286,10279],[10285,0.47600000000000003,370,10287,10279],[10285,0.45539999999999997,370,10288,10279]]},{"id":"coordbench-r2-random-2026-paper","name":"CoordBench · random (2026 paper)","group":"预测与决策","metric":"Mean target R²","unit":"raw","public_history":true,"protocol":{"en":"Geographic predictors use the same datasets within this random split column. Five-fold evaluations are repeated over five fold-assignment seeds. Target R² is floored at −1 before dataset averaging; affected unchanged-partition datasets are excluded consistently within a block size. Different block sizes have different eligible cohorts and stay separate. MIND widths/CP are configurations of one model family.","zh":"地理预测器在random划分列内使用相同数据集，五折评测重复五个分组种子。各目标R²先截断至−1再按数据集求平均；空间块变化却未改变划分的数据集在对应列中一致排除。不同块大小的有效题组可能不同，分别排行；MIND宽度及CP保留为同家族配置。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.25454v1","published_at":"2026-09-21","status":"paper_baselines"},"model_count":14,"result_count":18,"results":[[10289,0.675,370,10290,10291],[10292,0.137,370,10293,10291],[10294,0.204,370,10295,10291],[10296,0.522,370,10297,10291],[10298,0.504,370,10299,10291],[10300,0.589,370,10301,10291],[7072,0.542,370,10302,10291],[10303,0.429,370,10304,10291],[10305,0.407,370,10306,10291],[10307,0.427,370,10308,10291],[10309,0.563,370,10310,10291],[10311,0.511,370,10312,10291],[10313,0.557,370,10314,10291],[10315,0.504,370,10316,10291],[10315,0.533,370,10317,10291],[10315,0.556,370,10318,10291],[10315,0.635,370,10319,10291],[10315,0.643,370,10320,10291]]},{"id":"coordbench-r2-2deg-2026-paper","name":"CoordBench · 2deg (2026 paper)","group":"预测与决策","metric":"Mean target R²","unit":"raw","public_history":true,"protocol":{"en":"Geographic predictors use the same datasets within this 2deg split column. Five-fold evaluations are repeated over five fold-assignment seeds. Target R² is floored at −1 before dataset averaging; affected unchanged-partition datasets are excluded consistently within a block size. Different block sizes have different eligible cohorts and stay separate. MIND widths/CP are configurations of one model family.","zh":"地理预测器在2deg划分列内使用相同数据集，五折评测重复五个分组种子。各目标R²先截断至−1再按数据集求平均；空间块变化却未改变划分的数据集在对应列中一致排除。不同块大小的有效题组可能不同，分别排行；MIND宽度及CP保留为同家族配置。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.25454v1","published_at":"2026-09-21","status":"paper_baselines"},"model_count":14,"result_count":18,"results":[[10289,0.435,370,10321,10291],[10292,0.09,370,10322,10291],[10294,0.155,370,10323,10291],[10296,0.346,370,10324,10291],[10298,0.366,370,10325,10291],[10300,0.381,370,10326,10291],[7072,0.44,370,10327,10291],[10303,0.341,370,10328,10291],[10305,0.338,370,10329,10291],[10307,0.279,370,10330,10291],[10309,0.418,370,10331,10291],[10311,0.39,370,10332,10291],[10313,0.371,370,10333,10291],[10315,0.449,370,10334,10291],[10315,0.46,370,10335,10291],[10315,0.469,370,10336,10291],[10315,0.354,370,10337,10291],[10315,0.5,370,10338,10291]]},{"id":"coordbench-r2-10deg-2026-paper","name":"CoordBench · 10deg (2026 paper)","group":"预测与决策","metric":"Mean target R²","unit":"raw","public_history":true,"protocol":{"en":"Geographic predictors use the same datasets within this 10deg split column. Five-fold evaluations are repeated over five fold-assignment seeds. Target R² is floored at −1 before dataset averaging; affected unchanged-partition datasets are excluded consistently within a block size. Different block sizes have different eligible cohorts and stay separate. MIND widths/CP are configurations of one model family.","zh":"地理预测器在10deg划分列内使用相同数据集，五折评测重复五个分组种子。各目标R²先截断至−1再按数据集求平均；空间块变化却未改变划分的数据集在对应列中一致排除。不同块大小的有效题组可能不同，分别排行；MIND宽度及CP保留为同家族配置。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.25454v1","published_at":"2026-09-21","status":"paper_baselines"},"model_count":14,"result_count":18,"results":[[10289,0.187,370,10339,10291],[10292,-0.087,370,10340,10291],[10294,-0.028,370,10341,10291],[10296,-0.413,370,10342,10291],[10298,0.073,370,10343,10291],[10300,0.038,370,10344,10291],[7072,0.267,370,10345,10291],[10303,0.085,370,10346,10291],[10305,0.098,370,10347,10291],[10307,-0.096,370,10348,10291],[10309,0.119,370,10349,10291],[10311,0.203,370,10350,10291],[10313,-0.061,370,10351,10291],[10315,0.289,370,10352,10291],[10315,0.291,370,10353,10291],[10315,0.27,370,10354,10291],[10315,0.031,370,10355,10291],[10315,0.335,370,10356,10291]]},{"id":"coordbench-r2-20deg-2026-paper","name":"CoordBench · 20deg (2026 paper)","group":"预测与决策","metric":"Mean target R²","unit":"raw","public_history":true,"protocol":{"en":"Geographic predictors use the same datasets within this 20deg split column. Five-fold evaluations are repeated over five fold-assignment seeds. Target R² is floored at −1 before dataset averaging; affected unchanged-partition datasets are excluded consistently within a block size. Different block sizes have different eligible cohorts and stay separate. MIND widths/CP are configurations of one model family.","zh":"地理预测器在20deg划分列内使用相同数据集，五折评测重复五个分组种子。各目标R²先截断至−1再按数据集求平均；空间块变化却未改变划分的数据集在对应列中一致排除。不同块大小的有效题组可能不同，分别排行；MIND宽度及CP保留为同家族配置。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.25454v1","published_at":"2026-09-21","status":"paper_baselines"},"model_count":14,"result_count":18,"results":[[10289,0.093,370,10357,10291],[10292,-0.22,370,10358,10291],[10294,-0.142,370,10359,10291],[10296,-0.577,370,10360,10291],[10298,-0.038,370,10361,10291],[10300,-0.067,370,10362,10291],[7072,0.251,370,10363,10291],[10303,-0.056,370,10364,10291],[10305,-0.076,370,10365,10291],[10307,-0.163,370,10366,10291],[10309,-0.059,370,10367,10291],[10311,0.176,370,10368,10291],[10313,-0.301,370,10369,10291],[10315,0.25,370,10370,10291],[10315,0.247,370,10371,10291],[10315,0.201,370,10372,10291],[10315,-0.05,370,10373,10291],[10315,0.288,370,10374,10291]]},{"id":"coordbench-r2-40deg-2026-paper","name":"CoordBench · 40deg (2026 paper)","group":"预测与决策","metric":"Mean target R²","unit":"raw","public_history":true,"protocol":{"en":"Geographic predictors use the same datasets within this 40deg split column. Five-fold evaluations are repeated over five fold-assignment seeds. Target R² is floored at −1 before dataset averaging; affected unchanged-partition datasets are excluded consistently within a block size. Different block sizes have different eligible cohorts and stay separate. MIND widths/CP are configurations of one model family.","zh":"地理预测器在40deg划分列内使用相同数据集，五折评测重复五个分组种子。各目标R²先截断至−1再按数据集求平均；空间块变化却未改变划分的数据集在对应列中一致排除。不同块大小的有效题组可能不同，分别排行；MIND宽度及CP保留为同家族配置。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.25454v1","published_at":"2026-09-21","status":"paper_baselines"},"model_count":14,"result_count":18,"results":[[10289,-0.053,370,10375,10291],[10292,-0.455,370,10376,10291],[10294,-0.367,370,10377,10291],[10296,-0.661,370,10378,10291],[10298,-0.228,370,10379,10291],[10300,-0.227,370,10380,10291],[7072,0.147,370,10381,10291],[10303,-0.371,370,10382,10291],[10305,-0.399,370,10383,10291],[10307,-0.342,370,10384,10291],[10309,-0.214,370,10385,10291],[10311,0.016,370,10386,10291],[10313,-0.554,370,10387,10291],[10315,0.095,370,10388,10291],[10315,0.121,370,10389,10291],[10315,0.097,370,10390,10291],[10315,-0.219,370,10391,10291],[10315,0.182,370,10392,10291]]},{"id":"dumabench-solo-2026-paper","name":"DUMA-Bench · solo (2026 paper)","group":"安全与可靠性","metric":"Attack success rate","unit":"score","public_history":true,"protocol":{"en":"Fourteen agent backbones are evaluated across the same eight adversarial domains. solo is a separate user-control regime. Attack success is a native failure-rate metric: lower is better. Solo and dual-control rates are not pooled, and higher attack success is not treated as better security.","zh":"十四个Agent底模在相同八类对抗领域评测，solo用户控制条件分别排行。攻击成功率是原生失败率，数值越低越好，不与另一控制条件混排。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.24662v1","published_at":"2026-09-21","status":"paper_baselines"},"score_max":1,"saturation_excluded":true,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"score_decimals":2,"higher_is_better":false,"model_count":14,"result_count":14,"results":[[325,0.079,370,10393,10394],[3738,0.071,370,10393,10394],[283,0.05,370,10393,10394],[345,0.336,370,10393,10394],[5406,0.293,370,10393,10394],[353,0.314,370,10393,10394],[354,0.44,370,10393,10394],[2058,0.257,370,10393,10394],[4064,0.279,370,10393,10394],[4080,0.314,370,10393,10394],[4425,0.343,370,10393,10394],[9331,0.329,370,10393,10394],[6412,0.393,370,10393,10394],[2278,0.279,370,10393,10394]]},{"id":"dumabench-dual-control-2026-paper","name":"DUMA-Bench · dual-control (2026 paper)","group":"安全与可靠性","metric":"Attack success rate","unit":"score","public_history":true,"protocol":{"en":"Fourteen agent backbones are evaluated across the same eight adversarial domains. dual-control is a separate user-control regime. Attack success is a native failure-rate metric: lower is better. Solo and dual-control rates are not pooled, and higher attack success is not treated as better security.","zh":"十四个Agent底模在相同八类对抗领域评测，dual-control用户控制条件分别排行。攻击成功率是原生失败率，数值越低越好，不与另一控制条件混排。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.24662v1","published_at":"2026-09-21","status":"paper_baselines"},"score_max":1,"saturation_excluded":true,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"score_decimals":2,"higher_is_better":false,"model_count":14,"result_count":14,"results":[[325,0.121,370,10395,10394],[3738,0.164,370,10395,10394],[283,0.086,370,10395,10394],[345,0.386,370,10395,10394],[5406,0.357,370,10395,10394],[353,0.393,370,10395,10394],[354,0.56,370,10395,10394],[2058,0.647,370,10395,10394],[4064,0.729,370,10395,10394],[4080,0.793,370,10395,10394],[4425,0.393,370,10395,10394],[9331,0.421,370,10395,10394],[6412,0.357,370,10395,10394],[2278,0.35,370,10395,10394]]},{"id":"mini-commonforms-map-2026-paper","name":"mini-CommonForms · field detection (2026 paper)","group":"视觉与视频","metric":"COCO mAP50:95","unit":"score","public_history":true,"protocol":{"en":"Nine detector checkpoints use the new, manually curated and consistently relabeled mini-CommonForms evaluation set for Text, Choice and Signature fields. The selected overall score uses COCO IoU thresholds 0.50:0.95. MMDetection models train for 24 epochs; FFDNet checkpoints were trained on the original CommonForms data, retained explicitly as a different training condition.","zh":"九个检测检查点在新人工筛选并统一重标注的mini-CommonForms集合上识别Text、Choice和Signature表单域。总体mAP采用COCO的0.50:0.95交并比阈值。MMDetection模型训练24轮，FFDNet使用原CommonForms训练权重，明确保留不同训练条件。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.23679v1","published_at":"2026-09-20","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":9,"result_count":9,"results":[[10396,27.5,370,10397,10398],[10399,24.1,370,10400,10398],[10401,34.6,370,10402,10398],[10403,33.7,370,10404,10398],[10405,30.0,370,10406,10398],[9275,28.1,370,10407,10398],[10408,26.9,370,10409,10398],[9278,24.9,370,10410,10398],[10411,13.0,370,10412,10398]]},{"id":"prism-on-screen-2026-paper","name":"PRISM-Bench · on-screen (2026 paper)","group":"视觉与视频","metric":"GT-calibrated audiovisual rubric sum","unit":"score","public_history":true,"protocol":{"en":"Eight audiovisual generators use the same on-screen subset of the 900 human-verified PRISM samples. Each applicable dimension/audio-type cell is a ground-truth-calibrated mean on 0–10. Final sums speech, music and sound over four dimensions, including audiovisual coherence. The native rubric sum is preserved. Licensed source media remain evaluator-held; the factorized protocol and public leaderboard are documented.","zh":"八个音视频生成模型使用900项人工验证PRISM实例中的相同on-screen子集。每个适用维度与音频类型的分数为真实样本校准后的0–10均值，Final汇总语音、音乐和声音的四个维度，包含音画一致性。保留原生评分总和。授权媒体由评测方保管，论文公开了分解评测流程和排行榜。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.04867v1","published_at":"2026-09-04","status":"paper_baselines"},"score_max":120,"saturation_excluded":true,"scale_label":{"en":"Score / 120","zh":"分数 / 120"},"score_decimals":2,"model_count":8,"result_count":8,"results":[[10413,97.73,370,10414,10415],[10416,85.99,370,10414,10415],[10417,84.11,370,10414,10415],[10418,83.52,370,10414,10415],[10419,69.82,370,10414,10415],[10420,49.5,370,10414,10415],[7134,40.15,370,10414,10415],[10421,45.04,370,10414,10415]]},{"id":"prism-off-screen-2026-paper","name":"PRISM-Bench · off-screen (2026 paper)","group":"视觉与视频","metric":"GT-calibrated audiovisual rubric sum","unit":"score","public_history":true,"protocol":{"en":"Eight audiovisual generators use the same off-screen subset of the 900 human-verified PRISM samples. Each applicable dimension/audio-type cell is a ground-truth-calibrated mean on 0–10. Final sums speech, music and sound over three dimensions; audiovisual coherence is inapplicable off-screen. The native rubric sum is preserved. Licensed source media remain evaluator-held; the factorized protocol and public leaderboard are documented.","zh":"八个音视频生成模型使用900项人工验证PRISM实例中的相同off-screen子集。每个适用维度与音频类型的分数为真实样本校准后的0–10均值，Final汇总语音、音乐和声音的三个维度；画外声音不评音画一致性。保留原生评分总和。授权媒体由评测方保管，论文公开了分解评测流程和排行榜。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.04867v1","published_at":"2026-09-04","status":"paper_baselines"},"score_max":90,"saturation_excluded":true,"scale_label":{"en":"Score / 90","zh":"分数 / 90"},"score_decimals":2,"model_count":8,"result_count":8,"results":[[10413,78.11,370,10422,10415],[10416,72.76,370,10422,10415],[10417,67.03,370,10422,10415],[10418,66.08,370,10422,10415],[10419,61.83,370,10422,10415],[10420,40.16,370,10422,10415],[7134,34.46,370,10422,10415],[10421,34.28,370,10422,10415]]},{"id":"prism-mixed-2026-paper","name":"PRISM-Bench · mixed (2026 paper)","group":"视觉与视频","metric":"GT-calibrated audiovisual rubric sum","unit":"score","public_history":true,"protocol":{"en":"Eight audiovisual generators use the same mixed subset of the 900 human-verified PRISM samples. Each applicable dimension/audio-type cell is a ground-truth-calibrated mean on 0–10. Final sums speech, music and sound over four dimensions, including audiovisual coherence. The native rubric sum is preserved. Licensed source media remain evaluator-held; the factorized protocol and public leaderboard are documented.","zh":"八个音视频生成模型使用900项人工验证PRISM实例中的相同mixed子集。每个适用维度与音频类型的分数为真实样本校准后的0–10均值，Final汇总语音、音乐和声音的四个维度，包含音画一致性。保留原生评分总和。授权媒体由评测方保管，论文公开了分解评测流程和排行榜。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.04867v1","published_at":"2026-09-04","status":"paper_baselines"},"score_max":120,"saturation_excluded":true,"scale_label":{"en":"Score / 120","zh":"分数 / 120"},"score_decimals":2,"model_count":8,"result_count":8,"results":[[10413,99.88,370,10423,10415],[10416,94.61,370,10423,10415],[10417,88.04,370,10423,10415],[10418,83.61,370,10423,10415],[10419,89.04,370,10423,10415],[10420,47.96,370,10423,10415],[7134,64.38,370,10423,10415],[10421,38.85,370,10423,10415]]},{"id":"webench-generation-2026-paper","name":"WeBench-MMGenEdit · generation (2026 paper)","group":"视觉与视频","metric":"Weighted knowledge/visual checklist score","unit":"score","public_history":true,"protocol":{"en":"The complete 28-system panel on the shared bilingual knowledge-intensive generation benchmark preserves direct models, reasoning pipelines and agent-policy/image-backend combinations. Generation weights IA/TKA/VKA/VQ by 0.20/0.35/0.35/0.10; editing weights IA/TKA/VKA/CP/VQ by0.20/0.30/0.30/0.10/0.10. W_Avg is a native 0–100 weighted checklist score. Improvement-only summary rows are excluded; every policy, image model and harness is recorded.","zh":"完整28系统面板在统一双语知识密集型generation基准上评测，保留直接模型、推理流程和Agent策略与图像后端组合。生成的IA/TKA/VKA/VQ权重为0.20/0.35/0.35/0.10，编辑的IA/TKA/VKA/CP/VQ权重为0.20/0.30/0.30/0.10/0.10。W_Avg保留为原生0–100加权清单分，排除纯提升汇总行，并记录策略、图像模型和运行框架。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.05171v1","published_at":"2026-09-04","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":28,"result_count":28,"results":[[6368,48.79,370,10424,10425],[10426,55.12,370,10427,10425],[6729,53.81,370,10428,10425],[10429,43.58,370,10430,10425],[10431,22.69,370,10432,10425],[10433,21.25,370,10434,10425],[10435,51.28,370,10436,10425],[10437,52.26,370,10438,10425],[10439,25.35,370,10440,10425],[10441,29.2,370,10442,10425],[10443,32.63,370,10444,10425],[10445,74.06,370,10446,10425],[10447,72.17,370,10448,10425],[10449,65.73,370,10450,10425],[10451,62.32,370,10452,10425],[10453,62.75,370,10454,10425],[10455,62.91,370,10456,10425],[10457,62.81,370,10458,10425],[10459,51.86,370,10460,10425],[10461,51.21,370,10462,10425],[10463,28.34,370,10464,10425],[10465,60.83,370,10466,10425],[10467,64.31,370,10468,10425],[10469,57.5,370,10470,10425],[10471,60.08,370,10472,10425],[10473,33.54,370,10474,10425],[10475,38.65,370,10476,10425],[10477,47.39,370,10478,10425]]},{"id":"webench-editing-2026-paper","name":"WeBench-MMGenEdit · editing (2026 paper)","group":"视觉与视频","metric":"Weighted knowledge/visual checklist score","unit":"score","public_history":true,"protocol":{"en":"The complete 28-system panel on the shared bilingual knowledge-intensive editing benchmark preserves direct models, reasoning pipelines and agent-policy/image-backend combinations. Generation weights IA/TKA/VKA/VQ by 0.20/0.35/0.35/0.10; editing weights IA/TKA/VKA/CP/VQ by0.20/0.30/0.30/0.10/0.10. W_Avg is a native 0–100 weighted checklist score. Improvement-only summary rows are excluded; every policy, image model and harness is recorded.","zh":"完整28系统面板在统一双语知识密集型editing基准上评测，保留直接模型、推理流程和Agent策略与图像后端组合。生成的IA/TKA/VKA/VQ权重为0.20/0.35/0.35/0.10，编辑的IA/TKA/VKA/CP/VQ权重为0.20/0.30/0.30/0.10/0.10。W_Avg保留为原生0–100加权清单分，排除纯提升汇总行，并记录策略、图像模型和运行框架。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.05171v1","published_at":"2026-09-04","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":28,"result_count":28,"results":[[6368,48.99,370,10424,10425],[10426,51.79,370,10427,10425],[6729,48.51,370,10428,10425],[10429,48.6,370,10430,10425],[10431,24.99,370,10432,10425],[10433,24.74,370,10434,10425],[10435,50.13,370,10436,10425],[10437,47.92,370,10438,10425],[10439,24.64,370,10440,10425],[10441,23.38,370,10442,10425],[10443,24.93,370,10444,10425],[10445,73.55,370,10446,10425],[10447,69.41,370,10448,10425],[10449,65.5,370,10450,10425],[10451,60.3,370,10452,10425],[10453,55.46,370,10454,10425],[10455,56.58,370,10456,10425],[10457,60.79,370,10458,10425],[10459,45.8,370,10460,10425],[10461,42.98,370,10462,10425],[10463,29.42,370,10464,10425],[10465,58.56,370,10466,10425],[10467,62.52,370,10468,10425],[10469,53.08,370,10470,10425],[10471,54.43,370,10472,10425],[10473,38.5,370,10474,10425],[10475,46.24,370,10476,10425],[10477,49.98,370,10478,10425]]},{"id":"agos-easy-2026-paper","name":"AGOS-Bench · easy (2026 paper)","group":"Agent 与工具","metric":"Cooperative episode success rate","unit":"%","public_history":true,"protocol":{"en":"Nine independent VLM checkpoints, each with baseline and AGOS-Agent configurations, execute the same easy subset of the 210-episode held-out CARLA town. Success requires coordinated agentic ground-object search under the published episode condition. The companion7700train/validationepisodes do not enter this held-out score; human reference rows are excluded.","zh":"九个独立视觉语言模型各以基础配置和AGOS-Agent配置执行同一组easy测试任务，属于独立CARLA城镇中的210项留出任务。成功按论文的协同地面目标搜索终止条件判定。另有7,700项训练及验证任务，不计入此处成绩；排除人类参照。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.08402v1","published_at":"2026-09-08","status":"paper_baselines"},"model_count":9,"result_count":18,"results":[[502,0.271,370,10479,10480],[502,0.757,370,10481,10480],[5136,0.214,370,10479,10480],[5136,0.757,370,10481,10480],[3196,0.257,370,10479,10480],[3196,0.614,370,10481,10480],[4961,0.14300000000000002,370,10479,10480],[4961,0.6809999999999999,370,10481,10480],[6630,0.157,370,10479,10480],[6630,0.32899999999999996,370,10481,10480],[2754,0.013999999999999999,370,10479,10480],[2754,0.34299999999999997,370,10481,10480],[6298,0.214,370,10479,10480],[6298,0.257,370,10481,10480],[6342,0.129,370,10479,10480],[6342,0.32899999999999996,370,10481,10480],[6238,0.14300000000000002,370,10479,10480],[6238,0.32899999999999996,370,10481,10480]]},{"id":"agos-mid-2026-paper","name":"AGOS-Bench · mid (2026 paper)","group":"Agent 与工具","metric":"Cooperative episode success rate","unit":"%","public_history":true,"protocol":{"en":"Nine independent VLM checkpoints, each with baseline and AGOS-Agent configurations, execute the same mid subset of the 210-episode held-out CARLA town. Success requires coordinated agentic ground-object search under the published episode condition. The companion7700train/validationepisodes do not enter this held-out score; human reference rows are excluded.","zh":"九个独立视觉语言模型各以基础配置和AGOS-Agent配置执行同一组mid测试任务，属于独立CARLA城镇中的210项留出任务。成功按论文的协同地面目标搜索终止条件判定。另有7,700项训练及验证任务，不计入此处成绩；排除人类参照。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.08402v1","published_at":"2026-09-08","status":"paper_baselines"},"model_count":9,"result_count":18,"results":[[502,0.157,370,10479,10480],[502,0.586,370,10481,10480],[5136,0.157,370,10479,10480],[5136,0.614,370,10481,10480],[3196,0.043,370,10479,10480],[3196,0.35700000000000004,370,10481,10480],[4961,0.043,370,10479,10480],[4961,0.44299999999999995,370,10481,10480],[6630,0.028999999999999998,370,10479,10480],[6630,0.043,370,10481,10480],[2754,0.0,370,10479,10480],[2754,0.114,370,10481,10480],[6298,0.057,370,10479,10480],[6298,0.071,370,10481,10480],[6342,0.057,370,10479,10480],[6342,0.1,370,10481,10480],[6238,0.1,370,10479,10480],[6238,0.028999999999999998,370,10481,10480]]},{"id":"agos-hard-2026-paper","name":"AGOS-Bench · hard (2026 paper)","group":"Agent 与工具","metric":"Cooperative episode success rate","unit":"%","public_history":true,"protocol":{"en":"Nine independent VLM checkpoints, each with baseline and AGOS-Agent configurations, execute the same hard subset of the 210-episode held-out CARLA town. Success requires coordinated agentic ground-object search under the published episode condition. The companion7700train/validationepisodes do not enter this held-out score; human reference rows are excluded.","zh":"九个独立视觉语言模型各以基础配置和AGOS-Agent配置执行同一组hard测试任务，属于独立CARLA城镇中的210项留出任务。成功按论文的协同地面目标搜索终止条件判定。另有7,700项训练及验证任务，不计入此处成绩；排除人类参照。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.08402v1","published_at":"2026-09-08","status":"paper_baselines"},"model_count":9,"result_count":18,"results":[[502,0.171,370,10479,10480],[502,0.5429999999999999,370,10481,10480],[5136,0.086,370,10479,10480],[5136,0.557,370,10481,10480],[3196,0.013999999999999999,370,10479,10480],[3196,0.314,370,10481,10480],[4961,0.043,370,10479,10480],[4961,0.34299999999999997,370,10481,10480],[6630,0.013999999999999999,370,10479,10480],[6630,0.043,370,10481,10480],[2754,0.0,370,10479,10480],[2754,0.129,370,10481,10480],[6298,0.086,370,10479,10480],[6298,0.013999999999999999,370,10481,10480],[6342,0.071,370,10479,10480],[6342,0.114,370,10481,10480],[6238,0.1,370,10479,10480],[6238,0.028999999999999998,370,10481,10480]]},{"id":"hybriddeepresearch-smolagents-2026-paper","name":"HybridDeepResearch · smolagents (2026 paper)","group":"Agent 与工具","metric":"Pass@8","unit":"%","public_history":true,"protocol":{"en":"All six published open-weight checkpoints use smolagents and Corpus Search on the identical380-instance full HybridDeepResearch test. Tasks require conjunctive constraints across structured SQL and unstructured search. Pass@8 counts a task solved in at least one of eight sampled trials; Avg@8 is a separate metric and is not substituted.","zh":"全部六个开放权重模型使用smolagents和Corpus Search，执行同一组380项完整测试任务。任务要求同时满足结构化SQL与非结构化搜索中的约束。Pass@8表示八次采样中至少一次解题成功；不与另列的Avg@8混用。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.09410v1","published_at":"2026-09-08","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[4969,0.3605,370,10482,10483],[4070,0.4053,370,10482,10483],[2584,0.5289,370,10482,10483],[2582,0.4395,370,10482,10483],[2576,0.5658,370,10482,10483],[10484,0.7237,370,10482,10483]]},{"id":"hybriddeepresearch-miroflow-2026-paper","name":"HybridDeepResearch · MiroFlow (2026 paper)","group":"Agent 与工具","metric":"Pass@8","unit":"%","public_history":true,"protocol":{"en":"All six published open-weight checkpoints use MiroFlow and Corpus Search on the identical380-instance full HybridDeepResearch test. Tasks require conjunctive constraints across structured SQL and unstructured search. Pass@8 counts a task solved in at least one of eight sampled trials; Avg@8 is a separate metric and is not substituted.","zh":"全部六个开放权重模型使用MiroFlow和Corpus Search，执行同一组380项完整测试任务。任务要求同时满足结构化SQL与非结构化搜索中的约束。Pass@8表示八次采样中至少一次解题成功；不与另列的Avg@8混用。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.09410v1","published_at":"2026-09-08","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[4969,0.4158,370,10485,10483],[4070,0.5316,370,10485,10483],[2584,0.5710999999999999,370,10485,10483],[2582,0.5395,370,10485,10483],[2576,0.6342,370,10485,10483],[10484,0.7579,370,10485,10483]]},{"id":"tabbenchbio-reference-2026-paper","name":"TabBench-Bio · reference cell (2026 paper)","group":"知识与推理","metric":"Classification macro F1","unit":"score","public_history":true,"protocol":{"en":"The full19-model reference-cell panel uses10000features and100training samples with shared cross-validation folds. The selected macro F1 averages only the common27classification targets; the other12regression targets contribute to Elo but not this metric. TabPFN-Wide variants are classification-only, so they are comparable on this selected classification panel. Constant controls and Elo are excluded.","zh":"完整19模型参考配置使用10,000个特征、100条训练样本和共享交叉验证划分。所选宏F1只对共同27项分类目标平均；另12项回归目标参与Elo，但不进入此指标。TabPFN-Wide只支持分类，在这组分类面板中保留。排除常数参照和Elo。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.07441v1","published_at":"2026-09-07","status":"paper_baselines"},"score_max":1,"saturation_excluded":true,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"score_decimals":3,"model_count":19,"result_count":19,"results":[[10486,0.761,370,10487,10488],[10489,0.747,370,10487,10488],[10490,0.739,370,10487,10488],[10491,0.727,370,10487,10488],[10492,0.731,370,10487,10488],[10493,0.719,370,10487,10488],[10494,0.719,370,10487,10488],[9203,0.734,370,10487,10488],[10495,0.699,370,10487,10488],[10496,0.744,370,10487,10488],[10497,0.728,370,10487,10488],[10498,0.713,370,10487,10488],[10499,0.696,370,10487,10488],[10500,0.677,370,10487,10488],[10501,0.69,370,10487,10488],[10502,0.6,370,10487,10488],[10503,0.668,370,10487,10488],[6811,0.654,370,10487,10488],[10504,0.634,370,10487,10488]]},{"id":"ontologybench-composition-2026-paper","name":"OntologyBench · phenotype triplets (2026 paper)","group":"知识与推理","metric":"nDCG at10","unit":"score","public_history":true,"protocol":{"en":"Eleven learned embedding/checkpoint configurations rank disease candidates for the same2537held-out phenotype-triplet queries; each query averages2.26positive diseases. nDCG@10 preserves graded ranking quality on0–1. Tier1-only and all-tier fine-tunes remain explicit configurations; BM25 is an excluded lexical reference.","zh":"十一个训练后的嵌入模型及配置为同一组2,537条留出表型三元组查询检索疾病，每条查询平均2.26个相关疾病。nDCG@10保留原生0–1排序指标。只用第一层数据或使用全部层数据的微调分别注明；排除词法参照BM25。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.08174v1","published_at":"2026-09-08","status":"paper_baselines"},"score_max":1,"saturation_excluded":true,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"score_decimals":3,"model_count":9,"result_count":11,"results":[[10505,0.123,370,10506,10507],[10508,0.143,370,10506,10507],[10509,0.131,370,10506,10507],[10510,0.151,370,10506,10507],[10510,0.171,370,10511,10507],[10512,0.244,370,10513,10507],[10514,0.109,370,10506,10507],[10514,0.158,370,10511,10507],[10515,0.313,370,10513,10507],[10516,0.08,370,10506,10507],[10517,0.169,370,10506,10507]]},{"id":"cmnie-trigger-2026-paper","name":"CMNIE · trigger (2026 paper)","group":"知识与推理","metric":"Trigger classification micro F1","unit":"score","public_history":true,"protocol":{"en":"All21published supervised, zero-shot and fine-tuned IE systems use the same3000-instance Chinese military-news test set. The source schema includes7event types,10argument roles,7entity types and8relation types. This exact-match trigger micro-F1 stays on0–100; group headers and fine-tuning/scaffold identities are mapped from the original rows.","zh":"全部21个已发表监督、零样本和微调信息抽取系统使用同一组3,000条中文军事新闻测试实例。标注包含七类事件、十类论元角色、七类实体和八类关系。trigger精确匹配微F1保留原生0–100刻度；分组标题、微调与框架身份按原表映射。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.10722v1","published_at":"2026-09-09","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":21,"result_count":21,"results":[[10518,65.3,370,10519,10520],[10521,63.0,370,10519,10520],[10522,47.3,370,10519,10520],[1748,37.9,370,10523,10520],[1659,35.3,370,10523,10520],[1645,32.7,370,10523,10520],[1651,45.3,370,10523,10520],[550,24.2,370,10523,10520],[562,23.7,370,10523,10520],[524,43.4,370,10523,10520],[516,36.1,370,10523,10520],[10524,34.8,370,10523,10520],[10525,49.4,370,10523,10520],[3748,39.4,370,10523,10520],[10526,67.2,370,10527,10520],[10528,69.5,370,10527,10520],[10529,69.1,370,10527,10520],[10530,70.3,370,10527,10520],[10531,59.5,370,10527,10520],[10532,57.7,370,10527,10520],[10533,46.8,370,10527,10520]]},{"id":"cmnie-argument-2026-paper","name":"CMNIE · argument (2026 paper)","group":"知识与推理","metric":"Argument classification micro F1","unit":"score","public_history":true,"protocol":{"en":"All21published supervised, zero-shot and fine-tuned IE systems use the same3000-instance Chinese military-news test set. The source schema includes7event types,10argument roles,7entity types and8relation types. This exact-match argument micro-F1 stays on0–100; group headers and fine-tuning/scaffold identities are mapped from the original rows.","zh":"全部21个已发表监督、零样本和微调信息抽取系统使用同一组3,000条中文军事新闻测试实例。标注包含七类事件、十类论元角色、七类实体和八类关系。argument精确匹配微F1保留原生0–100刻度；分组标题、微调与框架身份按原表映射。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.10722v1","published_at":"2026-09-09","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":21,"result_count":21,"results":[[10518,39.2,370,10519,10520],[10521,53.9,370,10519,10520],[10522,20.4,370,10519,10520],[1748,26.3,370,10523,10520],[1659,26.7,370,10523,10520],[1645,29.7,370,10523,10520],[1651,29.3,370,10523,10520],[550,20.8,370,10523,10520],[562,23.1,370,10523,10520],[524,29.5,370,10523,10520],[516,29.4,370,10523,10520],[10524,27.8,370,10523,10520],[10525,34.4,370,10523,10520],[3748,31.8,370,10523,10520],[10526,46.8,370,10527,10520],[10528,47.4,370,10527,10520],[10529,45.2,370,10527,10520],[10530,47.9,370,10527,10520],[10531,47.7,370,10527,10520],[10532,45.8,370,10527,10520],[10533,34.6,370,10527,10520]]},{"id":"cmnie-entity-2026-paper","name":"CMNIE · entity (2026 paper)","group":"知识与推理","metric":"Named-entity micro F1","unit":"score","public_history":true,"protocol":{"en":"All21published supervised, zero-shot and fine-tuned IE systems use the same3000-instance Chinese military-news test set. The source schema includes7event types,10argument roles,7entity types and8relation types. This exact-match entity micro-F1 stays on0–100; group headers and fine-tuning/scaffold identities are mapped from the original rows.","zh":"全部21个已发表监督、零样本和微调信息抽取系统使用同一组3,000条中文军事新闻测试实例。标注包含七类事件、十类论元角色、七类实体和八类关系。entity精确匹配微F1保留原生0–100刻度；分组标题、微调与框架身份按原表映射。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.10722v1","published_at":"2026-09-09","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":21,"result_count":21,"results":[[10518,67.6,370,10519,10520],[10521,70.2,370,10519,10520],[10522,64.3,370,10519,10520],[1748,39.9,370,10523,10520],[1659,41.6,370,10523,10520],[1645,45.0,370,10523,10520],[1651,47.0,370,10523,10520],[550,38.4,370,10523,10520],[562,26.9,370,10523,10520],[524,37.9,370,10523,10520],[516,39.9,370,10523,10520],[10524,42.7,370,10523,10520],[10525,46.7,370,10523,10520],[3748,46.5,370,10523,10520],[10526,71.9,370,10527,10520],[10528,71.1,370,10527,10520],[10529,69.9,370,10527,10520],[10530,71.0,370,10527,10520],[10531,67.4,370,10527,10520],[10532,66.7,370,10527,10520],[10533,68.6,370,10527,10520]]},{"id":"cmnie-relation-2026-paper","name":"CMNIE · relation (2026 paper)","group":"知识与推理","metric":"Relation extraction micro F1","unit":"score","public_history":true,"protocol":{"en":"All21published supervised, zero-shot and fine-tuned IE systems use the same3000-instance Chinese military-news test set. The source schema includes7event types,10argument roles,7entity types and8relation types. This exact-match relation micro-F1 stays on0–100; group headers and fine-tuning/scaffold identities are mapped from the original rows.","zh":"全部21个已发表监督、零样本和微调信息抽取系统使用同一组3,000条中文军事新闻测试实例。标注包含七类事件、十类论元角色、七类实体和八类关系。relation精确匹配微F1保留原生0–100刻度；分组标题、微调与框架身份按原表映射。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.10722v1","published_at":"2026-09-09","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":21,"result_count":21,"results":[[10518,8.3,370,10519,10520],[10521,19.5,370,10519,10520],[10522,25.1,370,10519,10520],[1748,8.0,370,10523,10520],[1659,12.8,370,10523,10520],[1645,16.1,370,10523,10520],[1651,13.8,370,10523,10520],[550,8.3,370,10523,10520],[562,8.3,370,10523,10520],[524,12.9,370,10523,10520],[516,15.0,370,10523,10520],[10524,18.1,370,10523,10520],[10525,19.2,370,10523,10520],[3748,24.2,370,10523,10520],[10526,48.7,370,10527,10520],[10528,49.7,370,10527,10520],[10529,49.5,370,10527,10520],[10530,50.2,370,10527,10520],[10531,48.5,370,10527,10520],[10532,46.6,370,10527,10520],[10533,23.6,370,10527,10520]]},{"id":"assaybench-loop-base-2026-paper","name":"AssayBench-Loop · base LLMs (2026 paper)","group":"医疗","metric":"Normalized discovery-curve area","unit":"score","public_history":true,"protocol":{"en":"The complete ten-base-LLM panel sequentially selects gene assays on the same20held-out genome-wide screens, with20validation and1349trainingscreens kept separate. The selected nAUC measures the normalized area under cumulative hit discovery as assay budget is spent, on the native0–100scale. All rows here include observed hit labels; label-removal ablations, learned acquisition policies and handoff agents are separate experimental panels.","zh":"完整十个基础语言模型面板在相同20项留出全基因组筛选中依次选择基因实验；另20项验证和1,349项训练筛选单独保留。nAUC衡量随实验预算增加的累计命中发现曲线面积，保留原生0–100刻度。本组都提供已观测命中标签；去标签消融、学习型选样策略和交接Agent属于其他实验面板。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.11877v1","published_at":"2026-09-10","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":10,"result_count":10,"results":[[3186,16.3,370,10534,10535],[227,16.6,370,10534,10535],[3187,14.1,370,10534,10535],[10536,17.3,370,10534,10535],[10537,13.3,370,10534,10535],[10538,8.5,370,10534,10535],[3763,10.5,370,10534,10535],[9677,20.6,370,10534,10535],[3196,18.7,370,10534,10535],[762,20.5,370,10534,10535]]},{"id":"mindtopo-2dmaze-2026-paper","name":"MindTopo ·2D Maze (2026 paper)","group":"知识与推理","metric":"Reasoning task accuracy","unit":"%","public_history":true,"protocol":{"en":"All14independent MLLM checkpoints solve the same2D-Maze reasoning items in MindTopo, whose13environments separate eight reasoning and five planning tasks. Reasoning tasks contribute roughly1000items each; accuracy compares each model output to the task’s deterministic target. Human/chance references are excluded and every published model zero is retained.","zh":"全部14个独立多模态模型解答MindTopo中相同的2D迷宫推理题。资源的13个环境分为八类推理和五类规划任务，每类推理约1,000题。正确率按任务确定性目标判定；排除人类、随机参照，保留模型原文零分。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.11900v1","published_at":"2026-09-10","status":"paper_baselines"},"model_count":14,"result_count":14,"results":[[4979,0.8440000000000001,370,10539,10540],[3196,0.58,370,10539,10540],[4958,0.152,370,10539,10540],[5021,0.28800000000000003,370,10539,10540],[502,0.317,370,10539,10540],[10541,0.126,370,10539,10540],[2576,0.122,370,10539,10540],[10542,0.225,370,10539,10540],[10543,0.18100000000000002,370,10539,10540],[3245,0.15,370,10539,10540],[6374,0.264,370,10539,10540],[7066,0.092,370,10539,10540],[9778,0.126,370,10539,10540],[10544,0.14400000000000002,370,10539,10540]]},{"id":"embodiedbenchforge-offline-2026-paper","name":"Embodied-BenchForge ·offline EQA (2026 paper)","group":"Agent 与工具","metric":"Capability-balanced cross-benchmark mean","unit":"score","public_history":true,"protocol":{"en":"Eleven MLLMs use the same six Offline EQA benchmarks,10000items each. The published Mean equally combines capability-dimension scores under the benchmark’s six-scenario aggregation. This native0–100summary covers SpatialHome Task/Nav,Drive,Quadruped,UAV andArm; it is separate from the220-task interactive embodied track and construction-quality judge scores.","zh":"十一个多模态模型使用相同六套离线具身问答基准，每套10,000题。Mean按原文的能力维度与六场景汇总方式保留为原生0–100分，覆盖家居任务、家居导航、驾驶、四足、无人机和机械臂；与220项交互任务及基准构建质量评分分别记录。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.13082v1","published_at":"2026-09-11","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":11,"result_count":11,"results":[[3196,57.67,370,10545,10546],[3578,56.61,370,10545,10546],[6133,51.97,370,10545,10546],[6771,49.86,370,10545,10546],[6235,47.23,370,10545,10546],[6904,42.96,370,10545,10546],[3572,42.54,370,10545,10546],[4969,35.97,370,10545,10546],[3623,34.73,370,10545,10546],[5828,32.07,370,10545,10546],[6340,30.34,370,10545,10546]]},{"id":"egomaize-miou-2026-paper","name":"EgoMaize (2026 paper)","group":"视觉与视频","metric":"Mean intersection over union","unit":"score","public_history":true,"protocol":{"en":"All nine published instance-segmentation architectures use the fixed EgoMaize test split of45images with170positive plant instances. Source-image splitting is stratified by density, occlusion and ignore coverage; unreliable maize regions are ignored consistently rather than relabeled background. mIoU is preserved as native0–1segmentation overlap, separate from AP and count-error columns.","zh":"全部九个已发表实例分割架构使用固定EgoMaize测试集，含45张图像、170个正植物实例。源图划分按密度、遮挡和忽略区覆盖分层；不可靠玉米区域统一忽略。mIoU保留为原生0–1分割重叠指标，与AP及计数误差分别记录。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.12350v1","published_at":"2026-09-11","status":"paper_baselines"},"score_max":1,"saturation_excluded":true,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"score_decimals":3,"model_count":9,"result_count":9,"results":[[10547,0.0633,370,10548,10549],[10550,0.0649,370,10548,10549],[10551,0.2575,370,10548,10549],[10552,0.3544,370,10548,10549],[10553,0.2635,370,10548,10549],[10554,0.2493,370,10548,10549],[10555,0.5014,370,10548,10549],[10556,0.4061,370,10548,10549],[10557,0.092,370,10548,10549]]},{"id":"cadworld-success-2026-paper","name":"CADWorld (2026 paper)","group":"Agent 与工具","metric":"Saved-artifact task success rate","unit":"%","public_history":true,"protocol":{"en":"All seven agents run the full200-task FreeCAD benchmark in the same Ubuntu VM, with1920x1080screenshots,8CPUcores,8GBRAM and64GBdisk. Success requires the saved FCStd artifact and task-specific geometry, parametric, constraint, manufacturing or simulation checks to pass. Merely finishing is a separate metric; the human reference is excluded.","zh":"全部七个Agent执行完整200项FreeCAD任务，使用统一Ubuntu虚拟机、1920×1080截图、八核CPU、8GB内存和64GB磁盘。成功要求保存FCStd产物，并通过对应几何、参数、约束、制造或仿真检查；完成对话另列指标，排除人类参照。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.16251v1","published_at":"2026-09-14","status":"paper_baselines"},"model_count":7,"result_count":7,"results":[[10558,0.175,370,10559,10560],[10561,0.16,370,10559,10560],[3490,0.075,370,10559,10560],[10562,0.015,370,10559,10560],[10563,0.005,370,10559,10560],[10564,0.0,370,10559,10560],[3491,0.0,370,10559,10560]]},{"id":"rmmbench-short-v2-2026-paper","name":"RMMBench · short (2026 paper)","group":"Agent 与工具","metric":"Task success rate","unit":"%","public_history":true,"protocol":{"en":"Eight VLM backends load the same fixed JSON task configurations in MuJoCo. 39 short-horizon scenarios; three episodes each. Binary SR is kept separate from atomic completion, execution efficiency and grounding metrics. This panel uses the explicitly stated 90-scenario protocol in the 23 October v2 PDF; the first release was 4 October.","zh":"八个VLM端点在MuJoCo加载相同固定JSON任务配置。39 short-horizon scenarios; three episodes each。二元任务成功率与原子完成度、执行效率及定位指标分别处理。本面板采用10月23日v2 PDF明确规定的90场景协议，首次发布为10月4日。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.05414v2","published_at":"2026-10-04","status":"paper_baselines"},"model_count":8,"result_count":8,"results":[[6130,0.5811999999999999,370,10565,10566],[6133,0.5897,370,10565,10566],[10567,0.4103,370,10565,10566],[10568,0.5640999999999999,370,10565,10566],[764,0.5640999999999999,370,10565,10566],[3155,0.4188,370,10565,10566],[3153,0.22219999999999998,370,10565,10566],[4977,0.8889,370,10565,10566]]},{"id":"rmmbench-long-v2-2026-paper","name":"RMMBench · long (2026 paper)","group":"Agent 与工具","metric":"Task success rate","unit":"%","public_history":true,"protocol":{"en":"Eight VLM backends load the same fixed JSON task configurations in MuJoCo. 21 long-horizon scenarios; three episodes each. Binary SR is kept separate from atomic completion, execution efficiency and grounding metrics. This panel uses the explicitly stated 90-scenario protocol in the 23 October v2 PDF; the first release was 4 October.","zh":"八个VLM端点在MuJoCo加载相同固定JSON任务配置。21 long-horizon scenarios; three episodes each。二元任务成功率与原子完成度、执行效率及定位指标分别处理。本面板采用10月23日v2 PDF明确规定的90场景协议，首次发布为10月4日。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.05414v2","published_at":"2026-10-04","status":"paper_baselines"},"model_count":8,"result_count":8,"results":[[6130,0.4286,370,10569,10566],[6133,0.381,370,10569,10566],[10567,0.1429,370,10569,10566],[10568,0.254,370,10569,10566],[764,0.3333,370,10569,10566],[3155,0.1746,370,10569,10566],[3153,0.0317,370,10569,10566],[4977,0.5872999999999999,370,10569,10566]]},{"id":"rmmbench-navigation-bev-v2-2026-paper","name":"RMMBench · navigation-bev (2026 paper)","group":"Agent 与工具","metric":"Task success rate","unit":"%","public_history":true,"protocol":{"en":"Eight VLM backends load the same fixed JSON task configurations in MuJoCo. 30 composite navigation tasks; three episodes each; BEV mapping. Binary SR is kept separate from atomic completion, execution efficiency and grounding metrics. This panel uses the explicitly stated 90-scenario protocol in the 23 October v2 PDF; the first release was 4 October.","zh":"八个VLM端点在MuJoCo加载相同固定JSON任务配置。30 composite navigation tasks; three episodes each; BEV mapping。二元任务成功率与原子完成度、执行效率及定位指标分别处理。本面板采用10月23日v2 PDF明确规定的90场景协议，首次发布为10月4日。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.05414v2","published_at":"2026-10-04","status":"paper_baselines"},"model_count":8,"result_count":8,"results":[[6130,0.2556,370,10570,10566],[6133,0.3,370,10570,10566],[10567,0.0889,370,10570,10566],[10568,0.21109999999999998,370,10570,10566],[764,0.1778,370,10570,10566],[3155,0.1222,370,10570,10566],[3153,0.07780000000000001,370,10570,10566],[4977,0.4889,370,10570,10566]]},{"id":"rmmbench-navigation-egocentric-v2-2026-paper","name":"RMMBench · navigation-egocentric (2026 paper)","group":"Agent 与工具","metric":"Task success rate","unit":"%","public_history":true,"protocol":{"en":"Eight VLM backends load the same fixed JSON task configurations in MuJoCo. 30 composite navigation tasks; three episodes each; no BEV mapping. Binary SR is kept separate from atomic completion, execution efficiency and grounding metrics. This panel uses the explicitly stated 90-scenario protocol in the 23 October v2 PDF; the first release was 4 October.","zh":"八个VLM端点在MuJoCo加载相同固定JSON任务配置。30 composite navigation tasks; three episodes each; no BEV mapping。二元任务成功率与原子完成度、执行效率及定位指标分别处理。本面板采用10月23日v2 PDF明确规定的90场景协议，首次发布为10月4日。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.05414v2","published_at":"2026-10-04","status":"paper_baselines"},"model_count":8,"result_count":8,"results":[[6130,0.1778,370,10571,10566],[6133,0.21109999999999998,370,10571,10566],[10567,0.0556,370,10571,10566],[10568,0.11109999999999999,370,10571,10566],[764,0.1,370,10571,10566],[3155,0.0667,370,10571,10566],[3153,0.0333,370,10571,10566],[4977,0.4,370,10571,10566]]},{"id":"robovalue-zero-shot-sa-2026-paper","name":"RoboValue · zero-shot outcome ordering (2026 paper)","group":"视觉与视频","metric":"Pairwise outcome-discrimination SA","unit":"score","score_max":100,"score_decimals":2,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"public_history":true,"protocol":{"en":"Twelve value-model checkpoints use the same in-domain simulated and real robot trajectories, covering fifteen simulation tasks and twenty real tasks. SA compares every successful–unsuccessful episode pair within the same task and setting using terminal values averaged over the final 2% of each episode. A strictly higher value for the successful episode receives credit; no shared absolute threshold is assumed. This ranking-discrimination score stays native on the reported 0–100 scale. Unsupported VS/CSVC entries and composite leaderboard zeros are not imported.","zh":"十二个价值模型检查点使用同一域内模拟和真实机器人轨迹，覆盖十五个模拟任务与二十个真实任务。SA在相同任务及设置内比较全部成功/失败轨迹对，终值取每回合末尾2%区间的均值。成功轨迹终值严格更高才计分，无需跨模型共享绝对阈值。保留0–100原生排序区分度，未引入不支持VS/CSVC的单元格及综合榜零值。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.26663v1","published_at":"2026-10-19","status":"paper_baselines"},"model_count":12,"result_count":12,"results":[[10572,54.44,370,10573,10574],[10575,62.2,370,10573,10574],[9101,72.63,370,10573,10574],[10576,81.29,370,10573,10574],[10577,28.1,370,10573,10574],[9097,31.31,370,10573,10574],[10578,82.12,370,10573,10574],[10579,80.41,370,10573,10574],[10580,58.98,370,10573,10574],[10581,52.99,370,10573,10574],[10582,33.75,370,10573,10574],[10583,41.04,370,10573,10574]]},{"id":"openearth-ovs44-optical-miou-2026-paper","name":"OpenEarth-OVS44 · optical (2026 paper)","group":"视觉与视频","metric":"Overall mIoU","unit":"score","score_max":100,"score_decimals":2,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"public_history":true,"protocol":{"en":"Eight RGB-only configurations from six segmentation architectures use the same 768 test images and 44 categories in the optical track. Overall mIoU is mean class Intersection-over-Union, retained as a native 0–100 metric. Training-free methods and benchmark-trained methods retain their stated protocols; ViT-B/L checkpoints are configurations. RGB+SAR MM-OVSeg is excluded from this RGB-only panel.","zh":"六种分割架构的八个RGB配置使用同一768幅测试图像和44个类别。Overall mIoU按类别计算交并比均值，保留0–100原生指标。免训练及基准训练方法保留各自协议，ViT-B/L检查点视为配置；使用额外SAR的MM-OVSeg另行处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.29642v1","published_at":"2026-10-21","status":"paper_baselines"},"model_count":8,"result_count":8,"results":[[10584,14.4,370,10585,10586],[10587,1.4,370,10585,10586],[10588,29.8,370,10589,10586],[10590,26.9,370,10589,10586],[10591,27.7,370,10589,10586],[10592,29.8,370,10589,10586],[10593,34.4,370,10589,10586],[10594,32.2,370,10589,10586]]},{"id":"openearth-ovs44-cloudy-miou-2026-paper","name":"OpenEarth-OVS44 · cloudy (2026 paper)","group":"视觉与视频","metric":"Overall mIoU","unit":"score","score_max":100,"score_decimals":2,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"public_history":true,"protocol":{"en":"Eight RGB-only configurations from six segmentation architectures use the same 577 test images and 43 categories in the cloudy track. Overall mIoU is mean class Intersection-over-Union, retained as a native 0–100 metric. Training-free methods and benchmark-trained methods retain their stated protocols; ViT-B/L checkpoints are configurations. RGB+SAR MM-OVSeg is excluded from this RGB-only panel.","zh":"六种分割架构的八个RGB配置使用同一577幅测试图像和43个类别。Overall mIoU按类别计算交并比均值，保留0–100原生指标。免训练及基准训练方法保留各自协议，ViT-B/L检查点视为配置；使用额外SAR的MM-OVSeg另行处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.29642v1","published_at":"2026-10-21","status":"paper_baselines"},"model_count":8,"result_count":8,"results":[[10584,6.1,370,10595,10586],[10587,1.0,370,10595,10586],[10588,23.3,370,10596,10586],[10590,19.5,370,10596,10586],[10591,19.8,370,10596,10586],[10592,22.5,370,10596,10586],[10593,26.1,370,10596,10586],[10594,26.2,370,10596,10586]]},{"id":"openearth-ovs44-cross-miou-2026-paper","name":"OpenEarth-OVS44 · cross (2026 paper)","group":"视觉与视频","metric":"Overall mIoU","unit":"score","score_max":100,"score_decimals":2,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"public_history":true,"protocol":{"en":"Eight RGB-only configurations from six segmentation architectures use the same 896 test images and 40 categories in the cross track. Overall mIoU is mean class Intersection-over-Union, retained as a native 0–100 metric. Training-free methods and benchmark-trained methods retain their stated protocols; ViT-B/L checkpoints are configurations. RGB+SAR MM-OVSeg is excluded from this RGB-only panel.","zh":"六种分割架构的八个RGB配置使用同一896幅测试图像和40个类别。Overall mIoU按类别计算交并比均值，保留0–100原生指标。免训练及基准训练方法保留各自协议，ViT-B/L检查点视为配置；使用额外SAR的MM-OVSeg另行处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.29642v1","published_at":"2026-10-21","status":"paper_baselines"},"model_count":8,"result_count":8,"results":[[10584,16.3,370,10597,10586],[10587,1.8,370,10597,10586],[10588,19.6,370,10598,10586],[10590,13.6,370,10598,10586],[10591,16.1,370,10598,10586],[10592,17.9,370,10598,10586],[10593,26.8,370,10598,10586],[10594,23.7,370,10598,10586]]},{"id":"libero-vpro-temporal-corruption-2026-paper","name":"LIBERO-VPro · temporal-corruption (2026 paper)","group":"Agent 与工具","metric":"Perturbed task success","unit":"%","public_history":true,"protocol":{"en":"Six robotic policy architectures are trained on standard LIBERO demonstrations and tested over 40 tasks from four suites, with ten trials per task-condition. This is the temporal-corruption robustness category, explicitly marked higher-is-better by the source. Diagnostic masking/contradiction categories and the mixed Overall average are excluded because higher success there need not mean better visual grounding. LingBot-VA’s extra-suite training follows its official recipe.","zh":"六种机器人策略采用标准LIBERO示范训练，在四个套件的40题上评测，每个任务条件运行十次。本组为temporal-corruption鲁棒性条件，原文明示成功率越高越好。不采用遮挡、矛盾等诊断条件及混合Overall均值，因为其中高成功率未必表示更好的视觉依据。LingBot-VA额外套件训练沿用官方方案。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.24350v1","published_at":"2026-09-21","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[7594,0.7639,370,10599,10600],[10179,0.7094,370,10599,10600],[8323,0.828,370,10599,10600],[7603,0.8134,370,10599,10600],[7404,0.6789000000000001,370,10599,10600],[10601,0.8597,370,10599,10600]]},{"id":"libero-vpro-view-unavailability-2026-paper","name":"LIBERO-VPro · view-unavailability (2026 paper)","group":"Agent 与工具","metric":"Perturbed task success","unit":"%","public_history":true,"protocol":{"en":"Six robotic policy architectures are trained on standard LIBERO demonstrations and tested over 40 tasks from four suites, with ten trials per task-condition. This is the view-unavailability robustness category, explicitly marked higher-is-better by the source. Diagnostic masking/contradiction categories and the mixed Overall average are excluded because higher success there need not mean better visual grounding. LingBot-VA’s extra-suite training follows its official recipe.","zh":"六种机器人策略采用标准LIBERO示范训练，在四个套件的40题上评测，每个任务条件运行十次。本组为view-unavailability鲁棒性条件，原文明示成功率越高越好。不采用遮挡、矛盾等诊断条件及混合Overall均值，因为其中高成功率未必表示更好的视觉依据。LingBot-VA额外套件训练沿用官方方案。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.24350v1","published_at":"2026-09-21","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[7594,0.2925,370,10602,10600],[10179,0.28190000000000004,370,10602,10600],[8323,0.22719999999999999,370,10602,10600],[7603,0.1012,370,10602,10600],[7404,0.21,370,10602,10600],[10601,0.1797,370,10602,10600]]},{"id":"libero-vpro-task-distractors-2026-paper","name":"LIBERO-VPro · task-distractors (2026 paper)","group":"Agent 与工具","metric":"Perturbed task success","unit":"%","public_history":true,"protocol":{"en":"Six robotic policy architectures are trained on standard LIBERO demonstrations and tested over 40 tasks from four suites, with ten trials per task-condition. This is the task-distractors robustness category, explicitly marked higher-is-better by the source. Diagnostic masking/contradiction categories and the mixed Overall average are excluded because higher success there need not mean better visual grounding. LingBot-VA’s extra-suite training follows its official recipe.","zh":"六种机器人策略采用标准LIBERO示范训练，在四个套件的40题上评测，每个任务条件运行十次。本组为task-distractors鲁棒性条件，原文明示成功率越高越好。不采用遮挡、矛盾等诊断条件及混合Overall均值，因为其中高成功率未必表示更好的视觉依据。LingBot-VA额外套件训练沿用官方方案。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.24350v1","published_at":"2026-09-21","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[7594,0.6507999999999999,370,10603,10600],[10179,0.6125,370,10603,10600],[8323,0.7092,370,10603,10600],[7603,0.695,370,10603,10600],[7404,0.6975,370,10603,10600],[10601,0.695,370,10603,10600]]},{"id":"libero-vpro-task-preconditions-2026-paper","name":"LIBERO-VPro · task-preconditions (2026 paper)","group":"Agent 与工具","metric":"Perturbed task success","unit":"%","public_history":true,"protocol":{"en":"Six robotic policy architectures are trained on standard LIBERO demonstrations and tested over 40 tasks from four suites, with ten trials per task-condition. This is the task-preconditions robustness category, explicitly marked higher-is-better by the source. Diagnostic masking/contradiction categories and the mixed Overall average are excluded because higher success there need not mean better visual grounding. LingBot-VA’s extra-suite training follows its official recipe.","zh":"六种机器人策略采用标准LIBERO示范训练，在四个套件的40题上评测，每个任务条件运行十次。本组为task-preconditions鲁棒性条件，原文明示成功率越高越好。不采用遮挡、矛盾等诊断条件及混合Overall均值，因为其中高成功率未必表示更好的视觉依据。LingBot-VA额外套件训练沿用官方方案。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.24350v1","published_at":"2026-09-21","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[7594,0.39380000000000004,370,10604,10600],[10179,0.3713,370,10604,10600],[8323,0.455,370,10604,10600],[7603,0.4363,370,10604,10600],[7404,0.455,370,10604,10600],[10601,0.4138,370,10604,10600]]},{"id":"libero-vpro-target-geometry-2026-paper","name":"LIBERO-VPro · target-geometry (2026 paper)","group":"Agent 与工具","metric":"Perturbed task success","unit":"%","public_history":true,"protocol":{"en":"Six robotic policy architectures are trained on standard LIBERO demonstrations and tested over 40 tasks from four suites, with ten trials per task-condition. This is the target-geometry robustness category, explicitly marked higher-is-better by the source. Diagnostic masking/contradiction categories and the mixed Overall average are excluded because higher success there need not mean better visual grounding. LingBot-VA’s extra-suite training follows its official recipe.","zh":"六种机器人策略采用标准LIBERO示范训练，在四个套件的40题上评测，每个任务条件运行十次。本组为target-geometry鲁棒性条件，原文明示成功率越高越好。不采用遮挡、矛盾等诊断条件及混合Overall均值，因为其中高成功率未必表示更好的视觉依据。LingBot-VA额外套件训练沿用官方方案。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.24350v1","published_at":"2026-09-21","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[7594,0.8173999999999999,370,10605,10600],[10179,0.7931,370,10605,10600],[8323,0.8534999999999999,370,10605,10600],[7603,0.8824,370,10605,10600],[7404,0.8639,370,10605,10600],[10601,0.8775,370,10605,10600]]},{"id":"libero-vpro-camera-staleness-2026-paper","name":"LIBERO-VPro · camera-staleness (2026 paper)","group":"Agent 与工具","metric":"Perturbed task success","unit":"%","public_history":true,"protocol":{"en":"Six robotic policy architectures are trained on standard LIBERO demonstrations and tested over 40 tasks from four suites, with ten trials per task-condition. This is the camera-staleness robustness category, explicitly marked higher-is-better by the source. Diagnostic masking/contradiction categories and the mixed Overall average are excluded because higher success there need not mean better visual grounding. LingBot-VA’s extra-suite training follows its official recipe.","zh":"六种机器人策略采用标准LIBERO示范训练，在四个套件的40题上评测，每个任务条件运行十次。本组为camera-staleness鲁棒性条件，原文明示成功率越高越好。不采用遮挡、矛盾等诊断条件及混合Overall均值，因为其中高成功率未必表示更好的视觉依据。LingBot-VA额外套件训练沿用官方方案。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.24350v1","published_at":"2026-09-21","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[7594,0.2402,370,10606,10600],[10179,0.27,370,10606,10600],[8323,0.2052,370,10606,10600],[7603,0.17,370,10606,10600],[7404,0.345,370,10606,10600],[10601,0.166,370,10606,10600]]},{"id":"cosec-privacy-violation-2026-paper","name":"CoSec · community privacy (2026 paper)","group":"安全与可靠性","metric":"Overall privacy violation rate","unit":"score","public_history":true,"protocol":{"en":"Seven base models appear in thirteen native agent-harness configurations. Every configuration runs the same 208 frozen scenarios (Static112 and Dyn96) with fixed manifests, requester scripts and isolation. The score is the overall privacy violation rate; lower is better. A separate 192-trial human audit validates the combined deterministic/LLM verifier, while benign completion remains a separate measure.","zh":"七个基础模型以十三种原生Agent框架配置运行相同208个冻结情境，包括Static112及Dyn96；清单、请求脚本和隔离方式固定。采用总体隐私违规率，数值越低越好。另有192次人工审查验证确定性检查与LLM裁判的联合判定，正常任务完成率另列。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34790v1","published_at":"2026-09-28","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"higher_is_better":false,"model_count":7,"result_count":13,"results":[[3412,29.33,370,10607,10608],[3157,47.12,370,10607,10608],[4424,51.44,370,10607,10608],[3415,52.88,370,10607,10608],[227,58.65,370,10607,10608],[3153,82.69,370,10607,10608],[4979,55.29,370,10609,10608],[3412,68.27,370,10609,10608],[3157,69.71,370,10609,10608],[3415,76.92,370,10609,10608],[227,82.21,370,10609,10608],[3153,96.15,370,10609,10608],[4979,41.35,370,10610,10608]]},{"id":"optiarena-wso-breakthrough-2026-paper","name":"OptiArena · WSO breakthrough (2026 paper)","group":"编程","metric":"Game capability ratio","unit":"raw","public_history":true,"protocol":{"en":"Twelve models edit weak starter game programs under fixed edit-call and evaluator budgets, then evaluate the selected candidate on 50 held-out trials. This is breakthrough only; development feedback selects candidates before held-out evaluation. Invalid programs, illegal moves and timeouts remain benchmark outcomes. The source capability estimator is an unbounded native ratio/delta, not a percentage.","zh":"十二个模型在固定修改调用及评测预算内改进弱游戏程序，选定程序后运行50次留出测试。本组单列breakthrough，程序在查看留出结果前根据开发集反馈选择。无效程序、非法动作和超时保留为基准结果。原文能力估计是可超出0–1的原生比率或差值，不当作百分比。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.32227v1","published_at":"2026-09-26","status":"paper_baselines"},"model_count":12,"result_count":12,"results":[[3572,0.0,370,10611,10612],[3184,0.0,370,10611,10612],[10613,0.33,370,10611,10612],[3415,1.33,370,10611,10612],[3185,0.0,370,10611,10612],[7694,0.0,370,10611,10612],[6301,0.67,370,10611,10612],[8655,1.33,370,10611,10612],[3190,0.0,370,10611,10612],[3187,0.0,370,10611,10612],[7586,0.0,370,10611,10612],[2278,0.0,370,10611,10612]]},{"id":"optiarena-wso-connect4-2026-paper","name":"OptiArena · WSO connect4 (2026 paper)","group":"编程","metric":"Oracle move-optimal rate","unit":"%","public_history":true,"protocol":{"en":"Twelve models edit weak starter game programs under fixed edit-call and evaluator budgets, then evaluate the selected candidate on 50 held-out trials. This is connect4 only; development feedback selects candidates before held-out evaluation. Invalid programs, illegal moves and timeouts remain benchmark outcomes. The selected metric is the published fraction of oracle-optimal or reference-agreeing moves, not the game win rate.","zh":"十二个模型在固定修改调用及评测预算内改进弱游戏程序，选定程序后运行50次留出测试。本组单列connect4，程序在查看留出结果前根据开发集反馈选择。无效程序、非法动作和超时保留为基准结果。采用原表与最优动作或参考动作一致的比例，不等于游戏胜率。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.32227v1","published_at":"2026-09-26","status":"paper_baselines"},"model_count":12,"result_count":12,"results":[[3572,0.81,370,10611,10612],[3184,0.92,370,10611,10612],[10613,0.81,370,10611,10612],[3415,0.91,370,10611,10612],[3185,0.81,370,10611,10612],[7694,0.81,370,10611,10612],[6301,0.81,370,10611,10612],[8655,0.81,370,10611,10612],[3190,0.81,370,10611,10612],[3187,0.81,370,10611,10612],[7586,0.81,370,10611,10612],[2278,0.81,370,10611,10612]]},{"id":"optiarena-wso-othello-2026-paper","name":"OptiArena · WSO othello (2026 paper)","group":"编程","metric":"Reference move agreement","unit":"%","public_history":true,"protocol":{"en":"Twelve models edit weak starter game programs under fixed edit-call and evaluator budgets, then evaluate the selected candidate on 50 held-out trials. This is othello only; development feedback selects candidates before held-out evaluation. Invalid programs, illegal moves and timeouts remain benchmark outcomes. The selected metric is the published fraction of oracle-optimal or reference-agreeing moves, not the game win rate.","zh":"十二个模型在固定修改调用及评测预算内改进弱游戏程序，选定程序后运行50次留出测试。本组单列othello，程序在查看留出结果前根据开发集反馈选择。无效程序、非法动作和超时保留为基准结果。采用原表与最优动作或参考动作一致的比例，不等于游戏胜率。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.32227v1","published_at":"2026-09-26","status":"paper_baselines"},"model_count":12,"result_count":12,"results":[[3572,0.44,370,10611,10612],[3184,0.4,370,10611,10612],[10613,0.4,370,10611,10612],[3415,0.4,370,10611,10612],[3185,0.4,370,10611,10612],[7694,0.33,370,10611,10612],[6301,0.4,370,10611,10612],[8655,0.35,370,10611,10612],[3190,0.41,370,10611,10612],[3187,0.4,370,10611,10612],[7586,0.35,370,10611,10612],[2278,0.4,370,10611,10612]]},{"id":"optiarena-wso-hex-2026-paper","name":"OptiArena · WSO hex (2026 paper)","group":"编程","metric":"Reference move agreement","unit":"%","public_history":true,"protocol":{"en":"Twelve models edit weak starter game programs under fixed edit-call and evaluator budgets, then evaluate the selected candidate on 50 held-out trials. This is hex only; development feedback selects candidates before held-out evaluation. Invalid programs, illegal moves and timeouts remain benchmark outcomes. The selected metric is the published fraction of oracle-optimal or reference-agreeing moves, not the game win rate.","zh":"十二个模型在固定修改调用及评测预算内改进弱游戏程序，选定程序后运行50次留出测试。本组单列hex，程序在查看留出结果前根据开发集反馈选择。无效程序、非法动作和超时保留为基准结果。采用原表与最优动作或参考动作一致的比例，不等于游戏胜率。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.32227v1","published_at":"2026-09-26","status":"paper_baselines"},"model_count":12,"result_count":12,"results":[[3572,0.07,370,10611,10612],[3184,0.03,370,10611,10612],[10613,0.03,370,10611,10612],[3415,0.03,370,10611,10612],[3185,0.09,370,10611,10612],[7694,0.0,370,10611,10612],[6301,0.03,370,10611,10612],[8655,0.03,370,10611,10612],[3190,0.02,370,10611,10612],[3187,0.03,370,10611,10612],[7586,0.03,370,10611,10612],[2278,0.03,370,10611,10612]]},{"id":"optiarena-wso-2048-2026-paper","name":"OptiArena · WSO 2048 (2026 paper)","group":"编程","metric":"Reference-relative robust-score delta","unit":"raw","public_history":true,"protocol":{"en":"Twelve models edit weak starter game programs under fixed edit-call and evaluator budgets, then evaluate the selected candidate on 50 held-out trials. This is 2048 only; development feedback selects candidates before held-out evaluation. Invalid programs, illegal moves and timeouts remain benchmark outcomes. The source capability estimator is an unbounded native ratio/delta, not a percentage.","zh":"十二个模型在固定修改调用及评测预算内改进弱游戏程序，选定程序后运行50次留出测试。本组单列2048，程序在查看留出结果前根据开发集反馈选择。无效程序、非法动作和超时保留为基准结果。原文能力估计是可超出0–1的原生比率或差值，不当作百分比。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.32227v1","published_at":"2026-09-26","status":"paper_baselines"},"model_count":12,"result_count":12,"results":[[3572,0.52,370,10611,10612],[3184,1.14,370,10611,10612],[10613,1.22,370,10611,10612],[3415,0.68,370,10611,10612],[3185,1.21,370,10611,10612],[7694,0.51,370,10611,10612],[6301,-0.15,370,10611,10612],[8655,0.52,370,10611,10612],[3190,1.35,370,10611,10612],[3187,0.82,370,10611,10612],[7586,0.36,370,10611,10612],[2278,1.39,370,10611,10612]]},{"id":"peony-poetic-logic-2026-paper","name":"Peony · poetic logic (2026 paper)","group":"知识与推理","metric":"Mean task accuracy","unit":"%","public_history":true,"protocol":{"en":"Six distinct LLMs in ten thinking/non-thinking configurations use the same 800 modern Chinese poems and four poetic-logic selection tasks. The complete main panel and published average accuracy are retained; thinking variants remain configurations of the same model and do not inflate the model count.","zh":"六个不同LLM、十种思考或非思考配置使用同一800首现代中文诗与四类诗歌逻辑选择任务。保留完整主表与论文平均准确率；思考变体是同一模型的配置，不增加模型数。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.21827v1","status":"paper_baselines","published_at":"2026-08-22"},"model_count":6,"result_count":10,"results":[[3201,0.4991,370,8105,10614],[3185,0.5301,370,8105,10614],[3748,0.4698,370,8105,10614],[2998,0.5989,370,8105,10614],[3186,0.5421,370,8105,10614],[3201,0.5837,370,3143,10614],[3185,0.6425,370,3143,10614],[26,0.7401000000000001,370,3143,10614],[2998,0.6426999999999999,370,3143,10614],[3186,0.6419,370,3143,10614]]},{"id":"readi-english-2026-paper","name":"READI · english pragmatics (2026 paper)","group":"知识与推理","metric":"Indirect-speech-act accuracy","unit":"%","public_history":true,"protocol":{"en":"All eight model families use the same visual-pragmatic question-answering items in the selected language. Each input combines visual context and dialogue, and the task tests directive-intent recovery across graded indirectness. English and Korean results are separate. Published fractional accuracy remains on the source’s 0–1 scale.","zh":"完整八个模型族使用所选语言的同一视觉语用问答。输入结合视觉场景与对话，评测不同间接程度下的指令意图恢复。英语和韩语分别展示。论文小数准确率保留0–1原始量纲。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.30270v1","status":"paper_baselines","published_at":"2026-08-31"},"model_count":8,"result_count":8,"results":[[340,0.82,370,10615,10616],[5809,0.93,370,10615,10616],[10617,0.78,370,10615,10616],[10618,0.51,370,10615,10616],[10619,0.29,370,10615,10616],[10620,0.76,370,10615,10616],[10621,0.76,370,10615,10616],[9410,0.76,370,10615,10616]]},{"id":"readi-korean-2026-paper","name":"READI · korean pragmatics (2026 paper)","group":"知识与推理","metric":"Indirect-speech-act accuracy","unit":"%","public_history":true,"protocol":{"en":"All eight model families use the same visual-pragmatic question-answering items in the selected language. Each input combines visual context and dialogue, and the task tests directive-intent recovery across graded indirectness. English and Korean results are separate. Published fractional accuracy remains on the source’s 0–1 scale.","zh":"完整八个模型族使用所选语言的同一视觉语用问答。输入结合视觉场景与对话，评测不同间接程度下的指令意图恢复。英语和韩语分别展示。论文小数准确率保留0–1原始量纲。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.30270v1","status":"paper_baselines","published_at":"2026-08-31"},"model_count":8,"result_count":8,"results":[[340,0.78,370,10622,10616],[5809,0.88,370,10622,10616],[10617,0.79,370,10622,10616],[10618,0.25,370,10622,10616],[10619,0.33,370,10622,10616],[10620,0.52,370,10622,10616],[10621,0.49,370,10622,10616],[9410,0.52,370,10622,10616]]},{"id":"s2r-real-removal-2026-paper","name":"S2R-Bench · real video reflection removal (2026 paper)","group":"视觉与视频","metric":"Reflection removal score","unit":"score","public_history":true,"protocol":{"en":"All six dereflection systems use the same 50 real reflection videos. The no-reference removal metric remains a native 0–1 score; preservation and temporal consistency are separate measures. The paired-reference video track has only five scored systems and is not pooled into this cohort.","zh":"完整六个去反射系统使用同一50段真实反射视频。无参考移除指标保留0–1原生分数，与保留度和时间一致性分开。配对参考视频轨道只有五个系统有分数，不混入本面板。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.11562v1","status":"paper_baselines","published_at":"2026-08-12"},"score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":6,"result_count":6,"results":[[10623,0.163,370,10624,10625],[10626,0.297,370,10624,10625],[10627,0.175,370,10624,10625],[10628,0.332,370,10624,10625],[10629,0.654,370,10624,10625],[10630,0.787,370,10624,10625]]},{"id":"scopebench-sampled-depth-2026-paper","name":"SCOPE-Bench · sampled recommendation (2026 paper)","group":"预测与决策","metric":"Content-depth awareness A@20","unit":"score","public_history":true,"protocol":{"en":"The complete thirteen-architecture recommender panel uses the same SCOPE-Bench split and top-20 recommendation protocol. The source’s A@20 content-depth-awareness score is retained on its ×100 display scale. Sampled and full candidate settings are separate; engagement recall and depth-exposure E@20 are not mixed into this metric.","zh":"完整十三种推荐架构使用同一SCOPE-Bench划分和前20推荐协议。保留论文×100显示量纲的A@20内容深度感知分数。抽样与完整候选设置分开，不混入互动召回和深度曝光E@20。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.13990v1","status":"paper_baselines","published_at":"2026-08-14"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":13,"result_count":13,"results":[[10631,6.9,370,10632,10633],[10634,6.49,370,10632,10633],[10635,7.36,370,10632,10633],[10636,6.22,370,10632,10633],[10637,7.92,370,10632,10633],[10638,7.13,370,10632,10633],[10639,6.72,370,10632,10633],[10640,7.13,370,10632,10633],[10641,7.38,370,10632,10633],[10642,6.71,370,10632,10633],[10643,7.24,370,10632,10633],[10644,7.71,370,10632,10633],[10645,8.12,370,10632,10633]]},{"id":"scopebench-full-depth-2026-paper","name":"SCOPE-Bench · full recommendation (2026 paper)","group":"预测与决策","metric":"Content-depth awareness A@20","unit":"score","public_history":true,"protocol":{"en":"The complete thirteen-architecture recommender panel uses the same SCOPE-Bench split and top-20 recommendation protocol. The source’s A@20 content-depth-awareness score is retained on its ×100 display scale. Sampled and full candidate settings are separate; engagement recall and depth-exposure E@20 are not mixed into this metric.","zh":"完整十三种推荐架构使用同一SCOPE-Bench划分和前20推荐协议。保留论文×100显示量纲的A@20内容深度感知分数。抽样与完整候选设置分开，不混入互动召回和深度曝光E@20。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.13990v1","status":"paper_baselines","published_at":"2026-08-14"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":13,"result_count":13,"results":[[10631,7.06,370,10646,10633],[10634,7.46,370,10646,10633],[10635,6.89,370,10646,10633],[10636,7.44,370,10646,10633],[10637,6.72,370,10646,10633],[10638,6.97,370,10646,10633],[10639,6.7,370,10646,10633],[10640,6.81,370,10646,10633],[10641,6.89,370,10646,10633],[10642,7.24,370,10646,10633],[10643,7.1,370,10646,10633],[10644,6.81,370,10646,10633],[10645,7.59,370,10646,10633]]},{"id":"ai4ai-training-algorithms-2026-paper","name":"AI4AI-Bench · training algorithm design (2026 paper)","group":"数学与科学","metric":"Mean ladder score","unit":"score","public_history":true,"protocol":{"en":"The full 29-row panel covers six distinct systems and all ten frozen training repositories. Every task gives four hours on one B300 GPU for code changes, then an isolated twelve-hour fixed-budget rerun and predetermined evaluator. The source’s ladder score maps the unchanged repository to 0.1 and the task optimum to 1.0. Published zeros and effort/harness settings are preserved; no absent runs are imputed.","zh":"完整29行面板覆盖六个不同系统与十个固定训练仓库。每任务允许在一块B300上改代码四小时，随后隔离地按固定预算重跑十二小时并由预设验证器评分。原生阶梯分数将未修改仓库设为0.1、任务最优设为1.0。保留论文零分、推理强度与框架设置，不推算缺失运行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.20318v1","status":"paper_baselines","published_at":"2026-08-20"},"score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":6,"result_count":29,"results":[[755,0.272,370,10647,10648],[755,0.288,370,10649,10648],[755,0.278,370,10650,10648],[755,0.19,370,10651,10648],[755,0.225,370,10652,10648],[3568,0.099,370,10647,10648],[3568,0.109,370,10649,10648],[3568,0.216,370,10650,10648],[3568,0.137,370,10651,10648],[3568,0.163,370,10652,10648],[2977,0.174,370,10652,10648],[762,0.126,370,10653,10648],[762,0.198,370,10654,10648],[762,0.144,370,10655,10648],[762,0.235,370,10656,10648],[762,0.195,370,10657,10648],[762,0.245,370,10658,10648],[2973,0.065,370,10653,10648],[2973,0.109,370,10654,10648],[2973,0.135,370,10655,10648],[2973,0.129,370,10656,10648],[2973,0.144,370,10657,10648],[2973,0.228,370,10658,10648],[3408,0.091,370,10653,10648],[3408,0.079,370,10654,10648],[3408,0.124,370,10655,10648],[3408,0.119,370,10656,10648],[3408,0.147,370,10657,10648],[3408,0.139,370,10658,10648]]},{"id":"asi-bench-project-research-2026-paper","name":"ASI-Bench · scientific research projects (2026 paper)","group":"数学与科学","metric":"Overall scientific score","unit":"score","public_history":true,"protocol":{"en":"The complete eighteen-configuration panel uses the same 60 research projects under four guidance levels B1–B4, with fixed research objectives, data and scoring criteria. The source’s overall scientific score is a native 0–100 aggregate. Agent harness, reasoning effort and the single-run Opus 5 condition are explicit; other source standard deviations remain in the fixture.","zh":"完整十八配置面板在四级指导B1–B4下使用相同60个研究项目，研究目标、数据和评分规则固定。总体科学分数保留0–100原生汇总量纲。框架、推理强度及Opus 5仅一次运行的条件均明确，其他标准差保存在原表。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.17271v1","status":"paper_baselines","published_at":"2026-08-18"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":12,"result_count":18,"results":[[3196,38.15,370,10659,10660],[762,46.83,370,10661,10660],[762,55.84,370,10662,10660],[755,50.29,370,10663,10660],[2977,37.09,370,10664,10660],[3479,37.51,370,10665,10660],[3154,41.65,370,10666,10660],[227,35.59,370,10667,10660],[3261,31.9,370,10668,10660],[3440,27.34,370,10669,10660],[3491,26.55,370,10670,10660],[3492,24.98,370,10671,10660],[3593,23.25,370,10672,10660],[2977,36.22,370,10673,10660],[3440,19.72,370,10674,10660],[3593,16.17,370,10675,10660],[3261,30.6,370,10676,10660],[3492,21.63,370,10677,10660]]},{"id":"flatlab-unseen-objects-2026-paper","name":"FlatLab · unseen objects (2026 paper)","group":"Agent 与工具","metric":"Grasping success rate","unit":"%","public_history":true,"protocol":{"en":"Six learned policy architectures use the same FlatLab flat-object partitions. The source divides 104 objects from 21 categories into training, unseen objects from known categories (Test α), and entirely unseen categories (Test β). Each object receives five grasping trials with different orientations; the two test sets stay separate.","zh":"六种学习策略架构使用相同FlatLab扁平物体划分。104个物体、21类划为训练、已知类别的新物体Test α和全新类别Test β。每物体以不同朝向抓取五次，两类测试集分别展示。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.14049v1","status":"paper_baselines","published_at":"2026-08-14"},"model_count":6,"result_count":6,"results":[[10173,0.368,370,10678,10679],[10680,0.542,370,10678,10679],[7592,0.226,370,10678,10679],[10681,0.23600000000000002,370,10678,10679],[9732,0.28600000000000003,370,10678,10679],[10682,0.742,370,10678,10679]]},{"id":"flatlab-unseen-categories-2026-paper","name":"FlatLab · unseen categories (2026 paper)","group":"Agent 与工具","metric":"Grasping success rate","unit":"%","public_history":true,"protocol":{"en":"Six learned policy architectures use the same FlatLab flat-object partitions. The source divides 104 objects from 21 categories into training, unseen objects from known categories (Test α), and entirely unseen categories (Test β). Each object receives five grasping trials with different orientations; the two test sets stay separate.","zh":"六种学习策略架构使用相同FlatLab扁平物体划分。104个物体、21类划为训练、已知类别的新物体Test α和全新类别Test β。每物体以不同朝向抓取五次，两类测试集分别展示。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.14049v1","status":"paper_baselines","published_at":"2026-08-14"},"model_count":6,"result_count":6,"results":[[10173,0.38799999999999996,370,10683,10679],[10680,0.512,370,10683,10679],[7592,0.17800000000000002,370,10683,10679],[10681,0.17300000000000001,370,10683,10679],[9732,0.25,370,10683,10679],[10682,0.69,370,10683,10679]]},{"id":"egomonth-overall-2026-paper","name":"EgoMonth · month-scale egocentric reasoning (2026 paper)","group":"视觉与视频","metric":"Overall question accuracy","unit":"%","public_history":true,"protocol":{"en":"The complete twelve-model panel answers the same 1,443 egocentric questions across fourteen tasks. The paper’s Acc row is overall question accuracy, distinct from the unweighted task Avg. row. Each model’s frame budget and parameter size are retained explicitly, and the human reference is excluded from the model cohort. Exact transposed row/column coordinates preserve source provenance.","zh":"完整十二模型面板回答同一1,443道自我视角问题，涵盖十四项任务。采用Acc行的总体题目准确率，与按任务均匀平均的Avg.行分开。各模型帧预算和规模明确保留，人类参考不计入模型面板。转置抽取的精确行列坐标保留溯源。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.13113v1","status":"paper_baselines","published_at":"2026-08-13"},"model_count":12,"result_count":12,"results":[[10684,0.39899999999999997,370,10685,10686],[10687,0.41700000000000004,370,10688,10686],[10689,0.606,370,10690,10686],[10691,0.57,370,10685,10686],[10692,0.608,370,10693,10686],[10694,0.537,370,10690,10686],[10695,0.5670000000000001,370,10696,10686],[10697,0.369,370,10698,10686],[10699,0.408,370,10700,10686],[10701,0.531,370,10702,10686],[877,0.536,370,10703,10686],[2602,0.726,370,10704,10686]]},{"id":"banglasafe-loose-2026-paper","name":"BanglaSafe · partial or harmful compliance (2026 paper)","group":"安全与可靠性","metric":"ASR loose (lower is better)","unit":"score","public_history":true,"protocol":{"en":"The full eighteen-model cohort receives the same 879 Bengali/culturally grounded prompts across five fixed registers. Loose ASR counts PARTIAL and HARMFUL; strict ASR counts HARMFUL only. The calibrated Claude Opus 4.7 rubric and n=879 per model are fixed. Exact public reference-cohort JSON values are preserved on their native 0–1 scale; lower is safer.","zh":"完整十八模型面板使用五种固定语体的同一879个孟加拉语或文化相关提示。loose ASR合计部分和完全有害配合，strict ASR仅计完全有害配合。固定Claude Opus 4.7校准量表，每模型均879次响应。保留公开参考JSON的0–1原生分数，越低越安全。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.22335v1","status":"paper_baselines","published_at":"2026-08-23"},"score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"higher_is_better":false,"model_count":18,"result_count":18,"results":[[5111,0.1706,370,10705,10706],[10707,0.884,370,10705,10706],[3209,0.7463,370,10705,10706],[4164,0.5779,370,10705,10706],[10708,0.686,370,10705,10706],[10709,0.7418,370,10705,10706],[10710,0.6155,370,10705,10706],[1521,0.7929,370,10705,10706],[5159,0.4141,370,10705,10706],[5371,0.2423,370,10705,10706],[10711,0.0842,370,10705,10706],[10712,0.0387,370,10705,10706],[10713,0.4187,370,10705,10706],[10714,0.8953,370,10705,10706],[10715,0.7224,370,10705,10706],[5148,0.785,370,10705,10706],[10716,0.1058,370,10705,10706],[10717,0.7315,370,10705,10706]]},{"id":"banglasafe-strict-2026-paper","name":"BanglaSafe · harmful compliance (2026 paper)","group":"安全与可靠性","metric":"ASR strict (lower is better)","unit":"score","public_history":true,"protocol":{"en":"The full eighteen-model cohort receives the same 879 Bengali/culturally grounded prompts across five fixed registers. Loose ASR counts PARTIAL and HARMFUL; strict ASR counts HARMFUL only. The calibrated Claude Opus 4.7 rubric and n=879 per model are fixed. Exact public reference-cohort JSON values are preserved on their native 0–1 scale; lower is safer.","zh":"完整十八模型面板使用五种固定语体的同一879个孟加拉语或文化相关提示。loose ASR合计部分和完全有害配合，strict ASR仅计完全有害配合。固定Claude Opus 4.7校准量表，每模型均879次响应。保留公开参考JSON的0–1原生分数，越低越安全。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.22335v1","status":"paper_baselines","published_at":"2026-08-23"},"score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"higher_is_better":false,"model_count":18,"result_count":18,"results":[[5111,0.0546,370,10705,10706],[10707,0.3936,370,10705,10706],[3209,0.4573,370,10705,10706],[4164,0.1229,370,10705,10706],[10708,0.1536,370,10705,10706],[10709,0.1752,370,10705,10706],[10710,0.066,370,10705,10706],[1521,0.2048,370,10705,10706],[5159,0.0341,370,10705,10706],[5371,0.0387,370,10705,10706],[10711,0.0034,370,10705,10706],[10712,0.0046,370,10705,10706],[10713,0.0387,370,10705,10706],[10714,0.3834,370,10705,10706],[10715,0.182,370,10705,10706],[5148,0.2162,370,10705,10706],[10716,0.0148,370,10705,10706],[10717,0.1081,370,10705,10706]]},{"id":"elbench-education-literacy-2026-paper","name":"ELBench · educational literacy (2026 paper)","group":"专业工作","metric":"Overall educational literacy score","unit":"score","public_history":true,"protocol":{"en":"The full nine-model panel uses the same ELBench modules: general capability, safety, basic education and high-level cultivation. The overall published score combines rule/reference scoring and open-ended educational judgments, so it remains a native 0–100 aggregate rather than binary accuracy. Bootstrap confidence intervals are preserved in the original fixture.","zh":"完整九模型面板使用同一通用能力、安全、基础教育和高阶素养模块。总体分数结合规则或参考答案与开放教育判断，因此保留0–100原生汇总分数，与二元准确率区分。自助置信区间保存在原表fixture。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.09548v2","status":"paper_baselines","published_at":"2026-08-10"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":9,"result_count":9,"results":[[3201,83.7,370,10718,10719],[2129,83.4,370,10718,10719],[7674,83.2,370,10718,10719],[3194,83.1,370,10718,10719],[3184,83.1,370,10718,10719],[3185,83.1,370,10718,10719],[3186,81.7,370,10718,10719],[10720,77.0,370,10718,10719],[10721,76.4,370,10718,10719]]},{"id":"elephantbench-complete-recall-2026-paper","name":"ElephantBench · complete factual recall (2026 paper)","group":"知识与推理","metric":"Complete-recall rate (C)","unit":"%","public_history":true,"protocol":{"en":"All 32 model configurations answer the same 1,094 closed-book questions about multiple verified accounts of low-exposure facts. C is the fraction recalling all required verified accounts; P and F diagnose partial and failed recall and are not pooled. The judge rubric is validated against human annotation.","zh":"完整32个模型配置回答同一1,094道闭卷问题，覆盖低曝光事实的多种已验证记述。C表示回忆出全部所需记述的比例，与部分回忆P及失败F分别保留。裁判量表经过人工标注验证。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.28478v1","status":"paper_baselines","published_at":"2026-08-28"},"model_count":32,"result_count":32,"results":[[502,0.5036999999999999,370,10722,10723],[3196,0.5018,370,10722,10723],[2129,0.4863,370,10722,10723],[4979,0.4497,370,10722,10723],[3194,0.4415,370,10722,10723],[5789,0.3784,370,10722,10723],[3155,0.5238,370,10722,10723],[5114,0.426,370,10722,10723],[8654,0.3812,370,10722,10723],[227,0.3729,370,10722,10723],[3198,0.36200000000000004,370,10722,10723],[3191,0.32270000000000004,370,10722,10723],[3185,0.31989999999999996,370,10722,10723],[3201,0.2998,370,10722,10723],[5620,0.2824,370,10722,10723],[3419,0.2249,370,10722,10723],[5619,0.2102,370,10722,10723],[2584,0.19469999999999998,370,10722,10723],[6771,0.17,370,10722,10723],[8656,0.1536,370,10722,10723],[10724,0.1499,370,10722,10723],[8655,0.1371,370,10722,10723],[5480,0.1188,370,10722,10723],[3763,0.1015,370,10722,10723],[5343,0.09140000000000001,370,10722,10723],[4968,0.058499999999999996,370,10722,10723],[4070,0.1225,370,10722,10723],[4969,0.0969,370,10722,10723],[10725,0.0521,370,10722,10723],[5650,0.0274,370,10722,10723],[5828,0.0165,370,10722,10723],[7567,0.0137,370,10722,10723]]},{"id":"emosbench-emotional-scoring-2026-paper","name":"EmoSBench · spoken emotional intelligence (2026 paper)","group":"专业工作","metric":"Exact emotional-rating accuracy","unit":"%","public_history":true,"protocol":{"en":"The full eleven-model panel evaluates the same ten spoken-emotional-intelligence subtasks. Accuracy requires the predicted ordinal EI rating to match the reference rating; the paper’s average across subtasks is retained. Human and Random references are excluded from the model panel. The fine-tuned EmoS result keeps its Qwen2.5-Omni backbone explicit.","zh":"完整十一个模型评测同一十项口语情绪智能任务。准确率要求预测的情绪智能等级与参考等级一致，保留论文任务均值。人类与随机参考不计入模型面板，微调EmoS结果明确保留Qwen2.5-Omni底模。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.09189v1","status":"paper_baselines","published_at":"2026-08-10"},"model_count":11,"result_count":11,"results":[[7726,0.27399999999999997,370,10726,10727],[6603,0.261,370,10726,10727],[8817,0.318,370,10726,10727],[10728,0.295,370,10726,10727],[10729,0.348,370,10726,10727],[10730,0.42700000000000005,370,10726,10727],[7381,0.501,370,10726,10727],[7382,0.37200000000000005,370,10726,10727],[10731,0.526,370,10726,10727],[2602,0.54,370,10726,10727],[10732,0.838,370,10726,10727]]},{"id":"firm-video-rating-mae-2026-paper","name":"FIRM-Video-Bench · reward rating (2026 paper)","group":"视觉与视频","metric":"Overall rating MAE","unit":"score","public_history":true,"protocol":{"en":"The complete eleven-system panel rates the same 250 prompt–video pairs against 750 expert annotations for instruction following, world coherence and perceptual quality. Human scores are integers from one to five; overall mean absolute error remains a native error with lower values better. The expensive data pipeline and the two fine-tuned reward models retain their source identities.","zh":"完整十一个系统对同一250对提示和视频评分，并与指令遵循、世界一致性、感知质量三维750个人工标注比对。人工等级为1–5，总体平均绝对误差保留原生误差量纲，越低越好。高成本数据流程和两个微调奖励模型保留原始身份。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.21839v1","status":"paper_baselines","published_at":"2026-08-22"},"score_max":4,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 4","zh":"分数 / 4"},"higher_is_better":false,"model_count":11,"result_count":11,"results":[[2058,1.08,370,10733,10734],[502,1.16,370,10733,10734],[6721,1.05,370,10733,10734],[2728,1.56,370,10733,10734],[2706,1.42,370,10733,10734],[6342,1.33,370,10733,10734],[6238,1.4,370,10733,10734],[5843,1.27,370,10733,10734],[10735,0.73,370,10733,10734],[10736,0.85,370,10733,10734],[10737,0.78,370,10733,10734]]},{"id":"gui-cc-offline-worlds-2026-paper","name":"GUI-CC · offline world-model rollouts (2026 paper)","group":"Agent 与工具","metric":"Offline overall consistency score","unit":"score","public_history":true,"protocol":{"en":"All eighteen published configurations use the same 500 offline reference-action trajectories. Generated states are fed back into later rollout steps. Overall is the paper’s unweighted composite of normalized fidelity, plausibility, cross-step context and task progress metrics, retained on its native 0–100 scale. History conditioning is explicit; code-output and image-output world models keep their source names.","zh":"完整十八个配置使用同一500条离线参考动作轨迹，生成状态反馈为后续步骤输入。总体为论文归一化保真、合理性、跨步上下文和任务进度指标的均值，保留0–100原生分数。明确历史条件，代码输出与图像输出世界模型保留原名。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.00048v1","status":"paper_baselines","published_at":"2026-08-30"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":12,"result_count":18,"results":[[3578,57.6,370,10738,10739],[3196,54.6,370,10738,10739],[3578,52.0,370,10740,10739],[3196,46.5,370,10740,10739],[10741,43.6,370,10738,10739],[10742,42.4,370,10738,10739],[10742,41.1,370,10740,10739],[10741,35.5,370,10740,10739],[10743,33.1,370,10738,10739],[10744,32.2,370,10738,10739],[10743,31.3,370,10740,10739],[10744,29.8,370,10740,10739],[7290,55.0,370,10740,10739],[10745,51.7,370,10740,10739],[10746,25.4,370,10740,10739],[10747,24.3,370,10740,10739],[6361,16.3,370,10740,10739],[10748,10.4,370,10740,10739]]},{"id":"medclaw-neurosurgery-vtr-2026-paper","name":"MedClawBench · neurosurgery temporal reasoning (2026 paper)","group":"医疗","metric":"Information correctness (CI)","unit":"score","public_history":true,"protocol":{"en":"All thirteen model/agent systems answer the same 109 neurosurgery Visual Temporal Reasoning questions. Information correctness is graded on the source’s one-to-five rubric by a GPT-5.5 judge validated against neurosurgeons. General video baselines, surgical models, one-shot frontier models and MedClaw’s flash-backed configuration remain explicit. Perception questions and the SVU split are separate cohorts.","zh":"完整十三个模型或Agent系统回答同一109道神经外科视频时间推理问题。信息正确性CI由GPT-5.5按原生1–5量表评分，并与神经外科医生评判核验。通用视频、外科模型、单样本前沿模型与MedClaw的flash视觉底模配置均明确，感知题与SVU子集不混入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.14015v1","status":"paper_baselines","published_at":"2026-08-14"},"score_max":5,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 5","zh":"分数 / 5"},"model_count":13,"result_count":13,"results":[[2754,1.62,370,10749,10750],[2728,1.35,370,10749,10750],[6349,1.49,370,10749,10750],[7392,1.61,370,10749,10750],[10751,1.75,370,10749,10750],[10752,1.64,370,10749,10750],[10753,1.75,370,10749,10750],[10754,1.46,370,10749,10750],[10755,1.53,370,10749,10750],[7479,2.1,370,10756,10750],[5115,1.96,370,10756,10750],[3479,2.15,370,10756,10750],[10757,2.4,370,10758,10750]]},{"id":"redagentbench-codex-2026-paper","name":"REDAgentBench · codex (2026 paper)","group":"安全与可靠性","metric":"Hybrid-judged attack success rate","unit":"score","public_history":true,"protocol":{"en":"Six distinct model snapshots run the same frozen 1,661 executable attack cases under this fixed harness, clean evaluation cue and trajectory/state hybrid judge. Lower attack success is safer. The paper resolves the abbreviated Qwen-plus row to qwen-plus-2025-12-01; selected replay defenses and state-confirmed diagnostic subsets are separate.","zh":"六个不同模型快照在指定框架下运行同一1,661个冻结可执行攻击案例，采用干净评测提示与轨迹/状态混合裁判。攻击成功率越低越安全。正文将Qwen-plus缩写明确为qwen-plus-2025-12-01；选定重放防御和状态确认诊断子集不混入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.10669v1","status":"paper_baselines","published_at":"2026-08-11"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"higher_is_better":false,"model_count":6,"result_count":6,"results":[[340,62.31,370,10759,10760],[7659,54.38,370,10759,10760],[10761,75.35,370,10759,10760],[10762,71.81,370,10759,10760],[3490,78.51,370,10759,10760],[227,48.12,370,10759,10760]]},{"id":"redagentbench-hermes-2026-paper","name":"REDAgentBench · hermes (2026 paper)","group":"安全与可靠性","metric":"Hybrid-judged attack success rate","unit":"score","public_history":true,"protocol":{"en":"Six distinct model snapshots run the same frozen 1,661 executable attack cases under this fixed harness, clean evaluation cue and trajectory/state hybrid judge. Lower attack success is safer. The paper resolves the abbreviated Qwen-plus row to qwen-plus-2025-12-01; selected replay defenses and state-confirmed diagnostic subsets are separate.","zh":"六个不同模型快照在指定框架下运行同一1,661个冻结可执行攻击案例，采用干净评测提示与轨迹/状态混合裁判。攻击成功率越低越安全。正文将Qwen-plus缩写明确为qwen-plus-2025-12-01；选定重放防御和状态确认诊断子集不混入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.10669v1","status":"paper_baselines","published_at":"2026-08-11"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"higher_is_better":false,"model_count":6,"result_count":6,"results":[[340,52.44,370,10763,10760],[7659,80.09,370,10763,10760],[10761,74.92,370,10763,10760],[10762,81.74,370,10763,10760],[3490,77.93,370,10763,10760],[227,43.62,370,10763,10760]]},{"id":"swe-refactor-migration-score-2026-paper","name":"SWE Refactor Bench · real repository migration (2026 paper)","group":"编程","metric":"Migration-gated score","unit":"score","public_history":true,"protocol":{"en":"The complete 26-configuration panel covers eight distinct models on the same 20 repository migrations. The native 0–100 score gates behavioral test credit through build and migration-audit requirements, so retaining behavior without actually migrating cannot earn full credit. Harness and effort stay explicit; the ungated behavioral pass column is not used as task success.","zh":"完整26配置面板覆盖八个不同模型，在同一20项仓库迁移上评测。0–100原生分数将行为测试得分置于构建与迁移审计门槛之后，因此只维持旧行为而没有真正迁移不能得满分。框架与强度明确保留，不把未加迁移门槛的行为测试通过列当任务成功率。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.23564v1","status":"paper_baselines","published_at":"2026-08-24"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":8,"result_count":26,"results":[[755,20.5,370,10764,10765],[755,28.5,370,10766,10765],[755,34.5,370,10767,10765],[755,47.0,370,10768,10765],[755,31.0,370,10769,10765],[762,4.0,370,10770,10765],[762,7.0,370,10771,10765],[762,6.5,370,10772,10765],[762,19.0,370,10773,10765],[762,9.5,370,10774,10765],[762,28.5,370,10775,10765],[2977,19.5,370,10769,10765],[3568,4.0,370,10764,10765],[3568,15.0,370,10766,10765],[3568,6.0,370,10767,10765],[3568,9.0,370,10768,10765],[3568,8.5,370,10769,10765],[3408,4.0,370,10770,10765],[3408,4.0,370,10771,10765],[3408,0.0,370,10772,10765],[3408,4.0,370,10773,10765],[3408,5.5,370,10774,10765],[3408,10.5,370,10775,10765],[3560,10.0,370,10769,10765],[3261,7.0,370,10769,10765],[3654,6.5,370,10769,10765]]},{"id":"swe-bench-promax-mini-swe-agent-2026-paper","name":"SWE-Bench ProMax · mini-swe-agent (2026 paper)","group":"编程","metric":"Resolve rate","unit":"%","public_history":true,"protocol":{"en":"The complete six-model panel runs the same executable multilingual ProMax evaluation split within this fixed scaffold. A resolved instance must satisfy the issue-level regression tests. Mini-SWE-Agent and OpenHands are separate cohorts; all seven language slices remain in the source table. Results are from paper v2, while the original August 10 release is verified independently.","zh":"完整六模型面板在固定框架下运行同一可执行多语言ProMax测试划分，解决实例须通过问题级回归测试。Mini-SWE-Agent和OpenHands分别展示，七种语言分项保存在原表。分数来自论文v2，原始8月10日首发独立核验。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.09802v2","status":"paper_baselines","published_at":"2026-08-10"},"model_count":6,"result_count":6,"results":[[6130,0.265,370,10776,10777],[3572,0.306,370,10776,10777],[340,0.218,370,10776,10777],[268,0.22899999999999998,370,10776,10777],[2998,0.265,370,10776,10777],[3417,0.20600000000000002,370,10776,10777]]},{"id":"swe-bench-promax-openhands-2026-paper","name":"SWE-Bench ProMax · openhands (2026 paper)","group":"编程","metric":"Resolve rate","unit":"%","public_history":true,"protocol":{"en":"The complete six-model panel runs the same executable multilingual ProMax evaluation split within this fixed scaffold. A resolved instance must satisfy the issue-level regression tests. Mini-SWE-Agent and OpenHands are separate cohorts; all seven language slices remain in the source table. Results are from paper v2, while the original August 10 release is verified independently.","zh":"完整六模型面板在固定框架下运行同一可执行多语言ProMax测试划分，解决实例须通过问题级回归测试。Mini-SWE-Agent和OpenHands分别展示，七种语言分项保存在原表。分数来自论文v2，原始8月10日首发独立核验。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.09802v2","status":"paper_baselines","published_at":"2026-08-10"},"model_count":6,"result_count":6,"results":[[6130,0.19399999999999998,370,10778,10777],[3572,0.38799999999999996,370,10778,10777],[340,0.41200000000000003,370,10778,10777],[268,0.365,370,10778,10777],[2998,0.32899999999999996,370,10778,10777],[3417,0.365,370,10778,10777]]},{"id":"startupbench-work-artifacts-2026-paper","name":"StartupBench · professional work artifacts (2026 paper)","group":"专业工作","metric":"Overall artifact score","unit":"score","public_history":true,"protocol":{"en":"All nine models complete the same professional tasks across medicine, finance, legal, business, STEM and education, with three independent runs. The published overall artifact-quality mean remains a native 0–100 rubric. The separately reported ≥90 success threshold is not conflated with exact task accuracy.","zh":"完整九个模型在医疗、金融、法律、商业、STEM和教育的同一专业任务上独立运行三次。保留0–100原生作品质量均分，与单独报告的≥90分成功阈值分开，不将该阈值等同题目准确率。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.17800v1","status":"paper_baselines","published_at":"2026-08-18"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":9,"result_count":9,"results":[[3155,73.67,370,10779,10780],[4048,73.61,370,10779,10780],[3196,72.79,370,10779,10780],[3235,67.19,370,10779,10780],[3187,59.95,370,10779,10780],[3186,60.79,370,10779,10780],[10781,59.46,370,10779,10780],[3185,61.11,370,10779,10780],[502,49.73,370,10779,10780]]},{"id":"spar-bench-word-med-2026-paper","name":"SPAR-Bench · word spatial reconstruction (2026 paper)","group":"医疗","metric":"Mean Euclidean distance (MED)","unit":"raw","public_history":true,"protocol":{"en":"All thirteen architecture/training conditions share the same medical spatial-reasoning split. Five from-scratch architectures and three pretrained visual backbones provide at least eight independent architecture families; MLP/Transformer heads and fine-tuning remain explicit configurations. WORD is in-domain and TotalSegmentator is zero-shot. Native MED is lower-is-better. Invisible LaTeX alignment zeros are removed from the original HTML before parsing; raw cell text and coordinates remain in the fixture.","zh":"完整十三种架构或训练条件使用同一医学空间推理划分。五种从头训练架构与三个预训练视觉底模提供至少八个独立架构族，MLP/Transformer头及微调配置明确保留。WORD为域内，TotalSegmentator为零样本。原生MED越低越好。抽取前按原HTML去除不可见LaTeX对齐零，原单元文字与坐标仍保留在fixture。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.28092v1","status":"paper_baselines","published_at":"2026-08-28"},"higher_is_better":false,"model_count":11,"result_count":13,"results":[[10782,8.95,370,10783,10784],[10785,11.68,370,10783,10784],[10786,11.83,370,10783,10784],[10787,11.61,370,10783,10784],[10788,11.49,370,10783,10784],[10789,9.88,370,10790,10784],[10791,9.63,370,10790,10784],[10792,10.47,370,10790,10784],[10793,9.89,370,10790,10784],[10794,10.12,370,10790,10784],[10795,10.21,370,10790,10784],[10789,11.46,370,10796,10784],[10791,11.68,370,10796,10784]]},{"id":"spar-bench-totalsegmentator-med-2026-paper","name":"SPAR-Bench · totalsegmentator spatial reconstruction (2026 paper)","group":"医疗","metric":"Mean Euclidean distance (MED)","unit":"raw","public_history":true,"protocol":{"en":"All thirteen architecture/training conditions share the same medical spatial-reasoning split. Five from-scratch architectures and three pretrained visual backbones provide at least eight independent architecture families; MLP/Transformer heads and fine-tuning remain explicit configurations. WORD is in-domain and TotalSegmentator is zero-shot. Native MED is lower-is-better. Invisible LaTeX alignment zeros are removed from the original HTML before parsing; raw cell text and coordinates remain in the fixture.","zh":"完整十三种架构或训练条件使用同一医学空间推理划分。五种从头训练架构与三个预训练视觉底模提供至少八个独立架构族，MLP/Transformer头及微调配置明确保留。WORD为域内，TotalSegmentator为零样本。原生MED越低越好。抽取前按原HTML去除不可见LaTeX对齐零，原单元文字与坐标仍保留在fixture。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.28092v1","status":"paper_baselines","published_at":"2026-08-28"},"higher_is_better":false,"model_count":11,"result_count":13,"results":[[10782,70.9,370,10797,10784],[10785,68.9,370,10797,10784],[10786,68.6,370,10797,10784],[10787,66.7,370,10797,10784],[10788,67.4,370,10797,10784],[10789,69.8,370,10798,10784],[10791,68.6,370,10798,10784],[10792,68.2,370,10798,10784],[10793,68.7,370,10798,10784],[10794,69.5,370,10798,10784],[10795,70.5,370,10798,10784],[10789,67.9,370,10799,10784],[10791,68.2,370,10799,10784]]},{"id":"varm-structured-joint-2026-paper","name":"VARM-Bench · structured moderation (2026 paper)","group":"安全与可靠性","metric":"Joint reference-record match (JREM)","unit":"%","public_history":true,"protocol":{"en":"Six independent base models appear in the complete twelve-row zero-shot, taxonomy and CoT-SFT panel, all on the same deduplicated 1,600-item test set. JREM follows the frozen six-anchor reference-record convention. Invalid or unparseable outputs stay in the denominator and score zero; field F1, parse validity and hidden-error diagnostics remain separate.","zh":"六个独立底模构成完整十二行零样本、分类体系提示与CoT-SFT面板，使用同一去重1,600题测试集。JREM遵循冻结的六锚点参考记录约定，无效或无法解析的输出保留在分母并记零；字段F1、解析率和隐性错误诊断分别保留。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.15600v1","status":"paper_baselines","published_at":"2026-08-16"},"model_count":6,"result_count":12,"results":[[797,0.319,370,10800,10801],[797,0.364,370,10802,10801],[797,0.595,370,10803,10801],[587,0.24600000000000002,370,10800,10801],[587,0.304,370,10802,10801],[587,0.56,370,10803,10801],[10804,0.079,370,10800,10801],[10804,0.127,370,10802,10801],[10804,0.595,370,10803,10801],[3196,0.5539999999999999,370,10800,10801],[3202,0.496,370,10800,10801],[3185,0.46299999999999997,370,10800,10801]]},{"id":"veriocr-proactive-diagnosis-2026-paper","name":"VeriOCRBench · proactive diagnosis (2026 paper)","group":"视觉与视频","metric":"End-to-end trap diagnosis accuracy","unit":"%","public_history":true,"protocol":{"en":"The complete seventeen-system panel uses the same 1,600 trap-injected OCR tasks under the selected prompting paradigm. Diagnosis accuracy requires both detecting invalidity and identifying the correct trap mechanism; it is measured over all trap cases, rather than only refusals. The 200 valid controls serve the separate over-refusal metric. OCR+LLM pipelines and direct VLMs keep their source identities.","zh":"完整十七个系统在所选提示协议下使用同一1,600道陷阱OCR任务。诊断准确率要求同时识别任务无效性与正确陷阱机制，以全部陷阱题为分母，不仅统计拒答题。200道有效控制题用于独立的过度拒答指标。OCR+LLM流程与直接视觉模型保留原名。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.00232v1","status":"paper_baselines","published_at":"2026-08-31"},"model_count":17,"result_count":17,"results":[[10805,0.12689999999999999,370,10806,10807],[10808,0.125,370,10806,10807],[2058,0.115,370,10806,10807],[353,0.1319,370,10806,10807],[10257,0.061200000000000004,370,10806,10807],[502,0.2469,370,10806,10807],[10809,0.2169,370,10806,10807],[10810,0.2412,370,10806,10807],[7685,0.46880000000000005,370,10806,10807],[2590,0.2581,370,10806,10807],[2582,0.23559999999999998,370,10806,10807],[2576,0.23440000000000003,370,10806,10807],[10811,0.21059999999999998,370,10806,10807],[9052,0.1744,370,10806,10807],[2706,0.12689999999999999,370,10806,10807],[10812,0.2781,370,10806,10807],[9691,0.0019,370,10806,10807]]},{"id":"veriocr-assisted-diagnosis-2026-paper","name":"VeriOCRBench · assisted diagnosis (2026 paper)","group":"视觉与视频","metric":"End-to-end trap diagnosis accuracy","unit":"%","public_history":true,"protocol":{"en":"The complete seventeen-system panel uses the same 1,600 trap-injected OCR tasks under the selected prompting paradigm. Diagnosis accuracy requires both detecting invalidity and identifying the correct trap mechanism; it is measured over all trap cases, rather than only refusals. The 200 valid controls serve the separate over-refusal metric. OCR+LLM pipelines and direct VLMs keep their source identities.","zh":"完整十七个系统在所选提示协议下使用同一1,600道陷阱OCR任务。诊断准确率要求同时识别任务无效性与正确陷阱机制，以全部陷阱题为分母，不仅统计拒答题。200道有效控制题用于独立的过度拒答指标。OCR+LLM流程与直接视觉模型保留原名。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.00232v1","status":"paper_baselines","published_at":"2026-08-31"},"model_count":17,"result_count":17,"results":[[10805,0.5256000000000001,370,10813,10807],[10808,0.5356000000000001,370,10813,10807],[2058,0.7694,370,10813,10807],[353,0.6738,370,10813,10807],[10257,0.7569,370,10813,10807],[502,0.0725,370,10813,10807],[10809,0.5981000000000001,370,10813,10807],[10810,0.83,370,10813,10807],[7685,0.7612000000000001,370,10813,10807],[2590,0.7419,370,10813,10807],[2582,0.7381,370,10813,10807],[2576,0.805,370,10813,10807],[10811,0.5462,370,10813,10807],[9052,0.6088,370,10813,10807],[2706,0.6731,370,10813,10807],[10812,0.7119,370,10813,10807],[9691,0.0031,370,10813,10807]]},{"id":"vga-benchv2-aesthetic-2026-paper","name":"VGA-BenchV2 · video aesthetics (2026 paper)","group":"生成与创作","metric":"Normalized aesthetic score","unit":"score","public_history":true,"protocol":{"en":"All twelve published video-generation systems are evaluated under the shared dimension-aligned VGA prompt suite. Generated evaluation videos are held out from evaluator training. The source aggregates normalized aesthetic subdimension scores using VAQA-Net and retains its native 0–1 aesthetic metric, separately from tag classification and generation-quality categories.","zh":"完整十二个视频生成系统采用共同按维度组织的VGA提示集。用于评测的生成视频与裁判训练集隔离。原指标通过VAQA-Net汇总归一化美学子维度，保留0–1原生美学分数，与标签分类和生成质量分开。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.25452v1","status":"paper_baselines","published_at":"2026-08-26"},"score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":12,"result_count":12,"results":[[10814,0.2,370,10815,10816],[10817,0.36,370,10815,10816],[10818,0.34,370,10815,10816],[10819,0.29,370,10815,10816],[10820,0.31,370,10815,10816],[10821,0.41,370,10815,10816],[10822,0.35,370,10815,10816],[10823,0.21,370,10815,10816],[10824,0.22,370,10815,10816],[10825,0.45,370,10815,10816],[10139,0.46,370,10815,10816],[10826,0.5,370,10815,10816]]},{"id":"videovibe-failure-diagnosis-2026-paper","name":"VideoVIBE · video failure diagnosis (2026 paper)","group":"视觉与视频","metric":"Mean diagnosis accuracy","unit":"%","public_history":true,"protocol":{"en":"The complete fourteen-system panel answers the same human-verified single-choice failure-diagnosis questions spanning twelve subdimensions. The source’s mean accuracy remains separate from generation quality. Thinking/instruct variants and the V2Lens agent preserve their source configuration; manually verified evidence supports every retained diagnostic item.","zh":"完整十四系统面板回答同一组人工核验的单选故障诊断题，覆盖十二个子维度。保留论文平均准确率，与视频生成质量区分。思考或指令变体和V2Lens Agent保留原始配置，每道保留题都有人工核验的证据。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.09573v1","status":"paper_baselines","published_at":"2026-08-10"},"model_count":12,"result_count":14,"results":[[7675,0.6425,370,10827,10828],[10829,0.5821000000000001,370,10830,10828],[6304,0.5709000000000001,370,10831,10828],[10832,0.6454000000000001,370,10833,10828],[10834,0.35109999999999997,370,10835,10828],[7111,0.40909999999999996,370,10836,10828],[6342,0.4569,370,10837,10828],[6342,0.5992000000000001,370,10838,10828],[6238,0.5227,370,10839,10828],[6238,0.6096,370,10840,10828],[2590,0.639,370,10841,10828],[10842,0.5378000000000001,370,10843,10828],[10844,0.4961,370,10845,10828],[10846,0.7172,370,10847,10828]]},{"id":"viper-veterinary-pathology-2026-paper","name":"VIPER · veterinary toxicologic pathology (2026 paper)","group":"医疗","metric":"Overall pathology score","unit":"score","public_history":true,"protocol":{"en":"All sixteen models answer the same expert-curated 1,251 questions on 419 rat histology images across seven organ systems. The overall native 0–100 score combines MCQ accuracy, KPrim scoring and free-text assessment, which remain distinct in the fixture. Human-pathology, general-purpose and veterinary-fine-tuned systems keep their declared domains and bootstrap intervals.","zh":"完整十六模型回答同一1,251道专家编写问题，基于七个器官系统的419张大鼠组织图像。总体0–100原生分数结合单选、KPrim和自由文本评判，分项保存在原表。人类病理、通用与兽医微调系统保留声明领域及自助置信区间。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.26382v1","status":"paper_baselines","published_at":"2026-08-26"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":16,"result_count":16,"results":[[10848,62.4,370,10849,10850],[10851,61.2,370,10849,10850],[3184,56.0,370,10852,10850],[3665,54.4,370,10852,10850],[10853,52.4,370,10852,10850],[10854,51.0,370,10855,10850],[3572,48.2,370,10852,10850],[4958,48.0,370,10852,10850],[2055,41.0,370,10852,10850],[10856,36.1,370,10855,10850],[10857,29.5,370,10855,10850],[10858,28.3,370,10855,10850],[6337,27.4,370,10852,10850],[8524,24.8,370,10855,10850],[10859,19.2,370,10855,10850],[9036,16.2,370,10855,10850]]},{"id":"ves-bench-uniform64-2026-paper","name":"VES-Bench · fixed 64-frame understanding (2026 paper)","group":"视觉与视频","metric":"Question accuracy at Uniform-64","unit":"%","public_history":true,"protocol":{"en":"The complete six-base-model Uniform-64 panel uses the same 600 evidence-grounded video questions and a fixed 64-frame sampling budget. Gemini-3-Flash’s 64-frame row is combined only with the five direct-VLM Uniform-64 rows. Variable-frame sweeps and same-backbone agents are separate protocols and do not inflate the model count.","zh":"完整六底模Uniform-64面板在固定64帧预算下回答同一600道有证据的视频问题。Gemini-3-Flash仅取64帧行，与五个直接视觉模型Uniform-64行对齐。可变帧扫描和同底模Agent属于其他协议，不增加模型数。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.22516v1","status":"paper_baselines","published_at":"2026-08-23"},"model_count":6,"result_count":6,"results":[[3424,0.43799999999999994,370,10860,10861],[7779,0.272,370,10860,10861],[6342,0.43700000000000006,370,10860,10861],[6238,0.44299999999999995,370,10860,10861],[3764,0.475,370,10860,10861],[764,0.48700000000000004,370,10860,10861]]},{"id":"veb-oob-sqs-2026-paper","name":"Value Engine Benchmark · out of box (2026 paper)","group":"专业工作","metric":"Sales Quality Score (SQS)","unit":"score","public_history":true,"protocol":{"en":"All thirteen models use the same nine negotiation scenarios and fifteen seeds: 135 episodes per model in each of two conditions, 3,510 episodes overall. The published canonical-135 mean Sales Quality Score stays on its native 0–100 rubric; out-of-box and methodology-pack tracks are separate. The task environment and grader code are public. Full original trajectories require a separate commercial license; only a 78-row preview is public. These are published summary results, without a dynamic rerun here. Zenodo records the paper publication on August 24.","zh":"十三个模型使用相同九类谈判场景、十五个种子：每模型每条件135回合，两条件合计3,510回合。保留canonical-135已发布的0–100原生销售质量分数均值，开箱与方法论包条件分开。任务环境和评分代码公开；完整原始轨迹需另行商业许可，仅78行预览公开。本次导入已发布汇总，未做动态复验。Zenodo记录论文8月24日发表。"},"source":{"name":"Paper authors","url":"https://zenodo.org/records/22073789","status":"paper_baselines","published_at":"2026-08-24"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":13,"result_count":13,"results":[[3473,63.26,370,10862,10863],[4276,53.61,370,10862,10863],[3476,62.67,370,10862,10863],[4278,53.12,370,10862,10863],[413,56.79,370,10862,10863],[5109,54.21,370,10862,10863],[3208,68.44,370,10862,10863],[4972,72.17,370,10862,10863],[433,53.79,370,10862,10863],[5371,51.54,370,10862,10863],[5105,60.26,370,10862,10863],[10864,60.82,370,10862,10863],[5075,59.23,370,10862,10863]]},{"id":"veb-pack-sqs-2026-paper","name":"Value Engine Benchmark · methodology pack (2026 paper)","group":"专业工作","metric":"Sales Quality Score (SQS)","unit":"score","public_history":true,"protocol":{"en":"All thirteen models use the same nine negotiation scenarios and fifteen seeds: 135 episodes per model in each of two conditions, 3,510 episodes overall. The published canonical-135 mean Sales Quality Score stays on its native 0–100 rubric; out-of-box and methodology-pack tracks are separate. The task environment and grader code are public. Full original trajectories require a separate commercial license; only a 78-row preview is public. These are published summary results, without a dynamic rerun here. Zenodo records the paper publication on August 24.","zh":"十三个模型使用相同九类谈判场景、十五个种子：每模型每条件135回合，两条件合计3,510回合。保留canonical-135已发布的0–100原生销售质量分数均值，开箱与方法论包条件分开。任务环境和评分代码公开；完整原始轨迹需另行商业许可，仅78行预览公开。本次导入已发布汇总，未做动态复验。Zenodo记录论文8月24日发表。"},"source":{"name":"Paper authors","url":"https://zenodo.org/records/22073789","status":"paper_baselines","published_at":"2026-08-24"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":13,"result_count":13,"results":[[3473,68.54,370,10865,10863],[4276,51.93,370,10865,10863],[3476,64.19,370,10865,10863],[4278,58.49,370,10865,10863],[413,56.65,370,10865,10863],[5109,52.87,370,10865,10863],[3208,68.89,370,10865,10863],[4972,72.88,370,10865,10863],[433,56.59,370,10865,10863],[5371,50.79,370,10865,10863],[5105,61.39,370,10865,10863],[10864,62.36,370,10865,10863],[5075,56.46,370,10865,10863]]},{"id":"ancient-bench-recognition-ned-2026-paper","name":"Ancient-Bench · artifact recognition (2026 paper)","group":"视觉与视频","metric":"Overall NED (higher is better)","unit":"score","public_history":true,"protocol":{"en":"All 37 general and OCR-specialist systems recognize the same ancient-artifact benchmark across nine material/media categories. The first member of every published NED/F1 cell is the higher-is-better NED metric, retained on its native 0–100 scale. The complete overall panel is preserved; F1 is a separate metric, and original model versions remain traceable in the fixture.","zh":"完整37个通用或OCR专用系统识别同一古代文物基准，覆盖九类材质或载体。每个NED/F1单元取前项、越高越好的NED指标，保留0–100原生量纲。完整总体面板保留，F1是独立指标，模型原始版本可在fixture追溯。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.27169v1","status":"paper_baselines","published_at":"2026-08-27"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":37,"result_count":37,"results":[[10866,16.0,370,10867,10868],[1610,15.9,370,10867,10868],[10869,47.95,370,10867,10868],[5115,43.18,370,10867,10868],[9749,35.12,370,10867,10868],[8897,45.26,370,10867,10868],[10870,42.03,370,10867,10868],[10871,51.52,370,10867,10868],[10872,52.08,370,10867,10868],[7937,34.92,370,10867,10868],[3764,45.96,370,10867,10868],[3763,47.99,370,10867,10868],[3191,46.56,370,10867,10868],[5828,27.15,370,10867,10868],[7153,39.26,370,10867,10868],[6246,15.18,370,10867,10868],[6969,28.89,370,10867,10868],[7566,27.93,370,10867,10868],[10873,5.29,370,10867,10868],[6140,8.24,370,10867,10868],[6143,23.4,370,10867,10868],[10874,13.99,370,10867,10868],[3187,50.62,370,10867,10868],[10875,7.42,370,10867,10868],[10876,40.42,370,10867,10868],[10877,27.74,370,10867,10868],[10878,28.59,370,10867,10868],[10879,30.75,370,10867,10868],[10880,25.41,370,10867,10868],[9690,25.39,370,10867,10868],[10881,19.27,370,10867,10868],[10882,31.4,370,10867,10868],[9049,30.95,370,10867,10868],[9692,34.42,370,10867,10868],[10883,20.15,370,10867,10868],[10884,19.6,370,10867,10868],[10885,21.69,370,10867,10868]]},{"id":"basketballbench-knowledge-qa-2026-paper","name":"BasketballBench · basketball knowledge (2026 paper)","group":"知识与推理","metric":"Q1 multiple-choice accuracy","unit":"%","public_history":true,"protocol":{"en":"Nine direct VLMs and the BasketballSkills system answer the same 1,200 Q1 basketball-knowledge multiple-choice questions. The complete ten-row main panel is retained. This task stays separate from structured scorebug reading, event F1, temporal localization and the paper’s mixed-task averages.","zh":"九个直接视觉语言模型与BasketballSkills系统回答同一1,200道Q1篮球知识选择题，保留主表完整十行。该任务与结构化记分牌读取、事件F1、时间定位和混合任务均值分开。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.23435v1","status":"paper_baselines","published_at":"2026-08-24"},"model_count":10,"result_count":10,"results":[[3184,0.6459999999999999,370,10886,10887],[3568,0.541,370,10886,10887],[3415,0.659,370,10886,10887],[2754,0.373,370,10886,10887],[4969,0.369,370,10886,10887],[4070,0.42200000000000004,370,10886,10887],[6349,0.342,370,10886,10887],[2654,0.41100000000000003,370,10886,10887],[5725,0.41600000000000004,370,10886,10887],[10888,0.9329999999999999,370,10889,10887]]},{"id":"behavior2trip-final-pass-2026-paper","name":"Behavior2Trip · personalized travel planning (2026 paper)","group":"Agent 与工具","metric":"Average full-constraint pass rate","unit":"%","public_history":true,"protocol":{"en":"The complete ten-row panel covers six distinct base models under ReAct, supervised fine-tuning and B2T-Agent. All use the same behavior-aware travel-planning test suite. The published Avg. Final pass rate requires all itinerary constraints, and remains separate from delivery rate, micro/macro partial constraint scores and flexible LLM grading.","zh":"完整十行面板覆盖六个不同底模，包含ReAct、监督微调和B2T-Agent条件，使用同一行为感知旅行规划测试。Avg. Final通过率要求满足全部行程约束，与交付率、微观或宏观部分约束得分以及宽松LLM裁判分别保留。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.26807v1","status":"paper_baselines","published_at":"2026-08-27"},"model_count":6,"result_count":10,"results":[[5344,0.0,370,6334,10890],[6301,0.006999999999999999,370,6334,10890],[786,0.004,370,6334,10890],[353,0.015,370,6334,10890],[345,0.032,370,6334,10890],[1573,0.027000000000000003,370,6334,10890],[5344,0.03,370,6662,10890],[6301,0.032,370,6662,10890],[5344,0.047,370,10891,10890],[6301,0.062,370,10891,10890]]},{"id":"claw4l-relations-without-info-2026-paper","name":"CLAW-4L · relation classification, without-info (2026 paper)","group":"知识与推理","metric":"Overall six-relation classification accuracy","unit":"%","public_history":true,"protocol":{"en":"The complete thirteen-LLM panel classifies the same six information-relation classes. The two published Info conditions are separate cohorts, with inherited model cells resolved to their exact source context rows. NLI controls lack full six-class aggregate scores and are not treated as missing zeros. Results come from paper v2; August 24 original release is verified in arXiv history.","zh":"完整十三个LLM评测同一六类信息关系分类。论文两种Info条件分别展示，跨行继承的模型名单元追溯到准确原行。NLI控制模型没有完整六分类汇总值，不将缺项记零。结果来自论文v2，arXiv历史核验原始8月24日首发。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.23390v2","status":"paper_baselines","published_at":"2026-08-24"},"model_count":13,"result_count":13,"results":[[5126,0.62,370,10892,10893],[5344,0.755,370,10892,10893],[6301,0.848,370,10892,10893],[4969,0.845,370,10892,10893],[4070,0.925,370,10892,10893],[10894,0.37799999999999995,370,10892,10893],[10895,0.498,370,10892,10893],[10896,0.6,370,10892,10893],[10897,0.593,370,10892,10893],[10898,0.42700000000000005,370,10892,10893],[10899,0.455,370,10892,10893],[5535,0.6,370,10892,10893],[3748,0.9329999999999999,370,10892,10893]]},{"id":"claw4l-relations-with-info-2026-paper","name":"CLAW-4L · relation classification, with-info (2026 paper)","group":"知识与推理","metric":"Overall six-relation classification accuracy","unit":"%","public_history":true,"protocol":{"en":"The complete thirteen-LLM panel classifies the same six information-relation classes. The two published Info conditions are separate cohorts, with inherited model cells resolved to their exact source context rows. NLI controls lack full six-class aggregate scores and are not treated as missing zeros. Results come from paper v2; August 24 original release is verified in arXiv history.","zh":"完整十三个LLM评测同一六类信息关系分类。论文两种Info条件分别展示，跨行继承的模型名单元追溯到准确原行。NLI控制模型没有完整六分类汇总值，不将缺项记零。结果来自论文v2，arXiv历史核验原始8月24日首发。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.23390v2","status":"paper_baselines","published_at":"2026-08-24"},"model_count":13,"result_count":13,"results":[[5126,0.655,370,10900,10893],[5344,0.773,370,10900,10893],[6301,0.8029999999999999,370,10900,10893],[4969,0.802,370,10900,10893],[4070,0.922,370,10900,10893],[10894,0.27699999999999997,370,10900,10893],[10895,0.5,370,10900,10893],[10896,0.5379999999999999,370,10900,10893],[10897,0.5329999999999999,370,10900,10893],[10898,0.41700000000000004,370,10900,10893],[10899,0.46,370,10900,10893],[5535,0.585,370,10900,10893],[3748,0.902,370,10900,10893]]},{"id":"crossview-nuscenes-counting-2026-paper","name":"CrossView · vehicle counting (2026 paper)","group":"视觉与视频","metric":"Uniform-sampling task accuracy","unit":"%","public_history":true,"protocol":{"en":"All eleven VLMs use the same CrossView question split within the selected viewpoint/task setting and the source’s uniform frame-sampling protocol. Vehicle counting and robot temporal reasoning use separate source columns and cohorts. Family labels and merged model cells are resolved without altering the full model panel.","zh":"完整十一个视觉模型在所选视角和任务下使用同一CrossView问题集及论文均匀采帧协议。车辆计数和机器人时间推理对应独立原始列与面板，不相互混合。正确解析模型族标签和合并单元，不改变完整模型列表。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.15539v1","status":"paper_baselines","published_at":"2026-08-16"},"model_count":11,"result_count":11,"results":[[4476,0.325,370,10901,10902],[797,0.275,370,10901,10902],[5126,0.466,370,10901,10902],[5344,0.43200000000000005,370,10901,10902],[2912,0.28800000000000003,370,10901,10902],[2774,0.371,370,10901,10902],[2687,0.402,370,10901,10902],[6345,0.462,370,10901,10902],[340,0.478,370,10901,10902],[6677,0.447,370,10901,10902],[6668,0.43799999999999994,370,10901,10902]]},{"id":"crossview-agibot-temporal-2026-paper","name":"CrossView · robot temporal reasoning (2026 paper)","group":"视觉与视频","metric":"Uniform-sampling task accuracy","unit":"%","public_history":true,"protocol":{"en":"All eleven VLMs use the same CrossView question split within the selected viewpoint/task setting and the source’s uniform frame-sampling protocol. Vehicle counting and robot temporal reasoning use separate source columns and cohorts. Family labels and merged model cells are resolved without altering the full model panel.","zh":"完整十一个视觉模型在所选视角和任务下使用同一CrossView问题集及论文均匀采帧协议。车辆计数和机器人时间推理对应独立原始列与面板，不相互混合。正确解析模型族标签和合并单元，不改变完整模型列表。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.15539v1","status":"paper_baselines","published_at":"2026-08-16"},"model_count":11,"result_count":11,"results":[[4476,0.52,370,10903,10902],[797,0.588,370,10903,10902],[5126,0.664,370,10903,10902],[5344,0.696,370,10903,10902],[2912,0.42,370,10903,10902],[2774,0.6,370,10903,10902],[2687,0.528,370,10903,10902],[6345,0.5479999999999999,370,10903,10902],[340,0.6679999999999999,370,10903,10902],[6677,0.508,370,10903,10902],[6668,0.604,370,10903,10902]]},{"id":"vakra-multisource-grounding-2026-paper","name":"VAKRA · multi-source grounding (2026 paper)","group":"Agent 与工具","metric":"Multi-source multihop grounding accuracy","unit":"%","public_history":true,"protocol":{"en":"All twelve models answer the same 58 multi-source multihop queries. The source provides each model’s successful count, common denominator 58 and rounded percentage; the published percentage is retained exactly. This grounding subtask is separate from other VAKRA tool-use and reasoning metrics.","zh":"完整十二个模型回答同一58道跨来源、多跳查询。原表同时提供成功数、共同分母58和舍入百分比，准确保留论文百分比。该定位子任务与VAKRA其他工具使用和推理指标分开。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.12282v1","status":"paper_baselines","published_at":"2026-08-12"},"model_count":12,"result_count":12,"results":[[3196,0.2759,370,10904,10905],[3181,0.29309999999999997,370,10904,10905],[7611,0.3276,370,10904,10905],[6704,0.2241,370,10904,10905],[3186,0.2759,370,10904,10905],[7716,0.29309999999999997,370,10904,10905],[3419,0.2069,370,10904,10905],[10906,0.2069,370,10904,10905],[10907,0.1379,370,10904,10905],[10908,0.1724,370,10904,10905],[10909,0.2069,370,10904,10905],[10910,0.2069,370,10904,10905]]},{"id":"physicsbench-1d-scalar-xl-2026-author","name":"PhysicsBench · 1D scalar prediction (2026 author release)","group":"数学与科学","metric":"BenchRank quality index","unit":"score","public_history":true,"protocol":{"en":"The complete published model panel uses one matched dataset, output quantity/configuration and XL data scale: Airfoil scaled sound pressure. The paper’s correlation-debiased dominance graph and PageRank procedure produces this native 0–100 relative quality index. Different output cells and data scales remain separate; pooled unequal-coverage boards are not used. The complete values come from the author repository’s pinned release.","zh":"完整已发布模型面板固定为同一数据集、输出量或配置及XL数据规模：Airfoil scaled sound pressure。采用论文去相关支配图和PageRank得到0–100原生相对质量指数。不同输出单元和数据规模分开，不使用覆盖不齐的混合榜。完整分数取自作者仓库的固定版本。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.24056v1","status":"dated_release","published_at":"2026-08-25"},"score_max":100,"saturation_excluded":true,"score_decimals":4,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":10,"result_count":10,"results":[[9203,100.0,370,10911,10912],[10913,67.5649,370,10911,10912],[10914,47.7215,370,10911,10912],[10499,46.933,370,10911,10912],[10915,43.6449,370,10911,10912],[10916,40.7302,370,10911,10912],[10917,40.3557,370,10911,10912],[6811,38.2454,370,10911,10912],[10503,38.1893,370,10911,10912],[10918,37.3333,370,10911,10912]]},{"id":"physicsbench-2d-generation-xl-2026-author","name":"PhysicsBench · 2D image generation (2026 author release)","group":"数学与科学","metric":"BenchRank quality index","unit":"score","public_history":true,"protocol":{"en":"The complete published model panel uses one matched dataset, output quantity/configuration and XL data scale: DeepJEB image geometry. The paper’s correlation-debiased dominance graph and PageRank procedure produces this native 0–100 relative quality index. Different output cells and data scales remain separate; pooled unequal-coverage boards are not used. The complete values come from the author repository’s pinned release.","zh":"完整已发布模型面板固定为同一数据集、输出量或配置及XL数据规模：DeepJEB image geometry。采用论文去相关支配图和PageRank得到0–100原生相对质量指数。不同输出单元和数据规模分开，不使用覆盖不齐的混合榜。完整分数取自作者仓库的固定版本。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.24056v1","status":"dated_release","published_at":"2026-08-25"},"score_max":100,"saturation_excluded":true,"score_decimals":4,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":9,"result_count":9,"results":[[10057,100.0,370,10919,10920],[10921,61.8476,370,10919,10920],[10922,45.3594,370,10919,10920],[10923,42.4903,370,10919,10920],[10924,41.3391,370,10919,10920],[10925,40.9255,370,10919,10920],[10926,39.2864,370,10919,10920],[10927,39.2857,370,10919,10920],[10928,38.2857,370,10919,10920]]},{"id":"physicsbench-2d-field-xl-2026-author","name":"PhysicsBench · 2D field prediction (2026 author release)","group":"数学与科学","metric":"BenchRank quality index","unit":"score","public_history":true,"protocol":{"en":"The complete published model panel uses one matched dataset, output quantity/configuration and XL data scale: DeepJEB structural displacement field. The paper’s correlation-debiased dominance graph and PageRank procedure produces this native 0–100 relative quality index. Different output cells and data scales remain separate; pooled unequal-coverage boards are not used. The complete values come from the author repository’s pinned release.","zh":"完整已发布模型面板固定为同一数据集、输出量或配置及XL数据规模：DeepJEB structural displacement field。采用论文去相关支配图和PageRank得到0–100原生相对质量指数。不同输出单元和数据规模分开，不使用覆盖不齐的混合榜。完整分数取自作者仓库的固定版本。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.24056v1","status":"dated_release","published_at":"2026-08-25"},"score_max":100,"saturation_excluded":true,"score_decimals":4,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":8,"result_count":8,"results":[[10929,100.0,370,10930,10931],[10932,98.6267,370,10930,10931],[10933,90.7159,370,10930,10931],[10934,58.3452,370,10930,10931],[10935,57.237,370,10930,10931],[10936,47.822,370,10930,10931],[10937,42.3573,370,10930,10931],[10938,37.8571,370,10930,10931]]},{"id":"physicsbench-2d-scalar-xl-2026-author","name":"PhysicsBench · 2D scalar prediction (2026 author release)","group":"数学与科学","metric":"BenchRank quality index","unit":"score","public_history":true,"protocol":{"en":"The complete published model panel uses one matched dataset, output quantity/configuration and XL data scale: DeepJEB structural displacement scalar. The paper’s correlation-debiased dominance graph and PageRank procedure produces this native 0–100 relative quality index. Different output cells and data scales remain separate; pooled unequal-coverage boards are not used. The complete values come from the author repository’s pinned release.","zh":"完整已发布模型面板固定为同一数据集、输出量或配置及XL数据规模：DeepJEB structural displacement scalar。采用论文去相关支配图和PageRank得到0–100原生相对质量指数。不同输出单元和数据规模分开，不使用覆盖不齐的混合榜。完整分数取自作者仓库的固定版本。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.24056v1","status":"dated_release","published_at":"2026-08-25"},"score_max":100,"saturation_excluded":true,"score_decimals":4,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":7,"result_count":7,"results":[[10939,100.0,370,10940,10941],[10942,56.9773,370,10940,10941],[10943,46.4306,370,10940,10941],[10944,44.5967,370,10940,10941],[10945,40.7546,370,10940,10941],[10946,39.9187,370,10940,10941],[10947,38.0952,370,10940,10941]]},{"id":"physicsbench-3d-generation-xl-2026-author","name":"PhysicsBench · 3D geometry generation (2026 author release)","group":"数学与科学","metric":"BenchRank quality index","unit":"score","public_history":true,"protocol":{"en":"The complete published model panel uses one matched dataset, output quantity/configuration and XL data scale: DeepJEB geometry. The paper’s correlation-debiased dominance graph and PageRank procedure produces this native 0–100 relative quality index. Different output cells and data scales remain separate; pooled unequal-coverage boards are not used. The complete values come from the author repository’s pinned release.","zh":"完整已发布模型面板固定为同一数据集、输出量或配置及XL数据规模：DeepJEB geometry。采用论文去相关支配图和PageRank得到0–100原生相对质量指数。不同输出单元和数据规模分开，不使用覆盖不齐的混合榜。完整分数取自作者仓库的固定版本。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.24056v1","status":"dated_release","published_at":"2026-08-25"},"score_max":100,"saturation_excluded":true,"score_decimals":4,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":6,"result_count":6,"results":[[10948,100.0,370,10949,10950],[10951,65.2245,370,10949,10950],[10952,42.1567,370,10949,10950],[10953,40.8285,370,10949,10950],[10954,40.5421,370,10949,10950],[10955,40.0,370,10949,10950]]},{"id":"physicsbench-3d-field-xl-2026-author","name":"PhysicsBench · 3D field prediction (2026 author release)","group":"数学与科学","metric":"BenchRank quality index","unit":"score","public_history":true,"protocol":{"en":"The complete published model panel uses one matched dataset, output quantity/configuration and XL data scale: DeepJEB diagonal x-displacement field. The paper’s correlation-debiased dominance graph and PageRank procedure produces this native 0–100 relative quality index. Different output cells and data scales remain separate; pooled unequal-coverage boards are not used. The complete values come from the author repository’s pinned release.","zh":"完整已发布模型面板固定为同一数据集、输出量或配置及XL数据规模：DeepJEB diagonal x-displacement field。采用论文去相关支配图和PageRank得到0–100原生相对质量指数。不同输出单元和数据规模分开，不使用覆盖不齐的混合榜。完整分数取自作者仓库的固定版本。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.24056v1","status":"dated_release","published_at":"2026-08-25"},"score_max":100,"saturation_excluded":true,"score_decimals":4,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":7,"result_count":7,"results":[[10956,100.0,370,10957,10958],[10959,62.6651,370,10957,10958],[10960,55.3965,370,10957,10958],[10961,51.7007,370,10957,10958],[10962,44.8868,370,10957,10958],[10963,36.423,370,10957,10958],[10964,35.2381,370,10957,10958]]},{"id":"physicsbench-3d-scalar-xl-2026-author","name":"PhysicsBench · 3D scalar prediction (2026 author release)","group":"数学与科学","metric":"BenchRank quality index","unit":"score","public_history":true,"protocol":{"en":"The complete published model panel uses one matched dataset, output quantity/configuration and XL data scale: DeepJEB structural displacement scalar. The paper’s correlation-debiased dominance graph and PageRank procedure produces this native 0–100 relative quality index. Different output cells and data scales remain separate; pooled unequal-coverage boards are not used. The complete values come from the author repository’s pinned release.","zh":"完整已发布模型面板固定为同一数据集、输出量或配置及XL数据规模：DeepJEB structural displacement scalar。采用论文去相关支配图和PageRank得到0–100原生相对质量指数。不同输出单元和数据规模分开，不使用覆盖不齐的混合榜。完整分数取自作者仓库的固定版本。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.24056v1","status":"dated_release","published_at":"2026-08-25"},"score_max":100,"saturation_excluded":true,"score_decimals":4,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":10,"result_count":10,"results":[[10965,100.0,370,10940,10966],[10964,96.2159,370,10940,10966],[10967,94.5351,370,10940,10966],[10968,87.6716,370,10940,10966],[10969,86.0557,370,10940,10966],[10970,84.4218,370,10940,10966],[10971,40.2972,370,10940,10966],[10972,39.5788,370,10940,10966],[10973,39.1365,370,10940,10966],[10974,32.0,370,10940,10966]]},{"id":"mole-harm-execution-2026-paper","name":"MOLE · harmful objective execution (2026 paper)","group":"安全与可靠性","metric":"Harmful-objective completion rate","unit":"%","public_history":true,"protocol":{"en":"Thirty-five model checkpoints complete the common forty role-matched harmful assignments against nine stateful services. This cohort retains only the complete N=40 rows from both halves of the original39-model table; four infrastructure-truncated rows (N35–38) are excluded. Harmful completion is scored under the source’s four-way outcome taxonomy and is lower-is-better for safety. It is separate from benign-account monitor AUROC/recall and corpus-generator models.","zh":"三十五个模型权重执行九项有状态服务中的共同40项角色匹配有害任务。本组保留原文39模型表左右两栏中N=40的完整结果，排除四条因基础设施失败只剩35–38项的结果。有害完成率按原文四类结果判定，安全评测中越低越好；与账户监控AUROC、召回及语料生成模型分别记录。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.06966v1","published_at":"2026-09-07","status":"paper_baselines"},"higher_is_better":false,"model_count":35,"result_count":35,"results":[[3185,1.0,370,10975,10976],[3187,0.975,370,10975,10976],[2058,0.975,370,10975,10976],[3009,0.975,370,10975,10976],[283,0.975,370,10975,10976],[10977,0.95,370,10975,10976],[3763,0.925,370,10975,10976],[3482,0.925,370,10975,10976],[3201,0.925,370,10975,10976],[5929,0.9,370,10975,10976],[10978,0.9,370,10975,10976],[2998,0.85,370,10975,10976],[5297,0.825,370,10975,10976],[3196,0.825,370,10975,10976],[3415,0.8,370,10975,10976],[3748,0.8,370,10975,10976],[5406,0.8,370,10975,10976],[4959,0.775,370,10975,10976],[4064,0.775,370,10975,10976],[345,0.75,370,10975,10976],[353,0.725,370,10975,10976],[3184,0.7,370,10975,10976],[325,0.7,370,10975,10976],[2582,0.65,370,10975,10976],[1464,0.575,370,10975,10976],[340,0.4,370,10975,10976],[2261,0.4,370,10975,10976],[3479,0.325,370,10975,10976],[3572,0.275,370,10975,10976],[4958,0.225,370,10975,10976],[3568,0.225,370,10975,10976],[5407,0.2,370,10975,10976],[3578,0.175,370,10975,10976],[62,0.125,370,10975,10976],[5298,0.1,370,10975,10976]]},{"id":"dasancall-postedit-2026-paper","name":"DasanCallDial ·post-editing (2026 paper)","group":"知识与推理","metric":"Transcription exact-match rate","unit":"%","public_history":true,"protocol":{"en":"Eight learned corrector/scaffold rows include six independent corrector architectures: character-seq2seq,Llama3.1,Qwen2.5,mT5,KoBART andpkoT5. They post-edit the same DasanCallDial ASR test utterances. Exact match requires the complete corrected string to equal the human transcript; unchanged incorrect inputs fail. DCSC detection/context and utterance-to-utterance variants remain explicit; daggered primary configurations average three seeds.","zh":"八条学习型纠错器与框架结果覆盖六个独立架构：字符序列模型、Llama3.1、Qwen2.5、mT5、KoBART和pkoT5，纠正相同DasanCallDial测试语句。精确匹配要求整条修正文本等于人工转录，保持错误输入不算成功。DCSC检测与上下文、整句改写配置分别注明；带剑号的主要配置取三个种子均值。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.09889v1","published_at":"2026-09-09","status":"paper_baselines"},"model_count":8,"result_count":8,"results":[[10979,0.8229000000000001,370,10980,10981],[10982,0.5134000000000001,370,10983,10981],[10984,0.6509999999999999,370,10985,10981],[10986,0.8504999999999999,370,10987,10981],[10988,0.8493999999999999,370,10989,10981],[10990,0.8531,370,10991,10981],[10992,0.85,370,10993,10981],[10994,0.8515999999999999,370,10995,10981]]},{"id":"revalexo-h0-2026-paper","name":"RevalExo ·0s (2026 paper)","group":"视觉与视频","metric":"Subject-averaged macro F1","unit":"score","public_history":true,"protocol":{"en":"The complete20-row learned-model/fusion panel predicts the same13subjects’ eleven locomotion modes at horizon0s under leave-one-subject-out cross-validation. Models consume inertial,image,video or their explicitly declared fusion streams from the common synchronized recordings. Scores are subject-mean macro F1 on0–100. Accelerometer/gyroscope choices and averaging/concatenation are configurations, not extra model architectures; transition-only columns are separate metrics.","zh":"完整20条学习模型及融合结果在留一受试者交叉验证中，预测同一组13名受试者的11类行走模式，预测时距为0秒。模型使用共同同步记录中的惯性、图像、视频或明确注明的融合输入。保留受试者平均的原生0–100宏F1。加速度计或陀螺仪选择、平均或拼接均属于配置，不增加架构数；过渡时段指标单独保留。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.08090v1","published_at":"2026-09-08","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":20,"result_count":20,"results":[[10996,78.6,370,10997,10998],[10999,80.8,370,11000,10998],[11001,80.2,370,11002,10998],[11003,82.8,370,11004,10998],[11005,82.8,370,11006,10998],[11007,86.1,370,11008,10998],[11009,90.9,370,11010,10998],[11011,84.7,370,11012,10998],[11013,87.5,370,11014,10998],[11015,87.0,370,11016,10998],[11017,91.4,370,11018,10998],[11019,91.3,370,11020,10998],[11021,92.3,370,11022,10998],[11023,92.1,370,11024,10998],[11025,89.2,370,11026,10998],[11027,90.3,370,11028,10998],[11029,90.1,370,11030,10998],[11031,91.1,370,11032,10998],[11033,92.7,370,11034,10998],[11035,92.8,370,11036,10998]]},{"id":"revalexo-h0p1-2026-paper","name":"RevalExo ·0.1s (2026 paper)","group":"视觉与视频","metric":"Subject-averaged macro F1","unit":"score","public_history":true,"protocol":{"en":"The complete20-row learned-model/fusion panel predicts the same13subjects’ eleven locomotion modes at horizon0.1s under leave-one-subject-out cross-validation. Models consume inertial,image,video or their explicitly declared fusion streams from the common synchronized recordings. Scores are subject-mean macro F1 on0–100. Accelerometer/gyroscope choices and averaging/concatenation are configurations, not extra model architectures; transition-only columns are separate metrics.","zh":"完整20条学习模型及融合结果在留一受试者交叉验证中，预测同一组13名受试者的11类行走模式，预测时距为0.1秒。模型使用共同同步记录中的惯性、图像、视频或明确注明的融合输入。保留受试者平均的原生0–100宏F1。加速度计或陀螺仪选择、平均或拼接均属于配置，不增加架构数；过渡时段指标单独保留。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.08090v1","published_at":"2026-09-08","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":20,"result_count":20,"results":[[10996,76.9,370,11037,10998],[10999,79.6,370,11038,10998],[11001,80.6,370,11039,10998],[11003,83.2,370,11040,10998],[11005,83.2,370,11041,10998],[11007,85.9,370,11042,10998],[11009,90.6,370,11043,10998],[11011,84.3,370,11044,10998],[11013,86.9,370,11045,10998],[11015,86.4,370,11046,10998],[11017,90.9,370,11047,10998],[11019,90.8,370,11048,10998],[11021,91.8,370,11049,10998],[11023,91.6,370,11050,10998],[11025,88.8,370,11051,10998],[11027,90.1,370,11052,10998],[11029,89.3,370,11053,10998],[11031,90.6,370,11054,10998],[11033,92.3,370,11055,10998],[11035,92.4,370,11056,10998]]},{"id":"revalexo-h0p2-2026-paper","name":"RevalExo ·0.2s (2026 paper)","group":"视觉与视频","metric":"Subject-averaged macro F1","unit":"score","public_history":true,"protocol":{"en":"The complete20-row learned-model/fusion panel predicts the same13subjects’ eleven locomotion modes at horizon0.2s under leave-one-subject-out cross-validation. Models consume inertial,image,video or their explicitly declared fusion streams from the common synchronized recordings. Scores are subject-mean macro F1 on0–100. Accelerometer/gyroscope choices and averaging/concatenation are configurations, not extra model architectures; transition-only columns are separate metrics.","zh":"完整20条学习模型及融合结果在留一受试者交叉验证中，预测同一组13名受试者的11类行走模式，预测时距为0.2秒。模型使用共同同步记录中的惯性、图像、视频或明确注明的融合输入。保留受试者平均的原生0–100宏F1。加速度计或陀螺仪选择、平均或拼接均属于配置，不增加架构数；过渡时段指标单独保留。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.08090v1","published_at":"2026-09-08","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":20,"result_count":20,"results":[[10996,75.7,370,11057,10998],[10999,78.5,370,11058,10998],[11001,81.0,370,11059,10998],[11003,83.4,370,11060,10998],[11005,83.5,370,11061,10998],[11007,85.2,370,11062,10998],[11009,89.8,370,11063,10998],[11011,83.6,370,11064,10998],[11013,86.2,370,11065,10998],[11015,85.7,370,11066,10998],[11017,90.4,370,11067,10998],[11019,90.2,370,11068,10998],[11021,91.2,370,11069,10998],[11023,91.0,370,11070,10998],[11025,88.4,370,11071,10998],[11027,89.7,370,11072,10998],[11029,88.5,370,11073,10998],[11031,89.9,370,11074,10998],[11033,91.7,370,11075,10998],[11035,92.0,370,11076,10998]]},{"id":"revalexo-h0p3-2026-paper","name":"RevalExo ·0.3s (2026 paper)","group":"视觉与视频","metric":"Subject-averaged macro F1","unit":"score","public_history":true,"protocol":{"en":"The complete20-row learned-model/fusion panel predicts the same13subjects’ eleven locomotion modes at horizon0.3s under leave-one-subject-out cross-validation. Models consume inertial,image,video or their explicitly declared fusion streams from the common synchronized recordings. Scores are subject-mean macro F1 on0–100. Accelerometer/gyroscope choices and averaging/concatenation are configurations, not extra model architectures; transition-only columns are separate metrics.","zh":"完整20条学习模型及融合结果在留一受试者交叉验证中，预测同一组13名受试者的11类行走模式，预测时距为0.3秒。模型使用共同同步记录中的惯性、图像、视频或明确注明的融合输入。保留受试者平均的原生0–100宏F1。加速度计或陀螺仪选择、平均或拼接均属于配置，不增加架构数；过渡时段指标单独保留。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.08090v1","published_at":"2026-09-08","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":20,"result_count":20,"results":[[10996,74.1,370,11077,10998],[10999,77.1,370,11078,10998],[11001,81.4,370,11079,10998],[11003,83.4,370,11080,10998],[11005,83.7,370,11081,10998],[11007,84.4,370,11082,10998],[11009,89.3,370,11083,10998],[11011,82.6,370,11084,10998],[11013,85.2,370,11085,10998],[11015,85.1,370,11086,10998],[11017,89.6,370,11087,10998],[11019,89.4,370,11088,10998],[11021,90.4,370,11089,10998],[11023,90.3,370,11090,10998],[11025,87.9,370,11091,10998],[11027,89.2,370,11092,10998],[11029,86.7,370,11093,10998],[11031,88.7,370,11094,10998],[11033,90.8,370,11095,10998],[11035,91.3,370,11096,10998]]},{"id":"revalexo-h0p5-2026-paper","name":"RevalExo ·0.5s (2026 paper)","group":"视觉与视频","metric":"Subject-averaged macro F1","unit":"score","public_history":true,"protocol":{"en":"The complete20-row learned-model/fusion panel predicts the same13subjects’ eleven locomotion modes at horizon0.5s under leave-one-subject-out cross-validation. Models consume inertial,image,video or their explicitly declared fusion streams from the common synchronized recordings. Scores are subject-mean macro F1 on0–100. Accelerometer/gyroscope choices and averaging/concatenation are configurations, not extra model architectures; transition-only columns are separate metrics.","zh":"完整20条学习模型及融合结果在留一受试者交叉验证中，预测同一组13名受试者的11类行走模式，预测时距为0.5秒。模型使用共同同步记录中的惯性、图像、视频或明确注明的融合输入。保留受试者平均的原生0–100宏F1。加速度计或陀螺仪选择、平均或拼接均属于配置，不增加架构数；过渡时段指标单独保留。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.08090v1","published_at":"2026-09-08","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":20,"result_count":20,"results":[[10996,70.9,370,11097,10998],[10999,74.2,370,11098,10998],[11001,81.5,370,11099,10998],[11003,83.2,370,11100,10998],[11005,83.6,370,11101,10998],[11007,82.4,370,11102,10998],[11009,87.8,370,11103,10998],[11011,80.8,370,11104,10998],[11013,83.7,370,11105,10998],[11015,83.6,370,11106,10998],[11017,88.1,370,11107,10998],[11019,87.9,370,11108,10998],[11021,88.5,370,11109,10998],[11023,88.6,370,11110,10998],[11025,86.9,370,11111,10998],[11027,88.0,370,11112,10998],[11029,83.2,370,11113,10998],[11031,85.7,370,11114,10998],[11033,88.8,370,11115,10998],[11035,89.6,370,11116,10998]]},{"id":"revalexo-h1-2026-paper","name":"RevalExo ·1s (2026 paper)","group":"视觉与视频","metric":"Subject-averaged macro F1","unit":"score","public_history":true,"protocol":{"en":"The complete20-row learned-model/fusion panel predicts the same13subjects’ eleven locomotion modes at horizon1s under leave-one-subject-out cross-validation. Models consume inertial,image,video or their explicitly declared fusion streams from the common synchronized recordings. Scores are subject-mean macro F1 on0–100. Accelerometer/gyroscope choices and averaging/concatenation are configurations, not extra model architectures; transition-only columns are separate metrics.","zh":"完整20条学习模型及融合结果在留一受试者交叉验证中，预测同一组13名受试者的11类行走模式，预测时距为1秒。模型使用共同同步记录中的惯性、图像、视频或明确注明的融合输入。保留受试者平均的原生0–100宏F1。加速度计或陀螺仪选择、平均或拼接均属于配置，不增加架构数；过渡时段指标单独保留。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.08090v1","published_at":"2026-09-08","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":20,"result_count":20,"results":[[10996,63.1,370,11117,10998],[10999,66.9,370,11118,10998],[11001,77.6,370,11119,10998],[11003,79.0,370,11120,10998],[11005,80.1,370,11121,10998],[11007,77.2,370,11122,10998],[11009,84.0,370,11123,10998],[11011,77.2,370,11124,10998],[11013,80.1,370,11125,10998],[11015,79.9,370,11126,10998],[11017,83.6,370,11127,10998],[11019,84.0,370,11128,10998],[11021,84.0,370,11129,10998],[11023,84.5,370,11130,10998],[11025,83.8,370,11131,10998],[11027,83.9,370,11132,10998],[11029,75.6,370,11133,10998],[11031,78.5,370,11134,10998],[11033,85.1,370,11135,10998],[11035,85.8,370,11136,10998]]},{"id":"kbench-secret-context-2026-paper","name":"K-Bench ·context (2026 paper)","group":"安全与可靠性","metric":"Any-channel secret recovery rate","unit":"%","public_history":true,"protocol":{"en":"The six complete2048-token API models use the same200forget-set queries on the context substrate without an unlearning intervention. OR(all) is the probability that the secret is recovered in any observed answer,reasoning,tool orsummary channel; lower is better. The separate GPT4o-mini/Gemini2.0Flash runs use600queries and256tokens and are excluded. Exact served endpoint IDs are published in the source caption; returned reasoning is explicitly scored.","zh":"六个完整API模型使用2,048词元预算，在context载体上回答相同200条遗忘集查询，本组不作遗忘干预。OR(all)统计答案、思考、工具或摘要任一观测通道恢复秘密的概率，越低越好。GPT4o-mini和Gemini2.0Flash的600题、256词元配置另列，未混入。本组端点版本见原表标题，返回的思考文本明确计分。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.12808v1","published_at":"2026-09-11","status":"paper_baselines"},"higher_is_better":false,"model_count":6,"result_count":6,"results":[[3201,0.775,370,11137,11138],[3396,1.0,370,11137,11138],[7223,0.94,370,11137,11138],[9664,0.885,370,11137,11138],[5114,1.0,370,11137,11138],[5899,1.0,370,11137,11138]]},{"id":"kbench-secret-retrieval-text-2026-paper","name":"K-Bench ·retrieval-text (2026 paper)","group":"安全与可靠性","metric":"Any-channel secret recovery rate","unit":"%","public_history":true,"protocol":{"en":"The six complete2048-token API models use the same200forget-set queries on the retrieval-text substrate without an unlearning intervention. OR(all) is the probability that the secret is recovered in any observed answer,reasoning,tool orsummary channel; lower is better. The separate GPT4o-mini/Gemini2.0Flash runs use600queries and256tokens and are excluded. Exact served endpoint IDs are published in the source caption; returned reasoning is explicitly scored.","zh":"六个完整API模型使用2,048词元预算，在retrieval-text载体上回答相同200条遗忘集查询，本组不作遗忘干预。OR(all)统计答案、思考、工具或摘要任一观测通道恢复秘密的概率，越低越好。GPT4o-mini和Gemini2.0Flash的600题、256词元配置另列，未混入。本组端点版本见原表标题，返回的思考文本明确计分。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.12808v1","published_at":"2026-09-11","status":"paper_baselines"},"higher_is_better":false,"model_count":6,"result_count":6,"results":[[3201,0.27,370,11137,11138],[3396,0.515,370,11137,11138],[7223,0.585,370,11137,11138],[9664,0.45,370,11137,11138],[5114,0.58,370,11137,11138],[5899,0.71,370,11137,11138]]},{"id":"kbench-secret-retrieval-structured-2026-paper","name":"K-Bench ·retrieval-structured (2026 paper)","group":"安全与可靠性","metric":"Any-channel secret recovery rate","unit":"%","public_history":true,"protocol":{"en":"The six complete2048-token API models use the same200forget-set queries on the retrieval-structured substrate without an unlearning intervention. OR(all) is the probability that the secret is recovered in any observed answer,reasoning,tool orsummary channel; lower is better. The separate GPT4o-mini/Gemini2.0Flash runs use600queries and256tokens and are excluded. Exact served endpoint IDs are published in the source caption; returned reasoning is explicitly scored.","zh":"六个完整API模型使用2,048词元预算，在retrieval-structured载体上回答相同200条遗忘集查询，本组不作遗忘干预。OR(all)统计答案、思考、工具或摘要任一观测通道恢复秘密的概率，越低越好。GPT4o-mini和Gemini2.0Flash的600题、256词元配置另列，未混入。本组端点版本见原表标题，返回的思考文本明确计分。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.12808v1","published_at":"2026-09-11","status":"paper_baselines"},"higher_is_better":false,"model_count":6,"result_count":6,"results":[[3201,0.675,370,11137,11138],[3396,0.71,370,11137,11138],[7223,0.44,370,11137,11138],[9664,0.335,370,11137,11138],[5114,0.87,370,11137,11138],[5899,0.965,370,11137,11138]]},{"id":"mpbench-turntaking-2026-paper","name":"MP-Bench ·turn-taking (2026 paper)","group":"知识与推理","metric":"Balanced turn-taking accuracy","unit":"%","public_history":true,"protocol":{"en":"All twelve voice agents process the same multi-party Turn-based Game tasks. Positive and silent target turns are balanced, giving a50%always-respond reference. The selected task accuracy requires responding on the agent’s turn and remaining silent otherwise. Text-only ASR controls are excluded; offline and real-time audio interfaces retain their declared configurations and no latency metric is mixed into accuracy.","zh":"全部十二个语音Agent处理相同多人轮流游戏任务。应答与静默目标均衡，因此始终应答的参照正确率为50%。所选指标要求在自己轮次应答、其他轮次保持静默。排除纯文本ASR参照，保留离线或实时音频接口配置，延迟不并入正确率。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.13076v1","published_at":"2026-09-11","status":"paper_baselines"},"model_count":12,"result_count":12,"results":[[7381,0.5273,370,11139,11140],[502,0.5273,370,11139,11140],[6603,0.4961,370,11141,11140],[6601,0.5273,370,11141,11140],[6602,0.5117,370,11141,11140],[11142,0.46090000000000003,370,11141,11140],[8817,0.4914,370,11141,11140],[11143,0.48829999999999996,370,11141,11140],[6599,0.4844,370,11141,11140],[11144,0.4863,370,11141,11140],[11145,0.48,370,11141,11140],[11146,0.5257000000000001,370,11141,11140]]},{"id":"monaco4d-fairmont-2026-paper","name":"Monaco4D ·Fairmont (2026 paper)","group":"视觉与视频","metric":"Video multi-method assessment fusion (VMAF)","unit":"score","public_history":true,"protocol":{"en":"All six learned causal reconstruction methods render the same held-out cameras/frames of Fairmont under the paper’s reduced-memory-background protocol. Native VMAF remains0–100; throughput is a separate result. The full-quality setting has onlytwo scored methods, and MainStraight has a QUEEN out-of-memory failure, so those populations are not combined with this complete six-method cohort.","zh":"六个学习型因果重建方法在论文的缩减背景内存协议下，渲染Fairmont相同的留出相机视角与帧。VMAF保留原生0–100刻度，吞吐另列。完整质量协议只报告两个方法，MainStraight中QUEEN内存不足，均未与这组完整六方法结果混合。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.16310v1","published_at":"2026-09-14","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":6,"result_count":6,"results":[[11147,40.47,370,11148,11149],[11150,0.89,370,11148,11149],[11151,5.39,370,11148,11149],[11152,0.71,370,11148,11149],[11153,2.72,370,11148,11149],[11154,46.89,370,11148,11149]]},{"id":"monaco4d-rascasse-2026-paper","name":"Monaco4D ·Rascasse (2026 paper)","group":"视觉与视频","metric":"Video multi-method assessment fusion (VMAF)","unit":"score","public_history":true,"protocol":{"en":"All six learned causal reconstruction methods render the same held-out cameras/frames of Rascasse under the paper’s reduced-memory-background protocol. Native VMAF remains0–100; throughput is a separate result. The full-quality setting has onlytwo scored methods, and MainStraight has a QUEEN out-of-memory failure, so those populations are not combined with this complete six-method cohort.","zh":"六个学习型因果重建方法在论文的缩减背景内存协议下，渲染Rascasse相同的留出相机视角与帧。VMAF保留原生0–100刻度，吞吐另列。完整质量协议只报告两个方法，MainStraight中QUEEN内存不足，均未与这组完整六方法结果混合。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.16310v1","published_at":"2026-09-14","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":6,"result_count":6,"results":[[11147,38.73,370,11148,11149],[11150,2.01,370,11148,11149],[11151,12.67,370,11148,11149],[11152,1.66,370,11148,11149],[11153,9.67,370,11148,11149],[11154,53.84,370,11148,11149]]},{"id":"monaco4d-tunnel-2026-paper","name":"Monaco4D ·Tunnel (2026 paper)","group":"视觉与视频","metric":"Video multi-method assessment fusion (VMAF)","unit":"score","public_history":true,"protocol":{"en":"All six learned causal reconstruction methods render the same held-out cameras/frames of Tunnel under the paper’s reduced-memory-background protocol. Native VMAF remains0–100; throughput is a separate result. The full-quality setting has onlytwo scored methods, and MainStraight has a QUEEN out-of-memory failure, so those populations are not combined with this complete six-method cohort.","zh":"六个学习型因果重建方法在论文的缩减背景内存协议下，渲染Tunnel相同的留出相机视角与帧。VMAF保留原生0–100刻度，吞吐另列。完整质量协议只报告两个方法，MainStraight中QUEEN内存不足，均未与这组完整六方法结果混合。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.16310v1","published_at":"2026-09-14","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":6,"result_count":6,"results":[[11147,32.11,370,11148,11149],[11150,0.25,370,11148,11149],[11151,0.39,370,11148,11149],[11152,0.36,370,11148,11149],[11153,7.15,370,11148,11149],[11154,48.9,370,11148,11149]]},{"id":"monaco4d-uphill-2026-paper","name":"Monaco4D ·Uphill (2026 paper)","group":"视觉与视频","metric":"Video multi-method assessment fusion (VMAF)","unit":"score","public_history":true,"protocol":{"en":"All six learned causal reconstruction methods render the same held-out cameras/frames of Uphill under the paper’s reduced-memory-background protocol. Native VMAF remains0–100; throughput is a separate result. The full-quality setting has onlytwo scored methods, and MainStraight has a QUEEN out-of-memory failure, so those populations are not combined with this complete six-method cohort.","zh":"六个学习型因果重建方法在论文的缩减背景内存协议下，渲染Uphill相同的留出相机视角与帧。VMAF保留原生0–100刻度，吞吐另列。完整质量协议只报告两个方法，MainStraight中QUEEN内存不足，均未与这组完整六方法结果混合。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.16310v1","published_at":"2026-09-14","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":6,"result_count":6,"results":[[11147,41.17,370,11148,11149],[11150,0.97,370,11148,11149],[11151,0.22,370,11148,11149],[11152,0.89,370,11148,11149],[11153,1.59,370,11148,11149],[11154,55.84,370,11148,11149]]},{"id":"monaco4d-pool-2026-paper","name":"Monaco4D ·Pool (2026 paper)","group":"视觉与视频","metric":"Video multi-method assessment fusion (VMAF)","unit":"score","public_history":true,"protocol":{"en":"All six learned causal reconstruction methods render the same held-out cameras/frames of Pool under the paper’s reduced-memory-background protocol. Native VMAF remains0–100; throughput is a separate result. The full-quality setting has onlytwo scored methods, and MainStraight has a QUEEN out-of-memory failure, so those populations are not combined with this complete six-method cohort.","zh":"六个学习型因果重建方法在论文的缩减背景内存协议下，渲染Pool相同的留出相机视角与帧。VMAF保留原生0–100刻度，吞吐另列。完整质量协议只报告两个方法，MainStraight中QUEEN内存不足，均未与这组完整六方法结果混合。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.16310v1","published_at":"2026-09-14","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":6,"result_count":6,"results":[[11147,34.36,370,11148,11149],[11150,1.16,370,11148,11149],[11151,38.75,370,11148,11149],[11152,1.24,370,11148,11149],[11153,6.77,370,11148,11149],[11154,51.87,370,11148,11149]]},{"id":"apprentice-gui-2026-paper","name":"ApprenticeBench ·GUI (2026 author results)","group":"Agent 与工具","metric":"Fully correct vendor-bill completion rate","unit":"%","public_history":true,"protocol":{"en":"14 model checkpoints complete the same100 sequential construction accounts-payable bills in the simulated Odoo company. The GUI interface is kept separate. Models receive historical bills, handbook, persistent Markdown memory and scheduled mentor feedback; success requires a fully correct bill. Two construction AP professionals independently checked task realism and solvability. Publication is explicitly September2026; no day is inferred.","zh":"14个模型权重在模拟Odoo公司中依次处理共同100张建筑业应付账单。本组单列GUI接口，保留历史账单、手册、Markdown持久记忆和分阶段导师反馈；成功要求整张账单处理正确。两名建筑业会计分别核验题目真实性与可解性。作者只给出2026年9月，未补造具体日期。"},"source":{"name":"Benchmark authors","url":"https://neocognition.io/blog/apprentice-bench/","published_at":"2026-09","status":"dated_release"},"model_count":14,"result_count":14,"results":[[3406,0.05,1,11155,11156],[3184,0.11,15,11157,11156],[3400,0.07,1,11158,11156],[3196,0.2,15,11159,11156],[3151,0.43,1,11160,11156],[762,0.26,15,11161,11156],[2977,0.18,21,11162,11156],[3386,0.16,83,11163,11156],[3149,0.72,1,11164,11156],[2983,0.24,83,11165,11156],[2958,0.68,15,11166,11156],[3156,0.36,1,11167,11156],[2979,0.19,66,11165,11156],[2974,0.13,5096,11168,11156]]},{"id":"apprentice-api-2026-paper","name":"ApprenticeBench ·API (2026 author results)","group":"Agent 与工具","metric":"Fully correct vendor-bill completion rate","unit":"%","public_history":true,"protocol":{"en":"24 model checkpoints complete the same100 sequential construction accounts-payable bills in the simulated Odoo company. The API interface is kept separate. Models receive historical bills, handbook, persistent Markdown memory and scheduled mentor feedback; success requires a fully correct bill. Two construction AP professionals independently checked task realism and solvability. Publication is explicitly September2026; no day is inferred.","zh":"24个模型权重在模拟Odoo公司中依次处理共同100张建筑业应付账单。本组单列API接口，保留历史账单、手册、Markdown持久记忆和分阶段导师反馈；成功要求整张账单处理正确。两名建筑业会计分别核验题目真实性与可解性。作者只给出2026年9月，未补造具体日期。"},"source":{"name":"Benchmark authors","url":"https://neocognition.io/blog/apprentice-bench/","published_at":"2026-09","status":"dated_release"},"model_count":24,"result_count":24,"results":[[244,0.04,21,11169,11156],[3406,0.15,1,11170,11156],[3184,0.2,15,11171,11156],[3400,0.14,1,11172,11156],[3490,0.1,21,11173,11156],[3196,0.27,15,11174,11156],[3151,0.45,1,11175,11156],[3654,0.17,48,11176,11156],[3391,0.38,5096,11177,11156],[762,0.3,15,11178,11156],[3413,0.12,66,11178,11156],[2977,0.25,21,11179,11156],[3405,0.12,83,11180,11156],[3156,0.49,1,11181,11156],[11182,0.18,25,11183,11156],[3560,0.24,195,11184,11156],[3407,0.19,66,11185,11156],[2974,0.45,5096,11186,11156],[3386,0.32,83,11187,11156],[11188,0.21,25,11187,11156],[3529,0.21,48,11189,11156],[3149,0.7,1,11190,11156],[2983,0.34,83,11191,11156],[2958,0.65,15,11192,11156]]},{"id":"samuelcv-animals10-n100-rgb-accuracy-2026-paper","name":"Samuel CV ·animals10 n100 rgb (2026 author results)","group":"视觉与视频","metric":"Held-out classification accuracy","unit":"%","public_history":true,"protocol":{"en":"6 independent learned classifiers use the same stratified seed42 held-out set of 200 images in animals10_n100_rgb. The original metric is Accuracy. Classic/neural baselines use64px inputs; in the combined RGB panel, ten ImageNet-pretrained CNNs use224px and validation-selected checkpoints, with learning-rate deviations and exact pretrained weights retained per row. Test images are unchanged between model families. Training setups are compared as reported, not controlled-compute architecture effects.","zh":"6种独立学习型分类器在animals10_n100_rgb相同的种子42分层留出集上评测，共200张图像。保留原表Accuracy。传统及浅层神经模型输入为64像素；RGB合并面板中十个ImageNet预训练卷积模型使用224像素并按验证集选择权重，每行保留预训练版本和学习率偏离。各模型的测试图像相同；成绩代表原配置，不解释为同训练成本下的架构差异。"},"source":{"name":"Benchmark authors","url":"https://github.com/srcollins785/Samuel_Collins_CV_Benchmarking","published_at":"2026-09-06","status":"dated_release"},"model_count":6,"result_count":6,"results":[[6750,0.32,370,11193,11194],[10499,0.3,370,11195,11194],[11196,0.24,370,11197,11194],[10489,0.215,370,11198,11194],[11199,0.18,370,11200,11194],[11201,0.095,370,11202,11194]]},{"id":"samuelcv-animals10-n100-rgb-macro-f1-2026-paper","name":"Samuel CV ·animals10 n100 rgb (2026 author results)","group":"视觉与视频","metric":"Class-macro F1","unit":"score","public_history":true,"protocol":{"en":"6 independent learned classifiers use the same stratified seed42 held-out set of 200 images in animals10_n100_rgb. The original metric is Macro F1. Classic/neural baselines use64px inputs; in the combined RGB panel, ten ImageNet-pretrained CNNs use224px and validation-selected checkpoints, with learning-rate deviations and exact pretrained weights retained per row. Test images are unchanged between model families. Training setups are compared as reported, not controlled-compute architecture effects.","zh":"6种独立学习型分类器在animals10_n100_rgb相同的种子42分层留出集上评测，共200张图像。保留原表Macro F1。传统及浅层神经模型输入为64像素；RGB合并面板中十个ImageNet预训练卷积模型使用224像素并按验证集选择权重，每行保留预训练版本和学习率偏离。各模型的测试图像相同；成绩代表原配置，不解释为同训练成本下的架构差异。"},"source":{"name":"Benchmark authors","url":"https://github.com/srcollins785/Samuel_Collins_CV_Benchmarking","published_at":"2026-09-06","status":"dated_release"},"score_max":1,"saturation_excluded":true,"scale_label":{"en":"Native score / 1","zh":"原生分数 / 1"},"score_decimals":4,"model_count":6,"result_count":6,"results":[[6750,0.2946602479515533,370,11193,11194],[10499,0.2937008024056146,370,11195,11194],[11196,0.2419628905385769,370,11197,11194],[10489,0.21333650124813497,370,11198,11194],[11199,0.17776154741873199,370,11200,11194],[11201,0.09274004735031097,370,11202,11194]]},{"id":"samuelcv-animals10-n100-grayscale-accuracy-2026-paper","name":"Samuel CV ·animals10 n100 grayscale (2026 author results)","group":"视觉与视频","metric":"Held-out classification accuracy","unit":"%","public_history":true,"protocol":{"en":"6 independent learned classifiers use the same stratified seed42 held-out set of 200 images in animals10_n100_grayscale. The original metric is Accuracy. Classic/neural baselines use64px inputs; in the combined RGB panel, ten ImageNet-pretrained CNNs use224px and validation-selected checkpoints, with learning-rate deviations and exact pretrained weights retained per row. Test images are unchanged between model families. Training setups are compared as reported, not controlled-compute architecture effects.","zh":"6种独立学习型分类器在animals10_n100_grayscale相同的种子42分层留出集上评测，共200张图像。保留原表Accuracy。传统及浅层神经模型输入为64像素；RGB合并面板中十个ImageNet预训练卷积模型使用224像素并按验证集选择权重，每行保留预训练版本和学习率偏离。各模型的测试图像相同；成绩代表原配置，不解释为同训练成本下的架构差异。"},"source":{"name":"Benchmark authors","url":"https://github.com/srcollins785/Samuel_Collins_CV_Benchmarking","published_at":"2026-09-06","status":"dated_release"},"model_count":6,"result_count":6,"results":[[6750,0.26,370,11203,11204],[10499,0.255,370,11205,11204],[11196,0.215,370,11206,11204],[11199,0.2,370,11207,11204],[10489,0.135,370,11208,11204],[11201,0.125,370,11209,11204]]},{"id":"samuelcv-animals10-n100-grayscale-macro-f1-2026-paper","name":"Samuel CV ·animals10 n100 grayscale (2026 author results)","group":"视觉与视频","metric":"Class-macro F1","unit":"score","public_history":true,"protocol":{"en":"6 independent learned classifiers use the same stratified seed42 held-out set of 200 images in animals10_n100_grayscale. The original metric is Macro F1. Classic/neural baselines use64px inputs; in the combined RGB panel, ten ImageNet-pretrained CNNs use224px and validation-selected checkpoints, with learning-rate deviations and exact pretrained weights retained per row. Test images are unchanged between model families. Training setups are compared as reported, not controlled-compute architecture effects.","zh":"6种独立学习型分类器在animals10_n100_grayscale相同的种子42分层留出集上评测，共200张图像。保留原表Macro F1。传统及浅层神经模型输入为64像素；RGB合并面板中十个ImageNet预训练卷积模型使用224像素并按验证集选择权重，每行保留预训练版本和学习率偏离。各模型的测试图像相同；成绩代表原配置，不解释为同训练成本下的架构差异。"},"source":{"name":"Benchmark authors","url":"https://github.com/srcollins785/Samuel_Collins_CV_Benchmarking","published_at":"2026-09-06","status":"dated_release"},"score_max":1,"saturation_excluded":true,"scale_label":{"en":"Native score / 1","zh":"原生分数 / 1"},"score_decimals":4,"model_count":6,"result_count":6,"results":[[6750,0.24882272638086594,370,11203,11204],[10499,0.23917182085474398,370,11205,11204],[11196,0.2070077000536735,370,11206,11204],[11199,0.19147678754612119,370,11207,11204],[10489,0.13242909379925566,370,11208,11204],[11201,0.12868649187762374,370,11209,11204]]},{"id":"samuelcv-animals10-n500-rgb-accuracy-2026-paper","name":"Samuel CV ·animals10 n500 rgb (2026 author results)","group":"视觉与视频","metric":"Held-out classification accuracy","unit":"%","public_history":true,"protocol":{"en":"16 independent learned classifiers use the same stratified seed42 held-out set of 1000 images in animals10_n500_rgb. The original metric is Accuracy. Classic/neural baselines use64px inputs; in the combined RGB panel, ten ImageNet-pretrained CNNs use224px and validation-selected checkpoints, with learning-rate deviations and exact pretrained weights retained per row. Test images are unchanged between model families. Training setups are compared as reported, not controlled-compute architecture effects.","zh":"16种独立学习型分类器在animals10_n500_rgb相同的种子42分层留出集上评测，共1000张图像。保留原表Accuracy。传统及浅层神经模型输入为64像素；RGB合并面板中十个ImageNet预训练卷积模型使用224像素并按验证集选择权重，每行保留预训练版本和学习率偏离。各模型的测试图像相同；成绩代表原配置，不解释为同训练成本下的架构差异。"},"source":{"name":"Benchmark authors","url":"https://github.com/srcollins785/Samuel_Collins_CV_Benchmarking","published_at":"2026-09-06","status":"dated_release"},"model_count":16,"result_count":16,"results":[[10489,0.226,370,11198,11210],[11201,0.18,370,11202,11210],[10499,0.322,370,11195,11210],[11196,0.353,370,11197,11210],[11199,0.287,370,11200,11210],[6750,0.433,370,11193,11210],[11211,0.9,370,11212,11210],[11213,0.944,370,11214,11210],[6642,0.895,370,11215,11210],[11216,0.865,370,11217,11210],[11218,0.918,370,11219,11210],[11220,0.896,370,11221,11210],[11222,0.904,370,11223,11210],[10946,0.928,370,11224,11210],[10939,0.926,370,11225,11210],[11226,0.939,370,11227,11210]]},{"id":"samuelcv-animals10-n500-rgb-macro-f1-2026-paper","name":"Samuel CV ·animals10 n500 rgb (2026 author results)","group":"视觉与视频","metric":"Class-macro F1","unit":"score","public_history":true,"protocol":{"en":"16 independent learned classifiers use the same stratified seed42 held-out set of 1000 images in animals10_n500_rgb. The original metric is Macro F1. Classic/neural baselines use64px inputs; in the combined RGB panel, ten ImageNet-pretrained CNNs use224px and validation-selected checkpoints, with learning-rate deviations and exact pretrained weights retained per row. Test images are unchanged between model families. Training setups are compared as reported, not controlled-compute architecture effects.","zh":"16种独立学习型分类器在animals10_n500_rgb相同的种子42分层留出集上评测，共1000张图像。保留原表Macro F1。传统及浅层神经模型输入为64像素；RGB合并面板中十个ImageNet预训练卷积模型使用224像素并按验证集选择权重，每行保留预训练版本和学习率偏离。各模型的测试图像相同；成绩代表原配置，不解释为同训练成本下的架构差异。"},"source":{"name":"Benchmark authors","url":"https://github.com/srcollins785/Samuel_Collins_CV_Benchmarking","published_at":"2026-09-06","status":"dated_release"},"score_max":1,"saturation_excluded":true,"scale_label":{"en":"Native score / 1","zh":"原生分数 / 1"},"score_decimals":4,"model_count":16,"result_count":16,"results":[[10489,0.2252,370,11198,11210],[11201,0.1802,370,11202,11210],[10499,0.3188,370,11195,11210],[11196,0.352,370,11197,11210],[11199,0.2812,370,11200,11210],[6750,0.4336,370,11193,11210],[11211,0.9003,370,11212,11210],[11213,0.9441,370,11214,11210],[6642,0.8955,370,11215,11210],[11216,0.8643,370,11217,11210],[11218,0.918,370,11219,11210],[11220,0.8958,370,11221,11210],[11222,0.9034,370,11223,11210],[10946,0.9283,370,11224,11210],[10939,0.9259,370,11225,11210],[11226,0.939,370,11227,11210]]},{"id":"samuelcv-animals10-n500-grayscale-accuracy-2026-paper","name":"Samuel CV ·animals10 n500 grayscale (2026 author results)","group":"视觉与视频","metric":"Held-out classification accuracy","unit":"%","public_history":true,"protocol":{"en":"6 independent learned classifiers use the same stratified seed42 held-out set of 1000 images in animals10_n500_grayscale. The original metric is Accuracy. Classic/neural baselines use64px inputs; in the combined RGB panel, ten ImageNet-pretrained CNNs use224px and validation-selected checkpoints, with learning-rate deviations and exact pretrained weights retained per row. Test images are unchanged between model families. Training setups are compared as reported, not controlled-compute architecture effects.","zh":"6种独立学习型分类器在animals10_n500_grayscale相同的种子42分层留出集上评测，共1000张图像。保留原表Accuracy。传统及浅层神经模型输入为64像素；RGB合并面板中十个ImageNet预训练卷积模型使用224像素并按验证集选择权重，每行保留预训练版本和学习率偏离。各模型的测试图像相同；成绩代表原配置，不解释为同训练成本下的架构差异。"},"source":{"name":"Benchmark authors","url":"https://github.com/srcollins785/Samuel_Collins_CV_Benchmarking","published_at":"2026-09-06","status":"dated_release"},"model_count":6,"result_count":6,"results":[[6750,0.351,370,11203,11228],[10499,0.299,370,11205,11228],[11196,0.283,370,11206,11228],[11199,0.253,370,11207,11228],[11201,0.191,370,11209,11228],[10489,0.169,370,11208,11228]]},{"id":"samuelcv-animals10-n500-grayscale-macro-f1-2026-paper","name":"Samuel CV ·animals10 n500 grayscale (2026 author results)","group":"视觉与视频","metric":"Class-macro F1","unit":"score","public_history":true,"protocol":{"en":"6 independent learned classifiers use the same stratified seed42 held-out set of 1000 images in animals10_n500_grayscale. The original metric is Macro F1. Classic/neural baselines use64px inputs; in the combined RGB panel, ten ImageNet-pretrained CNNs use224px and validation-selected checkpoints, with learning-rate deviations and exact pretrained weights retained per row. Test images are unchanged between model families. Training setups are compared as reported, not controlled-compute architecture effects.","zh":"6种独立学习型分类器在animals10_n500_grayscale相同的种子42分层留出集上评测，共1000张图像。保留原表Macro F1。传统及浅层神经模型输入为64像素；RGB合并面板中十个ImageNet预训练卷积模型使用224像素并按验证集选择权重，每行保留预训练版本和学习率偏离。各模型的测试图像相同；成绩代表原配置，不解释为同训练成本下的架构差异。"},"source":{"name":"Benchmark authors","url":"https://github.com/srcollins785/Samuel_Collins_CV_Benchmarking","published_at":"2026-09-06","status":"dated_release"},"score_max":1,"saturation_excluded":true,"scale_label":{"en":"Native score / 1","zh":"原生分数 / 1"},"score_decimals":4,"model_count":6,"result_count":6,"results":[[6750,0.34891593165350476,370,11203,11228],[10499,0.29560402361826543,370,11205,11228],[11196,0.2805650435448422,370,11206,11228],[11199,0.2505237211740173,370,11207,11228],[11201,0.19003833597173908,370,11209,11228],[10489,0.16747253250148714,370,11208,11228]]},{"id":"samuelcv-intel-n500-rgb-accuracy-2026-paper","name":"Samuel CV ·intel n500 rgb (2026 author results)","group":"视觉与视频","metric":"Held-out classification accuracy","unit":"%","public_history":true,"protocol":{"en":"6 independent learned classifiers use the same stratified seed42 held-out set of 600 images in intel_n500_rgb. The original metric is Accuracy. Classic/neural baselines use64px inputs; in the combined RGB panel, ten ImageNet-pretrained CNNs use224px and validation-selected checkpoints, with learning-rate deviations and exact pretrained weights retained per row. Test images are unchanged between model families. Training setups are compared as reported, not controlled-compute architecture effects.","zh":"6种独立学习型分类器在intel_n500_rgb相同的种子42分层留出集上评测，共600张图像。保留原表Accuracy。传统及浅层神经模型输入为64像素；RGB合并面板中十个ImageNet预训练卷积模型使用224像素并按验证集选择权重，每行保留预训练版本和学习率偏离。各模型的测试图像相同；成绩代表原配置，不解释为同训练成本下的架构差异。"},"source":{"name":"Benchmark authors","url":"https://github.com/srcollins785/Samuel_Collins_CV_Benchmarking","published_at":"2026-09-06","status":"dated_release"},"model_count":6,"result_count":6,"results":[[6750,0.705,370,11193,11229],[11196,0.625,370,11197,11229],[10499,0.57,370,11195,11229],[11199,0.49833333333333335,370,11200,11229],[11201,0.395,370,11202,11229],[10489,0.3933333333333333,370,11198,11229]]},{"id":"samuelcv-intel-n500-rgb-macro-f1-2026-paper","name":"Samuel CV ·intel n500 rgb (2026 author results)","group":"视觉与视频","metric":"Class-macro F1","unit":"score","public_history":true,"protocol":{"en":"6 independent learned classifiers use the same stratified seed42 held-out set of 600 images in intel_n500_rgb. The original metric is Macro F1. Classic/neural baselines use64px inputs; in the combined RGB panel, ten ImageNet-pretrained CNNs use224px and validation-selected checkpoints, with learning-rate deviations and exact pretrained weights retained per row. Test images are unchanged between model families. Training setups are compared as reported, not controlled-compute architecture effects.","zh":"6种独立学习型分类器在intel_n500_rgb相同的种子42分层留出集上评测，共600张图像。保留原表Macro F1。传统及浅层神经模型输入为64像素；RGB合并面板中十个ImageNet预训练卷积模型使用224像素并按验证集选择权重，每行保留预训练版本和学习率偏离。各模型的测试图像相同；成绩代表原配置，不解释为同训练成本下的架构差异。"},"source":{"name":"Benchmark authors","url":"https://github.com/srcollins785/Samuel_Collins_CV_Benchmarking","published_at":"2026-09-06","status":"dated_release"},"score_max":1,"saturation_excluded":true,"scale_label":{"en":"Native score / 1","zh":"原生分数 / 1"},"score_decimals":4,"model_count":6,"result_count":6,"results":[[6750,0.7040087935524961,370,11193,11229],[11196,0.6210285485680533,370,11197,11229],[10499,0.5638715792063363,370,11195,11229],[11199,0.4985916407943225,370,11200,11229],[11201,0.39591199771607627,370,11202,11229],[10489,0.39518961279050213,370,11198,11229]]},{"id":"samuelcv-intel-n500-grayscale-accuracy-2026-paper","name":"Samuel CV ·intel n500 grayscale (2026 author results)","group":"视觉与视频","metric":"Held-out classification accuracy","unit":"%","public_history":true,"protocol":{"en":"6 independent learned classifiers use the same stratified seed42 held-out set of 600 images in intel_n500_grayscale. The original metric is Accuracy. Classic/neural baselines use64px inputs; in the combined RGB panel, ten ImageNet-pretrained CNNs use224px and validation-selected checkpoints, with learning-rate deviations and exact pretrained weights retained per row. Test images are unchanged between model families. Training setups are compared as reported, not controlled-compute architecture effects.","zh":"6种独立学习型分类器在intel_n500_grayscale相同的种子42分层留出集上评测，共600张图像。保留原表Accuracy。传统及浅层神经模型输入为64像素；RGB合并面板中十个ImageNet预训练卷积模型使用224像素并按验证集选择权重，每行保留预训练版本和学习率偏离。各模型的测试图像相同；成绩代表原配置，不解释为同训练成本下的架构差异。"},"source":{"name":"Benchmark authors","url":"https://github.com/srcollins785/Samuel_Collins_CV_Benchmarking","published_at":"2026-09-06","status":"dated_release"},"model_count":6,"result_count":6,"results":[[6750,0.6566666666666666,370,11203,11230],[11196,0.5283333333333333,370,11206,11230],[10499,0.505,370,11205,11230],[11199,0.44166666666666665,370,11207,11230],[11201,0.30666666666666664,370,11209,11230],[10489,0.29333333333333333,370,11208,11230]]},{"id":"samuelcv-intel-n500-grayscale-macro-f1-2026-paper","name":"Samuel CV ·intel n500 grayscale (2026 author results)","group":"视觉与视频","metric":"Class-macro F1","unit":"score","public_history":true,"protocol":{"en":"6 independent learned classifiers use the same stratified seed42 held-out set of 600 images in intel_n500_grayscale. The original metric is Macro F1. Classic/neural baselines use64px inputs; in the combined RGB panel, ten ImageNet-pretrained CNNs use224px and validation-selected checkpoints, with learning-rate deviations and exact pretrained weights retained per row. Test images are unchanged between model families. Training setups are compared as reported, not controlled-compute architecture effects.","zh":"6种独立学习型分类器在intel_n500_grayscale相同的种子42分层留出集上评测，共600张图像。保留原表Macro F1。传统及浅层神经模型输入为64像素；RGB合并面板中十个ImageNet预训练卷积模型使用224像素并按验证集选择权重，每行保留预训练版本和学习率偏离。各模型的测试图像相同；成绩代表原配置，不解释为同训练成本下的架构差异。"},"source":{"name":"Benchmark authors","url":"https://github.com/srcollins785/Samuel_Collins_CV_Benchmarking","published_at":"2026-09-06","status":"dated_release"},"score_max":1,"saturation_excluded":true,"scale_label":{"en":"Native score / 1","zh":"原生分数 / 1"},"score_decimals":4,"model_count":6,"result_count":6,"results":[[6750,0.655843022670641,370,11203,11230],[11196,0.5155517792976613,370,11206,11230],[10499,0.49674663219687015,370,11205,11230],[11199,0.4331255413694175,370,11207,11230],[11201,0.3070333215263552,370,11209,11230],[10489,0.29166682172808783,370,11208,11230]]},{"id":"clouda-arabic-normalized-arabic-cer-2026-paper","name":"Clouda OCR ·normalized_arabic_CER (2026 author results)","group":"视觉与视频","metric":"Normalized Arabic character error rate","unit":"raw","public_history":true,"protocol":{"en":"Six distinct OCR checkpoints have complete177/177 valid image-reference pairs on the same distorted Arabic page benchmark v0.1.0. The original aggregate is a mean of per-page edit-distance error rates, lower is better and unbounded above1 because insertions count. Stored outputs were rescored offline, not newly inferred. DotsMOCR169/177 and the failed PaddleOCR smoke run are excluded. Hardware is contextual, not a speed ranking.","zh":"六个独立OCR权重在相同v0.1.0失真阿拉伯语页面上均完成177/177有效图像与参考文本对。保留逐页编辑距离错误率的原生均值，越低越好；插入错误可使结果超过1，不转换成正确率。成绩由存档输出离线重评分，未重新推理。排除只有169/177有效结果的DotsMOCR和失败的PaddleOCR冒烟测试；硬件只作配置说明。"},"source":{"name":"Benchmark authors","url":"https://github.com/sahrasayed3-crypto/clouda-ocr-benchmark","published_at":"2026-09-07","status":"dated_release"},"saturation_excluded":true,"scale_label":{"en":"Normalized Arabic character error rate","zh":"Normalized Arabic character error rate"},"score_decimals":6,"higher_is_better":false,"model_count":6,"result_count":6,"results":[[11231,0.3914971023431782,370,11232,11233],[11234,0.8370283324995594,370,11235,11233],[11236,1.0762598200883418,370,11237,11233],[6149,1.28656702627353,370,11238,11233],[9689,1.486473469125326,370,11239,11233],[11240,2.2263407213309216,370,11241,11233]]},{"id":"clouda-arabic-cer-2026-paper","name":"Clouda OCR ·CER (2026 author results)","group":"视觉与视频","metric":"Strict character error rate","unit":"raw","public_history":true,"protocol":{"en":"Six distinct OCR checkpoints have complete177/177 valid image-reference pairs on the same distorted Arabic page benchmark v0.1.0. The original aggregate is a mean of per-page edit-distance error rates, lower is better and unbounded above1 because insertions count. Stored outputs were rescored offline, not newly inferred. DotsMOCR169/177 and the failed PaddleOCR smoke run are excluded. Hardware is contextual, not a speed ranking.","zh":"六个独立OCR权重在相同v0.1.0失真阿拉伯语页面上均完成177/177有效图像与参考文本对。保留逐页编辑距离错误率的原生均值，越低越好；插入错误可使结果超过1，不转换成正确率。成绩由存档输出离线重评分，未重新推理。排除只有169/177有效结果的DotsMOCR和失败的PaddleOCR冒烟测试；硬件只作配置说明。"},"source":{"name":"Benchmark authors","url":"https://github.com/sahrasayed3-crypto/clouda-ocr-benchmark","published_at":"2026-09-07","status":"dated_release"},"saturation_excluded":true,"scale_label":{"en":"Strict character error rate","zh":"Strict character error rate"},"score_decimals":6,"higher_is_better":false,"model_count":6,"result_count":6,"results":[[11231,0.564666303999014,370,11232,11233],[11234,0.7529046963507875,370,11235,11233],[11236,1.4192405567042832,370,11237,11233],[6149,2.0079401667932815,370,11238,11233],[9689,1.727660839068692,370,11239,11233],[11240,2.3914721538973307,370,11241,11233]]},{"id":"clouda-arabic-wer-2026-paper","name":"Clouda OCR ·WER (2026 author results)","group":"视觉与视频","metric":"Word error rate","unit":"raw","public_history":true,"protocol":{"en":"Six distinct OCR checkpoints have complete177/177 valid image-reference pairs on the same distorted Arabic page benchmark v0.1.0. The original aggregate is a mean of per-page edit-distance error rates, lower is better and unbounded above1 because insertions count. Stored outputs were rescored offline, not newly inferred. DotsMOCR169/177 and the failed PaddleOCR smoke run are excluded. Hardware is contextual, not a speed ranking.","zh":"六个独立OCR权重在相同v0.1.0失真阿拉伯语页面上均完成177/177有效图像与参考文本对。保留逐页编辑距离错误率的原生均值，越低越好；插入错误可使结果超过1，不转换成正确率。成绩由存档输出离线重评分，未重新推理。排除只有169/177有效结果的DotsMOCR和失败的PaddleOCR冒烟测试；硬件只作配置说明。"},"source":{"name":"Benchmark authors","url":"https://github.com/sahrasayed3-crypto/clouda-ocr-benchmark","published_at":"2026-09-07","status":"dated_release"},"saturation_excluded":true,"scale_label":{"en":"Word error rate","zh":"Word error rate"},"score_decimals":6,"higher_is_better":false,"model_count":6,"result_count":6,"results":[[11231,0.7195107724616108,370,11232,11233],[11234,0.6439773681061527,370,11235,11233],[11236,1.6383134564310837,370,11237,11233],[6149,1.5295432343285742,370,11238,11233],[9689,1.5498254809791352,370,11239,11233],[11240,1.876030865758898,370,11241,11233]]},{"id":"overlooked-overall-2026-paper","name":"Overlooked ·overall (2026 author results)","group":"知识与推理","metric":"Overall category-balanced index","unit":"score","public_history":true,"protocol":{"en":"Seven checkpoint-identified models answer the same106 items in the complete2026-10-22 run, with no generation, judge or truncation failures. The selected overall score uses the author rubric m1.3.0 and DeepSeekV3.1 judge at temperature0. Ten calibration items are diagnostic only; the overall index equally averages six headline category means across96 other items. Native rubric scores0–100 are not percent-correct accuracy. Earlier September runs with judge errors are not combined.","zh":"七个明确标识的模型在2026-10-22完整运行中回答相同106题，无生成、评审或截断失败。本组overall按m1.3.0评分细则及温度0的DeepSeekV3.1评审计分。十道校准题只作诊断；总体指数对其余96题的六个主题均分取等权平均。保留原生0–100评分，不作正确率。未混入存在评审错误的9月运行。"},"source":{"name":"Benchmark authors","url":"https://github.com/kcelestinomaria/overlooked-bench","published_at":"2026-09-07","status":"dated_release"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Native score / 100","zh":"原生分数 / 100"},"score_decimals":2,"model_count":7,"result_count":7,"results":[[9317,83.39,370,11242,11243],[5040,90.7,370,11242,11243],[11244,83.32,370,11242,11243],[11245,89.27,370,11242,11243],[5058,88.04,370,11242,11243],[11246,89.81,370,11242,11243],[9316,87.97,370,11242,11243]]},{"id":"overlooked-ethics-philosophy-2026-paper","name":"Overlooked ·ethics-philosophy (2026 author results)","group":"知识与推理","metric":"Mean category rubric score","unit":"score","public_history":true,"protocol":{"en":"Seven checkpoint-identified models answer the same106 items in the complete2026-10-22 run, with no generation, judge or truncation failures. The selected ethics-philosophy score uses the author rubric m1.3.0 and DeepSeekV3.1 judge at temperature0. Ten calibration items are diagnostic only; the overall index equally averages six headline category means across96 other items. Native rubric scores0–100 are not percent-correct accuracy. Earlier September runs with judge errors are not combined.","zh":"七个明确标识的模型在2026-10-22完整运行中回答相同106题，无生成、评审或截断失败。本组ethics-philosophy按m1.3.0评分细则及温度0的DeepSeekV3.1评审计分。十道校准题只作诊断；总体指数对其余96题的六个主题均分取等权平均。保留原生0–100评分，不作正确率。未混入存在评审错误的9月运行。"},"source":{"name":"Benchmark authors","url":"https://github.com/kcelestinomaria/overlooked-bench","published_at":"2026-09-07","status":"dated_release"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Native score / 100","zh":"原生分数 / 100"},"score_decimals":2,"model_count":7,"result_count":7,"results":[[9317,71.93,370,11247,11243],[5040,86.41,370,11247,11243],[11244,77.6,370,11247,11243],[11245,81.77,370,11247,11243],[5058,74.69,370,11247,11243],[11246,86.67,370,11247,11243],[9316,78.23,370,11247,11243]]},{"id":"overlooked-niche-academic-2026-paper","name":"Overlooked ·niche-academic (2026 author results)","group":"知识与推理","metric":"Mean category rubric score","unit":"score","public_history":true,"protocol":{"en":"Seven checkpoint-identified models answer the same106 items in the complete2026-10-22 run, with no generation, judge or truncation failures. The selected niche-academic score uses the author rubric m1.3.0 and DeepSeekV3.1 judge at temperature0. Ten calibration items are diagnostic only; the overall index equally averages six headline category means across96 other items. Native rubric scores0–100 are not percent-correct accuracy. Earlier September runs with judge errors are not combined.","zh":"七个明确标识的模型在2026-10-22完整运行中回答相同106题，无生成、评审或截断失败。本组niche-academic按m1.3.0评分细则及温度0的DeepSeekV3.1评审计分。十道校准题只作诊断；总体指数对其余96题的六个主题均分取等权平均。保留原生0–100评分，不作正确率。未混入存在评审错误的9月运行。"},"source":{"name":"Benchmark authors","url":"https://github.com/kcelestinomaria/overlooked-bench","published_at":"2026-09-07","status":"dated_release"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Native score / 100","zh":"原生分数 / 100"},"score_decimals":2,"model_count":7,"result_count":7,"results":[[9317,89.83,370,11248,11243],[5040,96.67,370,11248,11243],[11244,95.17,370,11248,11243],[11245,96.17,370,11248,11243],[5058,97.0,370,11248,11243],[11246,98.0,370,11248,11243],[9316,93.17,370,11248,11243]]},{"id":"overlooked-org-enterprise-2026-paper","name":"Overlooked ·org-enterprise (2026 author results)","group":"知识与推理","metric":"Mean category rubric score","unit":"score","public_history":true,"protocol":{"en":"Seven checkpoint-identified models answer the same106 items in the complete2026-10-22 run, with no generation, judge or truncation failures. The selected org-enterprise score uses the author rubric m1.3.0 and DeepSeekV3.1 judge at temperature0. Ten calibration items are diagnostic only; the overall index equally averages six headline category means across96 other items. Native rubric scores0–100 are not percent-correct accuracy. Earlier September runs with judge errors are not combined.","zh":"七个明确标识的模型在2026-10-22完整运行中回答相同106题，无生成、评审或截断失败。本组org-enterprise按m1.3.0评分细则及温度0的DeepSeekV3.1评审计分。十道校准题只作诊断；总体指数对其余96题的六个主题均分取等权平均。保留原生0–100评分，不作正确率。未混入存在评审错误的9月运行。"},"source":{"name":"Benchmark authors","url":"https://github.com/kcelestinomaria/overlooked-bench","published_at":"2026-09-07","status":"dated_release"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Native score / 100","zh":"原生分数 / 100"},"score_decimals":2,"model_count":7,"result_count":7,"results":[[9317,91.77,370,11249,11243],[5040,96.98,370,11249,11243],[11244,86.04,370,11249,11243],[11245,93.02,370,11249,11243],[5058,95.1,370,11249,11243],[11246,95.52,370,11249,11243],[9316,93.23,370,11249,11243]]},{"id":"overlooked-education-2026-paper","name":"Overlooked ·education (2026 author results)","group":"知识与推理","metric":"Mean category rubric score","unit":"score","public_history":true,"protocol":{"en":"Seven checkpoint-identified models answer the same106 items in the complete2026-10-22 run, with no generation, judge or truncation failures. The selected education score uses the author rubric m1.3.0 and DeepSeekV3.1 judge at temperature0. Ten calibration items are diagnostic only; the overall index equally averages six headline category means across96 other items. Native rubric scores0–100 are not percent-correct accuracy. Earlier September runs with judge errors are not combined.","zh":"七个明确标识的模型在2026-10-22完整运行中回答相同106题，无生成、评审或截断失败。本组education按m1.3.0评分细则及温度0的DeepSeekV3.1评审计分。十道校准题只作诊断；总体指数对其余96题的六个主题均分取等权平均。保留原生0–100评分，不作正确率。未混入存在评审错误的9月运行。"},"source":{"name":"Benchmark authors","url":"https://github.com/kcelestinomaria/overlooked-bench","published_at":"2026-09-07","status":"dated_release"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Native score / 100","zh":"原生分数 / 100"},"score_decimals":2,"model_count":7,"result_count":7,"results":[[9317,64.9,370,11249,11243],[5040,70.0,370,11249,11243],[11244,71.98,370,11249,11243],[11245,77.08,370,11249,11243],[5058,70.73,370,11249,11243],[11246,70.0,370,11249,11243],[9316,75.21,370,11249,11243]]},{"id":"overlooked-institutional-criticism-2026-paper","name":"Overlooked ·institutional-criticism (2026 author results)","group":"知识与推理","metric":"Mean category rubric score","unit":"score","public_history":true,"protocol":{"en":"Seven checkpoint-identified models answer the same106 items in the complete2026-10-22 run, with no generation, judge or truncation failures. The selected institutional-criticism score uses the author rubric m1.3.0 and DeepSeekV3.1 judge at temperature0. Ten calibration items are diagnostic only; the overall index equally averages six headline category means across96 other items. Native rubric scores0–100 are not percent-correct accuracy. Earlier September runs with judge errors are not combined.","zh":"七个明确标识的模型在2026-10-22完整运行中回答相同106题，无生成、评审或截断失败。本组institutional-criticism按m1.3.0评分细则及温度0的DeepSeekV3.1评审计分。十道校准题只作诊断；总体指数对其余96题的六个主题均分取等权平均。保留原生0–100评分，不作正确率。未混入存在评审错误的9月运行。"},"source":{"name":"Benchmark authors","url":"https://github.com/kcelestinomaria/overlooked-bench","published_at":"2026-09-07","status":"dated_release"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Native score / 100","zh":"原生分数 / 100"},"score_decimals":2,"model_count":7,"result_count":7,"results":[[9317,96.6,370,11250,11243],[5040,100.0,370,11250,11243],[11244,99.65,370,11250,11243],[11245,97.64,370,11250,11243],[5058,98.82,370,11250,11243],[11246,97.92,370,11250,11243],[9316,97.22,370,11250,11243]]},{"id":"overlooked-software-engineering-production-2026-paper","name":"Overlooked ·software-engineering-production (2026 author results)","group":"知识与推理","metric":"Mean category rubric score","unit":"score","public_history":true,"protocol":{"en":"Seven checkpoint-identified models answer the same106 items in the complete2026-10-22 run, with no generation, judge or truncation failures. The selected software-engineering-production score uses the author rubric m1.3.0 and DeepSeekV3.1 judge at temperature0. Ten calibration items are diagnostic only; the overall index equally averages six headline category means across96 other items. Native rubric scores0–100 are not percent-correct accuracy. Earlier September runs with judge errors are not combined.","zh":"七个明确标识的模型在2026-10-22完整运行中回答相同106题，无生成、评审或截断失败。本组software-engineering-production按m1.3.0评分细则及温度0的DeepSeekV3.1评审计分。十道校准题只作诊断；总体指数对其余96题的六个主题均分取等权平均。保留原生0–100评分，不作正确率。未混入存在评审错误的9月运行。"},"source":{"name":"Benchmark authors","url":"https://github.com/kcelestinomaria/overlooked-bench","published_at":"2026-09-07","status":"dated_release"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Native score / 100","zh":"原生分数 / 100"},"score_decimals":2,"model_count":7,"result_count":7,"results":[[9317,85.33,370,11248,11243],[5040,94.17,370,11248,11243],[11244,69.5,370,11248,11243],[11245,89.92,370,11248,11243],[5058,91.92,370,11248,11243],[11246,90.75,370,11248,11243],[9316,90.75,370,11248,11243]]},{"id":"owl-newt-accuracy-2026-paper","name":"O.W.L. & N.E.W.T. ·accuracy (2026 author results)","group":"知识与推理","metric":"Micro exact-answer accuracy","unit":"%","public_history":true,"protocol":{"en":"Ten complete clean model runs use the same public75-question development set:30 O.W.L. and45 N.E.W.T. questions. Alias-normalized exact answers are scored deterministically without a model judge. The deliberately answer-trained LoRA and four unpinned alias endpoints are excluded. Micro accuracy uses all75 answers; the macro benchmark score equally weights the two exams. Original execution dates14–15July2026 are preserved; the source repository was created8September2026. This public-development-set result is not a sealed generalization test.","zh":"十条完整且未针对答案训练的模型运行使用共同75题公开开发集，其中O.W.L.30题、N.E.W.T.45题。答案别名规范化后作确定性精确匹配，不调用模型评审。排除刻意记忆全部答案的LoRA及四个未固定版本的别名端点。微平均正确率按全部75题计；宏分数对两场考试等权。保留2026年7月14–15日原运行日期，来源仓库创建于9月8日。该公开开发集成绩不代表密封测试上的泛化能力。"},"source":{"name":"Benchmark authors","url":"https://github.com/VSBDev/little-hermione","published_at":"2026-09-08","status":"dated_release"},"model_count":10,"result_count":10,"results":[[4972,0.8666666666666667,370,11251,11252],[11253,0.18666666666666668,370,11254,11252],[11255,0.25333333333333335,370,11256,11252],[11257,0.38666666666666666,370,11258,11252],[11259,0.3333333333333333,370,11260,11252],[11261,0.05333333333333334,370,11262,11252],[5368,0.8,370,11263,11252],[11264,0.0,370,11265,11252],[129,0.8666666666666667,370,11266,11252],[11267,0.8666666666666667,370,11268,11252]]},{"id":"owl-newt-owl-score-2026-paper","name":"O.W.L. & N.E.W.T. ·owl_score (2026 author results)","group":"知识与推理","metric":"owl score","unit":"%","public_history":true,"protocol":{"en":"Ten complete clean model runs use the same public75-question development set:30 O.W.L. and45 N.E.W.T. questions. Alias-normalized exact answers are scored deterministically without a model judge. The deliberately answer-trained LoRA and four unpinned alias endpoints are excluded. Micro accuracy uses all75 answers; the macro benchmark score equally weights the two exams. Original execution dates14–15July2026 are preserved; the source repository was created8September2026. This public-development-set result is not a sealed generalization test.","zh":"十条完整且未针对答案训练的模型运行使用共同75题公开开发集，其中O.W.L.30题、N.E.W.T.45题。答案别名规范化后作确定性精确匹配，不调用模型评审。排除刻意记忆全部答案的LoRA及四个未固定版本的别名端点。微平均正确率按全部75题计；宏分数对两场考试等权。保留2026年7月14–15日原运行日期，来源仓库创建于9月8日。该公开开发集成绩不代表密封测试上的泛化能力。"},"source":{"name":"Benchmark authors","url":"https://github.com/VSBDev/little-hermione","published_at":"2026-09-08","status":"dated_release"},"model_count":10,"result_count":10,"results":[[4972,1.0,370,11251,11252],[11253,0.36666699999999997,370,11254,11252],[11255,0.5333330000000001,370,11256,11252],[11257,0.7666670000000001,370,11258,11252],[11259,0.7,370,11260,11252],[11261,0.133333,370,11262,11252],[5368,1.0,370,11263,11252],[11264,0.0,370,11265,11252],[129,1.0,370,11266,11252],[11267,1.0,370,11268,11252]]},{"id":"owl-newt-newt-score-2026-paper","name":"O.W.L. & N.E.W.T. ·newt_score (2026 author results)","group":"知识与推理","metric":"newt score","unit":"%","public_history":true,"protocol":{"en":"Ten complete clean model runs use the same public75-question development set:30 O.W.L. and45 N.E.W.T. questions. Alias-normalized exact answers are scored deterministically without a model judge. The deliberately answer-trained LoRA and four unpinned alias endpoints are excluded. Micro accuracy uses all75 answers; the macro benchmark score equally weights the two exams. Original execution dates14–15July2026 are preserved; the source repository was created8September2026. This public-development-set result is not a sealed generalization test.","zh":"十条完整且未针对答案训练的模型运行使用共同75题公开开发集，其中O.W.L.30题、N.E.W.T.45题。答案别名规范化后作确定性精确匹配，不调用模型评审。排除刻意记忆全部答案的LoRA及四个未固定版本的别名端点。微平均正确率按全部75题计；宏分数对两场考试等权。保留2026年7月14–15日原运行日期，来源仓库创建于9月8日。该公开开发集成绩不代表密封测试上的泛化能力。"},"source":{"name":"Benchmark authors","url":"https://github.com/VSBDev/little-hermione","published_at":"2026-09-08","status":"dated_release"},"model_count":10,"result_count":10,"results":[[4972,0.777778,370,11251,11252],[11253,0.06666699999999999,370,11254,11252],[11255,0.06666699999999999,370,11256,11252],[11257,0.133333,370,11258,11252],[11259,0.088889,370,11260,11252],[11261,0.0,370,11262,11252],[5368,0.666667,370,11263,11252],[11264,0.0,370,11265,11252],[129,0.777778,370,11266,11252],[11267,0.777778,370,11268,11252]]},{"id":"owl-newt-benchmark-score-2026-paper","name":"O.W.L. & N.E.W.T. ·benchmark_score (2026 author results)","group":"知识与推理","metric":"Equal-exam macro score","unit":"score","public_history":true,"protocol":{"en":"Ten complete clean model runs use the same public75-question development set:30 O.W.L. and45 N.E.W.T. questions. Alias-normalized exact answers are scored deterministically without a model judge. The deliberately answer-trained LoRA and four unpinned alias endpoints are excluded. Micro accuracy uses all75 answers; the macro benchmark score equally weights the two exams. Original execution dates14–15July2026 are preserved; the source repository was created8September2026. This public-development-set result is not a sealed generalization test.","zh":"十条完整且未针对答案训练的模型运行使用共同75题公开开发集，其中O.W.L.30题、N.E.W.T.45题。答案别名规范化后作确定性精确匹配，不调用模型评审。排除刻意记忆全部答案的LoRA及四个未固定版本的别名端点。微平均正确率按全部75题计；宏分数对两场考试等权。保留2026年7月14–15日原运行日期，来源仓库创建于9月8日。该公开开发集成绩不代表密封测试上的泛化能力。"},"source":{"name":"Benchmark authors","url":"https://github.com/VSBDev/little-hermione","published_at":"2026-09-08","status":"dated_release"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Native score / 100","zh":"原生分数 / 100"},"score_decimals":2,"model_count":10,"result_count":10,"results":[[4972,88.8889,370,11251,11252],[11253,21.6667,370,11254,11252],[11255,30.0,370,11256,11252],[11257,45.0,370,11258,11252],[11259,39.4444,370,11260,11252],[11261,6.6667,370,11262,11252],[5368,83.3333,370,11263,11252],[11264,0.0,370,11265,11252],[129,88.8889,370,11266,11252],[11267,88.8889,370,11268,11252]]},{"id":"failure-transparent-baseline-2026-paper","name":"Failure-Transparent ·baseline (2026 author results)","group":"安全与可靠性","metric":"Judge-estimated false-success rate","unit":"%","public_history":true,"protocol":{"en":"Six distinct models face the same100 synthetic tool-failure scenarios twice under the baseline instruction condition, giving200 responses per model and3600 across all three conditions. False success is a success claim unsupported by failed tool evidence, labeled by GPT5.6Luna. This unified six-model extension is post-confirmatory and not human-validated. Its three instruction populations remain separate; lower false-success is better.","zh":"六个独立模型在baseline指令下分别重复回答相同100个合成工具失败情境，每模型200条回答，三个条件共3600条。GPT5.6Luna判定是否在缺乏成功工具证据时仍声称完成。这是原研究之后扩展的六模型分析，尚未经人工验证；三个指令条件分别记录，错误声称成功的比例越低越好。"},"source":{"name":"Benchmark authors","url":"https://github.com/junru-zhu/failure-transparent-agents","published_at":"2026-09-10","status":"dated_release"},"higher_is_better":false,"model_count":6,"result_count":6,"results":[[5107,0.34,370,11269,11270],[11271,0.295,370,11269,11270],[11272,0.265,370,11269,11270],[11273,0.155,370,11269,11270],[11274,0.06,370,11269,11270],[11275,0.25,370,11269,11270]]},{"id":"failure-transparent-transparency-2026-paper","name":"Failure-Transparent ·transparency (2026 author results)","group":"安全与可靠性","metric":"Judge-estimated false-success rate","unit":"%","public_history":true,"protocol":{"en":"Six distinct models face the same100 synthetic tool-failure scenarios twice under the transparency instruction condition, giving200 responses per model and3600 across all three conditions. False success is a success claim unsupported by failed tool evidence, labeled by GPT5.6Luna. This unified six-model extension is post-confirmatory and not human-validated. Its three instruction populations remain separate; lower false-success is better.","zh":"六个独立模型在transparency指令下分别重复回答相同100个合成工具失败情境，每模型200条回答，三个条件共3600条。GPT5.6Luna判定是否在缺乏成功工具证据时仍声称完成。这是原研究之后扩展的六模型分析，尚未经人工验证；三个指令条件分别记录，错误声称成功的比例越低越好。"},"source":{"name":"Benchmark authors","url":"https://github.com/junru-zhu/failure-transparent-agents","published_at":"2026-09-10","status":"dated_release"},"higher_is_better":false,"model_count":6,"result_count":6,"results":[[5107,0.03,370,11276,11270],[11271,0.195,370,11276,11270],[11272,0.205,370,11276,11270],[11273,0.065,370,11276,11270],[11274,0.005,370,11276,11270],[11275,0.055,370,11276,11270]]},{"id":"failure-transparent-evidence-contract-2026-paper","name":"Failure-Transparent ·evidence_contract (2026 author results)","group":"安全与可靠性","metric":"Judge-estimated false-success rate","unit":"%","public_history":true,"protocol":{"en":"Six distinct models face the same100 synthetic tool-failure scenarios twice under the evidence_contract instruction condition, giving200 responses per model and3600 across all three conditions. False success is a success claim unsupported by failed tool evidence, labeled by GPT5.6Luna. This unified six-model extension is post-confirmatory and not human-validated. Its three instruction populations remain separate; lower false-success is better.","zh":"六个独立模型在evidence_contract指令下分别重复回答相同100个合成工具失败情境，每模型200条回答，三个条件共3600条。GPT5.6Luna判定是否在缺乏成功工具证据时仍声称完成。这是原研究之后扩展的六模型分析，尚未经人工验证；三个指令条件分别记录，错误声称成功的比例越低越好。"},"source":{"name":"Benchmark authors","url":"https://github.com/junru-zhu/failure-transparent-agents","published_at":"2026-09-10","status":"dated_release"},"higher_is_better":false,"model_count":6,"result_count":6,"results":[[5107,0.0,370,11277,11270],[11271,0.0,370,11277,11270],[11272,0.02,370,11277,11270],[11273,0.005,370,11277,11270],[11274,0.005,370,11277,11270],[11275,0.02,370,11277,11270]]},{"id":"steam-sentiment-accuracy-2026-paper","name":"Steam Review Sentiment ·Accuracy (2026 author results)","group":"知识与推理","metric":"Held-out sentiment Accuracy","unit":"%","public_history":true,"protocol":{"en":"Six independent classifier architectures share1377 held-out Steam reviews (613 negative,764 positive), with4129 training and1377 validation rows at seed42. Empty NLTK-preprocessed reviews are removed:7002 sampled reviews become6883 before splitting. TF-IDF fits training data only; recurrent models and fine-tuned DistilBERT reuse identical row splits. The original rounded test table is retained. Macro-F1 remains native0–100 and is not percent-correct.","zh":"六种独立分类架构共享1377条留出Steam评论，其中负面613条、正面764条；训练4129条、验证1377条，划分种子42。NLTK预处理后过滤空文本，7002条抽样评论变为6883条再划分。TF-IDF只在训练集拟合，循环网络与微调DistilBERT复用相同样本划分。保留原表四舍五入成绩，宏F1保留原生0–100刻度。"},"source":{"name":"Benchmark authors","url":"https://github.com/qbalhkim/steam-review-sentiment","published_at":"2026-09-10","status":"dated_release"},"model_count":6,"result_count":6,"results":[[11278,0.8853,370,11279,11280],[10489,0.8504,370,11281,11280],[11282,0.8453,370,11281,11280],[11283,0.841,370,11281,11280],[11284,0.8423999999999999,370,11285,11280],[11286,0.8373,370,11287,11280]]},{"id":"steam-sentiment-macrof1-2026-paper","name":"Steam Review Sentiment ·Macro-F1 (2026 author results)","group":"知识与推理","metric":"Held-out sentiment Macro-F1","unit":"score","public_history":true,"protocol":{"en":"Six independent classifier architectures share1377 held-out Steam reviews (613 negative,764 positive), with4129 training and1377 validation rows at seed42. Empty NLTK-preprocessed reviews are removed:7002 sampled reviews become6883 before splitting. TF-IDF fits training data only; recurrent models and fine-tuned DistilBERT reuse identical row splits. The original rounded test table is retained. Macro-F1 remains native0–100 and is not percent-correct.","zh":"六种独立分类架构共享1377条留出Steam评论，其中负面613条、正面764条；训练4129条、验证1377条，划分种子42。NLTK预处理后过滤空文本，7002条抽样评论变为6883条再划分。TF-IDF只在训练集拟合，循环网络与微调DistilBERT复用相同样本划分。保留原表四舍五入成绩，宏F1保留原生0–100刻度。"},"source":{"name":"Benchmark authors","url":"https://github.com/qbalhkim/steam-review-sentiment","published_at":"2026-09-10","status":"dated_release"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Native score / 100","zh":"原生分数 / 100"},"score_decimals":2,"model_count":6,"result_count":6,"results":[[11278,88.4,370,11279,11280],[10489,84.73,370,11281,11280],[11282,84.18,370,11281,11280],[11283,83.88,370,11281,11280],[11284,83.84,370,11285,11280],[11286,83.36,370,11287,11280]]},{"id":"landuse-native-accuracy-macro-2026-paper","name":"Land-use Relevance ·Language-macro accuracy (2026 author results)","group":"知识与推理","metric":"Language-macro accuracy","unit":"%","public_history":true,"protocol":{"en":"The complete52-row panel includes43 distinct model checkpoints and their explicit runtime/scoring configurations, each evaluated on85 languages x300 labeled sentences. It classifies whether a sentence contains remotely observable land/environment information. Native fixed rules are used, with greedy English-prompt decoding or the declared classifier thresholds. Metrics are unweighted means of per-language metrics. Encoder350M15-language results and benchmark-tuned threshold upper bounds are excluded. Backend/DSpark/logprob variants do not count as independent models.","zh":"完整52条结果包含43个独立模型权重及其明确的运行或评分配置，每行均评测85种语言各300条已标注句子，判断内容是否涉及遥感可观测的土地或环境信息。使用原生固定判定规则，包括英文提示词下的贪心生成或声明的分类器阈值；指标对各语言成绩等权平均。排除Encoder350M的15语言结果和在测试标签上选阈值的上界。后端、DSpark与logprob变体不增加模型数。"},"source":{"name":"Benchmark authors","url":"https://huggingface.co/datasets/NoeFlandre/benchmark-llms-landuse-relevance","published_at":"2026-09-13","status":"dated_release"},"model_count":43,"result_count":52,"results":[[11288,0.7851,370,11289,11290],[11291,0.7629,370,11292,11290],[11291,0.7623,370,11293,11290],[11291,0.7602,370,11294,11290],[11295,0.7622,370,11296,11290],[11297,0.7472,370,11298,11290],[11299,0.7728,370,11300,11290],[11301,0.7553,370,11302,11290],[11303,0.6116,370,11304,11290],[11305,0.7047,370,11306,11290],[11305,0.6995,370,11307,11290],[11291,0.5353,370,11308,11290],[11309,0.5333,370,11310,11290],[11311,0.5333,370,11312,11290],[11313,0.5332,370,11314,11290],[11315,0.5327,370,11316,11290],[11317,0.5408,370,11318,11290],[11305,0.6936,370,11319,11290],[11320,0.5351,370,11321,11290],[11322,0.5412,370,11323,11290],[11324,0.6833,370,11325,11290],[11326,0.6688,370,11327,11290],[11328,0.6389,370,11329,11290],[11330,0.6038,370,11331,11290],[11330,0.6024,370,11332,11290],[11330,0.6024,370,11333,11290],[11334,0.6061,370,11335,11290],[11334,0.6059,370,11336,11290],[11334,0.6059,370,11337,11290],[11338,0.6026,370,11339,11290],[11340,0.6071,370,11341,11290],[11342,0.5909,370,11343,11290],[11344,0.5657,370,11345,11290],[11346,0.5863,370,11347,11290],[11348,0.5688,370,11349,11290],[11350,0.5613,370,11351,11290],[11352,0.5561,370,11353,11290],[11354,0.5276,370,11355,11290],[11356,0.5139,370,11357,11290],[11358,0.4959,370,11359,11290],[11360,0.4843,370,11361,11290],[11362,0.5111,370,11363,11290],[11364,0.5035,370,11365,11290],[11366,0.4893,370,11367,11290],[11368,0.4878,370,11369,11290],[11370,0.4761,370,11371,11290],[11372,0.474,370,11373,11290],[11374,0.4705,370,11375,11290],[11376,0.4673,370,11377,11290],[11378,0.4669,370,11379,11290],[11380,0.7064,370,11381,11290],[11382,0.6468,370,11383,11290]]},{"id":"landuse-native-balanced-accuracy-macro-2026-paper","name":"Land-use Relevance ·Language-macro balanced accuracy (2026 author results)","group":"知识与推理","metric":"Language-macro balanced accuracy","unit":"%","public_history":true,"protocol":{"en":"The complete52-row panel includes43 distinct model checkpoints and their explicit runtime/scoring configurations, each evaluated on85 languages x300 labeled sentences. It classifies whether a sentence contains remotely observable land/environment information. Native fixed rules are used, with greedy English-prompt decoding or the declared classifier thresholds. Metrics are unweighted means of per-language metrics. Encoder350M15-language results and benchmark-tuned threshold upper bounds are excluded. Backend/DSpark/logprob variants do not count as independent models.","zh":"完整52条结果包含43个独立模型权重及其明确的运行或评分配置，每行均评测85种语言各300条已标注句子，判断内容是否涉及遥感可观测的土地或环境信息。使用原生固定判定规则，包括英文提示词下的贪心生成或声明的分类器阈值；指标对各语言成绩等权平均。排除Encoder350M的15语言结果和在测试标签上选阈值的上界。后端、DSpark与logprob变体不增加模型数。"},"source":{"name":"Benchmark authors","url":"https://huggingface.co/datasets/NoeFlandre/benchmark-llms-landuse-relevance","published_at":"2026-09-13","status":"dated_release"},"model_count":43,"result_count":52,"results":[[11288,0.7775,370,11289,11290],[11291,0.773,370,11292,11290],[11291,0.7744,370,11293,11290],[11291,0.7536,370,11294,11290],[11295,0.755,370,11296,11290],[11297,0.745,370,11298,11290],[11299,0.78,370,11300,11290],[11301,0.7606,370,11302,11290],[11303,0.5882,370,11304,11290],[11305,0.7056,370,11306,11290],[11305,0.7047,370,11307,11290],[11291,0.5022,370,11308,11290],[11309,0.5,370,11310,11290],[11311,0.5,370,11312,11290],[11313,0.5001,370,11314,11290],[11315,0.4994,370,11316,11290],[11317,0.5094,370,11318,11290],[11305,0.6999,370,11319,11290],[11320,0.5049,370,11321,11290],[11322,0.5201,370,11323,11290],[11324,0.6999,370,11325,11290],[11326,0.6869,370,11327,11290],[11328,0.6577,370,11329,11290],[11330,0.6158,370,11331,11290],[11330,0.6148,370,11332,11290],[11330,0.6148,370,11333,11290],[11334,0.6239,370,11335,11290],[11334,0.6237,370,11336,11290],[11334,0.6237,370,11337,11290],[11338,0.6221,370,11339,11290],[11340,0.6277,370,11341,11290],[11342,0.6099,370,11343,11290],[11344,0.5846,370,11345,11290],[11346,0.6113,370,11347,11290],[11348,0.5929,370,11349,11290],[11350,0.5864,370,11351,11290],[11352,0.5815,370,11353,11290],[11354,0.5564,370,11355,11290],[11356,0.5438,370,11357,11290],[11358,0.5247,370,11359,11290],[11360,0.5121,370,11361,11290],[11362,0.5415,370,11363,11290],[11364,0.5341,370,11365,11290],[11366,0.5209,370,11367,11290],[11368,0.5202,370,11369,11290],[11370,0.5089,370,11371,11290],[11372,0.5066,370,11373,11290],[11374,0.5035,370,11375,11290],[11376,0.5006,370,11377,11290],[11378,0.5002,370,11379,11290],[11380,0.7134,370,11381,11290],[11382,0.6561,370,11383,11290]]},{"id":"landuse-native-f1-macro-2026-paper","name":"Land-use Relevance ·Language-macro F1 (2026 author results)","group":"知识与推理","metric":"Language-macro F1","unit":"score","public_history":true,"protocol":{"en":"The complete52-row panel includes43 distinct model checkpoints and their explicit runtime/scoring configurations, each evaluated on85 languages x300 labeled sentences. It classifies whether a sentence contains remotely observable land/environment information. Native fixed rules are used, with greedy English-prompt decoding or the declared classifier thresholds. Metrics are unweighted means of per-language metrics. Encoder350M15-language results and benchmark-tuned threshold upper bounds are excluded. Backend/DSpark/logprob variants do not count as independent models.","zh":"完整52条结果包含43个独立模型权重及其明确的运行或评分配置，每行均评测85种语言各300条已标注句子，判断内容是否涉及遥感可观测的土地或环境信息。使用原生固定判定规则，包括英文提示词下的贪心生成或声明的分类器阈值；指标对各语言成绩等权平均。排除Encoder350M的15语言结果和在测试标签上选阈值的上界。后端、DSpark与logprob变体不增加模型数。"},"source":{"name":"Benchmark authors","url":"https://huggingface.co/datasets/NoeFlandre/benchmark-llms-landuse-relevance","published_at":"2026-09-13","status":"dated_release"},"score_max":1,"saturation_excluded":true,"scale_label":{"en":"Native score / 1","zh":"原生分数 / 1"},"score_decimals":4,"model_count":43,"result_count":52,"results":[[11288,0.8151,370,11289,11290],[11291,0.8106,370,11292,11290],[11291,0.8104,370,11293,11290],[11291,0.8029,370,11294,11290],[11295,0.7905,370,11296,11290],[11297,0.7634,370,11298,11290],[11299,0.7509,370,11300,11290],[11301,0.7442,370,11302,11290],[11303,0.7217,370,11304,11290],[11305,0.7179,370,11306,11290],[11305,0.6992,370,11307,11290],[11291,0.6965,370,11308,11290],[11309,0.6957,370,11310,11290],[11311,0.6957,370,11312,11290],[11313,0.6951,370,11314,11290],[11315,0.695,370,11316,11290],[11317,0.6949,370,11318,11290],[11305,0.6915,370,11319,11290],[11320,0.6866,370,11321,11290],[11322,0.6571,370,11323,11290],[11324,0.5936,370,11325,11290],[11326,0.5625,370,11327,11290],[11328,0.5209,370,11329,11290],[11330,0.516,370,11331,11290],[11330,0.5095,370,11332,11290],[11330,0.5095,370,11333,11290],[11334,0.4896,370,11335,11290],[11334,0.4889,370,11336,11290],[11334,0.4889,370,11337,11290],[11338,0.4598,370,11339,11290],[11340,0.4551,370,11341,11290],[11342,0.4432,370,11343,11290],[11344,0.4061,370,11345,11290],[11346,0.3664,370,11347,11290],[11348,0.3469,370,11349,11290],[11350,0.3306,370,11351,11290],[11352,0.3173,370,11353,11290],[11354,0.2125,370,11355,11290],[11356,0.1657,370,11357,11290],[11358,0.1627,370,11359,11290],[11360,0.1599,370,11361,11290],[11362,0.1547,370,11363,11290],[11364,0.1324,370,11365,11290],[11366,0.0843,370,11367,11290],[11368,0.08,370,11369,11290],[11370,0.0346,370,11371,11290],[11372,0.0337,370,11373,11290],[11374,0.0191,370,11375,11290],[11376,0.0022,370,11377,11290],[11378,0.0013,370,11379,11290],[11380,0.6773,370,11381,11290],[11382,0.608,370,11383,11290]]},{"id":"vidhalloc-factual-2026-paper","name":"VidHalLoc ·factual (2026 paper)","group":"视觉与视频","metric":"Factual acceptance accuracy","unit":"%","public_history":true,"protocol":{"en":"All15 methods assess the same800 held-out paired factual/counterfactual instances, sampled100 per category from the2000-example VidHalLoc corpus and covering436 unique videos. Joint accuracy requires accepting the factual answer and rejecting its paired hallucination. Four dedicated detector thresholds are selected on a disjoint100-instance/100-video calibration set and frozen before evaluation. Original video inputs/frames and agent scaffolds remain explicit. The panel includes eight distinct standalone VLMs in addition to agents/detectors.","zh":"全部15种方法判断共同800组事实与反事实配对样本，来自2000条语料中的八类各100条，覆盖436个独立视频。联合正确要求同时接受事实回答并拒绝配对幻觉。四个专用检测器在与评测无视频重叠的100条、100视频校准集上选阈值，随后固定。保留各方法原视频输入、采帧及Agent配置；除Agent和检测器外，面板本身已有八个独立视觉语言模型。"},"source":{"name":"Benchmark authors","url":"https://arxiv.org/abs/2609.09895v1","published_at":"2026-09-09","status":"paper_baselines"},"model_count":15,"result_count":15,"results":[[6133,0.9825,370,11384,11385],[2058,0.8362999999999999,370,11386,11385],[11387,0.7238,370,11388,11385],[11389,0.8787999999999999,370,11390,11385],[11391,0.5588000000000001,370,11392,11385],[11393,0.9087999999999999,370,11394,11385],[3763,0.955,370,11395,11385],[11396,0.9813,370,11397,11385],[11398,0.3663,370,11399,11385],[11400,0.8075,370,11401,11385],[11402,0.8837999999999999,370,11403,11385],[11404,0.4775,370,11405,11385],[11406,0.6663,370,11407,11385],[11408,0.675,370,11409,11385],[11410,0.4425,370,11411,11385]]},{"id":"vidhalloc-counterfactual-2026-paper","name":"VidHalLoc ·counterfactual (2026 paper)","group":"视觉与视频","metric":"Counterfactual rejection accuracy","unit":"%","public_history":true,"protocol":{"en":"All15 methods assess the same800 held-out paired factual/counterfactual instances, sampled100 per category from the2000-example VidHalLoc corpus and covering436 unique videos. Joint accuracy requires accepting the factual answer and rejecting its paired hallucination. Four dedicated detector thresholds are selected on a disjoint100-instance/100-video calibration set and frozen before evaluation. Original video inputs/frames and agent scaffolds remain explicit. The panel includes eight distinct standalone VLMs in addition to agents/detectors.","zh":"全部15种方法判断共同800组事实与反事实配对样本，来自2000条语料中的八类各100条，覆盖436个独立视频。联合正确要求同时接受事实回答并拒绝配对幻觉。四个专用检测器在与评测无视频重叠的100条、100视频校准集上选阈值，随后固定。保留各方法原视频输入、采帧及Agent配置；除Agent和检测器外，面板本身已有八个独立视觉语言模型。"},"source":{"name":"Benchmark authors","url":"https://arxiv.org/abs/2609.09895v1","published_at":"2026-09-09","status":"paper_baselines"},"model_count":15,"result_count":15,"results":[[6133,0.8462999999999999,370,11384,11385],[2058,0.9275,370,11386,11385],[11387,0.2563,370,11388,11385],[11389,0.8338,370,11390,11385],[11391,0.9488,370,11392,11385],[11393,0.7212999999999999,370,11394,11385],[3763,0.845,370,11395,11385],[11396,0.48,370,11397,11385],[11398,0.8975,370,11399,11385],[11400,0.5313,370,11401,11385],[11402,0.81,370,11403,11385],[11404,0.5688,370,11405,11385],[11406,0.3713,370,11407,11385],[11408,0.5925,370,11409,11385],[11410,0.8887999999999999,370,11411,11385]]},{"id":"vidhalloc-joint-2026-paper","name":"VidHalLoc ·joint (2026 paper)","group":"视觉与视频","metric":"Joint factual-and-counterfactual accuracy","unit":"%","public_history":true,"protocol":{"en":"All15 methods assess the same800 held-out paired factual/counterfactual instances, sampled100 per category from the2000-example VidHalLoc corpus and covering436 unique videos. Joint accuracy requires accepting the factual answer and rejecting its paired hallucination. Four dedicated detector thresholds are selected on a disjoint100-instance/100-video calibration set and frozen before evaluation. Original video inputs/frames and agent scaffolds remain explicit. The panel includes eight distinct standalone VLMs in addition to agents/detectors.","zh":"全部15种方法判断共同800组事实与反事实配对样本，来自2000条语料中的八类各100条，覆盖436个独立视频。联合正确要求同时接受事实回答并拒绝配对幻觉。四个专用检测器在与评测无视频重叠的100条、100视频校准集上选阈值，随后固定。保留各方法原视频输入、采帧及Agent配置；除Agent和检测器外，面板本身已有八个独立视觉语言模型。"},"source":{"name":"Benchmark authors","url":"https://arxiv.org/abs/2609.09895v1","published_at":"2026-09-09","status":"paper_baselines"},"model_count":15,"result_count":15,"results":[[6133,0.8362999999999999,370,11384,11385],[2058,0.795,370,11386,11385],[11387,0.0213,370,11388,11385],[11389,0.73,370,11390,11385],[11391,0.5263,370,11392,11385],[11393,0.6487999999999999,370,11394,11385],[3763,0.81,370,11395,11385],[11396,0.4675,370,11397,11385],[11398,0.315,370,11399,11385],[11400,0.3975,370,11401,11385],[11402,0.715,370,11403,11385],[11404,0.0725,370,11405,11385],[11406,0.065,370,11407,11385],[11408,0.33130000000000004,370,11409,11385],[11410,0.34630000000000005,370,11411,11385]]},{"id":"liberoctrl-camera-l1-2026-paper","name":"LIBERO-CTRL ·camera L1 (2026 paper)","group":"Agent 与工具","metric":"Perturbed rollout success rate","unit":"%","public_history":true,"protocol":{"en":"Six distinct policies share the same400 camera/L1 rollouts: four LIBERO suites, ten tasks per suite, ten perturbation configurations per task, normalized severity radius2. Exact task/initial-state/configuration IDs match across policies; success counts independently reproduce all author summary cells. MINERVA is excluded because language and simultaneous inputs are identity controls for its lookup policy. Nominal2000-rollout LIBERO scores are not mixed into the new perturbation benchmark.","zh":"六个独立策略共享camera/L1下400次扰动测试：四个LIBERO套件、每套件十任务、每任务十扰动配置，标准化严重度半径为2。已逐项确认各策略任务、初态及配置编号一致，独立重算成功次数并核对作者汇总。MINERVA采用指令查表，语言和同步扰动对其含有身份控制，因此排除。未将普通LIBERO的2000次无扰动成绩混入新基准。"},"source":{"name":"Benchmark authors","url":"https://arxiv.org/abs/2609.15940v1","published_at":"2026-09-14","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[8323,0.915,370,11412,11413],[7594,0.94,370,11414,11413],[7598,0.6375,370,11415,11413],[9600,0.6375,370,11416,11413],[11417,0.9525,370,11418,11413],[11419,0.73,370,11420,11413]]},{"id":"liberoctrl-camera-l2-2026-paper","name":"LIBERO-CTRL ·camera L2 (2026 paper)","group":"Agent 与工具","metric":"Perturbed rollout success rate","unit":"%","public_history":true,"protocol":{"en":"Six distinct policies share the same400 camera/L2 rollouts: four LIBERO suites, ten tasks per suite, ten perturbation configurations per task, normalized severity radius4. Exact task/initial-state/configuration IDs match across policies; success counts independently reproduce all author summary cells. MINERVA is excluded because language and simultaneous inputs are identity controls for its lookup policy. Nominal2000-rollout LIBERO scores are not mixed into the new perturbation benchmark.","zh":"六个独立策略共享camera/L2下400次扰动测试：四个LIBERO套件、每套件十任务、每任务十扰动配置，标准化严重度半径为4。已逐项确认各策略任务、初态及配置编号一致，独立重算成功次数并核对作者汇总。MINERVA采用指令查表，语言和同步扰动对其含有身份控制，因此排除。未将普通LIBERO的2000次无扰动成绩混入新基准。"},"source":{"name":"Benchmark authors","url":"https://arxiv.org/abs/2609.15940v1","published_at":"2026-09-14","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[8323,0.8075,370,11421,11413],[7594,0.875,370,11422,11413],[7598,0.24,370,11423,11413],[9600,0.4475,370,11424,11413],[11417,0.855,370,11425,11413],[11419,0.6375,370,11426,11413]]},{"id":"liberoctrl-camera-l3-2026-paper","name":"LIBERO-CTRL ·camera L3 (2026 paper)","group":"Agent 与工具","metric":"Perturbed rollout success rate","unit":"%","public_history":true,"protocol":{"en":"Six distinct policies share the same400 camera/L3 rollouts: four LIBERO suites, ten tasks per suite, ten perturbation configurations per task, normalized severity radius8. Exact task/initial-state/configuration IDs match across policies; success counts independently reproduce all author summary cells. MINERVA is excluded because language and simultaneous inputs are identity controls for its lookup policy. Nominal2000-rollout LIBERO scores are not mixed into the new perturbation benchmark.","zh":"六个独立策略共享camera/L3下400次扰动测试：四个LIBERO套件、每套件十任务、每任务十扰动配置，标准化严重度半径为8。已逐项确认各策略任务、初态及配置编号一致，独立重算成功次数并核对作者汇总。MINERVA采用指令查表，语言和同步扰动对其含有身份控制，因此排除。未将普通LIBERO的2000次无扰动成绩混入新基准。"},"source":{"name":"Benchmark authors","url":"https://arxiv.org/abs/2609.15940v1","published_at":"2026-09-14","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[8323,0.56,370,11427,11413],[7594,0.6975,370,11428,11413],[7598,0.0375,370,11429,11413],[9600,0.2475,370,11430,11413],[11417,0.6675,370,11431,11413],[11419,0.63,370,11432,11413]]},{"id":"liberoctrl-lighting-l1-2026-paper","name":"LIBERO-CTRL ·lighting L1 (2026 paper)","group":"Agent 与工具","metric":"Perturbed rollout success rate","unit":"%","public_history":true,"protocol":{"en":"Six distinct policies share the same400 lighting/L1 rollouts: four LIBERO suites, ten tasks per suite, ten perturbation configurations per task, normalized severity radius2. Exact task/initial-state/configuration IDs match across policies; success counts independently reproduce all author summary cells. MINERVA is excluded because language and simultaneous inputs are identity controls for its lookup policy. Nominal2000-rollout LIBERO scores are not mixed into the new perturbation benchmark.","zh":"六个独立策略共享lighting/L1下400次扰动测试：四个LIBERO套件、每套件十任务、每任务十扰动配置，标准化严重度半径为2。已逐项确认各策略任务、初态及配置编号一致，独立重算成功次数并核对作者汇总。MINERVA采用指令查表，语言和同步扰动对其含有身份控制，因此排除。未将普通LIBERO的2000次无扰动成绩混入新基准。"},"source":{"name":"Benchmark authors","url":"https://arxiv.org/abs/2609.15940v1","published_at":"2026-09-14","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[8323,0.9825,370,11433,11413],[7594,0.9625,370,11434,11413],[7598,0.9325,370,11435,11413],[9600,0.7525,370,11436,11413],[11417,0.9825,370,11437,11413],[11419,0.7825,370,11438,11413]]},{"id":"liberoctrl-lighting-l2-2026-paper","name":"LIBERO-CTRL ·lighting L2 (2026 paper)","group":"Agent 与工具","metric":"Perturbed rollout success rate","unit":"%","public_history":true,"protocol":{"en":"Six distinct policies share the same400 lighting/L2 rollouts: four LIBERO suites, ten tasks per suite, ten perturbation configurations per task, normalized severity radius4. Exact task/initial-state/configuration IDs match across policies; success counts independently reproduce all author summary cells. MINERVA is excluded because language and simultaneous inputs are identity controls for its lookup policy. Nominal2000-rollout LIBERO scores are not mixed into the new perturbation benchmark.","zh":"六个独立策略共享lighting/L2下400次扰动测试：四个LIBERO套件、每套件十任务、每任务十扰动配置，标准化严重度半径为4。已逐项确认各策略任务、初态及配置编号一致，独立重算成功次数并核对作者汇总。MINERVA采用指令查表，语言和同步扰动对其含有身份控制，因此排除。未将普通LIBERO的2000次无扰动成绩混入新基准。"},"source":{"name":"Benchmark authors","url":"https://arxiv.org/abs/2609.15940v1","published_at":"2026-09-14","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[8323,0.98,370,11439,11413],[7594,0.9525,370,11440,11413],[7598,0.9225,370,11441,11413],[9600,0.735,370,11442,11413],[11417,0.9825,370,11443,11413],[11419,0.775,370,11444,11413]]},{"id":"liberoctrl-lighting-l3-2026-paper","name":"LIBERO-CTRL ·lighting L3 (2026 paper)","group":"Agent 与工具","metric":"Perturbed rollout success rate","unit":"%","public_history":true,"protocol":{"en":"Six distinct policies share the same400 lighting/L3 rollouts: four LIBERO suites, ten tasks per suite, ten perturbation configurations per task, normalized severity radius8. Exact task/initial-state/configuration IDs match across policies; success counts independently reproduce all author summary cells. MINERVA is excluded because language and simultaneous inputs are identity controls for its lookup policy. Nominal2000-rollout LIBERO scores are not mixed into the new perturbation benchmark.","zh":"六个独立策略共享lighting/L3下400次扰动测试：四个LIBERO套件、每套件十任务、每任务十扰动配置，标准化严重度半径为8。已逐项确认各策略任务、初态及配置编号一致，独立重算成功次数并核对作者汇总。MINERVA采用指令查表，语言和同步扰动对其含有身份控制，因此排除。未将普通LIBERO的2000次无扰动成绩混入新基准。"},"source":{"name":"Benchmark authors","url":"https://arxiv.org/abs/2609.15940v1","published_at":"2026-09-14","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[8323,0.955,370,11445,11413],[7594,0.9525,370,11446,11413],[7598,0.8875,370,11447,11413],[9600,0.7075,370,11448,11413],[11417,0.9525,370,11449,11413],[11419,0.7675,370,11450,11413]]},{"id":"liberoctrl-robot-l1-2026-paper","name":"LIBERO-CTRL ·robot L1 (2026 paper)","group":"Agent 与工具","metric":"Perturbed rollout success rate","unit":"%","public_history":true,"protocol":{"en":"Six distinct policies share the same400 robot/L1 rollouts: four LIBERO suites, ten tasks per suite, ten perturbation configurations per task, normalized severity radius2. Exact task/initial-state/configuration IDs match across policies; success counts independently reproduce all author summary cells. MINERVA is excluded because language and simultaneous inputs are identity controls for its lookup policy. Nominal2000-rollout LIBERO scores are not mixed into the new perturbation benchmark.","zh":"六个独立策略共享robot/L1下400次扰动测试：四个LIBERO套件、每套件十任务、每任务十扰动配置，标准化严重度半径为2。已逐项确认各策略任务、初态及配置编号一致，独立重算成功次数并核对作者汇总。MINERVA采用指令查表，语言和同步扰动对其含有身份控制，因此排除。未将普通LIBERO的2000次无扰动成绩混入新基准。"},"source":{"name":"Benchmark authors","url":"https://arxiv.org/abs/2609.15940v1","published_at":"2026-09-14","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[8323,0.94,370,11451,11413],[7594,0.8975,370,11452,11413],[7598,0.905,370,11453,11413],[9600,0.6325,370,11454,11413],[11417,0.9825,370,11455,11413],[11419,0.73,370,11456,11413]]},{"id":"liberoctrl-robot-l2-2026-paper","name":"LIBERO-CTRL ·robot L2 (2026 paper)","group":"Agent 与工具","metric":"Perturbed rollout success rate","unit":"%","public_history":true,"protocol":{"en":"Six distinct policies share the same400 robot/L2 rollouts: four LIBERO suites, ten tasks per suite, ten perturbation configurations per task, normalized severity radius4. Exact task/initial-state/configuration IDs match across policies; success counts independently reproduce all author summary cells. MINERVA is excluded because language and simultaneous inputs are identity controls for its lookup policy. Nominal2000-rollout LIBERO scores are not mixed into the new perturbation benchmark.","zh":"六个独立策略共享robot/L2下400次扰动测试：四个LIBERO套件、每套件十任务、每任务十扰动配置，标准化严重度半径为4。已逐项确认各策略任务、初态及配置编号一致，独立重算成功次数并核对作者汇总。MINERVA采用指令查表，语言和同步扰动对其含有身份控制，因此排除。未将普通LIBERO的2000次无扰动成绩混入新基准。"},"source":{"name":"Benchmark authors","url":"https://arxiv.org/abs/2609.15940v1","published_at":"2026-09-14","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[8323,0.875,370,11457,11413],[7594,0.7525,370,11458,11413],[7598,0.8175,370,11459,11413],[9600,0.46,370,11460,11413],[11417,0.925,370,11461,11413],[11419,0.5275,370,11462,11413]]},{"id":"liberoctrl-robot-l3-2026-paper","name":"LIBERO-CTRL ·robot L3 (2026 paper)","group":"Agent 与工具","metric":"Perturbed rollout success rate","unit":"%","public_history":true,"protocol":{"en":"Six distinct policies share the same400 robot/L3 rollouts: four LIBERO suites, ten tasks per suite, ten perturbation configurations per task, normalized severity radius8. Exact task/initial-state/configuration IDs match across policies; success counts independently reproduce all author summary cells. MINERVA is excluded because language and simultaneous inputs are identity controls for its lookup policy. Nominal2000-rollout LIBERO scores are not mixed into the new perturbation benchmark.","zh":"六个独立策略共享robot/L3下400次扰动测试：四个LIBERO套件、每套件十任务、每任务十扰动配置，标准化严重度半径为8。已逐项确认各策略任务、初态及配置编号一致，独立重算成功次数并核对作者汇总。MINERVA采用指令查表，语言和同步扰动对其含有身份控制，因此排除。未将普通LIBERO的2000次无扰动成绩混入新基准。"},"source":{"name":"Benchmark authors","url":"https://arxiv.org/abs/2609.15940v1","published_at":"2026-09-14","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[8323,0.6675,370,11463,11413],[7594,0.4475,370,11464,11413],[7598,0.525,370,11465,11413],[9600,0.2475,370,11466,11413],[11417,0.7275,370,11467,11413],[11419,0.2575,370,11468,11413]]},{"id":"liberoctrl-sensor-l1-2026-paper","name":"LIBERO-CTRL ·sensor L1 (2026 paper)","group":"Agent 与工具","metric":"Perturbed rollout success rate","unit":"%","public_history":true,"protocol":{"en":"Six distinct policies share the same400 sensor/L1 rollouts: four LIBERO suites, ten tasks per suite, ten perturbation configurations per task, normalized severity radius2. Exact task/initial-state/configuration IDs match across policies; success counts independently reproduce all author summary cells. MINERVA is excluded because language and simultaneous inputs are identity controls for its lookup policy. Nominal2000-rollout LIBERO scores are not mixed into the new perturbation benchmark.","zh":"六个独立策略共享sensor/L1下400次扰动测试：四个LIBERO套件、每套件十任务、每任务十扰动配置，标准化严重度半径为2。已逐项确认各策略任务、初态及配置编号一致，独立重算成功次数并核对作者汇总。MINERVA采用指令查表，语言和同步扰动对其含有身份控制，因此排除。未将普通LIBERO的2000次无扰动成绩混入新基准。"},"source":{"name":"Benchmark authors","url":"https://arxiv.org/abs/2609.15940v1","published_at":"2026-09-14","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[8323,0.9675,370,11469,11413],[7594,0.975,370,11470,11413],[7598,0.935,370,11471,11413],[9600,0.76,370,11472,11413],[11417,0.95,370,11473,11413],[11419,0.7925,370,11474,11413]]},{"id":"liberoctrl-sensor-l2-2026-paper","name":"LIBERO-CTRL ·sensor L2 (2026 paper)","group":"Agent 与工具","metric":"Perturbed rollout success rate","unit":"%","public_history":true,"protocol":{"en":"Six distinct policies share the same400 sensor/L2 rollouts: four LIBERO suites, ten tasks per suite, ten perturbation configurations per task, normalized severity radius4. Exact task/initial-state/configuration IDs match across policies; success counts independently reproduce all author summary cells. MINERVA is excluded because language and simultaneous inputs are identity controls for its lookup policy. Nominal2000-rollout LIBERO scores are not mixed into the new perturbation benchmark.","zh":"六个独立策略共享sensor/L2下400次扰动测试：四个LIBERO套件、每套件十任务、每任务十扰动配置，标准化严重度半径为4。已逐项确认各策略任务、初态及配置编号一致，独立重算成功次数并核对作者汇总。MINERVA采用指令查表，语言和同步扰动对其含有身份控制，因此排除。未将普通LIBERO的2000次无扰动成绩混入新基准。"},"source":{"name":"Benchmark authors","url":"https://arxiv.org/abs/2609.15940v1","published_at":"2026-09-14","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[8323,0.97,370,11475,11413],[7594,0.97,370,11476,11413],[7598,0.795,370,11477,11413],[9600,0.755,370,11478,11413],[11417,0.72,370,11479,11413],[11419,0.78,370,11480,11413]]},{"id":"liberoctrl-sensor-l3-2026-paper","name":"LIBERO-CTRL ·sensor L3 (2026 paper)","group":"Agent 与工具","metric":"Perturbed rollout success rate","unit":"%","public_history":true,"protocol":{"en":"Six distinct policies share the same400 sensor/L3 rollouts: four LIBERO suites, ten tasks per suite, ten perturbation configurations per task, normalized severity radius8. Exact task/initial-state/configuration IDs match across policies; success counts independently reproduce all author summary cells. MINERVA is excluded because language and simultaneous inputs are identity controls for its lookup policy. Nominal2000-rollout LIBERO scores are not mixed into the new perturbation benchmark.","zh":"六个独立策略共享sensor/L3下400次扰动测试：四个LIBERO套件、每套件十任务、每任务十扰动配置，标准化严重度半径为8。已逐项确认各策略任务、初态及配置编号一致，独立重算成功次数并核对作者汇总。MINERVA采用指令查表，语言和同步扰动对其含有身份控制，因此排除。未将普通LIBERO的2000次无扰动成绩混入新基准。"},"source":{"name":"Benchmark authors","url":"https://arxiv.org/abs/2609.15940v1","published_at":"2026-09-14","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[8323,0.875,370,11481,11413],[7594,0.8725,370,11482,11413],[7598,0.2,370,11483,11413],[9600,0.435,370,11484,11413],[11417,0.365,370,11485,11413],[11419,0.6725,370,11486,11413]]},{"id":"liberoctrl-actuation-l1-2026-paper","name":"LIBERO-CTRL ·actuation L1 (2026 paper)","group":"Agent 与工具","metric":"Perturbed rollout success rate","unit":"%","public_history":true,"protocol":{"en":"Six distinct policies share the same400 actuation/L1 rollouts: four LIBERO suites, ten tasks per suite, ten perturbation configurations per task, normalized severity radius2. Exact task/initial-state/configuration IDs match across policies; success counts independently reproduce all author summary cells. MINERVA is excluded because language and simultaneous inputs are identity controls for its lookup policy. Nominal2000-rollout LIBERO scores are not mixed into the new perturbation benchmark.","zh":"六个独立策略共享actuation/L1下400次扰动测试：四个LIBERO套件、每套件十任务、每任务十扰动配置，标准化严重度半径为2。已逐项确认各策略任务、初态及配置编号一致，独立重算成功次数并核对作者汇总。MINERVA采用指令查表，语言和同步扰动对其含有身份控制，因此排除。未将普通LIBERO的2000次无扰动成绩混入新基准。"},"source":{"name":"Benchmark authors","url":"https://arxiv.org/abs/2609.15940v1","published_at":"2026-09-14","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[8323,0.9825,370,11487,11413],[7594,0.9625,370,11488,11413],[7598,0.9575,370,11489,11413],[9600,0.74,370,11490,11413],[11417,0.985,370,11491,11413],[11419,0.8,370,11492,11413]]},{"id":"liberoctrl-actuation-l2-2026-paper","name":"LIBERO-CTRL ·actuation L2 (2026 paper)","group":"Agent 与工具","metric":"Perturbed rollout success rate","unit":"%","public_history":true,"protocol":{"en":"Six distinct policies share the same400 actuation/L2 rollouts: four LIBERO suites, ten tasks per suite, ten perturbation configurations per task, normalized severity radius4. Exact task/initial-state/configuration IDs match across policies; success counts independently reproduce all author summary cells. MINERVA is excluded because language and simultaneous inputs are identity controls for its lookup policy. Nominal2000-rollout LIBERO scores are not mixed into the new perturbation benchmark.","zh":"六个独立策略共享actuation/L2下400次扰动测试：四个LIBERO套件、每套件十任务、每任务十扰动配置，标准化严重度半径为4。已逐项确认各策略任务、初态及配置编号一致，独立重算成功次数并核对作者汇总。MINERVA采用指令查表，语言和同步扰动对其含有身份控制，因此排除。未将普通LIBERO的2000次无扰动成绩混入新基准。"},"source":{"name":"Benchmark authors","url":"https://arxiv.org/abs/2609.15940v1","published_at":"2026-09-14","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[8323,0.9625,370,11493,11413],[7594,0.9525,370,11494,11413],[7598,0.93,370,11495,11413],[9600,0.7225,370,11496,11413],[11417,0.96,370,11497,11413],[11419,0.7525,370,11498,11413]]},{"id":"liberoctrl-actuation-l3-2026-paper","name":"LIBERO-CTRL ·actuation L3 (2026 paper)","group":"Agent 与工具","metric":"Perturbed rollout success rate","unit":"%","public_history":true,"protocol":{"en":"Six distinct policies share the same400 actuation/L3 rollouts: four LIBERO suites, ten tasks per suite, ten perturbation configurations per task, normalized severity radius8. Exact task/initial-state/configuration IDs match across policies; success counts independently reproduce all author summary cells. MINERVA is excluded because language and simultaneous inputs are identity controls for its lookup policy. Nominal2000-rollout LIBERO scores are not mixed into the new perturbation benchmark.","zh":"六个独立策略共享actuation/L3下400次扰动测试：四个LIBERO套件、每套件十任务、每任务十扰动配置，标准化严重度半径为8。已逐项确认各策略任务、初态及配置编号一致，独立重算成功次数并核对作者汇总。MINERVA采用指令查表，语言和同步扰动对其含有身份控制，因此排除。未将普通LIBERO的2000次无扰动成绩混入新基准。"},"source":{"name":"Benchmark authors","url":"https://arxiv.org/abs/2609.15940v1","published_at":"2026-09-14","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[8323,0.915,370,11499,11413],[7594,0.92,370,11500,11413],[7598,0.86,370,11501,11413],[9600,0.6275,370,11502,11413],[11417,0.9225,370,11503,11413],[11419,0.6275,370,11504,11413]]},{"id":"liberoctrl-language-l1-2026-paper","name":"LIBERO-CTRL ·language L1 (2026 paper)","group":"Agent 与工具","metric":"Perturbed rollout success rate","unit":"%","public_history":true,"protocol":{"en":"Six distinct policies share the same400 language/L1 rollouts: four LIBERO suites, ten tasks per suite, ten perturbation configurations per task, normalized severity radius2. Exact task/initial-state/configuration IDs match across policies; success counts independently reproduce all author summary cells. MINERVA is excluded because language and simultaneous inputs are identity controls for its lookup policy. Nominal2000-rollout LIBERO scores are not mixed into the new perturbation benchmark.","zh":"六个独立策略共享language/L1下400次扰动测试：四个LIBERO套件、每套件十任务、每任务十扰动配置，标准化严重度半径为2。已逐项确认各策略任务、初态及配置编号一致，独立重算成功次数并核对作者汇总。MINERVA采用指令查表，语言和同步扰动对其含有身份控制，因此排除。未将普通LIBERO的2000次无扰动成绩混入新基准。"},"source":{"name":"Benchmark authors","url":"https://arxiv.org/abs/2609.15940v1","published_at":"2026-09-14","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[8323,0.8625,370,11505,11413],[7594,0.9075,370,11506,11413],[7598,0.875,370,11507,11413],[9600,0.3875,370,11508,11413],[11417,0.96,370,11509,11413],[11419,0.0175,370,11510,11413]]},{"id":"liberoctrl-language-l2-2026-paper","name":"LIBERO-CTRL ·language L2 (2026 paper)","group":"Agent 与工具","metric":"Perturbed rollout success rate","unit":"%","public_history":true,"protocol":{"en":"Six distinct policies share the same400 language/L2 rollouts: four LIBERO suites, ten tasks per suite, ten perturbation configurations per task, normalized severity radius4. Exact task/initial-state/configuration IDs match across policies; success counts independently reproduce all author summary cells. MINERVA is excluded because language and simultaneous inputs are identity controls for its lookup policy. Nominal2000-rollout LIBERO scores are not mixed into the new perturbation benchmark.","zh":"六个独立策略共享language/L2下400次扰动测试：四个LIBERO套件、每套件十任务、每任务十扰动配置，标准化严重度半径为4。已逐项确认各策略任务、初态及配置编号一致，独立重算成功次数并核对作者汇总。MINERVA采用指令查表，语言和同步扰动对其含有身份控制，因此排除。未将普通LIBERO的2000次无扰动成绩混入新基准。"},"source":{"name":"Benchmark authors","url":"https://arxiv.org/abs/2609.15940v1","published_at":"2026-09-14","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[8323,0.7975,370,11511,11413],[7594,0.9,370,11512,11413],[7598,0.865,370,11513,11413],[9600,0.2875,370,11514,11413],[11417,0.9475,370,11515,11413],[11419,0.0,370,11516,11413]]},{"id":"liberoctrl-language-l3-2026-paper","name":"LIBERO-CTRL ·language L3 (2026 paper)","group":"Agent 与工具","metric":"Perturbed rollout success rate","unit":"%","public_history":true,"protocol":{"en":"Six distinct policies share the same400 language/L3 rollouts: four LIBERO suites, ten tasks per suite, ten perturbation configurations per task, normalized severity radius8. Exact task/initial-state/configuration IDs match across policies; success counts independently reproduce all author summary cells. MINERVA is excluded because language and simultaneous inputs are identity controls for its lookup policy. Nominal2000-rollout LIBERO scores are not mixed into the new perturbation benchmark.","zh":"六个独立策略共享language/L3下400次扰动测试：四个LIBERO套件、每套件十任务、每任务十扰动配置，标准化严重度半径为8。已逐项确认各策略任务、初态及配置编号一致，独立重算成功次数并核对作者汇总。MINERVA采用指令查表，语言和同步扰动对其含有身份控制，因此排除。未将普通LIBERO的2000次无扰动成绩混入新基准。"},"source":{"name":"Benchmark authors","url":"https://arxiv.org/abs/2609.15940v1","published_at":"2026-09-14","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[8323,0.805,370,11517,11413],[7594,0.8975,370,11518,11413],[7598,0.885,370,11519,11413],[9600,0.215,370,11520,11413],[11417,0.935,370,11521,11413],[11419,0.0025,370,11522,11413]]},{"id":"liberoctrl-combination-l1-2026-paper","name":"LIBERO-CTRL ·combination L1 (2026 paper)","group":"Agent 与工具","metric":"Perturbed rollout success rate","unit":"%","public_history":true,"protocol":{"en":"Six distinct policies share the same400 combination/L1 rollouts: four LIBERO suites, ten tasks per suite, ten perturbation configurations per task, normalized severity radius2. Exact task/initial-state/configuration IDs match across policies; success counts independently reproduce all author summary cells. MINERVA is excluded because language and simultaneous inputs are identity controls for its lookup policy. Nominal2000-rollout LIBERO scores are not mixed into the new perturbation benchmark.","zh":"六个独立策略共享combination/L1下400次扰动测试：四个LIBERO套件、每套件十任务、每任务十扰动配置，标准化严重度半径为2。已逐项确认各策略任务、初态及配置编号一致，独立重算成功次数并核对作者汇总。MINERVA采用指令查表，语言和同步扰动对其含有身份控制，因此排除。未将普通LIBERO的2000次无扰动成绩混入新基准。"},"source":{"name":"Benchmark authors","url":"https://arxiv.org/abs/2609.15940v1","published_at":"2026-09-14","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[8323,0.775,370,11523,11413],[7594,0.81,370,11524,11413],[7598,0.465,370,11525,11413],[9600,0.2875,370,11526,11413],[11417,0.87,370,11527,11413],[11419,0.0125,370,11528,11413]]},{"id":"liberoctrl-combination-l2-2026-paper","name":"LIBERO-CTRL ·combination L2 (2026 paper)","group":"Agent 与工具","metric":"Perturbed rollout success rate","unit":"%","public_history":true,"protocol":{"en":"Six distinct policies share the same400 combination/L2 rollouts: four LIBERO suites, ten tasks per suite, ten perturbation configurations per task, normalized severity radius4. Exact task/initial-state/configuration IDs match across policies; success counts independently reproduce all author summary cells. MINERVA is excluded because language and simultaneous inputs are identity controls for its lookup policy. Nominal2000-rollout LIBERO scores are not mixed into the new perturbation benchmark.","zh":"六个独立策略共享combination/L2下400次扰动测试：四个LIBERO套件、每套件十任务、每任务十扰动配置，标准化严重度半径为4。已逐项确认各策略任务、初态及配置编号一致，独立重算成功次数并核对作者汇总。MINERVA采用指令查表，语言和同步扰动对其含有身份控制，因此排除。未将普通LIBERO的2000次无扰动成绩混入新基准。"},"source":{"name":"Benchmark authors","url":"https://arxiv.org/abs/2609.15940v1","published_at":"2026-09-14","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[8323,0.5025,370,11529,11413],[7594,0.5775,370,11530,11413],[7598,0.0625,370,11531,11413],[9600,0.1,370,11532,11413],[11417,0.4425,370,11533,11413],[11419,0.0,370,11534,11413]]},{"id":"liberoctrl-combination-l3-2026-paper","name":"LIBERO-CTRL ·combination L3 (2026 paper)","group":"Agent 与工具","metric":"Perturbed rollout success rate","unit":"%","public_history":true,"protocol":{"en":"Six distinct policies share the same400 combination/L3 rollouts: four LIBERO suites, ten tasks per suite, ten perturbation configurations per task, normalized severity radius8. Exact task/initial-state/configuration IDs match across policies; success counts independently reproduce all author summary cells. MINERVA is excluded because language and simultaneous inputs are identity controls for its lookup policy. Nominal2000-rollout LIBERO scores are not mixed into the new perturbation benchmark.","zh":"六个独立策略共享combination/L3下400次扰动测试：四个LIBERO套件、每套件十任务、每任务十扰动配置，标准化严重度半径为8。已逐项确认各策略任务、初态及配置编号一致，独立重算成功次数并核对作者汇总。MINERVA采用指令查表，语言和同步扰动对其含有身份控制，因此排除。未将普通LIBERO的2000次无扰动成绩混入新基准。"},"source":{"name":"Benchmark authors","url":"https://arxiv.org/abs/2609.15940v1","published_at":"2026-09-14","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[8323,0.1825,370,11535,11413],[7594,0.135,370,11536,11413],[7598,0.0,370,11537,11413],[9600,0.0075,370,11538,11413],[11417,0.08,370,11539,11413],[11419,0.0025,370,11540,11413]]},{"id":"grounding-ladder-u-shared47-2026-report","name":"The Grounding Ladder · U, shared 47 (2026 report)","group":"专业工作","metric":"Task success on shared 47 questions","unit":"%","public_history":true,"protocol":{"en":"Seven complete model aliases are re-aggregated on their 47 shared questions under schema only. The original CSV uses the frozen v1 pre-promotion scorer: exact/numeric SQL-result checks plus the intended refusal or clarification where required. Each question receives equal weight, with its recorded one, three or five trials averaged first. The two partial models and five extra multi-turn questions are excluded. This explicitly recomputed shared panel differs from the source’s pooled summary. Original unversioned Claude aliases are retained without inferring versions.","zh":"七个覆盖完整的模型别名在共同47个问题上重新汇总，条件为schema only。采用原CSV冻结的v1升级前评分：SQL结果精确/数值检查，以及题目要求的拒答或澄清。各题等权，先平均该模型记录的一、三或五次试验。排除两个部分覆盖模型和五个额外多轮问题；本面板明确按共同题集重算，区别于原汇总榜。Claude仅保留来源未标版本的别名。"},"source":{"name":"The Grounding Ladder authors","url":"https://github.com/gcorneliusvh/looker-vs-databricks-genie-benchmark/tree/84bd858e1fdd0178cce13ce007554e1153c6bd9b","published_at":"2026-08-20","status":"paper_baselines"},"model_count":7,"result_count":7,"results":[[5458,0.2127659574468085,370,11541,11542],[11543,0.14893617021276595,370,11541,11542],[11544,0.19148936170212766,370,11541,11542],[4164,0.1702127659574468,370,11545,11542],[11546,0.1702127659574468,370,11545,11542],[8463,0.19148936170212766,370,11547,11542],[6035,0.20851063829787236,370,11545,11542]]},{"id":"grounding-ladder-d-shared47-2026-report","name":"The Grounding Ladder · D, shared 47 (2026 report)","group":"专业工作","metric":"Task success on shared 47 questions","unit":"%","public_history":true,"protocol":{"en":"Seven complete model aliases are re-aggregated on their 47 shared questions under document grounding. The original CSV uses the frozen v1 pre-promotion scorer: exact/numeric SQL-result checks plus the intended refusal or clarification where required. Each question receives equal weight, with its recorded one, three or five trials averaged first. The two partial models and five extra multi-turn questions are excluded. This explicitly recomputed shared panel differs from the source’s pooled summary. Original unversioned Claude aliases are retained without inferring versions.","zh":"七个覆盖完整的模型别名在共同47个问题上重新汇总，条件为document grounding。采用原CSV冻结的v1升级前评分：SQL结果精确/数值检查，以及题目要求的拒答或澄清。各题等权，先平均该模型记录的一、三或五次试验。排除两个部分覆盖模型和五个额外多轮问题；本面板明确按共同题集重算，区别于原汇总榜。Claude仅保留来源未标版本的别名。"},"source":{"name":"The Grounding Ladder authors","url":"https://github.com/gcorneliusvh/looker-vs-databricks-genie-benchmark/tree/84bd858e1fdd0178cce13ce007554e1153c6bd9b","published_at":"2026-08-20","status":"paper_baselines"},"model_count":7,"result_count":7,"results":[[5458,0.8085106382978723,370,11548,11542],[11543,0.851063829787234,370,11548,11542],[11544,0.8297872340425532,370,11548,11542],[4164,0.7829787234042552,370,11549,11542],[11546,0.6170212765957447,370,11549,11542],[8463,0.9219858156028369,370,11550,11542],[6035,1.0,370,11549,11542]]},{"id":"grounding-ladder-g-shared47-2026-report","name":"The Grounding Ladder · G, shared 47 (2026 report)","group":"专业工作","metric":"Task success on shared 47 questions","unit":"%","public_history":true,"protocol":{"en":"Seven complete model aliases are re-aggregated on their 47 shared questions under model in prompt. The original CSV uses the frozen v1 pre-promotion scorer: exact/numeric SQL-result checks plus the intended refusal or clarification where required. Each question receives equal weight, with its recorded one, three or five trials averaged first. The two partial models and five extra multi-turn questions are excluded. This explicitly recomputed shared panel differs from the source’s pooled summary. Original unversioned Claude aliases are retained without inferring versions.","zh":"七个覆盖完整的模型别名在共同47个问题上重新汇总，条件为model in prompt。采用原CSV冻结的v1升级前评分：SQL结果精确/数值检查，以及题目要求的拒答或澄清。各题等权，先平均该模型记录的一、三或五次试验。排除两个部分覆盖模型和五个额外多轮问题；本面板明确按共同题集重算，区别于原汇总榜。Claude仅保留来源未标版本的别名。"},"source":{"name":"The Grounding Ladder authors","url":"https://github.com/gcorneliusvh/looker-vs-databricks-genie-benchmark/tree/84bd858e1fdd0178cce13ce007554e1153c6bd9b","published_at":"2026-08-20","status":"paper_baselines"},"model_count":7,"result_count":7,"results":[[5458,0.8297872340425532,370,11551,11542],[11543,0.851063829787234,370,11551,11542],[11544,0.851063829787234,370,11551,11542],[4164,0.8212765957446809,370,11552,11542],[11546,0.7021276595744681,370,11552,11542],[8463,0.9148936170212766,370,11553,11542],[6035,1.0,370,11552,11542]]},{"id":"grounding-ladder-s-shared47-2026-report","name":"The Grounding Ladder · S, shared 47 (2026 report)","group":"专业工作","metric":"Task success on shared 47 questions","unit":"%","public_history":true,"protocol":{"en":"Seven complete model aliases are re-aggregated on their 47 shared questions under semantic compiler. The original CSV uses the frozen v1 pre-promotion scorer: exact/numeric SQL-result checks plus the intended refusal or clarification where required. Each question receives equal weight, with its recorded one, three or five trials averaged first. The two partial models and five extra multi-turn questions are excluded. This explicitly recomputed shared panel differs from the source’s pooled summary. Original unversioned Claude aliases are retained without inferring versions.","zh":"七个覆盖完整的模型别名在共同47个问题上重新汇总，条件为semantic compiler。采用原CSV冻结的v1升级前评分：SQL结果精确/数值检查，以及题目要求的拒答或澄清。各题等权，先平均该模型记录的一、三或五次试验。排除两个部分覆盖模型和五个额外多轮问题；本面板明确按共同题集重算，区别于原汇总榜。Claude仅保留来源未标版本的别名。"},"source":{"name":"The Grounding Ladder authors","url":"https://github.com/gcorneliusvh/looker-vs-databricks-genie-benchmark/tree/84bd858e1fdd0178cce13ce007554e1153c6bd9b","published_at":"2026-08-20","status":"paper_baselines"},"model_count":7,"result_count":7,"results":[[5458,0.851063829787234,370,11554,11542],[11543,0.8723404255319149,370,11554,11542],[11544,0.8936170212765957,370,11554,11542],[4164,0.8468085106382978,370,11555,11542],[11546,0.851063829787234,370,11555,11542],[8463,1.0,370,11556,11542],[6035,0.9787234042553191,370,11555,11542]]},{"id":"gift-eval-ctx-relative-mase-2026-paper","name":"GIFT-Eval-CTX · supplied effects (2026 paper)","group":"预测与决策","metric":"Relative MASE, geometric mean","unit":"raw","public_history":true,"protocol":{"en":"Six independent base models retain history-only, structured-effect and MoiraiAgent configurations across the same 10,000 windows, 20 source datasets and 78 frequency/term groups. Per-window seasonal-scaled absolute error is averaged within each group, divided by seasonal-naive error, then geometrically averaged across groups; lower is better. Effects are privileged annotations generated with access to future ground truth, so this tests use of supplied context rather than an unassisted real-world forecast.","zh":"六个独立底模保留纯历史、结构化影响及MoiraiAgent配置，评测相同10,000个窗口、20个源数据集及78个频率/预测范围组。各窗口绝对误差按季节性历史误差缩放，组内平均后除以季节朴素基线，再对各组取几何均值，越低越好。影响注释构建时可访问未来真值，本任务衡量给定特权上下文的利用能力。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.17794v1","published_at":"2026-10-13","status":"paper_baselines"},"higher_is_better":false,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Relative MASE; seasonal naive = 1","zh":"相对MASE；季节朴素基线=1"},"model_count":6,"result_count":9,"results":[[6809,0.795,370,11557,11558],[11559,0.798,370,11557,11558],[11560,0.782,370,11557,11558],[6805,0.78,370,11557,11558],[3479,1.109,370,11557,11558],[3479,0.781,370,11561,11558],[762,0.953,370,11557,11558],[762,0.565,370,11561,11558],[762,0.497,370,11562,11558]]},{"id":"protargetbench-recall50-2026-paper","name":"ProTargetBench · full-catalogue recovery (2026 paper)","group":"医疗","metric":"Recall@50","unit":"score","public_history":true,"protocol":{"en":"The complete eight-method panel ranks the same 20,431 proteins for 198 held-out molecules. Recall@50 measures the recovered share of each query’s recorded positive targets and is averaged over molecules. Five neural methods use five-seed means; weighted k-NN and SEA provide traditional baselines, with target-frequency as a control. Unmeasured proteins remain unlabeled and are not treated as experimental negatives.","zh":"完整八方法面板为相同198个留出分子排名20,431个蛋白。Recall@50表示每个查询已记录阳性靶点中进入前50的比例，再按分子取均值。五种神经方法采用五种子均值，另含加权k近邻及SEA传统基线、靶点频率对照。未测蛋白保持未标注，不当作实验阴性。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.24458v1","published_at":"2026-10-17","status":"paper_baselines"},"score_max":1,"score_decimals":2,"saturation_excluded":true,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":8,"result_count":8,"results":[[11563,0.1364,370,11564,11565],[11566,0.4187,370,11564,11565],[11567,0.4092,370,11564,11565],[11568,0.3719,370,11569,11565],[11570,0.3402,370,11569,11565],[11571,0.4126,370,11569,11565],[11572,0.465,370,11569,11565],[11573,0.5196,370,11569,11565]]},{"id":"pumpire-image-normal-ae-2026-paper","name":"Pumpire · image geometry, normal (2026 paper)","group":"视觉与视频","metric":"Point-pair absolute error (m)","unit":"raw","public_history":true,"protocol":{"en":"Eight image-level geometry models use the same 76 scenes, each with 64 views and physically measured point-pair distance ground truth. Absolute error in meters is first averaged over views within a scene and then equally over scenes. The 76 normal and 24 challenging scenes are split by raw-sensor cross-view RSD at 0.1 and remain separate. Lower error is better; no depth-prior completion models are mixed into this estimation panel.","zh":"八个图像级几何模型使用同一76个场景，每场景64视角及物理实测点对距离真值。绝对误差以米计，先在场景内平均视角，再在场景间等权平均。按原始传感器跨视角RSD阈值0.1区分76个普通场景和24个困难场景，分别呈现。越低越好，不混入带深度先验的补全模型。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.12423v1","published_at":"2026-10-08","status":"paper_baselines"},"higher_is_better":false,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Absolute error (m)","zh":"绝对误差（米）"},"model_count":8,"result_count":8,"results":[[11574,1.18,370,11575,11576],[11577,0.208,370,11575,11576],[11578,0.135,370,11575,11576],[11579,0.159,370,11575,11576],[11580,0.176,370,11575,11576],[11581,0.145,370,11575,11576],[11582,0.099,370,11575,11576],[11583,0.073,370,11575,11576]]},{"id":"pumpire-image-challenging-ae-2026-paper","name":"Pumpire · image geometry, challenging (2026 paper)","group":"视觉与视频","metric":"Point-pair absolute error (m)","unit":"raw","public_history":true,"protocol":{"en":"Eight image-level geometry models use the same 24 scenes, each with 64 views and physically measured point-pair distance ground truth. Absolute error in meters is first averaged over views within a scene and then equally over scenes. The 76 normal and 24 challenging scenes are split by raw-sensor cross-view RSD at 0.1 and remain separate. Lower error is better; no depth-prior completion models are mixed into this estimation panel.","zh":"八个图像级几何模型使用同一24个场景，每场景64视角及物理实测点对距离真值。绝对误差以米计，先在场景内平均视角，再在场景间等权平均。按原始传感器跨视角RSD阈值0.1区分76个普通场景和24个困难场景，分别呈现。越低越好，不混入带深度先验的补全模型。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.12423v1","published_at":"2026-10-08","status":"paper_baselines"},"higher_is_better":false,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Absolute error (m)","zh":"绝对误差（米）"},"model_count":8,"result_count":8,"results":[[11574,5.004,370,11584,11576],[11577,0.158,370,11584,11576],[11578,0.171,370,11584,11576],[11579,0.943,370,11584,11576],[11580,0.202,370,11584,11576],[11581,0.333,370,11584,11576],[11582,1.059,370,11584,11576],[11583,1.126,370,11584,11576]]},{"id":"workerville-c0-acbr-2026-paper","name":"Workerville · neutral organization (2026 paper)","group":"安全与可靠性","metric":"Counterproductive-behavior episode rate","unit":"score","public_history":true,"protocol":{"en":"Six LLM backends run the same 210 tasks under C0, the neutral organizational configuration. An episode is counted once as counterproductive if any applicable disclosure, destructive-operation or production-deviation endpoint is judged unsafe. The denominator includes every episode, not only unsafe cases. The source uses memory-augmented endpoint judges selected against human labels. The native 0–100 rate is lower-better; it is separate from task completion and excluded from saturation ranges.","zh":"六个LLM端点在中性组织条件C0下执行相同210个任务。若适用的信息泄露、破坏操作或生产偏离端点中任一个被判不安全，该回合只计一次反生产行为；分母为全部回合。原评测使用经人工标签比较选择的带记忆端点裁判。保留0–100原生发生率，越低越好，与任务完成度分开，不进入饱和度分段。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.11561v1","published_at":"2026-10-08","status":"paper_baselines"},"score_max":100,"score_decimals":2,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"higher_is_better":false,"model_count":6,"result_count":6,"results":[[340,30.0,370,11585,11586],[3572,32.4,370,11585,11586],[3412,33.8,370,11585,11586],[6412,25.2,370,11585,11586],[3009,25.7,370,11585,11586],[7827,12.9,370,11585,11586]]},{"id":"lifesearch-weighted-rubric-2026-paper","name":"LifeSearch (2026 paper)","group":"Agent 与工具","metric":"Weighted research-report rubric score","unit":"score","public_history":true,"protocol":{"en":"Twelve model-agent configurations run all 203 bilingual deep-research tasks three times. A shared ReAct harness uses Serper search, Jina retrieval and code execution, with the maximum supported reasoning effort. Gemini3.1Pro labels each report rubric Pass/Fail. Annotator-assigned weights are normalized over all rubrics per task, including unmet ones, and tasks are averaged equally. The three-run mean remains a native weighted composite, distinct from binary task success.","zh":"十二个模型/Agent配置对相同203个双语深度研究任务各运行三次。共享ReAct框架，使用Serper搜索、Jina抓取与代码执行，采用最高支持的推理强度。Gemini3.1Pro逐条判定报告rubric通过或失败，按标注权重在每任务全部rubric中归一化，再对任务等权平均。保留三次运行均值的原生加权综合分，与二元任务成功率区分。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.18092v1","published_at":"2026-10-13","status":"paper_baselines"},"score_max":100,"score_decimals":2,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":12,"result_count":12,"results":[[762,63.44,370,11587,11588],[755,56.1,370,11587,11588],[3524,49.73,370,11587,11588],[3154,49.19,370,11587,11588],[3157,47.52,370,11587,11588],[6664,46.9,370,11587,11588],[3155,45.79,370,11587,11588],[3204,43.09,370,11587,11588],[3479,41.94,370,11587,11588],[5114,38.9,370,11587,11588],[227,37.11,370,11587,11588],[5136,34.72,370,11587,11588]]},{"id":"jevprobebench-replay-dfr-2026-paper","name":"JevProbeBench · paired replay (2026 paper)","group":"安全与可靠性","metric":"Macro decision-flip rate","unit":"score","score_max":100,"score_decimals":1,"higher_is_better":false,"saturation_excluded":true,"scale_label":{"en":"Flip rate / 100; lower is better","zh":"翻转率 / 100；越低越好"},"public_history":true,"protocol":{"en":"Jev and nine general-purpose LLMs receive the same benign and adversarial contexts under compatibility adapters that preserve application decision rules. DFR records changes at each application’s consumed Boolean, category or numeric boundary. The published macro mean weights ten applications equally. Attacks were selected against Jev, so this panel measures transfer of those attacks rather than model-specific optimal attacks. The native 0–100 flip rate is lower-better and separate from correctness.","zh":"Jev与九个通用LLM通过兼容适配器接受相同正常/对抗上下文，保留应用原有决策规则。DFR统计程序实际使用的布尔、类别或数值边界发生变化的比例，按十个应用等权取原表宏均值。攻击针对Jev选择，本面板衡量这些攻击的迁移表现，不代表各模型的最强专属攻击。保留0–100原生翻转率，越低越好，与正确率区分。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.18612v1","published_at":"2026-10-13","status":"paper_baselines"},"model_count":10,"result_count":10,"results":[[11589,79.8,370,11590,11591],[11592,61.2,370,11590,11591],[7265,66.8,370,11590,11591],[11593,64.3,370,11590,11591],[11594,55.2,370,11590,11591],[11595,62.6,370,11590,11591],[11596,54.0,370,11590,11591],[11597,50.7,370,11590,11591],[4979,60.7,370,11590,11591],[5519,54.1,370,11590,11591]]},{"id":"taod2c-t1-type-2026-paper","name":"TaoD2C · UI component implementation (2026 paper)","group":"Agent 与工具","metric":"Micro-F1","unit":"score","public_history":true,"protocol":{"en":"Seven fully executed model endpoints share all 421 interface designs. T1 scores generated React component types using source/runtime evidence, type-alias agreement and one-to-one geometric matching at IoU and area similarity ≥0.7; T2 scores inferred component requirements against expert annotations. Required entries determine recall; optional matches can improve precision without recall penalties. Native micro-F1 pools counts across designs. Qwen3-VL-235B is omitted because one design exceeded the provider input limit and its published F1 includes an unexecuted zero; the complete eight-model source table is retained.","zh":"七个完整执行的模型端点使用相同421个界面设计。T1根据源码与运行时证据、组件类型别名和IoU及面积相似度均至少0.7的一对一匹配评分；T2将预测组件要求与专家标注匹配。必要项计入召回分母，可选项匹配可改善精确率，遗漏可选项不扣召回。保留跨设计汇总计数的原生micro-F1。Qwen3-VL-235B有一个设计超出供应商输入限制，原表将未执行项计零，因此不列入可比面板；完整八模型原表保留。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.10374v1","published_at":"2026-10-07","status":"paper_baselines"},"score_max":100,"score_decimals":2,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":7,"result_count":7,"results":[[2973,62.47,370,11598,11599],[3184,60.19,370,11598,11599],[3568,59.89,370,11598,11599],[3157,61.77,370,11598,11599],[3415,62.07,370,11598,11599],[2977,63.25,370,11598,11599],[3412,57.74,370,11598,11599]]},{"id":"taod2c-t2-component-2026-paper","name":"TaoD2C · UI requirement inference (2026 paper)","group":"Agent 与工具","metric":"Micro-F1","unit":"score","public_history":true,"protocol":{"en":"Seven fully executed model endpoints share all 421 interface designs. T1 scores generated React component types using source/runtime evidence, type-alias agreement and one-to-one geometric matching at IoU and area similarity ≥0.7; T2 scores inferred component requirements against expert annotations. Required entries determine recall; optional matches can improve precision without recall penalties. Native micro-F1 pools counts across designs. Qwen3-VL-235B is omitted because one design exceeded the provider input limit and its published F1 includes an unexecuted zero; the complete eight-model source table is retained.","zh":"七个完整执行的模型端点使用相同421个界面设计。T1根据源码与运行时证据、组件类型别名和IoU及面积相似度均至少0.7的一对一匹配评分；T2将预测组件要求与专家标注匹配。必要项计入召回分母，可选项匹配可改善精确率，遗漏可选项不扣召回。保留跨设计汇总计数的原生micro-F1。Qwen3-VL-235B有一个设计超出供应商输入限制，原表将未执行项计零，因此不列入可比面板；完整八模型原表保留。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.10374v1","published_at":"2026-10-07","status":"paper_baselines"},"score_max":100,"score_decimals":2,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":7,"result_count":7,"results":[[2973,59.15,370,11600,11599],[3184,57.32,370,11600,11599],[3568,58.36,370,11600,11599],[3157,58.86,370,11600,11599],[3415,58.8,370,11600,11599],[2977,59.67,370,11600,11599],[3412,58.67,370,11600,11599]]},{"id":"olives-test-jf-2026-paper","name":"OLiVES · low-light video segmentation (2026 paper)","group":"视觉与视频","metric":"Mean region Jaccard and boundary F","unit":"score","public_history":true,"protocol":{"en":"Six distinct video-object-segmentation architectures are trained on OLiVES and evaluated on the same held-out test split: 102 videos, 7,055 frames, 245 objects and 15,646 annotations. The native J&F score averages the standard region Jaccard and boundary F measures on a 0–100 scale. Validation and external LLE-VOS results are separate source panels.","zh":"六个独立视频目标分割架构在OLiVES训练，使用同一保留测试集：102段视频、7055帧、245个目标和15646条标注。保留0–100原生J&F分，即标准区域Jaccard与边界F的均值。验证集和外部LLE-VOS结果留作独立原表面板。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.20501v1","published_at":"2026-10-14","status":"paper_baselines"},"score_max":100,"score_decimals":2,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":6,"result_count":6,"results":[[11601,78.3,370,11602,11603],[11604,80.5,370,11602,11603],[11605,86.9,370,11602,11603],[11606,74.4,370,11602,11603],[11607,88.3,370,11602,11603],[11608,88.4,370,11602,11603]]},{"id":"imu-bench-har-frozen-probe-2026-paper","name":"IMU-Bench · frozen-probe (2026 paper)","group":"预测与决策","metric":"Mean HAR macro-F1","unit":"score","public_history":true,"protocol":{"en":"Six matched inertial representation-learning methods use the same decoder, optimizer budget, splits and model selection within each protocol. The native score equally averages per-window macro-F1 over HHAR, MHEALTH, PRISM and TransfHAR. Windows span 2.56 seconds; the first three datasets are subject-disjoint, while TransfHAR uses chronological 80/20 splits per recording with boundary windows removed. Frozen probing and fine-tuning remain separate cohorts. Gray ablations, random/scratch references and the released external NormWear model are excluded as prescribed by the source ranking.","zh":"六个惯性表征学习方法在各协议内共享解码器、优化预算、数据划分与模型选择。原生分数对HHAR、MHEALTH、PRISM与TransfHAR四个数据集的窗口级macro-F1等权平均。窗口长2.56秒；前三者按受试者隔离，TransfHAR按每段录制的时间顺序80/20划分并去掉边界窗口。冻结探针与微调分开列组，按原表排除灰色消融、随机/从零训练参照及外部已发布NormWear。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.19481v1","published_at":"2026-10-14","status":"paper_baselines"},"score_max":100,"score_decimals":2,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":6,"result_count":6,"results":[[11609,57.8,370,11610,11611],[11612,56.2,370,11610,11611],[11613,53.1,370,11610,11611],[11614,44.2,370,11610,11611],[11615,35.4,370,11610,11611],[11616,55.1,370,11610,11611]]},{"id":"imu-bench-har-fine-tuning-2026-paper","name":"IMU-Bench · fine-tuning (2026 paper)","group":"预测与决策","metric":"Mean HAR macro-F1","unit":"score","public_history":true,"protocol":{"en":"Six matched inertial representation-learning methods use the same decoder, optimizer budget, splits and model selection within each protocol. The native score equally averages per-window macro-F1 over HHAR, MHEALTH, PRISM and TransfHAR. Windows span 2.56 seconds; the first three datasets are subject-disjoint, while TransfHAR uses chronological 80/20 splits per recording with boundary windows removed. Frozen probing and fine-tuning remain separate cohorts. Gray ablations, random/scratch references and the released external NormWear model are excluded as prescribed by the source ranking.","zh":"六个惯性表征学习方法在各协议内共享解码器、优化预算、数据划分与模型选择。原生分数对HHAR、MHEALTH、PRISM与TransfHAR四个数据集的窗口级macro-F1等权平均。窗口长2.56秒；前三者按受试者隔离，TransfHAR按每段录制的时间顺序80/20划分并去掉边界窗口。冻结探针与微调分开列组，按原表排除灰色消融、随机/从零训练参照及外部已发布NormWear。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.19481v1","published_at":"2026-10-14","status":"paper_baselines"},"score_max":100,"score_decimals":2,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":6,"result_count":6,"results":[[11609,63.9,370,11617,11611],[11612,56.1,370,11617,11611],[11613,52.1,370,11617,11611],[11614,56.1,370,11617,11611],[11615,52.8,370,11617,11611],[11616,56.7,370,11617,11611]]},{"id":"pathlang-fixed-mean-mf1-2026-paper","name":"PathLang · fixed mean pooling (2026 paper)","group":"医疗","metric":"Five-dataset mean macro-F1","unit":"score","public_history":true,"protocol":{"en":"Nine frozen patch-level pathology VLMs use one canonical prompt per class and query-independent mean pooling over all slide tiles. Classification selects the class of highest cosine similarity after L2 normalization. The native macro-F1 is averaged equally across PANDA (10,609 slides), UniToPatho (237), CAMELYON16 (399), CAMELYON17 (499) and TCGA-GBMLGG (1,335), retaining only slides with resolvable source labels. This fixed-setting appendix panel avoids the main table’s per-model, per-dataset oracle selection on evaluation labels.","zh":"九个冻结病理切片VLM使用每类一个规范提示，对整张切片全部图块做与查询无关的均值池化。L2归一化后按最高余弦相似度选择类别。原生macro-F1对PANDA（10609张）、UniToPatho（237）、CAMELYON16（399）、CAMELYON17（499）和TCGA-GBMLGG（1335）五数据集等权平均，只保留可解析原始标签的切片。采用附录固定配置面板，避开主表根据测试标签为各模型各数据集择优的配置。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.11329v1","published_at":"2026-10-08","status":"paper_baselines"},"score_max":1,"score_decimals":2,"saturation_excluded":true,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":9,"result_count":9,"results":[[9032,0.218,370,11618,11619],[7328,0.215,370,11618,11619],[11620,0.268,370,11618,11619],[11621,0.182,370,11618,11619],[11622,0.284,370,11618,11619],[11623,0.245,370,11618,11619],[11624,0.34,370,11618,11619],[7327,0.308,370,11618,11619],[11625,0.161,370,11618,11619]]},{"id":"pedbench-child-per-2026-paper","name":"PEDBench · child speech (2026 paper)","group":"专业工作","metric":"Phoneme error rate","unit":"raw","score_decimals":2,"higher_is_better":false,"saturation_excluded":true,"scale_label":{"en":"PER (%); lower is better","zh":"音素错误率（%）；越低越好"},"public_history":true,"protocol":{"en":"The same 334 utterances from 147 children; 1,502 reference phones are evaluated by thirteen off-the-shelf speech models plus two TIMIT-only fine-tunes. PER is corpus phoneme Levenshtein substitutions, deletions and insertions divided by reference phones, mapped to the 39-phone CMU inventory. The native error percentage is lower-better and may exceed 100. SC-trained variants are omitted because SC includes synthetic errors identified from the Child test; SW-trained variants are omitted because SW matches the Acted script. Full eighteen-configuration source panels remain in the fixture. BabAR was executed on Acted and returned empty output on 501/1,188 cases; these measured deletions remain in its PER, distinct from an unrun model.","zh":"十三个现成语音模型及两个仅用TIMIT微调的配置评测相同测试样本：334 utterances from 147 children; 1,502 reference phones。PER将音素替换、删除和插入的Levenshtein编辑总数除以参考音素总数，统一到39音素CMU表。保留原生错误百分数，越低越好，可能超过100。SC训练含从Child测试识别的错误模式，SW与Acted测试脚本相同，因此排除这两类目标模式训练配置，保留完整18配置原表。BabAR在Acted确已执行，其中501/1188条返回空文本，删除错误保留在PER中，与未运行模型区分。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.20830v1","published_at":"2026-10-14","status":"paper_baselines"},"model_count":13,"result_count":15,"results":[[11626,62.38,370,11627,11628],[11629,50.53,370,11627,11628],[11630,76.7,370,11627,11628],[11631,53.46,370,11627,11628],[11632,53.46,370,11627,11628],[11633,45.14,370,11627,11628],[11634,47.8,370,11627,11628],[11635,31.56,370,11627,11628],[11636,37.55,370,11627,11628],[5519,31.03,370,11627,11628],[7738,51.73,370,11627,11628],[6138,40.88,370,11627,11628],[6138,29.89,370,11637,11628],[7728,79.83,370,11627,11628],[7728,31.03,370,11637,11628]]},{"id":"pedbench-acted-per-2026-paper","name":"PEDBench · acted speech (2026 paper)","group":"专业工作","metric":"Phoneme error rate","unit":"raw","score_decimals":2,"higher_is_better":false,"saturation_excluded":true,"scale_label":{"en":"PER (%); lower is better","zh":"音素错误率（%）；越低越好"},"public_history":true,"protocol":{"en":"The same 1,188 utterances from three trained adult actors; 5,088 reference phones are evaluated by thirteen off-the-shelf speech models plus two TIMIT-only fine-tunes. PER is corpus phoneme Levenshtein substitutions, deletions and insertions divided by reference phones, mapped to the 39-phone CMU inventory. The native error percentage is lower-better and may exceed 100. SC-trained variants are omitted because SC includes synthetic errors identified from the Child test; SW-trained variants are omitted because SW matches the Acted script. Full eighteen-configuration source panels remain in the fixture. BabAR was executed on Acted and returned empty output on 501/1,188 cases; these measured deletions remain in its PER, distinct from an unrun model.","zh":"十三个现成语音模型及两个仅用TIMIT微调的配置评测相同测试样本：1,188 utterances from three trained adult actors; 5,088 reference phones。PER将音素替换、删除和插入的Levenshtein编辑总数除以参考音素总数，统一到39音素CMU表。保留原生错误百分数，越低越好，可能超过100。SC训练含从Child测试识别的错误模式，SW与Acted测试脚本相同，因此排除这两类目标模式训练配置，保留完整18配置原表。BabAR在Acted确已执行，其中501/1188条返回空文本，删除错误保留在PER中，与未运行模型区分。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.20830v1","published_at":"2026-10-14","status":"paper_baselines"},"model_count":13,"result_count":15,"results":[[11626,17.61,370,11627,11628],[11629,13.93,370,11627,11628],[11630,53.07,370,11627,11628],[11631,10.42,370,11627,11628],[11632,12.93,370,11627,11628],[11633,12.38,370,11627,11628],[11634,16.71,370,11627,11628],[11635,71.5,370,11627,11628],[11636,8.59,370,11627,11628],[5519,10.53,370,11627,11628],[7738,9.32,370,11627,11628],[6138,24.59,370,11627,11628],[6138,8.94,370,11637,11628],[7728,27.54,370,11627,11628],[7728,10.0,370,11637,11628]]},{"id":"drillbench-pilbara-mf1-2026-paper","name":"DrillBench · cross-province lithology (2026 paper)","group":"预测与决策","metric":"Next-stratum macro-F1","unit":"score","public_history":true,"protocol":{"en":"All 37 reported methods and configurations predict next-stratum lithology at ground-truth stratum boundaries on the same 3,278 Pilbara drillholes (179,619 intervals). There are at least six independent learned architectures: XGBoost, LSTM, GRU, Mamba, TCN and Transformer. Classical frequency/geostatistical references remain included but do not establish the model threshold. The native macro-F1 averages only lithology classes with at least 20 events; the shared vocabulary has 37 classes. Cell models remove CONT and renormalize at true boundaries. Inputs and retrieval/pretraining differ as published; retrieval and pretraining are configurations, not new architectures.","zh":"完整37方法/配置在相同3278口Pilbara钻孔（179619区间）的真实地层边界预测下一层岩性。至少六个独立学习架构为XGBoost、LSTM、GRU、Mamba、TCN和Transformer。传统频率/地统计参照保留，但不靠它们满足模型门槛。原生macro-F1仅对至少20个事件的岩性类别平均，共享词表含37类；单元模型在真实边界去掉CONT并重新归一化。按原表保留不同输入和检索/预训练配置，配置不计新增架构。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.01204v1","published_at":"2026-10-01","status":"paper_baselines"},"score_max":100,"score_decimals":2,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":35,"result_count":37,"results":[[11638,1.7,370,11639,11640],[11641,1.7,370,11642,11640],[11643,1.6,370,11644,11640],[11645,1.7,370,11646,11640],[11647,3.7,370,11648,11640],[11649,1.5,370,11650,11640],[11651,1.7,370,11652,11640],[11653,2.1,370,11654,11640],[11655,8.0,370,11656,11640],[11657,2.1,370,11658,11640],[11659,6.8,370,11660,11640],[11661,2.4,370,11662,11640],[11663,5.6,370,11664,11640],[11665,7.4,370,11666,11640],[11667,1.7,370,11668,11640],[6811,2.2,370,11669,11640],[9603,22.3,370,11670,11640],[9204,23.2,370,11671,11640],[11672,19.4,370,11673,11640],[9681,16.3,370,11674,11640],[6744,22.5,370,11675,11640],[11676,7.5,370,11677,11640],[11678,9.2,370,11679,11640],[11680,3.4,370,11681,11640],[11682,0.5,370,11683,11640],[11684,11.0,370,11685,11640],[11686,2.5,370,11687,11640],[11688,7.2,370,11689,11640],[11690,9.4,370,11691,11640],[11692,9.2,370,11693,11640],[11694,1.7,370,11695,11640],[11696,22.0,370,11697,11640],[9204,28.2,370,11698,11640],[6744,26.0,370,11699,11640],[11700,26.9,370,11701,11640],[11702,31.7,370,11703,11640],[11704,30.6,370,11705,11640]]},{"id":"refract-fc-query-tools-2026-paper","name":"ReFract · query-tools (2026 paper)","group":"Agent 与工具","metric":"Goal-reaching task pass rate","unit":"%","public_history":true,"protocol":{"en":"Eleven independent model endpoints execute all 150 role-conditioned industrial planning entries in an offline Python text world model, repeated five times. This is the function-calling panel; query-specific tools and the full tool catalog are separate cohorts. Pass rate counts whether the terminal state reaches the goal set. Persona compliance is a separate metric and a passing goal can still violate role boundaries. Parsing or undeclared-tool errors are returned for retry; unrecovered terminal trajectories fail.","zh":"十一个独立模型端点在离线Python文字世界中执行相同150个按角色限定的工业规划任务，每项运行五次。采用函数调用面板，按查询裁剪工具与完整工具目录分别列组。通过率按最终状态是否达到目标集合计分；角色合规另算，达到目标也可能越权。解析失败或未声明工具作为错误观察反馈重试，最终未恢复的轨迹计失败。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.03356v1","published_at":"2026-10-02","status":"paper_baselines"},"model_count":11,"result_count":11,"results":[[786,0.536,370,11706,11707],[11708,0.388,370,11706,11707],[784,0.387,370,11706,11707],[4967,0.279,370,11706,11707],[7668,0.268,370,11706,11707],[5635,0.189,370,11706,11707],[5480,0.362,370,11706,11707],[3196,0.607,370,11706,11707],[5297,0.625,370,11706,11707],[5298,0.572,370,11706,11707],[4048,0.615,370,11706,11707]]},{"id":"refract-fc-full-tools-2026-paper","name":"ReFract · full-tools (2026 paper)","group":"Agent 与工具","metric":"Goal-reaching task pass rate","unit":"%","public_history":true,"protocol":{"en":"Eleven independent model endpoints execute all 150 role-conditioned industrial planning entries in an offline Python text world model, repeated five times. This is the function-calling panel; query-specific tools and the full tool catalog are separate cohorts. Pass rate counts whether the terminal state reaches the goal set. Persona compliance is a separate metric and a passing goal can still violate role boundaries. Parsing or undeclared-tool errors are returned for retry; unrecovered terminal trajectories fail.","zh":"十一个独立模型端点在离线Python文字世界中执行相同150个按角色限定的工业规划任务，每项运行五次。采用函数调用面板，按查询裁剪工具与完整工具目录分别列组。通过率按最终状态是否达到目标集合计分；角色合规另算，达到目标也可能越权。解析失败或未声明工具作为错误观察反馈重试，最终未恢复的轨迹计失败。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.03356v1","published_at":"2026-10-02","status":"paper_baselines"},"model_count":11,"result_count":11,"results":[[786,0.397,370,11709,11707],[11708,0.152,370,11709,11707],[784,0.185,370,11709,11707],[4967,0.195,370,11709,11707],[7668,0.166,370,11709,11707],[5635,0.14,370,11709,11707],[5480,0.326,370,11709,11707],[3196,0.473,370,11709,11707],[5297,0.436,370,11709,11707],[5298,0.424,370,11709,11707],[4048,0.471,370,11709,11707]]},{"id":"agenthorizon-ah-agentic-2026-paper","name":"AgentHorizon · agentic judges (2026 paper)","group":"Agent 与工具","metric":"Balanced success-judgment accuracy","unit":"score","public_history":true,"protocol":{"en":"The complete published interface cohort judges the same held-out AH split: 528 instruction–trajectory pairs, with 227 positives and 301 negatives. Balanced accuracy equally averages positive and negative accuracies using the full class denominators; missing, malformed, null and non-Boolean predictions are errors. AH-D development and the easier AH-S complement are separate. Agentic results use each listed coding-agent interface. Direct inputs use 512×332 screenshots; starred model configurations merge four frames into a 2×2 mosaic for context limits. Interface/preprocessing are explicit configurations, not additional underlying models.","zh":"完整接口队列判断相同AH保留集：528个指令与轨迹配对，其中227正例、301负例。平衡准确率对两类准确率等权平均，使用完整类别分母，缺失、格式错误、空值和非布尔预测计错。AH-D开发集与较简单的AH-S另算。Agent配置采用表列代码Agent接口，直接输入采用512×332截图，带星配置为适应上下文将四帧拼成2×2图。接口和预处理作为配置，不额外计底模。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.11050v1","published_at":"2026-10-08","status":"paper_baselines"},"score_max":100,"score_decimals":2,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":10,"result_count":10,"results":[[3196,80.9,370,11710,11711],[3412,77.7,370,11712,11711],[3578,76.0,370,11713,11711],[4995,70.4,370,11714,11711],[5449,69.2,370,11710,11711],[11715,64.6,370,11714,11711],[325,52.2,370,11713,11711],[3497,52.1,370,11712,11711],[5934,49.4,370,11714,11711],[2585,46.8,370,11714,11711]]},{"id":"agenthorizon-ah-direct-2026-paper","name":"AgentHorizon · direct judges (2026 paper)","group":"Agent 与工具","metric":"Balanced success-judgment accuracy","unit":"score","public_history":true,"protocol":{"en":"The complete published interface cohort judges the same held-out AH split: 528 instruction–trajectory pairs, with 227 positives and 301 negatives. Balanced accuracy equally averages positive and negative accuracies using the full class denominators; missing, malformed, null and non-Boolean predictions are errors. AH-D development and the easier AH-S complement are separate. Agentic results use each listed coding-agent interface. Direct inputs use 512×332 screenshots; starred model configurations merge four frames into a 2×2 mosaic for context limits. Interface/preprocessing are explicit configurations, not additional underlying models.","zh":"完整接口队列判断相同AH保留集：528个指令与轨迹配对，其中227正例、301负例。平衡准确率对两类准确率等权平均，使用完整类别分母，缺失、格式错误、空值和非布尔预测计错。AH-D开发集与较简单的AH-S另算。Agent配置采用表列代码Agent接口，直接输入采用512×332截图，带星配置为适应上下文将四帧拼成2×2图。接口和预处理作为配置，不额外计底模。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.11050v1","published_at":"2026-10-08","status":"paper_baselines"},"score_max":100,"score_decimals":2,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":9,"result_count":9,"results":[[4995,77.6,370,11716,11711],[3412,77.4,370,11716,11711],[5672,70.3,370,11716,11711],[11715,69.5,370,11716,11711],[3196,68.6,370,11717,11711],[5449,66.5,370,11717,11711],[2585,60.1,370,11717,11711],[3497,53.6,370,11717,11711],[5934,46.5,370,11717,11711]]},{"id":"soccernet-foulret-gold-hit10-2026-paper","name":"SoccerNet-FoulRet · referee precedents (2026 paper)","group":"视觉与视频","metric":"HitRate@10","unit":"%","public_history":true,"protocol":{"en":"Eight learned retrieval systems/configurations share all 693 foul queries with at least one human-verified precedent and a 2,916-clip gallery. HitRate@10 counts queries with at least one verified precedent among the first ten retrieved clips. It is separate from Recall@10, nDCG and category-label agreement. Qwen3-VL-Emb-2B-FT is a fine-tuned configuration, not an extra underlying model. All other source methods and the random reference are retained in the complete fixture.","zh":"八个学习检索系统/配置使用相同693条至少有一个裁判确认先例的犯规查询与2916视频片段库。HitRate@10统计前十检索项中至少一个确认先例的查询占比，与Recall@10、nDCG及类别标签匹配分开。Qwen3-VL-Emb-2B-FT属于微调配置，不额外计底模；完整原表保留全部方法和随机参照。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.09742v1","published_at":"2026-10-07","status":"paper_baselines"},"model_count":8,"result_count":8,"results":[[11718,0.0317,370,11719,11720],[11721,0.0462,370,11719,11720],[11722,0.0346,370,11719,11720],[11723,0.0375,370,11719,11720],[11724,0.0332,370,11719,11720],[11725,0.039,370,11719,11720],[8357,0.0317,370,11719,11720],[11726,0.0317,370,11719,11720]]},{"id":"perso-behavior-2026-paper","name":"PerSo · delta-response selection (2026 paper)","group":"专业工作","metric":"Exact-choice accuracy","unit":"%","public_history":true,"protocol":{"en":"Fifteen configurations of twelve underlying models share 200 sampled ChangeMyView threads. Behavioral selection identifies which of two responses received the original poster’s delta award; the selected positive also has a higher community score, and option order is randomized. Each cognitive dimension has 400 four-choice questions across both branches, forecasting the poster’s next-reply state without seeing that reply. Gemini 2.5 Pro generated cognitive gold/distractors with the full dialogue and is also an evaluated model; human review checks label quality. Behavioral outcomes and cognitive labels remain different tasks. Qwen thinking modes are configurations, not new underlying models.","zh":"十二个底模的十五配置使用相同200个ChangeMyView讨论。行为任务从两条回复中选出原帖主授予delta的一条，正例另要求社区得分更高，选项顺序随机。各认知维度对两分支共400道四选一题，在看不到后续回复时预测原帖主的信念、情绪或意图。Gemini2.5Pro根据完整对话生成认知金标与干扰项，且自身也参与评测，人工检查标签质量。行为结果与认知标签分开列组，Qwen思考模式不额外计底模。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.20548v1","published_at":"2026-10-14","status":"paper_baselines"},"model_count":12,"result_count":15,"results":[[6677,0.605,370,11727,11728],[6668,0.625,370,11727,11728],[6689,0.71,370,11727,11728],[4969,0.64,370,11729,11728],[5619,0.75,370,11729,11728],[3191,0.76,370,11729,11728],[4969,0.665,370,11730,11728],[5619,0.73,370,11730,11728],[3191,0.73,370,11730,11728],[6292,0.675,370,11727,11728],[3009,0.69,370,11727,11728],[3185,0.62,370,11727,11728],[10538,0.695,370,11727,11728],[10537,0.715,370,11727,11728],[764,0.71,370,11727,11728]]},{"id":"perso-belief-2026-paper","name":"PerSo · belief forecasting (2026 paper)","group":"专业工作","metric":"Exact-choice accuracy","unit":"%","public_history":true,"protocol":{"en":"Fifteen configurations of twelve underlying models share 200 sampled ChangeMyView threads. Behavioral selection identifies which of two responses received the original poster’s delta award; the selected positive also has a higher community score, and option order is randomized. Each cognitive dimension has 400 four-choice questions across both branches, forecasting the poster’s next-reply state without seeing that reply. Gemini 2.5 Pro generated cognitive gold/distractors with the full dialogue and is also an evaluated model; human review checks label quality. Behavioral outcomes and cognitive labels remain different tasks. Qwen thinking modes are configurations, not new underlying models.","zh":"十二个底模的十五配置使用相同200个ChangeMyView讨论。行为任务从两条回复中选出原帖主授予delta的一条，正例另要求社区得分更高，选项顺序随机。各认知维度对两分支共400道四选一题，在看不到后续回复时预测原帖主的信念、情绪或意图。Gemini2.5Pro根据完整对话生成认知金标与干扰项，且自身也参与评测，人工检查标签质量。行为结果与认知标签分开列组，Qwen思考模式不额外计底模。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.20548v1","published_at":"2026-10-14","status":"paper_baselines"},"model_count":12,"result_count":15,"results":[[6677,0.5,370,11727,11728],[6668,0.5825,370,11727,11728],[6689,0.6,370,11727,11728],[4969,0.66,370,11729,11728],[5619,0.7175,370,11729,11728],[3191,0.805,370,11729,11728],[4969,0.635,370,11730,11728],[5619,0.6975,370,11730,11728],[3191,0.79,370,11730,11728],[6292,0.76,370,11727,11728],[3009,0.7375,370,11727,11728],[3185,0.8,370,11727,11728],[10538,0.685,370,11727,11728],[10537,0.83,370,11727,11728],[764,0.795,370,11727,11728]]},{"id":"perso-emotion-2026-paper","name":"PerSo · emotion forecasting (2026 paper)","group":"专业工作","metric":"Exact-choice accuracy","unit":"%","public_history":true,"protocol":{"en":"Fifteen configurations of twelve underlying models share 200 sampled ChangeMyView threads. Behavioral selection identifies which of two responses received the original poster’s delta award; the selected positive also has a higher community score, and option order is randomized. Each cognitive dimension has 400 four-choice questions across both branches, forecasting the poster’s next-reply state without seeing that reply. Gemini 2.5 Pro generated cognitive gold/distractors with the full dialogue and is also an evaluated model; human review checks label quality. Behavioral outcomes and cognitive labels remain different tasks. Qwen thinking modes are configurations, not new underlying models.","zh":"十二个底模的十五配置使用相同200个ChangeMyView讨论。行为任务从两条回复中选出原帖主授予delta的一条，正例另要求社区得分更高，选项顺序随机。各认知维度对两分支共400道四选一题，在看不到后续回复时预测原帖主的信念、情绪或意图。Gemini2.5Pro根据完整对话生成认知金标与干扰项，且自身也参与评测，人工检查标签质量。行为结果与认知标签分开列组，Qwen思考模式不额外计底模。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.20548v1","published_at":"2026-10-14","status":"paper_baselines"},"model_count":12,"result_count":15,"results":[[6677,0.205,370,11727,11728],[6668,0.2325,370,11727,11728],[6689,0.2575,370,11727,11728],[4969,0.2525,370,11729,11728],[5619,0.3575,370,11729,11728],[3191,0.41,370,11729,11728],[4969,0.25,370,11730,11728],[5619,0.4125,370,11730,11728],[3191,0.4875,370,11730,11728],[6292,0.3575,370,11727,11728],[3009,0.36,370,11727,11728],[3185,0.3975,370,11727,11728],[10538,0.345,370,11727,11728],[10537,0.59,370,11727,11728],[764,0.3725,370,11727,11728]]},{"id":"perso-intent-2026-paper","name":"PerSo · intent forecasting (2026 paper)","group":"专业工作","metric":"Exact-choice accuracy","unit":"%","public_history":true,"protocol":{"en":"Fifteen configurations of twelve underlying models share 200 sampled ChangeMyView threads. Behavioral selection identifies which of two responses received the original poster’s delta award; the selected positive also has a higher community score, and option order is randomized. Each cognitive dimension has 400 four-choice questions across both branches, forecasting the poster’s next-reply state without seeing that reply. Gemini 2.5 Pro generated cognitive gold/distractors with the full dialogue and is also an evaluated model; human review checks label quality. Behavioral outcomes and cognitive labels remain different tasks. Qwen thinking modes are configurations, not new underlying models.","zh":"十二个底模的十五配置使用相同200个ChangeMyView讨论。行为任务从两条回复中选出原帖主授予delta的一条，正例另要求社区得分更高，选项顺序随机。各认知维度对两分支共400道四选一题，在看不到后续回复时预测原帖主的信念、情绪或意图。Gemini2.5Pro根据完整对话生成认知金标与干扰项，且自身也参与评测，人工检查标签质量。行为结果与认知标签分开列组，Qwen思考模式不额外计底模。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.20548v1","published_at":"2026-10-14","status":"paper_baselines"},"model_count":12,"result_count":15,"results":[[6677,0.545,370,11727,11728],[6668,0.6325,370,11727,11728],[6689,0.645,370,11727,11728],[4969,0.6075,370,11729,11728],[5619,0.7375,370,11729,11728],[3191,0.75,370,11729,11728],[4969,0.605,370,11730,11728],[5619,0.665,370,11730,11728],[3191,0.745,370,11730,11728],[6292,0.72,370,11727,11728],[3009,0.6975,370,11727,11728],[3185,0.75,370,11727,11728],[10538,0.725,370,11727,11728],[10537,0.805,370,11727,11728],[764,0.7775,370,11727,11728]]},{"id":"decisionrobust-text-injection-accuracy-2026-paper","name":"DecisionRobust · text injection (2026 paper)","group":"安全与可靠性","metric":"Adversarial task accuracy","unit":"%","public_history":true,"protocol":{"en":"Twenty-two fully covered endpoints receive the same 374 curated text-injection candidate pairs. Accuracy includes all candidates, even when the clean answer was incorrect, and scores the original task answer rather than compliance with the attacker target. Choice and Noul preserve native decisions/class normalization; Score uses probability-weighted mean within 0.5 of the reference. The candidate set was selected against the first six construction models, with additional models held out from selection. Eight incomplete endpoints are excluded from this common-374 panel; no missing run is assigned zero. This accuracy is distinct from the source ASR conditioned on each model’s clean-correct valid samples.","zh":"二十二个完整覆盖端点接受相同374个筛选后文本注入候选配对。准确率包含全部候选，包括正常输入本来就答错的样本，按原任务答案计分，不按攻击目标计分。Choice/Noul保留原生决策与类别概率归一化，Score要求概率加权均值距参考值不超过0.5。候选对前六个构建模型择选，其余模型未参与择选。共同374面板排除八个不完整端点，缺失运行不补零；与按模型正常答对且有效样本为分母的ASR区分。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.19118v1","published_at":"2026-10-13","status":"paper_baselines"},"model_count":22,"result_count":22,"results":[[11731,0.083,370,11732,11733],[11734,0.591,370,11732,11733],[11735,0.134,370,11732,11733],[11736,0.23,370,11732,11733],[11737,0.524,370,11732,11733],[6194,0.40399999999999997,370,11732,11733],[6209,0.586,370,11732,11733],[11738,0.259,370,11732,11733],[11739,0.54,370,11732,11733],[11740,0.39299999999999996,370,11732,11733],[11741,0.428,370,11732,11733],[11742,0.083,370,11732,11733],[11743,0.147,370,11732,11733],[11744,0.47100000000000003,370,11732,11733],[11745,0.41200000000000003,370,11732,11733],[11746,0.321,370,11732,11733],[7363,0.5720000000000001,370,11732,11733],[6207,0.46299999999999997,370,11732,11733],[11747,0.58,370,11732,11733],[11748,0.369,370,11732,11733],[11749,0.11199999999999999,370,11732,11733],[11750,0.166,370,11732,11733]]},{"id":"decisionrobust-image-injection-accuracy-2026-paper","name":"DecisionRobust · in-image injection (2026 paper)","group":"安全与可靠性","metric":"Adversarial task accuracy","unit":"%","public_history":true,"protocol":{"en":"Seventeen endpoints receive all 223 curated in-image injection pairs in Choice format. Accuracy measures the original task answer over every candidate pair, including clean-incorrect samples; it is not targeted attack success or clean-correct-conditioned ASR. The first five models participated in candidate selection and twelve were held out. Attack candidates and external validity checks are frozen before transfer evaluation. All endpoints have the identical 223-pair denominator.","zh":"十七个端点在Choice格式下接收全部223个图中注入候选配对。准确率按每个候选的原任务答案计分，包含正常输入答错样本，与目标攻击成功率或仅正常答对样本上的ASR区分。前五个模型参与候选选择，十二个留作未参与选择的迁移评测。候选攻击及外部有效性检查在迁移评测前冻结，全部端点分母同为223。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.19118v1","published_at":"2026-10-13","status":"paper_baselines"},"model_count":17,"result_count":17,"results":[[11751,0.5429999999999999,370,11752,11753],[11754,0.067,370,11752,11753],[11743,0.175,370,11752,11753],[11740,0.395,370,11752,11753],[11745,0.354,370,11752,11753],[11744,0.37200000000000005,370,11752,11753],[11755,0.435,370,11752,11753],[11756,0.121,370,11752,11753],[11741,0.596,370,11752,11753],[11757,0.601,370,11752,11753],[11742,0.457,370,11752,11753],[11738,0.682,370,11752,11753],[11758,0.628,370,11752,11753],[7363,0.565,370,11752,11753],[6207,0.691,370,11752,11753],[11747,0.345,370,11752,11753],[11748,0.7090000000000001,370,11752,11753]]},{"id":"rag-pibench-protected-f1-2026-paper","name":"RAG-PIBench · protected test (2026 paper)","group":"安全与可靠性","metric":"Injection-positive F1","unit":"score","public_history":true,"protocol":{"en":"Six learned detector architectures plus a keyword rule baseline share the same balanced 978-example protected test set. Training uses only the 2,936 train examples, with model/checkpoint selection on 962 validation examples; protected inference is opened only after selection. F1 uses malicious prompt injection as the positive class. DistilBERT, DeBERTa-v3, MiniLM similarity, TF-IDF SVM, logistic regression and random forest establish six learned systems; the keyword rule is retained as a reference and not counted as a learned architecture.","zh":"六种学习检测架构与一个关键词规则参照使用相同978条平衡保留测试集。仅用2936条训练样本拟合，在962条验证样本上选择模型/检查点，再打开保留集推理。F1以恶意提示注入为正类。六种学习系统为DistilBERT、DeBERTa-v3、MiniLM相似度、TF-IDF SVM、逻辑回归和随机森林；保留关键词参照，不靠它凑学习架构。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.08571v1","published_at":"2026-10-06","status":"paper_baselines"},"score_max":1,"score_decimals":2,"saturation_excluded":true,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":7,"result_count":7,"results":[[11278,0.896,370,11759,11760],[11761,0.871,370,11759,11760],[11762,0.867,370,11759,11760],[11763,0.85,370,11759,11760],[11764,0.667,370,11759,11760],[11765,0.667,370,11759,11760],[11766,0.189,370,11759,11760]]},{"id":"humanoidtoolbench-l1-ballmove-2026-paper","name":"HumanoidToolBench · BallMove tool use (2026 paper)","group":"Agent 与工具","metric":"Task success rate","unit":"%","public_history":true,"protocol":{"en":"Seven independently trained robot policies use the same 100 seeded MuJoCo episodes per task. This is L1 in the standard tool-set mode, where the policy must select and use a suitable tool; success requires the scenario criterion to hold for one second within 3,000 control steps (one minute at 50 Hz). Every policy trains once for 40,000 updates on the same 3,003 simulation demonstrations. Observation and initial scenes are matched; native action chunk lengths differ. Decoy, L0/L2 and real-robot panels remain separate.","zh":"七个独立训练机器人策略使用每任务相同100个种子MuJoCo回合。采用标准工具集合的L1，需要选择并使用适合工具，在3000控制步（一分钟、50Hz）内让任务成功条件持续一秒。每策略在相同3003条模拟示范上训练一次、40000更新；观察和初始场景匹配，动作块长度保留各模型设置。干扰工具、L0/L2与真机面板另算。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.02089v1","published_at":"2026-10-01","status":"paper_baselines"},"model_count":7,"result_count":7,"results":[[11767,0.0,370,11768,11769],[11770,0.0,370,11768,11769],[8323,0.04,370,11768,11769],[10171,0.09,370,11768,11769],[7601,0.64,370,11768,11769],[11771,0.0,370,11768,11769],[7603,0.76,370,11768,11769]]},{"id":"humanoidtoolbench-l1-ballretrieve-2026-paper","name":"HumanoidToolBench · BallRetrieve tool use (2026 paper)","group":"Agent 与工具","metric":"Task success rate","unit":"%","public_history":true,"protocol":{"en":"Seven independently trained robot policies use the same 100 seeded MuJoCo episodes per task. This is L1 in the standard tool-set mode, where the policy must select and use a suitable tool; success requires the scenario criterion to hold for one second within 3,000 control steps (one minute at 50 Hz). Every policy trains once for 40,000 updates on the same 3,003 simulation demonstrations. Observation and initial scenes are matched; native action chunk lengths differ. Decoy, L0/L2 and real-robot panels remain separate.","zh":"七个独立训练机器人策略使用每任务相同100个种子MuJoCo回合。采用标准工具集合的L1，需要选择并使用适合工具，在3000控制步（一分钟、50Hz）内让任务成功条件持续一秒。每策略在相同3003条模拟示范上训练一次、40000更新；观察和初始场景匹配，动作块长度保留各模型设置。干扰工具、L0/L2与真机面板另算。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.02089v1","published_at":"2026-10-01","status":"paper_baselines"},"model_count":7,"result_count":7,"results":[[11767,0.0,370,11772,11769],[11770,0.0,370,11772,11769],[8323,0.0,370,11772,11769],[10171,0.0,370,11772,11769],[7601,0.36,370,11772,11769],[11771,0.0,370,11772,11769],[7603,0.57,370,11772,11769]]},{"id":"humanoidtoolbench-l1-icebreak-2026-paper","name":"HumanoidToolBench · IceBreak tool use (2026 paper)","group":"Agent 与工具","metric":"Task success rate","unit":"%","public_history":true,"protocol":{"en":"Seven independently trained robot policies use the same 100 seeded MuJoCo episodes per task. This is L1 in the standard tool-set mode, where the policy must select and use a suitable tool; success requires the scenario criterion to hold for one second within 3,000 control steps (one minute at 50 Hz). Every policy trains once for 40,000 updates on the same 3,003 simulation demonstrations. Observation and initial scenes are matched; native action chunk lengths differ. Decoy, L0/L2 and real-robot panels remain separate.","zh":"七个独立训练机器人策略使用每任务相同100个种子MuJoCo回合。采用标准工具集合的L1，需要选择并使用适合工具，在3000控制步（一分钟、50Hz）内让任务成功条件持续一秒。每策略在相同3003条模拟示范上训练一次、40000更新；观察和初始场景匹配，动作块长度保留各模型设置。干扰工具、L0/L2与真机面板另算。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.02089v1","published_at":"2026-10-01","status":"paper_baselines"},"model_count":7,"result_count":7,"results":[[11767,0.0,370,11773,11769],[11770,0.02,370,11773,11769],[8323,0.12,370,11773,11769],[10171,0.13,370,11773,11769],[7601,0.66,370,11773,11769],[11771,0.01,370,11773,11769],[7603,0.86,370,11773,11769]]},{"id":"longlongtextbench-mean6-2026-paper","name":"LongLongTextBench (2026 paper)","group":"视觉与视频","metric":"Mean-6 OCR target-unit coverage","unit":"score","public_history":true,"protocol":{"en":"Twelve image generators each produce four images for every one of 240 tasks, with no best-of-four selection. Each language × length group has 40 tasks. Qwen2.5-VL-7B OCR extracts English words or Chinese characters; multiset intersection counts target units without ordering or extra-unit penalties. Coverage pools target-unit counts within each group, then averages six groups equally. Native coverage is not exact text accuracy. Ideogram safety-refusal images are executed model behavior and receive zero.","zh":"十二图像生成器对全部240任务各生成四图，不作best-of-four择选。每语言×长度组40任务。Qwen2.5-VL-7B OCR提取英文词或中文字，以多重集合交集统计目标单位，不检查顺序也不直接惩罚额外单位。各组先汇总目标单位覆盖率，再对六组等权平均。保留原生覆盖分，与精确文本正确率区分。Ideogram安全拒绝图属于已执行的模型行为，按原协议计零。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.23374v1","published_at":"2026-10-16","status":"paper_baselines"},"score_max":100,"score_decimals":2,"saturation_excluded":true,"scale_label":{"en":"Native score / 100","zh":"原生分数 / 100"},"model_count":12,"result_count":12,"results":[[6381,55.3,370,11774,11775],[11776,94.3,370,11774,11775],[6366,96.6,370,11774,11775],[7290,99.0,370,11774,11775],[6360,11.8,370,11774,11775],[11777,31.8,370,11774,11775],[11778,40.8,370,11774,11775],[11779,43.9,370,11774,11775],[11780,55.7,370,11774,11775],[11781,63.5,370,11774,11775],[11782,64.9,370,11774,11775],[11783,96.3,370,11774,11775]]},{"id":"into-the-unknown-strict-valid-2026-paper","name":"Into the Unknown (2026 paper)","group":"数学与科学","metric":"Strict valid-submission rate","unit":"%","public_history":true,"protocol":{"en":"Nine model-agent systems each execute all fifteen open scientific follow-up tasks with original paper, data and code, internet access and scientist-specified CPU/GPU resources. No wall-time cap is imposed; up to five agent relaunch rounds are allowed. GPT-6-Astra grades each submission four times, with per-node 3/4 majority and ties failing. A valid paper must satisfy every integrity, completeness and scientist-success node (G1∩G2∩G3). All zeroes represent judged executed submissions, not missing runs. The judge model is also an evaluated agent; source reliability analyses and scientist-audit scope remain part of the paper-baseline interpretation.","zh":"九个模型Agent系统对全部十五个开放科学后续研究任务实际执行，提供原论文、数据、代码、联网和科学家规定的CPU/GPU资源，不设总时长上限，最多重启五轮。GPT-6-Astra对每份成果评分四次，各rubric节点须四次中至少三次通过，平票失败。有效论文须同时满足所有诚信、完整性和科学家成功条件节点（G1∩G2∩G3）。全零来自已执行并已评分的成果，不是缺失运行。裁判模型也在被测Agent中，须结合原文可靠性分析与科学家审查范围理解。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.19159v1","published_at":"2026-10-13","status":"paper_baselines"},"model_count":9,"result_count":9,"results":[[3194,0.0,370,11784,11785],[4983,0.0,370,11784,11785],[4960,0.0,370,11784,11785],[5275,0.0,370,11784,11785],[4979,0.0,370,11786,11785],[4977,0.0,370,11786,11785],[3404,0.0,370,11787,11785],[3155,0.0,370,11788,11785],[3154,0.0,370,11789,11785]]},{"id":"scenepercept-bench-srcc-2026-paper","name":"ScenePercept-Bench (2026 paper)","group":"视觉与视频","metric":"Spearman rank correlation","unit":"raw","public_history":true,"protocol":{"en":"Fifteen source model/method configurations predict scenario-conditioned image-perception scores on the same 2,100-image held-out test set. Ten-thousand-five-hundred images each serve SFT and RL splits. SRCC compares predicted scores with source ground-truth annotations, not a percentage of correct images. Qwen/Gemini annotators and GPT-5.3 refinement generated labels, with a human-annotated subset retained in test; GPT-5.3 is itself evaluated. Starred Q-Align, VisualQuality-R1, RAPO and SCPA are benchmark-trained configurations.","zh":"十五个模型/方法配置在相同2100图保留测试集上预测场景条件下的视觉感知评分。SFT和RL各用10500图。SRCC比较模型分数与原标注的秩相关，不是图像正确百分比。标签由Qwen/Gemini标注及GPT-5.3细化，人工标注子集保留在测试中，GPT-5.3自身也参与评测。带星的Q-Align、VisualQuality-R1、RAPO与SCPA按原训练协议列作配置。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.17661v1","published_at":"2026-10-13","status":"paper_baselines"},"score_decimals":3,"saturation_excluded":true,"scale_label":{"en":"Spearman correlation","zh":"Spearman相关系数"},"model_count":15,"result_count":15,"results":[[7801,0.574,370,11790,11791],[6943,0.712,370,11790,11791],[353,0.638,370,11790,11791],[4959,0.763,370,11790,11791],[2754,0.516,370,11790,11791],[6342,0.504,370,11790,11791],[11792,0.492,370,11790,11791],[11793,0.506,370,11790,11791],[6391,0.537,370,11790,11791],[6343,0.544,370,11790,11791],[9144,0.659,370,11794,11791],[11795,0.585,370,11790,11791],[11796,0.753,370,11794,11791],[11797,0.89,370,11794,11791],[11798,0.907,370,11794,11791]]},{"id":"orbia-world-models-overall-2026-paper","name":"Orbia · world-models (2026 paper)","group":"视觉与视频","metric":"Six-tier composite score","unit":"score","public_history":true,"protocol":{"en":"The full published family panel uses Orbia’s 720 exploration-and-revisit cases: 480 short cases and 240 paired minute-long extensions, with 859 anchor targets. Native interfaces receive camera/action adapters or text instructions with the initial image. The 0–100 composite weights control alignment and visual quality 25% each, input preservation, generated-content persistence and 3D self-consistency 10% each, and temporal stability 20%. Long-horizon components retain the paper’s 30/70 short/long weighting and calibrated metric mappings. World models and text-image-to-video generators remain separate families; this is not a task-success percentage.","zh":"完整原表家族面板使用Orbia的720个探索/重访样本：480个短样本和240个配对分钟扩展，共859锚点目标。通过各自接口转换相机/动作条件或首图加文本。0–100综合分对控制对齐和视觉质量各权重25%，输入保留、生成内容持久性和三维自洽各10%，时间稳定性20%；长时分量保留原文短/长30/70权重与标定映射。世界模型与文图生视频家族分开列组，综合分不作任务成功百分比。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.20693v1","published_at":"2026-10-14","status":"paper_baselines"},"score_max":100,"score_decimals":2,"saturation_excluded":true,"scale_label":{"en":"Native score / 100","zh":"原生分数 / 100"},"model_count":19,"result_count":19,"results":[[11799,76.32,370,11800,11801],[11802,74.77,370,11800,11801],[11803,72.7,370,11800,11801],[11804,72.24,370,11800,11801],[11805,71.95,370,11800,11801],[11806,68.07,370,11800,11801],[7242,67.01,370,11800,11801],[11807,66.72,370,11800,11801],[7243,65.98,370,11800,11801],[7508,65.5,370,11800,11801],[7240,65.08,370,11800,11801],[8428,64.34,370,11800,11801],[8303,62.75,370,11800,11801],[11808,62.73,370,11800,11801],[11809,62.72,370,11800,11801],[7241,61.79,370,11800,11801],[11810,60.64,370,11800,11801],[7244,58.87,370,11800,11801],[11811,52.3,370,11800,11801]]},{"id":"orbia-text-image-video-overall-2026-paper","name":"Orbia · text-image-video (2026 paper)","group":"视觉与视频","metric":"Six-tier composite score","unit":"score","public_history":true,"protocol":{"en":"The full published family panel uses Orbia’s 720 exploration-and-revisit cases: 480 short cases and 240 paired minute-long extensions, with 859 anchor targets. Native interfaces receive camera/action adapters or text instructions with the initial image. The 0–100 composite weights control alignment and visual quality 25% each, input preservation, generated-content persistence and 3D self-consistency 10% each, and temporal stability 20%. Long-horizon components retain the paper’s 30/70 short/long weighting and calibrated metric mappings. World models and text-image-to-video generators remain separate families; this is not a task-success percentage.","zh":"完整原表家族面板使用Orbia的720个探索/重访样本：480个短样本和240个配对分钟扩展，共859锚点目标。通过各自接口转换相机/动作条件或首图加文本。0–100综合分对控制对齐和视觉质量各权重25%，输入保留、生成内容持久性和三维自洽各10%，时间稳定性20%；长时分量保留原文短/长30/70权重与标定映射。世界模型与文图生视频家族分开列组，综合分不作任务成功百分比。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.20693v1","published_at":"2026-10-14","status":"paper_baselines"},"score_max":100,"score_decimals":2,"saturation_excluded":true,"scale_label":{"en":"Native score / 100","zh":"原生分数 / 100"},"model_count":8,"result_count":8,"results":[[11812,67.44,370,11813,11801],[7132,65.23,370,11813,11801],[11814,63.72,370,11813,11801],[11815,63.27,370,11813,11801],[7125,62.0,370,11813,11801],[11816,61.25,370,11813,11801],[7130,60.55,370,11813,11801],[9066,57.18,370,11813,11801]]},{"id":"keep-bench-parallel-allrec-2026-paper","name":"KEEP-Bench · parallel evidence (2026 paper)","group":"Agent 与工具","metric":"Complete-evidence preservation rate","unit":"%","public_history":true,"protocol":{"en":"Sixteen length-controlled compressors share all 900 parallel cases. All-Rec counts a case only when every required evidence proposition is preserved; parallel chains must retain both terminal answers and bind facts to the correct disambiguated entity. Conditional propositions require the stated condition to remain recoverable. GPT-5.2 with high reasoning judges each target proposition once; a 300-decision human audit reports 93% overall exact agreement. The default sentence budget is K=2h for h-hop cases. RECOMP abstractive has no length control and is excluded, while its original source results remain in the complete table. Model-specific output token limits and compression ratios remain as published; this is complete-evidence preservation, not downstream answer accuracy.","zh":"十六个可控制长度的压缩方法使用相同900个parallel样本。All-Rec要求样本的每个必要证据命题均保留；并行链须保留两个终点答案，且事实对应消歧后的正确实体，条件命题须保留其适用条件。GPT-5.2以高推理强度对每命题判断一次，300个判断的人工审查整体精确一致率93%。默认句数预算对h跳样本为K=2h。RECOMP抽象式无法控制长度，因此排除，完整原表仍保留其分数。各模型输出token上限与实际压缩比按原文保留，指标为完整证据保留率，与下游答题准确率区分。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.26827v1","published_at":"2026-10-19","status":"paper_baselines"},"model_count":16,"result_count":16,"results":[[11817,0.0,370,11818,11819],[11820,0.013999999999999999,370,11818,11819],[11821,0.03,370,11818,11819],[1828,0.0,370,11822,11819],[1804,0.027999999999999997,370,11822,11819],[562,0.13,370,11822,11819],[5126,0.14400000000000002,370,11822,11819],[5344,0.23800000000000002,370,11822,11819],[6301,0.33,370,11822,11819],[786,0.42100000000000004,370,11822,11819],[10873,0.023,370,11822,11819],[6140,0.24600000000000002,370,11822,11819],[6240,0.439,370,11822,11819],[6143,0.5760000000000001,370,11822,11819],[4958,0.319,370,11822,11819],[6214,0.638,370,11822,11819]]},{"id":"keep-bench-conditional-allrec-2026-paper","name":"KEEP-Bench · conditional evidence (2026 paper)","group":"Agent 与工具","metric":"Complete-evidence preservation rate","unit":"%","public_history":true,"protocol":{"en":"Sixteen length-controlled compressors share all 1364 conditional cases. All-Rec counts a case only when every required evidence proposition is preserved; parallel chains must retain both terminal answers and bind facts to the correct disambiguated entity. Conditional propositions require the stated condition to remain recoverable. GPT-5.2 with high reasoning judges each target proposition once; a 300-decision human audit reports 93% overall exact agreement. The default sentence budget is K=2h for h-hop cases. RECOMP abstractive has no length control and is excluded, while its original source results remain in the complete table. Model-specific output token limits and compression ratios remain as published; this is complete-evidence preservation, not downstream answer accuracy.","zh":"十六个可控制长度的压缩方法使用相同1364个conditional样本。All-Rec要求样本的每个必要证据命题均保留；并行链须保留两个终点答案，且事实对应消歧后的正确实体，条件命题须保留其适用条件。GPT-5.2以高推理强度对每命题判断一次，300个判断的人工审查整体精确一致率93%。默认句数预算对h跳样本为K=2h。RECOMP抽象式无法控制长度，因此排除，完整原表仍保留其分数。各模型输出token上限与实际压缩比按原文保留，指标为完整证据保留率，与下游答题准确率区分。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.26827v1","published_at":"2026-10-19","status":"paper_baselines"},"model_count":16,"result_count":16,"results":[[11817,0.19899999999999998,370,11818,11819],[11820,0.025,370,11818,11819],[11821,0.045,370,11818,11819],[1828,0.001,370,11822,11819],[1804,0.067,370,11822,11819],[562,0.235,370,11822,11819],[5126,0.266,370,11822,11819],[5344,0.293,370,11822,11819],[6301,0.431,370,11822,11819],[786,0.446,370,11822,11819],[10873,0.083,370,11822,11819],[6140,0.27899999999999997,370,11822,11819],[6240,0.545,370,11822,11819],[6143,0.7120000000000001,370,11822,11819],[4958,0.37,370,11822,11819],[6214,0.7,370,11822,11819]]},{"id":"swe-deceptionbench-neutral-active-2026-paper","name":"SWE-DeceptionBench · active (2026 paper)","group":"安全与可靠性","metric":"Deceptive-response rate","unit":"score","public_history":true,"protocol":{"en":"Seven model endpoints answer the same 120 Chinese/English neutral-condition prompts per deception type, covering six software lifecycle stages. Active prompts contain infeasible or conflicting requirements without explicitly requesting deception; Kimi-K2.6 judges request, thought and response into honest, deceptive, or hallucination/capability failure. Passive prompts explicitly request misleading conduct and the judge labels whether the final answer assists it. The common judge was selected by agreement with human annotations. Native event rates are lower-better and separate from capability failure, reward/pressure conditions and task success.","zh":"七个模型端点对每种类型相同120条中英双语中性条件提示作答，覆盖六个软件生命周期阶段。主动类型在不明示欺骗的情况下给出不可行或冲突要求，Kimi-K2.6根据请求、thought和回复区分诚实、欺骗与幻觉/能力失败；被动类型明确请求误导，按最终回复是否协助判断。统一裁判经人工标签比较后选择。保留越低越好的原生事件率，与能力失败、奖励/压力条件及任务成功率分开。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.13626v1","published_at":"2026-10-09","status":"paper_baselines"},"score_max":100,"score_decimals":2,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"higher_is_better":false,"model_count":7,"result_count":7,"results":[[3194,3.33,370,11823,11824],[3185,74.17,370,11823,11824],[2129,90.83,370,11823,11824],[227,62.5,370,11823,11824],[3196,2.5,370,11823,11824],[3187,80.0,370,11823,11824],[3202,35.83,370,11823,11824]]},{"id":"swe-deceptionbench-neutral-passive-2026-paper","name":"SWE-DeceptionBench · passive (2026 paper)","group":"安全与可靠性","metric":"Deceptive-response rate","unit":"score","public_history":true,"protocol":{"en":"Seven model endpoints answer the same 120 Chinese/English neutral-condition prompts per deception type, covering six software lifecycle stages. Active prompts contain infeasible or conflicting requirements without explicitly requesting deception; Kimi-K2.6 judges request, thought and response into honest, deceptive, or hallucination/capability failure. Passive prompts explicitly request misleading conduct and the judge labels whether the final answer assists it. The common judge was selected by agreement with human annotations. Native event rates are lower-better and separate from capability failure, reward/pressure conditions and task success.","zh":"七个模型端点对每种类型相同120条中英双语中性条件提示作答，覆盖六个软件生命周期阶段。主动类型在不明示欺骗的情况下给出不可行或冲突要求，Kimi-K2.6根据请求、thought和回复区分诚实、欺骗与幻觉/能力失败；被动类型明确请求误导，按最终回复是否协助判断。统一裁判经人工标签比较后选择。保留越低越好的原生事件率，与能力失败、奖励/压力条件及任务成功率分开。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.13626v1","published_at":"2026-10-09","status":"paper_baselines"},"score_max":100,"score_decimals":2,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"higher_is_better":false,"model_count":7,"result_count":7,"results":[[3194,0.0,370,11825,11824],[3185,98.33,370,11825,11824],[2129,90.83,370,11825,11824],[227,60.83,370,11825,11824],[3196,44.17,370,11825,11824],[3187,99.17,370,11825,11824],[3202,56.67,370,11825,11824]]},{"id":"paranoiaeval-evidence-excess-2026-paper","name":"ParanoiaEval · excess risk treatment (2026 paper)","group":"安全与可靠性","metric":"Risk-treatment violation rate","unit":"score","public_history":true,"protocol":{"en":"Eight native coding-agent configurations run all 200 evidence-controlled task pairs from 50 repositories three times (9,600 runs overall). VR uses every evidence-explicit E+ run, 200×3 per model, and counts an unnecessary avoidance, transfer, mitigation or acceptance treatment once per run. It is separate from task completion and evidence-response differences. GPT-5.6 Sol in Codex is the primary human-calibrated behavior judge and also an evaluated configuration. Claude models use Claude Code; GPT models use Codex.","zh":"八个原生代码Agent配置对50仓库的全部200个证据控制任务对各运行三次，总计9600回合。VR以每模型200×3个证据明确的E+回合为分母，按回合统计不必要的回避、转移、缓解或接受处理，与任务完成度及证据响应差值分开。Codex中的GPT-5.6Sol为经人工校准的主要行为裁判，也作为被测配置。Claude使用ClaudeCode，GPT使用Codex。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.08662v1","published_at":"2026-10-06","status":"paper_baselines"},"score_max":100,"score_decimals":2,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"higher_is_better":false,"model_count":8,"result_count":8,"results":[[11826,58.7,370,11827,11828],[7537,22.2,370,11827,11828],[10537,11.2,370,11827,11828],[10538,17.7,370,11827,11828],[4979,21.0,370,11829,11828],[5727,18.2,370,11829,11828],[5616,19.2,370,11829,11828],[3196,25.5,370,11829,11828]]},{"id":"reqclarifybench-persona-mean-align-2026-paper","name":"ReqClarifyBench · requirement alignment (2026 paper)","group":"Agent 与工具","metric":"Atomic requirement alignment","unit":"raw","score_decimals":3,"saturation_excluded":true,"scale_label":{"en":"Alignment −1 to +1","zh":"对齐分 −1至+1"},"public_history":true,"protocol":{"en":"Six standalone LLMs clarify all 341 repository-level requirements tasks under Expert Nitpicker, Vague Requester and Mind Changer personas, with up to twenty interaction rounds. The same GPT-5.6-Sol user simulator has the human-validated detailed requirements and answers at temperature 0.7; targets use temperature 0. DeepSeek-V4.1-Flash judges atomic requirements: covered=+1, omitted=0, violated=−1. ReqAlign averages over requirements in the document, and the source Avg column equally averages the three personas. This signed native alignment score ranges −1 to +1; it is separate from question effectiveness, BERTScore and generated-code success. The simulator model is itself one evaluated target.","zh":"六个独立LLM对全部341个仓库级需求任务分别与挑剔专家、模糊请求者和改变主意三种用户交互，最多二十轮。统一GPT-5.6Sol用户模拟器掌握人工验证的详细需求，以温度0.7回答；被测模型温度0。DeepSeek-V4.1Flash逐原子需求判断：覆盖+1、遗漏0、违背−1。ReqAlign先对文档原子需求平均，原表Avg再对三种用户等权平均。保留−1至+1原生需求对齐分，与提问有效率、BERTScore和生成代码成功分开。模拟器模型本身也在被测目标中。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.13022v1","published_at":"2026-10-09","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[4979,0.727,370,11830,11831],[755,0.577,370,11830,11831],[502,0.357,370,11830,11831],[3185,0.498,370,11830,11831],[784,0.282,370,11830,11831],[3154,0.521,370,11830,11831]]},{"id":"wpbench-wind-farm-a-short-2026-paper","name":"WPBench · Wind Farm A short (2026 paper)","group":"预测与决策","metric":"Normalized MAE","unit":"raw","public_history":true,"protocol":{"en":"Wind-power forecasting models use the same published splits and non-overlapping rolling test windows for Wind Farm A. The source Short row averages 12/24-step horizons with 576-step history. Trainable models tune only on validation data. Native nMAE is dimensionless, lower is better; structurally unsupported methods are missing, not scored zero. Dataset and horizon-group cohorts stay separate.","zh":"风电预测模型在Wind Farm A上使用相同数据划分与不重叠的滚动测试窗口。Short行平均12/24步预测，历史窗口576步；可训练模型只在验证集调参。保留无量纲nMAE，数值越低越好。结构上不适用的方法缺失，不补零；数据集及预测跨度分别排行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.24444v1","published_at":"2026-09-21","status":"paper_baselines"},"higher_is_better":false,"model_count":15,"result_count":15,"results":[[9464,0.096,370,11832,11833],[11834,0.095,370,11832,11833],[11835,0.1,370,11832,11833],[9161,0.107,370,11832,11833],[9682,0.104,370,11832,11833],[11836,0.089,370,11832,11833],[6812,0.096,370,11832,11833],[11837,0.101,370,11832,11833],[11838,0.091,370,11832,11833],[11839,0.081,370,11832,11833],[11840,0.1,370,11832,11833],[11841,0.102,370,11832,11833],[11842,0.11,370,11832,11833],[11843,0.107,370,11832,11833],[11844,0.082,370,11832,11833]]},{"id":"wpbench-wind-farm-a-long-2026-paper","name":"WPBench · Wind Farm A long (2026 paper)","group":"预测与决策","metric":"Normalized MAE","unit":"raw","public_history":true,"protocol":{"en":"Wind-power forecasting models use the same published splits and non-overlapping rolling test windows for Wind Farm A. The source Long row averages 72/144-step horizons with 576-step history. Trainable models tune only on validation data. Native nMAE is dimensionless, lower is better; structurally unsupported methods are missing, not scored zero. Dataset and horizon-group cohorts stay separate.","zh":"风电预测模型在Wind Farm A上使用相同数据划分与不重叠的滚动测试窗口。Long行平均72/144步预测，历史窗口576步；可训练模型只在验证集调参。保留无量纲nMAE，数值越低越好。结构上不适用的方法缺失，不补零；数据集及预测跨度分别排行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.24444v1","published_at":"2026-09-21","status":"paper_baselines"},"higher_is_better":false,"model_count":15,"result_count":15,"results":[[9464,0.179,370,11845,11833],[11834,0.168,370,11845,11833],[11835,0.167,370,11845,11833],[9161,0.183,370,11845,11833],[9682,0.178,370,11845,11833],[11836,0.171,370,11845,11833],[6812,0.179,370,11845,11833],[11837,0.212,370,11845,11833],[11838,0.173,370,11845,11833],[11839,0.167,370,11845,11833],[11840,0.188,370,11845,11833],[11841,0.185,370,11845,11833],[11842,0.195,370,11845,11833],[11843,0.214,370,11845,11833],[11844,0.17,370,11845,11833]]},{"id":"wpbench-wind-farm-b-short-2026-paper","name":"WPBench · Wind Farm B short (2026 paper)","group":"预测与决策","metric":"Normalized MAE","unit":"raw","public_history":true,"protocol":{"en":"Wind-power forecasting models use the same published splits and non-overlapping rolling test windows for Wind Farm B. The source Short row averages 12/24-step horizons with 576-step history. Trainable models tune only on validation data. Native nMAE is dimensionless, lower is better; structurally unsupported methods are missing, not scored zero. Dataset and horizon-group cohorts stay separate.","zh":"风电预测模型在Wind Farm B上使用相同数据划分与不重叠的滚动测试窗口。Short行平均12/24步预测，历史窗口576步；可训练模型只在验证集调参。保留无量纲nMAE，数值越低越好。结构上不适用的方法缺失，不补零；数据集及预测跨度分别排行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.24444v1","published_at":"2026-09-21","status":"paper_baselines"},"higher_is_better":false,"model_count":15,"result_count":15,"results":[[9464,0.098,370,11832,11833],[11834,0.106,370,11832,11833],[11835,0.105,370,11832,11833],[9161,0.13,370,11832,11833],[9682,0.124,370,11832,11833],[11836,0.11,370,11832,11833],[6812,0.095,370,11832,11833],[11837,0.108,370,11832,11833],[11838,0.1,370,11832,11833],[11839,0.093,370,11832,11833],[11840,0.112,370,11832,11833],[11841,0.116,370,11832,11833],[11842,0.114,370,11832,11833],[11843,0.137,370,11832,11833],[11844,0.093,370,11832,11833]]},{"id":"wpbench-wind-farm-b-long-2026-paper","name":"WPBench · Wind Farm B long (2026 paper)","group":"预测与决策","metric":"Normalized MAE","unit":"raw","public_history":true,"protocol":{"en":"Wind-power forecasting models use the same published splits and non-overlapping rolling test windows for Wind Farm B. The source Long row averages 72/144-step horizons with 576-step history. Trainable models tune only on validation data. Native nMAE is dimensionless, lower is better; structurally unsupported methods are missing, not scored zero. Dataset and horizon-group cohorts stay separate.","zh":"风电预测模型在Wind Farm B上使用相同数据划分与不重叠的滚动测试窗口。Long行平均72/144步预测，历史窗口576步；可训练模型只在验证集调参。保留无量纲nMAE，数值越低越好。结构上不适用的方法缺失，不补零；数据集及预测跨度分别排行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.24444v1","published_at":"2026-09-21","status":"paper_baselines"},"higher_is_better":false,"model_count":15,"result_count":15,"results":[[9464,0.17,370,11845,11833],[11834,0.173,370,11845,11833],[11835,0.179,370,11845,11833],[9161,0.186,370,11845,11833],[9682,0.19,370,11845,11833],[11836,0.206,370,11845,11833],[6812,0.165,370,11845,11833],[11837,0.224,370,11845,11833],[11838,0.184,370,11845,11833],[11839,0.178,370,11845,11833],[11840,0.186,370,11845,11833],[11841,0.173,370,11845,11833],[11842,0.182,370,11845,11833],[11843,0.226,370,11845,11833],[11844,0.17,370,11845,11833]]},{"id":"wpbench-chalmers-short-2026-paper","name":"WPBench · Chalmers short (2026 paper)","group":"预测与决策","metric":"Normalized MAE","unit":"raw","public_history":true,"protocol":{"en":"Wind-power forecasting models use the same published splits and non-overlapping rolling test windows for Chalmers. The source Short row averages 12/24-step horizons with 576-step history. Trainable models tune only on validation data. Native nMAE is dimensionless, lower is better; structurally unsupported methods are missing, not scored zero. Dataset and horizon-group cohorts stay separate.","zh":"风电预测模型在Chalmers上使用相同数据划分与不重叠的滚动测试窗口。Short行平均12/24步预测，历史窗口576步；可训练模型只在验证集调参。保留无量纲nMAE，数值越低越好。结构上不适用的方法缺失，不补零；数据集及预测跨度分别排行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.24444v1","published_at":"2026-09-21","status":"paper_baselines"},"higher_is_better":false,"model_count":15,"result_count":15,"results":[[9464,0.003,370,11832,11833],[11834,0.002,370,11832,11833],[11835,0.001,370,11832,11833],[9161,0.003,370,11832,11833],[9682,0.004,370,11832,11833],[11836,0.001,370,11832,11833],[6812,0.004,370,11832,11833],[11837,0.002,370,11832,11833],[11838,0.001,370,11832,11833],[11839,0.001,370,11832,11833],[11840,0.002,370,11832,11833],[11841,0.005,370,11832,11833],[11842,0.006,370,11832,11833],[11843,0.722,370,11832,11833],[11844,0.001,370,11832,11833]]},{"id":"wpbench-chalmers-long-2026-paper","name":"WPBench · Chalmers long (2026 paper)","group":"预测与决策","metric":"Normalized MAE","unit":"raw","public_history":true,"protocol":{"en":"Wind-power forecasting models use the same published splits and non-overlapping rolling test windows for Chalmers. The source Long row averages 72/144-step horizons with 576-step history. Trainable models tune only on validation data. Native nMAE is dimensionless, lower is better; structurally unsupported methods are missing, not scored zero. Dataset and horizon-group cohorts stay separate.","zh":"风电预测模型在Chalmers上使用相同数据划分与不重叠的滚动测试窗口。Long行平均72/144步预测，历史窗口576步；可训练模型只在验证集调参。保留无量纲nMAE，数值越低越好。结构上不适用的方法缺失，不补零；数据集及预测跨度分别排行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.24444v1","published_at":"2026-09-21","status":"paper_baselines"},"higher_is_better":false,"model_count":15,"result_count":15,"results":[[9464,0.011,370,11845,11833],[11834,0.004,370,11845,11833],[11835,0.004,370,11845,11833],[9161,0.004,370,11845,11833],[9682,0.004,370,11845,11833],[11836,0.004,370,11845,11833],[6812,0.014,370,11845,11833],[11837,0.005,370,11845,11833],[11838,0.008,370,11845,11833],[11839,0.005,370,11845,11833],[11840,0.004,370,11845,11833],[11841,0.013,370,11845,11833],[11842,0.008,370,11845,11833],[11843,0.568,370,11845,11833],[11844,0.005,370,11845,11833]]},{"id":"wpbench-maelstrom-1-short-2026-paper","name":"WPBench · Maelstrom 1 short (2026 paper)","group":"预测与决策","metric":"Normalized MAE","unit":"raw","public_history":true,"protocol":{"en":"Wind-power forecasting models use the same published splits and non-overlapping rolling test windows for Maelstrom 1. The source Short row averages 12/24-step horizons with 576-step history. Trainable models tune only on validation data. Native nMAE is dimensionless, lower is better; structurally unsupported methods are missing, not scored zero. Dataset and horizon-group cohorts stay separate.","zh":"风电预测模型在Maelstrom 1上使用相同数据划分与不重叠的滚动测试窗口。Short行平均12/24步预测，历史窗口576步；可训练模型只在验证集调参。保留无量纲nMAE，数值越低越好。结构上不适用的方法缺失，不补零；数据集及预测跨度分别排行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.24444v1","published_at":"2026-09-21","status":"paper_baselines"},"higher_is_better":false,"model_count":15,"result_count":15,"results":[[9464,0.069,370,11832,11833],[11834,0.068,370,11832,11833],[11835,0.072,370,11832,11833],[9161,0.086,370,11832,11833],[9682,0.076,370,11832,11833],[11836,0.069,370,11832,11833],[6812,0.069,370,11832,11833],[11837,0.071,370,11832,11833],[11838,0.075,370,11832,11833],[11839,0.067,370,11832,11833],[11840,0.072,370,11832,11833],[11841,0.072,370,11832,11833],[11842,0.07,370,11832,11833],[11843,0.072,370,11832,11833],[11844,0.065,370,11832,11833]]},{"id":"wpbench-maelstrom-1-long-2026-paper","name":"WPBench · Maelstrom 1 long (2026 paper)","group":"预测与决策","metric":"Normalized MAE","unit":"raw","public_history":true,"protocol":{"en":"Wind-power forecasting models use the same published splits and non-overlapping rolling test windows for Maelstrom 1. The source Long row averages 72/144-step horizons with 576-step history. Trainable models tune only on validation data. Native nMAE is dimensionless, lower is better; structurally unsupported methods are missing, not scored zero. Dataset and horizon-group cohorts stay separate.","zh":"风电预测模型在Maelstrom 1上使用相同数据划分与不重叠的滚动测试窗口。Long行平均72/144步预测，历史窗口576步；可训练模型只在验证集调参。保留无量纲nMAE，数值越低越好。结构上不适用的方法缺失，不补零；数据集及预测跨度分别排行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.24444v1","published_at":"2026-09-21","status":"paper_baselines"},"higher_is_better":false,"model_count":15,"result_count":15,"results":[[9464,0.117,370,11845,11833],[11834,0.114,370,11845,11833],[11835,0.111,370,11845,11833],[9161,0.12,370,11845,11833],[9682,0.117,370,11845,11833],[11836,0.12,370,11845,11833],[6812,0.118,370,11845,11833],[11837,0.133,370,11845,11833],[11838,0.115,370,11845,11833],[11839,0.11,370,11845,11833],[11840,0.125,370,11845,11833],[11841,0.118,370,11845,11833],[11842,0.124,370,11845,11833],[11843,0.121,370,11845,11833],[11844,0.117,370,11845,11833]]},{"id":"wpbench-maelstrom-2-short-2026-paper","name":"WPBench · Maelstrom 2 short (2026 paper)","group":"预测与决策","metric":"Normalized MAE","unit":"raw","public_history":true,"protocol":{"en":"Wind-power forecasting models use the same published splits and non-overlapping rolling test windows for Maelstrom 2. The source Short row averages 12/24-step horizons with 576-step history. Trainable models tune only on validation data. Native nMAE is dimensionless, lower is better; structurally unsupported methods are missing, not scored zero. Dataset and horizon-group cohorts stay separate.","zh":"风电预测模型在Maelstrom 2上使用相同数据划分与不重叠的滚动测试窗口。Short行平均12/24步预测，历史窗口576步；可训练模型只在验证集调参。保留无量纲nMAE，数值越低越好。结构上不适用的方法缺失，不补零；数据集及预测跨度分别排行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.24444v1","published_at":"2026-09-21","status":"paper_baselines"},"higher_is_better":false,"model_count":15,"result_count":15,"results":[[9464,0.06,370,11832,11833],[11834,0.059,370,11832,11833],[11835,0.06,370,11832,11833],[9161,0.071,370,11832,11833],[9682,0.065,370,11832,11833],[11836,0.058,370,11832,11833],[6812,0.06,370,11832,11833],[11837,0.059,370,11832,11833],[11838,0.061,370,11832,11833],[11839,0.057,370,11832,11833],[11840,0.06,370,11832,11833],[11841,0.059,370,11832,11833],[11842,0.06,370,11832,11833],[11843,0.061,370,11832,11833],[11844,0.056,370,11832,11833]]},{"id":"wpbench-maelstrom-2-long-2026-paper","name":"WPBench · Maelstrom 2 long (2026 paper)","group":"预测与决策","metric":"Normalized MAE","unit":"raw","public_history":true,"protocol":{"en":"Wind-power forecasting models use the same published splits and non-overlapping rolling test windows for Maelstrom 2. The source Long row averages 72/144-step horizons with 576-step history. Trainable models tune only on validation data. Native nMAE is dimensionless, lower is better; structurally unsupported methods are missing, not scored zero. Dataset and horizon-group cohorts stay separate.","zh":"风电预测模型在Maelstrom 2上使用相同数据划分与不重叠的滚动测试窗口。Long行平均72/144步预测，历史窗口576步；可训练模型只在验证集调参。保留无量纲nMAE，数值越低越好。结构上不适用的方法缺失，不补零；数据集及预测跨度分别排行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.24444v1","published_at":"2026-09-21","status":"paper_baselines"},"higher_is_better":false,"model_count":15,"result_count":15,"results":[[9464,0.093,370,11845,11833],[11834,0.088,370,11845,11833],[11835,0.086,370,11845,11833],[9161,0.095,370,11845,11833],[9682,0.09,370,11845,11833],[11836,0.092,370,11845,11833],[6812,0.094,370,11845,11833],[11837,0.099,370,11845,11833],[11838,0.089,370,11845,11833],[11839,0.087,370,11845,11833],[11840,0.095,370,11845,11833],[11841,0.094,370,11845,11833],[11842,0.094,370,11845,11833],[11843,0.098,370,11845,11833],[11844,0.093,370,11845,11833]]},{"id":"wpbench-maelstrom-3-short-2026-paper","name":"WPBench · Maelstrom 3 short (2026 paper)","group":"预测与决策","metric":"Normalized MAE","unit":"raw","public_history":true,"protocol":{"en":"Wind-power forecasting models use the same published splits and non-overlapping rolling test windows for Maelstrom 3. The source Short row averages 12/24-step horizons with 576-step history. Trainable models tune only on validation data. Native nMAE is dimensionless, lower is better; structurally unsupported methods are missing, not scored zero. Dataset and horizon-group cohorts stay separate.","zh":"风电预测模型在Maelstrom 3上使用相同数据划分与不重叠的滚动测试窗口。Short行平均12/24步预测，历史窗口576步；可训练模型只在验证集调参。保留无量纲nMAE，数值越低越好。结构上不适用的方法缺失，不补零；数据集及预测跨度分别排行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.24444v1","published_at":"2026-09-21","status":"paper_baselines"},"higher_is_better":false,"model_count":15,"result_count":15,"results":[[9464,0.078,370,11832,11833],[11834,0.076,370,11832,11833],[11835,0.08,370,11832,11833],[9161,0.085,370,11832,11833],[9682,0.084,370,11832,11833],[11836,0.076,370,11832,11833],[6812,0.077,370,11832,11833],[11837,0.076,370,11832,11833],[11838,0.078,370,11832,11833],[11839,0.075,370,11832,11833],[11840,0.08,370,11832,11833],[11841,0.077,370,11832,11833],[11842,0.079,370,11832,11833],[11843,0.079,370,11832,11833],[11844,0.074,370,11832,11833]]},{"id":"wpbench-maelstrom-3-long-2026-paper","name":"WPBench · Maelstrom 3 long (2026 paper)","group":"预测与决策","metric":"Normalized MAE","unit":"raw","public_history":true,"protocol":{"en":"Wind-power forecasting models use the same published splits and non-overlapping rolling test windows for Maelstrom 3. The source Long row averages 72/144-step horizons with 576-step history. Trainable models tune only on validation data. Native nMAE is dimensionless, lower is better; structurally unsupported methods are missing, not scored zero. Dataset and horizon-group cohorts stay separate.","zh":"风电预测模型在Maelstrom 3上使用相同数据划分与不重叠的滚动测试窗口。Long行平均72/144步预测，历史窗口576步；可训练模型只在验证集调参。保留无量纲nMAE，数值越低越好。结构上不适用的方法缺失，不补零；数据集及预测跨度分别排行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.24444v1","published_at":"2026-09-21","status":"paper_baselines"},"higher_is_better":false,"model_count":15,"result_count":15,"results":[[9464,0.123,370,11845,11833],[11834,0.121,370,11845,11833],[11835,0.119,370,11845,11833],[9161,0.13,370,11845,11833],[9682,0.13,370,11845,11833],[11836,0.122,370,11845,11833],[6812,0.123,370,11845,11833],[11837,0.138,370,11845,11833],[11838,0.123,370,11845,11833],[11839,0.118,370,11845,11833],[11840,0.131,370,11845,11833],[11841,0.125,370,11845,11833],[11842,0.127,370,11845,11833],[11843,0.127,370,11845,11833],[11844,0.127,370,11845,11833]]},{"id":"wpbench-maelstrom-4-short-2026-paper","name":"WPBench · Maelstrom 4 short (2026 paper)","group":"预测与决策","metric":"Normalized MAE","unit":"raw","public_history":true,"protocol":{"en":"Wind-power forecasting models use the same published splits and non-overlapping rolling test windows for Maelstrom 4. The source Short row averages 12/24-step horizons with 576-step history. Trainable models tune only on validation data. Native nMAE is dimensionless, lower is better; structurally unsupported methods are missing, not scored zero. Dataset and horizon-group cohorts stay separate.","zh":"风电预测模型在Maelstrom 4上使用相同数据划分与不重叠的滚动测试窗口。Short行平均12/24步预测，历史窗口576步；可训练模型只在验证集调参。保留无量纲nMAE，数值越低越好。结构上不适用的方法缺失，不补零；数据集及预测跨度分别排行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.24444v1","published_at":"2026-09-21","status":"paper_baselines"},"higher_is_better":false,"model_count":15,"result_count":15,"results":[[9464,0.076,370,11832,11833],[11834,0.074,370,11832,11833],[11835,0.078,370,11832,11833],[9161,0.081,370,11832,11833],[9682,0.081,370,11832,11833],[11836,0.075,370,11832,11833],[6812,0.076,370,11832,11833],[11837,0.076,370,11832,11833],[11838,0.078,370,11832,11833],[11839,0.074,370,11832,11833],[11840,0.078,370,11832,11833],[11841,0.079,370,11832,11833],[11842,0.077,370,11832,11833],[11843,0.078,370,11832,11833],[11844,0.072,370,11832,11833]]},{"id":"wpbench-maelstrom-4-long-2026-paper","name":"WPBench · Maelstrom 4 long (2026 paper)","group":"预测与决策","metric":"Normalized MAE","unit":"raw","public_history":true,"protocol":{"en":"Wind-power forecasting models use the same published splits and non-overlapping rolling test windows for Maelstrom 4. The source Long row averages 72/144-step horizons with 576-step history. Trainable models tune only on validation data. Native nMAE is dimensionless, lower is better; structurally unsupported methods are missing, not scored zero. Dataset and horizon-group cohorts stay separate.","zh":"风电预测模型在Maelstrom 4上使用相同数据划分与不重叠的滚动测试窗口。Long行平均72/144步预测，历史窗口576步；可训练模型只在验证集调参。保留无量纲nMAE，数值越低越好。结构上不适用的方法缺失，不补零；数据集及预测跨度分别排行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.24444v1","published_at":"2026-09-21","status":"paper_baselines"},"higher_is_better":false,"model_count":15,"result_count":15,"results":[[9464,0.126,370,11845,11833],[11834,0.123,370,11845,11833],[11835,0.122,370,11845,11833],[9161,0.134,370,11845,11833],[9682,0.129,370,11845,11833],[11836,0.126,370,11845,11833],[6812,0.128,370,11845,11833],[11837,0.144,370,11845,11833],[11838,0.123,370,11845,11833],[11839,0.121,370,11845,11833],[11840,0.136,370,11845,11833],[11841,0.127,370,11845,11833],[11842,0.133,370,11845,11833],[11843,0.135,370,11845,11833],[11844,0.132,370,11845,11833]]},{"id":"wpbench-wind-farm-site-1-short-2026-paper","name":"WPBench · Wind Farm Site 1 short (2026 paper)","group":"预测与决策","metric":"Normalized MAE","unit":"raw","public_history":true,"protocol":{"en":"Wind-power forecasting models use the same published splits and non-overlapping rolling test windows for Wind Farm Site 1. The source Short row averages 12/24-step horizons with 576-step history. Trainable models tune only on validation data. Native nMAE is dimensionless, lower is better; structurally unsupported methods are missing, not scored zero. Dataset and horizon-group cohorts stay separate.","zh":"风电预测模型在Wind Farm Site 1上使用相同数据划分与不重叠的滚动测试窗口。Short行平均12/24步预测，历史窗口576步；可训练模型只在验证集调参。保留无量纲nMAE，数值越低越好。结构上不适用的方法缺失，不补零；数据集及预测跨度分别排行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.24444v1","published_at":"2026-09-21","status":"paper_baselines"},"higher_is_better":false,"model_count":16,"result_count":16,"results":[[9464,0.096,370,11846,11833],[11834,0.096,370,11846,11833],[11835,0.129,370,11846,11833],[9161,0.109,370,11846,11833],[9682,0.117,370,11846,11833],[11836,0.098,370,11846,11833],[6812,0.096,370,11846,11833],[11847,0.095,370,11846,11833],[11837,0.103,370,11846,11833],[11838,0.099,370,11846,11833],[11839,0.092,370,11846,11833],[11840,0.106,370,11846,11833],[11841,0.101,370,11846,11833],[11842,0.108,370,11846,11833],[11843,0.118,370,11846,11833],[11844,0.09,370,11846,11833]]},{"id":"wpbench-wind-farm-site-1-long-2026-paper","name":"WPBench · Wind Farm Site 1 long (2026 paper)","group":"预测与决策","metric":"Normalized MAE","unit":"raw","public_history":true,"protocol":{"en":"Wind-power forecasting models use the same published splits and non-overlapping rolling test windows for Wind Farm Site 1. The source Long row averages 72/144-step horizons with 576-step history. Trainable models tune only on validation data. Native nMAE is dimensionless, lower is better; structurally unsupported methods are missing, not scored zero. Dataset and horizon-group cohorts stay separate.","zh":"风电预测模型在Wind Farm Site 1上使用相同数据划分与不重叠的滚动测试窗口。Long行平均72/144步预测，历史窗口576步；可训练模型只在验证集调参。保留无量纲nMAE，数值越低越好。结构上不适用的方法缺失，不补零；数据集及预测跨度分别排行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.24444v1","published_at":"2026-09-21","status":"paper_baselines"},"higher_is_better":false,"model_count":16,"result_count":16,"results":[[9464,0.172,370,11848,11833],[11834,0.17,370,11848,11833],[11835,0.182,370,11848,11833],[9161,0.186,370,11848,11833],[9682,0.191,370,11848,11833],[11836,0.187,370,11848,11833],[6812,0.167,370,11848,11833],[11847,0.167,370,11848,11833],[11837,0.201,370,11848,11833],[11838,0.173,370,11848,11833],[11839,0.167,370,11848,11833],[11840,0.185,370,11848,11833],[11841,0.174,370,11848,11833],[11842,0.175,370,11848,11833],[11843,0.188,370,11848,11833],[11844,0.191,370,11848,11833]]},{"id":"wpbench-wind-farm-site-2-short-2026-paper","name":"WPBench · Wind Farm Site 2 short (2026 paper)","group":"预测与决策","metric":"Normalized MAE","unit":"raw","public_history":true,"protocol":{"en":"Wind-power forecasting models use the same published splits and non-overlapping rolling test windows for Wind Farm Site 2. The source Short row averages 12/24-step horizons with 576-step history. Trainable models tune only on validation data. Native nMAE is dimensionless, lower is better; structurally unsupported methods are missing, not scored zero. Dataset and horizon-group cohorts stay separate.","zh":"风电预测模型在Wind Farm Site 2上使用相同数据划分与不重叠的滚动测试窗口。Short行平均12/24步预测，历史窗口576步；可训练模型只在验证集调参。保留无量纲nMAE，数值越低越好。结构上不适用的方法缺失，不补零；数据集及预测跨度分别排行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.24444v1","published_at":"2026-09-21","status":"paper_baselines"},"higher_is_better":false,"model_count":16,"result_count":16,"results":[[9464,0.108,370,11846,11833],[11834,0.107,370,11846,11833],[11835,0.16,370,11846,11833],[9161,0.128,370,11846,11833],[9682,0.126,370,11846,11833],[11836,0.109,370,11846,11833],[6812,0.105,370,11846,11833],[11847,0.104,370,11846,11833],[11837,0.112,370,11846,11833],[11838,0.107,370,11846,11833],[11839,0.095,370,11846,11833],[11840,0.116,370,11846,11833],[11841,0.12,370,11846,11833],[11842,0.115,370,11846,11833],[11843,0.136,370,11846,11833],[11844,0.094,370,11846,11833]]},{"id":"wpbench-wind-farm-site-2-long-2026-paper","name":"WPBench · Wind Farm Site 2 long (2026 paper)","group":"预测与决策","metric":"Normalized MAE","unit":"raw","public_history":true,"protocol":{"en":"Wind-power forecasting models use the same published splits and non-overlapping rolling test windows for Wind Farm Site 2. The source Long row averages 72/144-step horizons with 576-step history. Trainable models tune only on validation data. Native nMAE is dimensionless, lower is better; structurally unsupported methods are missing, not scored zero. Dataset and horizon-group cohorts stay separate.","zh":"风电预测模型在Wind Farm Site 2上使用相同数据划分与不重叠的滚动测试窗口。Long行平均72/144步预测，历史窗口576步；可训练模型只在验证集调参。保留无量纲nMAE，数值越低越好。结构上不适用的方法缺失，不补零；数据集及预测跨度分别排行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.24444v1","published_at":"2026-09-21","status":"paper_baselines"},"higher_is_better":false,"model_count":16,"result_count":16,"results":[[9464,0.209,370,11848,11833],[11834,0.212,370,11848,11833],[11835,0.233,370,11848,11833],[9161,0.22,370,11848,11833],[9682,0.222,370,11848,11833],[11836,0.218,370,11848,11833],[6812,0.198,370,11848,11833],[11847,0.209,370,11848,11833],[11837,0.245,370,11848,11833],[11838,0.213,370,11848,11833],[11839,0.203,370,11848,11833],[11840,0.22,370,11848,11833],[11841,0.214,370,11848,11833],[11842,0.214,370,11848,11833],[11843,0.235,370,11848,11833],[11844,0.237,370,11848,11833]]},{"id":"wpbench-wind-farm-site-3-short-2026-paper","name":"WPBench · Wind Farm Site 3 short (2026 paper)","group":"预测与决策","metric":"Normalized MAE","unit":"raw","public_history":true,"protocol":{"en":"Wind-power forecasting models use the same published splits and non-overlapping rolling test windows for Wind Farm Site 3. The source Short row averages 12/24-step horizons with 576-step history. Trainable models tune only on validation data. Native nMAE is dimensionless, lower is better; structurally unsupported methods are missing, not scored zero. Dataset and horizon-group cohorts stay separate.","zh":"风电预测模型在Wind Farm Site 3上使用相同数据划分与不重叠的滚动测试窗口。Short行平均12/24步预测，历史窗口576步；可训练模型只在验证集调参。保留无量纲nMAE，数值越低越好。结构上不适用的方法缺失，不补零；数据集及预测跨度分别排行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.24444v1","published_at":"2026-09-21","status":"paper_baselines"},"higher_is_better":false,"model_count":16,"result_count":16,"results":[[9464,0.067,370,11846,11833],[11834,0.066,370,11846,11833],[11835,0.098,370,11846,11833],[9161,0.083,370,11846,11833],[9682,0.078,370,11846,11833],[11836,0.067,370,11846,11833],[6812,0.069,370,11846,11833],[11847,0.068,370,11846,11833],[11837,0.064,370,11846,11833],[11838,0.064,370,11846,11833],[11839,0.061,370,11846,11833],[11840,0.07,370,11846,11833],[11841,0.075,370,11846,11833],[11842,0.073,370,11846,11833],[11843,0.104,370,11846,11833],[11844,0.059,370,11846,11833]]},{"id":"wpbench-wind-farm-site-3-long-2026-paper","name":"WPBench · Wind Farm Site 3 long (2026 paper)","group":"预测与决策","metric":"Normalized MAE","unit":"raw","public_history":true,"protocol":{"en":"Wind-power forecasting models use the same published splits and non-overlapping rolling test windows for Wind Farm Site 3. The source Long row averages 72/144-step horizons with 576-step history. Trainable models tune only on validation data. Native nMAE is dimensionless, lower is better; structurally unsupported methods are missing, not scored zero. Dataset and horizon-group cohorts stay separate.","zh":"风电预测模型在Wind Farm Site 3上使用相同数据划分与不重叠的滚动测试窗口。Long行平均72/144步预测，历史窗口576步；可训练模型只在验证集调参。保留无量纲nMAE，数值越低越好。结构上不适用的方法缺失，不补零；数据集及预测跨度分别排行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.24444v1","published_at":"2026-09-21","status":"paper_baselines"},"higher_is_better":false,"model_count":16,"result_count":16,"results":[[9464,0.133,370,11848,11833],[11834,0.125,370,11848,11833],[11835,0.125,370,11848,11833],[9161,0.131,370,11848,11833],[9682,0.129,370,11848,11833],[11836,0.126,370,11848,11833],[6812,0.143,370,11848,11833],[11847,0.133,370,11848,11833],[11837,0.137,370,11848,11833],[11838,0.128,370,11848,11833],[11839,0.118,370,11848,11833],[11840,0.136,370,11848,11833],[11841,0.138,370,11848,11833],[11842,0.139,370,11848,11833],[11843,0.173,370,11848,11833],[11844,0.118,370,11848,11833]]},{"id":"wpbench-wind-farm-site-4-short-2026-paper","name":"WPBench · Wind Farm Site 4 short (2026 paper)","group":"预测与决策","metric":"Normalized MAE","unit":"raw","public_history":true,"protocol":{"en":"Wind-power forecasting models use the same published splits and non-overlapping rolling test windows for Wind Farm Site 4. The source Short row averages 12/24-step horizons with 576-step history. Trainable models tune only on validation data. Native nMAE is dimensionless, lower is better; structurally unsupported methods are missing, not scored zero. Dataset and horizon-group cohorts stay separate.","zh":"风电预测模型在Wind Farm Site 4上使用相同数据划分与不重叠的滚动测试窗口。Short行平均12/24步预测，历史窗口576步；可训练模型只在验证集调参。保留无量纲nMAE，数值越低越好。结构上不适用的方法缺失，不补零；数据集及预测跨度分别排行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.24444v1","published_at":"2026-09-21","status":"paper_baselines"},"higher_is_better":false,"model_count":16,"result_count":16,"results":[[9464,0.073,370,11846,11833],[11834,0.076,370,11846,11833],[11835,0.124,370,11846,11833],[9161,0.102,370,11846,11833],[9682,0.102,370,11846,11833],[11836,0.074,370,11846,11833],[6812,0.079,370,11846,11833],[11847,0.074,370,11846,11833],[11837,0.079,370,11846,11833],[11838,0.073,370,11846,11833],[11839,0.068,370,11846,11833],[11840,0.089,370,11846,11833],[11841,0.083,370,11846,11833],[11842,0.081,370,11846,11833],[11843,0.102,370,11846,11833],[11844,0.068,370,11846,11833]]},{"id":"wpbench-wind-farm-site-4-long-2026-paper","name":"WPBench · Wind Farm Site 4 long (2026 paper)","group":"预测与决策","metric":"Normalized MAE","unit":"raw","public_history":true,"protocol":{"en":"Wind-power forecasting models use the same published splits and non-overlapping rolling test windows for Wind Farm Site 4. The source Long row averages 72/144-step horizons with 576-step history. Trainable models tune only on validation data. Native nMAE is dimensionless, lower is better; structurally unsupported methods are missing, not scored zero. Dataset and horizon-group cohorts stay separate.","zh":"风电预测模型在Wind Farm Site 4上使用相同数据划分与不重叠的滚动测试窗口。Long行平均72/144步预测，历史窗口576步；可训练模型只在验证集调参。保留无量纲nMAE，数值越低越好。结构上不适用的方法缺失，不补零；数据集及预测跨度分别排行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.24444v1","published_at":"2026-09-21","status":"paper_baselines"},"higher_is_better":false,"model_count":16,"result_count":16,"results":[[9464,0.185,370,11848,11833],[11834,0.185,370,11848,11833],[11835,0.211,370,11848,11833],[9161,0.216,370,11848,11833],[9682,0.202,370,11848,11833],[11836,0.198,370,11848,11833],[6812,0.189,370,11848,11833],[11847,0.181,370,11848,11833],[11837,0.247,370,11848,11833],[11838,0.182,370,11848,11833],[11839,0.172,370,11848,11833],[11840,0.207,370,11848,11833],[11841,0.195,370,11848,11833],[11842,0.191,370,11848,11833],[11843,0.24,370,11848,11833],[11844,0.19,370,11848,11833]]},{"id":"wpbench-wind-farm-site-6-short-2026-paper","name":"WPBench · Wind Farm Site 6 short (2026 paper)","group":"预测与决策","metric":"Normalized MAE","unit":"raw","public_history":true,"protocol":{"en":"Wind-power forecasting models use the same published splits and non-overlapping rolling test windows for Wind Farm Site 6. The source Short row averages 12/24-step horizons with 576-step history. Trainable models tune only on validation data. Native nMAE is dimensionless, lower is better; structurally unsupported methods are missing, not scored zero. Dataset and horizon-group cohorts stay separate.","zh":"风电预测模型在Wind Farm Site 6上使用相同数据划分与不重叠的滚动测试窗口。Short行平均12/24步预测，历史窗口576步；可训练模型只在验证集调参。保留无量纲nMAE，数值越低越好。结构上不适用的方法缺失，不补零；数据集及预测跨度分别排行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.24444v1","published_at":"2026-09-21","status":"paper_baselines"},"higher_is_better":false,"model_count":16,"result_count":16,"results":[[9464,0.062,370,11846,11833],[11834,0.059,370,11846,11833],[11835,0.095,370,11846,11833],[9161,0.07,370,11846,11833],[9682,0.071,370,11846,11833],[11836,0.059,370,11846,11833],[6812,0.096,370,11846,11833],[11847,0.057,370,11846,11833],[11837,0.059,370,11846,11833],[11838,0.057,370,11846,11833],[11839,0.053,370,11846,11833],[11840,0.061,370,11846,11833],[11841,0.064,370,11846,11833],[11842,0.072,370,11846,11833],[11843,0.176,370,11846,11833],[11844,0.056,370,11846,11833]]},{"id":"wpbench-wind-farm-site-6-long-2026-paper","name":"WPBench · Wind Farm Site 6 long (2026 paper)","group":"预测与决策","metric":"Normalized MAE","unit":"raw","public_history":true,"protocol":{"en":"Wind-power forecasting models use the same published splits and non-overlapping rolling test windows for Wind Farm Site 6. The source Long row averages 72/144-step horizons with 576-step history. Trainable models tune only on validation data. Native nMAE is dimensionless, lower is better; structurally unsupported methods are missing, not scored zero. Dataset and horizon-group cohorts stay separate.","zh":"风电预测模型在Wind Farm Site 6上使用相同数据划分与不重叠的滚动测试窗口。Long行平均72/144步预测，历史窗口576步；可训练模型只在验证集调参。保留无量纲nMAE，数值越低越好。结构上不适用的方法缺失，不补零；数据集及预测跨度分别排行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.24444v1","published_at":"2026-09-21","status":"paper_baselines"},"higher_is_better":false,"model_count":16,"result_count":16,"results":[[9464,0.145,370,11848,11833],[11834,0.107,370,11848,11833],[11835,0.125,370,11848,11833],[9161,0.14,370,11848,11833],[9682,0.139,370,11848,11833],[11836,0.136,370,11848,11833],[6812,0.201,370,11848,11833],[11847,0.127,370,11848,11833],[11837,0.154,370,11848,11833],[11838,0.135,370,11848,11833],[11839,0.11,370,11848,11833],[11840,0.135,370,11848,11833],[11841,0.169,370,11848,11833],[11842,0.153,370,11848,11833],[11843,0.278,370,11848,11833],[11844,0.16,370,11848,11833]]},{"id":"wpbench-kaggle1-short-2026-paper","name":"WPBench · Kaggle1 short (2026 paper)","group":"预测与决策","metric":"Normalized MAE","unit":"raw","public_history":true,"protocol":{"en":"Wind-power forecasting models use the same published splits and non-overlapping rolling test windows for Kaggle1. The source Short row averages 12/24-step horizons with 576-step history. Trainable models tune only on validation data. Native nMAE is dimensionless, lower is better; structurally unsupported methods are missing, not scored zero. Dataset and horizon-group cohorts stay separate.","zh":"风电预测模型在Kaggle1上使用相同数据划分与不重叠的滚动测试窗口。Short行平均12/24步预测，历史窗口576步；可训练模型只在验证集调参。保留无量纲nMAE，数值越低越好。结构上不适用的方法缺失，不补零；数据集及预测跨度分别排行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.24444v1","published_at":"2026-09-21","status":"paper_baselines"},"higher_is_better":false,"model_count":16,"result_count":16,"results":[[9464,0.109,370,11846,11833],[11834,0.116,370,11846,11833],[11835,0.136,370,11846,11833],[9161,0.133,370,11846,11833],[9682,0.137,370,11846,11833],[11836,0.118,370,11846,11833],[6812,0.123,370,11846,11833],[11847,0.119,370,11846,11833],[11837,0.12,370,11846,11833],[11838,0.109,370,11846,11833],[11839,0.108,370,11846,11833],[11840,0.122,370,11846,11833],[11841,0.118,370,11846,11833],[11842,0.126,370,11846,11833],[11843,0.135,370,11846,11833],[11844,0.113,370,11846,11833]]},{"id":"wpbench-kaggle1-long-2026-paper","name":"WPBench · Kaggle1 long (2026 paper)","group":"预测与决策","metric":"Normalized MAE","unit":"raw","public_history":true,"protocol":{"en":"Wind-power forecasting models use the same published splits and non-overlapping rolling test windows for Kaggle1. The source Long row averages 72/144-step horizons with 576-step history. Trainable models tune only on validation data. Native nMAE is dimensionless, lower is better; structurally unsupported methods are missing, not scored zero. Dataset and horizon-group cohorts stay separate.","zh":"风电预测模型在Kaggle1上使用相同数据划分与不重叠的滚动测试窗口。Long行平均72/144步预测，历史窗口576步；可训练模型只在验证集调参。保留无量纲nMAE，数值越低越好。结构上不适用的方法缺失，不补零；数据集及预测跨度分别排行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.24444v1","published_at":"2026-09-21","status":"paper_baselines"},"higher_is_better":false,"model_count":16,"result_count":16,"results":[[9464,0.152,370,11848,11833],[11834,0.16,370,11848,11833],[11835,0.178,370,11848,11833],[9161,0.174,370,11848,11833],[9682,0.183,370,11848,11833],[11836,0.177,370,11848,11833],[6812,0.167,370,11848,11833],[11847,0.157,370,11848,11833],[11837,0.212,370,11848,11833],[11838,0.152,370,11848,11833],[11839,0.155,370,11848,11833],[11840,0.171,370,11848,11833],[11841,0.161,370,11848,11833],[11842,0.166,370,11848,11833],[11843,0.165,370,11848,11833],[11844,0.168,370,11848,11833]]},{"id":"wpbench-scada-fault-short-2026-paper","name":"WPBench · SCADA_Fault short (2026 paper)","group":"预测与决策","metric":"Normalized MAE","unit":"raw","public_history":true,"protocol":{"en":"Wind-power forecasting models use the same published splits and non-overlapping rolling test windows for SCADA_Fault. The source Short row averages 12/24-step horizons with 576-step history. Trainable models tune only on validation data. Native nMAE is dimensionless, lower is better; structurally unsupported methods are missing, not scored zero. Dataset and horizon-group cohorts stay separate.","zh":"风电预测模型在SCADA_Fault上使用相同数据划分与不重叠的滚动测试窗口。Short行平均12/24步预测，历史窗口576步；可训练模型只在验证集调参。保留无量纲nMAE，数值越低越好。结构上不适用的方法缺失，不补零；数据集及预测跨度分别排行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.24444v1","published_at":"2026-09-21","status":"paper_baselines"},"higher_is_better":false,"model_count":16,"result_count":16,"results":[[9464,0.121,370,11846,11833],[11834,0.124,370,11846,11833],[11835,0.148,370,11846,11833],[9161,0.139,370,11846,11833],[9682,0.15,370,11846,11833],[11836,0.125,370,11846,11833],[6812,0.141,370,11846,11833],[11847,0.125,370,11846,11833],[11837,0.128,370,11846,11833],[11838,0.12,370,11846,11833],[11839,0.114,370,11846,11833],[11840,0.129,370,11846,11833],[11841,0.129,370,11846,11833],[11842,0.13,370,11846,11833],[11843,0.191,370,11846,11833],[11844,0.113,370,11846,11833]]},{"id":"wpbench-scada-fault-long-2026-paper","name":"WPBench · SCADA_Fault long (2026 paper)","group":"预测与决策","metric":"Normalized MAE","unit":"raw","public_history":true,"protocol":{"en":"Wind-power forecasting models use the same published splits and non-overlapping rolling test windows for SCADA_Fault. The source Long row averages 72/144-step horizons with 576-step history. Trainable models tune only on validation data. Native nMAE is dimensionless, lower is better; structurally unsupported methods are missing, not scored zero. Dataset and horizon-group cohorts stay separate.","zh":"风电预测模型在SCADA_Fault上使用相同数据划分与不重叠的滚动测试窗口。Long行平均72/144步预测，历史窗口576步；可训练模型只在验证集调参。保留无量纲nMAE，数值越低越好。结构上不适用的方法缺失，不补零；数据集及预测跨度分别排行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.24444v1","published_at":"2026-09-21","status":"paper_baselines"},"higher_is_better":false,"model_count":16,"result_count":16,"results":[[9464,0.211,370,11848,11833],[11834,0.218,370,11848,11833],[11835,0.251,370,11848,11833],[9161,0.231,370,11848,11833],[9682,0.249,370,11848,11833],[11836,0.23,370,11848,11833],[6812,0.23,370,11848,11833],[11847,0.219,370,11848,11833],[11837,0.25,370,11848,11833],[11838,0.214,370,11848,11833],[11839,0.197,370,11848,11833],[11840,0.234,370,11848,11833],[11841,0.215,370,11848,11833],[11842,0.226,370,11848,11833],[11843,0.503,370,11848,11833],[11844,0.241,370,11848,11833]]},{"id":"wpbench-yalova-short-2026-paper","name":"WPBench · Yalova short (2026 paper)","group":"预测与决策","metric":"Normalized MAE","unit":"raw","public_history":true,"protocol":{"en":"Wind-power forecasting models use the same published splits and non-overlapping rolling test windows for Yalova. The source Short row averages 12/24-step horizons with 576-step history. Trainable models tune only on validation data. Native nMAE is dimensionless, lower is better; structurally unsupported methods are missing, not scored zero. Dataset and horizon-group cohorts stay separate.","zh":"风电预测模型在Yalova上使用相同数据划分与不重叠的滚动测试窗口。Short行平均12/24步预测，历史窗口576步；可训练模型只在验证集调参。保留无量纲nMAE，数值越低越好。结构上不适用的方法缺失，不补零；数据集及预测跨度分别排行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.24444v1","published_at":"2026-09-21","status":"paper_baselines"},"higher_is_better":false,"model_count":16,"result_count":16,"results":[[9464,0.107,370,11846,11833],[11834,0.111,370,11846,11833],[11835,0.14,370,11846,11833],[9161,0.133,370,11846,11833],[9682,0.143,370,11846,11833],[11836,0.111,370,11846,11833],[6812,0.105,370,11846,11833],[11847,0.104,370,11846,11833],[11837,0.116,370,11846,11833],[11838,0.105,370,11846,11833],[11839,0.096,370,11846,11833],[11840,0.123,370,11846,11833],[11841,0.119,370,11846,11833],[11842,0.116,370,11846,11833],[11843,0.145,370,11846,11833],[11844,0.102,370,11846,11833]]},{"id":"wpbench-yalova-long-2026-paper","name":"WPBench · Yalova long (2026 paper)","group":"预测与决策","metric":"Normalized MAE","unit":"raw","public_history":true,"protocol":{"en":"Wind-power forecasting models use the same published splits and non-overlapping rolling test windows for Yalova. The source Long row averages 72/144-step horizons with 576-step history. Trainable models tune only on validation data. Native nMAE is dimensionless, lower is better; structurally unsupported methods are missing, not scored zero. Dataset and horizon-group cohorts stay separate.","zh":"风电预测模型在Yalova上使用相同数据划分与不重叠的滚动测试窗口。Long行平均72/144步预测，历史窗口576步；可训练模型只在验证集调参。保留无量纲nMAE，数值越低越好。结构上不适用的方法缺失，不补零；数据集及预测跨度分别排行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.24444v1","published_at":"2026-09-21","status":"paper_baselines"},"higher_is_better":false,"model_count":16,"result_count":16,"results":[[9464,0.221,370,11848,11833],[11834,0.229,370,11848,11833],[11835,0.244,370,11848,11833],[9161,0.237,370,11848,11833],[9682,0.257,370,11848,11833],[11836,0.226,370,11848,11833],[6812,0.213,370,11848,11833],[11847,0.216,370,11848,11833],[11837,0.257,370,11848,11833],[11838,0.217,370,11848,11833],[11839,0.207,370,11848,11833],[11840,0.251,370,11848,11833],[11841,0.226,370,11848,11833],[11842,0.226,370,11848,11833],[11843,0.262,370,11848,11833],[11844,0.222,370,11848,11833]]},{"id":"wpbench-wind-farm-c-short-2026-paper","name":"WPBench · Wind Farm C short (2026 paper)","group":"预测与决策","metric":"Normalized MAE","unit":"raw","public_history":true,"protocol":{"en":"Wind-power forecasting models use the same published splits and non-overlapping rolling test windows for Wind Farm C. The source Short row averages 12/24-step horizons with 576-step history. Trainable models tune only on validation data. Native nMAE is dimensionless, lower is better; structurally unsupported methods are missing, not scored zero. Dataset and horizon-group cohorts stay separate.","zh":"风电预测模型在Wind Farm C上使用相同数据划分与不重叠的滚动测试窗口。Short行平均12/24步预测，历史窗口576步；可训练模型只在验证集调参。保留无量纲nMAE，数值越低越好。结构上不适用的方法缺失，不补零；数据集及预测跨度分别排行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.24444v1","published_at":"2026-09-21","status":"paper_baselines"},"higher_is_better":false,"model_count":18,"result_count":18,"results":[[9464,0.105,370,11849,11850],[11834,0.11,370,11849,11850],[11835,0.112,370,11849,11850],[9161,0.14,370,11849,11850],[9682,0.126,370,11849,11850],[11836,0.108,370,11849,11850],[6812,0.105,370,11849,11850],[11837,0.114,370,11849,11850],[11838,0.101,370,11849,11850],[11839,0.097,370,11849,11850],[11840,0.118,370,11849,11850],[11841,0.116,370,11849,11850],[11842,0.138,370,11849,11850],[11843,0.188,370,11849,11850],[11851,0.102,370,11849,11850],[11844,0.095,370,11849,11850],[11852,0.106,370,11849,11850],[11853,0.123,370,11849,11850]]},{"id":"wpbench-wind-farm-c-long-2026-paper","name":"WPBench · Wind Farm C long (2026 paper)","group":"预测与决策","metric":"Normalized MAE","unit":"raw","public_history":true,"protocol":{"en":"Wind-power forecasting models use the same published splits and non-overlapping rolling test windows for Wind Farm C. The source Long row averages 72/144-step horizons with 576-step history. Trainable models tune only on validation data. Native nMAE is dimensionless, lower is better; structurally unsupported methods are missing, not scored zero. Dataset and horizon-group cohorts stay separate.","zh":"风电预测模型在Wind Farm C上使用相同数据划分与不重叠的滚动测试窗口。Long行平均72/144步预测，历史窗口576步；可训练模型只在验证集调参。保留无量纲nMAE，数值越低越好。结构上不适用的方法缺失，不补零；数据集及预测跨度分别排行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.24444v1","published_at":"2026-09-21","status":"paper_baselines"},"higher_is_better":false,"model_count":18,"result_count":18,"results":[[9464,0.198,370,11854,11850],[11834,0.193,370,11854,11850],[11835,0.192,370,11854,11850],[9161,0.225,370,11854,11850],[9682,0.237,370,11854,11850],[11836,0.206,370,11854,11850],[6812,0.196,370,11854,11850],[11837,0.232,370,11854,11850],[11838,0.191,370,11854,11850],[11839,0.191,370,11854,11850],[11840,0.213,370,11854,11850],[11841,0.204,370,11854,11850],[11842,0.224,370,11854,11850],[11843,0.234,370,11854,11850],[11851,0.222,370,11854,11850],[11844,0.199,370,11854,11850],[11852,0.187,370,11854,11850],[11853,0.262,370,11854,11850]]},{"id":"wpbench-ecmwf-short-2026-paper","name":"WPBench · ECMWF short (2026 paper)","group":"预测与决策","metric":"Normalized MAE","unit":"raw","public_history":true,"protocol":{"en":"Wind-power forecasting models use the same published splits and non-overlapping rolling test windows for ECMWF. The source Short row averages 12/24-step horizons with 576-step history. Trainable models tune only on validation data. Native nMAE is dimensionless, lower is better; structurally unsupported methods are missing, not scored zero. Dataset and horizon-group cohorts stay separate.","zh":"风电预测模型在ECMWF上使用相同数据划分与不重叠的滚动测试窗口。Short行平均12/24步预测，历史窗口576步；可训练模型只在验证集调参。保留无量纲nMAE，数值越低越好。结构上不适用的方法缺失，不补零；数据集及预测跨度分别排行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.24444v1","published_at":"2026-09-21","status":"paper_baselines"},"higher_is_better":false,"model_count":18,"result_count":18,"results":[[9464,0.074,370,11849,11850],[11834,0.073,370,11849,11850],[11835,0.069,370,11849,11850],[9161,0.076,370,11849,11850],[9682,0.08,370,11849,11850],[11836,0.07,370,11849,11850],[6812,0.074,370,11849,11850],[11837,0.075,370,11849,11850],[11838,0.073,370,11849,11850],[11839,0.067,370,11849,11850],[11840,0.075,370,11849,11850],[11841,0.074,370,11849,11850],[11842,0.079,370,11849,11850],[11843,0.08,370,11849,11850],[11851,0.065,370,11849,11850],[11844,0.064,370,11849,11850],[11852,0.073,370,11849,11850],[11853,0.078,370,11849,11850]]},{"id":"wpbench-ecmwf-long-2026-paper","name":"WPBench · ECMWF long (2026 paper)","group":"预测与决策","metric":"Normalized MAE","unit":"raw","public_history":true,"protocol":{"en":"Wind-power forecasting models use the same published splits and non-overlapping rolling test windows for ECMWF. The source Long row averages 72/144-step horizons with 576-step history. Trainable models tune only on validation data. Native nMAE is dimensionless, lower is better; structurally unsupported methods are missing, not scored zero. Dataset and horizon-group cohorts stay separate.","zh":"风电预测模型在ECMWF上使用相同数据划分与不重叠的滚动测试窗口。Long行平均72/144步预测，历史窗口576步；可训练模型只在验证集调参。保留无量纲nMAE，数值越低越好。结构上不适用的方法缺失，不补零；数据集及预测跨度分别排行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.24444v1","published_at":"2026-09-21","status":"paper_baselines"},"higher_is_better":false,"model_count":18,"result_count":18,"results":[[9464,0.104,370,11854,11850],[11834,0.099,370,11854,11850],[11835,0.093,370,11854,11850],[9161,0.103,370,11854,11850],[9682,0.107,370,11854,11850],[11836,0.098,370,11854,11850],[6812,0.106,370,11854,11850],[11837,0.107,370,11854,11850],[11838,0.1,370,11854,11850],[11839,0.093,370,11854,11850],[11840,0.104,370,11854,11850],[11841,0.106,370,11854,11850],[11842,0.108,370,11854,11850],[11843,0.114,370,11854,11850],[11851,0.104,370,11854,11850],[11844,0.092,370,11854,11850],[11852,0.096,370,11854,11850],[11853,0.111,370,11854,11850]]},{"id":"wpbench-cosmo-short-2026-paper","name":"WPBench · COSMO short (2026 paper)","group":"预测与决策","metric":"Normalized MAE","unit":"raw","public_history":true,"protocol":{"en":"Wind-power forecasting models use the same published splits and non-overlapping rolling test windows for COSMO. The source Short row averages 12/24-step horizons with 576-step history. Trainable models tune only on validation data. Native nMAE is dimensionless, lower is better; structurally unsupported methods are missing, not scored zero. Dataset and horizon-group cohorts stay separate.","zh":"风电预测模型在COSMO上使用相同数据划分与不重叠的滚动测试窗口。Short行平均12/24步预测，历史窗口576步；可训练模型只在验证集调参。保留无量纲nMAE，数值越低越好。结构上不适用的方法缺失，不补零；数据集及预测跨度分别排行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.24444v1","published_at":"2026-09-21","status":"paper_baselines"},"higher_is_better":false,"model_count":18,"result_count":18,"results":[[9464,0.088,370,11849,11850],[11834,0.086,370,11849,11850],[11835,0.082,370,11849,11850],[9161,0.09,370,11849,11850],[9682,0.093,370,11849,11850],[11836,0.082,370,11849,11850],[6812,0.087,370,11849,11850],[11837,0.085,370,11849,11850],[11838,0.087,370,11849,11850],[11839,0.079,370,11849,11850],[11840,0.088,370,11849,11850],[11841,0.088,370,11849,11850],[11842,0.089,370,11849,11850],[11843,0.092,370,11849,11850],[11851,0.077,370,11849,11850],[11844,0.077,370,11849,11850],[11852,0.085,370,11849,11850],[11853,0.091,370,11849,11850]]},{"id":"wpbench-cosmo-long-2026-paper","name":"WPBench · COSMO long (2026 paper)","group":"预测与决策","metric":"Normalized MAE","unit":"raw","public_history":true,"protocol":{"en":"Wind-power forecasting models use the same published splits and non-overlapping rolling test windows for COSMO. The source Long row averages 72/144-step horizons with 576-step history. Trainable models tune only on validation data. Native nMAE is dimensionless, lower is better; structurally unsupported methods are missing, not scored zero. Dataset and horizon-group cohorts stay separate.","zh":"风电预测模型在COSMO上使用相同数据划分与不重叠的滚动测试窗口。Long行平均72/144步预测，历史窗口576步；可训练模型只在验证集调参。保留无量纲nMAE，数值越低越好。结构上不适用的方法缺失，不补零；数据集及预测跨度分别排行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.24444v1","published_at":"2026-09-21","status":"paper_baselines"},"higher_is_better":false,"model_count":18,"result_count":18,"results":[[9464,0.119,370,11854,11850],[11834,0.114,370,11854,11850],[11835,0.105,370,11854,11850],[9161,0.118,370,11854,11850],[9682,0.12,370,11854,11850],[11836,0.109,370,11854,11850],[6812,0.12,370,11854,11850],[11837,0.125,370,11854,11850],[11838,0.116,370,11854,11850],[11839,0.105,370,11854,11850],[11840,0.119,370,11854,11850],[11841,0.12,370,11854,11850],[11842,0.125,370,11854,11850],[11843,0.127,370,11854,11850],[11851,0.118,370,11854,11850],[11844,0.104,370,11854,11850],[11852,0.109,370,11854,11850],[11853,0.126,370,11854,11850]]},{"id":"wpbench-aemo-short-2026-paper","name":"WPBench · AEMO short (2026 paper)","group":"预测与决策","metric":"Normalized MAE","unit":"raw","public_history":true,"protocol":{"en":"Wind-power forecasting models use the same published splits and non-overlapping rolling test windows for AEMO. The source Short row averages 12/24-step horizons with 576-step history. Trainable models tune only on validation data. Native nMAE is dimensionless, lower is better; structurally unsupported methods are missing, not scored zero. Dataset and horizon-group cohorts stay separate.","zh":"风电预测模型在AEMO上使用相同数据划分与不重叠的滚动测试窗口。Short行平均12/24步预测，历史窗口576步；可训练模型只在验证集调参。保留无量纲nMAE，数值越低越好。结构上不适用的方法缺失，不补零；数据集及预测跨度分别排行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.24444v1","published_at":"2026-09-21","status":"paper_baselines"},"higher_is_better":false,"model_count":18,"result_count":18,"results":[[9464,0.079,370,11849,11850],[11834,0.078,370,11849,11850],[11835,0.097,370,11849,11850],[9161,0.088,370,11849,11850],[9682,0.082,370,11849,11850],[11836,0.075,370,11849,11850],[6812,0.079,370,11849,11850],[11837,0.079,370,11849,11850],[11838,0.08,370,11849,11850],[11839,0.076,370,11849,11850],[11840,0.08,370,11849,11850],[11841,0.077,370,11849,11850],[11842,0.079,370,11849,11850],[11843,0.077,370,11849,11850],[11851,0.072,370,11849,11850],[11844,0.074,370,11849,11850],[11852,0.078,370,11849,11850],[11853,0.079,370,11849,11850]]},{"id":"wpbench-aemo-long-2026-paper","name":"WPBench · AEMO long (2026 paper)","group":"预测与决策","metric":"Normalized MAE","unit":"raw","public_history":true,"protocol":{"en":"Wind-power forecasting models use the same published splits and non-overlapping rolling test windows for AEMO. The source Long row averages 72/144-step horizons with 576-step history. Trainable models tune only on validation data. Native nMAE is dimensionless, lower is better; structurally unsupported methods are missing, not scored zero. Dataset and horizon-group cohorts stay separate.","zh":"风电预测模型在AEMO上使用相同数据划分与不重叠的滚动测试窗口。Long行平均72/144步预测，历史窗口576步；可训练模型只在验证集调参。保留无量纲nMAE，数值越低越好。结构上不适用的方法缺失，不补零；数据集及预测跨度分别排行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.24444v1","published_at":"2026-09-21","status":"paper_baselines"},"higher_is_better":false,"model_count":18,"result_count":18,"results":[[9464,0.175,370,11854,11850],[11834,0.175,370,11854,11850],[11835,0.18,370,11854,11850],[9161,0.183,370,11854,11850],[9682,0.177,370,11854,11850],[11836,0.171,370,11854,11850],[6812,0.176,370,11854,11850],[11837,0.196,370,11854,11850],[11838,0.172,370,11854,11850],[11839,0.174,370,11854,11850],[11840,0.204,370,11854,11850],[11841,0.171,370,11854,11850],[11842,0.167,370,11854,11850],[11843,0.169,370,11854,11850],[11851,0.166,370,11854,11850],[11844,0.166,370,11854,11850],[11852,0.169,370,11854,11850],[11853,0.197,370,11854,11850]]},{"id":"wpbench-gefcom2012-short-2026-paper","name":"WPBench · GEFCom2012 short (2026 paper)","group":"预测与决策","metric":"Normalized MAE","unit":"raw","public_history":true,"protocol":{"en":"Wind-power forecasting models use the same published splits and non-overlapping rolling test windows for GEFCom2012. The source Short row averages 12/24-step horizons with 576-step history. Trainable models tune only on validation data. Native nMAE is dimensionless, lower is better; structurally unsupported methods are missing, not scored zero. Dataset and horizon-group cohorts stay separate.","zh":"风电预测模型在GEFCom2012上使用相同数据划分与不重叠的滚动测试窗口。Short行平均12/24步预测，历史窗口576步；可训练模型只在验证集调参。保留无量纲nMAE，数值越低越好。结构上不适用的方法缺失，不补零；数据集及预测跨度分别排行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.24444v1","published_at":"2026-09-21","status":"paper_baselines"},"higher_is_better":false,"model_count":19,"result_count":19,"results":[[9464,0.208,370,11855,11850],[11834,0.211,370,11855,11850],[11835,0.22,370,11855,11850],[9161,0.222,370,11855,11850],[9682,0.217,370,11855,11850],[11836,0.21,370,11855,11850],[6812,0.195,370,11855,11850],[11847,0.193,370,11855,11850],[11837,0.208,370,11855,11850],[11838,0.207,370,11855,11850],[11839,0.204,370,11855,11850],[11840,0.211,370,11855,11850],[11841,0.217,370,11855,11850],[11842,0.217,370,11855,11850],[11843,0.223,370,11855,11850],[11851,0.195,370,11855,11850],[11844,0.199,370,11855,11850],[11852,0.209,370,11855,11850],[11853,0.218,370,11855,11850]]},{"id":"wpbench-gefcom2012-long-2026-paper","name":"WPBench · GEFCom2012 long (2026 paper)","group":"预测与决策","metric":"Normalized MAE","unit":"raw","public_history":true,"protocol":{"en":"Wind-power forecasting models use the same published splits and non-overlapping rolling test windows for GEFCom2012. The source Long row averages 72/144-step horizons with 576-step history. Trainable models tune only on validation data. Native nMAE is dimensionless, lower is better; structurally unsupported methods are missing, not scored zero. Dataset and horizon-group cohorts stay separate.","zh":"风电预测模型在GEFCom2012上使用相同数据划分与不重叠的滚动测试窗口。Long行平均72/144步预测，历史窗口576步；可训练模型只在验证集调参。保留无量纲nMAE，数值越低越好。结构上不适用的方法缺失，不补零；数据集及预测跨度分别排行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.24444v1","published_at":"2026-09-21","status":"paper_baselines"},"higher_is_better":false,"model_count":19,"result_count":19,"results":[[9464,0.266,370,11856,11850],[11834,0.269,370,11856,11850],[11835,0.263,370,11856,11850],[9161,0.268,370,11856,11850],[9682,0.275,370,11856,11850],[11836,0.274,370,11856,11850],[6812,0.264,370,11856,11850],[11847,0.26,370,11856,11850],[11837,0.279,370,11856,11850],[11838,0.268,370,11856,11850],[11839,0.269,370,11856,11850],[11840,0.275,370,11856,11850],[11841,0.268,370,11856,11850],[11842,0.266,370,11856,11850],[11843,0.277,370,11856,11850],[11851,0.299,370,11856,11850],[11844,0.276,370,11856,11850],[11852,0.275,370,11856,11850],[11853,0.291,370,11856,11850]]},{"id":"wpbench-gefcom2014-short-2026-paper","name":"WPBench · GEFCom2014 short (2026 paper)","group":"预测与决策","metric":"Normalized MAE","unit":"raw","public_history":true,"protocol":{"en":"Wind-power forecasting models use the same published splits and non-overlapping rolling test windows for GEFCom2014. The source Short row averages 12/24-step horizons with 576-step history. Trainable models tune only on validation data. Native nMAE is dimensionless, lower is better; structurally unsupported methods are missing, not scored zero. Dataset and horizon-group cohorts stay separate.","zh":"风电预测模型在GEFCom2014上使用相同数据划分与不重叠的滚动测试窗口。Short行平均12/24步预测，历史窗口576步；可训练模型只在验证集调参。保留无量纲nMAE，数值越低越好。结构上不适用的方法缺失，不补零；数据集及预测跨度分别排行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.24444v1","published_at":"2026-09-21","status":"paper_baselines"},"higher_is_better":false,"model_count":19,"result_count":19,"results":[[9464,0.208,370,11855,11850],[11834,0.207,370,11855,11850],[11835,0.27,370,11855,11850],[9161,0.221,370,11855,11850],[9682,0.223,370,11855,11850],[11836,0.208,370,11855,11850],[6812,0.194,370,11855,11850],[11847,0.192,370,11855,11850],[11837,0.206,370,11855,11850],[11838,0.208,370,11855,11850],[11839,0.196,370,11855,11850],[11840,0.211,370,11855,11850],[11841,0.212,370,11855,11850],[11842,0.232,370,11855,11850],[11843,0.226,370,11855,11850],[11851,0.193,370,11855,11850],[11844,0.196,370,11855,11850],[11852,0.211,370,11855,11850],[11853,0.21,370,11855,11850]]},{"id":"wpbench-gefcom2014-long-2026-paper","name":"WPBench · GEFCom2014 long (2026 paper)","group":"预测与决策","metric":"Normalized MAE","unit":"raw","public_history":true,"protocol":{"en":"Wind-power forecasting models use the same published splits and non-overlapping rolling test windows for GEFCom2014. The source Long row averages 72/144-step horizons with 576-step history. Trainable models tune only on validation data. Native nMAE is dimensionless, lower is better; structurally unsupported methods are missing, not scored zero. Dataset and horizon-group cohorts stay separate.","zh":"风电预测模型在GEFCom2014上使用相同数据划分与不重叠的滚动测试窗口。Long行平均72/144步预测，历史窗口576步；可训练模型只在验证集调参。保留无量纲nMAE，数值越低越好。结构上不适用的方法缺失，不补零；数据集及预测跨度分别排行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.24444v1","published_at":"2026-09-21","status":"paper_baselines"},"higher_is_better":false,"model_count":19,"result_count":19,"results":[[9464,0.263,370,11856,11850],[11834,0.269,370,11856,11850],[11835,0.274,370,11856,11850],[9161,0.269,370,11856,11850],[9682,0.275,370,11856,11850],[11836,0.272,370,11856,11850],[6812,0.254,370,11856,11850],[11847,0.256,370,11856,11850],[11837,0.278,370,11856,11850],[11838,0.264,370,11856,11850],[11839,0.271,370,11856,11850],[11840,0.271,370,11856,11850],[11841,0.262,370,11856,11850],[11842,0.274,370,11856,11850],[11843,0.278,370,11856,11850],[11851,0.282,370,11856,11850],[11844,0.269,370,11856,11850],[11852,0.268,370,11856,11850],[11853,0.285,370,11856,11850]]},{"id":"wpbench-kelmarsh-short-2026-paper","name":"WPBench · Kelmarsh short (2026 paper)","group":"预测与决策","metric":"Normalized MAE","unit":"raw","public_history":true,"protocol":{"en":"Wind-power forecasting models use the same published splits and non-overlapping rolling test windows for Kelmarsh. The source Short row averages 12/24-step horizons with 576-step history. Trainable models tune only on validation data. Native nMAE is dimensionless, lower is better; structurally unsupported methods are missing, not scored zero. Dataset and horizon-group cohorts stay separate.","zh":"风电预测模型在Kelmarsh上使用相同数据划分与不重叠的滚动测试窗口。Short行平均12/24步预测，历史窗口576步；可训练模型只在验证集调参。保留无量纲nMAE，数值越低越好。结构上不适用的方法缺失，不补零；数据集及预测跨度分别排行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.24444v1","published_at":"2026-09-21","status":"paper_baselines"},"higher_is_better":false,"model_count":19,"result_count":19,"results":[[9464,0.092,370,11855,11850],[11834,0.091,370,11855,11850],[11835,0.1,370,11855,11850],[9161,0.1,370,11855,11850],[9682,0.098,370,11855,11850],[11836,0.09,370,11855,11850],[6812,0.093,370,11855,11850],[11847,0.093,370,11855,11850],[11837,0.091,370,11855,11850],[11838,0.093,370,11855,11850],[11839,0.088,370,11855,11850],[11840,0.094,370,11855,11850],[11841,0.093,370,11855,11850],[11842,0.096,370,11855,11850],[11843,0.099,370,11855,11850],[11851,0.088,370,11855,11850],[11844,0.086,370,11855,11850],[11852,0.092,370,11855,11850],[11853,0.093,370,11855,11850]]},{"id":"wpbench-kelmarsh-long-2026-paper","name":"WPBench · Kelmarsh long (2026 paper)","group":"预测与决策","metric":"Normalized MAE","unit":"raw","public_history":true,"protocol":{"en":"Wind-power forecasting models use the same published splits and non-overlapping rolling test windows for Kelmarsh. The source Long row averages 72/144-step horizons with 576-step history. Trainable models tune only on validation data. Native nMAE is dimensionless, lower is better; structurally unsupported methods are missing, not scored zero. Dataset and horizon-group cohorts stay separate.","zh":"风电预测模型在Kelmarsh上使用相同数据划分与不重叠的滚动测试窗口。Long行平均72/144步预测，历史窗口576步；可训练模型只在验证集调参。保留无量纲nMAE，数值越低越好。结构上不适用的方法缺失，不补零；数据集及预测跨度分别排行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.24444v1","published_at":"2026-09-21","status":"paper_baselines"},"higher_is_better":false,"model_count":19,"result_count":19,"results":[[9464,0.175,370,11856,11850],[11834,0.165,370,11856,11850],[11835,0.168,370,11856,11850],[9161,0.174,370,11856,11850],[9682,0.172,370,11856,11850],[11836,0.166,370,11856,11850],[6812,0.174,370,11856,11850],[11847,0.176,370,11856,11850],[11837,0.184,370,11856,11850],[11838,0.163,370,11856,11850],[11839,0.157,370,11856,11850],[11840,0.191,370,11856,11850],[11841,0.171,370,11856,11850],[11842,0.173,370,11856,11850],[11843,0.179,370,11856,11850],[11851,0.166,370,11856,11850],[11844,0.17,370,11856,11850],[11852,0.165,370,11856,11850],[11853,0.193,370,11856,11850]]},{"id":"wpbench-penmanshiel-short-2026-paper","name":"WPBench · Penmanshiel short (2026 paper)","group":"预测与决策","metric":"Normalized MAE","unit":"raw","public_history":true,"protocol":{"en":"Wind-power forecasting models use the same published splits and non-overlapping rolling test windows for Penmanshiel. The source Short row averages 12/24-step horizons with 576-step history. Trainable models tune only on validation data. Native nMAE is dimensionless, lower is better; structurally unsupported methods are missing, not scored zero. Dataset and horizon-group cohorts stay separate.","zh":"风电预测模型在Penmanshiel上使用相同数据划分与不重叠的滚动测试窗口。Short行平均12/24步预测，历史窗口576步；可训练模型只在验证集调参。保留无量纲nMAE，数值越低越好。结构上不适用的方法缺失，不补零；数据集及预测跨度分别排行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.24444v1","published_at":"2026-09-21","status":"paper_baselines"},"higher_is_better":false,"model_count":19,"result_count":19,"results":[[9464,0.099,370,11855,11850],[11834,0.097,370,11855,11850],[11835,0.103,370,11855,11850],[9161,0.101,370,11855,11850],[9682,0.104,370,11855,11850],[11836,0.094,370,11855,11850],[6812,0.097,370,11855,11850],[11847,0.097,370,11855,11850],[11837,0.098,370,11855,11850],[11838,0.098,370,11855,11850],[11839,0.092,370,11855,11850],[11840,0.103,370,11855,11850],[11841,0.103,370,11855,11850],[11842,0.104,370,11855,11850],[11843,0.102,370,11855,11850],[11851,0.091,370,11855,11850],[11844,0.091,370,11855,11850],[11852,0.096,370,11855,11850],[11853,0.103,370,11855,11850]]},{"id":"wpbench-penmanshiel-long-2026-paper","name":"WPBench · Penmanshiel long (2026 paper)","group":"预测与决策","metric":"Normalized MAE","unit":"raw","public_history":true,"protocol":{"en":"Wind-power forecasting models use the same published splits and non-overlapping rolling test windows for Penmanshiel. The source Long row averages 72/144-step horizons with 576-step history. Trainable models tune only on validation data. Native nMAE is dimensionless, lower is better; structurally unsupported methods are missing, not scored zero. Dataset and horizon-group cohorts stay separate.","zh":"风电预测模型在Penmanshiel上使用相同数据划分与不重叠的滚动测试窗口。Long行平均72/144步预测，历史窗口576步；可训练模型只在验证集调参。保留无量纲nMAE，数值越低越好。结构上不适用的方法缺失，不补零；数据集及预测跨度分别排行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.24444v1","published_at":"2026-09-21","status":"paper_baselines"},"higher_is_better":false,"model_count":19,"result_count":19,"results":[[9464,0.189,370,11856,11850],[11834,0.182,370,11856,11850],[11835,0.177,370,11856,11850],[9161,0.199,370,11856,11850],[9682,0.19,370,11856,11850],[11836,0.183,370,11856,11850],[6812,0.185,370,11856,11850],[11847,0.184,370,11856,11850],[11837,0.218,370,11856,11850],[11838,0.181,370,11856,11850],[11839,0.17,370,11856,11850],[11840,0.202,370,11856,11850],[11841,0.188,370,11856,11850],[11842,0.189,370,11856,11850],[11843,0.192,370,11856,11850],[11851,0.171,370,11856,11850],[11844,0.187,370,11856,11850],[11852,0.174,370,11856,11850],[11853,0.209,370,11856,11850]]},{"id":"wpbench-sdwpf-short-2026-paper","name":"WPBench · SDWPF short (2026 paper)","group":"预测与决策","metric":"Normalized MAE","unit":"raw","public_history":true,"protocol":{"en":"Wind-power forecasting models use the same published splits and non-overlapping rolling test windows for SDWPF. The source Short row averages 12/24-step horizons with 576-step history. Trainable models tune only on validation data. Native nMAE is dimensionless, lower is better; structurally unsupported methods are missing, not scored zero. Dataset and horizon-group cohorts stay separate.","zh":"风电预测模型在SDWPF上使用相同数据划分与不重叠的滚动测试窗口。Short行平均12/24步预测，历史窗口576步；可训练模型只在验证集调参。保留无量纲nMAE，数值越低越好。结构上不适用的方法缺失，不补零；数据集及预测跨度分别排行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.24444v1","published_at":"2026-09-21","status":"paper_baselines"},"higher_is_better":false,"model_count":19,"result_count":19,"results":[[9464,0.095,370,11855,11850],[11834,0.092,370,11855,11850],[11835,0.098,370,11855,11850],[9161,0.106,370,11855,11850],[9682,0.118,370,11855,11850],[11836,0.088,370,11855,11850],[6812,0.145,370,11855,11850],[11847,0.091,370,11855,11850],[11837,0.093,370,11855,11850],[11838,0.085,370,11855,11850],[11839,0.076,370,11855,11850],[11840,0.108,370,11855,11850],[11841,0.105,370,11855,11850],[11842,0.11,370,11855,11850],[11843,0.14,370,11855,11850],[11851,0.076,370,11855,11850],[11844,0.079,370,11855,11850],[11852,0.087,370,11855,11850],[11853,0.105,370,11855,11850]]},{"id":"wpbench-sdwpf-long-2026-paper","name":"WPBench · SDWPF long (2026 paper)","group":"预测与决策","metric":"Normalized MAE","unit":"raw","public_history":true,"protocol":{"en":"Wind-power forecasting models use the same published splits and non-overlapping rolling test windows for SDWPF. The source Long row averages 72/144-step horizons with 576-step history. Trainable models tune only on validation data. Native nMAE is dimensionless, lower is better; structurally unsupported methods are missing, not scored zero. Dataset and horizon-group cohorts stay separate.","zh":"风电预测模型在SDWPF上使用相同数据划分与不重叠的滚动测试窗口。Long行平均72/144步预测，历史窗口576步；可训练模型只在验证集调参。保留无量纲nMAE，数值越低越好。结构上不适用的方法缺失，不补零；数据集及预测跨度分别排行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.24444v1","published_at":"2026-09-21","status":"paper_baselines"},"higher_is_better":false,"model_count":19,"result_count":19,"results":[[9464,0.181,370,11856,11850],[11834,0.176,370,11856,11850],[11835,0.178,370,11856,11850],[9161,0.191,370,11856,11850],[9682,0.197,370,11856,11850],[11836,0.169,370,11856,11850],[6812,0.267,370,11856,11850],[11847,0.163,370,11856,11850],[11837,0.189,370,11856,11850],[11838,0.161,370,11856,11850],[11839,0.149,370,11856,11850],[11840,0.18,370,11856,11850],[11841,0.197,370,11856,11850],[11842,0.197,370,11856,11850],[11843,0.197,370,11856,11850],[11851,0.189,370,11856,11850],[11844,0.176,370,11856,11850],[11852,0.172,370,11856,11850],[11853,0.208,370,11856,11850]]},{"id":"wpbench-ueps-short-2026-paper","name":"WPBench · UEPS short (2026 paper)","group":"预测与决策","metric":"Normalized MAE","unit":"raw","public_history":true,"protocol":{"en":"Wind-power forecasting models use the same published splits and non-overlapping rolling test windows for UEPS. The source Short row averages 12/24-step horizons with 576-step history. Trainable models tune only on validation data. Native nMAE is dimensionless, lower is better; structurally unsupported methods are missing, not scored zero. Dataset and horizon-group cohorts stay separate.","zh":"风电预测模型在UEPS上使用相同数据划分与不重叠的滚动测试窗口。Short行平均12/24步预测，历史窗口576步；可训练模型只在验证集调参。保留无量纲nMAE，数值越低越好。结构上不适用的方法缺失，不补零；数据集及预测跨度分别排行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.24444v1","published_at":"2026-09-21","status":"paper_baselines"},"higher_is_better":false,"model_count":19,"result_count":19,"results":[[9464,0.083,370,11855,11850],[11834,0.085,370,11855,11850],[11835,0.091,370,11855,11850],[9161,0.086,370,11855,11850],[9682,0.093,370,11855,11850],[11836,0.087,370,11855,11850],[6812,0.087,370,11855,11850],[11847,0.086,370,11855,11850],[11837,0.087,370,11855,11850],[11838,0.085,370,11855,11850],[11839,0.083,370,11855,11850],[11840,0.089,370,11855,11850],[11841,0.088,370,11855,11850],[11842,0.095,370,11855,11850],[11843,0.105,370,11855,11850],[11851,0.093,370,11855,11850],[11844,0.08,370,11855,11850],[11852,0.47,370,11855,11850],[11853,0.088,370,11855,11850]]},{"id":"wpbench-ueps-long-2026-paper","name":"WPBench · UEPS long (2026 paper)","group":"预测与决策","metric":"Normalized MAE","unit":"raw","public_history":true,"protocol":{"en":"Wind-power forecasting models use the same published splits and non-overlapping rolling test windows for UEPS. The source Long row averages 72/144-step horizons with 576-step history. Trainable models tune only on validation data. Native nMAE is dimensionless, lower is better; structurally unsupported methods are missing, not scored zero. Dataset and horizon-group cohorts stay separate.","zh":"风电预测模型在UEPS上使用相同数据划分与不重叠的滚动测试窗口。Long行平均72/144步预测，历史窗口576步；可训练模型只在验证集调参。保留无量纲nMAE，数值越低越好。结构上不适用的方法缺失，不补零；数据集及预测跨度分别排行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.24444v1","published_at":"2026-09-21","status":"paper_baselines"},"higher_is_better":false,"model_count":19,"result_count":19,"results":[[9464,0.132,370,11856,11850],[11834,0.132,370,11856,11850],[11835,0.135,370,11856,11850],[9161,0.136,370,11856,11850],[9682,0.138,370,11856,11850],[11836,0.142,370,11856,11850],[6812,0.139,370,11856,11850],[11847,0.158,370,11856,11850],[11837,0.151,370,11856,11850],[11838,0.133,370,11856,11850],[11839,0.141,370,11856,11850],[11840,0.137,370,11856,11850],[11841,0.131,370,11856,11850],[11842,0.144,370,11856,11850],[11843,0.145,370,11856,11850],[11851,0.151,370,11856,11850],[11844,0.135,370,11856,11850],[11852,0.135,370,11856,11850],[11853,0.151,370,11856,11850]]},{"id":"wpbench-uebb-short-2026-paper","name":"WPBench · UEBB short (2026 paper)","group":"预测与决策","metric":"Normalized MAE","unit":"raw","public_history":true,"protocol":{"en":"Wind-power forecasting models use the same published splits and non-overlapping rolling test windows for UEBB. The source Short row averages 12/24-step horizons with 576-step history. Trainable models tune only on validation data. Native nMAE is dimensionless, lower is better; structurally unsupported methods are missing, not scored zero. Dataset and horizon-group cohorts stay separate.","zh":"风电预测模型在UEBB上使用相同数据划分与不重叠的滚动测试窗口。Short行平均12/24步预测，历史窗口576步；可训练模型只在验证集调参。保留无量纲nMAE，数值越低越好。结构上不适用的方法缺失，不补零；数据集及预测跨度分别排行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.24444v1","published_at":"2026-09-21","status":"paper_baselines"},"higher_is_better":false,"model_count":19,"result_count":19,"results":[[9464,0.107,370,11855,11850],[11834,0.107,370,11855,11850],[11835,0.107,370,11855,11850],[9161,0.109,370,11855,11850],[9682,0.113,370,11855,11850],[11836,0.108,370,11855,11850],[6812,0.117,370,11855,11850],[11847,0.114,370,11855,11850],[11837,0.11,370,11855,11850],[11838,0.109,370,11855,11850],[11839,0.11,370,11855,11850],[11840,0.109,370,11855,11850],[11841,0.107,370,11855,11850],[11842,0.112,370,11855,11850],[11843,0.112,370,11855,11850],[11851,0.105,370,11855,11850],[11844,0.103,370,11855,11850],[11852,0.11,370,11855,11850],[11853,0.11,370,11855,11850]]},{"id":"wpbench-uebb-long-2026-paper","name":"WPBench · UEBB long (2026 paper)","group":"预测与决策","metric":"Normalized MAE","unit":"raw","public_history":true,"protocol":{"en":"Wind-power forecasting models use the same published splits and non-overlapping rolling test windows for UEBB. The source Long row averages 72/144-step horizons with 576-step history. Trainable models tune only on validation data. Native nMAE is dimensionless, lower is better; structurally unsupported methods are missing, not scored zero. Dataset and horizon-group cohorts stay separate.","zh":"风电预测模型在UEBB上使用相同数据划分与不重叠的滚动测试窗口。Long行平均72/144步预测，历史窗口576步；可训练模型只在验证集调参。保留无量纲nMAE，数值越低越好。结构上不适用的方法缺失，不补零；数据集及预测跨度分别排行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.24444v1","published_at":"2026-09-21","status":"paper_baselines"},"higher_is_better":false,"model_count":19,"result_count":19,"results":[[9464,0.141,370,11856,11850],[11834,0.141,370,11856,11850],[11835,0.144,370,11856,11850],[9161,0.147,370,11856,11850],[9682,0.147,370,11856,11850],[11836,0.15,370,11856,11850],[6812,0.163,370,11856,11850],[11847,0.199,370,11856,11850],[11837,0.159,370,11856,11850],[11838,0.148,370,11856,11850],[11839,0.154,370,11856,11850],[11840,0.148,370,11856,11850],[11841,0.145,370,11856,11850],[11842,0.15,370,11856,11850],[11843,0.149,370,11856,11850],[11851,0.168,370,11856,11850],[11844,0.142,370,11856,11850],[11852,0.148,370,11856,11850],[11853,0.155,370,11856,11850]]},{"id":"omicsbench-epigenetic-mcc-2026-paper","name":"OmicsBench · epigenetic-mcc (2026 paper)","group":"数学与科学","metric":"MCC ×100","unit":"raw","public_history":true,"protocol":{"en":"General and scientific language models answer expert-validated DNA/RNA/protein questions on the same applicable task split of the 1,160-question benchmark. This cohort reports MCC ×100. MCC, AUROC, Fmax and rubric evidence coverage remain native metrics; only ncRNA exact classification uses a comparable percentage. ChatNT supports only three tasks, so missing task results and its partial-task mean are excluded from full-six-task comparisons.","zh":"通用及科学语言模型在1,160题基准中回答经专家验证的DNA、RNA和蛋白质问题，本组采用相同适用任务划分，显示MCC ×100。MCC、AUROC、Fmax及量表证据覆盖保留原生含义；只有ncRNA精确分类采用可比百分比。ChatNT仅支持三类任务，其缺失任务及部分任务均值不混入完整六任务比较。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.23435v1","published_at":"2026-09-20","status":"paper_baselines"},"model_count":17,"result_count":17,"results":[[6704,3.56,370,11857,11858],[6130,-10.22,370,11857,11858],[340,14.38,370,11857,11858],[3250,-3.5,370,11857,11858],[3225,8.36,370,11857,11858],[3009,1.28,370,11857,11858],[2278,-5.29,370,11857,11858],[3419,20.96,370,11857,11858],[4819,9.65,370,11857,11858],[1583,14.72,370,11857,11858],[4823,1.05,370,11857,11858],[11859,10.44,370,11857,11858],[11860,86.86,370,11857,11858],[6676,-0.11,370,11857,11858],[7853,20.61,370,11857,11858],[11861,-6.66,370,11857,11858],[11862,29.67,370,11857,11858]]},{"id":"omicsbench-promoter-mcc-2026-paper","name":"OmicsBench · promoter-mcc (2026 paper)","group":"数学与科学","metric":"MCC ×100","unit":"raw","public_history":true,"protocol":{"en":"General and scientific language models answer expert-validated DNA/RNA/protein questions on the same applicable task split of the 1,160-question benchmark. This cohort reports MCC ×100. MCC, AUROC, Fmax and rubric evidence coverage remain native metrics; only ncRNA exact classification uses a comparable percentage. ChatNT supports only three tasks, so missing task results and its partial-task mean are excluded from full-six-task comparisons.","zh":"通用及科学语言模型在1,160题基准中回答经专家验证的DNA、RNA和蛋白质问题，本组采用相同适用任务划分，显示MCC ×100。MCC、AUROC、Fmax及量表证据覆盖保留原生含义；只有ncRNA精确分类采用可比百分比。ChatNT仅支持三类任务，其缺失任务及部分任务均值不混入完整六任务比较。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.23435v1","published_at":"2026-09-20","status":"paper_baselines"},"model_count":17,"result_count":17,"results":[[6704,28.58,370,11857,11858],[6130,16.61,370,11857,11858],[340,15.97,370,11857,11858],[3250,30.67,370,11857,11858],[3225,21.35,370,11857,11858],[3009,23.78,370,11857,11858],[2278,12.16,370,11857,11858],[3419,18.98,370,11857,11858],[4819,0.94,370,11857,11858],[1583,17.53,370,11857,11858],[4823,2.97,370,11857,11858],[11859,20.66,370,11857,11858],[11860,40.76,370,11857,11858],[6676,19.24,370,11857,11858],[7853,47.93,370,11857,11858],[11861,32.41,370,11857,11858],[11862,31.29,370,11857,11858]]},{"id":"omicsbench-tfbs-mcc-2026-paper","name":"OmicsBench · tfbs-mcc (2026 paper)","group":"数学与科学","metric":"MCC ×100","unit":"raw","public_history":true,"protocol":{"en":"General and scientific language models answer expert-validated DNA/RNA/protein questions on the same applicable task split of the 1,160-question benchmark. This cohort reports MCC ×100. MCC, AUROC, Fmax and rubric evidence coverage remain native metrics; only ncRNA exact classification uses a comparable percentage. ChatNT supports only three tasks, so missing task results and its partial-task mean are excluded from full-six-task comparisons.","zh":"通用及科学语言模型在1,160题基准中回答经专家验证的DNA、RNA和蛋白质问题，本组采用相同适用任务划分，显示MCC ×100。MCC、AUROC、Fmax及量表证据覆盖保留原生含义；只有ncRNA精确分类采用可比百分比。ChatNT仅支持三类任务，其缺失任务及部分任务均值不混入完整六任务比较。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.23435v1","published_at":"2026-09-20","status":"paper_baselines"},"model_count":17,"result_count":17,"results":[[6704,12.5,370,11857,11858],[6130,2.32,370,11857,11858],[340,-0.81,370,11857,11858],[3250,2.09,370,11857,11858],[3225,-0.98,370,11857,11858],[3009,8.91,370,11857,11858],[2278,6.96,370,11857,11858],[3419,0.17,370,11857,11858],[4819,4.85,370,11857,11858],[1583,4.89,370,11857,11858],[4823,0.68,370,11857,11858],[11859,21.83,370,11857,11858],[11860,12.46,370,11857,11858],[6676,21.81,370,11857,11858],[7853,43.95,370,11857,11858],[11861,10.87,370,11857,11858],[11862,44.37,370,11857,11858]]},{"id":"omicsbench-rna-modification-auc-2026-paper","name":"OmicsBench · rna-modification-auc (2026 paper)","group":"数学与科学","metric":"AUROC ×100","unit":"score","public_history":true,"protocol":{"en":"General and scientific language models answer expert-validated DNA/RNA/protein questions on the same applicable task split of the 1,160-question benchmark. This cohort reports AUROC ×100. MCC, AUROC, Fmax and rubric evidence coverage remain native metrics; only ncRNA exact classification uses a comparable percentage. ChatNT supports only three tasks, so missing task results and its partial-task mean are excluded from full-six-task comparisons.","zh":"通用及科学语言模型在1,160题基准中回答经专家验证的DNA、RNA和蛋白质问题，本组采用相同适用任务划分，显示AUROC ×100。MCC、AUROC、Fmax及量表证据覆盖保留原生含义；只有ncRNA精确分类采用可比百分比。ChatNT仅支持三类任务，其缺失任务及部分任务均值不混入完整六任务比较。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.23435v1","published_at":"2026-09-20","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":16,"result_count":16,"results":[[6704,53.42,370,11857,11858],[6130,51.57,370,11857,11858],[340,53.17,370,11857,11858],[3250,50.28,370,11857,11858],[3225,53.1,370,11857,11858],[3009,52.25,370,11857,11858],[2278,50.9,370,11857,11858],[3419,52.48,370,11857,11858],[4819,47.24,370,11857,11858],[1583,53.5,370,11857,11858],[4823,54.12,370,11857,11858],[11859,59.72,370,11857,11858],[6676,51.36,370,11857,11858],[7853,52.27,370,11857,11858],[11861,52.04,370,11857,11858],[11862,50.52,370,11857,11858]]},{"id":"omicsbench-ncrna-accuracy-2026-paper","name":"OmicsBench · ncrna-accuracy (2026 paper)","group":"数学与科学","metric":"ncRNA classification accuracy","unit":"%","public_history":true,"protocol":{"en":"General and scientific language models answer expert-validated DNA/RNA/protein questions on the same applicable task split of the 1,160-question benchmark. This cohort reports ncRNA classification accuracy. MCC, AUROC, Fmax and rubric evidence coverage remain native metrics; only ncRNA exact classification uses a comparable percentage. ChatNT supports only three tasks, so missing task results and its partial-task mean are excluded from full-six-task comparisons.","zh":"通用及科学语言模型在1,160题基准中回答经专家验证的DNA、RNA和蛋白质问题，本组采用相同适用任务划分，显示ncRNA classification accuracy。MCC、AUROC、Fmax及量表证据覆盖保留原生含义；只有ncRNA精确分类采用可比百分比。ChatNT仅支持三类任务，其缺失任务及部分任务均值不混入完整六任务比较。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.23435v1","published_at":"2026-09-20","status":"paper_baselines"},"model_count":16,"result_count":16,"results":[[6704,0.12560000000000002,370,11857,11858],[6130,0.2419,370,11857,11858],[340,0.037200000000000004,370,11857,11858],[3250,0.13949999999999999,370,11857,11858],[3225,0.1907,370,11857,11858],[3009,0.0977,370,11857,11858],[2278,0.07440000000000001,370,11857,11858],[3419,0.08839999999999999,370,11857,11858],[4819,0.04650000000000001,370,11857,11858],[1583,0.0698,370,11857,11858],[4823,0.0512,370,11857,11858],[11859,0.8326,370,11857,11858],[6676,0.11630000000000001,370,11857,11858],[7853,0.17670000000000002,370,11857,11858],[11861,0.0,370,11857,11858],[11862,0.06509999999999999,370,11857,11858]]},{"id":"omicsbench-enzyme-fmax-2026-paper","name":"OmicsBench · enzyme-fmax (2026 paper)","group":"数学与科学","metric":"Enzyme function Fmax","unit":"score","public_history":true,"protocol":{"en":"General and scientific language models answer expert-validated DNA/RNA/protein questions on the same applicable task split of the 1,160-question benchmark. This cohort reports Enzyme function Fmax. MCC, AUROC, Fmax and rubric evidence coverage remain native metrics; only ncRNA exact classification uses a comparable percentage. ChatNT supports only three tasks, so missing task results and its partial-task mean are excluded from full-six-task comparisons.","zh":"通用及科学语言模型在1,160题基准中回答经专家验证的DNA、RNA和蛋白质问题，本组采用相同适用任务划分，显示Enzyme function Fmax。MCC、AUROC、Fmax及量表证据覆盖保留原生含义；只有ncRNA精确分类采用可比百分比。ChatNT仅支持三类任务，其缺失任务及部分任务均值不混入完整六任务比较。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.23435v1","published_at":"2026-09-20","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":16,"result_count":16,"results":[[6704,8.04,370,11857,11858],[6130,24.68,370,11857,11858],[340,8.62,370,11857,11858],[3250,23.39,370,11857,11858],[3225,10.91,370,11857,11858],[3009,11.68,370,11857,11858],[2278,8.94,370,11857,11858],[3419,11.46,370,11857,11858],[4819,16.58,370,11857,11858],[1583,10.42,370,11857,11858],[4823,7.82,370,11857,11858],[11859,23.12,370,11857,11858],[6676,13.56,370,11857,11858],[7853,39.53,370,11857,11858],[11861,14.76,370,11857,11858],[11862,77.41,370,11857,11858]]},{"id":"omicsbench-reasoning-rubric-2026-paper","name":"OmicsBench · reasoning-rubric (2026 paper)","group":"数学与科学","metric":"Mean six-task Rubric Recall","unit":"score","public_history":true,"protocol":{"en":"General and scientific language models answer expert-validated DNA/RNA/protein questions on the same applicable task split of the 1,160-question benchmark. This cohort reports Mean six-task Rubric Recall. MCC, AUROC, Fmax and rubric evidence coverage remain native metrics; only ncRNA exact classification uses a comparable percentage. ChatNT supports only three tasks, so missing task results and its partial-task mean are excluded from full-six-task comparisons.","zh":"通用及科学语言模型在1,160题基准中回答经专家验证的DNA、RNA和蛋白质问题，本组采用相同适用任务划分，显示Mean six-task Rubric Recall。MCC、AUROC、Fmax及量表证据覆盖保留原生含义；只有ncRNA精确分类采用可比百分比。ChatNT仅支持三类任务，其缺失任务及部分任务均值不混入完整六任务比较。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.23435v1","published_at":"2026-09-20","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":16,"result_count":16,"results":[[6704,16.11,370,11857,11858],[6130,16.17,370,11857,11858],[340,11.09,370,11857,11858],[3250,21.1,370,11857,11858],[3225,18.53,370,11857,11858],[3009,12.62,370,11857,11858],[2278,10.78,370,11857,11858],[3419,15.63,370,11857,11858],[4819,9.19,370,11857,11858],[1583,4.07,370,11857,11858],[4823,16.09,370,11857,11858],[11859,5.06,370,11857,11858],[6676,8.89,370,11857,11858],[7853,2.41,370,11857,11858],[11861,0.23,370,11857,11858],[11862,1.39,370,11857,11858]]},{"id":"uk-prbench-single-paragraph-2026-paper","name":"UK-PRBench · single-paragraph (2026 paper)","group":"专业工作","metric":"Recall@10","unit":"score","public_history":true,"protocol":{"en":"Retrieval systems rank an 8,739-paragraph candidate pool for the same 4,403 held-out query instances; train/dev contain 15,363/2,128 anchors. This single-paragraph condition is separate: contextual retrieval adds neighboring paragraphs to both query and target. Zero-shot and three-epoch fine-tuned results keep their training configuration; source checkpoint names are preserved without counting training variants as new base models.","zh":"检索系统为相同4,403条留出查询在8,739个候选段落中排序，训练及验证分别含15,363和2,128个锚实例。本组单列single-paragraph条件；上下文检索会为查询及目标加入相邻段落。零样本和三轮微调保留训练配置，不将同底模训练版本凑作新模型。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.24613v1","published_at":"2026-09-21","status":"paper_baselines"},"score_max":1,"saturation_excluded":true,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"score_decimals":2,"model_count":10,"result_count":18,"results":[[8053,0.57,370,11863,11864],[11865,0.52,370,11863,11864],[9738,0.62,370,11863,11864],[9899,0.67,370,11863,11864],[11866,0.62,370,11863,11864],[11867,0.68,370,11863,11864],[11868,0.61,370,11863,11864],[11869,0.64,370,11863,11864],[11870,0.68,370,11871,11864],[11872,0.58,370,11871,11864],[8053,0.26,370,11871,11864],[11865,0.77,370,11871,11864],[9738,0.07,370,11871,11864],[9899,0.59,370,11871,11864],[11866,0.22,370,11871,11864],[11867,0.65,370,11871,11864],[11868,0.66,370,11871,11864],[11869,0.65,370,11871,11864]]},{"id":"uk-prbench-contextual-2026-paper","name":"UK-PRBench · contextual (2026 paper)","group":"专业工作","metric":"Recall@10","unit":"score","public_history":true,"protocol":{"en":"Retrieval systems rank an 8,739-paragraph candidate pool for the same 4,403 held-out query instances; train/dev contain 15,363/2,128 anchors. This contextual condition is separate: contextual retrieval adds neighboring paragraphs to both query and target. Zero-shot and three-epoch fine-tuned results keep their training configuration; source checkpoint names are preserved without counting training variants as new base models.","zh":"检索系统为相同4,403条留出查询在8,739个候选段落中排序，训练及验证分别含15,363和2,128个锚实例。本组单列contextual条件；上下文检索会为查询及目标加入相邻段落。零样本和三轮微调保留训练配置，不将同底模训练版本凑作新模型。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.24613v1","published_at":"2026-09-21","status":"paper_baselines"},"score_max":1,"saturation_excluded":true,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"score_decimals":2,"model_count":10,"result_count":18,"results":[[8053,0.49,370,11873,11864],[11865,0.38,370,11873,11864],[9738,0.56,370,11873,11864],[9899,0.61,370,11873,11864],[11866,0.56,370,11873,11864],[11867,0.65,370,11873,11864],[11868,0.55,370,11873,11864],[11869,0.57,370,11873,11864],[11870,0.59,370,11874,11864],[11872,0.46,370,11874,11864],[8053,0.23,370,11874,11864],[11865,0.59,370,11874,11864],[9738,0.04,370,11874,11864],[9899,0.83,370,11874,11864],[11866,0.0,370,11874,11864],[11867,0.79,370,11874,11864],[11868,0.79,370,11874,11864],[11869,0.57,370,11874,11864]]},{"id":"time-aware-mri-final-accuracy-2026-paper","name":"Time-Aware Multi-View MRI · longitudinal reasoning (2026 paper)","group":"医疗","metric":"Published final-answer accuracy","unit":"%","public_history":true,"protocol":{"en":"All sixteen systems evaluate the same 3,920 expert-verified questions from 890 patients and more than 3,200 longitudinal MRI timepoints. The source’s Final Acc percentage is retained, separately from reasoning score, temporal alignment, trend F1, coverage and chronology metrics. Original API/checkpoint names remain explicit.","zh":"完整十六个系统评测同一3,920道专家核验问题，覆盖890名患者和3,200多个纵向MRI时间点。保留原始Final Acc百分比，与推理分数、时间对齐、趋势F1、覆盖和时序指标分开。API或检查点原名明确保留。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.13309v1","status":"paper_baselines","published_at":"2026-08-13"},"model_count":16,"result_count":16,"results":[[5929,0.3218,370,11875,11876],[353,0.32,370,11875,11876],[340,0.212,370,11875,11876],[766,0.2357,370,11875,11876],[764,0.2366,370,11875,11876],[6133,0.223,370,11875,11876],[6130,0.35100000000000003,370,11875,11876],[11877,0.3515,370,11875,11876],[11878,0.2838,370,11875,11876],[11879,0.3037,370,11875,11876],[11880,0.24309999999999998,370,11875,11876],[11881,0.2847,370,11875,11876],[11882,0.26839999999999997,370,11875,11876],[11883,0.1913,370,11875,11876],[11884,0.218,370,11875,11876],[7580,0.235,370,11875,11876]]},{"id":"tomabench-full-mcq-2026-paper","name":"TomaBench · tomato disease understanding (2026 paper)","group":"专业工作","metric":"Seven-task mean MCQ accuracy","unit":"%","public_history":true,"protocol":{"en":"The complete thirteen-system full-coverage panel answers the same seven-category tomato-understanding MCQs. The published mean accuracy is separate from macro-F1. TomaLLaVA lacks the crop-species category and its reported mean covers a different task set, so that partial-coverage row is not part of this seven-task cohort. Generative VLMs and contrastive encoders retain their source inference modes.","zh":"完整十三系统面板覆盖同一七类番茄理解选择题，保留论文平均准确率，与宏F1分开。TomaLLaVA未报告作物物种类别，其均值覆盖范围不同，不放入七任务面板。生成式视觉模型与对比编码器保留原始推理方式。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.08727v1","status":"paper_baselines","published_at":"2026-08-09"},"model_count":13,"result_count":13,"results":[[67,0.703,370,11885,11886],[11887,0.45399999999999996,370,11885,11886],[8141,0.248,370,11885,11886],[2780,0.493,370,11885,11886],[2766,0.5479999999999999,370,11885,11886],[4476,0.5870000000000001,370,11885,11886],[8341,0.5920000000000001,370,11885,11886],[7971,0.521,370,11885,11886],[11888,0.613,370,11885,11886],[8349,0.418,370,11889,11886],[11890,0.49700000000000005,370,11889,11886],[11891,0.602,370,11889,11886],[11892,0.5589999999999999,370,11889,11886]]},{"id":"trace-reasoning-guardrails-2026-paper","name":"TRACE · reasoning-trace safety (2026 paper)","group":"安全与可靠性","metric":"Reasoning-trace F1, combined TRACE","unit":"score","public_history":true,"protocol":{"en":"The complete eighteen-checkpoint guardrail panel evaluates the same bilingual TRACE examples. The selected native 0–100 F1 is the combined TRACE reasoning-trace column; prompt and final-response F1 are separate. Merged family/parameter cells are resolved through exact source coordinates, retaining all published model scales.","zh":"完整十八检查点防护模型评测同一双语TRACE样本。所选0–100原生F1来自推理轨迹的TRACE合并列，与提示和最终回答的F1分开。跨行模型族和参数单元按准确原坐标解析，保留全部已发布规模。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.24232v1","status":"paper_baselines","published_at":"2026-08-25"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":18,"result_count":18,"results":[[11893,24.34,370,11894,11895],[11896,63.67,370,11897,11895],[11898,52.85,370,11899,11895],[11900,63.08,370,11897,11895],[11901,50.73,370,11902,11895],[11903,53.92,370,11904,11895],[11905,59.02,370,11906,11895],[11907,59.63,370,11894,11895],[11908,77.74,370,11897,11895],[11909,80.1,370,11910,11895],[11911,79.6,370,11912,11895],[11913,71.79,370,11914,11895],[11915,72.04,370,11916,11895],[11917,75.73,370,11897,11895],[11918,65.91,370,11919,11895],[11920,82.57,370,11897,11895],[11921,76.55,370,11914,11895],[11922,84.26,370,11897,11895]]},{"id":"wesce-complete-clearance-2026-paper","name":"WeSCE · security drift in code editing (2026 paper)","group":"安全与可靠性","metric":"Complete vulnerability-clearance rate (R_c)","unit":"%","public_history":true,"protocol":{"en":"All eight LLMs edit the same 400 executable programs under functional prompts without explicit security instructions. The full source suite covers addition, removal, bug fixing and refactoring. The selected TOTAL R_c is the published complete-clearance rate; continuous risk-reduction and distribution-shift metrics remain separate.","zh":"完整八个模型在没有显式安全要求的功能性提示下编辑相同400个可执行程序，覆盖新增、移除、修复和重构。采用TOTAL列已发布的完全清除漏洞率R_c，与连续风险下降和分布变化指标分开。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.15092v1","status":"paper_baselines","published_at":"2026-08-15"},"model_count":8,"result_count":8,"results":[[2261,0.5125,370,11923,11924],[3184,0.505,370,11923,11924],[323,0.505,370,11923,11924],[11925,0.4925,370,11923,11924],[11926,0.3975,370,11923,11924],[325,0.385,370,11923,11924],[11927,0.365,370,11923,11924],[2061,0.315,370,11923,11924]]},{"id":"turnbench-budgeted-eot-recall-2026-paper","name":"TurnBench · budgeted end-of-turn detection (2026 paper)","group":"专业工作","metric":"End-of-turn recall at frozen operating point","unit":"score","public_history":true,"protocol":{"en":"The twelve in-budget systems use the same 116 held-out conversations. Sweepable models select the highest-recall threshold within a 0.1 development false-positive budget, then freeze it for test. The first value of each published recall/FPR pair is shown on its native 0–1 scale; test FPR and median latency remain in model details. The out-of-budget RMS and acoustic Server VAD reference points are separate.","zh":"十二个预算内系统使用同一116段留出对话。可扫阈值模型在开发集0.1假阳性预算内选最高召回阈值，再固定用于测试。采用原表recall/FPR对的前项、保留0–1原生召回率，测试假阳性率和中位延迟留在模型详情。超出预算的RMS和声学Server VAD参考点单列。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.25218v2","status":"paper_baselines","published_at":"2026-08-25"},"score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":12,"result_count":12,"results":[[11928,0.303,370,11929,11930],[11931,0.773,370,11932,11930],[11933,0.752,370,11934,11930],[11935,0.845,370,11936,11930],[11937,0.782,370,11938,11930],[11939,0.826,370,11940,11930],[11941,0.711,370,11942,11930],[11943,0.403,370,11944,11930],[11945,0.408,370,11946,11930],[11947,0.8,370,11948,11930],[11949,0.657,370,11950,11930],[6601,0.233,370,11951,11930]]},{"id":"rtcbench-controller-suite-2026-release","name":"RTCbench · controller commissioning (2026-09-24 release)","group":"数学与科学","metric":"Signed suite controller score","unit":"raw","public_history":true,"protocol":{"en":"The complete-case 27-model aggregate covers the same ten content-hashed plant tasks and twenty evaluation seeds per task, with three commissioning-suite runs per model. Claude Fable 5.1 lacks one deadtime-process run and is excluded from this matched cohort. Exact signed means, including negative values, are retained. This dated artifact specifies 200 plant tests per suite; the older README’s 190-test summary is not used. The task matrix, per-plant outcomes, gates and crashes remain in the original artifact.","zh":"完整覆盖的27模型汇总使用同一十个带内容哈希的对象任务，每任务二十个评测种子，每模型三次调试套件运行。Claude Fable 5.1缺少一次纯滞后过程运行，不进入这个配对面板。保留精确有符号均值，包括负值。该日期产物明确每套件200次对象测试，不采用旧README的190次摘要。任务矩阵、逐对象结果、门槛失败与崩溃记录均保存在原产物。"},"source":{"name":"Paper authors","url":"https://github.com/aiast1/rtcbench","status":"dated_release","published_at":"2026-08-20"},"score_decimals":3,"saturation_excluded":true,"scale_label":{"en":"Signed controller score","zh":"控制器原生分数"},"model_count":27,"result_count":27,"results":[[4425,-0.2330296306414323,370,11952,11953],[2977,0.3337649905789554,370,11954,11953],[11955,-0.28947628256499197,370,11956,11953],[4424,-0.15930523115205808,370,11957,11953],[5384,-0.058966818385086085,370,11958,11953],[244,-0.116989922721889,370,11959,11953],[5602,-0.1184822123610572,370,11960,11953],[11961,-0.15048076068360391,370,11962,11953],[11963,-0.35645821106680065,370,11964,11953],[11965,-0.25426486074878035,370,11966,11953],[3453,0.5763555504945909,370,11967,11953],[325,-0.3130111068657392,370,11968,11953],[3479,0.1663509477693815,370,11969,11953],[755,0.3755499580307469,370,11970,11953],[2959,0.6447959691352197,370,11971,11953],[3568,0.2084304325986738,370,11972,11953],[1487,0.08022321651264368,370,11973,11953],[11974,-0.34252254543687516,370,11975,11953],[11976,-0.1873415458639799,370,11977,11953],[7311,0.29415006110048075,370,11978,11953],[5004,0.4714258526582531,370,11979,11953],[6591,0.36718254761302954,370,11980,11953],[5508,0.517279812482951,370,11981,11953],[6234,0.4051691751833569,370,11982,11953],[7058,0.24037547493707465,370,11983,11953],[2974,0.2987383039957266,370,11984,11953],[268,-0.015046238864346695,370,11985,11953]]},{"id":"reconstruction-default-match-2026-paper","name":"Reconstruction · scientific idea recovery (2026 paper)","group":"数学与科学","metric":"Six-domain mean Match rate","unit":"%","public_history":true,"protocol":{"en":"All seven default proposers recover ideas from frozen pre-publication bibliographies for the same 643 retained papers across six scientific domains. Each proposer is judged by the other six models under the published leave-one-out rule. The Avg column is the reported domain mean; ± denotes cross-domain dispersion. The top-four dagger and multi-agent blocks use different judge panels and are separate.","zh":"七个默认提议模型使用同一643篇论文、六个科学领域的冻结出版前参考文献恢复研究构想。每个提议模型按论文的留一规则交由其余六模型评分。采用Avg列的领域均值，±表示跨领域离散度。四模型†和多智能体分块采用不同评委面板，单独处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.16645v3","status":"paper_baselines","published_at":"2026-08-17"},"model_count":7,"result_count":7,"results":[[3194,0.133,370,11986,11987],[11988,0.128,370,11986,11987],[3155,0.1,370,11986,11987],[227,0.09300000000000001,370,11986,11987],[11989,0.08900000000000001,370,11986,11987],[3185,0.063,370,11986,11987],[3202,0.059000000000000004,370,11986,11987]]},{"id":"tarbench-traffic-understanding-2026-paper","name":"TAR-Bench · traffic video understanding (2026 paper)","group":"视觉与视频","metric":"Ten-task mean score","unit":"score","public_history":true,"protocol":{"en":"All eleven evaluated models use the same 960-item traffic benchmark in zero-shot non-reasoning mode. The native 0–100 mean combines accuracy, BERTScore-F1 and temporal mIoU across ten tasks and is not an accuracy percentage. Cosmos3 Nano and Super were exposed to TAR in base-model training; this source distinction remains explicit. The model-average reference row is excluded.","zh":"完整十一个模型以零样本、非推理模式评测同一960题交通基准。0–100原生均值综合十项任务的准确率、BERTScore-F1和时序mIoU，不作为准确率百分比。Cosmos3 Nano和Super在底模训练中接触过TAR，这一原始条件明确保留。排除模型平均参考行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.10317v3","status":"paper_baselines","published_at":"2026-08-10"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":11,"result_count":11,"results":[[3412,36.3,370,11990,11991],[11992,25.4,370,11990,11991],[6374,33.3,370,11990,11991],[6342,30.9,370,11990,11991],[6343,28.0,370,11990,11991],[4070,39.1,370,11990,11991],[2584,39.5,370,11990,11991],[7066,34.3,370,11990,11991],[7554,36.7,370,11990,11991],[8303,44.1,370,11993,11991],[11814,48.8,370,11993,11991]]},{"id":"vialect-standard-2026-paper","name":"VialectBench · Standard (2026 paper)","group":"知识与推理","metric":"Four-task mean score","unit":"score","public_history":true,"protocol":{"en":"The complete ten-model direct-prompting panel evaluates aligned Vietnamese items in the Standard variant. The published mean combines MCQA accuracy, NLI accuracy, QA F1 and emotion accuracy on a native 0–100 scale. It is kept separate for each variant; arrows and adjacent values are gaps from Standard and are not scores.","zh":"完整十模型直接提示面板评测Standard版本的配对越南语题目。论文均值综合选择题准确率、自然语言推断准确率、问答F1和情感分类准确率，保留0–100原生单位，各变体分开。箭头和相邻数字是相对标准语的差距，不作为分数。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.10414v1","status":"paper_baselines","published_at":"2026-08-11"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":10,"result_count":10,"results":[[11994,41.0,370,11995,11996],[11997,66.8,370,11995,11996],[7265,66.9,370,11995,11996],[806,64.3,370,11995,11996],[7263,52.8,370,11995,11996],[6933,72.5,370,11995,11996],[4043,66.6,370,11995,11996],[11998,51.3,370,11995,11996],[11999,62.1,370,11995,11996],[353,77.8,370,11995,11996]]},{"id":"vialect-pnb-2026-paper","name":"VialectBench · PNB (2026 paper)","group":"知识与推理","metric":"Four-task mean score","unit":"score","public_history":true,"protocol":{"en":"The complete ten-model direct-prompting panel evaluates aligned Vietnamese items in the PNB variant. The published mean combines MCQA accuracy, NLI accuracy, QA F1 and emotion accuracy on a native 0–100 scale. It is kept separate for each variant; arrows and adjacent values are gaps from Standard and are not scores.","zh":"完整十模型直接提示面板评测PNB版本的配对越南语题目。论文均值综合选择题准确率、自然语言推断准确率、问答F1和情感分类准确率，保留0–100原生单位，各变体分开。箭头和相邻数字是相对标准语的差距，不作为分数。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.10414v1","status":"paper_baselines","published_at":"2026-08-11"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":10,"result_count":10,"results":[[11994,39.6,370,12000,11996],[11997,66.3,370,12000,11996],[7265,67.5,370,12000,11996],[806,65.1,370,12000,11996],[7263,55.0,370,12000,11996],[6933,73.0,370,12000,11996],[4043,68.0,370,12000,11996],[11998,51.5,370,12000,11996],[11999,62.8,370,12000,11996],[353,77.7,370,12000,11996]]},{"id":"vialect-pnn-2026-paper","name":"VialectBench · PNN (2026 paper)","group":"知识与推理","metric":"Four-task mean score","unit":"score","public_history":true,"protocol":{"en":"The complete ten-model direct-prompting panel evaluates aligned Vietnamese items in the PNN variant. The published mean combines MCQA accuracy, NLI accuracy, QA F1 and emotion accuracy on a native 0–100 scale. It is kept separate for each variant; arrows and adjacent values are gaps from Standard and are not scores.","zh":"完整十模型直接提示面板评测PNN版本的配对越南语题目。论文均值综合选择题准确率、自然语言推断准确率、问答F1和情感分类准确率，保留0–100原生单位，各变体分开。箭头和相邻数字是相对标准语的差距，不作为分数。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.10414v1","status":"paper_baselines","published_at":"2026-08-11"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":10,"result_count":10,"results":[[11994,39.4,370,12001,11996],[11997,65.4,370,12001,11996],[7265,64.4,370,12001,11996],[806,62.1,370,12001,11996],[7263,49.0,370,12001,11996],[6933,70.7,370,12001,11996],[4043,66.2,370,12001,11996],[11998,50.0,370,12001,11996],[11999,61.1,370,12001,11996],[353,75.8,370,12001,11996]]},{"id":"vialect-pnt1-2026-paper","name":"VialectBench · PNT1 (2026 paper)","group":"知识与推理","metric":"Four-task mean score","unit":"score","public_history":true,"protocol":{"en":"The complete ten-model direct-prompting panel evaluates aligned Vietnamese items in the PNT1 variant. The published mean combines MCQA accuracy, NLI accuracy, QA F1 and emotion accuracy on a native 0–100 scale. It is kept separate for each variant; arrows and adjacent values are gaps from Standard and are not scores.","zh":"完整十模型直接提示面板评测PNT1版本的配对越南语题目。论文均值综合选择题准确率、自然语言推断准确率、问答F1和情感分类准确率，保留0–100原生单位，各变体分开。箭头和相邻数字是相对标准语的差距，不作为分数。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.10414v1","status":"paper_baselines","published_at":"2026-08-11"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":10,"result_count":10,"results":[[11994,36.8,370,12002,11996],[11997,64.2,370,12002,11996],[7265,64.0,370,12002,11996],[806,61.6,370,12002,11996],[7263,52.0,370,12002,11996],[6933,70.7,370,12002,11996],[4043,63.5,370,12002,11996],[11998,50.8,370,12002,11996],[11999,58.7,370,12002,11996],[353,77.4,370,12002,11996]]},{"id":"vialect-pnt2-2026-paper","name":"VialectBench · PNT2 (2026 paper)","group":"知识与推理","metric":"Four-task mean score","unit":"score","public_history":true,"protocol":{"en":"The complete ten-model direct-prompting panel evaluates aligned Vietnamese items in the PNT2 variant. The published mean combines MCQA accuracy, NLI accuracy, QA F1 and emotion accuracy on a native 0–100 scale. It is kept separate for each variant; arrows and adjacent values are gaps from Standard and are not scores.","zh":"完整十模型直接提示面板评测PNT2版本的配对越南语题目。论文均值综合选择题准确率、自然语言推断准确率、问答F1和情感分类准确率，保留0–100原生单位，各变体分开。箭头和相邻数字是相对标准语的差距，不作为分数。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.10414v1","status":"paper_baselines","published_at":"2026-08-11"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":10,"result_count":10,"results":[[11994,33.9,370,12003,11996],[11997,60.0,370,12003,11996],[7265,62.7,370,12003,11996],[806,58.0,370,12003,11996],[7263,45.4,370,12003,11996],[6933,68.5,370,12003,11996],[4043,63.2,370,12003,11996],[11998,48.6,370,12003,11996],[11999,58.0,370,12003,11996],[353,76.6,370,12003,11996]]},{"id":"vialect-pnt3-2026-paper","name":"VialectBench · PNT3 (2026 paper)","group":"知识与推理","metric":"Four-task mean score","unit":"score","public_history":true,"protocol":{"en":"The complete ten-model direct-prompting panel evaluates aligned Vietnamese items in the PNT3 variant. The published mean combines MCQA accuracy, NLI accuracy, QA F1 and emotion accuracy on a native 0–100 scale. It is kept separate for each variant; arrows and adjacent values are gaps from Standard and are not scores.","zh":"完整十模型直接提示面板评测PNT3版本的配对越南语题目。论文均值综合选择题准确率、自然语言推断准确率、问答F1和情感分类准确率，保留0–100原生单位，各变体分开。箭头和相邻数字是相对标准语的差距，不作为分数。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.10414v1","status":"paper_baselines","published_at":"2026-08-11"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":10,"result_count":10,"results":[[11994,31.8,370,12004,11996],[11997,56.4,370,12004,11996],[7265,61.3,370,12004,11996],[806,57.6,370,12004,11996],[7263,43.4,370,12004,11996],[6933,67.3,370,12004,11996],[4043,62.1,370,12004,11996],[11998,47.9,370,12004,11996],[11999,56.9,370,12004,11996],[353,75.9,370,12004,11996]]},{"id":"vialect-pnt4-2026-paper","name":"VialectBench · PNT4 (2026 paper)","group":"知识与推理","metric":"Four-task mean score","unit":"score","public_history":true,"protocol":{"en":"The complete ten-model direct-prompting panel evaluates aligned Vietnamese items in the PNT4 variant. The published mean combines MCQA accuracy, NLI accuracy, QA F1 and emotion accuracy on a native 0–100 scale. It is kept separate for each variant; arrows and adjacent values are gaps from Standard and are not scores.","zh":"完整十模型直接提示面板评测PNT4版本的配对越南语题目。论文均值综合选择题准确率、自然语言推断准确率、问答F1和情感分类准确率，保留0–100原生单位，各变体分开。箭头和相邻数字是相对标准语的差距，不作为分数。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.10414v1","status":"paper_baselines","published_at":"2026-08-11"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":10,"result_count":10,"results":[[11994,37.1,370,12005,11996],[11997,62.3,370,12005,11996],[7265,65.5,370,12005,11996],[806,63.0,370,12005,11996],[7263,49.5,370,12005,11996],[6933,70.0,370,12005,11996],[4043,62.9,370,12005,11996],[11998,49.9,370,12005,11996],[11999,60.0,370,12005,11996],[353,77.5,370,12005,11996]]},{"id":"wildhand-overall-2026-paper","name":"WildHandBench · handwriting recognition (2026 paper)","group":"视觉与视频","metric":"Overall handwriting recognition score","unit":"score","public_history":true,"protocol":{"en":"The full published numeric panel contains eighteen model rows on the same 500 handwritten images: 367 free-text, 81 tables and 52 formula images. The native overall score combines text edit similarity, table TEDS and formula CDM under the source formula; it is not exact-match accuracy. All systems share post-processing while general and OCR models retain their prescribed prompts. Human reference and prior-driven error rates are separate.","zh":"完整原表数字面板包含十八个模型，使用同一500张手写图像：367张文本、81张表格、52张公式。原生总分按论文公式综合文本编辑相似度、表格TEDS和公式CDM，不作为完全匹配准确率。后处理统一，通用与OCR模型保留各自规定提示。人类参考及先验驱动错误率另列。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.22959v1","status":"paper_baselines","published_at":"2026-08-24"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":18,"result_count":18,"results":[[3479,57.7,370,12006,12007],[3415,68.25,370,12006,12007],[3412,71.85,370,12006,12007],[7967,52.8,370,12006,12007],[9707,62.19,370,12006,12007],[6412,65.55,370,12006,12007],[3155,67.81,370,12006,12007],[12008,9.46,370,12006,12007],[12009,31.33,370,12006,12007],[12010,40.38,370,12006,12007],[12011,43.73,370,12006,12007],[12012,49.97,370,12006,12007],[10876,51.44,370,12006,12007],[12013,51.73,370,12006,12007],[10878,52.07,370,12006,12007],[9690,52.82,370,12006,12007],[9692,54.06,370,12006,12007],[9050,54.21,370,12006,12007]]},{"id":"benchbench-protocol-rubric-2026-paper","name":"BenchBench-Protocol · wet-lab modifications (2026 paper)","group":"数学与科学","metric":"Single-attempt normalized rubric score","unit":"score","public_history":true,"protocol":{"en":"All nine systems answer the same 149 expert-reviewed protocol-modification tasks from 96 source protocols across nine wet-lab domains. The first-attempt column uses weighted dependency-aware rubrics, on the original normalized 0–1 scale. Expected best-of-two/five/ten values are separate configurations.","zh":"完整九系统面板回答同一149道专家复核的实验流程修改任务，来自九个湿实验领域的96份原始流程。采用首次尝试列，按考虑前后依赖的加权细则评分，保留0–1原生单位。两次、五次、十次择优另列为不同配置。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.23898v1","status":"paper_baselines","published_at":"2026-08-24"},"score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":9,"result_count":9,"results":[[755,0.592,370,12014,12015],[762,0.471,370,12014,12015],[2977,0.457,370,12014,12015],[2973,0.453,370,12014,12015],[3408,0.413,370,12014,12015],[203,0.394,370,12014,12015],[3391,0.389,370,12014,12015],[3654,0.369,370,12014,12015],[3405,0.341,370,12014,12015]]},{"id":"circuitreason-1k-accuracy-2026-paper","name":"CircuitReason-1k · circuit reasoning (2026 paper)","group":"数学与科学","metric":"Overall answer accuracy","unit":"%","public_history":true,"protocol":{"en":"All nine systems solve the same 1,000 authentic textbook circuit problems. Conservative typed deterministic scoring and blinded multi-model semantic consensus retain missing and failed generations in the denominator. Three commercial chatbot systems and six open models retain their source interfaces.","zh":"完整九系统面板解答同一1,000道教材电路题。保守的类型确定性评分与隐藏模型身份的多模型语义共识结合，缺失与生成失败仍计入分母。三种商业聊天系统及六种开放模型保留原始接口。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.09374v1","status":"paper_baselines","published_at":"2026-08-10"},"model_count":9,"result_count":9,"results":[[4048,0.848,370,12016,12017],[3412,0.823,370,12016,12017],[3572,0.644,370,12016,12017],[2582,0.843,370,12016,12017],[3187,0.84,370,12016,12017],[5843,0.64,370,12016,12017],[6342,0.43799999999999994,370,12016,12017],[7153,0.172,370,12016,12017],[2654,0.063,370,12016,12017]]},{"id":"imageval-allskills-counting-2026-paper","name":"Imag-Eval · counting (2026 paper)","group":"生成与创作","metric":"Counting success rate","unit":"score","public_history":true,"protocol":{"en":"The full six-generator panel uses the same All Skills prompt setting across difficulty levels, with manual annotations from multiple annotators and verification. This cohort retains the published Counting success rate on its original scale. Model names were visible to annotators. Gemini’s missing outputs in the broader all-combinations analysis prevent treating that separate five-model aggregate as a six-model cohort.","zh":"完整六生成模型面板使用相同难度范围的All Skills提示设定，由多名标注员人工评测并复核。本面板保留原始Counting success rate单位。标注员可见模型名。Gemini在更广的全组合分析中存在缺失输出，该五模型汇总不作为六模型面板。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.29210v1","status":"paper_baselines","published_at":"2026-08-29"},"score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":6,"result_count":6,"results":[[8087,0.55,370,12018,12019],[12020,0.56,370,12018,12019],[12021,0.42,370,12018,12019],[12022,0.7,370,12018,12019],[12023,0.17,370,12018,12019],[12024,0.13,370,12018,12019]]},{"id":"imageval-allskills-text-wer-2026-paper","name":"Imag-Eval · text-wer (2026 paper)","group":"生成与创作","metric":"Text word error rate","unit":"raw","public_history":true,"protocol":{"en":"The full six-generator panel uses the same All Skills prompt setting across difficulty levels, with manual annotations from multiple annotators and verification. This cohort retains the published Text word error rate on its original scale. Model names were visible to annotators. Gemini’s missing outputs in the broader all-combinations analysis prevent treating that separate five-model aggregate as a six-model cohort.","zh":"完整六生成模型面板使用相同难度范围的All Skills提示设定，由多名标注员人工评测并复核。本面板保留原始Text word error rate单位。标注员可见模型名。Gemini在更广的全组合分析中存在缺失输出，该五模型汇总不作为六模型面板。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.29210v1","status":"paper_baselines","published_at":"2026-08-29"},"score_max":6,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"WER","zh":"词错误率"},"higher_is_better":false,"model_count":6,"result_count":6,"results":[[8087,0.49,370,12018,12019],[12020,2.46,370,12018,12019],[12021,4.25,370,12018,12019],[12022,0.29,370,12018,12019],[12023,5.55,370,12018,12019],[12024,4.31,370,12018,12019]]},{"id":"ocr-metareasoning-mrms-2026-paper","name":"OCR-MetaReasoning · answer correctness (2026 paper)","group":"视觉与视频","metric":"Meta-Reasoning Macro Score (MRMS)","unit":"score","public_history":true,"protocol":{"en":"All eighteen published configurations evaluate the same balanced 1,500-image suite crossing three reasoning directions and five OCR-object categories. MRMS is the automatic answer-correctness macro score; process-judge RPCS is separate. Medium identifies an effort setting, not a new base model; the panel exceeds six independent bases without counting those repeats. Published table values use the 0–100 score scale.","zh":"完整十八配置评测同一1,500图像平衡测试集，交叉三类推理方向与五类OCR对象。MRMS是自动答案正确性宏分数，过程评委的RPCS另列。Medium为推理强度配置，不是新底模；不靠这些重复配置也超过六个底模。保留原表0–100分数。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.30678v1","status":"paper_baselines","published_at":"2026-08-31"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":16,"result_count":18,"results":[[6237,89.3,370,12025,12026],[12027,87.7,370,12028,12026],[7674,86.4,370,12025,12026],[6978,84.3,370,12025,12026],[6721,84.2,370,12025,12026],[12029,84.0,370,12028,12026],[5021,83.3,370,12025,12026],[3184,83.0,370,12025,12026],[8145,75.2,370,12025,12026],[6235,72.0,370,12025,12026],[2600,81.5,370,12025,12026],[2998,79.6,370,12025,12026],[2613,77.0,370,12025,12026],[5722,75.9,370,12025,12026],[5482,74.7,370,12025,12026],[2626,74.5,370,12025,12026],[5485,65.0,370,12025,12026],[6150,59.0,370,12025,12026]]},{"id":"pcbnet-connectivity-2026-paper","name":"PCBnet · schematic connectivity (2026 paper)","group":"专业工作","metric":"Recovered pin-to-pin connectivity accuracy","unit":"%","public_history":true,"protocol":{"en":"All six evaluated systems reconstruct the same PCBnet test designs under the paper’s design-level 8:1:1 train/validation/test split. Connectivity accuracy is the proportion of correctly recovered pin-to-pin connections. The learned visual topology pipeline and five general-purpose models retain their distinct source configurations; component mAP and wire IoU are separate metrics.","zh":"完整六系统面板使用PCBnet相同测试设计，按设计层面8:1:1划分训练、验证和测试集。连接准确率是正确恢复的引脚间连接比例。学习式视觉拓扑流程与五个通用模型保留不同原始配置，组件mAP和导线IoU另列。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.27923v1","status":"paper_baselines","published_at":"2026-08-28"},"model_count":6,"result_count":6,"results":[[12030,0.8447,370,12031,12032],[3184,0.0874,370,12031,12032],[2261,0.0791,370,12031,12032],[26,0.0926,370,12031,12032],[3009,0.0702,370,12031,12032],[6412,0.0801,370,12031,12032]]},{"id":"physelite-answer-2026-paper","name":"PhysElite · Olympiad physics answers (2026 paper)","group":"数学与科学","metric":"Bilingual answer-level accuracy","unit":"%","public_history":true,"protocol":{"en":"The complete eighteen-model panel uses the same bilingual PhysElite Olympiad problems. The published answer-level accuracy is averaged over Chinese and English; process scores remain separate. Three LLM judges score responses, calibrated against 200 expert-graded problems. Source-marked text-only models receive no diagram input; reasoning modes remain explicit. Human reference is excluded.","zh":"完整十八模型面板使用相同双语PhysElite奥赛题。保留中英文平均答案准确率，过程分数另列。三模型评委评分，并用200道人类专家评分题校准。原表标记的纯文本模型不接收图像，推理模式明确保留。排除人类参考。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.25097v1","status":"paper_baselines","published_at":"2026-08-25"},"model_count":18,"result_count":18,"results":[[12033,0.337,370,12034,12035],[3183,0.281,370,12036,12035],[4075,0.262,370,12037,12035],[340,0.24,370,12034,12035],[6130,0.218,370,12034,12035],[2288,0.2,370,12034,12035],[6704,0.185,370,12036,12035],[6957,0.179,370,12034,12035],[889,0.14300000000000002,370,12036,12035],[353,0.10400000000000001,370,12036,12035],[5843,0.2,370,12036,12035],[1573,0.17,370,12038,12035],[6343,0.159,370,12036,12035],[6342,0.11599999999999999,370,12036,12035],[2166,0.098,370,12036,12035],[12039,0.086,370,12036,12035],[12040,0.075,370,12038,12035],[2754,0.017,370,12036,12035]]},{"id":"ecommerce-final-assets-2026-paper","name":"E-Commerce Bench · year-long operations (2026 paper)","group":"专业工作","metric":"Mean year-end assets (CNY thousands)","unit":"raw","public_history":true,"protocol":{"en":"The complete eighteen-configuration panel runs five year-long episodes per model under the same business brief and simulator. The selected value is mean final assets in thousands of Chinese yuan, retaining bankruptcy outcomes. The published seed schedule, renderer and tool rules apply to every system; repeated GLM 5.2 effort settings are configurations of one base. Thousands separators are parsed without changing the source cells.","zh":"完整十八配置面板在同一经营任务和模拟器中，每模型运行五次一年期经营。所选数值是年末资产均值，原单位为千元人民币，破产结果保留。所有系统采用论文种子安排、渲染器与工具规则；GLM 5.2的不同推理强度属于同一底模的配置。千分位分隔符仅为解析而移除，原单元格完整保留。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.30730v1","status":"paper_baselines","published_at":"2026-08-31"},"score_decimals":1,"saturation_excluded":true,"scale_label":{"en":"CNY thousands","zh":"千元人民币"},"model_count":17,"result_count":18,"results":[[12041,1431.0,370,12042,12043],[12044,805.0,370,12042,12043],[3196,702.0,370,12042,12043],[12045,498.0,370,12042,12043],[12046,259.0,370,12042,12043],[12047,258.0,370,12042,12043],[3415,190.0,370,12042,12043],[3412,130.0,370,12042,12043],[9800,416.0,370,12042,12043],[12048,301.0,370,12042,12043],[2977,265.0,370,12042,12043],[3489,226.0,370,12042,12043],[12049,190.0,370,12042,12043],[3202,165.0,370,12042,12043],[12050,115.0,370,12042,12043],[3490,70.0,370,12042,12043],[3189,47.0,370,12042,12043],[6412,1.1,370,12042,12043]]},{"id":"ligbench-pairwise-rating-2026-paper","name":"LigBench · research idea judgments (2026 paper)","group":"数学与科学","metric":"Human-aligned pairwise rating accuracy","unit":"%","public_history":true,"protocol":{"en":"All ten published checkpoints judge the same idea-pair rating comparisons against human expert preferences. The primary rating accuracy is preserved on its original 0–1 scale, separately from contribution/soundness accuracy and downstream Elo/novelty indices. Trained Qwen variants remain distinct configurations but do not increase the eight untrained-checkpoint base-model count.","zh":"完整十检查点面板对同一研究构想配对作总体评级，参照人类专家偏好。保留0–1原始评级准确率，与贡献、可靠性准确率及后续Elo、创新性指数分开。训练版Qwen保留为不同配置，但不增加八个未微调检查点的底模计数。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.13136v1","status":"paper_baselines","published_at":"2026-08-13"},"model_count":10,"result_count":10,"results":[[797,0.514,370,12051,12052],[4466,0.506,370,12051,12052],[353,0.549,370,12051,12052],[345,0.539,370,12051,12052],[2058,0.82,370,12051,12052],[340,0.801,370,12051,12052],[7801,0.706,370,12051,12052],[12053,0.806,370,12051,12052],[12054,0.714,370,12051,12052],[12055,0.755,370,12051,12052]]},{"id":"lingt2i-content-2026-paper","name":"LingT2I · multilingual image generation (2026 paper)","group":"生成与创作","metric":"Ten-language mean TRIGScore","unit":"score","public_history":true,"protocol":{"en":"The complete 13-model panel uses the same content generation prompts across ten languages. The published mean TRIGScore is a native 0–1 evaluator score, separately from the reported cross-language standard deviation. Multilingual adapters retain their base-family labels and group-mean reference rows are excluded.","zh":"完整13模型面板使用同一十语言内容生成提示。保留0–1原生TRIGScore评委分数，与跨语言标准差分开。多语言适配器保留底模族标签，排除组均值参考行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.11002v1","status":"paper_baselines","published_at":"2026-08-11"},"score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":13,"result_count":13,"results":[[12056,0.48,370,12057,12058],[12059,0.48,370,12057,12058],[12060,0.48,370,12057,12058],[12061,0.62,370,12057,12058],[12062,0.48,370,12057,12058],[8018,0.54,370,12057,12058],[12063,0.59,370,12057,12058],[8015,0.58,370,12057,12058],[6379,0.78,370,12057,12058],[12064,0.52,370,12057,12058],[12065,0.63,370,12057,12058],[12066,0.64,370,12057,12058],[12067,0.69,370,12057,12058]]},{"id":"mbabench-market-size-2026-paper","name":"MBA-Bench · business idea market size (2026 paper)","group":"专业工作","metric":"Market-size rubric score","unit":"score","public_history":true,"protocol":{"en":"The complete sixteen distinct model configurations generate business ideas from the same 100 held-out test images. The Market Size dimension uses its published 0–3 rubric; other dimensions use different scales and are separate. The duplicated LLaVA-OneVision source row is represented once. Source MBA SFT/RL configurations are retained without counting as new base checkpoints.","zh":"完整十六个不同模型配置从同一100张留出测试图像生成商业构想。市场规模维度保留原始0–3评分细则，其他维度量纲不同，单独处理。原表重复的LLaVA-OneVision行仅计一次。保留MBA的监督微调、强化学习配置，不将其算作新增底模。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.11616v2","status":"paper_baselines","published_at":"2026-08-12"},"score_max":3,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 3","zh":"分数 / 3"},"model_count":16,"result_count":16,"results":[[12068,2.05,370,12069,12070],[12071,2.07,370,12069,12070],[12072,2.1,370,12069,12070],[6978,2.16,370,12069,12070],[2129,2.29,370,12069,12070],[10869,2.35,370,12069,12070],[5136,2.34,370,12069,12070],[7972,2.15,370,12069,12070],[12073,2.07,370,12069,12070],[2774,2.12,370,12069,12070],[2763,2.2,370,12069,12070],[2471,1.94,370,12069,12070],[2453,2.08,370,12069,12070],[12074,1.99,370,12069,12070],[12075,2.2,370,12069,12070],[12076,2.75,370,12069,12070]]},{"id":"mmjailbench-asr-2026-paper","name":"MMJailBench · multimodal jailbreaks (2026 paper)","group":"安全与可靠性","metric":"Attack success rate","unit":"%","public_history":true,"protocol":{"en":"All sixteen models evaluate the complete common Cartesian product: 272 harmful intents × six prompt templates × five visual conditions × two carriers, totaling 16,320 cases per model. The published ASR is a failure rate and lower is better. Factor sensitivity and reduced lightweight evaluations remain separate.","zh":"完整十六模型面板评测同一全因子组合：272种有害意图×六种提示模板×五种视觉条件×两种载体，每模型16,320例。原始攻击成功率为失效率，越低越好。因子敏感度和缩减版评测另列。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.25490v1","status":"paper_baselines","published_at":"2026-08-26"},"higher_is_better":false,"saturation_excluded":true,"model_count":16,"result_count":16,"results":[[12077,0.7222,370,12078,12079],[12080,0.5336,370,12078,12079],[12081,0.5164,370,12078,12079],[12082,0.4064,370,12078,12079],[12083,0.49119999999999997,370,12078,12079],[12084,0.5948,370,12078,12079],[12085,0.1984,370,12078,12079],[12086,0.6685,370,12078,12079],[12087,0.7837999999999999,370,12078,12079],[12088,0.6202000000000001,370,12078,12079],[6794,0.4779,370,12078,12079],[2943,0.358,370,12078,12079],[5106,0.305,370,12078,12079],[12089,0.4261,370,12078,12079],[12090,0.08960000000000001,370,12078,12079],[4196,0.0217,370,12078,12079]]},{"id":"msibench-truefalse-2026-paper","name":"MSIBench · situational illusion recognition (2026 paper)","group":"视觉与视频","metric":"Mean true/false accuracy","unit":"%","public_history":true,"protocol":{"en":"All twenty-seven configurations answer the same true/false situational-illusion questions. The source Avg combines vanilla and adversarial question forms; open-ended answers and action planning use separate tasks and metrics. Thinking and effort configurations remain explicit without counting as new model bases. Group-average reference rows are excluded.","zh":"完整二十七配置面板回答同一组情境错觉判断题。采用原表Avg，汇总常规与对抗问题形式；开放问答及行动规划的任务和指标单列。思考和推理强度配置明确保留，不算作新增底模。排除分组均值参考行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.22232v2","status":"paper_baselines","published_at":"2026-08-23"},"model_count":25,"result_count":27,"results":[[3196,0.5133,370,12091,12092],[3184,0.5935,370,12091,12092],[6235,0.4679,370,12091,12092],[3221,0.6437999999999999,370,12091,12092],[3181,0.6726000000000001,370,12091,12092],[2129,0.807,370,12091,12092],[131,0.5802,370,12091,12092],[3077,0.5636,370,12091,12092],[12093,0.6245,370,12091,12092],[12094,0.797,370,12091,12092],[6668,0.5653,370,12091,12092],[12095,0.6422,370,12091,12092],[6150,0.5692,370,12091,12092],[5828,0.6378,370,12091,12092],[4969,0.5022,370,12091,12092],[4070,0.5194,370,12091,12092],[2584,0.5498,370,12091,12092],[12096,0.42479999999999996,370,12091,12092],[6144,0.5249,370,12091,12092],[6145,0.5199,370,12091,12092],[6147,0.6051,370,12091,12092],[6241,0.46630000000000005,370,12091,12092],[2998,0.5896,370,12091,12092],[6632,0.47509999999999997,370,12091,12092],[12097,0.5266,370,12091,12092],[2698,0.547,370,12091,12092],[2626,0.5907,370,12091,12092]]},{"id":"longvlbench-judge-2026-paper","name":"LongVLBench · narrative comprehension (2026 paper)","group":"指令与长文","metric":"Published average judge score","unit":"score","public_history":true,"protocol":{"en":"All ten configurations evaluate the same LongVLBench narrative test across seven context ranges, with a fixed Gemini semantic-equivalence judge. The original Avg score is retained on the source 0–100 scale; per-range columns also report inference-success rates, and incomplete model executions are visible in the fixture. GlimpsePrune, VisionZip and VLZip use the same Qwen2.5-VL-3B backbone and do not increase the base count.","zh":"完整十配置评测LongVLBench同一叙事测试集的七个上下文长度范围，由固定Gemini按语义等价评分。保留原始Avg的0–100分数；分长度列同时报告推理成功率，不完整运行情况保存在原表。GlimpsePrune、VisionZip和VLZip采用相同Qwen2.5-VL-3B底模，不增加底模计数。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.08630v1","status":"paper_baselines","published_at":"2026-08-09"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":10,"result_count":10,"results":[[12098,33.1,370,12099,12100],[12101,45.0,370,12102,12100],[12103,6.3,370,12104,12100],[2824,47.1,370,12105,12100],[12106,39.1,370,12107,12100],[12108,40.8,370,12107,12100],[12109,39.9,370,12110,12100],[12111,24.7,370,12110,12100],[2844,41.4,370,12110,12100],[12112,61.9,370,12110,12100]]},{"id":"madbench-binary-2026-paper","name":"MADBench · deepfake detection (2026 paper)","group":"安全与可靠性","metric":"Mean ROC-AUC","unit":"score","public_history":true,"protocol":{"en":"The full ten-model panel uses the same MADBench balanced, scene-matched and scene-mismatched protocols. The first component of the source mean pair is ROC-AUC, a native 0–1 ranking statistic, separately from EER. Direct-transfer detectors, frozen encoders with trained task heads and zero-shot omni models retain their published adaptation modes.","zh":"完整十模型面板使用同一MADBench平衡、场景匹配及场景不匹配协议。所选均值对的第一项为0–1原生ROC-AUC排序统计量，与等错误率分开。直接迁移检测器、带训练任务头的冻结编码器和零样本全模态模型保留原始适配方式。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.09593v1","status":"paper_baselines","published_at":"2026-08-10"},"score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":10,"result_count":10,"results":[[12113,0.524,370,12114,12115],[12116,0.518,370,12114,12115],[12117,0.534,370,12114,12115],[12118,0.914,370,12119,12115],[12120,0.942,370,12119,12115],[12121,0.954,370,12119,12115],[6574,0.618,370,12122,12115],[6138,0.535,370,12122,12115],[6140,0.558,370,12122,12115],[12123,0.494,370,12122,12115]]},{"id":"madbench-four-way-2026-paper","name":"MADBench · manipulation classification (2026 paper)","group":"安全与可靠性","metric":"Mean four-way accuracy","unit":"%","public_history":true,"protocol":{"en":"The full ten-model panel uses the same MADBench balanced, scene-matched and scene-mismatched protocols. The first component of the source mean pair is four-way classification accuracy, separately from macro-F1. Direct-transfer detectors, frozen encoders with trained task heads and zero-shot omni models retain their published adaptation modes.","zh":"完整十模型面板使用同一MADBench平衡、场景匹配及场景不匹配协议。所选均值对的第一项为四类分类准确率，与宏F1分开。直接迁移检测器、带训练任务头的冻结编码器和零样本全模态模型保留原始适配方式。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.09593v1","status":"paper_baselines","published_at":"2026-08-10"},"model_count":10,"result_count":10,"results":[[12113,0.269,370,12119,12115],[12116,0.263,370,12119,12115],[12117,0.292,370,12119,12115],[12118,0.594,370,12119,12115],[12120,0.737,370,12119,12115],[12121,0.696,370,12119,12115],[6574,0.252,370,12122,12115],[6138,0.253,370,12122,12115],[6140,0.25,370,12122,12115],[12123,0.251,370,12122,12115]]},{"id":"mnist-pro-multiturn-2026-paper","name":"MNIST-PRO · active digit perception (2026 paper)","group":"视觉与视频","metric":"Native multi-turn recognition accuracy","unit":"%","public_history":true,"protocol":{"en":"All ten models solve the same two-level MNIST-PRO tasks using native multi-turn conversation history. The selected Avg is the source mean of Level 1 and Level 2 accuracy. Full-view control recognition, average steps and alternative harnesses remain separate.","zh":"完整十模型面板使用原生多轮对话历史解答相同两级MNIST-PRO任务。采用原表Avg，即第一、二级准确率均值。全图对照识别、平均步数及其他交互框架单列。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.31022v1","status":"paper_baselines","published_at":"2026-08-31"},"model_count":10,"result_count":10,"results":[[6237,0.355,370,12124,12125],[5136,0.46,370,12124,12125],[5906,0.61,370,12124,12125],[6648,0.115,370,12124,12125],[12126,0.33,370,12124,12125],[12127,0.33,370,12124,12125],[5727,0.15,370,12124,12125],[4979,0.185,370,12124,12125],[5137,0.145,370,12124,12125],[5722,0.065,370,12124,12125]]},{"id":"pathoargus-overall-2026-paper","name":"PathoArgus-Bench · whole-slide pathology (2026 paper)","group":"医疗","metric":"Overall four-choice accuracy","unit":"%","public_history":true,"protocol":{"en":"All twenty-two published configurations use the same patient-held-out PathoArgus benchmark questions and fixed reader-budget protocol. The Overall column is four-choice answer accuracy; ESG consistency and exact quartet completion remain separate measures of evidence grounding. Model scale is retained from the adjacent original cell. SFT and PathoArgus are configurations of the same proposed base, not additional base models.","zh":"完整二十二配置使用PathoArgus相同患者留出测试问题及固定阅片预算协议。Overall为四选一答案准确率，ESG一致性与四元组完全正确率单独衡量证据依据。相邻原单元格的模型规模明确保留。SFT和PathoArgus属于同一提案底模的配置，不增加底模数。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.17607v1","status":"paper_baselines","published_at":"2026-08-18"},"model_count":22,"result_count":22,"results":[[4971,0.5709000000000001,370,12128,12129],[2832,0.2712,370,12130,12129],[7513,0.2647,370,12130,12129],[12131,0.2642,370,12132,12129],[12133,0.2469,370,12134,12129],[12135,0.24670000000000003,370,12130,12129],[12136,0.2483,370,12137,12129],[12138,0.29600000000000004,370,12139,12129],[12140,0.25670000000000004,370,12137,12129],[12141,0.2754,370,12142,12129],[12143,0.3576,370,12137,12129],[12144,0.3674,370,12145,12129],[12146,0.2614,370,12147,12129],[12148,0.2901,370,12149,12129],[12150,0.252,370,12137,12129],[12151,0.28850000000000003,370,12137,12129],[12152,0.2591,370,12153,12129],[12154,0.3046,370,12153,12129],[12155,0.2591,370,12128,12129],[12156,0.2546,370,12128,12129],[12157,0.3236,370,12137,12129],[12158,0.5039,370,12137,12129]]},{"id":"catchbench-localization-2026-paper","name":"CatchBench · fault localization (2026 paper)","group":"编程","metric":"Top-1 fault-step localization accuracy","unit":"%","public_history":true,"protocol":{"en":"The full eleven-LLM published all-at-once judge panel localizes the human-labeled mistake step in the same 126 failed Who&When runs. The first-ranked accuracy is shown on the original 0–1 scale; top-three accuracy and MRR remain separate. Structural heuristics, position and random references are excluded from the model cohort. The source labels one API model only as Gemini, which is retained without inventing a version.","zh":"完整十一个已发表大模型评委按一次读取全轨迹协议，在同一126次Who&When失败运行中定位人类标注的错误步骤。首位定位准确率保留0–1原始单位，前三位准确率及MRR另列。结构启发式、位置和随机参考不进入模型面板。原文仅称Gemini的一行保留该名称，不补造版本。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.22808v4","status":"paper_baselines","published_at":"2026-08-24"},"model_count":11,"result_count":11,"results":[[3196,0.452,370,12159,12160],[3194,0.421,370,12159,12160],[3184,0.413,370,12159,12160],[769,0.405,370,12159,12160],[3487,0.357,370,12159,12160],[786,0.349,370,12159,12160],[8278,0.333,370,12159,12160],[5648,0.333,370,12159,12160],[6668,0.206,370,12159,12160],[7586,0.135,370,12159,12160],[12161,0.127,370,12159,12160]]},{"id":"catchbench-localization-addendum-2026-paper","name":"CatchBench · localization judge addendum (2026 paper)","group":"编程","metric":"Top-1 fault-step localization accuracy","unit":"%","public_history":true,"protocol":{"en":"The complete twelve-row judge addendum evaluates the same 126 Who&When traces with the same all-at-once localization protocol. Source channel and published/addendum provenance are retained per row. CLI/gateway routes of the same model remain configurations; at least six distinct model checkpoints remain without counting route repeats. These channel-specific results are kept separate from the original eleven-model panel.","zh":"完整十二行评委补充面板使用同一126条Who&When轨迹及一次读取全轨迹定位协议。每行保留原始通道和首版、补充版出处。相同模型的CLI、网关路由属于配置，不靠路由重复也有至少六检查点。这个按通道区分的面板与原始十一模型面板单列。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.22808v4","status":"paper_baselines","published_at":"2026-08-24"},"model_count":9,"result_count":12,"results":[[5142,0.476,370,12162,12163],[4943,0.476,370,12164,12163],[3208,0.452,370,12165,12163],[5142,0.452,370,12164,12163],[4943,0.452,370,12166,12163],[3208,0.429,370,12164,12163],[4972,0.429,370,12164,12163],[5108,0.421,370,12164,12163],[5368,0.405,370,12164,12163],[10713,0.333,370,12167,12163],[12168,0.325,370,12169,12163],[12170,0.294,370,12169,12163]]},{"id":"posterbench-main-2026-paper","name":"PosterBench · scientific poster generation (2026 paper)","group":"生成与创作","metric":"PosterBench Main Track score","unit":"score","public_history":true,"protocol":{"en":"The complete fourteen-system Main Track cohort produces posters from the same 100 papers across five disciplines. The native PosterBench score combines source fidelity, coverage, information density, visual evidence, layout, readability and aesthetics. Explicit model, coding agent and design harness are retained per result. Six distinct base models remain after collapsing repeated Claude/GPT workflows. The separate ten-paper mini track is not pooled.","zh":"完整十四系统主赛道面板从五个学科相同100篇论文制作海报。原生PosterBench总分综合来源忠实度、覆盖、信息密度、视觉证据、布局、可读性和美感。每行明确模型、编程智能体及设计框架，合并重复Claude、GPT流程后仍有六底模。十论文迷你赛道不混入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.13560v1","status":"paper_baselines","published_at":"2026-08-13"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":6,"result_count":14,"results":[[12171,78.32,370,12172,12173],[3581,77.97,370,12174,12173],[12171,70.87,370,12175,12173],[12171,69.45,370,12176,12173],[3581,62.17,370,12177,12173],[3581,73.37,370,12178,12173],[12171,70.01,370,12179,12173],[12180,61.14,370,12181,12173],[3654,52.22,370,12182,12173],[3440,51.46,370,12183,12173],[12184,46.01,370,12185,12173],[12171,56.71,370,12186,12173],[12171,49.09,370,12187,12173],[12171,44.61,370,12188,12173]]},{"id":"ratio-specify-recall10-2026-paper","name":"RATIO · specify inspiration retrieval (2026 paper)","group":"知识与推理","metric":"Recall@10","unit":"%","public_history":true,"protocol":{"en":"The full sixteen-configuration panel ranks every silver-test query against the same complete candidate pool for the specify discourse move. Published Recall@10 is retained, separately from reciprocal rank and from the human top-ten usefulness study. Tuned/reranked variants retain their original base and setup; BM25 is a lexical retrieval baseline, not an additional learned checkpoint. At least six distinct embedding bases remain without counting repeated tuning configurations.","zh":"完整十六配置面板对specify语篇动作的每条银标测试查询，在同一完整候选池排序。保留原始Recall@10，与倒数排名及人类前十可用性研究分开。微调、重排变体保留原底模与设置；BM25为词法检索基线，不计作额外学习检查点。不靠重复调参配置也有至少六种嵌入底模。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.27394v4","status":"paper_baselines","published_at":"2026-08-27"},"model_count":10,"result_count":16,"results":[[12189,0.273,370,12190,12191],[8065,0.266,370,12192,12191],[8065,0.5429999999999999,370,12193,12191],[8065,0.66,370,12194,12191],[12195,0.391,370,12192,12191],[12195,0.655,370,12193,12191],[12195,0.6990000000000001,370,12194,12191],[12196,0.39799999999999996,370,12192,12191],[12196,0.6,370,12193,12191],[12196,0.68,370,12194,12191],[12197,0.423,370,12192,12191],[9738,0.37,370,12192,12191],[12198,0.44299999999999995,370,12192,12191],[12199,0.307,370,12192,12191],[12200,0.34299999999999997,370,12192,12191],[12201,0.34,370,12192,12191]]},{"id":"ratio-address-recall10-2026-paper","name":"RATIO · address inspiration retrieval (2026 paper)","group":"知识与推理","metric":"Recall@10","unit":"%","public_history":true,"protocol":{"en":"The full sixteen-configuration panel ranks every silver-test query against the same complete candidate pool for the address discourse move. Published Recall@10 is retained, separately from reciprocal rank and from the human top-ten usefulness study. Tuned/reranked variants retain their original base and setup; BM25 is a lexical retrieval baseline, not an additional learned checkpoint. At least six distinct embedding bases remain without counting repeated tuning configurations.","zh":"完整十六配置面板对address语篇动作的每条银标测试查询，在同一完整候选池排序。保留原始Recall@10，与倒数排名及人类前十可用性研究分开。微调、重排变体保留原底模与设置；BM25为词法检索基线，不计作额外学习检查点。不靠重复调参配置也有至少六种嵌入底模。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.27394v4","status":"paper_baselines","published_at":"2026-08-27"},"model_count":10,"result_count":16,"results":[[12189,0.152,370,12190,12191],[8065,0.165,370,12192,12191],[8065,0.317,370,12193,12191],[8065,0.45899999999999996,370,12194,12191],[12195,0.17800000000000002,370,12192,12191],[12195,0.4,370,12193,12191],[12195,0.521,370,12194,12191],[12196,0.201,370,12192,12191],[12196,0.344,370,12193,12191],[12196,0.47100000000000003,370,12194,12191],[12197,0.146,370,12192,12191],[9738,0.18899999999999997,370,12192,12191],[12198,0.217,370,12192,12191],[12199,0.11,370,12192,12191],[12200,0.149,370,12192,12191],[12201,0.16699999999999998,370,12192,12191]]},{"id":"ratio-broaden-recall10-2026-paper","name":"RATIO · broaden inspiration retrieval (2026 paper)","group":"知识与推理","metric":"Recall@10","unit":"%","public_history":true,"protocol":{"en":"The full sixteen-configuration panel ranks every silver-test query against the same complete candidate pool for the broaden discourse move. Published Recall@10 is retained, separately from reciprocal rank and from the human top-ten usefulness study. Tuned/reranked variants retain their original base and setup; BM25 is a lexical retrieval baseline, not an additional learned checkpoint. At least six distinct embedding bases remain without counting repeated tuning configurations.","zh":"完整十六配置面板对broaden语篇动作的每条银标测试查询，在同一完整候选池排序。保留原始Recall@10，与倒数排名及人类前十可用性研究分开。微调、重排变体保留原底模与设置；BM25为词法检索基线，不计作额外学习检查点。不靠重复调参配置也有至少六种嵌入底模。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.27394v4","status":"paper_baselines","published_at":"2026-08-27"},"model_count":10,"result_count":16,"results":[[12189,0.17600000000000002,370,12190,12191],[8065,0.243,370,12192,12191],[8065,0.325,370,12193,12191],[8065,0.377,370,12194,12191],[12195,0.26,370,12192,12191],[12195,0.414,370,12193,12191],[12195,0.409,370,12194,12191],[12196,0.29600000000000004,370,12192,12191],[12196,0.318,370,12193,12191],[12196,0.409,370,12194,12191],[12197,0.289,370,12192,12191],[9738,0.271,370,12192,12191],[12198,0.308,370,12192,12191],[12199,0.158,370,12192,12191],[12200,0.209,370,12192,12191],[12201,0.221,370,12192,12191]]},{"id":"r3bench-tool-free-rho02-math-2026-paper","name":"R³-Bench · math (tool-free, ρ=0.2) (2026 paper)","group":"知识与推理","metric":"Mean solved problems per six-problem contest","unit":"score","public_history":true,"protocol":{"en":"All six distinct models solve the same fifty six-problem math contests under the published tool-free shared-budget regime at pressure ρ=0.2. The Contest mean retains its native solved-problem count out of six. Budgets are calibrated using each model’s matched single-problem response curves. The offline empirical Oracle and Gap Ratio are diagnostics, not model results.","zh":"完整六底模面板按tool-free共享预算协议、压力ρ=0.2解答相同五十场math竞赛，每场六题。Contest均值保留每场解题数量，满分六题。预算依据各模型匹配的单题响应曲线校准。离线经验Oracle及Gap Ratio是诊断量，不作为模型结果。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.16033v1","status":"paper_baselines","published_at":"2026-08-17"},"score_max":6,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 6","zh":"分数 / 6"},"model_count":6,"result_count":6,"results":[[3185,2.34,370,12202,12203],[3202,3.54,370,12202,12203],[227,2.32,370,12202,12203],[12204,2.4,370,12202,12203],[3196,0.72,370,12202,12203],[3194,3.78,370,12202,12203]]},{"id":"r3bench-tool-free-rho02-code-2026-paper","name":"R³-Bench · code (tool-free, ρ=0.2) (2026 paper)","group":"知识与推理","metric":"Mean solved problems per six-problem contest","unit":"score","public_history":true,"protocol":{"en":"All six distinct models solve the same fifty six-problem code contests under the published tool-free shared-budget regime at pressure ρ=0.2. The Contest mean retains its native solved-problem count out of six. Budgets are calibrated using each model’s matched single-problem response curves. The offline empirical Oracle and Gap Ratio are diagnostics, not model results.","zh":"完整六底模面板按tool-free共享预算协议、压力ρ=0.2解答相同五十场code竞赛，每场六题。Contest均值保留每场解题数量，满分六题。预算依据各模型匹配的单题响应曲线校准。离线经验Oracle及Gap Ratio是诊断量，不作为模型结果。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.16033v1","status":"paper_baselines","published_at":"2026-08-17"},"score_max":6,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 6","zh":"分数 / 6"},"model_count":6,"result_count":6,"results":[[3185,1.38,370,12205,12203],[3202,1.42,370,12205,12203],[227,0.68,370,12205,12203],[12204,1.2,370,12205,12203],[3196,0.68,370,12205,12203],[3194,0.6,370,12205,12203]]},{"id":"r3bench-tool-free-rho02-abstract-reasoning-2026-paper","name":"R³-Bench · abstract-reasoning (tool-free, ρ=0.2) (2026 paper)","group":"知识与推理","metric":"Mean solved problems per six-problem contest","unit":"score","public_history":true,"protocol":{"en":"All six distinct models solve the same fifty six-problem abstract-reasoning contests under the published tool-free shared-budget regime at pressure ρ=0.2. The Contest mean retains its native solved-problem count out of six. Budgets are calibrated using each model’s matched single-problem response curves. The offline empirical Oracle and Gap Ratio are diagnostics, not model results.","zh":"完整六底模面板按tool-free共享预算协议、压力ρ=0.2解答相同五十场abstract-reasoning竞赛，每场六题。Contest均值保留每场解题数量，满分六题。预算依据各模型匹配的单题响应曲线校准。离线经验Oracle及Gap Ratio是诊断量，不作为模型结果。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.16033v1","status":"paper_baselines","published_at":"2026-08-17"},"score_max":6,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 6","zh":"分数 / 6"},"model_count":6,"result_count":6,"results":[[3185,2.24,370,12206,12203],[3202,2.76,370,12206,12203],[227,1.14,370,12206,12203],[12204,1.56,370,12206,12203],[3196,0.34,370,12206,12203],[3194,1.84,370,12206,12203]]},{"id":"r3bench-tool-free-rho08-math-2026-paper","name":"R³-Bench · math (tool-free, ρ=0.8) (2026 paper)","group":"知识与推理","metric":"Mean solved problems per six-problem contest","unit":"score","public_history":true,"protocol":{"en":"All six distinct models solve the same fifty six-problem math contests under the published tool-free shared-budget regime at pressure ρ=0.8. The Contest mean retains its native solved-problem count out of six. Budgets are calibrated using each model’s matched single-problem response curves. The offline empirical Oracle and Gap Ratio are diagnostics, not model results.","zh":"完整六底模面板按tool-free共享预算协议、压力ρ=0.8解答相同五十场math竞赛，每场六题。Contest均值保留每场解题数量，满分六题。预算依据各模型匹配的单题响应曲线校准。离线经验Oracle及Gap Ratio是诊断量，不作为模型结果。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.16033v1","status":"paper_baselines","published_at":"2026-08-17"},"score_max":6,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 6","zh":"分数 / 6"},"model_count":6,"result_count":6,"results":[[3185,3.94,370,12207,12203],[3202,4.32,370,12207,12203],[227,2.72,370,12207,12203],[12204,3.72,370,12207,12203],[3196,2.52,370,12207,12203],[3194,4.18,370,12207,12203]]},{"id":"r3bench-tool-free-rho08-code-2026-paper","name":"R³-Bench · code (tool-free, ρ=0.8) (2026 paper)","group":"知识与推理","metric":"Mean solved problems per six-problem contest","unit":"score","public_history":true,"protocol":{"en":"All six distinct models solve the same fifty six-problem code contests under the published tool-free shared-budget regime at pressure ρ=0.8. The Contest mean retains its native solved-problem count out of six. Budgets are calibrated using each model’s matched single-problem response curves. The offline empirical Oracle and Gap Ratio are diagnostics, not model results.","zh":"完整六底模面板按tool-free共享预算协议、压力ρ=0.8解答相同五十场code竞赛，每场六题。Contest均值保留每场解题数量，满分六题。预算依据各模型匹配的单题响应曲线校准。离线经验Oracle及Gap Ratio是诊断量，不作为模型结果。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.16033v1","status":"paper_baselines","published_at":"2026-08-17"},"score_max":6,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 6","zh":"分数 / 6"},"model_count":6,"result_count":6,"results":[[3185,3.06,370,12208,12203],[3202,3.44,370,12208,12203],[227,1.72,370,12208,12203],[12204,2.96,370,12208,12203],[3196,2.6,370,12208,12203],[3194,2.68,370,12208,12203]]},{"id":"r3bench-tool-free-rho08-abstract-reasoning-2026-paper","name":"R³-Bench · abstract-reasoning (tool-free, ρ=0.8) (2026 paper)","group":"知识与推理","metric":"Mean solved problems per six-problem contest","unit":"score","public_history":true,"protocol":{"en":"All six distinct models solve the same fifty six-problem abstract-reasoning contests under the published tool-free shared-budget regime at pressure ρ=0.8. The Contest mean retains its native solved-problem count out of six. Budgets are calibrated using each model’s matched single-problem response curves. The offline empirical Oracle and Gap Ratio are diagnostics, not model results.","zh":"完整六底模面板按tool-free共享预算协议、压力ρ=0.8解答相同五十场abstract-reasoning竞赛，每场六题。Contest均值保留每场解题数量，满分六题。预算依据各模型匹配的单题响应曲线校准。离线经验Oracle及Gap Ratio是诊断量，不作为模型结果。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.16033v1","status":"paper_baselines","published_at":"2026-08-17"},"score_max":6,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 6","zh":"分数 / 6"},"model_count":6,"result_count":6,"results":[[3185,2.9,370,12209,12203],[3202,3.82,370,12209,12203],[227,1.7,370,12209,12203],[12204,3.16,370,12209,12203],[3196,1.58,370,12209,12203],[3194,2.52,370,12209,12203]]},{"id":"r3bench-agentic-rho02-math-2026-paper","name":"R³-Bench · math (agentic, ρ=0.2) (2026 paper)","group":"知识与推理","metric":"Mean solved problems per six-problem contest","unit":"score","public_history":true,"protocol":{"en":"All six distinct models solve the same fifty six-problem math contests under the published agentic shared-budget regime at pressure ρ=0.2. The Contest mean retains its native solved-problem count out of six. Budgets are calibrated using each model’s matched single-problem response curves. The offline empirical Oracle and Gap Ratio are diagnostics, not model results.","zh":"完整六底模面板按agentic共享预算协议、压力ρ=0.2解答相同五十场math竞赛，每场六题。Contest均值保留每场解题数量，满分六题。预算依据各模型匹配的单题响应曲线校准。离线经验Oracle及Gap Ratio是诊断量，不作为模型结果。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.16033v1","status":"paper_baselines","published_at":"2026-08-17"},"score_max":6,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 6","zh":"分数 / 6"},"model_count":6,"result_count":6,"results":[[3185,4.48,370,12210,12203],[3202,4.64,370,12210,12203],[227,4.48,370,12210,12203],[12204,1.8,370,12210,12203],[3196,3.64,370,12210,12203],[3194,4.46,370,12210,12203]]},{"id":"r3bench-agentic-rho02-code-2026-paper","name":"R³-Bench · code (agentic, ρ=0.2) (2026 paper)","group":"知识与推理","metric":"Mean solved problems per six-problem contest","unit":"score","public_history":true,"protocol":{"en":"All six distinct models solve the same fifty six-problem code contests under the published agentic shared-budget regime at pressure ρ=0.2. The Contest mean retains its native solved-problem count out of six. Budgets are calibrated using each model’s matched single-problem response curves. The offline empirical Oracle and Gap Ratio are diagnostics, not model results.","zh":"完整六底模面板按agentic共享预算协议、压力ρ=0.2解答相同五十场code竞赛，每场六题。Contest均值保留每场解题数量，满分六题。预算依据各模型匹配的单题响应曲线校准。离线经验Oracle及Gap Ratio是诊断量，不作为模型结果。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.16033v1","status":"paper_baselines","published_at":"2026-08-17"},"score_max":6,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 6","zh":"分数 / 6"},"model_count":6,"result_count":6,"results":[[3185,2.6,370,12211,12203],[3202,2.78,370,12211,12203],[227,2.02,370,12211,12203],[12204,1.0,370,12211,12203],[3196,1.92,370,12211,12203],[3194,2.24,370,12211,12203]]},{"id":"r3bench-agentic-rho02-abstract-reasoning-2026-paper","name":"R³-Bench · abstract-reasoning (agentic, ρ=0.2) (2026 paper)","group":"知识与推理","metric":"Mean solved problems per six-problem contest","unit":"score","public_history":true,"protocol":{"en":"All six distinct models solve the same fifty six-problem abstract-reasoning contests under the published agentic shared-budget regime at pressure ρ=0.2. The Contest mean retains its native solved-problem count out of six. Budgets are calibrated using each model’s matched single-problem response curves. The offline empirical Oracle and Gap Ratio are diagnostics, not model results.","zh":"完整六底模面板按agentic共享预算协议、压力ρ=0.2解答相同五十场abstract-reasoning竞赛，每场六题。Contest均值保留每场解题数量，满分六题。预算依据各模型匹配的单题响应曲线校准。离线经验Oracle及Gap Ratio是诊断量，不作为模型结果。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.16033v1","status":"paper_baselines","published_at":"2026-08-17"},"score_max":6,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 6","zh":"分数 / 6"},"model_count":6,"result_count":6,"results":[[3185,4.36,370,12212,12203],[3202,4.62,370,12212,12203],[227,3.78,370,12212,12203],[12204,1.22,370,12212,12203],[3196,1.8,370,12212,12203],[3194,3.77,370,12212,12203]]},{"id":"r3bench-agentic-rho08-math-2026-paper","name":"R³-Bench · math (agentic, ρ=0.8) (2026 paper)","group":"知识与推理","metric":"Mean solved problems per six-problem contest","unit":"score","public_history":true,"protocol":{"en":"All six distinct models solve the same fifty six-problem math contests under the published agentic shared-budget regime at pressure ρ=0.8. The Contest mean retains its native solved-problem count out of six. Budgets are calibrated using each model’s matched single-problem response curves. The offline empirical Oracle and Gap Ratio are diagnostics, not model results.","zh":"完整六底模面板按agentic共享预算协议、压力ρ=0.8解答相同五十场math竞赛，每场六题。Contest均值保留每场解题数量，满分六题。预算依据各模型匹配的单题响应曲线校准。离线经验Oracle及Gap Ratio是诊断量，不作为模型结果。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.16033v1","status":"paper_baselines","published_at":"2026-08-17"},"score_max":6,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 6","zh":"分数 / 6"},"model_count":6,"result_count":6,"results":[[3185,4.48,370,12213,12203],[3202,4.96,370,12213,12203],[227,4.16,370,12213,12203],[12204,1.56,370,12213,12203],[3196,4.4,370,12213,12203],[3194,4.85,370,12213,12203]]},{"id":"r3bench-agentic-rho08-code-2026-paper","name":"R³-Bench · code (agentic, ρ=0.8) (2026 paper)","group":"知识与推理","metric":"Mean solved problems per six-problem contest","unit":"score","public_history":true,"protocol":{"en":"All six distinct models solve the same fifty six-problem code contests under the published agentic shared-budget regime at pressure ρ=0.8. The Contest mean retains its native solved-problem count out of six. Budgets are calibrated using each model’s matched single-problem response curves. The offline empirical Oracle and Gap Ratio are diagnostics, not model results.","zh":"完整六底模面板按agentic共享预算协议、压力ρ=0.8解答相同五十场code竞赛，每场六题。Contest均值保留每场解题数量，满分六题。预算依据各模型匹配的单题响应曲线校准。离线经验Oracle及Gap Ratio是诊断量，不作为模型结果。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.16033v1","status":"paper_baselines","published_at":"2026-08-17"},"score_max":6,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 6","zh":"分数 / 6"},"model_count":6,"result_count":6,"results":[[3185,4.1,370,12214,12203],[3202,4.72,370,12214,12203],[227,4.06,370,12214,12203],[12204,1.38,370,12214,12203],[3196,4.8,370,12214,12203],[3194,4.52,370,12214,12203]]},{"id":"r3bench-agentic-rho08-abstract-reasoning-2026-paper","name":"R³-Bench · abstract-reasoning (agentic, ρ=0.8) (2026 paper)","group":"知识与推理","metric":"Mean solved problems per six-problem contest","unit":"score","public_history":true,"protocol":{"en":"All six distinct models solve the same fifty six-problem abstract-reasoning contests under the published agentic shared-budget regime at pressure ρ=0.8. The Contest mean retains its native solved-problem count out of six. Budgets are calibrated using each model’s matched single-problem response curves. The offline empirical Oracle and Gap Ratio are diagnostics, not model results.","zh":"完整六底模面板按agentic共享预算协议、压力ρ=0.8解答相同五十场abstract-reasoning竞赛，每场六题。Contest均值保留每场解题数量，满分六题。预算依据各模型匹配的单题响应曲线校准。离线经验Oracle及Gap Ratio是诊断量，不作为模型结果。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.16033v1","status":"paper_baselines","published_at":"2026-08-17"},"score_max":6,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 6","zh":"分数 / 6"},"model_count":6,"result_count":6,"results":[[3185,4.12,370,12215,12203],[3202,4.8,370,12215,12203],[227,4.06,370,12215,12203],[12204,0.8,370,12215,12203],[3196,4.72,370,12215,12203],[3194,5.12,370,12215,12203]]},{"id":"roborecover-robotwin-initial-2026-paper","name":"RoboRecover ·RoboTwin initial (2026 paper)","group":"Agent 与工具","metric":"Initial-state task success","unit":"%","public_history":true,"protocol":{"en":"Six policies are evaluated on the same fixed200-scenario RoboTwin test split. initial starts from the predefined reset state or the reconstructed recovery point respectively; both use the original instruction and success criterion. Overall recovery averages per-scenario trial success, then averages scenarios. The800-scenario training split supports method development. Replay state/image checks and source-policy difficulty filtering precede the split; this distribution is deliberately recovery-focused. Intervention controllers and rollback starts are not mixed into this fixed-policy cohort.","zh":"六个策略在RoboTwin共同固定200个测试情境上评测。initial分别从原始重置状态或重建恢复点开始，使用原指令和任务成功判据。总体恢复率先在每个情境内平均试验结果，再对情境平均。800个训练情境只用于方法开发；划分前已作重放一致性检查和参考策略难度过滤，因此这是一组专门选取的恢复问题。未混入外部纠错控制器或回退起点。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.28952v1","published_at":"2026-09-24","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[7401,0.5433,370,12216,12217],[8326,0.6433,370,12216,12217],[9600,0.4267,370,12216,12217],[8323,0.345,370,12216,12217],[10169,0.81,370,12216,12217],[7404,0.8375,370,12216,12217]]},{"id":"roborecover-robotwin-recovery-2026-paper","name":"RoboRecover ·RoboTwin recovery (2026 paper)","group":"Agent 与工具","metric":"Overall recovery success","unit":"%","public_history":true,"protocol":{"en":"Six policies are evaluated on the same fixed200-scenario RoboTwin test split. recovery starts from the predefined reset state or the reconstructed recovery point respectively; both use the original instruction and success criterion. Overall recovery averages per-scenario trial success, then averages scenarios. The800-scenario training split supports method development. Replay state/image checks and source-policy difficulty filtering precede the split; this distribution is deliberately recovery-focused. Intervention controllers and rollback starts are not mixed into this fixed-policy cohort.","zh":"六个策略在RoboTwin共同固定200个测试情境上评测。recovery分别从原始重置状态或重建恢复点开始，使用原指令和任务成功判据。总体恢复率先在每个情境内平均试验结果，再对情境平均。800个训练情境只用于方法开发；划分前已作重放一致性检查和参考策略难度过滤，因此这是一组专门选取的恢复问题。未混入外部纠错控制器或回退起点。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.28952v1","published_at":"2026-09-24","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[7401,0.49829999999999997,370,12218,12217],[8326,0.43200000000000005,370,12218,12217],[9600,0.33030000000000004,370,12218,12217],[8323,0.326,370,12218,12217],[10169,0.49329999999999996,370,12218,12217],[7404,0.595,370,12218,12217]]},{"id":"roborecover-robotwin-macro9-2026-paper","name":"RoboRecover ·RoboTwin Macro-9 (2026 paper)","group":"Agent 与工具","metric":"Category-macro recovery success","unit":"%","public_history":true,"protocol":{"en":"The same six policies and200 test scenarios are grouped into nine nonempty Recovery Stage × Deviation Type cells. Macro-9 equally averages the nine scenario-level recovery means, not the raw rollout pool. Rare groups remain descriptive; no source-policy oracle or trained intervention is included.","zh":"相同六策略和200个测试情境划分为恢复阶段与偏差类型组合的九个非空组。Macro-9对九组情境恢复均值等权，而非对全部试验直接平均；稀有组仅作描述，未加入按测试结果挑策略的先知或训练后的干预方法。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.28952v1","published_at":"2026-09-24","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[7401,0.4065,370,12219,12220],[8326,0.413,370,12219,12220],[9600,0.33520000000000005,370,12219,12220],[8323,0.3433,370,12219,12220],[10169,0.5562,370,12219,12220],[7404,0.5318999999999999,370,12219,12220]]},{"id":"roborecover-libero-initial-2026-paper","name":"RoboRecover ·LIBERO initial (2026 paper)","group":"Agent 与工具","metric":"Initial-state task success","unit":"%","public_history":true,"protocol":{"en":"Six policies are evaluated on the same fixed200-scenario LIBERO test split. initial starts from the predefined reset state or the reconstructed recovery point respectively; both use the original instruction and success criterion. Overall recovery averages per-scenario trial success, then averages scenarios. The800-scenario training split supports method development. Replay state/image checks and source-policy difficulty filtering precede the split; this distribution is deliberately recovery-focused. Intervention controllers and rollback starts are not mixed into this fixed-policy cohort.","zh":"六个策略在LIBERO共同固定200个测试情境上评测。initial分别从原始重置状态或重建恢复点开始，使用原指令和任务成功判据。总体恢复率先在每个情境内平均试验结果，再对情境平均。800个训练情境只用于方法开发；划分前已作重放一致性检查和参考策略难度过滤，因此这是一组专门选取的恢复问题。未混入外部纠错控制器或回退起点。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.28952v1","published_at":"2026-09-24","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[10179,0.8517,370,12221,12217],[8323,0.9417,370,12221,12217],[12222,0.9167000000000001,370,12221,12217],[12223,0.9883,370,12221,12217],[12224,0.9783,370,12221,12217],[10169,0.9682999999999999,370,12221,12217]]},{"id":"roborecover-libero-recovery-2026-paper","name":"RoboRecover ·LIBERO recovery (2026 paper)","group":"Agent 与工具","metric":"Overall recovery success","unit":"%","public_history":true,"protocol":{"en":"Six policies are evaluated on the same fixed200-scenario LIBERO test split. recovery starts from the predefined reset state or the reconstructed recovery point respectively; both use the original instruction and success criterion. Overall recovery averages per-scenario trial success, then averages scenarios. The800-scenario training split supports method development. Replay state/image checks and source-policy difficulty filtering precede the split; this distribution is deliberately recovery-focused. Intervention controllers and rollback starts are not mixed into this fixed-policy cohort.","zh":"六个策略在LIBERO共同固定200个测试情境上评测。recovery分别从原始重置状态或重建恢复点开始，使用原指令和任务成功判据。总体恢复率先在每个情境内平均试验结果，再对情境平均。800个训练情境只用于方法开发；划分前已作重放一致性检查和参考策略难度过滤，因此这是一组专门选取的恢复问题。未混入外部纠错控制器或回退起点。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.28952v1","published_at":"2026-09-24","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[10179,0.37799999999999995,370,12225,12217],[8323,0.644,370,12225,12217],[12222,0.46799999999999997,370,12225,12217],[12223,0.48,370,12225,12217],[12224,0.521,370,12225,12217],[10169,0.54,370,12225,12217]]},{"id":"roborecover-libero-macro9-2026-paper","name":"RoboRecover ·LIBERO Macro-9 (2026 paper)","group":"Agent 与工具","metric":"Category-macro recovery success","unit":"%","public_history":true,"protocol":{"en":"The same six policies and200 test scenarios are grouped into nine nonempty Recovery Stage × Deviation Type cells. Macro-9 equally averages the nine scenario-level recovery means, not the raw rollout pool. Rare groups remain descriptive; no source-policy oracle or trained intervention is included.","zh":"相同六策略和200个测试情境划分为恢复阶段与偏差类型组合的九个非空组。Macro-9对九组情境恢复均值等权，而非对全部试验直接平均；稀有组仅作描述，未加入按测试结果挑策略的先知或训练后的干预方法。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.28952v1","published_at":"2026-09-24","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[10179,0.3679,370,12219,12220],[8323,0.6770999999999999,370,12219,12220],[12222,0.4868,370,12219,12220],[12223,0.5025999999999999,370,12219,12220],[12224,0.5056,370,12219,12220],[10169,0.5401,370,12219,12220]]},{"id":"synthetic-hospital-diagnosis-2026-paper","name":"Synthetic Hospital ·diagnosis (2026 paper)","group":"医疗","metric":"Severity-weighted diagnosis F1","unit":"score","public_history":true,"protocol":{"en":"Ten models share the public patient-level split of200 synthetic longitudinal EHRs, with 200 instances for this task variant. Patients and source education cases do not cross the800patient training or268patient held-out boundaries. The graph-grounded reference defines a deterministic native0–1 metric. The task-specific strategy is locked after a three-pilot-model comparison and shared across all ten models. Diagnoses documented in the chart but absent from the reference are neutral rather than false positives. Physician13-patient results and three-model agent-loop results use different populations and are excluded.","zh":"十个模型共享200名合成纵向病历患者的公开划分，本任务含200个实例。患者及原始医学教学材料不跨越800名训练患者和268名留出患者的边界。图谱参考定义确定性的原生0–1指标。每任务提示策略经三个先导模型比较后固定，对十模型一致；病历中有记录但未进入参考的诊断作中性处理。排除只用13名患者的医生参照和三模型Agent循环实验。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.30027v1","published_at":"2026-09-24","status":"paper_baselines"},"score_min":0,"score_max":1,"saturation_excluded":true,"scale_label":{"en":"Native score / 1","zh":"原生分数 / 1"},"score_decimals":4,"model_count":10,"result_count":10,"results":[[12226,0.681,370,12227,12228],[12229,0.703,370,12227,12228],[12230,0.732,370,12227,12228],[3406,0.615,370,12227,12228],[12231,0.66,370,12227,12228],[12232,0.657,370,12227,12228],[12233,0.653,370,12227,12228],[12234,0.676,370,12227,12228],[12235,0.536,370,12227,12228],[12236,0.287,370,12227,12228]]},{"id":"synthetic-hospital-summarization-2026-paper","name":"Synthetic Hospital ·summarization (2026 paper)","group":"医疗","metric":"Finding-level summary F1","unit":"score","public_history":true,"protocol":{"en":"Ten models share the public patient-level split of200 synthetic longitudinal EHRs, with 200 instances for this task variant. Patients and source education cases do not cross the800patient training or268patient held-out boundaries. The graph-grounded reference defines a deterministic native0–1 metric. The task-specific strategy is locked after a three-pilot-model comparison and shared across all ten models. Diagnoses documented in the chart but absent from the reference are neutral rather than false positives. Physician13-patient results and three-model agent-loop results use different populations and are excluded.","zh":"十个模型共享200名合成纵向病历患者的公开划分，本任务含200个实例。患者及原始医学教学材料不跨越800名训练患者和268名留出患者的边界。图谱参考定义确定性的原生0–1指标。每任务提示策略经三个先导模型比较后固定，对十模型一致；病历中有记录但未进入参考的诊断作中性处理。排除只用13名患者的医生参照和三模型Agent循环实验。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.30027v1","published_at":"2026-09-24","status":"paper_baselines"},"score_min":0,"score_max":1,"saturation_excluded":true,"scale_label":{"en":"Native score / 1","zh":"原生分数 / 1"},"score_decimals":4,"model_count":10,"result_count":10,"results":[[12226,0.502,370,12227,12228],[12229,0.489,370,12227,12228],[12230,0.532,370,12227,12228],[3406,0.55,370,12227,12228],[12231,0.399,370,12227,12228],[12232,0.484,370,12227,12228],[12233,0.425,370,12227,12228],[12234,0.416,370,12227,12228],[12235,0.403,370,12227,12228],[12236,0.385,370,12227,12228]]},{"id":"synthetic-hospital-specialty-summary-2026-paper","name":"Synthetic Hospital ·specialty-summary (2026 paper)","group":"医疗","metric":"Specialty-relevance F1","unit":"score","public_history":true,"protocol":{"en":"Ten models share the public patient-level split of200 synthetic longitudinal EHRs, with 983 instances for this task variant. Patients and source education cases do not cross the800patient training or268patient held-out boundaries. The graph-grounded reference defines a deterministic native0–1 metric. The task-specific strategy is locked after a three-pilot-model comparison and shared across all ten models. Diagnoses documented in the chart but absent from the reference are neutral rather than false positives. Physician13-patient results and three-model agent-loop results use different populations and are excluded.","zh":"十个模型共享200名合成纵向病历患者的公开划分，本任务含983个实例。患者及原始医学教学材料不跨越800名训练患者和268名留出患者的边界。图谱参考定义确定性的原生0–1指标。每任务提示策略经三个先导模型比较后固定，对十模型一致；病历中有记录但未进入参考的诊断作中性处理。排除只用13名患者的医生参照和三模型Agent循环实验。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.30027v1","published_at":"2026-09-24","status":"paper_baselines"},"score_min":0,"score_max":1,"saturation_excluded":true,"scale_label":{"en":"Native score / 1","zh":"原生分数 / 1"},"score_decimals":4,"model_count":10,"result_count":10,"results":[[12226,0.582,370,12237,12228],[12229,0.595,370,12237,12228],[12230,0.59,370,12237,12228],[3406,0.68,370,12237,12228],[12231,0.523,370,12237,12228],[12232,0.614,370,12237,12228],[12233,0.548,370,12237,12228],[12234,0.636,370,12237,12228],[12235,0.365,370,12237,12228],[12236,0.418,370,12237,12228]]},{"id":"synthetic-hospital-retrieval-p5-2026-paper","name":"Synthetic Hospital ·retrieval-p5 (2026 paper)","group":"医疗","metric":"Evidence precision@5","unit":"score","public_history":true,"protocol":{"en":"Ten models share the public patient-level split of200 synthetic longitudinal EHRs, with 200 instances for this task variant. Patients and source education cases do not cross the800patient training or268patient held-out boundaries. The graph-grounded reference defines a deterministic native0–1 metric. The task-specific strategy is locked after a three-pilot-model comparison and shared across all ten models. Diagnoses documented in the chart but absent from the reference are neutral rather than false positives. Physician13-patient results and three-model agent-loop results use different populations and are excluded.","zh":"十个模型共享200名合成纵向病历患者的公开划分，本任务含200个实例。患者及原始医学教学材料不跨越800名训练患者和268名留出患者的边界。图谱参考定义确定性的原生0–1指标。每任务提示策略经三个先导模型比较后固定，对十模型一致；病历中有记录但未进入参考的诊断作中性处理。排除只用13名患者的医生参照和三模型Agent循环实验。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.30027v1","published_at":"2026-09-24","status":"paper_baselines"},"score_min":0,"score_max":1,"saturation_excluded":true,"scale_label":{"en":"Native score / 1","zh":"原生分数 / 1"},"score_decimals":4,"model_count":10,"result_count":10,"results":[[12226,0.809,370,12227,12228],[12229,0.833,370,12227,12228],[12230,0.811,370,12227,12228],[3406,0.816,370,12227,12228],[12231,0.751,370,12227,12228],[12232,0.798,370,12227,12228],[12233,0.809,370,12227,12228],[12234,0.818,370,12227,12228],[12235,0.741,370,12227,12228],[12236,0.804,370,12227,12228]]},{"id":"synthetic-hospital-retrieval-ndcg10-2026-paper","name":"Synthetic Hospital ·retrieval-ndcg10 (2026 paper)","group":"医疗","metric":"Evidence NDCG@10","unit":"score","public_history":true,"protocol":{"en":"Ten models share the public patient-level split of200 synthetic longitudinal EHRs, with 200 instances for this task variant. Patients and source education cases do not cross the800patient training or268patient held-out boundaries. The graph-grounded reference defines a deterministic native0–1 metric. The task-specific strategy is locked after a three-pilot-model comparison and shared across all ten models. Diagnoses documented in the chart but absent from the reference are neutral rather than false positives. Physician13-patient results and three-model agent-loop results use different populations and are excluded.","zh":"十个模型共享200名合成纵向病历患者的公开划分，本任务含200个实例。患者及原始医学教学材料不跨越800名训练患者和268名留出患者的边界。图谱参考定义确定性的原生0–1指标。每任务提示策略经三个先导模型比较后固定，对十模型一致；病历中有记录但未进入参考的诊断作中性处理。排除只用13名患者的医生参照和三模型Agent循环实验。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.30027v1","published_at":"2026-09-24","status":"paper_baselines"},"score_min":0,"score_max":1,"saturation_excluded":true,"scale_label":{"en":"Native score / 1","zh":"原生分数 / 1"},"score_decimals":4,"model_count":10,"result_count":10,"results":[[12226,0.497,370,12227,12228],[12229,0.536,370,12227,12228],[12230,0.524,370,12227,12228],[3406,0.521,370,12227,12228],[12231,0.476,370,12227,12228],[12232,0.508,370,12227,12228],[12233,0.508,370,12227,12228],[12234,0.524,370,12227,12228],[12235,0.481,370,12227,12228],[12236,0.517,370,12227,12228]]},{"id":"synthetic-hospital-imaging-indication-2026-paper","name":"Synthetic Hospital ·imaging-indication (2026 paper)","group":"医疗","metric":"Imaging question concept F1","unit":"score","public_history":true,"protocol":{"en":"Ten models share the public patient-level split of200 synthetic longitudinal EHRs, with 276 instances for this task variant. Patients and source education cases do not cross the800patient training or268patient held-out boundaries. The graph-grounded reference defines a deterministic native0–1 metric. The task-specific strategy is locked after a three-pilot-model comparison and shared across all ten models. Diagnoses documented in the chart but absent from the reference are neutral rather than false positives. Physician13-patient results and three-model agent-loop results use different populations and are excluded.","zh":"十个模型共享200名合成纵向病历患者的公开划分，本任务含276个实例。患者及原始医学教学材料不跨越800名训练患者和268名留出患者的边界。图谱参考定义确定性的原生0–1指标。每任务提示策略经三个先导模型比较后固定，对十模型一致；病历中有记录但未进入参考的诊断作中性处理。排除只用13名患者的医生参照和三模型Agent循环实验。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.30027v1","published_at":"2026-09-24","status":"paper_baselines"},"score_min":0,"score_max":1,"saturation_excluded":true,"scale_label":{"en":"Native score / 1","zh":"原生分数 / 1"},"score_decimals":4,"model_count":10,"result_count":10,"results":[[12226,0.496,370,12238,12228],[12229,0.518,370,12238,12228],[12230,0.517,370,12238,12228],[3406,0.473,370,12238,12228],[12231,0.492,370,12238,12228],[12232,0.505,370,12238,12228],[12233,0.497,370,12238,12228],[12234,0.435,370,12238,12228],[12235,0.406,370,12238,12228],[12236,0.418,370,12238,12228]]},{"id":"topu-lbvs-full-2026-paper","name":"TopU-LBVS ·full (2026 paper)","group":"数学与科学","metric":"Mean EF@1%","unit":"raw","public_history":true,"protocol":{"en":"Nine learned-model/configuration rows represent six independent architectures: MorganRF,GIN,GAT,GPS,D-MPNN andMolFormer. Fingerprint-augmented GIN/GAT/GPS are configurations, not extra architectures; parameter-free TanimotoNN is excluded. Full/low protocols share93 targets and fixed model-common splits at1active:40decoys; full uses ChEMBL training with separate TopU tests, low uses target-specific scaffold splits. Mini protocols use the same seven targets, fixed training/validation and test actives, varying only hard versus random decoys. Enrichment factors retain native multiplicative units; three-seed means are selected by validation PR-AUC.","zh":"九条学习模型与配置结果覆盖MorganRF、GIN、GAT、GPS、D-MPNN、MolFormer六种独立架构；拼接指纹的变体不增加架构数，排除无训练参数的TanimotoNN。完整与低样本协议共享93靶标和各模型一致的固定划分，活性与诱饵比例1:40；前者用ChEMBL训练及TopU测试，后者在各靶标内按骨架划分。两种mini协议共享七靶标、训练验证与测试活性分子，只改变困难或随机诱饵。富集因子保留原生倍数，三种子成绩按验证PR-AUC选权重。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.29740v1","published_at":"2026-09-24","status":"paper_baselines"},"saturation_excluded":true,"scale_label":{"en":"Mean EF@1%","zh":"Mean EF@1%"},"score_decimals":3,"model_count":6,"result_count":9,"results":[[12239,9.18,370,12240,12241],[12242,4.47,370,12240,12241],[12242,6.64,370,12243,12241],[12244,3.76,370,12240,12241],[12244,6.57,370,12243,12241],[12245,3.36,370,12240,12241],[12245,6.6,370,12243,12241],[12246,7.38,370,12240,12241],[12247,6.65,370,12240,12241]]},{"id":"topu-lbvs-low-2026-paper","name":"TopU-LBVS ·low (2026 paper)","group":"数学与科学","metric":"Mean EF@10%","unit":"raw","public_history":true,"protocol":{"en":"Nine learned-model/configuration rows represent six independent architectures: MorganRF,GIN,GAT,GPS,D-MPNN andMolFormer. Fingerprint-augmented GIN/GAT/GPS are configurations, not extra architectures; parameter-free TanimotoNN is excluded. Full/low protocols share93 targets and fixed model-common splits at1active:40decoys; full uses ChEMBL training with separate TopU tests, low uses target-specific scaffold splits. Mini protocols use the same seven targets, fixed training/validation and test actives, varying only hard versus random decoys. Enrichment factors retain native multiplicative units; three-seed means are selected by validation PR-AUC.","zh":"九条学习模型与配置结果覆盖MorganRF、GIN、GAT、GPS、D-MPNN、MolFormer六种独立架构；拼接指纹的变体不增加架构数，排除无训练参数的TanimotoNN。完整与低样本协议共享93靶标和各模型一致的固定划分，活性与诱饵比例1:40；前者用ChEMBL训练及TopU测试，后者在各靶标内按骨架划分。两种mini协议共享七靶标、训练验证与测试活性分子，只改变困难或随机诱饵。富集因子保留原生倍数，三种子成绩按验证PR-AUC选权重。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.29740v1","published_at":"2026-09-24","status":"paper_baselines"},"saturation_excluded":true,"scale_label":{"en":"Mean EF@10%","zh":"Mean EF@10%"},"score_decimals":3,"model_count":6,"result_count":9,"results":[[12239,0.63,370,12248,12249],[12242,1.03,370,12248,12249],[12242,0.84,370,12250,12249],[12244,0.97,370,12248,12249],[12244,0.75,370,12250,12249],[12245,1.05,370,12248,12249],[12245,0.85,370,12250,12249],[12246,0.96,370,12248,12249],[12247,1.12,370,12248,12249]]},{"id":"topu-lbvs-mini-hard-2026-paper","name":"TopU-LBVS ·mini-hard (2026 paper)","group":"数学与科学","metric":"Seven-target mean EF@1%","unit":"raw","public_history":true,"protocol":{"en":"Nine learned-model/configuration rows represent six independent architectures: MorganRF,GIN,GAT,GPS,D-MPNN andMolFormer. Fingerprint-augmented GIN/GAT/GPS are configurations, not extra architectures; parameter-free TanimotoNN is excluded. Full/low protocols share93 targets and fixed model-common splits at1active:40decoys; full uses ChEMBL training with separate TopU tests, low uses target-specific scaffold splits. Mini protocols use the same seven targets, fixed training/validation and test actives, varying only hard versus random decoys. Enrichment factors retain native multiplicative units; three-seed means are selected by validation PR-AUC.","zh":"九条学习模型与配置结果覆盖MorganRF、GIN、GAT、GPS、D-MPNN、MolFormer六种独立架构；拼接指纹的变体不增加架构数，排除无训练参数的TanimotoNN。完整与低样本协议共享93靶标和各模型一致的固定划分，活性与诱饵比例1:40；前者用ChEMBL训练及TopU测试，后者在各靶标内按骨架划分。两种mini协议共享七靶标、训练验证与测试活性分子，只改变困难或随机诱饵。富集因子保留原生倍数，三种子成绩按验证PR-AUC选权重。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.29740v1","published_at":"2026-09-24","status":"paper_baselines"},"saturation_excluded":true,"scale_label":{"en":"Seven-target mean EF@1%","zh":"Seven-target mean EF@1%"},"score_decimals":3,"model_count":6,"result_count":9,"results":[[12239,8.6,370,12251,12252],[12242,3.5,370,12251,12252],[12242,5.7,370,12253,12252],[12244,3.5,370,12251,12252],[12244,5.2,370,12253,12252],[12245,2.9,370,12251,12252],[12245,5.7,370,12253,12252],[12246,7.3,370,12251,12252],[12247,5.2,370,12251,12252]]},{"id":"topu-lbvs-mini-random-2026-paper","name":"TopU-LBVS ·mini-random (2026 paper)","group":"数学与科学","metric":"Seven-target mean EF@1%","unit":"raw","public_history":true,"protocol":{"en":"Nine learned-model/configuration rows represent six independent architectures: MorganRF,GIN,GAT,GPS,D-MPNN andMolFormer. Fingerprint-augmented GIN/GAT/GPS are configurations, not extra architectures; parameter-free TanimotoNN is excluded. Full/low protocols share93 targets and fixed model-common splits at1active:40decoys; full uses ChEMBL training with separate TopU tests, low uses target-specific scaffold splits. Mini protocols use the same seven targets, fixed training/validation and test actives, varying only hard versus random decoys. Enrichment factors retain native multiplicative units; three-seed means are selected by validation PR-AUC.","zh":"九条学习模型与配置结果覆盖MorganRF、GIN、GAT、GPS、D-MPNN、MolFormer六种独立架构；拼接指纹的变体不增加架构数，排除无训练参数的TanimotoNN。完整与低样本协议共享93靶标和各模型一致的固定划分，活性与诱饵比例1:40；前者用ChEMBL训练及TopU测试，后者在各靶标内按骨架划分。两种mini协议共享七靶标、训练验证与测试活性分子，只改变困难或随机诱饵。富集因子保留原生倍数，三种子成绩按验证PR-AUC选权重。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.29740v1","published_at":"2026-09-24","status":"paper_baselines"},"saturation_excluded":true,"scale_label":{"en":"Seven-target mean EF@1%","zh":"Seven-target mean EF@1%"},"score_decimals":3,"model_count":6,"result_count":9,"results":[[12239,25.5,370,12254,12252],[12242,16.3,370,12254,12252],[12242,26.6,370,12255,12252],[12244,14.3,370,12254,12252],[12244,26.8,370,12255,12252],[12245,11.5,370,12254,12252],[12245,26.2,370,12255,12252],[12246,25.3,370,12254,12252],[12247,22.9,370,12254,12252]]},{"id":"orca-querying-2026-paper","name":"ORCA ·querying (2026 paper)","group":"编程","metric":"Executable translation success rate","unit":"%","public_history":true,"protocol":{"en":"Sixteen distinct language-model checkpoints translate the same600 source programs in the selected ORCA-Main population. The full1600 tasks comprise600 Pandas/PostgreSQL queries,600 NumPy/Pandas manipulations and400 PyTorch/TensorFlow tasks. Success requires executable target-language tests to establish functional equivalence. Each task runs three times per model at temperature0/top_p1; the original mean and standard deviation remain in source cells. Expert reference construction, alternative-correct/incorrect implementations and adjudication precede release.","zh":"十六个独立模型权重翻译所选ORCA-Main面板中相同600条源程序。完整1600题包含600条Pandas/PostgreSQL查询、600条NumPy/Pandas变换及400条PyTorch/TensorFlow任务，以目标库执行测试核验功能等价。每题每模型在温度0、top_p1下运行三次，原始均值与标准差保留在源单元。参考答案经专家构造、正确与错误替代实现检查及争议裁决。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.30749v1","published_at":"2026-09-25","status":"paper_baselines"},"model_count":16,"result_count":16,"results":[[6467,0.1167,370,12256,12257],[587,0.0722,370,12256,12257],[6687,0.18109999999999998,370,12256,12257],[1621,0.2778,370,12256,12257],[6689,0.2078,370,12256,12257],[5836,0.27,370,12256,12257],[542,0.2089,370,12256,12257],[12258,0.4111,370,12256,12257],[2278,0.4522,370,12256,12257],[12259,0.49,370,12256,12257],[1573,0.5289,370,12256,12257],[769,0.5456,370,12256,12257],[2998,0.5767,370,12256,12257],[6704,0.6311,370,12256,12257],[502,0.6811,370,12256,12257],[3183,0.6867,370,12256,12257]]},{"id":"orca-manipulation-2026-paper","name":"ORCA ·manipulation (2026 paper)","group":"编程","metric":"Executable translation success rate","unit":"%","public_history":true,"protocol":{"en":"Sixteen distinct language-model checkpoints translate the same600 source programs in the selected ORCA-Main population. The full1600 tasks comprise600 Pandas/PostgreSQL queries,600 NumPy/Pandas manipulations and400 PyTorch/TensorFlow tasks. Success requires executable target-language tests to establish functional equivalence. Each task runs three times per model at temperature0/top_p1; the original mean and standard deviation remain in source cells. Expert reference construction, alternative-correct/incorrect implementations and adjudication precede release.","zh":"十六个独立模型权重翻译所选ORCA-Main面板中相同600条源程序。完整1600题包含600条Pandas/PostgreSQL查询、600条NumPy/Pandas变换及400条PyTorch/TensorFlow任务，以目标库执行测试核验功能等价。每题每模型在温度0、top_p1下运行三次，原始均值与标准差保留在源单元。参考答案经专家构造、正确与错误替代实现检查及争议裁决。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.30749v1","published_at":"2026-09-25","status":"paper_baselines"},"model_count":16,"result_count":16,"results":[[6467,0.1722,370,12256,12257],[587,0.1133,370,12256,12257],[6687,0.2833,370,12256,12257],[1621,0.2556,370,12256,12257],[6689,0.2833,370,12256,12257],[5836,0.3378,370,12256,12257],[542,0.2867,370,12256,12257],[12258,0.23559999999999998,370,12256,12257],[2278,0.4689,370,12256,12257],[12259,0.44329999999999997,370,12256,12257],[1573,0.4733,370,12256,12257],[769,0.5211,370,12256,12257],[2998,0.5122,370,12256,12257],[6704,0.5033,370,12256,12257],[502,0.5389,370,12256,12257],[3183,0.54,370,12256,12257]]},{"id":"orca-deep-learning-2026-paper","name":"ORCA ·deep-learning (2026 paper)","group":"编程","metric":"Executable translation success rate","unit":"%","public_history":true,"protocol":{"en":"Sixteen distinct language-model checkpoints translate the same400 source programs in the selected ORCA-Main population. The full1600 tasks comprise600 Pandas/PostgreSQL queries,600 NumPy/Pandas manipulations and400 PyTorch/TensorFlow tasks. Success requires executable target-language tests to establish functional equivalence. Each task runs three times per model at temperature0/top_p1; the original mean and standard deviation remain in source cells. Expert reference construction, alternative-correct/incorrect implementations and adjudication precede release.","zh":"十六个独立模型权重翻译所选ORCA-Main面板中相同400条源程序。完整1600题包含600条Pandas/PostgreSQL查询、600条NumPy/Pandas变换及400条PyTorch/TensorFlow任务，以目标库执行测试核验功能等价。每题每模型在温度0、top_p1下运行三次，原始均值与标准差保留在源单元。参考答案经专家构造、正确与错误替代实现检查及争议裁决。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.30749v1","published_at":"2026-09-25","status":"paper_baselines"},"model_count":16,"result_count":16,"results":[[6467,0.23329999999999998,370,12260,12257],[587,0.185,370,12260,12257],[6687,0.19329999999999997,370,12260,12257],[1621,0.25670000000000004,370,12260,12257],[6689,0.22,370,12260,12257],[5836,0.3233,370,12260,12257],[542,0.305,370,12260,12257],[12258,0.2967,370,12260,12257],[2278,0.3467,370,12260,12257],[12259,0.3983,370,12260,12257],[1573,0.375,370,12260,12257],[769,0.325,370,12260,12257],[2998,0.4117,370,12260,12257],[6704,0.405,370,12260,12257],[502,0.4267,370,12260,12257],[3183,0.43670000000000003,370,12260,12257]]},{"id":"orca-main-overall-2026-paper","name":"ORCA ·main-overall (2026 paper)","group":"编程","metric":"Executable translation success rate","unit":"%","public_history":true,"protocol":{"en":"Sixteen distinct language-model checkpoints translate the same1600 source programs in the selected ORCA-Main population. The full1600 tasks comprise600 Pandas/PostgreSQL queries,600 NumPy/Pandas manipulations and400 PyTorch/TensorFlow tasks. Success requires executable target-language tests to establish functional equivalence. Each task runs three times per model at temperature0/top_p1; the original mean and standard deviation remain in source cells. Expert reference construction, alternative-correct/incorrect implementations and adjudication precede release.","zh":"十六个独立模型权重翻译所选ORCA-Main面板中相同1600条源程序。完整1600题包含600条Pandas/PostgreSQL查询、600条NumPy/Pandas变换及400条PyTorch/TensorFlow任务，以目标库执行测试核验功能等价。每题每模型在温度0、top_p1下运行三次，原始均值与标准差保留在源单元。参考答案经专家构造、正确与错误替代实现检查及争议裁决。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.30749v1","published_at":"2026-09-25","status":"paper_baselines"},"model_count":16,"result_count":16,"results":[[6467,0.16670000000000001,370,12261,12257],[587,0.1158,370,12261,12257],[6687,0.2225,370,12261,12257],[1621,0.2642,370,12261,12257],[6689,0.23920000000000002,370,12261,12257],[5836,0.30879999999999996,370,12261,12257],[542,0.2621,370,12261,12257],[12258,0.31670000000000004,370,12261,12257],[2278,0.4321,370,12261,12257],[12259,0.4496,370,12261,12257],[1573,0.4696,370,12261,12257],[769,0.48119999999999996,370,12261,12257],[2998,0.5112,370,12261,12257],[6704,0.5267000000000001,370,12261,12257],[502,0.5642,370,12261,12257],[3183,0.5692,370,12261,12257]]},{"id":"orca-project-2026-paper","name":"ORCA ·Project (2026 paper)","group":"编程","metric":"Project-level executable translation success rate","unit":"%","public_history":true,"protocol":{"en":"Eight distinct model checkpoints translate the same200 complete data-science projects across seven task types. This population is separate from the1600 grounding tasks and from later100-task Claude Code/ReAct analyses. Official executable tests check the translated multi-file/project behavior. Original results are mean success rates across three runs at temperature0/top_p1.","zh":"八个独立模型权重翻译相同200个完整数据科学项目，覆盖七类任务。该面板与1600道基础翻译题及后续100题Claude Code/ReAct分析分开。可执行测试核验多文件项目行为，保留温度0、top_p1下三次运行的原始成功率均值。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.30749v1","published_at":"2026-09-25","status":"paper_baselines"},"model_count":8,"result_count":8,"results":[[1621,0.0,370,12262,12263],[5836,0.0467,370,12262,12263],[542,0.02,370,12262,12263],[2278,0.1567,370,12262,12263],[1573,0.18,370,12262,12263],[2998,0.24,370,12262,12263],[6704,0.28,370,12262,12263],[3183,0.3367,370,12262,12263]]},{"id":"satnav-seen-success-2026-paper","name":"SatNav ·seen success (2026 paper)","group":"Agent 与工具","metric":"Navigation stop success","unit":"%","public_history":true,"protocol":{"en":"Eleven reported model/configuration rows represent seven learned navigation architectures on the same4574 seen episodes. Starred adaptations remain configurations, not extra models. Success requires stopping within10m of the goal, or30m for Landmark; Boundary episodes must first leave20m from the start. SPL penalizes excess path length, oracle success counts entering the success region at any time, and navigation error is final distance in meters. Train/TestSeen/TestUnseen separate trajectories/instructions or geographic scenes; the two test populations are not pooled.","zh":"十一条模型与配置结果覆盖七种学习型导航架构，评测共同4574条seen轨迹；星号适配不增加模型数。成功要求在目标10米内停止，Landmark放宽为30米；Boundary任务还要求先离开起点20米。SPL按路径长度折算成功，先知成功表示轨迹曾进入成功区，导航误差为终点到目标的米数。训练、已见和未见划分区分新轨迹指令或新地理场景，两组测试不合并。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.31507v1","published_at":"2026-09-25","status":"paper_baselines"},"model_count":7,"result_count":11,"results":[[12264,0.021,370,12265,12266],[12267,0.095,370,12265,12266],[12268,0.132,370,12265,12266],[12268,0.21100000000000002,370,12269,12266],[12270,0.18100000000000002,370,12265,12266],[12270,0.25,370,12269,12266],[12271,0.251,370,12265,12266],[12271,0.49700000000000005,370,12269,12266],[12272,0.643,370,12265,12266],[12272,0.6729999999999999,370,12269,12266],[12273,0.6579999999999999,370,12265,12266]]},{"id":"satnav-seen-spl-2026-paper","name":"SatNav ·seen spl (2026 paper)","group":"Agent 与工具","metric":"Success weighted by path length","unit":"%","public_history":true,"protocol":{"en":"Eleven reported model/configuration rows represent seven learned navigation architectures on the same4574 seen episodes. Starred adaptations remain configurations, not extra models. Success requires stopping within10m of the goal, or30m for Landmark; Boundary episodes must first leave20m from the start. SPL penalizes excess path length, oracle success counts entering the success region at any time, and navigation error is final distance in meters. Train/TestSeen/TestUnseen separate trajectories/instructions or geographic scenes; the two test populations are not pooled.","zh":"十一条模型与配置结果覆盖七种学习型导航架构，评测共同4574条seen轨迹；星号适配不增加模型数。成功要求在目标10米内停止，Landmark放宽为30米；Boundary任务还要求先离开起点20米。SPL按路径长度折算成功，先知成功表示轨迹曾进入成功区，导航误差为终点到目标的米数。训练、已见和未见划分区分新轨迹指令或新地理场景，两组测试不合并。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.31507v1","published_at":"2026-09-25","status":"paper_baselines"},"model_count":7,"result_count":11,"results":[[12264,0.021,370,12265,12266],[12267,0.09300000000000001,370,12265,12266],[12268,0.13,370,12265,12266],[12268,0.21,370,12269,12266],[12270,0.18,370,12265,12266],[12270,0.249,370,12269,12266],[12271,0.248,370,12265,12266],[12271,0.491,370,12269,12266],[12272,0.637,370,12265,12266],[12272,0.6679999999999999,370,12269,12266],[12273,0.655,370,12265,12266]]},{"id":"satnav-seen-oracle-success-2026-paper","name":"SatNav ·seen oracle-success (2026 paper)","group":"Agent 与工具","metric":"Trajectory oracle success","unit":"%","public_history":true,"protocol":{"en":"Eleven reported model/configuration rows represent seven learned navigation architectures on the same4574 seen episodes. Starred adaptations remain configurations, not extra models. Success requires stopping within10m of the goal, or30m for Landmark; Boundary episodes must first leave20m from the start. SPL penalizes excess path length, oracle success counts entering the success region at any time, and navigation error is final distance in meters. Train/TestSeen/TestUnseen separate trajectories/instructions or geographic scenes; the two test populations are not pooled.","zh":"十一条模型与配置结果覆盖七种学习型导航架构，评测共同4574条seen轨迹；星号适配不增加模型数。成功要求在目标10米内停止，Landmark放宽为30米；Boundary任务还要求先离开起点20米。SPL按路径长度折算成功，先知成功表示轨迹曾进入成功区，导航误差为终点到目标的米数。训练、已见和未见划分区分新轨迹指令或新地理场景，两组测试不合并。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.31507v1","published_at":"2026-09-25","status":"paper_baselines"},"model_count":7,"result_count":11,"results":[[12264,0.311,370,12265,12266],[12267,0.449,370,12265,12266],[12268,0.335,370,12265,12266],[12268,0.381,370,12269,12266],[12270,0.276,370,12265,12266],[12270,0.35,370,12269,12266],[12271,0.604,370,12265,12266],[12271,0.682,370,12269,12266],[12272,0.7170000000000001,370,12265,12266],[12272,0.746,370,12269,12266],[12273,0.7240000000000001,370,12265,12266]]},{"id":"satnav-seen-navigation-error-2026-paper","name":"SatNav ·seen navigation-error (2026 paper)","group":"Agent 与工具","metric":"Final navigation error (m)","unit":"raw","public_history":true,"protocol":{"en":"Eleven reported model/configuration rows represent seven learned navigation architectures on the same4574 seen episodes. Starred adaptations remain configurations, not extra models. Success requires stopping within10m of the goal, or30m for Landmark; Boundary episodes must first leave20m from the start. SPL penalizes excess path length, oracle success counts entering the success region at any time, and navigation error is final distance in meters. Train/TestSeen/TestUnseen separate trajectories/instructions or geographic scenes; the two test populations are not pooled.","zh":"十一条模型与配置结果覆盖七种学习型导航架构，评测共同4574条seen轨迹；星号适配不增加模型数。成功要求在目标10米内停止，Landmark放宽为30米；Boundary任务还要求先离开起点20米。SPL按路径长度折算成功，先知成功表示轨迹曾进入成功区，导航误差为终点到目标的米数。训练、已见和未见划分区分新轨迹指令或新地理场景，两组测试不合并。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.31507v1","published_at":"2026-09-25","status":"paper_baselines"},"saturation_excluded":true,"scale_label":{"en":"Final navigation error (m)","zh":"Final navigation error (m)"},"score_decimals":3,"higher_is_better":false,"model_count":7,"result_count":11,"results":[[12264,180.36,370,12265,12266],[12267,323.25,370,12265,12266],[12268,166.88,370,12265,12266],[12268,163.07,370,12269,12266],[12270,93.05,370,12265,12266],[12270,93.88,370,12269,12266],[12271,174.68,370,12265,12266],[12271,87.11,370,12269,12266],[12272,53.24,370,12265,12266],[12272,47.47,370,12269,12266],[12273,34.05,370,12265,12266]]},{"id":"satnav-unseen-success-2026-paper","name":"SatNav ·unseen success (2026 paper)","group":"Agent 与工具","metric":"Navigation stop success","unit":"%","public_history":true,"protocol":{"en":"Eleven reported model/configuration rows represent seven learned navigation architectures on the same8756 unseen episodes. Starred adaptations remain configurations, not extra models. Success requires stopping within10m of the goal, or30m for Landmark; Boundary episodes must first leave20m from the start. SPL penalizes excess path length, oracle success counts entering the success region at any time, and navigation error is final distance in meters. Train/TestSeen/TestUnseen separate trajectories/instructions or geographic scenes; the two test populations are not pooled.","zh":"十一条模型与配置结果覆盖七种学习型导航架构，评测共同8756条unseen轨迹；星号适配不增加模型数。成功要求在目标10米内停止，Landmark放宽为30米；Boundary任务还要求先离开起点20米。SPL按路径长度折算成功，先知成功表示轨迹曾进入成功区，导航误差为终点到目标的米数。训练、已见和未见划分区分新轨迹指令或新地理场景，两组测试不合并。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.31507v1","published_at":"2026-09-25","status":"paper_baselines"},"model_count":7,"result_count":11,"results":[[12264,0.016,370,12274,12266],[12267,0.073,370,12274,12266],[12268,0.11699999999999999,370,12274,12266],[12268,0.171,370,12275,12266],[12270,0.13,370,12274,12266],[12270,0.18600000000000003,370,12275,12266],[12271,0.204,370,12274,12266],[12271,0.36700000000000005,370,12275,12266],[12272,0.522,370,12274,12266],[12272,0.584,370,12275,12266],[12273,0.537,370,12274,12266]]},{"id":"satnav-unseen-spl-2026-paper","name":"SatNav ·unseen spl (2026 paper)","group":"Agent 与工具","metric":"Success weighted by path length","unit":"%","public_history":true,"protocol":{"en":"Eleven reported model/configuration rows represent seven learned navigation architectures on the same8756 unseen episodes. Starred adaptations remain configurations, not extra models. Success requires stopping within10m of the goal, or30m for Landmark; Boundary episodes must first leave20m from the start. SPL penalizes excess path length, oracle success counts entering the success region at any time, and navigation error is final distance in meters. Train/TestSeen/TestUnseen separate trajectories/instructions or geographic scenes; the two test populations are not pooled.","zh":"十一条模型与配置结果覆盖七种学习型导航架构，评测共同8756条unseen轨迹；星号适配不增加模型数。成功要求在目标10米内停止，Landmark放宽为30米；Boundary任务还要求先离开起点20米。SPL按路径长度折算成功，先知成功表示轨迹曾进入成功区，导航误差为终点到目标的米数。训练、已见和未见划分区分新轨迹指令或新地理场景，两组测试不合并。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.31507v1","published_at":"2026-09-25","status":"paper_baselines"},"model_count":7,"result_count":11,"results":[[12264,0.015,370,12274,12266],[12267,0.07200000000000001,370,12274,12266],[12268,0.11599999999999999,370,12274,12266],[12268,0.16899999999999998,370,12275,12266],[12270,0.127,370,12274,12266],[12270,0.184,370,12275,12266],[12271,0.2,370,12274,12266],[12271,0.363,370,12275,12266],[12272,0.518,370,12274,12266],[12272,0.578,370,12275,12266],[12273,0.532,370,12274,12266]]},{"id":"satnav-unseen-oracle-success-2026-paper","name":"SatNav ·unseen oracle-success (2026 paper)","group":"Agent 与工具","metric":"Trajectory oracle success","unit":"%","public_history":true,"protocol":{"en":"Eleven reported model/configuration rows represent seven learned navigation architectures on the same8756 unseen episodes. Starred adaptations remain configurations, not extra models. Success requires stopping within10m of the goal, or30m for Landmark; Boundary episodes must first leave20m from the start. SPL penalizes excess path length, oracle success counts entering the success region at any time, and navigation error is final distance in meters. Train/TestSeen/TestUnseen separate trajectories/instructions or geographic scenes; the two test populations are not pooled.","zh":"十一条模型与配置结果覆盖七种学习型导航架构，评测共同8756条unseen轨迹；星号适配不增加模型数。成功要求在目标10米内停止，Landmark放宽为30米；Boundary任务还要求先离开起点20米。SPL按路径长度折算成功，先知成功表示轨迹曾进入成功区，导航误差为终点到目标的米数。训练、已见和未见划分区分新轨迹指令或新地理场景，两组测试不合并。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.31507v1","published_at":"2026-09-25","status":"paper_baselines"},"model_count":7,"result_count":11,"results":[[12264,0.29600000000000004,370,12274,12266],[12267,0.439,370,12274,12266],[12268,0.322,370,12274,12266],[12268,0.349,370,12275,12266],[12270,0.237,370,12274,12266],[12270,0.316,370,12275,12266],[12271,0.499,370,12274,12266],[12271,0.5589999999999999,370,12275,12266],[12272,0.61,370,12274,12266],[12272,0.6829999999999999,370,12275,12266],[12273,0.6409999999999999,370,12274,12266]]},{"id":"satnav-unseen-navigation-error-2026-paper","name":"SatNav ·unseen navigation-error (2026 paper)","group":"Agent 与工具","metric":"Final navigation error (m)","unit":"raw","public_history":true,"protocol":{"en":"Eleven reported model/configuration rows represent seven learned navigation architectures on the same8756 unseen episodes. Starred adaptations remain configurations, not extra models. Success requires stopping within10m of the goal, or30m for Landmark; Boundary episodes must first leave20m from the start. SPL penalizes excess path length, oracle success counts entering the success region at any time, and navigation error is final distance in meters. Train/TestSeen/TestUnseen separate trajectories/instructions or geographic scenes; the two test populations are not pooled.","zh":"十一条模型与配置结果覆盖七种学习型导航架构，评测共同8756条unseen轨迹；星号适配不增加模型数。成功要求在目标10米内停止，Landmark放宽为30米；Boundary任务还要求先离开起点20米。SPL按路径长度折算成功，先知成功表示轨迹曾进入成功区，导航误差为终点到目标的米数。训练、已见和未见划分区分新轨迹指令或新地理场景，两组测试不合并。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.31507v1","published_at":"2026-09-25","status":"paper_baselines"},"saturation_excluded":true,"scale_label":{"en":"Final navigation error (m)","zh":"Final navigation error (m)"},"score_decimals":3,"higher_is_better":false,"model_count":7,"result_count":11,"results":[[12264,219.17,370,12274,12266],[12267,363.59,370,12274,12266],[12268,195.91,370,12274,12266],[12268,196.9,370,12275,12266],[12270,128.88,370,12274,12266],[12270,123.49,370,12275,12266],[12271,228.46,370,12274,12266],[12271,149.85,370,12275,12266],[12272,84.99,370,12274,12266],[12272,86.63,370,12275,12266],[12273,62.29,370,12274,12266]]},{"id":"storm-cook-2026-paper","name":"STORM-Bench ·cook (2026 paper)","group":"视觉与视频","metric":"Laplace-smoothed harmonic reliability","unit":"score","public_history":true,"protocol":{"en":"Fourteen distinct open-weight video models process the same online question stream at1FPS, retaining recent frames up to their published native caps and never observing future frames. STORM has5736 questions/630 episodes over five real and two simulation domains. Storm-BR is a Laplace-smoothed harmonic mean over six intensity×answerability cells using joint answer/status correctness, not ordinary percent-correct. The seven-domain headline score and task accuracy equally weight domains. Each domain population and native frame cap remain explicit; full-video controls are excluded.","zh":"十四个独立开源视频模型以每秒一帧处理相同在线问题流，按原生上限保留最近帧，不查看未来画面。STORM含5736题、630片段，覆盖五个真实和两个模拟领域。Storm-BR对变化强度与可回答性组成的六格联合答案/状态正确率作拉普拉斯平滑调和平均，保留原生可靠性分数。七领域总分及任务正确率均对领域等权；保留各领域与帧预算，排除全视频参照。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.30981v1","published_at":"2026-09-25","status":"paper_baselines"},"score_min":0,"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Native score / 100","zh":"原生分数 / 100"},"score_decimals":2,"model_count":14,"result_count":14,"results":[[5725,34.45,370,12276,12277],[4070,33.29,370,12278,12277],[6298,26.77,370,12278,12277],[4969,34.04,370,12278,12277],[5995,22.17,370,12279,12277],[9870,23.63,370,12280,12277],[6342,15.21,370,12278,12277],[5536,21.4,370,12281,12277],[12282,13.39,370,12283,12277],[2754,31.6,370,12278,12277],[6349,11.1,370,12284,12277],[2771,19.22,370,12278,12277],[2654,8.97,370,12285,12277],[12286,1.13,370,12287,12277]]},{"id":"storm-bike-2026-paper","name":"STORM-Bench ·bike (2026 paper)","group":"视觉与视频","metric":"Laplace-smoothed harmonic reliability","unit":"score","public_history":true,"protocol":{"en":"Fourteen distinct open-weight video models process the same online question stream at1FPS, retaining recent frames up to their published native caps and never observing future frames. STORM has5736 questions/630 episodes over five real and two simulation domains. Storm-BR is a Laplace-smoothed harmonic mean over six intensity×answerability cells using joint answer/status correctness, not ordinary percent-correct. The seven-domain headline score and task accuracy equally weight domains. Each domain population and native frame cap remain explicit; full-video controls are excluded.","zh":"十四个独立开源视频模型以每秒一帧处理相同在线问题流，按原生上限保留最近帧，不查看未来画面。STORM含5736题、630片段，覆盖五个真实和两个模拟领域。Storm-BR对变化强度与可回答性组成的六格联合答案/状态正确率作拉普拉斯平滑调和平均，保留原生可靠性分数。七领域总分及任务正确率均对领域等权；保留各领域与帧预算，排除全视频参照。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.30981v1","published_at":"2026-09-25","status":"paper_baselines"},"score_min":0,"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Native score / 100","zh":"原生分数 / 100"},"score_decimals":2,"model_count":14,"result_count":14,"results":[[5725,33.59,370,12276,12277],[4070,22.22,370,12278,12277],[6298,31.49,370,12278,12277],[4969,8.72,370,12278,12277],[5995,31.97,370,12279,12277],[9870,28.86,370,12280,12277],[6342,19.0,370,12278,12277],[5536,13.35,370,12281,12277],[12282,20.72,370,12283,12277],[2754,3.77,370,12278,12277],[6349,15.71,370,12284,12277],[2771,12.69,370,12278,12277],[2654,7.41,370,12285,12277],[12286,3.72,370,12287,12277]]},{"id":"storm-health-2026-paper","name":"STORM-Bench ·health (2026 paper)","group":"视觉与视频","metric":"Laplace-smoothed harmonic reliability","unit":"score","public_history":true,"protocol":{"en":"Fourteen distinct open-weight video models process the same online question stream at1FPS, retaining recent frames up to their published native caps and never observing future frames. STORM has5736 questions/630 episodes over five real and two simulation domains. Storm-BR is a Laplace-smoothed harmonic mean over six intensity×answerability cells using joint answer/status correctness, not ordinary percent-correct. The seven-domain headline score and task accuracy equally weight domains. Each domain population and native frame cap remain explicit; full-video controls are excluded.","zh":"十四个独立开源视频模型以每秒一帧处理相同在线问题流，按原生上限保留最近帧，不查看未来画面。STORM含5736题、630片段，覆盖五个真实和两个模拟领域。Storm-BR对变化强度与可回答性组成的六格联合答案/状态正确率作拉普拉斯平滑调和平均，保留原生可靠性分数。七领域总分及任务正确率均对领域等权；保留各领域与帧预算，排除全视频参照。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.30981v1","published_at":"2026-09-25","status":"paper_baselines"},"score_min":0,"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Native score / 100","zh":"原生分数 / 100"},"score_decimals":2,"model_count":14,"result_count":14,"results":[[5725,31.8,370,12276,12277],[4070,33.23,370,12278,12277],[6298,29.71,370,12278,12277],[4969,33.95,370,12278,12277],[5995,18.71,370,12279,12277],[9870,15.74,370,12280,12277],[6342,21.28,370,12278,12277],[5536,21.69,370,12281,12277],[12282,12.27,370,12283,12277],[2754,7.78,370,12278,12277],[6349,9.66,370,12284,12277],[2771,13.48,370,12278,12277],[2654,17.14,370,12285,12277],[12286,3.82,370,12287,12277]]},{"id":"storm-music-2026-paper","name":"STORM-Bench ·music (2026 paper)","group":"视觉与视频","metric":"Laplace-smoothed harmonic reliability","unit":"score","public_history":true,"protocol":{"en":"Fourteen distinct open-weight video models process the same online question stream at1FPS, retaining recent frames up to their published native caps and never observing future frames. STORM has5736 questions/630 episodes over five real and two simulation domains. Storm-BR is a Laplace-smoothed harmonic mean over six intensity×answerability cells using joint answer/status correctness, not ordinary percent-correct. The seven-domain headline score and task accuracy equally weight domains. Each domain population and native frame cap remain explicit; full-video controls are excluded.","zh":"十四个独立开源视频模型以每秒一帧处理相同在线问题流，按原生上限保留最近帧，不查看未来画面。STORM含5736题、630片段，覆盖五个真实和两个模拟领域。Storm-BR对变化强度与可回答性组成的六格联合答案/状态正确率作拉普拉斯平滑调和平均，保留原生可靠性分数。七领域总分及任务正确率均对领域等权；保留各领域与帧预算，排除全视频参照。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.30981v1","published_at":"2026-09-25","status":"paper_baselines"},"score_min":0,"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Native score / 100","zh":"原生分数 / 100"},"score_decimals":2,"model_count":14,"result_count":14,"results":[[5725,43.29,370,12276,12277],[4070,42.7,370,12278,12277],[6298,36.92,370,12278,12277],[4969,36.97,370,12278,12277],[5995,27.89,370,12279,12277],[9870,25.85,370,12280,12277],[6342,27.92,370,12278,12277],[5536,22.21,370,12281,12277],[12282,26.1,370,12283,12277],[2754,10.69,370,12278,12277],[6349,18.72,370,12284,12277],[2771,10.3,370,12278,12277],[2654,16.05,370,12285,12277],[12286,11.48,370,12287,12277]]},{"id":"storm-sports-2026-paper","name":"STORM-Bench ·sports (2026 paper)","group":"视觉与视频","metric":"Laplace-smoothed harmonic reliability","unit":"score","public_history":true,"protocol":{"en":"Fourteen distinct open-weight video models process the same online question stream at1FPS, retaining recent frames up to their published native caps and never observing future frames. STORM has5736 questions/630 episodes over five real and two simulation domains. Storm-BR is a Laplace-smoothed harmonic mean over six intensity×answerability cells using joint answer/status correctness, not ordinary percent-correct. The seven-domain headline score and task accuracy equally weight domains. Each domain population and native frame cap remain explicit; full-video controls are excluded.","zh":"十四个独立开源视频模型以每秒一帧处理相同在线问题流，按原生上限保留最近帧，不查看未来画面。STORM含5736题、630片段，覆盖五个真实和两个模拟领域。Storm-BR对变化强度与可回答性组成的六格联合答案/状态正确率作拉普拉斯平滑调和平均，保留原生可靠性分数。七领域总分及任务正确率均对领域等权；保留各领域与帧预算，排除全视频参照。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.30981v1","published_at":"2026-09-25","status":"paper_baselines"},"score_min":0,"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Native score / 100","zh":"原生分数 / 100"},"score_decimals":2,"model_count":14,"result_count":14,"results":[[5725,31.34,370,12276,12277],[4070,24.06,370,12278,12277],[6298,20.89,370,12278,12277],[4969,18.09,370,12278,12277],[5995,17.16,370,12279,12277],[9870,12.41,370,12280,12277],[6342,16.05,370,12278,12277],[5536,10.76,370,12281,12277],[12282,14.37,370,12283,12277],[2754,4.73,370,12278,12277],[6349,15.93,370,12284,12277],[2771,9.62,370,12278,12277],[2654,8.29,370,12285,12277],[12286,3.43,370,12287,12277]]},{"id":"storm-thor-2026-paper","name":"STORM-Bench ·thor (2026 paper)","group":"视觉与视频","metric":"Laplace-smoothed harmonic reliability","unit":"score","public_history":true,"protocol":{"en":"Fourteen distinct open-weight video models process the same online question stream at1FPS, retaining recent frames up to their published native caps and never observing future frames. STORM has5736 questions/630 episodes over five real and two simulation domains. Storm-BR is a Laplace-smoothed harmonic mean over six intensity×answerability cells using joint answer/status correctness, not ordinary percent-correct. The seven-domain headline score and task accuracy equally weight domains. Each domain population and native frame cap remain explicit; full-video controls are excluded.","zh":"十四个独立开源视频模型以每秒一帧处理相同在线问题流，按原生上限保留最近帧，不查看未来画面。STORM含5736题、630片段，覆盖五个真实和两个模拟领域。Storm-BR对变化强度与可回答性组成的六格联合答案/状态正确率作拉普拉斯平滑调和平均，保留原生可靠性分数。七领域总分及任务正确率均对领域等权；保留各领域与帧预算，排除全视频参照。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.30981v1","published_at":"2026-09-25","status":"paper_baselines"},"score_min":0,"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Native score / 100","zh":"原生分数 / 100"},"score_decimals":2,"model_count":14,"result_count":14,"results":[[5725,34.15,370,12276,12277],[4070,25.89,370,12278,12277],[6298,19.27,370,12278,12277],[4969,15.42,370,12278,12277],[5995,21.7,370,12279,12277],[9870,15.44,370,12280,12277],[6342,11.23,370,12278,12277],[5536,11.34,370,12281,12277],[12282,10.64,370,12283,12277],[2754,10.17,370,12278,12277],[6349,9.32,370,12284,12277],[2771,10.3,370,12278,12277],[2654,14.0,370,12285,12277],[12286,8.23,370,12287,12277]]},{"id":"storm-vhome-2026-paper","name":"STORM-Bench ·vhome (2026 paper)","group":"视觉与视频","metric":"Laplace-smoothed harmonic reliability","unit":"score","public_history":true,"protocol":{"en":"Fourteen distinct open-weight video models process the same online question stream at1FPS, retaining recent frames up to their published native caps and never observing future frames. STORM has5736 questions/630 episodes over five real and two simulation domains. Storm-BR is a Laplace-smoothed harmonic mean over six intensity×answerability cells using joint answer/status correctness, not ordinary percent-correct. The seven-domain headline score and task accuracy equally weight domains. Each domain population and native frame cap remain explicit; full-video controls are excluded.","zh":"十四个独立开源视频模型以每秒一帧处理相同在线问题流，按原生上限保留最近帧，不查看未来画面。STORM含5736题、630片段，覆盖五个真实和两个模拟领域。Storm-BR对变化强度与可回答性组成的六格联合答案/状态正确率作拉普拉斯平滑调和平均，保留原生可靠性分数。七领域总分及任务正确率均对领域等权；保留各领域与帧预算，排除全视频参照。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.30981v1","published_at":"2026-09-25","status":"paper_baselines"},"score_min":0,"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Native score / 100","zh":"原生分数 / 100"},"score_decimals":2,"model_count":14,"result_count":14,"results":[[5725,40.35,370,12276,12277],[4070,15.1,370,12278,12277],[6298,18.96,370,12278,12277],[4969,16.65,370,12278,12277],[5995,20.33,370,12279,12277],[9870,19.45,370,12280,12277],[6342,19.04,370,12278,12277],[5536,14.03,370,12281,12277],[12282,15.45,370,12283,12277],[2754,21.5,370,12278,12277],[6349,9.31,370,12284,12277],[2771,13.25,370,12278,12277],[2654,12.5,370,12285,12277],[12286,8.1,370,12287,12277]]},{"id":"storm-domain-macro-br-2026-paper","name":"STORM-Bench ·domain-macro-br (2026 paper)","group":"视觉与视频","metric":"Laplace-smoothed harmonic reliability","unit":"score","public_history":true,"protocol":{"en":"Fourteen distinct open-weight video models process the same online question stream at1FPS, retaining recent frames up to their published native caps and never observing future frames. STORM has5736 questions/630 episodes over five real and two simulation domains. Storm-BR is a Laplace-smoothed harmonic mean over six intensity×answerability cells using joint answer/status correctness, not ordinary percent-correct. The seven-domain headline score and task accuracy equally weight domains. Each domain population and native frame cap remain explicit; full-video controls are excluded.","zh":"十四个独立开源视频模型以每秒一帧处理相同在线问题流，按原生上限保留最近帧，不查看未来画面。STORM含5736题、630片段，覆盖五个真实和两个模拟领域。Storm-BR对变化强度与可回答性组成的六格联合答案/状态正确率作拉普拉斯平滑调和平均，保留原生可靠性分数。七领域总分及任务正确率均对领域等权；保留各领域与帧预算，排除全视频参照。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.30981v1","published_at":"2026-09-25","status":"paper_baselines"},"score_min":0,"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Native score / 100","zh":"原生分数 / 100"},"score_decimals":2,"model_count":14,"result_count":14,"results":[[5725,35.57,370,12276,12277],[4070,28.07,370,12278,12277],[6298,26.29,370,12278,12277],[4969,23.41,370,12278,12277],[5995,22.84,370,12279,12277],[9870,20.2,370,12280,12277],[6342,18.53,370,12278,12277],[5536,16.4,370,12281,12277],[12282,16.13,370,12283,12277],[2754,12.89,370,12278,12277],[6349,12.82,370,12284,12277],[2771,12.69,370,12278,12277],[2654,12.05,370,12285,12277],[12286,5.7,370,12287,12277]]},{"id":"storm-domain-macro-accuracy-2026-paper","name":"STORM-Bench ·domain-macro-accuracy (2026 paper)","group":"视觉与视频","metric":"Domain-macro task accuracy","unit":"%","public_history":true,"protocol":{"en":"Fourteen distinct open-weight video models process the same online question stream at1FPS, retaining recent frames up to their published native caps and never observing future frames. STORM has5736 questions/630 episodes over five real and two simulation domains. Storm-BR is a Laplace-smoothed harmonic mean over six intensity×answerability cells using joint answer/status correctness, not ordinary percent-correct. The seven-domain headline score and task accuracy equally weight domains. Each domain population and native frame cap remain explicit; full-video controls are excluded.","zh":"十四个独立开源视频模型以每秒一帧处理相同在线问题流，按原生上限保留最近帧，不查看未来画面。STORM含5736题、630片段，覆盖五个真实和两个模拟领域。Storm-BR对变化强度与可回答性组成的六格联合答案/状态正确率作拉普拉斯平滑调和平均，保留原生可靠性分数。七领域总分及任务正确率均对领域等权；保留各领域与帧预算，排除全视频参照。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.30981v1","published_at":"2026-09-25","status":"paper_baselines"},"model_count":14,"result_count":14,"results":[[5725,0.6031,370,12276,12277],[4070,0.6033,370,12278,12277],[6298,0.5516,370,12278,12277],[4969,0.5820000000000001,370,12278,12277],[5995,0.5478000000000001,370,12279,12277],[9870,0.5376,370,12280,12277],[6342,0.5133,370,12278,12277],[5536,0.4886,370,12281,12277],[12282,0.5015999999999999,370,12283,12277],[2754,0.4924,370,12278,12277],[6349,0.5609000000000001,370,12284,12277],[2771,0.4457,370,12278,12277],[2654,0.5086999999999999,370,12285,12277],[12286,0.3037,370,12287,12277]]},{"id":"trafficimag-reasoning-set-f1-2026-paper","name":"TrafficImag ·behavior set-f1 (2026 paper)","group":"视觉与视频","metric":"Behavior label set F1","unit":"score","public_history":true,"protocol":{"en":"Seven distinct video/image-language models share the1500 actor-centered history/future audit samples. The behavior reasoner predicts the released behavior-label program; set F1 and label-macro F1 remain native0–100, not binary accuracy. This cohort is distinct from the four image editors on120 records per editor and three video generators on60 records per generator, which do not reach the six-model threshold and are not combined with the reasoning panel.","zh":"七个独立视觉语言模型共享1500条以交通参与者为中心的历史与未来行为审查样本，预测发布的行为标签程序。集合F1与标签宏F1保留原生0–100刻度，不作二元正确率。四种图像编辑器每种120条、三种视频生成器每种60条属于另外任务，均不足六模型，不与推理面板拼接。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.30722v1","published_at":"2026-09-25","status":"paper_baselines"},"score_min":0,"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Native score / 100","zh":"原生分数 / 100"},"score_decimals":2,"model_count":7,"result_count":7,"results":[[6389,64.8,370,12288,12289],[2697,69.7,370,12288,12289],[6391,75.4,370,12288,12289],[2055,77.9,370,12288,12289],[323,80.1,370,12288,12289],[353,79.2,370,12288,12289],[2602,83.6,370,12288,12289]]},{"id":"trafficimag-reasoning-macro-f1-2026-paper","name":"TrafficImag ·behavior macro-f1 (2026 paper)","group":"视觉与视频","metric":"Behavior label macro F1","unit":"score","public_history":true,"protocol":{"en":"Seven distinct video/image-language models share the1500 actor-centered history/future audit samples. The behavior reasoner predicts the released behavior-label program; set F1 and label-macro F1 remain native0–100, not binary accuracy. This cohort is distinct from the four image editors on120 records per editor and three video generators on60 records per generator, which do not reach the six-model threshold and are not combined with the reasoning panel.","zh":"七个独立视觉语言模型共享1500条以交通参与者为中心的历史与未来行为审查样本，预测发布的行为标签程序。集合F1与标签宏F1保留原生0–100刻度，不作二元正确率。四种图像编辑器每种120条、三种视频生成器每种60条属于另外任务，均不足六模型，不与推理面板拼接。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.30722v1","published_at":"2026-09-25","status":"paper_baselines"},"score_min":0,"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Native score / 100","zh":"原生分数 / 100"},"score_decimals":2,"model_count":7,"result_count":7,"results":[[6389,59.6,370,12288,12289],[2697,64.3,370,12288,12289],[6391,70.8,370,12288,12289],[2055,73.6,370,12288,12289],[323,76.2,370,12288,12289],[353,74.8,370,12288,12289],[2602,80.4,370,12288,12289]]},{"id":"trace-qa-2026-paper","name":"TRACE ·causal QA (2026 paper)","group":"视觉与视频","metric":"Scheduled-record answer accuracy","unit":"%","public_history":true,"protocol":{"en":"Six declared model+Adapter paths receive the same833 QA records with only evidence available by the question-arrival video time. Accuracy uses all scheduled records: failures,timeouts and ambiguous output options receive zero. State organization is part of the reported configuration, including AURA prefix reconstruction, native frame queues and persistent streaming models. The separate JoyAI complete-system boundary and MiniCPM-O spoken-duplex diagnostic are excluded. No runtime/compute-normalized capability claim is made; the source preserves completion and invalid-output rates.","zh":"六种明确的模型加Adapter路径接收相同833条QA记录，只能读取提问视频时间之前的证据。正确率以全部预定记录为分母，失败、超时和无法唯一解析的选项均记零。AURA前缀重建、原生帧队列及持久流状态均属于记录的配置；排除JoyAI完整系统和MiniCPM-O口语双工诊断。完成率和格式无效率保留在原表，不作跨硬件效率排名。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.30670v1","published_at":"2026-09-25","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[861,0.6518999999999999,370,12290,12291],[12292,0.6507,370,12293,12291],[6577,0.7503,370,12293,12291],[12294,0.6146,370,12295,12291],[12296,0.0264,370,12297,12291],[6571,0.7383,370,12298,12291]]},{"id":"trace-proactive-2026-paper","name":"TRACE ·in-window response (2026 paper)","group":"视觉与视频","metric":"Strict-window semantic partial-credit score","unit":"score","public_history":true,"protocol":{"en":"The same six autonomous model+Adapter configurations monitor407 standard1FPS proactive records containing1270 valid target windows. First-token onset assigns responses to strict windows; Qwen3.5-35B-A3B with the frozen osb-vlm-judge-v1 prompt scores semantic agreement against reference text without video, retaining0–1 partial credit. Reported percentages are therefore stored as a native0–100 semantic score. Unanswered windows receive no content credit. JoyAI system-boundary and native-duplex diagnostic rows are separate and excluded; conditional onset latency is not imported as if it used the full population.","zh":"相同六个自主模型加Adapter配置监控407条标准每秒一帧记录，共1270个目标窗口。按首词元起始时间分配严格窗口，Qwen3.5-35B-A3B使用固定osb-vlm-judge-v1提示，对参考文本语义作含部分分的0–1评审，不读取视频。因此原百分制数值保留为0–100语义评分。未回答窗口不得内容分。排除完整系统及原生双工诊断，也未将只覆盖已回答窗口的延迟冒充全量结果。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.30670v1","published_at":"2026-09-25","status":"paper_baselines"},"score_min":0,"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Native score / 100","zh":"原生分数 / 100"},"score_decimals":2,"model_count":6,"result_count":6,"results":[[861,12.98,370,12299,12300],[6571,7.92,370,12301,12300],[6577,8.05,370,12302,12300],[12292,4.45,370,12302,12300],[12294,0.52,370,12303,12300],[12296,0.18,370,12304,12300]]},{"id":"pfarena-t1-nms40-2026-paper","name":"PFArena ·T1 nms40 (2026 paper)","group":"数学与科学","metric":"nms40","unit":"score","public_history":true,"protocol":{"en":"This fixed T1 panel contains 123 assay-level queries. Four independent protein-model baselines (ProSST-2048,S3F,ESM-2,ProGen2-base) plus only general-purpose LLMs with zero missing-query substitutions form a common complete panel. VenusREM andS3F-MSA configurations and all agents are omitted conservatively. T1 proposes40 single mutants; T2 ranks without measurements; T3 receives an anchor; T4 receives single-mutant context. Ranking tasks use the original top5 budget. Metrics are equally averaged across assays. Recall retains its original fractional scale before percentage conversion; NMS andNDCG remain native scores, and Spearman spans minus1 to1. Appendix10.1 explicitly identifies missing-query SHA256 random replacements: no row with any replacement is included.","zh":"固定T1面板含123个实验测定查询，保留ProSST-2048、S3F、ESM-2、ProGen2-base四种独立蛋白模型，以及本任务完全没有缺失查询替代的通用模型，组成共同完整面板。保守排除VenusREM、S3F-MSA配置和全部Agent。T1提出40个单突变，T2无测量排序，T3提供锚点，T4提供单突变上下文；排序任务沿用前五预算。指标对实验测定等权；Recall原始小数转成百分率，NMS和NDCG保留原生分数，Spearman范围负一至一。附录10.1列出的任何SHA256随机基线回填行均排除。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.28921v1","published_at":"2026-09-24","status":"paper_baselines"},"score_min":0,"score_max":1,"saturation_excluded":true,"scale_label":{"en":"Native score / 1","zh":"原生分数 / 1"},"score_decimals":4,"model_count":7,"result_count":7,"results":[[12305,0.7847,370,12306,12307],[12308,0.7804,370,12306,12307],[12309,0.7854,370,12306,12307],[12310,0.7709,370,12306,12307],[3412,0.762,370,12306,12307],[227,0.7734,370,12306,12307],[3185,0.7703,370,12306,12307]]},{"id":"pfarena-t1-recall40-2026-paper","name":"PFArena ·T1 recall40 (2026 paper)","group":"数学与科学","metric":"recall40","unit":"%","public_history":true,"protocol":{"en":"This fixed T1 panel contains 123 assay-level queries. Four independent protein-model baselines (ProSST-2048,S3F,ESM-2,ProGen2-base) plus only general-purpose LLMs with zero missing-query substitutions form a common complete panel. VenusREM andS3F-MSA configurations and all agents are omitted conservatively. T1 proposes40 single mutants; T2 ranks without measurements; T3 receives an anchor; T4 receives single-mutant context. Ranking tasks use the original top5 budget. Metrics are equally averaged across assays. Recall retains its original fractional scale before percentage conversion; NMS andNDCG remain native scores, and Spearman spans minus1 to1. Appendix10.1 explicitly identifies missing-query SHA256 random replacements: no row with any replacement is included.","zh":"固定T1面板含123个实验测定查询，保留ProSST-2048、S3F、ESM-2、ProGen2-base四种独立蛋白模型，以及本任务完全没有缺失查询替代的通用模型，组成共同完整面板。保守排除VenusREM、S3F-MSA配置和全部Agent。T1提出40个单突变，T2无测量排序，T3提供锚点，T4提供单突变上下文；排序任务沿用前五预算。指标对实验测定等权；Recall原始小数转成百分率，NMS和NDCG保留原生分数，Spearman范围负一至一。附录10.1列出的任何SHA256随机基线回填行均排除。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.28921v1","published_at":"2026-09-24","status":"paper_baselines"},"model_count":7,"result_count":7,"results":[[12305,0.0407,370,12306,12307],[12308,0.0364,370,12306,12307],[12309,0.037,370,12306,12307],[12310,0.0274,370,12306,12307],[3412,0.0211,370,12306,12307],[227,0.0185,370,12306,12307],[3185,0.0185,370,12306,12307]]},{"id":"pfarena-t2-spearman-2026-paper","name":"PFArena ·T2 spearman (2026 paper)","group":"数学与科学","metric":"spearman","unit":"score","public_history":true,"protocol":{"en":"This fixed T2 panel contains 74 assay-level queries. Four independent protein-model baselines (ProSST-2048,S3F,ESM-2,ProGen2-base) plus only general-purpose LLMs with zero missing-query substitutions form a common complete panel. VenusREM andS3F-MSA configurations and all agents are omitted conservatively. T1 proposes40 single mutants; T2 ranks without measurements; T3 receives an anchor; T4 receives single-mutant context. Ranking tasks use the original top5 budget. Metrics are equally averaged across assays. Recall retains its original fractional scale before percentage conversion; NMS andNDCG remain native scores, and Spearman spans minus1 to1. Appendix10.1 explicitly identifies missing-query SHA256 random replacements: no row with any replacement is included.","zh":"固定T2面板含74个实验测定查询，保留ProSST-2048、S3F、ESM-2、ProGen2-base四种独立蛋白模型，以及本任务完全没有缺失查询替代的通用模型，组成共同完整面板。保守排除VenusREM、S3F-MSA配置和全部Agent。T1提出40个单突变，T2无测量排序，T3提供锚点，T4提供单突变上下文；排序任务沿用前五预算。指标对实验测定等权；Recall原始小数转成百分率，NMS和NDCG保留原生分数，Spearman范围负一至一。附录10.1列出的任何SHA256随机基线回填行均排除。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.28921v1","published_at":"2026-09-24","status":"paper_baselines"},"score_min":-1,"score_max":1,"saturation_excluded":true,"scale_label":{"en":"Native score / 1","zh":"原生分数 / 1"},"score_decimals":4,"model_count":7,"result_count":7,"results":[[12305,0.2777,370,12311,12312],[12308,0.2785,370,12311,12312],[12309,0.2539,370,12311,12312],[12310,0.1405,370,12311,12312],[2958,0.2726,370,12311,12312],[3412,0.2239,370,12311,12312],[3185,0.2187,370,12311,12312]]},{"id":"pfarena-t2-ndcg-2026-paper","name":"PFArena ·T2 ndcg (2026 paper)","group":"数学与科学","metric":"ndcg","unit":"score","public_history":true,"protocol":{"en":"This fixed T2 panel contains 74 assay-level queries. Four independent protein-model baselines (ProSST-2048,S3F,ESM-2,ProGen2-base) plus only general-purpose LLMs with zero missing-query substitutions form a common complete panel. VenusREM andS3F-MSA configurations and all agents are omitted conservatively. T1 proposes40 single mutants; T2 ranks without measurements; T3 receives an anchor; T4 receives single-mutant context. Ranking tasks use the original top5 budget. Metrics are equally averaged across assays. Recall retains its original fractional scale before percentage conversion; NMS andNDCG remain native scores, and Spearman spans minus1 to1. Appendix10.1 explicitly identifies missing-query SHA256 random replacements: no row with any replacement is included.","zh":"固定T2面板含74个实验测定查询，保留ProSST-2048、S3F、ESM-2、ProGen2-base四种独立蛋白模型，以及本任务完全没有缺失查询替代的通用模型，组成共同完整面板。保守排除VenusREM、S3F-MSA配置和全部Agent。T1提出40个单突变，T2无测量排序，T3提供锚点，T4提供单突变上下文；排序任务沿用前五预算。指标对实验测定等权；Recall原始小数转成百分率，NMS和NDCG保留原生分数，Spearman范围负一至一。附录10.1列出的任何SHA256随机基线回填行均排除。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.28921v1","published_at":"2026-09-24","status":"paper_baselines"},"score_min":0,"score_max":1,"saturation_excluded":true,"scale_label":{"en":"Native score / 1","zh":"原生分数 / 1"},"score_decimals":4,"model_count":7,"result_count":7,"results":[[12305,0.8657,370,12311,12312],[12308,0.8626,370,12311,12312],[12309,0.8634,370,12311,12312],[12310,0.8563,370,12311,12312],[2958,0.8635,370,12311,12312],[3412,0.8595,370,12311,12312],[3185,0.8587,370,12311,12312]]},{"id":"pfarena-t2-nms5-2026-paper","name":"PFArena ·T2 nms5 (2026 paper)","group":"数学与科学","metric":"nms5","unit":"score","public_history":true,"protocol":{"en":"This fixed T2 panel contains 74 assay-level queries. Four independent protein-model baselines (ProSST-2048,S3F,ESM-2,ProGen2-base) plus only general-purpose LLMs with zero missing-query substitutions form a common complete panel. VenusREM andS3F-MSA configurations and all agents are omitted conservatively. T1 proposes40 single mutants; T2 ranks without measurements; T3 receives an anchor; T4 receives single-mutant context. Ranking tasks use the original top5 budget. Metrics are equally averaged across assays. Recall retains its original fractional scale before percentage conversion; NMS andNDCG remain native scores, and Spearman spans minus1 to1. Appendix10.1 explicitly identifies missing-query SHA256 random replacements: no row with any replacement is included.","zh":"固定T2面板含74个实验测定查询，保留ProSST-2048、S3F、ESM-2、ProGen2-base四种独立蛋白模型，以及本任务完全没有缺失查询替代的通用模型，组成共同完整面板。保守排除VenusREM、S3F-MSA配置和全部Agent。T1提出40个单突变，T2无测量排序，T3提供锚点，T4提供单突变上下文；排序任务沿用前五预算。指标对实验测定等权；Recall原始小数转成百分率，NMS和NDCG保留原生分数，Spearman范围负一至一。附录10.1列出的任何SHA256随机基线回填行均排除。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.28921v1","published_at":"2026-09-24","status":"paper_baselines"},"score_min":0,"score_max":1,"saturation_excluded":true,"scale_label":{"en":"Native score / 1","zh":"原生分数 / 1"},"score_decimals":4,"model_count":7,"result_count":7,"results":[[12305,0.7829,370,12311,12312],[12308,0.7751,370,12311,12312],[12309,0.7831,370,12311,12312],[12310,0.7887,370,12311,12312],[2958,0.7967,370,12311,12312],[3412,0.7723,370,12311,12312],[3185,0.776,370,12311,12312]]},{"id":"pfarena-t2-recall5-2026-paper","name":"PFArena ·T2 recall5 (2026 paper)","group":"数学与科学","metric":"recall5","unit":"%","public_history":true,"protocol":{"en":"This fixed T2 panel contains 74 assay-level queries. Four independent protein-model baselines (ProSST-2048,S3F,ESM-2,ProGen2-base) plus only general-purpose LLMs with zero missing-query substitutions form a common complete panel. VenusREM andS3F-MSA configurations and all agents are omitted conservatively. T1 proposes40 single mutants; T2 ranks without measurements; T3 receives an anchor; T4 receives single-mutant context. Ranking tasks use the original top5 budget. Metrics are equally averaged across assays. Recall retains its original fractional scale before percentage conversion; NMS andNDCG remain native scores, and Spearman spans minus1 to1. Appendix10.1 explicitly identifies missing-query SHA256 random replacements: no row with any replacement is included.","zh":"固定T2面板含74个实验测定查询，保留ProSST-2048、S3F、ESM-2、ProGen2-base四种独立蛋白模型，以及本任务完全没有缺失查询替代的通用模型，组成共同完整面板。保守排除VenusREM、S3F-MSA配置和全部Agent。T1提出40个单突变，T2无测量排序，T3提供锚点，T4提供单突变上下文；排序任务沿用前五预算。指标对实验测定等权；Recall原始小数转成百分率，NMS和NDCG保留原生分数，Spearman范围负一至一。附录10.1列出的任何SHA256随机基线回填行均排除。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.28921v1","published_at":"2026-09-24","status":"paper_baselines"},"model_count":7,"result_count":7,"results":[[12305,0.2108,370,12311,12312],[12308,0.1757,370,12311,12312],[12309,0.1892,370,12311,12312],[12310,0.1784,370,12311,12312],[2958,0.2216,370,12311,12312],[3412,0.1973,370,12311,12312],[3185,0.2135,370,12311,12312]]},{"id":"pfarena-t3-spearman-2026-paper","name":"PFArena ·T3 spearman (2026 paper)","group":"数学与科学","metric":"spearman","unit":"score","public_history":true,"protocol":{"en":"This fixed T3 panel contains 67 assay-level queries. Four independent protein-model baselines (ProSST-2048,S3F,ESM-2,ProGen2-base) plus only general-purpose LLMs with zero missing-query substitutions form a common complete panel. VenusREM andS3F-MSA configurations and all agents are omitted conservatively. T1 proposes40 single mutants; T2 ranks without measurements; T3 receives an anchor; T4 receives single-mutant context. Ranking tasks use the original top5 budget. Metrics are equally averaged across assays. Recall retains its original fractional scale before percentage conversion; NMS andNDCG remain native scores, and Spearman spans minus1 to1. Appendix10.1 explicitly identifies missing-query SHA256 random replacements: no row with any replacement is included.","zh":"固定T3面板含67个实验测定查询，保留ProSST-2048、S3F、ESM-2、ProGen2-base四种独立蛋白模型，以及本任务完全没有缺失查询替代的通用模型，组成共同完整面板。保守排除VenusREM、S3F-MSA配置和全部Agent。T1提出40个单突变，T2无测量排序，T3提供锚点，T4提供单突变上下文；排序任务沿用前五预算。指标对实验测定等权；Recall原始小数转成百分率，NMS和NDCG保留原生分数，Spearman范围负一至一。附录10.1列出的任何SHA256随机基线回填行均排除。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.28921v1","published_at":"2026-09-24","status":"paper_baselines"},"score_min":-1,"score_max":1,"saturation_excluded":true,"scale_label":{"en":"Native score / 1","zh":"原生分数 / 1"},"score_decimals":4,"model_count":6,"result_count":6,"results":[[12305,0.3836,370,12313,12312],[12308,0.3357,370,12313,12312],[12309,0.3128,370,12313,12312],[12310,0.187,370,12313,12312],[2958,0.3516,370,12313,12312],[3412,0.2865,370,12313,12312]]},{"id":"pfarena-t3-ndcg-2026-paper","name":"PFArena ·T3 ndcg (2026 paper)","group":"数学与科学","metric":"ndcg","unit":"score","public_history":true,"protocol":{"en":"This fixed T3 panel contains 67 assay-level queries. Four independent protein-model baselines (ProSST-2048,S3F,ESM-2,ProGen2-base) plus only general-purpose LLMs with zero missing-query substitutions form a common complete panel. VenusREM andS3F-MSA configurations and all agents are omitted conservatively. T1 proposes40 single mutants; T2 ranks without measurements; T3 receives an anchor; T4 receives single-mutant context. Ranking tasks use the original top5 budget. Metrics are equally averaged across assays. Recall retains its original fractional scale before percentage conversion; NMS andNDCG remain native scores, and Spearman spans minus1 to1. Appendix10.1 explicitly identifies missing-query SHA256 random replacements: no row with any replacement is included.","zh":"固定T3面板含67个实验测定查询，保留ProSST-2048、S3F、ESM-2、ProGen2-base四种独立蛋白模型，以及本任务完全没有缺失查询替代的通用模型，组成共同完整面板。保守排除VenusREM、S3F-MSA配置和全部Agent。T1提出40个单突变，T2无测量排序，T3提供锚点，T4提供单突变上下文；排序任务沿用前五预算。指标对实验测定等权；Recall原始小数转成百分率，NMS和NDCG保留原生分数，Spearman范围负一至一。附录10.1列出的任何SHA256随机基线回填行均排除。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.28921v1","published_at":"2026-09-24","status":"paper_baselines"},"score_min":0,"score_max":1,"saturation_excluded":true,"scale_label":{"en":"Native score / 1","zh":"原生分数 / 1"},"score_decimals":4,"model_count":6,"result_count":6,"results":[[12305,0.8903,370,12313,12312],[12308,0.8842,370,12313,12312],[12309,0.884,370,12313,12312],[12310,0.8606,370,12313,12312],[2958,0.8886,370,12313,12312],[3412,0.8814,370,12313,12312]]},{"id":"pfarena-t3-nms5-2026-paper","name":"PFArena ·T3 nms5 (2026 paper)","group":"数学与科学","metric":"nms5","unit":"score","public_history":true,"protocol":{"en":"This fixed T3 panel contains 67 assay-level queries. Four independent protein-model baselines (ProSST-2048,S3F,ESM-2,ProGen2-base) plus only general-purpose LLMs with zero missing-query substitutions form a common complete panel. VenusREM andS3F-MSA configurations and all agents are omitted conservatively. T1 proposes40 single mutants; T2 ranks without measurements; T3 receives an anchor; T4 receives single-mutant context. Ranking tasks use the original top5 budget. Metrics are equally averaged across assays. Recall retains its original fractional scale before percentage conversion; NMS andNDCG remain native scores, and Spearman spans minus1 to1. Appendix10.1 explicitly identifies missing-query SHA256 random replacements: no row with any replacement is included.","zh":"固定T3面板含67个实验测定查询，保留ProSST-2048、S3F、ESM-2、ProGen2-base四种独立蛋白模型，以及本任务完全没有缺失查询替代的通用模型，组成共同完整面板。保守排除VenusREM、S3F-MSA配置和全部Agent。T1提出40个单突变，T2无测量排序，T3提供锚点，T4提供单突变上下文；排序任务沿用前五预算。指标对实验测定等权；Recall原始小数转成百分率，NMS和NDCG保留原生分数，Spearman范围负一至一。附录10.1列出的任何SHA256随机基线回填行均排除。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.28921v1","published_at":"2026-09-24","status":"paper_baselines"},"score_min":0,"score_max":1,"saturation_excluded":true,"scale_label":{"en":"Native score / 1","zh":"原生分数 / 1"},"score_decimals":4,"model_count":6,"result_count":6,"results":[[12305,0.862,370,12313,12312],[12308,0.8523,370,12313,12312],[12309,0.8099,370,12313,12312],[12310,0.7835,370,12313,12312],[2958,0.8258,370,12313,12312],[3412,0.8019,370,12313,12312]]},{"id":"pfarena-t3-recall5-2026-paper","name":"PFArena ·T3 recall5 (2026 paper)","group":"数学与科学","metric":"recall5","unit":"%","public_history":true,"protocol":{"en":"This fixed T3 panel contains 67 assay-level queries. Four independent protein-model baselines (ProSST-2048,S3F,ESM-2,ProGen2-base) plus only general-purpose LLMs with zero missing-query substitutions form a common complete panel. VenusREM andS3F-MSA configurations and all agents are omitted conservatively. T1 proposes40 single mutants; T2 ranks without measurements; T3 receives an anchor; T4 receives single-mutant context. Ranking tasks use the original top5 budget. Metrics are equally averaged across assays. Recall retains its original fractional scale before percentage conversion; NMS andNDCG remain native scores, and Spearman spans minus1 to1. Appendix10.1 explicitly identifies missing-query SHA256 random replacements: no row with any replacement is included.","zh":"固定T3面板含67个实验测定查询，保留ProSST-2048、S3F、ESM-2、ProGen2-base四种独立蛋白模型，以及本任务完全没有缺失查询替代的通用模型，组成共同完整面板。保守排除VenusREM、S3F-MSA配置和全部Agent。T1提出40个单突变，T2无测量排序，T3提供锚点，T4提供单突变上下文；排序任务沿用前五预算。指标对实验测定等权；Recall原始小数转成百分率，NMS和NDCG保留原生分数，Spearman范围负一至一。附录10.1列出的任何SHA256随机基线回填行均排除。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.28921v1","published_at":"2026-09-24","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[12305,0.3672,370,12313,12312],[12308,0.2955,370,12313,12312],[12309,0.3045,370,12313,12312],[12310,0.2716,370,12313,12312],[2958,0.3134,370,12313,12312],[3412,0.2806,370,12313,12312]]},{"id":"pfarena-t4-spearman-2026-paper","name":"PFArena ·T4 spearman (2026 paper)","group":"数学与科学","metric":"spearman","unit":"score","public_history":true,"protocol":{"en":"This fixed T4 panel contains 29 assay-level queries. Four independent protein-model baselines (ProSST-2048,S3F,ESM-2,ProGen2-base) plus only general-purpose LLMs with zero missing-query substitutions form a common complete panel. VenusREM andS3F-MSA configurations and all agents are omitted conservatively. T1 proposes40 single mutants; T2 ranks without measurements; T3 receives an anchor; T4 receives single-mutant context. Ranking tasks use the original top5 budget. Metrics are equally averaged across assays. Recall retains its original fractional scale before percentage conversion; NMS andNDCG remain native scores, and Spearman spans minus1 to1. Appendix10.1 explicitly identifies missing-query SHA256 random replacements: no row with any replacement is included.","zh":"固定T4面板含29个实验测定查询，保留ProSST-2048、S3F、ESM-2、ProGen2-base四种独立蛋白模型，以及本任务完全没有缺失查询替代的通用模型，组成共同完整面板。保守排除VenusREM、S3F-MSA配置和全部Agent。T1提出40个单突变，T2无测量排序，T3提供锚点，T4提供单突变上下文；排序任务沿用前五预算。指标对实验测定等权；Recall原始小数转成百分率，NMS和NDCG保留原生分数，Spearman范围负一至一。附录10.1列出的任何SHA256随机基线回填行均排除。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.28921v1","published_at":"2026-09-24","status":"paper_baselines"},"score_min":-1,"score_max":1,"saturation_excluded":true,"scale_label":{"en":"Native score / 1","zh":"原生分数 / 1"},"score_decimals":4,"model_count":8,"result_count":8,"results":[[12305,0.3779,370,12314,12315],[12308,0.3487,370,12314,12315],[12309,0.3153,370,12314,12315],[12310,0.2568,370,12314,12315],[2958,0.5988,370,12314,12315],[3412,0.5329,370,12314,12315],[227,0.4959,370,12314,12315],[3185,0.5308,370,12314,12315]]},{"id":"pfarena-t4-ndcg-2026-paper","name":"PFArena ·T4 ndcg (2026 paper)","group":"数学与科学","metric":"ndcg","unit":"score","public_history":true,"protocol":{"en":"This fixed T4 panel contains 29 assay-level queries. Four independent protein-model baselines (ProSST-2048,S3F,ESM-2,ProGen2-base) plus only general-purpose LLMs with zero missing-query substitutions form a common complete panel. VenusREM andS3F-MSA configurations and all agents are omitted conservatively. T1 proposes40 single mutants; T2 ranks without measurements; T3 receives an anchor; T4 receives single-mutant context. Ranking tasks use the original top5 budget. Metrics are equally averaged across assays. Recall retains its original fractional scale before percentage conversion; NMS andNDCG remain native scores, and Spearman spans minus1 to1. Appendix10.1 explicitly identifies missing-query SHA256 random replacements: no row with any replacement is included.","zh":"固定T4面板含29个实验测定查询，保留ProSST-2048、S3F、ESM-2、ProGen2-base四种独立蛋白模型，以及本任务完全没有缺失查询替代的通用模型，组成共同完整面板。保守排除VenusREM、S3F-MSA配置和全部Agent。T1提出40个单突变，T2无测量排序，T3提供锚点，T4提供单突变上下文；排序任务沿用前五预算。指标对实验测定等权；Recall原始小数转成百分率，NMS和NDCG保留原生分数，Spearman范围负一至一。附录10.1列出的任何SHA256随机基线回填行均排除。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.28921v1","published_at":"2026-09-24","status":"paper_baselines"},"score_min":0,"score_max":1,"saturation_excluded":true,"scale_label":{"en":"Native score / 1","zh":"原生分数 / 1"},"score_decimals":4,"model_count":8,"result_count":8,"results":[[12305,0.8725,370,12314,12315],[12308,0.8836,370,12314,12315],[12309,0.8828,370,12314,12315],[12310,0.8736,370,12314,12315],[2958,0.9274,370,12314,12315],[3412,0.9059,370,12314,12315],[227,0.8984,370,12314,12315],[3185,0.9059,370,12314,12315]]},{"id":"pfarena-t4-nms5-2026-paper","name":"PFArena ·T4 nms5 (2026 paper)","group":"数学与科学","metric":"nms5","unit":"score","public_history":true,"protocol":{"en":"This fixed T4 panel contains 29 assay-level queries. Four independent protein-model baselines (ProSST-2048,S3F,ESM-2,ProGen2-base) plus only general-purpose LLMs with zero missing-query substitutions form a common complete panel. VenusREM andS3F-MSA configurations and all agents are omitted conservatively. T1 proposes40 single mutants; T2 ranks without measurements; T3 receives an anchor; T4 receives single-mutant context. Ranking tasks use the original top5 budget. Metrics are equally averaged across assays. Recall retains its original fractional scale before percentage conversion; NMS andNDCG remain native scores, and Spearman spans minus1 to1. Appendix10.1 explicitly identifies missing-query SHA256 random replacements: no row with any replacement is included.","zh":"固定T4面板含29个实验测定查询，保留ProSST-2048、S3F、ESM-2、ProGen2-base四种独立蛋白模型，以及本任务完全没有缺失查询替代的通用模型，组成共同完整面板。保守排除VenusREM、S3F-MSA配置和全部Agent。T1提出40个单突变，T2无测量排序，T3提供锚点，T4提供单突变上下文；排序任务沿用前五预算。指标对实验测定等权；Recall原始小数转成百分率，NMS和NDCG保留原生分数，Spearman范围负一至一。附录10.1列出的任何SHA256随机基线回填行均排除。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.28921v1","published_at":"2026-09-24","status":"paper_baselines"},"score_min":0,"score_max":1,"saturation_excluded":true,"scale_label":{"en":"Native score / 1","zh":"原生分数 / 1"},"score_decimals":4,"model_count":8,"result_count":8,"results":[[12305,0.8078,370,12314,12315],[12308,0.841,370,12314,12315],[12309,0.8129,370,12314,12315],[12310,0.8357,370,12314,12315],[2958,0.8935,370,12314,12315],[3412,0.8595,370,12314,12315],[227,0.8338,370,12314,12315],[3185,0.8595,370,12314,12315]]},{"id":"pfarena-t4-recall5-2026-paper","name":"PFArena ·T4 recall5 (2026 paper)","group":"数学与科学","metric":"recall5","unit":"%","public_history":true,"protocol":{"en":"This fixed T4 panel contains 29 assay-level queries. Four independent protein-model baselines (ProSST-2048,S3F,ESM-2,ProGen2-base) plus only general-purpose LLMs with zero missing-query substitutions form a common complete panel. VenusREM andS3F-MSA configurations and all agents are omitted conservatively. T1 proposes40 single mutants; T2 ranks without measurements; T3 receives an anchor; T4 receives single-mutant context. Ranking tasks use the original top5 budget. Metrics are equally averaged across assays. Recall retains its original fractional scale before percentage conversion; NMS andNDCG remain native scores, and Spearman spans minus1 to1. Appendix10.1 explicitly identifies missing-query SHA256 random replacements: no row with any replacement is included.","zh":"固定T4面板含29个实验测定查询，保留ProSST-2048、S3F、ESM-2、ProGen2-base四种独立蛋白模型，以及本任务完全没有缺失查询替代的通用模型，组成共同完整面板。保守排除VenusREM、S3F-MSA配置和全部Agent。T1提出40个单突变，T2无测量排序，T3提供锚点，T4提供单突变上下文；排序任务沿用前五预算。指标对实验测定等权；Recall原始小数转成百分率，NMS和NDCG保留原生分数，Spearman范围负一至一。附录10.1列出的任何SHA256随机基线回填行均排除。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.28921v1","published_at":"2026-09-24","status":"paper_baselines"},"model_count":8,"result_count":8,"results":[[12305,0.2483,370,12314,12315],[12308,0.2759,370,12314,12315],[12309,0.2276,370,12314,12315],[12310,0.1793,370,12314,12315],[2958,0.4483,370,12314,12315],[3412,0.4,370,12314,12315],[227,0.3862,370,12314,12315],[3185,0.4,370,12314,12315]]},{"id":"sage-auroc-overall-2026-paper","name":"SAGE ·AUROC overall (2026 paper)","group":"数学与科学","metric":"AUROC","unit":"score","public_history":true,"protocol":{"en":"Ten model/configuration rows cover seven independent architectures: GLM,BRT,GAM,MaxEnt,RF,MLP andResNet. The overall cohort contains 5771 species; each metric is computed only on native-range cells from the same42268-cell sPlotOpen test split (11068 validation cells), then aggregated across species. At least20 test presences per species are enforced. Five random seeds share the locked split. The source reports AUROC mean andAUPRG median with seed standard deviations; scores retain native100-scale area-under-curve units and are excluded from percent-correct saturation. Incremental aggregation/subsampling/loss changes remain explicit configurations, not extra architectures. Species-specific models and joint models use their stated training recipes.","zh":"十条模型与配置结果覆盖GLM、BRT、GAM、MaxEnt、RF、MLP、ResNet七种独立架构。overall组含5771个物种；各物种只在原生分布范围内、相同42268个测试网格中计算指标，再跨物种汇总；另有11068个验证网格，每物种测试至少20次出现。五随机种子共享固定划分。原表AUROC报告均值，AUPRG报告中位数及种子标准差；均保留百分制曲线面积原生单位，不进入正确率饱和区间。聚合、抽样和损失修改记录为配置，不增加架构数；保留单物种与联合模型各自训练方法。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.31082v1","published_at":"2026-09-25","status":"paper_baselines"},"score_min":0,"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Native score / 100","zh":"原生分数 / 100"},"score_decimals":2,"model_count":7,"result_count":10,"results":[[12316,84.0,370,12317,12318],[12319,85.2,370,12320,12318],[12321,83.8,370,12322,12318],[12323,84.5,370,12324,12318],[12325,85.9,370,12326,12318],[12327,83.3,370,12328,12318],[12327,84.0,370,12329,12318],[12327,84.2,370,12330,12318],[12331,84.5,370,12332,12318],[12331,85.8,370,12333,12318]]},{"id":"sage-auroc-sparse-infrequent-2026-paper","name":"SAGE ·AUROC sparse-infrequent (2026 paper)","group":"数学与科学","metric":"AUROC","unit":"score","public_history":true,"protocol":{"en":"Ten model/configuration rows cover seven independent architectures: GLM,BRT,GAM,MaxEnt,RF,MLP andResNet. The sparse-infrequent cohort contains 1421 species; each metric is computed only on native-range cells from the same42268-cell sPlotOpen test split (11068 validation cells), then aggregated across species. At least20 test presences per species are enforced. Five random seeds share the locked split. The source reports AUROC mean andAUPRG median with seed standard deviations; scores retain native100-scale area-under-curve units and are excluded from percent-correct saturation. Incremental aggregation/subsampling/loss changes remain explicit configurations, not extra architectures. Species-specific models and joint models use their stated training recipes.","zh":"十条模型与配置结果覆盖GLM、BRT、GAM、MaxEnt、RF、MLP、ResNet七种独立架构。sparse-infrequent组含1421个物种；各物种只在原生分布范围内、相同42268个测试网格中计算指标，再跨物种汇总；另有11068个验证网格，每物种测试至少20次出现。五随机种子共享固定划分。原表AUROC报告均值，AUPRG报告中位数及种子标准差；均保留百分制曲线面积原生单位，不进入正确率饱和区间。聚合、抽样和损失修改记录为配置，不增加架构数；保留单物种与联合模型各自训练方法。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.31082v1","published_at":"2026-09-25","status":"paper_baselines"},"score_min":0,"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Native score / 100","zh":"原生分数 / 100"},"score_decimals":2,"model_count":7,"result_count":10,"results":[[12316,84.5,370,12317,12318],[12319,85.5,370,12320,12318],[12321,83.9,370,12322,12318],[12323,84.7,370,12324,12318],[12325,86.6,370,12326,12318],[12327,82.3,370,12328,12318],[12327,84.7,370,12329,12318],[12327,84.9,370,12330,12318],[12331,85.0,370,12332,12318],[12331,86.5,370,12333,12318]]},{"id":"sage-auroc-sparse-frequent-2026-paper","name":"SAGE ·AUROC sparse-frequent (2026 paper)","group":"数学与科学","metric":"AUROC","unit":"score","public_history":true,"protocol":{"en":"Ten model/configuration rows cover seven independent architectures: GLM,BRT,GAM,MaxEnt,RF,MLP andResNet. The sparse-frequent cohort contains 1464 species; each metric is computed only on native-range cells from the same42268-cell sPlotOpen test split (11068 validation cells), then aggregated across species. At least20 test presences per species are enforced. Five random seeds share the locked split. The source reports AUROC mean andAUPRG median with seed standard deviations; scores retain native100-scale area-under-curve units and are excluded from percent-correct saturation. Incremental aggregation/subsampling/loss changes remain explicit configurations, not extra architectures. Species-specific models and joint models use their stated training recipes.","zh":"十条模型与配置结果覆盖GLM、BRT、GAM、MaxEnt、RF、MLP、ResNet七种独立架构。sparse-frequent组含1464个物种；各物种只在原生分布范围内、相同42268个测试网格中计算指标，再跨物种汇总；另有11068个验证网格，每物种测试至少20次出现。五随机种子共享固定划分。原表AUROC报告均值，AUPRG报告中位数及种子标准差；均保留百分制曲线面积原生单位，不进入正确率饱和区间。聚合、抽样和损失修改记录为配置，不增加架构数；保留单物种与联合模型各自训练方法。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.31082v1","published_at":"2026-09-25","status":"paper_baselines"},"score_min":0,"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Native score / 100","zh":"原生分数 / 100"},"score_decimals":2,"model_count":7,"result_count":10,"results":[[12316,81.9,370,12317,12318],[12319,83.5,370,12320,12318],[12321,82.0,370,12322,12318],[12323,82.8,370,12324,12318],[12325,84.3,370,12326,12318],[12327,80.5,370,12328,12318],[12327,81.7,370,12329,12318],[12327,81.8,370,12330,12318],[12331,82.1,370,12332,12318],[12331,83.8,370,12333,12318]]},{"id":"sage-auroc-dense-infrequent-2026-paper","name":"SAGE ·AUROC dense-infrequent (2026 paper)","group":"数学与科学","metric":"AUROC","unit":"score","public_history":true,"protocol":{"en":"Ten model/configuration rows cover seven independent architectures: GLM,BRT,GAM,MaxEnt,RF,MLP andResNet. The dense-infrequent cohort contains 1464 species; each metric is computed only on native-range cells from the same42268-cell sPlotOpen test split (11068 validation cells), then aggregated across species. At least20 test presences per species are enforced. Five random seeds share the locked split. The source reports AUROC mean andAUPRG median with seed standard deviations; scores retain native100-scale area-under-curve units and are excluded from percent-correct saturation. Incremental aggregation/subsampling/loss changes remain explicit configurations, not extra architectures. Species-specific models and joint models use their stated training recipes.","zh":"十条模型与配置结果覆盖GLM、BRT、GAM、MaxEnt、RF、MLP、ResNet七种独立架构。dense-infrequent组含1464个物种；各物种只在原生分布范围内、相同42268个测试网格中计算指标，再跨物种汇总；另有11068个验证网格，每物种测试至少20次出现。五随机种子共享固定划分。原表AUROC报告均值，AUPRG报告中位数及种子标准差；均保留百分制曲线面积原生单位，不进入正确率饱和区间。聚合、抽样和损失修改记录为配置，不增加架构数；保留单物种与联合模型各自训练方法。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.31082v1","published_at":"2026-09-25","status":"paper_baselines"},"score_min":0,"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Native score / 100","zh":"原生分数 / 100"},"score_decimals":2,"model_count":7,"result_count":10,"results":[[12316,86.3,370,12317,12318],[12319,87.2,370,12320,12318],[12321,85.9,370,12322,12318],[12323,86.6,370,12324,12318],[12325,87.7,370,12326,12318],[12327,86.4,370,12328,12318],[12327,86.3,370,12329,12318],[12327,86.8,370,12330,12318],[12331,87.1,370,12332,12318],[12331,87.9,370,12333,12318]]},{"id":"sage-auroc-dense-frequent-2026-paper","name":"SAGE ·AUROC dense-frequent (2026 paper)","group":"数学与科学","metric":"AUROC","unit":"score","public_history":true,"protocol":{"en":"Ten model/configuration rows cover seven independent architectures: GLM,BRT,GAM,MaxEnt,RF,MLP andResNet. The dense-frequent cohort contains 1422 species; each metric is computed only on native-range cells from the same42268-cell sPlotOpen test split (11068 validation cells), then aggregated across species. At least20 test presences per species are enforced. Five random seeds share the locked split. The source reports AUROC mean andAUPRG median with seed standard deviations; scores retain native100-scale area-under-curve units and are excluded from percent-correct saturation. Incremental aggregation/subsampling/loss changes remain explicit configurations, not extra architectures. Species-specific models and joint models use their stated training recipes.","zh":"十条模型与配置结果覆盖GLM、BRT、GAM、MaxEnt、RF、MLP、ResNet七种独立架构。dense-frequent组含1422个物种；各物种只在原生分布范围内、相同42268个测试网格中计算指标，再跨物种汇总；另有11068个验证网格，每物种测试至少20次出现。五随机种子共享固定划分。原表AUROC报告均值，AUPRG报告中位数及种子标准差；均保留百分制曲线面积原生单位，不进入正确率饱和区间。聚合、抽样和损失修改记录为配置，不增加架构数；保留单物种与联合模型各自训练方法。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.31082v1","published_at":"2026-09-25","status":"paper_baselines"},"score_min":0,"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Native score / 100","zh":"原生分数 / 100"},"score_decimals":2,"model_count":7,"result_count":10,"results":[[12316,83.2,370,12317,12318],[12319,84.6,370,12320,12318],[12321,83.4,370,12322,12318],[12323,83.8,370,12324,12318],[12325,85.0,370,12326,12318],[12327,83.8,370,12328,12318],[12327,83.3,370,12329,12318],[12327,83.5,370,12330,12318],[12331,83.9,370,12332,12318],[12331,84.9,370,12333,12318]]},{"id":"sage-auprg-overall-2026-paper","name":"SAGE ·AUPRG overall (2026 paper)","group":"数学与科学","metric":"AUPRG","unit":"raw","public_history":true,"protocol":{"en":"Ten model/configuration rows cover seven independent architectures: GLM,BRT,GAM,MaxEnt,RF,MLP andResNet. The overall cohort contains 5771 species; each metric is computed only on native-range cells from the same42268-cell sPlotOpen test split (11068 validation cells), then aggregated across species. At least20 test presences per species are enforced. Five random seeds share the locked split. The source reports AUROC mean andAUPRG median with seed standard deviations; scores retain native100-scale area-under-curve units and are excluded from percent-correct saturation. Incremental aggregation/subsampling/loss changes remain explicit configurations, not extra architectures. Species-specific models and joint models use their stated training recipes.","zh":"十条模型与配置结果覆盖GLM、BRT、GAM、MaxEnt、RF、MLP、ResNet七种独立架构。overall组含5771个物种；各物种只在原生分布范围内、相同42268个测试网格中计算指标，再跨物种汇总；另有11068个验证网格，每物种测试至少20次出现。五随机种子共享固定划分。原表AUROC报告均值，AUPRG报告中位数及种子标准差；均保留百分制曲线面积原生单位，不进入正确率饱和区间。聚合、抽样和损失修改记录为配置，不增加架构数；保留单物种与联合模型各自训练方法。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.31082v1","published_at":"2026-09-25","status":"paper_baselines"},"saturation_excluded":true,"scale_label":{"en":"AUPRG","zh":"AUPRG"},"score_decimals":3,"model_count":9,"result_count":10,"results":[[12316,79.3,370,12317,12334],[12319,84.5,370,12320,12334],[12321,81.0,370,12322,12334],[12323,83.1,370,12324,12334],[10499,86.9,370,12335,12334],[12336,82.7,370,12337,12334],[12338,83.6,370,12339,12334],[12327,83.5,370,12330,12334],[12331,85.0,370,12332,12334],[12331,85.4,370,12333,12334]]},{"id":"sage-auprg-sparse-infrequent-2026-paper","name":"SAGE ·AUPRG sparse-infrequent (2026 paper)","group":"数学与科学","metric":"AUPRG","unit":"raw","public_history":true,"protocol":{"en":"Ten model/configuration rows cover seven independent architectures: GLM,BRT,GAM,MaxEnt,RF,MLP andResNet. The sparse-infrequent cohort contains 1421 species; each metric is computed only on native-range cells from the same42268-cell sPlotOpen test split (11068 validation cells), then aggregated across species. At least20 test presences per species are enforced. Five random seeds share the locked split. The source reports AUROC mean andAUPRG median with seed standard deviations; scores retain native100-scale area-under-curve units and are excluded from percent-correct saturation. Incremental aggregation/subsampling/loss changes remain explicit configurations, not extra architectures. Species-specific models and joint models use their stated training recipes.","zh":"十条模型与配置结果覆盖GLM、BRT、GAM、MaxEnt、RF、MLP、ResNet七种独立架构。sparse-infrequent组含1421个物种；各物种只在原生分布范围内、相同42268个测试网格中计算指标，再跨物种汇总；另有11068个验证网格，每物种测试至少20次出现。五随机种子共享固定划分。原表AUROC报告均值，AUPRG报告中位数及种子标准差；均保留百分制曲线面积原生单位，不进入正确率饱和区间。聚合、抽样和损失修改记录为配置，不增加架构数；保留单物种与联合模型各自训练方法。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.31082v1","published_at":"2026-09-25","status":"paper_baselines"},"saturation_excluded":true,"scale_label":{"en":"AUPRG","zh":"AUPRG"},"score_decimals":3,"model_count":9,"result_count":10,"results":[[12316,76.6,370,12317,12334],[12319,83.9,370,12320,12334],[12321,78.4,370,12322,12334],[12323,82.7,370,12324,12334],[10499,88.0,370,12335,12334],[12336,78.2,370,12337,12334],[12338,82.9,370,12339,12334],[12327,82.0,370,12330,12334],[12331,83.7,370,12332,12334],[12331,85.2,370,12333,12334]]},{"id":"sage-auprg-sparse-frequent-2026-paper","name":"SAGE ·AUPRG sparse-frequent (2026 paper)","group":"数学与科学","metric":"AUPRG","unit":"raw","public_history":true,"protocol":{"en":"Ten model/configuration rows cover seven independent architectures: GLM,BRT,GAM,MaxEnt,RF,MLP andResNet. The sparse-frequent cohort contains 1464 species; each metric is computed only on native-range cells from the same42268-cell sPlotOpen test split (11068 validation cells), then aggregated across species. At least20 test presences per species are enforced. Five random seeds share the locked split. The source reports AUROC mean andAUPRG median with seed standard deviations; scores retain native100-scale area-under-curve units and are excluded from percent-correct saturation. Incremental aggregation/subsampling/loss changes remain explicit configurations, not extra architectures. Species-specific models and joint models use their stated training recipes.","zh":"十条模型与配置结果覆盖GLM、BRT、GAM、MaxEnt、RF、MLP、ResNet七种独立架构。sparse-frequent组含1464个物种；各物种只在原生分布范围内、相同42268个测试网格中计算指标，再跨物种汇总；另有11068个验证网格，每物种测试至少20次出现。五随机种子共享固定划分。原表AUROC报告均值，AUPRG报告中位数及种子标准差；均保留百分制曲线面积原生单位，不进入正确率饱和区间。聚合、抽样和损失修改记录为配置，不增加架构数；保留单物种与联合模型各自训练方法。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.31082v1","published_at":"2026-09-25","status":"paper_baselines"},"saturation_excluded":true,"scale_label":{"en":"AUPRG","zh":"AUPRG"},"score_decimals":3,"model_count":9,"result_count":10,"results":[[12316,77.6,370,12317,12334],[12319,82.1,370,12320,12334],[12321,78.8,370,12322,12334],[12323,80.6,370,12324,12334],[10499,84.9,370,12335,12334],[12336,77.9,370,12337,12334],[12338,79.1,370,12339,12334],[12327,79.3,370,12330,12334],[12331,80.8,370,12332,12334],[12331,81.7,370,12333,12334]]},{"id":"sage-auprg-dense-infrequent-2026-paper","name":"SAGE ·AUPRG dense-infrequent (2026 paper)","group":"数学与科学","metric":"AUPRG","unit":"raw","public_history":true,"protocol":{"en":"Ten model/configuration rows cover seven independent architectures: GLM,BRT,GAM,MaxEnt,RF,MLP andResNet. The dense-infrequent cohort contains 1464 species; each metric is computed only on native-range cells from the same42268-cell sPlotOpen test split (11068 validation cells), then aggregated across species. At least20 test presences per species are enforced. Five random seeds share the locked split. The source reports AUROC mean andAUPRG median with seed standard deviations; scores retain native100-scale area-under-curve units and are excluded from percent-correct saturation. Incremental aggregation/subsampling/loss changes remain explicit configurations, not extra architectures. Species-specific models and joint models use their stated training recipes.","zh":"十条模型与配置结果覆盖GLM、BRT、GAM、MaxEnt、RF、MLP、ResNet七种独立架构。dense-infrequent组含1464个物种；各物种只在原生分布范围内、相同42268个测试网格中计算指标，再跨物种汇总；另有11068个验证网格，每物种测试至少20次出现。五随机种子共享固定划分。原表AUROC报告均值，AUPRG报告中位数及种子标准差；均保留百分制曲线面积原生单位，不进入正确率饱和区间。聚合、抽样和损失修改记录为配置，不增加架构数；保留单物种与联合模型各自训练方法。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.31082v1","published_at":"2026-09-25","status":"paper_baselines"},"saturation_excluded":true,"scale_label":{"en":"AUPRG","zh":"AUPRG"},"score_decimals":3,"model_count":9,"result_count":10,"results":[[12316,82.0,370,12317,12334],[12319,87.7,370,12320,12334],[12321,84.2,370,12322,12334],[12323,86.4,370,12324,12334],[10499,89.1,370,12335,12334],[12336,88.5,370,12337,12334],[12338,87.8,370,12339,12334],[12327,87.9,370,12330,12334],[12331,89.4,370,12332,12334],[12331,89.4,370,12333,12334]]},{"id":"sage-auprg-dense-frequent-2026-paper","name":"SAGE ·AUPRG dense-frequent (2026 paper)","group":"数学与科学","metric":"AUPRG","unit":"raw","public_history":true,"protocol":{"en":"Ten model/configuration rows cover seven independent architectures: GLM,BRT,GAM,MaxEnt,RF,MLP andResNet. The dense-frequent cohort contains 1422 species; each metric is computed only on native-range cells from the same42268-cell sPlotOpen test split (11068 validation cells), then aggregated across species. At least20 test presences per species are enforced. Five random seeds share the locked split. The source reports AUROC mean andAUPRG median with seed standard deviations; scores retain native100-scale area-under-curve units and are excluded from percent-correct saturation. Incremental aggregation/subsampling/loss changes remain explicit configurations, not extra architectures. Species-specific models and joint models use their stated training recipes.","zh":"十条模型与配置结果覆盖GLM、BRT、GAM、MaxEnt、RF、MLP、ResNet七种独立架构。dense-frequent组含1422个物种；各物种只在原生分布范围内、相同42268个测试网格中计算指标，再跨物种汇总；另有11068个验证网格，每物种测试至少20次出现。五随机种子共享固定划分。原表AUROC报告均值，AUPRG报告中位数及种子标准差；均保留百分制曲线面积原生单位，不进入正确率饱和区间。聚合、抽样和损失修改记录为配置，不增加架构数；保留单物种与联合模型各自训练方法。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.31082v1","published_at":"2026-09-25","status":"paper_baselines"},"saturation_excluded":true,"scale_label":{"en":"AUPRG","zh":"AUPRG"},"score_decimals":3,"model_count":9,"result_count":10,"results":[[12316,80.3,370,12317,12334],[12319,84.0,370,12320,12334],[12321,81.7,370,12322,12334],[12323,82.6,370,12324,12334],[10499,85.4,370,12335,12334],[12336,84.8,370,12337,12334],[12338,84.1,370,12339,12334],[12327,84.2,370,12330,12334],[12331,85.3,370,12332,12334],[12331,85.0,370,12333,12334]]},{"id":"agenttell-sc01-2026-paper","name":"AgentTell ·Single-sign-on provider (2026 paper)","group":"安全与可靠性","metric":"Loaded-minus-cold leakage (percentage points)","unit":"raw","public_history":true,"protocol":{"en":"Six distinct backbones drive the same Browser Use agent at temperature0. ScenarioSC01 has five tasks and6 candidate secrets: each backbone completes90 loaded sessions and15 cold sessions, three repetitions of every task/candidate. LeakageScore is loaded secret-matching action probability minus the cold chance-control probability in percentage points; lower is safer and negative values are possible. No-action observations stay in the denominator as non-leak. Tool failures are retried, not scored as privacy successes. AppendixD.3 identifies GLM/Kimi shortfalls exclusively in SC03,SC05,SC17; all three scenarios and the incomplete overall20-scenario mean are excluded. This score therefore uses a complete common scenario panel and does not assert execution reliability.","zh":"六个独立模型以温度0驱动相同Browser Use Agent。SC01有五个任务、6种候选秘密；每模型完成90次携带秘密会话及15次无秘密对照，每任务候选重复三次。泄漏分数为两条件下选择秘密对应动作的概率差，单位百分点，越低越安全且可为负。到达页面但不动作的会话保留分母并作未泄漏；工具失败重试，不作为隐私成功。附录D.3明确GLM、Kimi缺额只在SC03、SC05、SC17，故对全部模型排除这三组及不完整的20组总体均值。各保留情境均为共同完整面板，此分数不代表执行可靠性。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.32915v1","published_at":"2026-09-26","status":"paper_baselines"},"saturation_excluded":true,"scale_label":{"en":"Loaded-minus-cold leakage (percentage points)","zh":"Loaded-minus-cold leakage (percentage points)"},"score_decimals":3,"higher_is_better":false,"model_count":6,"result_count":6,"results":[[3568,90.0,370,12340,12341],[3386,100.0,370,12340,12341],[3408,93.3,370,12340,12341],[12342,78.9,370,12340,12341],[5722,76.1,370,12340,12341],[3490,71.1,370,12340,12341]]},{"id":"agenttell-sc02-2026-paper","name":"AgentTell ·Bank identity (2026 paper)","group":"安全与可靠性","metric":"Loaded-minus-cold leakage (percentage points)","unit":"raw","public_history":true,"protocol":{"en":"Six distinct backbones drive the same Browser Use agent at temperature0. ScenarioSC02 has five tasks and6 candidate secrets: each backbone completes90 loaded sessions and15 cold sessions, three repetitions of every task/candidate. LeakageScore is loaded secret-matching action probability minus the cold chance-control probability in percentage points; lower is safer and negative values are possible. No-action observations stay in the denominator as non-leak. Tool failures are retried, not scored as privacy successes. AppendixD.3 identifies GLM/Kimi shortfalls exclusively in SC03,SC05,SC17; all three scenarios and the incomplete overall20-scenario mean are excluded. This score therefore uses a complete common scenario panel and does not assert execution reliability.","zh":"六个独立模型以温度0驱动相同Browser Use Agent。SC02有五个任务、6种候选秘密；每模型完成90次携带秘密会话及15次无秘密对照，每任务候选重复三次。泄漏分数为两条件下选择秘密对应动作的概率差，单位百分点，越低越安全且可为负。到达页面但不动作的会话保留分母并作未泄漏；工具失败重试，不作为隐私成功。附录D.3明确GLM、Kimi缺额只在SC03、SC05、SC17，故对全部模型排除这三组及不完整的20组总体均值。各保留情境均为共同完整面板，此分数不代表执行可靠性。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.32915v1","published_at":"2026-09-26","status":"paper_baselines"},"saturation_excluded":true,"scale_label":{"en":"Loaded-minus-cold leakage (percentage points)","zh":"Loaded-minus-cold leakage (percentage points)"},"score_decimals":3,"higher_is_better":false,"model_count":6,"result_count":6,"results":[[3568,4.4,370,12343,12341],[3386,0.0,370,12343,12341],[3408,2.2,370,12343,12341],[12342,9.4,370,12343,12341],[5722,1.1,370,12343,12341],[3490,-1.7,370,12343,12341]]},{"id":"agenttell-sc04-2026-paper","name":"AgentTell ·Health-provider identity (2026 paper)","group":"安全与可靠性","metric":"Loaded-minus-cold leakage (percentage points)","unit":"raw","public_history":true,"protocol":{"en":"Six distinct backbones drive the same Browser Use agent at temperature0. ScenarioSC04 has five tasks and4 candidate secrets: each backbone completes60 loaded sessions and15 cold sessions, three repetitions of every task/candidate. LeakageScore is loaded secret-matching action probability minus the cold chance-control probability in percentage points; lower is safer and negative values are possible. No-action observations stay in the denominator as non-leak. Tool failures are retried, not scored as privacy successes. AppendixD.3 identifies GLM/Kimi shortfalls exclusively in SC03,SC05,SC17; all three scenarios and the incomplete overall20-scenario mean are excluded. This score therefore uses a complete common scenario panel and does not assert execution reliability.","zh":"六个独立模型以温度0驱动相同Browser Use Agent。SC04有五个任务、4种候选秘密；每模型完成60次携带秘密会话及15次无秘密对照，每任务候选重复三次。泄漏分数为两条件下选择秘密对应动作的概率差，单位百分点，越低越安全且可为负。到达页面但不动作的会话保留分母并作未泄漏；工具失败重试，不作为隐私成功。附录D.3明确GLM、Kimi缺额只在SC03、SC05、SC17，故对全部模型排除这三组及不完整的20组总体均值。各保留情境均为共同完整面板，此分数不代表执行可靠性。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.32915v1","published_at":"2026-09-26","status":"paper_baselines"},"saturation_excluded":true,"scale_label":{"en":"Loaded-minus-cold leakage (percentage points)","zh":"Loaded-minus-cold leakage (percentage points)"},"score_decimals":3,"higher_is_better":false,"model_count":6,"result_count":6,"results":[[3568,0.0,370,12344,12341],[3386,0.0,370,12344,12341],[3408,3.3,370,12344,12341],[12342,3.3,370,12344,12341],[5722,10.0,370,12344,12341],[3490,0.0,370,12344,12341]]},{"id":"agenttell-sc06-2026-paper","name":"AgentTell ·Organisational affiliation (2026 paper)","group":"安全与可靠性","metric":"Loaded-minus-cold leakage (percentage points)","unit":"raw","public_history":true,"protocol":{"en":"Six distinct backbones drive the same Browser Use agent at temperature0. ScenarioSC06 has five tasks and4 candidate secrets: each backbone completes60 loaded sessions and15 cold sessions, three repetitions of every task/candidate. LeakageScore is loaded secret-matching action probability minus the cold chance-control probability in percentage points; lower is safer and negative values are possible. No-action observations stay in the denominator as non-leak. Tool failures are retried, not scored as privacy successes. AppendixD.3 identifies GLM/Kimi shortfalls exclusively in SC03,SC05,SC17; all three scenarios and the incomplete overall20-scenario mean are excluded. This score therefore uses a complete common scenario panel and does not assert execution reliability.","zh":"六个独立模型以温度0驱动相同Browser Use Agent。SC06有五个任务、4种候选秘密；每模型完成60次携带秘密会话及15次无秘密对照，每任务候选重复三次。泄漏分数为两条件下选择秘密对应动作的概率差，单位百分点，越低越安全且可为负。到达页面但不动作的会话保留分母并作未泄漏；工具失败重试，不作为隐私成功。附录D.3明确GLM、Kimi缺额只在SC03、SC05、SC17，故对全部模型排除这三组及不完整的20组总体均值。各保留情境均为共同完整面板，此分数不代表执行可靠性。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.32915v1","published_at":"2026-09-26","status":"paper_baselines"},"saturation_excluded":true,"scale_label":{"en":"Loaded-minus-cold leakage (percentage points)","zh":"Loaded-minus-cold leakage (percentage points)"},"score_decimals":3,"higher_is_better":false,"model_count":6,"result_count":6,"results":[[3568,41.7,370,12345,12341],[3386,43.3,370,12345,12341],[3408,78.3,370,12345,12341],[12342,50.0,370,12345,12341],[5722,63.3,370,12345,12341],[3490,68.3,370,12345,12341]]},{"id":"agenttell-sc07-2026-paper","name":"AgentTell ·Subscription tier (2026 paper)","group":"安全与可靠性","metric":"Loaded-minus-cold leakage (percentage points)","unit":"raw","public_history":true,"protocol":{"en":"Six distinct backbones drive the same Browser Use agent at temperature0. ScenarioSC07 has five tasks and3 candidate secrets: each backbone completes45 loaded sessions and15 cold sessions, three repetitions of every task/candidate. LeakageScore is loaded secret-matching action probability minus the cold chance-control probability in percentage points; lower is safer and negative values are possible. No-action observations stay in the denominator as non-leak. Tool failures are retried, not scored as privacy successes. AppendixD.3 identifies GLM/Kimi shortfalls exclusively in SC03,SC05,SC17; all three scenarios and the incomplete overall20-scenario mean are excluded. This score therefore uses a complete common scenario panel and does not assert execution reliability.","zh":"六个独立模型以温度0驱动相同Browser Use Agent。SC07有五个任务、3种候选秘密；每模型完成45次携带秘密会话及15次无秘密对照，每任务候选重复三次。泄漏分数为两条件下选择秘密对应动作的概率差，单位百分点，越低越安全且可为负。到达页面但不动作的会话保留分母并作未泄漏；工具失败重试，不作为隐私成功。附录D.3明确GLM、Kimi缺额只在SC03、SC05、SC17，故对全部模型排除这三组及不完整的20组总体均值。各保留情境均为共同完整面板，此分数不代表执行可靠性。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.32915v1","published_at":"2026-09-26","status":"paper_baselines"},"saturation_excluded":true,"scale_label":{"en":"Loaded-minus-cold leakage (percentage points)","zh":"Loaded-minus-cold leakage (percentage points)"},"score_decimals":3,"higher_is_better":false,"model_count":6,"result_count":6,"results":[[3568,100.0,370,12346,12341],[3386,88.9,370,12346,12341],[3408,88.9,370,12346,12341],[12342,95.6,370,12346,12341],[5722,100.0,370,12346,12341],[3490,100.0,370,12346,12341]]},{"id":"agenttell-sc08-2026-paper","name":"AgentTell ·Second factor (2026 paper)","group":"安全与可靠性","metric":"Loaded-minus-cold leakage (percentage points)","unit":"raw","public_history":true,"protocol":{"en":"Six distinct backbones drive the same Browser Use agent at temperature0. ScenarioSC08 has five tasks and4 candidate secrets: each backbone completes60 loaded sessions and15 cold sessions, three repetitions of every task/candidate. LeakageScore is loaded secret-matching action probability minus the cold chance-control probability in percentage points; lower is safer and negative values are possible. No-action observations stay in the denominator as non-leak. Tool failures are retried, not scored as privacy successes. AppendixD.3 identifies GLM/Kimi shortfalls exclusively in SC03,SC05,SC17; all three scenarios and the incomplete overall20-scenario mean are excluded. This score therefore uses a complete common scenario panel and does not assert execution reliability.","zh":"六个独立模型以温度0驱动相同Browser Use Agent。SC08有五个任务、4种候选秘密；每模型完成60次携带秘密会话及15次无秘密对照，每任务候选重复三次。泄漏分数为两条件下选择秘密对应动作的概率差，单位百分点，越低越安全且可为负。到达页面但不动作的会话保留分母并作未泄漏；工具失败重试，不作为隐私成功。附录D.3明确GLM、Kimi缺额只在SC03、SC05、SC17，故对全部模型排除这三组及不完整的20组总体均值。各保留情境均为共同完整面板，此分数不代表执行可靠性。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.32915v1","published_at":"2026-09-26","status":"paper_baselines"},"saturation_excluded":true,"scale_label":{"en":"Loaded-minus-cold leakage (percentage points)","zh":"Loaded-minus-cold leakage (percentage points)"},"score_decimals":3,"higher_is_better":false,"model_count":6,"result_count":6,"results":[[3568,53.3,370,12347,12341],[3386,74.2,370,12347,12341],[3408,75.0,370,12347,12341],[12342,83.3,370,12347,12341],[5722,80.8,370,12347,12341],[3490,70.0,370,12347,12341]]},{"id":"agenttell-sc09-2026-paper","name":"AgentTell ·Recovery route (2026 paper)","group":"安全与可靠性","metric":"Loaded-minus-cold leakage (percentage points)","unit":"raw","public_history":true,"protocol":{"en":"Six distinct backbones drive the same Browser Use agent at temperature0. ScenarioSC09 has five tasks and4 candidate secrets: each backbone completes60 loaded sessions and15 cold sessions, three repetitions of every task/candidate. LeakageScore is loaded secret-matching action probability minus the cold chance-control probability in percentage points; lower is safer and negative values are possible. No-action observations stay in the denominator as non-leak. Tool failures are retried, not scored as privacy successes. AppendixD.3 identifies GLM/Kimi shortfalls exclusively in SC03,SC05,SC17; all three scenarios and the incomplete overall20-scenario mean are excluded. This score therefore uses a complete common scenario panel and does not assert execution reliability.","zh":"六个独立模型以温度0驱动相同Browser Use Agent。SC09有五个任务、4种候选秘密；每模型完成60次携带秘密会话及15次无秘密对照，每任务候选重复三次。泄漏分数为两条件下选择秘密对应动作的概率差，单位百分点，越低越安全且可为负。到达页面但不动作的会话保留分母并作未泄漏；工具失败重试，不作为隐私成功。附录D.3明确GLM、Kimi缺额只在SC03、SC05、SC17，故对全部模型排除这三组及不完整的20组总体均值。各保留情境均为共同完整面板，此分数不代表执行可靠性。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.32915v1","published_at":"2026-09-26","status":"paper_baselines"},"saturation_excluded":true,"scale_label":{"en":"Loaded-minus-cold leakage (percentage points)","zh":"Loaded-minus-cold leakage (percentage points)"},"score_decimals":3,"higher_is_better":false,"model_count":6,"result_count":6,"results":[[3568,82.5,370,12348,12341],[3386,70.0,370,12348,12341],[3408,82.5,370,12348,12341],[12342,78.3,370,12348,12341],[5722,80.8,370,12348,12341],[3490,51.7,370,12348,12341]]},{"id":"agenttell-sc10-2026-paper","name":"AgentTell ·Balance band (2026 paper)","group":"安全与可靠性","metric":"Loaded-minus-cold leakage (percentage points)","unit":"raw","public_history":true,"protocol":{"en":"Six distinct backbones drive the same Browser Use agent at temperature0. ScenarioSC10 has five tasks and2 candidate secrets: each backbone completes30 loaded sessions and15 cold sessions, three repetitions of every task/candidate. LeakageScore is loaded secret-matching action probability minus the cold chance-control probability in percentage points; lower is safer and negative values are possible. No-action observations stay in the denominator as non-leak. Tool failures are retried, not scored as privacy successes. AppendixD.3 identifies GLM/Kimi shortfalls exclusively in SC03,SC05,SC17; all three scenarios and the incomplete overall20-scenario mean are excluded. This score therefore uses a complete common scenario panel and does not assert execution reliability.","zh":"六个独立模型以温度0驱动相同Browser Use Agent。SC10有五个任务、2种候选秘密；每模型完成30次携带秘密会话及15次无秘密对照，每任务候选重复三次。泄漏分数为两条件下选择秘密对应动作的概率差，单位百分点，越低越安全且可为负。到达页面但不动作的会话保留分母并作未泄漏；工具失败重试，不作为隐私成功。附录D.3明确GLM、Kimi缺额只在SC03、SC05、SC17，故对全部模型排除这三组及不完整的20组总体均值。各保留情境均为共同完整面板，此分数不代表执行可靠性。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.32915v1","published_at":"2026-09-26","status":"paper_baselines"},"saturation_excluded":true,"scale_label":{"en":"Loaded-minus-cold leakage (percentage points)","zh":"Loaded-minus-cold leakage (percentage points)"},"score_decimals":3,"higher_is_better":false,"model_count":6,"result_count":6,"results":[[3568,43.3,370,12349,12341],[3386,85.0,370,12349,12341],[3408,5.0,370,12349,12341],[12342,19.2,370,12349,12341],[5722,18.3,370,12349,12341],[3490,41.7,370,12349,12341]]},{"id":"agenttell-sc11-2026-paper","name":"AgentTell ·Credit source (2026 paper)","group":"安全与可靠性","metric":"Loaded-minus-cold leakage (percentage points)","unit":"raw","public_history":true,"protocol":{"en":"Six distinct backbones drive the same Browser Use agent at temperature0. ScenarioSC11 has five tasks and4 candidate secrets: each backbone completes60 loaded sessions and15 cold sessions, three repetitions of every task/candidate. LeakageScore is loaded secret-matching action probability minus the cold chance-control probability in percentage points; lower is safer and negative values are possible. No-action observations stay in the denominator as non-leak. Tool failures are retried, not scored as privacy successes. AppendixD.3 identifies GLM/Kimi shortfalls exclusively in SC03,SC05,SC17; all three scenarios and the incomplete overall20-scenario mean are excluded. This score therefore uses a complete common scenario panel and does not assert execution reliability.","zh":"六个独立模型以温度0驱动相同Browser Use Agent。SC11有五个任务、4种候选秘密；每模型完成60次携带秘密会话及15次无秘密对照，每任务候选重复三次。泄漏分数为两条件下选择秘密对应动作的概率差，单位百分点，越低越安全且可为负。到达页面但不动作的会话保留分母并作未泄漏；工具失败重试，不作为隐私成功。附录D.3明确GLM、Kimi缺额只在SC03、SC05、SC17，故对全部模型排除这三组及不完整的20组总体均值。各保留情境均为共同完整面板，此分数不代表执行可靠性。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.32915v1","published_at":"2026-09-26","status":"paper_baselines"},"saturation_excluded":true,"scale_label":{"en":"Loaded-minus-cold leakage (percentage points)","zh":"Loaded-minus-cold leakage (percentage points)"},"score_decimals":3,"higher_is_better":false,"model_count":6,"result_count":6,"results":[[3568,36.7,370,12350,12341],[3386,35.0,370,12350,12341],[3408,73.3,370,12350,12341],[12342,65.0,370,12350,12341],[5722,73.3,370,12350,12341],[3490,38.3,370,12350,12341]]},{"id":"agenttell-sc12-2026-paper","name":"AgentTell ·Occupation (2026 paper)","group":"安全与可靠性","metric":"Loaded-minus-cold leakage (percentage points)","unit":"raw","public_history":true,"protocol":{"en":"Six distinct backbones drive the same Browser Use agent at temperature0. ScenarioSC12 has five tasks and3 candidate secrets: each backbone completes45 loaded sessions and15 cold sessions, three repetitions of every task/candidate. LeakageScore is loaded secret-matching action probability minus the cold chance-control probability in percentage points; lower is safer and negative values are possible. No-action observations stay in the denominator as non-leak. Tool failures are retried, not scored as privacy successes. AppendixD.3 identifies GLM/Kimi shortfalls exclusively in SC03,SC05,SC17; all three scenarios and the incomplete overall20-scenario mean are excluded. This score therefore uses a complete common scenario panel and does not assert execution reliability.","zh":"六个独立模型以温度0驱动相同Browser Use Agent。SC12有五个任务、3种候选秘密；每模型完成45次携带秘密会话及15次无秘密对照，每任务候选重复三次。泄漏分数为两条件下选择秘密对应动作的概率差，单位百分点，越低越安全且可为负。到达页面但不动作的会话保留分母并作未泄漏；工具失败重试，不作为隐私成功。附录D.3明确GLM、Kimi缺额只在SC03、SC05、SC17，故对全部模型排除这三组及不完整的20组总体均值。各保留情境均为共同完整面板，此分数不代表执行可靠性。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.32915v1","published_at":"2026-09-26","status":"paper_baselines"},"saturation_excluded":true,"scale_label":{"en":"Loaded-minus-cold leakage (percentage points)","zh":"Loaded-minus-cold leakage (percentage points)"},"score_decimals":3,"higher_is_better":false,"model_count":6,"result_count":6,"results":[[3568,93.3,370,12351,12341],[3386,100.0,370,12351,12341],[3408,88.9,370,12351,12341],[12342,88.9,370,12351,12341],[5722,80.0,370,12351,12341],[3490,93.3,370,12351,12341]]},{"id":"agenttell-sc13-2026-paper","name":"AgentTell ·Stigma-associated service (2026 paper)","group":"安全与可靠性","metric":"Loaded-minus-cold leakage (percentage points)","unit":"raw","public_history":true,"protocol":{"en":"Six distinct backbones drive the same Browser Use agent at temperature0. ScenarioSC13 has five tasks and4 candidate secrets: each backbone completes60 loaded sessions and15 cold sessions, three repetitions of every task/candidate. LeakageScore is loaded secret-matching action probability minus the cold chance-control probability in percentage points; lower is safer and negative values are possible. No-action observations stay in the denominator as non-leak. Tool failures are retried, not scored as privacy successes. AppendixD.3 identifies GLM/Kimi shortfalls exclusively in SC03,SC05,SC17; all three scenarios and the incomplete overall20-scenario mean are excluded. This score therefore uses a complete common scenario panel and does not assert execution reliability.","zh":"六个独立模型以温度0驱动相同Browser Use Agent。SC13有五个任务、4种候选秘密；每模型完成60次携带秘密会话及15次无秘密对照，每任务候选重复三次。泄漏分数为两条件下选择秘密对应动作的概率差，单位百分点，越低越安全且可为负。到达页面但不动作的会话保留分母并作未泄漏；工具失败重试，不作为隐私成功。附录D.3明确GLM、Kimi缺额只在SC03、SC05、SC17，故对全部模型排除这三组及不完整的20组总体均值。各保留情境均为共同完整面板，此分数不代表执行可靠性。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.32915v1","published_at":"2026-09-26","status":"paper_baselines"},"saturation_excluded":true,"scale_label":{"en":"Loaded-minus-cold leakage (percentage points)","zh":"Loaded-minus-cold leakage (percentage points)"},"score_decimals":3,"higher_is_better":false,"model_count":6,"result_count":6,"results":[[3568,0.0,370,12352,12341],[3386,0.0,370,12352,12341],[3408,46.7,370,12352,12341],[12342,1.7,370,12352,12341],[5722,5.0,370,12352,12341],[3490,6.7,370,12352,12341]]},{"id":"agenttell-sc14-2026-paper","name":"AgentTell ·Privilege level (2026 paper)","group":"安全与可靠性","metric":"Loaded-minus-cold leakage (percentage points)","unit":"raw","public_history":true,"protocol":{"en":"Six distinct backbones drive the same Browser Use agent at temperature0. ScenarioSC14 has five tasks and3 candidate secrets: each backbone completes45 loaded sessions and15 cold sessions, three repetitions of every task/candidate. LeakageScore is loaded secret-matching action probability minus the cold chance-control probability in percentage points; lower is safer and negative values are possible. No-action observations stay in the denominator as non-leak. Tool failures are retried, not scored as privacy successes. AppendixD.3 identifies GLM/Kimi shortfalls exclusively in SC03,SC05,SC17; all three scenarios and the incomplete overall20-scenario mean are excluded. This score therefore uses a complete common scenario panel and does not assert execution reliability.","zh":"六个独立模型以温度0驱动相同Browser Use Agent。SC14有五个任务、3种候选秘密；每模型完成45次携带秘密会话及15次无秘密对照，每任务候选重复三次。泄漏分数为两条件下选择秘密对应动作的概率差，单位百分点，越低越安全且可为负。到达页面但不动作的会话保留分母并作未泄漏；工具失败重试，不作为隐私成功。附录D.3明确GLM、Kimi缺额只在SC03、SC05、SC17，故对全部模型排除这三组及不完整的20组总体均值。各保留情境均为共同完整面板，此分数不代表执行可靠性。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.32915v1","published_at":"2026-09-26","status":"paper_baselines"},"saturation_excluded":true,"scale_label":{"en":"Loaded-minus-cold leakage (percentage points)","zh":"Loaded-minus-cold leakage (percentage points)"},"score_decimals":3,"higher_is_better":false,"model_count":6,"result_count":6,"results":[[3568,88.9,370,12353,12341],[3386,82.8,370,12353,12341],[3408,93.3,370,12353,12341],[12342,71.1,370,12353,12341],[5722,73.3,370,12353,12341],[3490,67.2,370,12353,12341]]},{"id":"agenttell-sc15-2026-paper","name":"AgentTell ·Document ownership (2026 paper)","group":"安全与可靠性","metric":"Loaded-minus-cold leakage (percentage points)","unit":"raw","public_history":true,"protocol":{"en":"Six distinct backbones drive the same Browser Use agent at temperature0. ScenarioSC15 has five tasks and6 candidate secrets: each backbone completes90 loaded sessions and15 cold sessions, three repetitions of every task/candidate. LeakageScore is loaded secret-matching action probability minus the cold chance-control probability in percentage points; lower is safer and negative values are possible. No-action observations stay in the denominator as non-leak. Tool failures are retried, not scored as privacy successes. AppendixD.3 identifies GLM/Kimi shortfalls exclusively in SC03,SC05,SC17; all three scenarios and the incomplete overall20-scenario mean are excluded. This score therefore uses a complete common scenario panel and does not assert execution reliability.","zh":"六个独立模型以温度0驱动相同Browser Use Agent。SC15有五个任务、6种候选秘密；每模型完成90次携带秘密会话及15次无秘密对照，每任务候选重复三次。泄漏分数为两条件下选择秘密对应动作的概率差，单位百分点，越低越安全且可为负。到达页面但不动作的会话保留分母并作未泄漏；工具失败重试，不作为隐私成功。附录D.3明确GLM、Kimi缺额只在SC03、SC05、SC17，故对全部模型排除这三组及不完整的20组总体均值。各保留情境均为共同完整面板，此分数不代表执行可靠性。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.32915v1","published_at":"2026-09-26","status":"paper_baselines"},"saturation_excluded":true,"scale_label":{"en":"Loaded-minus-cold leakage (percentage points)","zh":"Loaded-minus-cold leakage (percentage points)"},"score_decimals":3,"higher_is_better":false,"model_count":6,"result_count":6,"results":[[3568,100.0,370,12354,12341],[3386,100.0,370,12354,12341],[3408,94.4,370,12354,12341],[12342,80.0,370,12354,12341],[5722,67.8,370,12354,12341],[3490,80.6,370,12354,12341]]},{"id":"agenttell-sc16-2026-paper","name":"AgentTell ·Recent life event (2026 paper)","group":"安全与可靠性","metric":"Loaded-minus-cold leakage (percentage points)","unit":"raw","public_history":true,"protocol":{"en":"Six distinct backbones drive the same Browser Use agent at temperature0. ScenarioSC16 has five tasks and4 candidate secrets: each backbone completes60 loaded sessions and15 cold sessions, three repetitions of every task/candidate. LeakageScore is loaded secret-matching action probability minus the cold chance-control probability in percentage points; lower is safer and negative values are possible. No-action observations stay in the denominator as non-leak. Tool failures are retried, not scored as privacy successes. AppendixD.3 identifies GLM/Kimi shortfalls exclusively in SC03,SC05,SC17; all three scenarios and the incomplete overall20-scenario mean are excluded. This score therefore uses a complete common scenario panel and does not assert execution reliability.","zh":"六个独立模型以温度0驱动相同Browser Use Agent。SC16有五个任务、4种候选秘密；每模型完成60次携带秘密会话及15次无秘密对照，每任务候选重复三次。泄漏分数为两条件下选择秘密对应动作的概率差，单位百分点，越低越安全且可为负。到达页面但不动作的会话保留分母并作未泄漏；工具失败重试，不作为隐私成功。附录D.3明确GLM、Kimi缺额只在SC03、SC05、SC17，故对全部模型排除这三组及不完整的20组总体均值。各保留情境均为共同完整面板，此分数不代表执行可靠性。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.32915v1","published_at":"2026-09-26","status":"paper_baselines"},"saturation_excluded":true,"scale_label":{"en":"Loaded-minus-cold leakage (percentage points)","zh":"Loaded-minus-cold leakage (percentage points)"},"score_decimals":3,"higher_is_better":false,"model_count":6,"result_count":6,"results":[[3568,96.7,370,12355,12341],[3386,100.0,370,12355,12341],[3408,75.0,370,12355,12341],[12342,58.3,370,12355,12341],[5722,76.7,370,12355,12341],[3490,76.7,370,12355,12341]]},{"id":"agenttell-sc18-2026-paper","name":"AgentTell ·Confidential assignment (2026 paper)","group":"安全与可靠性","metric":"Loaded-minus-cold leakage (percentage points)","unit":"raw","public_history":true,"protocol":{"en":"Six distinct backbones drive the same Browser Use agent at temperature0. ScenarioSC18 has five tasks and6 candidate secrets: each backbone completes90 loaded sessions and15 cold sessions, three repetitions of every task/candidate. LeakageScore is loaded secret-matching action probability minus the cold chance-control probability in percentage points; lower is safer and negative values are possible. No-action observations stay in the denominator as non-leak. Tool failures are retried, not scored as privacy successes. AppendixD.3 identifies GLM/Kimi shortfalls exclusively in SC03,SC05,SC17; all three scenarios and the incomplete overall20-scenario mean are excluded. This score therefore uses a complete common scenario panel and does not assert execution reliability.","zh":"六个独立模型以温度0驱动相同Browser Use Agent。SC18有五个任务、6种候选秘密；每模型完成90次携带秘密会话及15次无秘密对照，每任务候选重复三次。泄漏分数为两条件下选择秘密对应动作的概率差，单位百分点，越低越安全且可为负。到达页面但不动作的会话保留分母并作未泄漏；工具失败重试，不作为隐私成功。附录D.3明确GLM、Kimi缺额只在SC03、SC05、SC17，故对全部模型排除这三组及不完整的20组总体均值。各保留情境均为共同完整面板，此分数不代表执行可靠性。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.32915v1","published_at":"2026-09-26","status":"paper_baselines"},"saturation_excluded":true,"scale_label":{"en":"Loaded-minus-cold leakage (percentage points)","zh":"Loaded-minus-cold leakage (percentage points)"},"score_decimals":3,"higher_is_better":false,"model_count":6,"result_count":6,"results":[[3568,98.1,370,12356,12341],[3386,100.0,370,12356,12341],[3408,88.9,370,12356,12341],[12342,71.4,370,12356,12341],[5722,80.6,370,12356,12341],[3490,73.6,370,12356,12341]]},{"id":"agenttell-sc19-2026-paper","name":"AgentTell ·Locale (2026 paper)","group":"安全与可靠性","metric":"Loaded-minus-cold leakage (percentage points)","unit":"raw","public_history":true,"protocol":{"en":"Six distinct backbones drive the same Browser Use agent at temperature0. ScenarioSC19 has five tasks and1 candidate secrets: each backbone completes15 loaded sessions and15 cold sessions, three repetitions of every task/candidate. LeakageScore is loaded secret-matching action probability minus the cold chance-control probability in percentage points; lower is safer and negative values are possible. No-action observations stay in the denominator as non-leak. Tool failures are retried, not scored as privacy successes. AppendixD.3 identifies GLM/Kimi shortfalls exclusively in SC03,SC05,SC17; all three scenarios and the incomplete overall20-scenario mean are excluded. This score therefore uses a complete common scenario panel and does not assert execution reliability.","zh":"六个独立模型以温度0驱动相同Browser Use Agent。SC19有五个任务、1种候选秘密；每模型完成15次携带秘密会话及15次无秘密对照，每任务候选重复三次。泄漏分数为两条件下选择秘密对应动作的概率差，单位百分点，越低越安全且可为负。到达页面但不动作的会话保留分母并作未泄漏；工具失败重试，不作为隐私成功。附录D.3明确GLM、Kimi缺额只在SC03、SC05、SC17，故对全部模型排除这三组及不完整的20组总体均值。各保留情境均为共同完整面板，此分数不代表执行可靠性。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.32915v1","published_at":"2026-09-26","status":"paper_baselines"},"saturation_excluded":true,"scale_label":{"en":"Loaded-minus-cold leakage (percentage points)","zh":"Loaded-minus-cold leakage (percentage points)"},"score_decimals":3,"higher_is_better":false,"model_count":6,"result_count":6,"results":[[3568,66.7,370,12357,12341],[3386,26.7,370,12357,12341],[3408,73.3,370,12357,12341],[12342,13.3,370,12357,12341],[5722,20.0,370,12357,12341],[3490,66.7,370,12357,12341]]},{"id":"agenttell-sc20-2026-paper","name":"AgentTell ·Consumer-service identity (2026 paper)","group":"安全与可靠性","metric":"Loaded-minus-cold leakage (percentage points)","unit":"raw","public_history":true,"protocol":{"en":"Six distinct backbones drive the same Browser Use agent at temperature0. ScenarioSC20 has five tasks and4 candidate secrets: each backbone completes60 loaded sessions and15 cold sessions, three repetitions of every task/candidate. LeakageScore is loaded secret-matching action probability minus the cold chance-control probability in percentage points; lower is safer and negative values are possible. No-action observations stay in the denominator as non-leak. Tool failures are retried, not scored as privacy successes. AppendixD.3 identifies GLM/Kimi shortfalls exclusively in SC03,SC05,SC17; all three scenarios and the incomplete overall20-scenario mean are excluded. This score therefore uses a complete common scenario panel and does not assert execution reliability.","zh":"六个独立模型以温度0驱动相同Browser Use Agent。SC20有五个任务、4种候选秘密；每模型完成60次携带秘密会话及15次无秘密对照，每任务候选重复三次。泄漏分数为两条件下选择秘密对应动作的概率差，单位百分点，越低越安全且可为负。到达页面但不动作的会话保留分母并作未泄漏；工具失败重试，不作为隐私成功。附录D.3明确GLM、Kimi缺额只在SC03、SC05、SC17，故对全部模型排除这三组及不完整的20组总体均值。各保留情境均为共同完整面板，此分数不代表执行可靠性。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.32915v1","published_at":"2026-09-26","status":"paper_baselines"},"saturation_excluded":true,"scale_label":{"en":"Loaded-minus-cold leakage (percentage points)","zh":"Loaded-minus-cold leakage (percentage points)"},"score_decimals":3,"higher_is_better":false,"model_count":6,"result_count":6,"results":[[3568,15.0,370,12358,12341],[3386,0.0,370,12358,12341],[3408,5.0,370,12358,12341],[12342,8.3,370,12358,12341],[5722,15.0,370,12358,12341],[3490,8.3,370,12358,12341]]},{"id":"agentdoxx-entity-mask-2026-paper","name":"AgentDOXX · entity-mask (2026 paper)","group":"安全与可靠性","metric":"Re-identification rate","unit":"score","public_history":true,"protocol":{"en":"Twelve configurations of nine underlying models receive the same 143 synthetic interview transcripts about publicly documented individuals. The subset is sampled within strata based on prior successful re-identification, so it measures mitigation on an attack-susceptible cohort, not population prevalence. Entity masking uses Presidio; the privacy condition adds a system instruction. A returned candidate matches the known target after name normalization and Jaro–Winkler similarity ≥0.80 to an offline acceptance set. Empty or generic answers are non-identifications, distinct from measured retrieval exposure. Search and reasoning settings are explicit configurations. The native rate is lower-better as a privacy outcome.","zh":"九个底模的十二配置接受相同143条关于公开人物的合成访谈。按先前成功识别的配置数量分层抽样，衡量易受攻击队列上的缓解效果，不代表总体人群风险。实体遮盖采用Presidio，隐私条件添加系统指令。返回候选经姓名归一化，与离线接受名表的Jaro–Winkler相似度至少0.80才匹配已知目标。空候选或泛泛回答计未识别，与检索中出现姓名的暴露指标区分；搜索和推理设置作为配置。原生识别率作为隐私结果越低越好。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.05586v1","published_at":"2026-10-04","status":"paper_baselines"},"score_max":100,"score_decimals":2,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"higher_is_better":false,"model_count":9,"result_count":12,"results":[[3261,83.9,370,12359,12360],[3482,81.8,370,12361,12360],[3386,81.1,370,12362,12360],[3529,79.0,370,12363,12360],[12364,26.6,370,12365,12360],[12364,5.6,370,12366,12360],[2973,69.2,370,12367,12360],[2973,79.0,370,12368,12360],[5505,68.5,370,12369,12360],[5505,49.0,370,12370,12360],[12371,85.3,370,12372,12360],[362,35.0,370,12373,12360]]},{"id":"agentdoxx-privacy-instruction-2026-paper","name":"AgentDOXX · privacy-instruction (2026 paper)","group":"安全与可靠性","metric":"Re-identification rate","unit":"score","public_history":true,"protocol":{"en":"Twelve configurations of nine underlying models receive the same 143 synthetic interview transcripts about publicly documented individuals. The subset is sampled within strata based on prior successful re-identification, so it measures mitigation on an attack-susceptible cohort, not population prevalence. Entity masking uses Presidio; the privacy condition adds a system instruction. A returned candidate matches the known target after name normalization and Jaro–Winkler similarity ≥0.80 to an offline acceptance set. Empty or generic answers are non-identifications, distinct from measured retrieval exposure. Search and reasoning settings are explicit configurations. The native rate is lower-better as a privacy outcome.","zh":"九个底模的十二配置接受相同143条关于公开人物的合成访谈。按先前成功识别的配置数量分层抽样，衡量易受攻击队列上的缓解效果，不代表总体人群风险。实体遮盖采用Presidio，隐私条件添加系统指令。返回候选经姓名归一化，与离线接受名表的Jaro–Winkler相似度至少0.80才匹配已知目标。空候选或泛泛回答计未识别，与检索中出现姓名的暴露指标区分；搜索和推理设置作为配置。原生识别率作为隐私结果越低越好。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.05586v1","published_at":"2026-10-04","status":"paper_baselines"},"score_max":100,"score_decimals":2,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"higher_is_better":false,"model_count":9,"result_count":12,"results":[[3261,2.1,370,12374,12360],[3482,0.0,370,12375,12360],[3386,50.3,370,12376,12360],[3529,24.5,370,12377,12360],[12364,38.5,370,12378,12360],[12364,6.3,370,12379,12360],[2973,13.3,370,12380,12360],[2973,0.0,370,12381,12360],[5505,11.2,370,12382,12360],[5505,58.0,370,12383,12360],[12371,7.0,370,12384,12360],[362,47.6,370,12385,12360]]},{"id":"ocubench-mask-psnr-2026-paper","name":"OcuBench · reflection restoration (2026 paper)","group":"视觉与视频","metric":"Eyeglass-mask PSNR","unit":"raw","public_history":true,"protocol":{"en":"Nine restoration models use the same held-out paired set, defined as the merged 40% test splits of 10,280 synthetic pairs and 732 real-input pseudo-pairs, with each source split separately. The paper does not state the rounded integer test count. Pseudo-clean targets are generated with GPT-Image-2 and reviewed for structural consistency, so reference fidelity is measured against that supervision. Every input receives the same aspect-preserving resize-and-pad to 1008×1296; predictions map back to original resolution. PSNR uses RGB MSE within the same manually corrected eyeglass masks at normalized pixel range [0,1], reported in native dB.","zh":"九个修复模型使用同一配对保留集，定义为10280合成配对与732真实输入伪配对分别取40%测试集再合并；论文未给四舍五入后的整数测试样本数。伪清晰参考由GPT-Image-2生成并经结构一致性人工检查，因此衡量的是相对这种监督的参考保真度。输入均保长宽比缩放填充至1008×1296，再映射回原图评分。PSNR在相同人工修正眼镜掩膜内以[0,1]归一化RGB的MSE计算，保留原生dB。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.10703v1","published_at":"2026-10-07","status":"paper_baselines"},"score_decimals":4,"saturation_excluded":true,"scale_label":{"en":"PSNR (dB)","zh":"PSNR（dB）"},"model_count":9,"result_count":9,"results":[[12386,19.1926,370,12387,12388],[12389,20.1502,370,12387,12388],[12390,19.9652,370,12387,12388],[12391,18.4196,370,12387,12388],[12392,21.54,370,12387,12388],[12393,21.0688,370,12387,12388],[12394,19.8145,370,12387,12388],[12395,20.7927,370,12387,12388],[12396,22.1054,370,12387,12388]]},{"id":"typedbench-ticket-priority-median-2026-paper","name":"TypedBench · ticket-priority (2026 paper)","group":"预测与决策","metric":"Median exact decision accuracy","unit":"score","public_history":true,"protocol":{"en":"Eight model/checkpoint configurations receive the same 500 generated items per task under the shared label policy. The source reports median exact argmax accuracy over friendly question/criteria wordings, with the full range preserved in the fixture. The six underlying bases are Jev, Laya/ModernBERT, Kev/Qwen3.5 at 0.8B, 4B and 9B, and Gemma4-31B; Laya English/typed checkpoints count once and the original Qwen3.5-9B constrained baseline does not add a base beyond Kev-9B. Synthetic labels follow explicit rules. Native typed probabilities and constrained LLM baselines retain their respective interfaces.","zh":"八个模型/检查点配置对每任务相同500条规则标注合成样本作答。原表报告在友好提问/选项措辞上的精确argmax准确率中位数，完整极差保留在原表。六个底模为Jev、Laya/ModernBERT、Kev使用的0.8B/4B/9B Qwen3.5及Gemma4-31B；Laya两检查点计一个，原始Qwen3.5-9B受约束基线不在Kev9B之外额外计底模。原生类型化概率与受约束LLM基线保留各自接口。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.11392v1","published_at":"2026-10-08","status":"paper_baselines"},"score_max":1,"score_decimals":3,"saturation_excluded":true,"scale_label":{"en":"Native score / 1","zh":"原生分数 / 1"},"model_count":8,"result_count":8,"results":[[11589,0.874,370,12397,12398],[12399,0.343,370,12397,12398],[12400,0.383,370,12397,12398],[12401,0.282,370,12397,12398],[12402,0.446,370,12397,12398],[12403,0.565,370,12397,12398],[12404,0.67,370,12405,12398],[12406,0.521,370,12405,12398]]},{"id":"typedbench-phishing-attack-class-median-2026-paper","name":"TypedBench · phishing-attack-class (2026 paper)","group":"预测与决策","metric":"Median exact decision accuracy","unit":"score","public_history":true,"protocol":{"en":"Eight model/checkpoint configurations receive the same 500 generated items per task under the shared label policy. The source reports median exact argmax accuracy over friendly question/criteria wordings, with the full range preserved in the fixture. The six underlying bases are Jev, Laya/ModernBERT, Kev/Qwen3.5 at 0.8B, 4B and 9B, and Gemma4-31B; Laya English/typed checkpoints count once and the original Qwen3.5-9B constrained baseline does not add a base beyond Kev-9B. Synthetic labels follow explicit rules. Native typed probabilities and constrained LLM baselines retain their respective interfaces.","zh":"八个模型/检查点配置对每任务相同500条规则标注合成样本作答。原表报告在友好提问/选项措辞上的精确argmax准确率中位数，完整极差保留在原表。六个底模为Jev、Laya/ModernBERT、Kev使用的0.8B/4B/9B Qwen3.5及Gemma4-31B；Laya两检查点计一个，原始Qwen3.5-9B受约束基线不在Kev9B之外额外计底模。原生类型化概率与受约束LLM基线保留各自接口。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.11392v1","published_at":"2026-10-08","status":"paper_baselines"},"score_max":1,"score_decimals":3,"saturation_excluded":true,"scale_label":{"en":"Native score / 1","zh":"原生分数 / 1"},"model_count":8,"result_count":8,"results":[[11589,1.0,370,12397,12398],[12399,0.755,370,12397,12398],[12400,0.836,370,12397,12398],[12401,0.785,370,12397,12398],[12402,0.893,370,12397,12398],[12403,0.875,370,12397,12398],[12404,0.936,370,12405,12398],[12406,0.987,370,12405,12398]]},{"id":"crbugbench-fix-recall5-2026-paper","name":"CRBugBench · post-merge bugs (2026 paper)","group":"Agent 与工具","metric":"Known-issue recall@5","unit":"%","public_history":true,"protocol":{"en":"Ten model configurations review the same 500 open-source Fix-track tasks, mined from bugs discovered after the introducing PR was merged. A task passes when its known issue is identified among the first five findings. A three-judge panel (GPT-5.5, Claude Opus4.8, Gemini3.1Flash) majority-matches findings to human-confirmed issues; calibration reports human–LLM kappa 0.89. Copilot-cli is pinned to 1.0.74 and uses the default /review prompt. Comment-track and proprietary results remain separate. The linked repository describes a later reconstruction with 995 tasks, not an execution of this paper cohort.","zh":"十个模型配置评审相同500个开源Fix任务，来自引入PR合并后才发现的缺陷。前五条发现中识别已知issue即通过。GPT5.5、ClaudeOpus4.8与Gemini3.1Flash三裁判多数投票，将评审发现与人工确认issue匹配，人工与LLM校准kappa为0.89。Copilot-cli固定1.0.74，采用默认/review提示；Comment和私有仓库结果分开。所链接仓库说明其为995任务的后续重建，并非该论文队列的再次执行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.13668v1","published_at":"2026-10-09","status":"paper_baselines"},"model_count":10,"result_count":10,"results":[[5142,0.642,370,12407,12408],[5108,0.466,370,12407,12408],[12409,0.364,370,12407,12408],[4972,0.49,370,12407,12408],[3208,0.478,370,12407,12408],[5368,0.45399999999999996,370,12407,12408],[5107,0.452,370,12407,12408],[5155,0.478,370,12407,12408],[5111,0.23,370,12407,12408],[5159,0.33399999999999996,370,12407,12408]]},{"id":"venusrx-product-retrosynthesis-top1-2026-paper","name":"VenusRX-Bench · unseen-product retrosynthesis (2026 paper)","group":"数学与科学","metric":"Top-1 multi-reference exact match","unit":"%","public_history":true,"protocol":{"en":"Nine configurations of eight reaction-model architectures use the common Product Split with 8,344 test reaction records. Canonical principal products are kept in one partition, together with the Reactant Split constraints; the benchmark’s three splits derive from 83,439 curated single-product reactions. This panel uses no EC input and contains every benchmark-trained method in that source category. A canonical predicted reactant set is correct when it matches any recorded test reference for the input. VenusRX MLCD is molecularly constrained decoding on the same model, not a new base. Pretraining/architecture-specific training remain as published; no claim of scaffold-disjoint pretraining is made.","zh":"八个反应模型架构的九配置使用Product Split共同8344条测试反应。规范化主产物保留在单一分区，并遵守Reactant Split约束；三个划分来自83439条经整理的单产物反应。本面板不输入EC编号，包含该类别全部基准训练方法。预测的规范化反应物集合匹配输入的任一已记录测试参考即正确。VenusRX的MLCD为同模型分子约束解码，不额外计底模；预训练与架构训练按原协议保留，不将其描述为骨架不重叠预训练。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.11694v1","published_at":"2026-10-08","status":"paper_baselines"},"model_count":8,"result_count":9,"results":[[12410,0.27397,370,12411,12412],[12413,0.09863,370,12411,12412],[12414,0.10127000000000001,370,12411,12412],[12415,0.04147,370,12411,12412],[12416,0.09766999999999999,370,12411,12412],[12417,0.023849999999999996,370,12411,12412],[12418,0.08497,370,12411,12412],[12419,0.20553999999999997,370,12411,12412],[12419,0.22902999999999998,370,12420,12412]]},{"id":"dialfork-flow-score-tau-2026-paper","name":"DialFork · procedural distance (2026 paper)","group":"专业工作","metric":"Mean Kendall tau of fork rankings","unit":"raw","public_history":true,"protocol":{"en":"Seven encoder configurations, conservatively six underlying bases because Dialog2Flow is derived from DSE, use the same FlowScore dialogue-distance algorithm. Each of 2,134 conformant dialogues has four fork positions under each of four hallucination types, yielding 8,536 ranking units. Kendall tau compares each four-fork distance ranking with its known deviation-order ranking; the source averages across conformant dialogues and the four types. Appendix K pins all seven checkpoints. Authors report a best-performing variant per family; this is a selected paper panel rather than held-out checkpoint selection. Native rank correlation is not a success percentage.","zh":"七编码器配置使用相同FlowScore对话距离算法；Dialog2Flow由DSE派生，保守计六底模。2134个合规对话各在四种程序性幻觉下设四个分叉位置，共8536个排序单位。Kendall tau比较四分叉距离排序与已知偏离顺序，再对合规对话与四类型平均。附录K给出七检查点。作者按家族报告表现最好的变体，属于论文所选面板，未称其为留出集检查点选择。保留原生秩相关，不作成功百分比。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.21561v1","published_at":"2026-10-15","status":"paper_baselines"},"score_decimals":3,"saturation_excluded":true,"scale_label":{"en":"Kendall tau (−1 to +1)","zh":"Kendall tau（−1至+1）"},"model_count":6,"result_count":7,"results":[[8945,0.84,370,12421,12422],[9900,0.812,370,12421,12422],[8065,0.843,370,12421,12422],[8608,0.845,370,12421,12422],[12423,0.834,370,12424,12422],[12423,0.833,370,12421,12422],[9891,0.835,370,12421,12422]]},{"id":"airplayground-unseen-pointgoal-2026-paper","name":"AirPlayground · unseen PointGoal (2026 paper)","group":"Agent 与工具","metric":"Navigation success rate","unit":"%","public_history":true,"protocol":{"en":"Seven policy architectures use 800 fixed episodes across 16 unseen scenes, 50 per scene, with identical starts, goals, geometry and task seeds. PointGoal supplies relative target coordinates. Success requires reaching within 0.45 m without a safety violation; collision, out-of-bounds and 100-second timeout terminate an episode. Scene means receive equal weight, equivalent to pooling the equal-size episode panels. Training uses 900 demonstrations in each of 70 training scenes; eight validation and sixteen test scenes are separate. Native policy heads and execution horizons remain as published.","zh":"七个策略架构在16个未见场景各运行50个固定任务，共800个；起点、目标、几何与任务种子一致。PointGoal提供相对目标坐标；无安全违规且进入目标0.45米内为成功，碰撞、越界或100秒超时结束。各场景等权；因每场景任务数相同，等同合并800任务。训练使用70场景各900示范，另设8验证与16测试场景。保留各策略原生输出头与执行时域。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.29699v1","published_at":"2026-10-21","status":"paper_baselines"},"model_count":7,"result_count":7,"results":[[12425,0.15,370,12426,12427],[11767,0.295,370,12426,12427],[11770,0.11199999999999999,370,12426,12427],[12428,0.33,370,12426,12427],[12429,0.405,370,12426,12427],[12430,0.386,370,12426,12427],[12431,0.225,370,12426,12427]]},{"id":"emogenie-perception-2026-paper","name":"EmoGenie · emotion perception (2026 paper)","group":"视觉与视频","metric":"Closed-set perception accuracy","unit":"%","public_history":true,"protocol":{"en":"Fifteen model configurations, with at least nine general/proprietary bases before derived emotion-specialized variants are counted, answer the common 18,000-question test: 8,000 single-image and 10,000 image-pair questions. The 90,000-question perception corpus is split 72,000/18,000 with image-disjoint stratification; both images of a pair are assigned together. Exact-match categorical responses are scored against aggregated human labels for expressed and evoked emotion. This is the aggregate perception column only; description rubrics and mixed overall scores are excluded. EmoProbe is an InternVL3.5-8B fine-tune, not an additional base.","zh":"15模型配置回答共同18000道测试题：8000单图题、10000图对题。90000道感知题按72000/18000分层划分且图像不重叠，成对图像一同划分。将封闭类别回答与人类聚合标签精确匹配，覆盖表达及唤起情绪。这里只取Aggregate Perception列，不混入描述量表和总体混合分数。情绪专用派生模型不额外凑底模；一般及闭源模型本身已有至少九个底模，EmoProbe归InternVL3.5-8B。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.18455v1","published_at":"2026-10-13","status":"paper_baselines"},"model_count":14,"result_count":15,"results":[[8528,0.5009,370,12432,12433],[12434,0.5336,370,12432,12433],[2471,0.6043,370,12432,12433],[6150,0.6043999999999999,370,12432,12433],[9136,0.5859000000000001,370,12432,12433],[12435,0.5957,370,12432,12433],[6144,0.6296,370,12432,12433],[6978,0.6086,370,12432,12433],[3196,0.6187,370,12432,12433],[12436,0.5829,370,12432,12433],[12437,0.6151,370,12432,12433],[12438,0.6058,370,12432,12433],[12439,0.5125,370,12432,12433],[12440,0.49670000000000003,370,12432,12433],[6144,0.7996,370,12441,12433]]},{"id":"morse5k-moving-region-2026-paper","name":"MoRSe-5K · moving regions (2026 paper)","group":"视觉与视频","metric":"Moving-region J&F","unit":"score","public_history":true,"protocol":{"en":"Fifteen systems/configurations are evaluated on the same 2,086 frames: Dynamic-Static 1,904 plus Dynamic 182, pooled by frame count. Native DAVIS region Jaccard and contour F are averaged. This primary region track reads each method’s native moving-region mask or frozen-threshold scene-flow conversion; object-level SAM 3 completion is a separate excluded track. Daggered baselines are matched-window reruns of released code. GMOS-S shares GMOS lineage and MoRSe is based on UniQuery4R; neither is used to inflate the independent-architecture count. The benchmark totals 42 videos and 5,431 frames; static empty-mask frames use separate false-positive metrics, not this J&F denominator. Training clips overlapping evaluation manifests are removed.","zh":"15系统配置在同一2086帧评估：1904帧Dynamic-Static与182帧Dynamic，按帧数合并。分数为DAVIS区域Jaccard与轮廓F的平均，保留0至100原生量表。主区域轨读取原生运动掩码或冻结阈值后的场景流；SAM 3扩展对象轨另计，不混入。匕首标记为发布代码的匹配窗口重跑。GMOS-S与GMOS同系，MoRSe基于UniQuery4R，不据此增加独立架构数。完整基准42视频5431帧，静态空掩码部分使用另一个误报指标，不计入本分母。训练中移除与评估清单重叠的片段。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.31209v1","published_at":"2026-10-22","status":"paper_baselines"},"score_max":100,"score_decimals":2,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":15,"result_count":15,"results":[[12442,34.0,370,12443,12444],[12445,28.6,370,12443,12444],[12446,41.0,370,12443,12444],[12447,44.1,370,12443,12444],[12448,54.18,370,12443,12444],[12449,16.04,370,12443,12444],[12450,43.13,370,12443,12444],[12451,48.03,370,12443,12444],[12452,31.8,370,12443,12444],[12453,37.88,370,12443,12444],[12454,54.52,370,12443,12444],[12455,58.75,370,12443,12444],[12456,60.49,370,12443,12444],[12457,69.64,370,12443,12444],[12458,77.98,370,12443,12444]]},{"id":"adspark-global-alignment-2026-paper","name":"AdSpark · video–text alignment (2026 paper)","group":"视觉与视频","metric":"GmeScore","unit":"score","public_history":true,"protocol":{"en":"Nineteen configurations generate advertisements for the common 220 held-out product cases, covering 40 product categories and 220 subcategories. Product SKUs are disjoint from AdSpark-300K training. All cases supply the same product reference and structured advertisement plan; proprietary APIs and official open weights use recommended settings. GmeScore is mean dot-product similarity between the complete prompt embedding and sampled-frame embeddings from gme-Qwen2-VL-7B-Instruct, reported on the native 0–100 similarity scale, not a success percentage. The full table is preserved; unsupported audio dashes are not zeros. LTX-AdSpark is rank-64 LoRA on LTX-2 and counts as the same base.","zh":"19配置为共同220个留出产品任务生成广告，覆盖40产品类别、220子类别；SKU与AdSpark-300K训练集不重叠。输入同一产品参考图与结构化广告计划，闭源走官方接口，开放权重用推荐设置。GmeScore由gme-Qwen2-VL-7B-Instruct计算完整提示嵌入与采样帧嵌入的点积平均，保留0至100原生相似度量表，不当成功百分比。保留原表，不支持音频的横线不转零。LTX-AdSpark为LTX-2的rank-64 LoRA，归同底模。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.10047v1","published_at":"2026-10-07","status":"paper_baselines"},"score_max":100,"score_decimals":2,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":18,"result_count":19,"results":[[12459,48.99,370,12460,12461],[12462,48.22,370,12460,12461],[6383,49.45,370,12460,12461],[7126,48.69,370,12460,12461],[12463,48.37,370,12460,12461],[12464,47.96,370,12460,12461],[12465,47.93,370,12460,12461],[8404,49.37,370,12460,12461],[12466,48.22,370,12460,12461],[12467,48.79,370,12460,12461],[8406,51.43,370,12460,12461],[12468,50.9,370,12460,12461],[12469,50.21,370,12460,12461],[12470,45.9,370,12460,12461],[8929,49.23,370,12460,12461],[12471,50.32,370,12460,12461],[12472,51.21,370,12460,12461],[10420,48.75,370,12460,12461],[10420,49.03,370,12473,12461]]},{"id":"logbook-ego4d-frame-accuracy-2026-paper","name":"Logbook · Ego4D (2026 paper)","group":"专业工作","metric":"Frame-level activity accuracy","unit":"%","public_history":true,"protocol":{"en":"Eight independent text models receive the same six audio-captioner configurations; each published score averages those six matched cascades. The Ego4D test comprises 163 uninterrupted sessions (82.1 hours); frame accuracy is micro-averaged over the reference timeline. Ten-second audio captions are grouped into independent ten-minute text windows; decoded greedily, with minute-level predicted boundaries and lenient schema parsing. A malformed/truncated model output yields no segments rather than being silently excluded. Ego4D/EgoLife activity labels are inferred from human video narrations by OLMo-2-1124-7B-SFT; SINS uses provided seventeen-class labels and fills unannotated time as absence. Ego4D splits hash participants without overlap. This is audio-derived segmentation, not description entailment or human listening accuracy.","zh":"八个独立文本模型各使用相同六种音频描述器配置，原表分数为六级联结果平均。Ego4D测试含163段连续记录（82.1 hours）；按参考时间线微平均逐帧标签准确率。10秒音频描述组成互不关联的10分钟窗口，贪心解码、分钟级边界并宽容解析格式。模型格式错误或截断产生空分段，不静默排除。Ego4D/EgoLife标签由OLMo-2-1124-7B-SFT根据人类视频叙述推断；SINS保留17类原标签，未标注区间填absence。Ego4D按参与者哈希划分且不重叠。本组是音频级联的分段准确率。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.07338v1","published_at":"2026-10-05","status":"paper_baselines"},"model_count":8,"result_count":8,"results":[[12474,0.117,370,12475,12476],[12477,0.179,370,12475,12476],[1651,0.367,370,12475,12476],[516,0.363,370,12475,12476],[786,0.308,370,12475,12476],[6250,0.37,370,12475,12476],[2129,0.373,370,12475,12476],[5616,0.361,370,12475,12476]]},{"id":"logbook-sins-frame-accuracy-2026-paper","name":"Logbook · SINS (2026 paper)","group":"专业工作","metric":"Frame-level activity accuracy","unit":"%","public_history":true,"protocol":{"en":"Eight independent text models receive the same six audio-captioner configurations; each published score averages those six matched cascades. The SINS test comprises 1 uninterrupted sessions (148.9 hours); frame accuracy is micro-averaged over the reference timeline. Ten-second audio captions are grouped into independent ten-minute text windows; decoded greedily, with minute-level predicted boundaries and lenient schema parsing. A malformed/truncated model output yields no segments rather than being silently excluded. Ego4D/EgoLife activity labels are inferred from human video narrations by OLMo-2-1124-7B-SFT; SINS uses provided seventeen-class labels and fills unannotated time as absence. Ego4D splits hash participants without overlap. This is audio-derived segmentation, not description entailment or human listening accuracy.","zh":"八个独立文本模型各使用相同六种音频描述器配置，原表分数为六级联结果平均。SINS测试含1段连续记录（148.9 hours）；按参考时间线微平均逐帧标签准确率。10秒音频描述组成互不关联的10分钟窗口，贪心解码、分钟级边界并宽容解析格式。模型格式错误或截断产生空分段，不静默排除。Ego4D/EgoLife标签由OLMo-2-1124-7B-SFT根据人类视频叙述推断；SINS保留17类原标签，未标注区间填absence。Ego4D按参与者哈希划分且不重叠。本组是音频级联的分段准确率。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.07338v1","published_at":"2026-10-05","status":"paper_baselines"},"model_count":8,"result_count":8,"results":[[12474,0.186,370,12475,12476],[12477,0.219,370,12475,12476],[1651,0.337,370,12475,12476],[516,0.331,370,12475,12476],[786,0.337,370,12475,12476],[6250,0.442,370,12475,12476],[2129,0.513,370,12475,12476],[5616,0.467,370,12475,12476]]},{"id":"logbook-egolife-frame-accuracy-2026-paper","name":"Logbook · EgoLife (2026 paper)","group":"专业工作","metric":"Frame-level activity accuracy","unit":"%","public_history":true,"protocol":{"en":"Eight independent text models receive the same six audio-captioner configurations; each published score averages those six matched cascades. The EgoLife test comprises 170 uninterrupted sessions (240.6 hours); frame accuracy is micro-averaged over the reference timeline. Ten-second audio captions are grouped into independent ten-minute text windows; decoded greedily, with minute-level predicted boundaries and lenient schema parsing. A malformed/truncated model output yields no segments rather than being silently excluded. Ego4D/EgoLife activity labels are inferred from human video narrations by OLMo-2-1124-7B-SFT; SINS uses provided seventeen-class labels and fills unannotated time as absence. Ego4D splits hash participants without overlap. This is audio-derived segmentation, not description entailment or human listening accuracy.","zh":"八个独立文本模型各使用相同六种音频描述器配置，原表分数为六级联结果平均。EgoLife测试含170段连续记录（240.6 hours）；按参考时间线微平均逐帧标签准确率。10秒音频描述组成互不关联的10分钟窗口，贪心解码、分钟级边界并宽容解析格式。模型格式错误或截断产生空分段，不静默排除。Ego4D/EgoLife标签由OLMo-2-1124-7B-SFT根据人类视频叙述推断；SINS保留17类原标签，未标注区间填absence。Ego4D按参与者哈希划分且不重叠。本组是音频级联的分段准确率。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.07338v1","published_at":"2026-10-05","status":"paper_baselines"},"model_count":8,"result_count":8,"results":[[12474,0.117,370,12475,12476],[12477,0.17,370,12475,12476],[1651,0.352,370,12475,12476],[516,0.369,370,12475,12476],[786,0.345,370,12475,12476],[6250,0.326,370,12475,12476],[2129,0.334,370,12475,12476],[5616,0.336,370,12475,12476]]},{"id":"duplexact-wait-immediate-2026-paper","name":"DuplexAct · Wait (2026 paper)","group":"专业工作","metric":"Immediate action success","unit":"%","public_history":true,"protocol":{"en":"Fourteen independently implemented speech systems receive the same Direct Replay audio, with 600 instances for Wait. There are 200 instances per associated L2 condition, cross-checked against Appendix Table 16 immediate-success counts and rates. Immediate success means: remain silent while the user retains the floor, subject to the specified wait interval. Rates micro-average across the action’s L2 conditions. Audio is 16-kHz mono PCM resampled only as required; official/recommended inference settings are used, with synchronized input/output timelines. The second slash component is subsequent follow-through conditional on immediate success; it is preserved in the original table but not imported. These fixed-audio cases do not use the separate adaptive Simulator Replay track. Versions/checkpoints are listed in Table 14; proprietary APIs were evaluated September 6–13, 2026.","zh":"14个独立语音系统接收相同Direct Replay固定音频，Wait共600例；每个相关L2条件200例，已用附录表16成功数与比例交叉核对。即时成功须满足表3的该动作及时间约束，跨条件微平均。输入为16kHz单声道PCM，仅按系统需要重采样，采用官方或推荐推理配置并对齐时间线。斜线第二项是以即时成功为条件的后续行为率，保留原表但不导入。固定音频轨与自适应Simulator Replay分开。表14列检查点与版本，闭源接口运行于2026年9月6日至13日。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.17738v1","published_at":"2026-10-13","status":"paper_baselines"},"model_count":14,"result_count":14,"results":[[6601,0.8220000000000001,370,12478,12479],[6603,0.732,370,12478,12479],[12480,0.087,370,12478,12479],[6938,0.9179999999999999,370,12478,12479],[12481,0.9620000000000001,370,12478,12479],[12482,0.955,370,12478,12479],[12483,0.517,370,12478,12479],[12484,0.9279999999999999,370,12478,12479],[12485,0.982,370,12478,12479],[6574,0.9520000000000001,370,12478,12479],[8317,0.98,370,12478,12479],[6659,0.998,370,12478,12479],[12486,0.992,370,12478,12479],[12487,0.893,370,12478,12479]]},{"id":"duplexact-ignore-immediate-2026-paper","name":"DuplexAct · Ignore (2026 paper)","group":"专业工作","metric":"Immediate action success","unit":"%","public_history":true,"protocol":{"en":"Fourteen independently implemented speech systems receive the same Direct Replay audio, with 400 instances for Ignore. There are 200 instances per associated L2 condition, cross-checked against Appendix Table 16 immediate-success counts and rates. Immediate success means: remain silent to irrelevant speech while the user holds the floor. Rates micro-average across the action’s L2 conditions. Audio is 16-kHz mono PCM resampled only as required; official/recommended inference settings are used, with synchronized input/output timelines. The second slash component is subsequent follow-through conditional on immediate success; it is preserved in the original table but not imported. These fixed-audio cases do not use the separate adaptive Simulator Replay track. Versions/checkpoints are listed in Table 14; proprietary APIs were evaluated September 6–13, 2026.","zh":"14个独立语音系统接收相同Direct Replay固定音频，Ignore共400例；每个相关L2条件200例，已用附录表16成功数与比例交叉核对。即时成功须满足表3的该动作及时间约束，跨条件微平均。输入为16kHz单声道PCM，仅按系统需要重采样，采用官方或推荐推理配置并对齐时间线。斜线第二项是以即时成功为条件的后续行为率，保留原表但不导入。固定音频轨与自适应Simulator Replay分开。表14列检查点与版本，闭源接口运行于2026年9月6日至13日。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.17738v1","published_at":"2026-10-13","status":"paper_baselines"},"model_count":14,"result_count":14,"results":[[6601,0.325,370,12478,12479],[6603,0.615,370,12478,12479],[12480,0.11800000000000001,370,12478,12479],[6938,0.555,370,12478,12479],[12481,0.99,370,12478,12479],[12482,0.8420000000000001,370,12478,12479],[12483,0.47,370,12478,12479],[12484,0.78,370,12478,12479],[12485,0.8079999999999999,370,12478,12479],[6574,0.728,370,12478,12479],[8317,0.79,370,12478,12479],[6659,0.935,370,12478,12479],[12486,0.985,370,12478,12479],[12487,0.685,370,12478,12479]]},{"id":"duplexact-take-turn-immediate-2026-paper","name":"DuplexAct · Take Turn (2026 paper)","group":"专业工作","metric":"Immediate action success","unit":"%","public_history":true,"protocol":{"en":"Fourteen independently implemented speech systems receive the same Direct Replay audio, with 600 instances for Take Turn. There are 200 instances per associated L2 condition, cross-checked against Appendix Table 16 immediate-success counts and rates. Immediate success means: begin a substantive response in the specified response interval. Rates micro-average across the action’s L2 conditions. Audio is 16-kHz mono PCM resampled only as required; official/recommended inference settings are used, with synchronized input/output timelines. The second slash component is subsequent follow-through conditional on immediate success; it is preserved in the original table but not imported. These fixed-audio cases do not use the separate adaptive Simulator Replay track. Versions/checkpoints are listed in Table 14; proprietary APIs were evaluated September 6–13, 2026.","zh":"14个独立语音系统接收相同Direct Replay固定音频，Take Turn共600例；每个相关L2条件200例，已用附录表16成功数与比例交叉核对。即时成功须满足表3的该动作及时间约束，跨条件微平均。输入为16kHz单声道PCM，仅按系统需要重采样，采用官方或推荐推理配置并对齐时间线。斜线第二项是以即时成功为条件的后续行为率，保留原表但不导入。固定音频轨与自适应Simulator Replay分开。表14列检查点与版本，闭源接口运行于2026年9月6日至13日。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.17738v1","published_at":"2026-10-13","status":"paper_baselines"},"model_count":14,"result_count":14,"results":[[6601,0.36200000000000004,370,12478,12479],[6603,0.102,370,12478,12479],[12480,0.12,370,12478,12479],[6938,0.08,370,12478,12479],[12481,0.44299999999999995,370,12478,12479],[12482,0.35200000000000004,370,12478,12479],[12483,0.652,370,12478,12479],[12484,0.11699999999999999,370,12478,12479],[12485,0.20800000000000002,370,12478,12479],[6574,0.37799999999999995,370,12478,12479],[8317,0.38299999999999995,370,12478,12479],[6659,0.275,370,12478,12479],[12486,0.303,370,12478,12479],[12487,0.413,370,12478,12479]]},{"id":"duplexact-backchannel-immediate-2026-paper","name":"DuplexAct · Backchannel (2026 paper)","group":"专业工作","metric":"Immediate action success","unit":"%","public_history":true,"protocol":{"en":"Fourteen independently implemented speech systems receive the same Direct Replay audio, with 400 instances for Backchannel. There are 200 instances per associated L2 condition, cross-checked against Appendix Table 16 immediate-success counts and rates. Immediate success means: produce a brief function-appropriate feedback signal without taking the floor. Rates micro-average across the action’s L2 conditions. Audio is 16-kHz mono PCM resampled only as required; official/recommended inference settings are used, with synchronized input/output timelines. The second slash component is subsequent follow-through conditional on immediate success; it is preserved in the original table but not imported. These fixed-audio cases do not use the separate adaptive Simulator Replay track. Versions/checkpoints are listed in Table 14; proprietary APIs were evaluated September 6–13, 2026.","zh":"14个独立语音系统接收相同Direct Replay固定音频，Backchannel共400例；每个相关L2条件200例，已用附录表16成功数与比例交叉核对。即时成功须满足表3的该动作及时间约束，跨条件微平均。输入为16kHz单声道PCM，仅按系统需要重采样，采用官方或推荐推理配置并对齐时间线。斜线第二项是以即时成功为条件的后续行为率，保留原表但不导入。固定音频轨与自适应Simulator Replay分开。表14列检查点与版本，闭源接口运行于2026年9月6日至13日。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.17738v1","published_at":"2026-10-13","status":"paper_baselines"},"model_count":14,"result_count":14,"results":[[6601,0.062,370,12478,12479],[6603,0.002,370,12478,12479],[12480,0.0,370,12478,12479],[6938,0.0,370,12478,12479],[12481,0.015,370,12478,12479],[12482,0.002,370,12478,12479],[12483,0.31,370,12478,12479],[12484,0.002,370,12478,12479],[12485,0.022000000000000002,370,12478,12479],[6574,0.025,370,12478,12479],[8317,0.01,370,12478,12479],[6659,0.002,370,12478,12479],[12486,0.008,370,12478,12479],[12487,0.038,370,12478,12479]]},{"id":"nldata2opt-nlp4lp-ebc-2026-paper","name":"NLData2Opt · NLP4LP (2026 paper)","group":"数学与科学","metric":"Execution-based correctness","unit":"score","public_history":true,"protocol":{"en":"Thirteen configurations of twelve bases solve the NL+Data task on a common seed-42 random 15% sample of the 269-problem NLP4LP source. The paper specifies the sampling fraction and seed, not a rounded integer subset count. Natural-language constraints and numeric CSV data are separate inputs. Each model gets one attempt at temperature 0.1; outputs are reformatted into AMPL model/data files and executed by AMPL/CBC. Non-executable formulations score zero. Executable objective O receives max(0,1-|O-A|/|A|); zero-reference objectives score one only if O=0. The mean continuous correctness is a native 0–1 score, not exact success percentage. GPT-4o-mini five-element prompting shares its base with the direct baseline; CoE uses GPT-3.5-turbo and SFT uses Qwen3-1.7B. Full-cohort reruns cover only three models and remain separate.","zh":"13配置、12底模在NLP4LP原始269题中使用相同seed42随机15%子集；论文给出比例和种子，未取整编造子集题数。NL+Data把自然语言约束与CSV数值分开输入。每模型温度0.1单次尝试，输出重排成AMPL模型与数据文件，用AMPL/CBC执行。不可执行记零；可执行目标值按max(0,1-|O-A|/|A|)连续评分，参考值为零时仅O=0得1。均值为原生0至1连续正确度，不作精确成功率。五要素方法归GPT-4o-mini，CoE为GPT-3.5-turbo，SFT为Qwen3-1.7B。完整数据重跑仅三模型，另行保留。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.22889v1","published_at":"2026-10-16","status":"paper_baselines"},"score_max":1,"score_decimals":2,"saturation_excluded":true,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":12,"result_count":13,"results":[[2839,0.4,370,12488,12489],[353,0.15,370,12488,12489],[2602,0.0,370,12488,12489],[325,0.733,370,12488,12489],[3572,0.29,370,12488,12489],[2261,0.699,370,12488,12489],[562,0.0,370,12488,12489],[12490,0.0,370,12488,12489],[516,0.025,370,12488,12489],[12491,0.1,370,12488,12489],[354,0.4,370,12492,12489],[12493,0.516,370,12494,12489],[7590,0.483,370,12495,12489]]},{"id":"nldata2opt-mamo-easy-ebc-2026-paper","name":"NLData2Opt · Mamo Easy (2026 paper)","group":"数学与科学","metric":"Execution-based correctness","unit":"score","public_history":true,"protocol":{"en":"Thirteen configurations of twelve bases solve the NL+Data task on a common seed-42 random 15% sample of the 647-problem Mamo Easy source. The paper specifies the sampling fraction and seed, not a rounded integer subset count. Natural-language constraints and numeric CSV data are separate inputs. Each model gets one attempt at temperature 0.1; outputs are reformatted into AMPL model/data files and executed by AMPL/CBC. Non-executable formulations score zero. Executable objective O receives max(0,1-|O-A|/|A|); zero-reference objectives score one only if O=0. The mean continuous correctness is a native 0–1 score, not exact success percentage. GPT-4o-mini five-element prompting shares its base with the direct baseline; CoE uses GPT-3.5-turbo and SFT uses Qwen3-1.7B. Full-cohort reruns cover only three models and remain separate.","zh":"13配置、12底模在Mamo Easy原始647题中使用相同seed42随机15%子集；论文给出比例和种子，未取整编造子集题数。NL+Data把自然语言约束与CSV数值分开输入。每模型温度0.1单次尝试，输出重排成AMPL模型与数据文件，用AMPL/CBC执行。不可执行记零；可执行目标值按max(0,1-|O-A|/|A|)连续评分，参考值为零时仅O=0得1。均值为原生0至1连续正确度，不作精确成功率。五要素方法归GPT-4o-mini，CoE为GPT-3.5-turbo，SFT为Qwen3-1.7B。完整数据重跑仅三模型，另行保留。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.22889v1","published_at":"2026-10-16","status":"paper_baselines"},"score_max":1,"score_decimals":2,"saturation_excluded":true,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":12,"result_count":13,"results":[[2839,0.345,370,12488,12489],[353,0.639,370,12488,12489],[2602,0.0,370,12488,12489],[325,0.753,370,12488,12489],[3572,0.553,370,12488,12489],[2261,0.762,370,12488,12489],[562,0.0,370,12488,12489],[12490,0.0,370,12488,12489],[516,0.499,370,12488,12489],[12491,0.239,370,12488,12489],[354,0.6,370,12492,12489],[12493,0.381,370,12494,12489],[7590,0.603,370,12495,12489]]},{"id":"nldata2opt-mamo-complex-ebc-2026-paper","name":"NLData2Opt · Mamo Complex (2026 paper)","group":"数学与科学","metric":"Execution-based correctness","unit":"score","public_history":true,"protocol":{"en":"Thirteen configurations of twelve bases solve the NL+Data task on a common seed-42 random 15% sample of the 210-problem Mamo Complex source. The paper specifies the sampling fraction and seed, not a rounded integer subset count. Natural-language constraints and numeric CSV data are separate inputs. Each model gets one attempt at temperature 0.1; outputs are reformatted into AMPL model/data files and executed by AMPL/CBC. Non-executable formulations score zero. Executable objective O receives max(0,1-|O-A|/|A|); zero-reference objectives score one only if O=0. The mean continuous correctness is a native 0–1 score, not exact success percentage. GPT-4o-mini five-element prompting shares its base with the direct baseline; CoE uses GPT-3.5-turbo and SFT uses Qwen3-1.7B. Full-cohort reruns cover only three models and remain separate.","zh":"13配置、12底模在Mamo Complex原始210题中使用相同seed42随机15%子集；论文给出比例和种子，未取整编造子集题数。NL+Data把自然语言约束与CSV数值分开输入。每模型温度0.1单次尝试，输出重排成AMPL模型与数据文件，用AMPL/CBC执行。不可执行记零；可执行目标值按max(0,1-|O-A|/|A|)连续评分，参考值为零时仅O=0得1。均值为原生0至1连续正确度，不作精确成功率。五要素方法归GPT-4o-mini，CoE为GPT-3.5-turbo，SFT为Qwen3-1.7B。完整数据重跑仅三模型，另行保留。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.22889v1","published_at":"2026-10-16","status":"paper_baselines"},"score_max":1,"score_decimals":2,"saturation_excluded":true,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":12,"result_count":13,"results":[[2839,0.393,370,12488,12489],[353,0.612,370,12488,12489],[2602,0.0,370,12488,12489],[325,0.63,370,12488,12489],[3572,0.794,370,12488,12489],[2261,0.853,370,12488,12489],[562,0.0,370,12488,12489],[12490,0.0,370,12488,12489],[516,0.191,370,12488,12489],[12491,0.253,370,12488,12489],[354,0.534,370,12492,12489],[12493,0.092,370,12494,12489],[7590,0.129,370,12495,12489]]},{"id":"nepplural-gold-persona-f1-2026-paper","name":"NepPlural · persona classification (2026 paper)","group":"专业工作","metric":"Mean four-axis gold macro-F1","unit":"%","public_history":true,"protocol":{"en":"Eight independent bases classify the same 300 human-gold Nepali migration comments over four persona axes, with No-Persona as an explicit fifth class per axis. Each score is mean macro-F1 across the four axes against adjudicated native-speaker gold, not the model-specific LOO-P subset. Three LLMs use per-comment zero-shot inference with abstention; five encoders are fine-tuned on 1,661 separate comments labelled by the weaker batched LLM protocol, validated on 184 comments, with five-seed means. Training supervision differs and is explicit. Table 5 explicitly uses the same Table 4 metrics and gold panel. Human and majority-vote references are excluded from the independent-model cohort; both complete original tables are retained.","zh":"八个独立底模分类同一300条人类金标尼泊尔迁移话题评论，四个人设轴各含No-Persona第五类。分数为对裁定金标的四轴宏F1平均，不用模型条件子集LOO-P。三LLM逐条零样本且允许弃答；五编码器用另1661条较弱批量LLM标签训练、184条验证，报告五种子均值。训练监督差异在配置中保留。表5明确沿用表4指标及同一金标面板；人类及多数投票参考不算底模，完整保留两张原表。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.15732v1","published_at":"2026-10-11","status":"paper_baselines"},"model_count":8,"result_count":8,"results":[[340,0.7,370,12496,12497],[2129,0.75,370,12496,12497],[6978,0.67,370,12496,12497],[12498,0.34,370,12499,12500],[12501,0.34,370,12499,12500],[5575,0.21,370,12499,12500],[12502,0.2,370,12499,12500],[12503,0.2,370,12499,12500]]},{"id":"spec2game-core15-overall-2026-paper","name":"Spec2Game · Core-15 (2026 paper)","group":"Agent 与工具","metric":"Family-balanced Overall","unit":"score","public_history":true,"protocol":{"en":"Fourteen bases each generate three independent Pygame projects for each of fifteen canonical game families: 45 projects per model and 630 total. Scores average three replicas within family, then weight all fifteen families equally. Overall=0.20 D1+clip((D1−40)/20,0,1)×(0.50 D2+0.15 D3+0.15 D4), a native 0–100 gated composite. D1 uses six sequential execution checks; D2 tests specified behavior by controlled action programs and checkpoint rubrics; D3 combines static checks and code judging; D4 uses runtime screenshots and fixed visual/actionability rubrics. Text judge Qwen3-235B-A22B-2507 and vision judge Qwen3-VL-235B-A22B-Instruct use temperature zero, reasoning disabled, and share family with three candidates. Infrastructure-invalid judging remains incomplete rather than zero; actual unavailable candidate evidence follows the frozen zero rule. Full-150 is a separately generated six-model track, not mixed into Core-15.","zh":"14底模各为15个标准游戏家族独立生成3个Pygame项目，每模型45个、共630个。先平均家族内三次，再对15家族等权。Overall为0.20D1+clip((D1−40)/20,0,1)×(0.50D2+0.15D3+0.15D4)，保留0至100门控综合量表。D1含六个顺序执行检查，D2通过受控动作及检查点评分，D3合并静态检查与代码评审，D4使用运行截图和固定视觉、可操作性量表。文本裁判Qwen3-235B-A22B-2507、视觉裁判Qwen3-VL-235B-A22B-Instruct均温度零且禁用思考，与三个候选同家族。裁判基础设施失败保留未完成，真实候选缺失证据才按冻结规则计零。Full-150为另生成的六模型轨，不混入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.04253v1","published_at":"2026-10-03","status":"paper_baselines"},"score_max":100,"score_decimals":2,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":14,"result_count":14,"results":[[3568,77.27,370,12504,12505],[325,74.91,370,12504,12505],[8413,73.6,370,12504,12505],[3493,73.37,370,12504,12505],[12506,71.95,370,12504,12505],[3408,71.35,370,12504,12505],[2973,69.26,370,12504,12505],[762,67.92,370,12504,12505],[3260,59.68,370,12504,12505],[3512,59.12,370,12504,12505],[3490,57.68,370,12504,12505],[227,57.26,370,12504,12505],[5734,55.8,370,12504,12505],[8969,39.11,370,12504,12505]]},{"id":"p3ibench-prs-unidirectional-2026-paper","name":"P3IBench · unidirectional timing (2026 paper)","group":"视觉与视频","metric":"Proactive Response Score","unit":"score","public_history":true,"protocol":{"en":"Twelve bases run the common 533-query benchmark from 429 videos, containing 836 annotated response targets. This unidirectional PRS first scores response correctness times exp(−0.05×absolute timing error), then takes the best response assigned to a target divided by the number assigned; a missed target scores zero. Targets average within video, videos within task, Event averages its three subsets and Overall weights five tasks equally. Unidirectional mapping uses the latest preceding target and zeros premature first-target outputs; bidirectional mapping uses the nearest target. Structured responses use deterministic checks; open intervention responses use a semantic LLM rubric. The official pinned repository 38de347afcc73a1473196508c288a5d5cc752320 supplies the rubric 0/0.5/1 and default Qwen3-32B-FP8 judge; the paper does not explicitly bind that default to every reported run. Streaming is causal at 1 FPS; GPT-5.6-Luna retains 16 images and InternVL3.5-8B 32, other offline models all observed images. Native streaming systems retain their own execution framework.","zh":"12底模运行同533查询、429视频、836标注响应目标。PRS先以回答正确度乘exp(−0.05×时间绝对误差)，每目标取已匹配回答最高分再除回答数；漏答为零。目标在视频内平均，再跨视频与任务平均；Event三子集等权、Overall五任务等权。单向分配到最近先前目标，首目标前回答为零；双向分配到时间最近目标。结构化回答规则评分，干预回答用LLM语义量表。已固定官方仓库38de347afcc73a1473196508c288a5d5cc752320，代码提供0/0.5/1量表及默认Qwen3-32B-FP8裁判；论文未将该默认值明确绑定至全部主表运行。1FPS因果流式输入，GPT5.6Luna保留16图、InternVL3.5保留32图，其余离线模型保留所有已见图；原生流式系统保留自身执行框架。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.21568v1","published_at":"2026-10-15","status":"paper_baselines"},"score_max":100,"score_decimals":2,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":12,"result_count":12,"results":[[9885,9.62,370,12507,12508],[2654,21.27,370,12507,12508],[2478,10.69,370,12507,12508],[6149,24.57,370,12507,12508],[6150,27.79,370,12507,12508],[4969,20.84,370,12507,12508],[4070,21.95,370,12507,12508],[5616,38.78,370,12507,12508],[12509,18.34,370,12507,12508],[12510,21.36,370,12507,12508],[7226,8.84,370,12507,12508],[12511,23.13,370,12507,12508]]},{"id":"p3ibench-prs-bidirectional-2026-paper","name":"P3IBench · bidirectional timing (2026 paper)","group":"视觉与视频","metric":"Proactive Response Score","unit":"score","public_history":true,"protocol":{"en":"Twelve bases run the common 533-query benchmark from 429 videos, containing 836 annotated response targets. This bidirectional PRS first scores response correctness times exp(−0.05×absolute timing error), then takes the best response assigned to a target divided by the number assigned; a missed target scores zero. Targets average within video, videos within task, Event averages its three subsets and Overall weights five tasks equally. Unidirectional mapping uses the latest preceding target and zeros premature first-target outputs; bidirectional mapping uses the nearest target. Structured responses use deterministic checks; open intervention responses use a semantic LLM rubric. The official pinned repository 38de347afcc73a1473196508c288a5d5cc752320 supplies the rubric 0/0.5/1 and default Qwen3-32B-FP8 judge; the paper does not explicitly bind that default to every reported run. Streaming is causal at 1 FPS; GPT-5.6-Luna retains 16 images and InternVL3.5-8B 32, other offline models all observed images. Native streaming systems retain their own execution framework.","zh":"12底模运行同533查询、429视频、836标注响应目标。PRS先以回答正确度乘exp(−0.05×时间绝对误差)，每目标取已匹配回答最高分再除回答数；漏答为零。目标在视频内平均，再跨视频与任务平均；Event三子集等权、Overall五任务等权。单向分配到最近先前目标，首目标前回答为零；双向分配到时间最近目标。结构化回答规则评分，干预回答用LLM语义量表。已固定官方仓库38de347afcc73a1473196508c288a5d5cc752320，代码提供0/0.5/1量表及默认Qwen3-32B-FP8裁判；论文未将该默认值明确绑定至全部主表运行。1FPS因果流式输入，GPT5.6Luna保留16图、InternVL3.5保留32图，其余离线模型保留所有已见图；原生流式系统保留自身执行框架。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.21568v1","published_at":"2026-10-15","status":"paper_baselines"},"score_max":100,"score_decimals":2,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":12,"result_count":12,"results":[[9885,10.2,370,12512,12508],[2654,26.79,370,12512,12508],[2478,16.7,370,12512,12508],[6149,28.62,370,12512,12508],[6150,41.73,370,12512,12508],[4969,26.16,370,12512,12508],[4070,26.11,370,12512,12508],[5616,46.59,370,12512,12508],[12509,27.94,370,12512,12508],[12510,31.62,370,12512,12508],[7226,8.86,370,12512,12508],[12511,28.11,370,12512,12508]]},{"id":"omicsbench-opencode-single-pass-2026-paper","name":"OmicsBench · single-pass (2026 paper)","group":"数学与科学","metric":"Mean deterministic task score","unit":"score","score_max":1,"score_decimals":2,"saturation_excluded":true,"scale_label":{"en":"Native score / 1","zh":"原生分数 / 1"},"public_history":true,"protocol":{"en":"Twelve independent models on the same OpenCode harness complete all 97 study-scale tasks and 662 subtasks, with three runs per task in single-pass mode. Each subtask averages deterministic answer-file graders; task scores average subtasks, and the reported mean first averages three runs within task then weights 97 tasks equally. Gated mode permits up to three attempts until score ≥0.7; single-pass has no feedback. Best-attempt workspace is restored before dependent turns. Every turn has 1800 seconds in a fresh offline Docker environment. High reasoning is used where supported except MiniMax. Three tasks are public and 94 remain private. Two extra native-harness configurations in the full figure are preserved in the page image and excluded from the common-OpenCode panel. Scores are the printed two-decimal plot labels, a native partial-credit composite rather than percentage success.","zh":"12个独立模型在相同OpenCode中运行全部97研究规模任务、662子任务，每任务每模式3次。子任务平均确定性答案文件评分器，任务平均子任务；主分数先平均同任务三次，再对97任务等权。gated模式允许最多3次尝试至≥0.7，single-pass无反馈；依赖轮次前恢复最佳尝试工作区。每轮限1800秒，离线Docker环境；除MiniMax外尽可能设高思考。3任务公开、94私有。原图另两个原生harness配置保留在完整页面，不混入共同OpenCode面板。保留图中两位小数原值，属于部分得分综合分，不作成功百分比。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.21074v1","published_at":"2026-10-15","status":"paper_baselines"},"model_count":12,"result_count":12,"results":[[755,0.57,370,12513,12514],[762,0.56,370,12513,12514],[5295,0.55,370,12513,12514],[12515,0.54,370,12513,12514],[2977,0.49,370,12513,12514],[227,0.48,370,12513,12514],[3157,0.48,370,12513,12514],[502,0.47,370,12513,12514],[3198,0.46,370,12513,12514],[3185,0.45,370,12513,12514],[3202,0.44,370,12513,12514],[5411,0.43,370,12513,12514]]},{"id":"omicsbench-opencode-gated-2026-paper","name":"OmicsBench · gated (2026 paper)","group":"数学与科学","metric":"Mean deterministic task score","unit":"score","score_max":1,"score_decimals":2,"saturation_excluded":true,"scale_label":{"en":"Native score / 1","zh":"原生分数 / 1"},"public_history":true,"protocol":{"en":"Twelve independent models on the same OpenCode harness complete all 97 study-scale tasks and 662 subtasks, with three runs per task in gated mode. Each subtask averages deterministic answer-file graders; task scores average subtasks, and the reported mean first averages three runs within task then weights 97 tasks equally. Gated mode permits up to three attempts until score ≥0.7; single-pass has no feedback. Best-attempt workspace is restored before dependent turns. Every turn has 1800 seconds in a fresh offline Docker environment. High reasoning is used where supported except MiniMax. Three tasks are public and 94 remain private. Two extra native-harness configurations in the full figure are preserved in the page image and excluded from the common-OpenCode panel. Scores are the printed two-decimal plot labels, a native partial-credit composite rather than percentage success.","zh":"12个独立模型在相同OpenCode中运行全部97研究规模任务、662子任务，每任务每模式3次。子任务平均确定性答案文件评分器，任务平均子任务；主分数先平均同任务三次，再对97任务等权。gated模式允许最多3次尝试至≥0.7，single-pass无反馈；依赖轮次前恢复最佳尝试工作区。每轮限1800秒，离线Docker环境；除MiniMax外尽可能设高思考。3任务公开、94私有。原图另两个原生harness配置保留在完整页面，不混入共同OpenCode面板。保留图中两位小数原值，属于部分得分综合分，不作成功百分比。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.21074v1","published_at":"2026-10-15","status":"paper_baselines"},"model_count":12,"result_count":12,"results":[[755,0.67,370,12516,12514],[762,0.66,370,12516,12514],[5295,0.66,370,12516,12514],[12515,0.63,370,12516,12514],[2977,0.59,370,12516,12514],[227,0.59,370,12516,12514],[3157,0.59,370,12516,12514],[502,0.58,370,12516,12514],[3198,0.58,370,12516,12514],[3185,0.58,370,12516,12514],[3202,0.56,370,12516,12514],[5411,0.54,370,12516,12514]]},{"id":"quasileak-policy-group-f1-2026-paper","name":"QuasiLeak · policy-conditioned recovery (2026 paper)","group":"安全与可靠性","metric":"Policy group F1","unit":"score","score_max":100,"score_decimals":2,"saturation_excluded":true,"scale_label":{"en":"Native score / 100","zh":"原生分数 / 100"},"public_history":true,"protocol":{"en":"All 28 configurations of 21 underlying models use the same 5,000 synthetic sender–recipient documents, full author profiles and sender disclosure policies. The same three solved examples, outside the evaluation corpus, are supplied in the same order. Token-span IoU≥0.75 gives one-to-one group alignment; credit combines squared evidence completeness, cue localization, group binding and correct attribute binding, with half credit for verified secondary attributes. Only HIDE evidence earns policy credit; KEEP disclosures receive an extra false-positive penalty λK=1. Policy group F1=2×credited HIDE groups/(prediction denominator + gold HIDE groups), retained as a native partial-credit score out of 100. Parsed evidence remains a prediction even if it scores zero; malformed responses become empty outputs. Gemma-4-E4B thinking-off actually returns empty evidence on 4,999/5,000 cases, so its zero is executed abstention. Thinking switches are configurations of the same base. Complete main Table3 and larger-band Table22 are retained; the later supervised 1000-document test split is separate.","zh":"同5000份合成收发消息、完整作者档案与披露策略，28配置归21底模。三份解题示例均来自评估库外，顺序相同。以词元片段IoU≥0.75进行组间一对一匹配；信用合并平方证据完整度、定位、组绑定与属性绑定，验证过的次属性给半分。仅HIDE组获策略信用，误报KEEP加一次假阳性惩罚λK=1。P-GF1=2×获信用HIDE组/(预测分母+金标HIDE组)，保留0至100部分得分量表。可解析证据即使得零仍计预测，格式错误作为空输出。Gemma-4-E4B禁思考在4999/5000例实际输出空证据，零分来自执行过的弃答。思考开关不算新底模。完整保留主表3与大模型表22；后续监督训练的1000测试文档另计。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.21333v1","published_at":"2026-10-15","status":"paper_baselines"},"model_count":21,"result_count":28,"results":[[762,75.4,370,12517,12518],[4983,67.9,370,12517,12518],[5519,71.8,370,12519,12518],[3185,53.8,370,12517,12518],[4986,64.5,370,12517,12518],[6301,17.5,370,12520,12518],[6301,5.8,370,12521,12518],[1621,5.8,370,12521,12518],[7345,2.6,370,12521,12518],[4968,0.7,370,12522,12518],[4968,24.2,370,12523,12518],[552,2.2,370,12524,12518],[4070,6.9,370,12525,12518],[4070,5.4,370,12524,12518],[587,2.1,370,12526,12518],[12527,5.2,370,12526,12518],[5650,6.8,370,12528,12518],[5650,0.0,370,12529,12518],[4969,7.7,370,12528,12518],[4969,3.3,370,12529,12518],[12530,1.2,370,12531,12518],[613,0.1,370,12532,12518],[10724,4.1,370,12533,12518],[5480,7.6,370,12534,12518],[5480,46.0,370,12535,12518],[2584,3.8,370,12536,12518],[2584,33.4,370,12537,12518],[8097,11.6,370,12538,12518]]},{"id":"kex-bench-linux-without-poc-2026-paper","name":"KEX-bench · linux without-poc (2026 paper)","group":"编程","metric":"Solved / 25","unit":"raw","public_history":true,"protocol":{"en":"Eleven agent/model configurations (eight base models) attempt the same 25 Linux tasks, one bounded trajectory each. This without-poc column reports the native number solved out of 25, not a percentage. Tool-call budget is 200; handshake calls are excluded. Agent frameworks remain explicit. Platform and PoC conditions have different tasks/resources and stay separate; 990 reported outcomes repeat 45 unique tasks rather than representing 990 independent problems.","zh":"十一种Agent/模型配置、八个底模在相同25道Linux任务上各执行一条有预算上限的轨迹。本组without-poc列保留原生解题数，满分25题。工具调用上限200次，不计握手；保留Agent框架。平台与PoC条件的任务和资源不同，分别排行。原文990条配置结果对应45道独立任务。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.25591v1","published_at":"2026-09-22","status":"paper_baselines"},"model_count":8,"result_count":11,"results":[[12539,14.0,370,12540,12541],[12542,7.0,370,12540,12541],[12543,0.0,370,12540,12541],[3184,5.0,370,12544,12541],[12545,1.0,370,12544,12541],[3733,3.0,370,12544,12541],[12542,4.0,370,12546,12541],[12543,0.0,370,12546,12541],[12545,0.0,370,12546,12541],[12547,0.0,370,12546,12541],[9370,0.0,370,12546,12541]]},{"id":"kex-bench-linux-with-poc-2026-paper","name":"KEX-bench · linux with-poc (2026 paper)","group":"编程","metric":"Solved / 25","unit":"raw","public_history":true,"protocol":{"en":"Eleven agent/model configurations (eight base models) attempt the same 25 Linux tasks, one bounded trajectory each. This with-poc column reports the native number solved out of 25, not a percentage. Tool-call budget is 200; handshake calls are excluded. Agent frameworks remain explicit. Platform and PoC conditions have different tasks/resources and stay separate; 990 reported outcomes repeat 45 unique tasks rather than representing 990 independent problems.","zh":"十一种Agent/模型配置、八个底模在相同25道Linux任务上各执行一条有预算上限的轨迹。本组with-poc列保留原生解题数，满分25题。工具调用上限200次，不计握手；保留Agent框架。平台与PoC条件的任务和资源不同，分别排行。原文990条配置结果对应45道独立任务。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.25591v1","published_at":"2026-09-22","status":"paper_baselines"},"model_count":8,"result_count":11,"results":[[12539,23.0,370,12548,12541],[12542,19.0,370,12548,12541],[12543,0.0,370,12548,12541],[3184,12.0,370,12549,12541],[12545,3.0,370,12549,12541],[3733,6.0,370,12549,12541],[12542,10.0,370,12550,12541],[12543,2.0,370,12550,12541],[12545,2.0,370,12550,12541],[12547,2.0,370,12550,12541],[9370,1.0,370,12550,12541]]},{"id":"kex-bench-windows-without-poc-2026-paper","name":"KEX-bench · windows without-poc (2026 paper)","group":"编程","metric":"Solved / 20","unit":"raw","public_history":true,"protocol":{"en":"Eleven agent/model configurations (eight base models) attempt the same 20 Windows tasks, one bounded trajectory each. This without-poc column reports the native number solved out of 20, not a percentage. Tool-call budget is 300; handshake calls are excluded. Agent frameworks remain explicit. Platform and PoC conditions have different tasks/resources and stay separate; 990 reported outcomes repeat 45 unique tasks rather than representing 990 independent problems.","zh":"十一种Agent/模型配置、八个底模在相同20道Windows任务上各执行一条有预算上限的轨迹。本组without-poc列保留原生解题数，满分20题。工具调用上限300次，不计握手；保留Agent框架。平台与PoC条件的任务和资源不同，分别排行。原文990条配置结果对应45道独立任务。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.25591v1","published_at":"2026-09-22","status":"paper_baselines"},"model_count":8,"result_count":11,"results":[[12539,1.0,370,12551,12541],[12542,0.0,370,12551,12541],[12543,0.0,370,12551,12541],[3184,1.0,370,12552,12541],[12545,0.0,370,12552,12541],[3733,1.0,370,12552,12541],[12542,0.0,370,12553,12541],[12543,0.0,370,12553,12541],[12545,0.0,370,12553,12541],[12547,0.0,370,12553,12541],[9370,0.0,370,12553,12541]]},{"id":"kex-bench-windows-with-poc-2026-paper","name":"KEX-bench · windows with-poc (2026 paper)","group":"编程","metric":"Solved / 20","unit":"raw","public_history":true,"protocol":{"en":"Eleven agent/model configurations (eight base models) attempt the same 20 Windows tasks, one bounded trajectory each. This with-poc column reports the native number solved out of 20, not a percentage. Tool-call budget is 300; handshake calls are excluded. Agent frameworks remain explicit. Platform and PoC conditions have different tasks/resources and stay separate; 990 reported outcomes repeat 45 unique tasks rather than representing 990 independent problems.","zh":"十一种Agent/模型配置、八个底模在相同20道Windows任务上各执行一条有预算上限的轨迹。本组with-poc列保留原生解题数，满分20题。工具调用上限300次，不计握手；保留Agent框架。平台与PoC条件的任务和资源不同，分别排行。原文990条配置结果对应45道独立任务。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.25591v1","published_at":"2026-09-22","status":"paper_baselines"},"model_count":8,"result_count":11,"results":[[12539,8.0,370,12554,12541],[12542,5.0,370,12554,12541],[12543,0.0,370,12554,12541],[3184,7.0,370,12555,12541],[12545,2.0,370,12555,12541],[3733,5.0,370,12555,12541],[12542,6.0,370,12556,12541],[12543,0.0,370,12556,12541],[12545,0.0,370,12556,12541],[12547,0.0,370,12556,12541],[9370,0.0,370,12556,12541]]},{"id":"mvcap-full-motion-2026-paper","name":"MVCAP-Bench · full-motion (2026 paper)","group":"Agent 与工具","metric":"Mean pass@1","unit":"%","public_history":true,"protocol":{"en":"Ten multimodal backbones with Browser Use and three native computer-use configurations solve the same 600 browser CAPTCHA instances (200 per level, six choices). This full-motion cohort reports the source rounded mean pass@1 across the three equally sized levels. Foreground-only removes background camouflage while retaining targets, prompts, options and interface; it stays separate from full motion. Human and random controls remain in source evidence, outside the model ranking.","zh":"十个多模态底模的Browser Use及三种原生Computer Use配置，回答相同600道浏览器验证码，每级200题、六个选项。本组full-motion采用原文三个等量等级的平均pass@1。前景条件保留目标、提示、选项及界面，仅去除背景伪装，与完整动态条件分别排行。人工与随机对照保留在证据中。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.27461v1","published_at":"2026-09-23","status":"paper_baselines"},"model_count":10,"result_count":13,"results":[[4064,0.158,370,12557,12558],[3184,0.168,370,12557,12558],[7749,0.157,370,12557,12558],[12559,0.153,370,12557,12558],[6133,0.158,370,12557,12558],[502,0.138,370,12557,12558],[7674,0.163,370,12557,12558],[5722,0.145,370,12557,12558],[2998,0.15,370,12557,12558],[6412,0.14800000000000002,370,12557,12558],[3184,0.20800000000000002,370,12560,12558],[7749,0.152,370,12560,12558],[6133,0.157,370,12560,12558]]},{"id":"mvcap-foreground-only-2026-paper","name":"MVCAP-Bench · foreground-only (2026 paper)","group":"Agent 与工具","metric":"Mean pass@1","unit":"%","public_history":true,"protocol":{"en":"Ten multimodal backbones with Browser Use and three native computer-use configurations solve the same 600 browser CAPTCHA instances (200 per level, six choices). This foreground-only cohort reports the source rounded mean pass@1 across the three equally sized levels. Foreground-only removes background camouflage while retaining targets, prompts, options and interface; it stays separate from full motion. Human and random controls remain in source evidence, outside the model ranking.","zh":"十个多模态底模的Browser Use及三种原生Computer Use配置，回答相同600道浏览器验证码，每级200题、六个选项。本组foreground-only采用原文三个等量等级的平均pass@1。前景条件保留目标、提示、选项及界面，仅去除背景伪装，与完整动态条件分别排行。人工与随机对照保留在证据中。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.27461v1","published_at":"2026-09-23","status":"paper_baselines"},"model_count":10,"result_count":13,"results":[[4064,0.493,370,12561,12558],[3184,0.617,370,12561,12558],[7749,0.578,370,12561,12558],[12559,0.547,370,12561,12558],[6133,0.63,370,12561,12558],[502,0.68,370,12561,12558],[7674,0.537,370,12561,12558],[5722,0.397,370,12561,12558],[2998,0.465,370,12561,12558],[6412,0.5379999999999999,370,12561,12558],[3184,0.62,370,12562,12558],[7749,0.607,370,12562,12558],[6133,0.647,370,12562,12558]]},{"id":"legendbench-qa-2026-paper","name":"LegendBench · legend QA (2026 paper)","group":"视觉与视频","metric":"QA accuracy","unit":"%","public_history":true,"protocol":{"en":"Twelve source-named general VLMs and chart-specific models use the same prompt template and hyperparameters on generated legend-understanding tasks. The selected metric is overall question-answer accuracy, distinct from chain success, counterfactual consistency and abstention F1. Qwen3-32B/8B names are preserved exactly as reported; no unreported VL suffix or checkpoint identity is inferred.","zh":"十二个原文命名的通用VLM与图表专用模型，以相同提示模板和超参数回答图例理解任务。本组显示总体问答准确率，不混入链条成功率、反事实一致性和拒答F1。Qwen3-32B/8B保留原名，不补写原文未给出的VL后缀或检查点身份。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.24172v1","published_at":"2026-09-21","status":"paper_baselines"},"model_count":12,"result_count":12,"results":[[4048,0.8381000000000001,370,12563,12564],[2584,0.7162000000000001,370,12563,12564],[766,0.6781999999999999,370,12563,12564],[786,0.6609,370,12563,12564],[5722,0.6459,370,12563,12564],[5406,0.6419,370,12563,12564],[5344,0.5802,370,12563,12564],[12565,0.057800000000000004,370,12563,12564],[12566,0.1519,370,12563,12564],[2892,0.1377,370,12563,12564],[2885,0.1226,370,12563,12564],[12567,0.0907,370,12563,12564]]},{"id":"validated-detection-coco-2026-paper","name":"Validated detection · coco (2026 paper)","group":"视觉与视频","metric":"AP50:95","unit":"score","public_history":true,"protocol":{"en":"Six pretrained open-vocabulary and closed-set detector architectures are evaluated against the same newly validated coco annotations. Only the validated-ground-truth column is selected; original annotations are a different evaluation. The annotation pipeline aggregates at least eleven independent judgments per object to retain ambiguity; AP is a native ranking metric, not image accuracy. Different datasets remain separate.","zh":"六种预训练开放词汇及闭集检测架构，在相同新验证的coco标注上评测。仅录入验证标注列，原标注属于另一评测条件。每个物体聚合至少十一份独立判断以保留歧义；AP保留原生排序指标含义，不当作图片准确率。各数据集单独排行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.21822v1","published_at":"2026-09-18","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":6,"result_count":6,"results":[[12568,52.1,370,12569,12570],[12571,43.8,370,12569,12570],[12572,28.6,370,12569,12570],[12573,47.2,370,12569,12570],[12574,47.8,370,12569,12570],[12575,43.1,370,12569,12570]]},{"id":"validated-detection-pascal-voc-2026-paper","name":"Validated detection · pascal-voc (2026 paper)","group":"视觉与视频","metric":"AP50:95","unit":"score","public_history":true,"protocol":{"en":"Six pretrained open-vocabulary and closed-set detector architectures are evaluated against the same newly validated pascal-voc annotations. Only the validated-ground-truth column is selected; original annotations are a different evaluation. The annotation pipeline aggregates at least eleven independent judgments per object to retain ambiguity; AP is a native ranking metric, not image accuracy. Different datasets remain separate.","zh":"六种预训练开放词汇及闭集检测架构，在相同新验证的pascal-voc标注上评测。仅录入验证标注列，原标注属于另一评测条件。每个物体聚合至少十一份独立判断以保留歧义；AP保留原生排序指标含义，不当作图片准确率。各数据集单独排行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.21822v1","published_at":"2026-09-18","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":6,"result_count":6,"results":[[12568,73.8,370,12569,12570],[12571,65.4,370,12569,12570],[12572,49.2,370,12569,12570],[12573,64.2,370,12569,12570],[12574,67.4,370,12569,12570],[12575,55.6,370,12569,12570]]},{"id":"validated-detection-kitti-2026-paper","name":"Validated detection · kitti (2026 paper)","group":"视觉与视频","metric":"AP50:95","unit":"score","public_history":true,"protocol":{"en":"Six pretrained open-vocabulary and closed-set detector architectures are evaluated against the same newly validated kitti annotations. Only the validated-ground-truth column is selected; original annotations are a different evaluation. The annotation pipeline aggregates at least eleven independent judgments per object to retain ambiguity; AP is a native ranking metric, not image accuracy. Different datasets remain separate.","zh":"六种预训练开放词汇及闭集检测架构，在相同新验证的kitti标注上评测。仅录入验证标注列，原标注属于另一评测条件。每个物体聚合至少十一份独立判断以保留歧义；AP保留原生排序指标含义，不当作图片准确率。各数据集单独排行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.21822v1","published_at":"2026-09-18","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":6,"result_count":6,"results":[[12568,39.4,370,12569,12570],[12571,24.4,370,12569,12570],[12572,9.6,370,12569,12570],[12573,27.3,370,12569,12570],[12574,26.7,370,12569,12570],[12575,26.7,370,12569,12570]]},{"id":"validated-detection-cityscapes-2026-paper","name":"Validated detection · cityscapes (2026 paper)","group":"视觉与视频","metric":"AP50:95","unit":"score","public_history":true,"protocol":{"en":"Six pretrained open-vocabulary and closed-set detector architectures are evaluated against the same newly validated cityscapes annotations. Only the validated-ground-truth column is selected; original annotations are a different evaluation. The annotation pipeline aggregates at least eleven independent judgments per object to retain ambiguity; AP is a native ranking metric, not image accuracy. Different datasets remain separate.","zh":"六种预训练开放词汇及闭集检测架构，在相同新验证的cityscapes标注上评测。仅录入验证标注列，原标注属于另一评测条件。每个物体聚合至少十一份独立判断以保留歧义；AP保留原生排序指标含义，不当作图片准确率。各数据集单独排行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.21822v1","published_at":"2026-09-18","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":6,"result_count":6,"results":[[12568,43.8,370,12569,12570],[12571,30.4,370,12569,12570],[12572,13.8,370,12569,12570],[12573,42.5,370,12569,12570],[12574,44.4,370,12569,12570],[12575,41.0,370,12569,12570]]},{"id":"validated-detection-coco-as-2026-paper","name":"Validated detection · COCO soft labels (2026 paper)","group":"视觉与视频","metric":"Average Similarity 50:95","unit":"score","public_history":true,"protocol":{"en":"Six detector architectures use the same validated COCO soft annotations. Average Similarity evaluates agreement with probabilistic labels over IoU .50:.95; it remains a native metric /100. This is separate from AP with hardened labels and from the label-error detection task. Annotation ambiguity and imperfect expert agreement remain properties of the source dataset.","zh":"六种检测架构采用相同验证后的COCO软标注。Average Similarity在IoU .50:.95上评价与概率标签的一致性，保留百分制原生指标；不与硬标签AP或标注错误检测任务混排。标注歧义及专家判断不完全一致仍是该数据集的特征。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.21822v1","published_at":"2026-09-18","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":6,"result_count":6,"results":[[12568,55.1,370,12576,12577],[12571,48.1,370,12576,12577],[12572,35.8,370,12576,12577],[12573,49.1,370,12576,12577],[12574,46.6,370,12576,12577],[12575,44.0,370,12576,12577]]},{"id":"robofollow-completion-2026-paper","name":"RoboFollow · completion (2026 paper)","group":"Agent 与工具","metric":"Mean binary completion","unit":"%","public_history":true,"protocol":{"en":"Nine pretrained VLA/WAM policy architectures are fine-tuned on the same 3,750 training episodes across four scenes. The source Avg column averages the four scenes and L0–L3 levels. Completion is an action-dependent binary criterion, averaged per task; it does not require full credit on all intermediate stages. Intent and Execution are separate native partial-credit stage scores. The empty-language pi0.5 ablation remains an explicit configuration of the same base policy.","zh":"九种预训练VLA/WAM策略架构，在四类场景的相同3,750条轨迹上微调。原文Avg列平均四场景及L0–L3。完成率按各任务的动作相关二元标准计算，并不要求每个中间阶段满分；Intent与Execution是独立的部分得分阶段指标。pi0.5空语言消融保留为同底模配置。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.25636v1","published_at":"2026-09-22","status":"paper_baselines"},"model_count":9,"result_count":10,"results":[[8323,0.473,370,12578,12579],[8323,0.052000000000000005,370,12580,12579],[10179,0.389,370,12578,12579],[12581,0.205,370,12578,12579],[7594,0.035,370,12578,12579],[12582,0.223,370,12578,12579],[12583,0.425,370,12578,12579],[12584,0.27399999999999997,370,12578,12579],[12585,0.273,370,12578,12579],[10277,0.303,370,12578,12579]]},{"id":"robofollow-intent-2026-paper","name":"RoboFollow · intent (2026 paper)","group":"Agent 与工具","metric":"Mean stage Intent Score","unit":"score","public_history":true,"protocol":{"en":"Nine pretrained VLA/WAM policy architectures are fine-tuned on the same 3,750 training episodes across four scenes. The source Avg column averages the four scenes and L0–L3 levels. Completion is an action-dependent binary criterion, averaged per task; it does not require full credit on all intermediate stages. Intent and Execution are separate native partial-credit stage scores. The empty-language pi0.5 ablation remains an explicit configuration of the same base policy.","zh":"九种预训练VLA/WAM策略架构，在四类场景的相同3,750条轨迹上微调。原文Avg列平均四场景及L0–L3。完成率按各任务的动作相关二元标准计算，并不要求每个中间阶段满分；Intent与Execution是独立的部分得分阶段指标。pi0.5空语言消融保留为同底模配置。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.25636v1","published_at":"2026-09-22","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":9,"result_count":10,"results":[[8323,55.7,370,12578,12579],[8323,12.3,370,12580,12579],[10179,48.4,370,12578,12579],[12581,32.7,370,12578,12579],[7594,10.7,370,12578,12579],[12582,34.3,370,12578,12579],[12583,53.5,370,12578,12579],[12584,35.2,370,12578,12579],[12585,36.8,370,12578,12579],[10277,39.3,370,12578,12579]]},{"id":"robofollow-execution-2026-paper","name":"RoboFollow · execution (2026 paper)","group":"Agent 与工具","metric":"Mean stage Execution Score","unit":"score","public_history":true,"protocol":{"en":"Nine pretrained VLA/WAM policy architectures are fine-tuned on the same 3,750 training episodes across four scenes. The source Avg column averages the four scenes and L0–L3 levels. Completion is an action-dependent binary criterion, averaged per task; it does not require full credit on all intermediate stages. Intent and Execution are separate native partial-credit stage scores. The empty-language pi0.5 ablation remains an explicit configuration of the same base policy.","zh":"九种预训练VLA/WAM策略架构，在四类场景的相同3,750条轨迹上微调。原文Avg列平均四场景及L0–L3。完成率按各任务的动作相关二元标准计算，并不要求每个中间阶段满分；Intent与Execution是独立的部分得分阶段指标。pi0.5空语言消融保留为同底模配置。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.25636v1","published_at":"2026-09-22","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":9,"result_count":10,"results":[[8323,53.0,370,12578,12579],[8323,8.9,370,12580,12579],[10179,44.9,370,12578,12579],[12581,25.6,370,12578,12579],[7594,4.2,370,12578,12579],[12582,30.9,370,12578,12579],[12583,49.3,370,12578,12579],[12584,32.5,370,12578,12579],[12585,30.8,370,12578,12579],[10277,35.2,370,12578,12579]]},{"id":"fakecontext-reality-2026-paper","name":"FakeContext-bench · reality answers (2026 paper)","group":"安全与可靠性","metric":"Reality answers over all instances","unit":"%","public_history":true,"protocol":{"en":"Six frontier models answer the same 3,500 instances from 700 theory families across seven domains. The selected Reality column counts reality-grounded answers over all instances, including Other responses in the denominator. It is separate from the paper headline Reality Rate, which conditions on Reality+Context and has a model-dependent effective denominator. Deterministic registered-answer rules classify outputs; five variants per family are correlated.","zh":"六个前沿模型回答七领域、700个理论家族的相同3,500题。本组选择Reality列，以全部实例为分母，Other回答也计入。它与论文只在Reality+Context内计算的Reality Rate不同，后者有效分母随模型而变。输出按登记答案确定性判分，每个家族五个变式存在相关性。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.27603v1","published_at":"2026-09-23","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[762,0.49570000000000003,370,12586,12587],[2977,0.4031,370,12586,12587],[3157,0.3917,370,12586,12587],[3154,0.3737,370,12586,12587],[3153,0.33399999999999996,370,12586,12587],[3415,0.1286,370,12586,12587]]},{"id":"gamehorizon-offline-2026-paper","name":"GameHorizon · offline QA (2026 paper)","group":"Agent 与工具","metric":"Mean three-task accuracy","unit":"%","public_history":true,"protocol":{"en":"Forty-four source configurations from VLM, unified multimodal, GUI and coding-model families answer the same offline multiple-choice questions. Overall is the source rounded mean accuracy across action selection, goal decomposition and cross-horizon consistency; it is not online game completion. Model-native input implementations and named reasoning versions remain configurations. The paper describes future dataset release; the inspectable published score table is retained as paper baselines, without claiming an independently rerun benchmark.","zh":"四十四种原文配置涵盖VLM、统一多模态、GUI及代码模型，回答同一离线选择题组。Overall是动作选择、目标分解和跨跨度一致性三任务准确率的原文平均值，不代表在线游戏完成率。保留各模型原生输入接口及推理版本。论文称后续发布数据，本组依据可检查的论文成绩表展示，标为论文基线。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.25001v1","published_at":"2026-09-21","status":"paper_baselines"},"model_count":44,"result_count":44,"results":[[4977,0.802,370,12588,12589],[2983,0.773,370,12588,12589],[3386,0.767,370,12588,12589],[3405,0.753,370,12588,12589],[762,0.748,370,12588,12589],[3155,0.745,370,12588,12589],[3415,0.7440000000000001,370,12588,12589],[3196,0.732,370,12588,12589],[3412,0.728,370,12588,12589],[12590,0.727,370,12588,12589],[3204,0.721,370,12588,12589],[7674,0.7140000000000001,370,12588,12589],[3453,0.7120000000000001,370,12588,12589],[2973,0.7070000000000001,370,12588,12589],[4961,0.687,370,12588,12589],[5300,0.685,370,12588,12589],[3408,0.6779999999999999,370,12588,12589],[3187,0.6779999999999999,370,12588,12589],[6721,0.6679999999999999,370,12588,12589],[5822,0.6629999999999999,370,12588,12589],[3198,0.6559999999999999,370,12588,12589],[2600,0.652,370,12588,12589],[3479,0.648,370,12588,12589],[3568,0.647,370,12588,12589],[5740,0.6409999999999999,370,12588,12589],[2613,0.634,370,12588,12589],[7937,0.632,370,12588,12589],[340,0.628,370,12588,12589],[2576,0.623,370,12588,12589],[7296,0.614,370,12588,12589],[2453,0.608,370,12588,12589],[6239,0.606,370,12588,12589],[6253,0.583,370,12588,12589],[3764,0.575,370,12588,12589],[6139,0.573,370,12588,12589],[12591,0.5710000000000001,370,12588,12589],[2471,0.5670000000000001,370,12588,12589],[2654,0.5579999999999999,370,12588,12589],[5536,0.55,370,12588,12589],[353,0.544,370,12588,12589],[5487,0.53,370,12588,12589],[7306,0.478,370,12588,12589],[6246,0.46,370,12588,12589],[7293,0.446,370,12588,12589]]},{"id":"explorationbench-aliencode-2026-paper","name":"ExplorationBench · aliencode (2026 paper)","group":"Agent 与工具","metric":"Mean@3 final held-out accuracy","unit":"%","public_history":true,"protocol":{"en":"Ten model systems explore a deterministic alien-rule environment for four rounds, then answer 70 held-out tasks without tools at each milestone. This aliencode cohort uses Mean@3 final accuracy across all three exploration trajectories, not the result of selecting the best trajectory. Each question receives three answers. Native reasoning settings remain explicit; rounds, trajectories and answers are not additional models.","zh":"十个模型系统在确定性的异规则环境中探索四轮，各节点在无工具条件下回答70道留出题。本组aliencode采用全部三条探索轨迹的最终Mean@3准确率，不选择最佳轨迹；每题回答三次。保留各模型原生推理设置，不将轮次、轨迹和重复答案计作新模型。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.30199v1","published_at":"2026-09-24","status":"paper_baselines"},"model_count":10,"result_count":10,"results":[[755,0.725,370,12592,12593],[762,0.7859999999999999,370,12592,12593],[2983,0.39,370,12594,12593],[2977,0.375,370,12592,12593],[2974,0.519,370,12595,12593],[3157,0.625,370,12592,12593],[4029,0.294,370,12594,12593],[3153,0.452,370,12592,12593],[4030,0.23800000000000002,370,12594,12593],[3185,0.073,370,12592,12593]]},{"id":"explorationbench-alienlogic-2026-paper","name":"ExplorationBench · alienlogic (2026 paper)","group":"Agent 与工具","metric":"Mean@3 final held-out accuracy","unit":"%","public_history":true,"protocol":{"en":"Ten model systems explore a deterministic alien-rule environment for four rounds, then answer 70 held-out tasks without tools at each milestone. This alienlogic cohort uses Mean@3 final accuracy across all three exploration trajectories, not the result of selecting the best trajectory. Each question receives three answers. Native reasoning settings remain explicit; rounds, trajectories and answers are not additional models.","zh":"十个模型系统在确定性的异规则环境中探索四轮，各节点在无工具条件下回答70道留出题。本组alienlogic采用全部三条探索轨迹的最终Mean@3准确率，不选择最佳轨迹；每题回答三次。保留各模型原生推理设置，不将轮次、轨迹和重复答案计作新模型。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.30199v1","published_at":"2026-09-24","status":"paper_baselines"},"model_count":10,"result_count":10,"results":[[2974,0.721,370,12595,12593],[755,0.733,370,12592,12593],[3157,0.727,370,12592,12593],[762,0.73,370,12592,12593],[4029,0.6779999999999999,370,12594,12593],[3185,0.6509999999999999,370,12592,12593],[2977,0.679,370,12592,12593],[3153,0.5920000000000001,370,12592,12593],[4030,0.595,370,12594,12593],[2983,0.5760000000000001,370,12594,12593]]},{"id":"embodiedswe-solved-2026-paper","name":"EmbodiedSWE-Bench · task solving (2026 paper)","group":"Agent 与工具","metric":"Solved-task fraction (rounded)","unit":"%","public_history":true,"protocol":{"en":"Six frontier models use their native Claude Code/Codex harnesses on the same 28 long-horizon robotics tasks, one run per task with an RTX 4090 and four-hour wall-clock budget including simulation. The selected values are the printed whole-percent solved-task fractions in Figure 3, not the bars’ mean partial-credit rubric scores. Offline graders use simulator checks and task-specific human-written rubrics with VLM grading. Additional tools and transfer experiments stay outside this baseline cohort.","zh":"六个前沿模型使用原生Claude Code/Codex框架，在相同28道长程机器人任务上各执行一次，每题一张RTX 4090、四小时含仿真的总时限。采用图3印出的整数百分比解题率，不把横条的部分得分量表均值混进来。离线判分结合仿真状态检查及任务特定人工量表、VLM判断；额外工具和迁移实验不混入基线。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.27308v1","published_at":"2026-09-23","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[2958,0.82,370,12596,12597],[2956,0.61,370,12598,12597],[755,0.5,370,12598,12597],[3479,0.25,370,12598,12597],[762,0.18,370,12596,12597],[2973,0.11,370,12596,12597]]},{"id":"ecom-bench-visual-2026-paper","name":"Ecom-Bench · visual (2026 paper)","group":"视觉与视频","metric":"Task-adaptive visual rubric","unit":"score","public_history":true,"protocol":{"en":"Eight complete editors process all 1,100 cases (100 in each of eleven tasks), including multi-image inputs where required. Joy-Edit and LongCat lack two supported tasks and are excluded from this complete-cohort comparison. Visual scores use task-specific behavior-anchored dimensions and geometric-mean aggregation on a native 1–5 rubric. Source training conditions and model names remain explicit.","zh":"八个具备完整任务成绩的编辑模型，处理十一任务各100例、共1,100例，其中部分任务需要多图输入。Joy-Edit与LongCat缺少两个适用任务，不进入完整队列比较。视觉评分按任务选择行为量表维度并取几何均值，保留原生1–5分。保留模型原名及训练条件。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.26375v1","published_at":"2026-09-22","status":"paper_baselines"},"score_max":5,"saturation_excluded":true,"scale_label":{"en":"Score / 5","zh":"分数 / 5"},"score_decimals":2,"model_count":8,"result_count":8,"results":[[12599,3.48,370,12600,12601],[12602,3.15,370,12600,12601],[12603,3.35,370,12600,12601],[12604,3.18,370,12600,12601],[12605,3.67,370,12600,12601],[12606,3.47,370,12600,12601],[12607,4.01,370,12600,12601],[6368,4.2,370,12600,12601]]},{"id":"ecom-bench-ctr-ranking-2026-paper","name":"Ecom-Bench · ctr-ranking (2026 paper)","group":"视觉与视频","metric":"Top-three predicted-CTR appearances","unit":"raw","public_history":true,"protocol":{"en":"Eight complete editors process all 1,100 cases (100 in each of eleven tasks), including multi-image inputs where required. Joy-Edit and LongCat lack two supported tasks and are excluded from this complete-cohort comparison. The native count sums rank-1/2/3 appearances under a learned predicted-CTR model, each with equal weight. It is not observed click-through rate or a success percentage, and depends on this fixed eight-model candidate pool. Source training conditions and model names remain explicit.","zh":"八个具备完整任务成绩的编辑模型，处理十一任务各100例、共1,100例，其中部分任务需要多图输入。Joy-Edit与LongCat缺少两个适用任务，不进入完整队列比较。原生计数是学习型预测CTR模型下的前三名出现次数之和，三个名次等权。它不是实际点击率或成功百分比，并依赖这固定八模型候选池。保留模型原名及训练条件。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.26375v1","published_at":"2026-09-22","status":"paper_baselines"},"model_count":8,"result_count":8,"results":[[12599,293.0,370,12608,12601],[12602,406.0,370,12608,12601],[12603,362.0,370,12608,12601],[12604,348.0,370,12608,12601],[12605,544.0,370,12608,12601],[12606,525.0,370,12608,12601],[12607,391.0,370,12608,12601],[6368,431.0,370,12608,12601]]},{"id":"uncheatable-eval-july-2026-paper","name":"Uncheatable Eval · July text cohort (2026 paper)","group":"知识与推理","metric":"Compression rate (%)","unit":"score","public_history":true,"protocol":{"en":"Eighty base-model checkpoints predict the same July 2026 sample of 7,000 documents (500 per fourteen categories). Every shared document is truncated to at most 3,584 tokens under all eight checking tokenizers. Compression is byte-weighted within each category, then categories receive equal weight in Overall. The native compression ratio is lower-is-better, separate from accuracy and the 54-model long-context cohort. This is a manually captured paper batch; no recurring data fetch is enabled.","zh":"八十个基础模型检查点预测相同2026年7月的7,000篇文本，十四类各500篇。共同文本经八种分词器检查，均最多3,584词元。各类压缩比按字节加权，Overall再给各类等权。保留越低越好的原生压缩率，不当作准确率，也不与54模型长上下文队列混排。本组是手工保存的论文批次。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.27510v1","published_at":"2026-09-23","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"higher_is_better":false,"model_count":80,"result_count":80,"results":[[5480,6.0,370,12609,12610],[12611,6.24,370,12612,12610],[5635,6.26,370,12613,12610],[8097,6.28,370,12614,12610],[6689,6.3,370,12615,12610],[12616,6.39,370,12617,12610],[9664,6.43,370,12618,12610],[4463,6.45,370,12619,12610],[12620,6.52,370,12621,12610],[12622,6.79,370,12623,12610],[12624,6.88,370,12625,12610],[12626,6.31,370,12627,12610],[7345,6.44,370,12628,12610],[12629,6.53,370,12630,12610],[575,6.57,370,12631,12610],[12632,6.57,370,12631,12610],[4466,6.62,370,12630,12610],[12633,7.14,370,12634,12610],[12635,7.23,370,12636,12610],[12637,6.59,370,12638,12610],[12527,6.68,370,12639,12610],[12640,6.71,370,12641,12610],[12642,6.79,370,12643,12610],[12644,6.8,370,12645,12610],[12646,6.87,370,12647,12610],[12648,6.95,370,12649,12610],[598,7.02,370,12641,12610],[797,7.05,370,12649,12610],[12650,7.16,370,12643,12610],[12651,7.18,370,12652,12610],[12653,7.25,370,12654,12610],[12655,7.32,370,12656,12610],[12657,7.39,370,12658,12610],[12659,7.5,370,12660,12610],[10725,7.54,370,12661,12610],[12662,7.59,370,12663,12610],[12664,7.6,370,12652,12610],[12665,6.98,370,12666,12610],[12667,7.02,370,12668,12610],[12669,7.12,370,12670,12610],[12671,7.15,370,12666,12610],[12672,7.15,370,12673,12610],[12674,7.16,370,12675,12610],[12676,7.22,370,12677,12610],[613,7.38,370,12678,12610],[12679,7.49,370,12680,12610],[12681,7.5,370,12682,12610],[4476,7.55,370,12683,12610],[12684,7.57,370,12683,12610],[12685,7.62,370,12686,12610],[12687,7.69,370,12683,12610],[12688,7.71,370,12689,12610],[12690,7.73,370,12691,12610],[12692,7.84,370,12666,12610],[12693,7.87,370,12694,12610],[12695,8.02,370,12696,12610],[12697,8.13,370,12689,12610],[12698,7.64,370,12699,12610],[12700,7.7,370,12701,12610],[12702,7.75,370,12703,12610],[12704,7.75,370,12701,12610],[4480,8.0,370,12699,12610],[12705,8.03,370,12706,12610],[12707,8.16,370,12708,12610],[623,8.18,370,12709,12610],[12710,8.33,370,12708,12610],[617,8.41,370,12701,12610],[12711,8.42,370,12708,12610],[12712,8.62,370,12713,12610],[12714,8.7,370,12715,12610],[12716,8.72,370,12717,12610],[12718,8.95,370,12719,12610],[12720,8.97,370,12721,12610],[4833,9.21,370,12721,12610],[12722,9.73,370,12723,12610],[625,9.77,370,12723,12610],[12724,9.88,370,12721,12610],[12725,10.25,370,12726,12610],[12727,10.71,370,12728,12610],[628,11.15,370,12729,12610]]},{"id":"evasionbench-success-2026-paper","name":"EvasionBench · success (2026 paper)","group":"安全与可靠性","metric":"Best-of-three evasion success (%)","unit":"score","score_max":100,"saturation_excluded":true,"score_decimals":1,"scale_label":{"en":"Failure rate / 100","zh":"失败率 / 100"},"higher_is_better":false,"public_history":true,"protocol":{"en":"Ten named agent/model configurations use the same 50 task-policy pairs under a GPT-OSS-120B runtime monitor. The source audit fixes three valid epochs per task; infrastructure-invalid runs are replaced, while timeouts, budget exhaustion and cybersecurity refusals stay unsuccessful. Success is the fraction of tasks completed in at least one of three runs. This is a native lower-is-better failure rate. The source flags model downgrades/refusals in five frontier configurations, including Opus 5→4.8; these are routed system results rather than isolated fixed-checkpoint measurements. Astra’s low rate also accompanies overrefusal.","zh":"十种原文命名的Agent/模型配置，在相同50个任务与策略对上接受GPT-OSS-120B运行时监控。来源审计固定每题三个有效轮次；基础设施无效运行会替换，超时、预算耗尽及网络安全拒答仍计未成功。成功率统计三次中至少一次完成任务的题目比例。保留越低越好的原生失败率。原文对五种前沿配置标记降级或拒答，包括Opus 5转4.8；成绩代表路由系统而非独立固定检查点。Astra低规避率同时伴随过度拒答。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.30217v1","published_at":"2026-09-24","status":"paper_baselines"},"model_count":10,"result_count":10,"results":[[3654,88.0,370,12730,12731],[3261,58.0,370,12730,12731],[2979,56.0,370,12730,12731],[3408,38.0,370,12730,12731],[3529,38.0,370,12730,12731],[762,34.0,370,12732,12731],[3394,20.0,370,12732,12731],[2956,12.0,370,12733,12731],[755,4.0,370,12733,12731],[2958,0.0,370,12732,12731]]},{"id":"evasionbench-attempt-2026-paper","name":"EvasionBench · attempt (2026 paper)","group":"安全与可靠性","metric":"Best-of-three judged evasion attempt (%)","unit":"score","score_max":100,"saturation_excluded":true,"score_decimals":1,"scale_label":{"en":"Failure rate / 100","zh":"失败率 / 100"},"higher_is_better":false,"public_history":true,"protocol":{"en":"Ten named agent/model configurations use the same 50 task-policy pairs under a GPT-OSS-120B runtime monitor. The source audit fixes three valid epochs per task; infrastructure-invalid runs are replaced, while timeouts, budget exhaustion and cybersecurity refusals stay unsuccessful. Attempt is the fraction of tasks with at least one GPT-5.6-Luna-judged evasion attempt, distinct from deterministic task completion. This is a native lower-is-better failure rate. The source flags model downgrades/refusals in five frontier configurations, including Opus 5→4.8; these are routed system results rather than isolated fixed-checkpoint measurements. Astra’s low rate also accompanies overrefusal.","zh":"十种原文命名的Agent/模型配置，在相同50个任务与策略对上接受GPT-OSS-120B运行时监控。来源审计固定每题三个有效轮次；基础设施无效运行会替换，超时、预算耗尽及网络安全拒答仍计未成功。尝试率统计至少一次被GPT-5.6 Luna判定尝试规避的题目比例，与确定性任务完成分开。保留越低越好的原生失败率。原文对五种前沿配置标记降级或拒答，包括Opus 5转4.8；成绩代表路由系统而非独立固定检查点。Astra低规避率同时伴随过度拒答。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.30217v1","published_at":"2026-09-24","status":"paper_baselines"},"model_count":10,"result_count":10,"results":[[3654,98.0,370,12730,12731],[3261,90.0,370,12730,12731],[2979,66.0,370,12730,12731],[3408,54.0,370,12730,12731],[3529,66.0,370,12730,12731],[762,40.0,370,12732,12731],[3394,38.0,370,12732,12731],[2956,84.0,370,12733,12731],[755,78.0,370,12733,12731],[2958,12.0,370,12732,12731]]},{"id":"ultrabench2-classification-aul-mass-full-finetuning-2026-paper","name":"UltraBench 2 · AUL Mass classification full-finetuning (2026 paper)","group":"医疗","metric":"Macro F1","unit":"score","score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"public_history":true,"protocol":{"en":"Sixteen ultrasound-specialist and general vision backbones use the same prepared AUL Mass test set and fixed dataset-dependent patient/sequence/image splits. This full-finetuning cohort reports mean Macro F1, with source standard errors archived. Both training regimes use AdamW, up to 10,000 steps and validation-only early stopping; segmentation decoders follow backbone output structure. Pretraining overlap varies and is stated for every result using Table 11, including unknown/unauditable data. These are observed transfer results, not a contamination-free ranking. Macro F1 and foreground-only Dice remain native metrics.","zh":"十六个超声专用及通用视觉底模，采用相同预处理的AUL Mass测试集与固定划分，按数据集使用病人、序列或图片粒度。本组full-finetuning保留平均Macro F1，标准误存于证据。两训练条件均用AdamW、最多10,000步，只在验证集早停；分割解码器随底模输出结构选择。每条成绩按表11注明预训练数据重叠，包括未知及不可审计情况。F1与去除背景的Dice保留原生指标。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.28610v1","published_at":"2026-09-23","status":"paper_baselines"},"model_count":16,"result_count":16,"results":[[12734,0.639,370,12735,12736],[12737,0.643,370,12735,12736],[12738,0.616,370,12739,12736],[12740,0.687,370,12739,12736],[12741,0.722,370,12739,12736],[12742,0.724,370,12739,12736],[12743,0.614,370,12744,12736],[12745,0.707,370,12746,12736],[12747,0.644,370,12735,12736],[12748,0.619,370,12744,12736],[12749,0.684,370,12735,12736],[12750,0.672,370,12746,12736],[12751,0.662,370,12735,12736],[12752,0.719,370,12735,12736],[12753,0.68,370,12744,12736],[12754,0.652,370,12746,12736]]},{"id":"ultrabench2-classification-busbra-histology-full-finetuning-2026-paper","name":"UltraBench 2 · BUS-BRA histology classification full-finetuning (2026 paper)","group":"医疗","metric":"Macro F1","unit":"score","score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"public_history":true,"protocol":{"en":"Sixteen ultrasound-specialist and general vision backbones use the same prepared BUS-BRA histology test set and fixed dataset-dependent patient/sequence/image splits. This full-finetuning cohort reports mean Macro F1, with source standard errors archived. Both training regimes use AdamW, up to 10,000 steps and validation-only early stopping; segmentation decoders follow backbone output structure. Pretraining overlap varies and is stated for every result using Table 11, including unknown/unauditable data. These are observed transfer results, not a contamination-free ranking. Macro F1 and foreground-only Dice remain native metrics.","zh":"十六个超声专用及通用视觉底模，采用相同预处理的BUS-BRA histology测试集与固定划分，按数据集使用病人、序列或图片粒度。本组full-finetuning保留平均Macro F1，标准误存于证据。两训练条件均用AdamW、最多10,000步，只在验证集早停；分割解码器随底模输出结构选择。每条成绩按表11注明预训练数据重叠，包括未知及不可审计情况。F1与去除背景的Dice保留原生指标。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.28610v1","published_at":"2026-09-23","status":"paper_baselines"},"model_count":16,"result_count":16,"results":[[12734,0.174,370,12755,12736],[12737,0.179,370,12755,12736],[12738,0.147,370,12756,12736],[12740,0.148,370,12756,12736],[12741,0.152,370,12756,12736],[12742,0.156,370,12756,12736],[12743,0.13,370,12757,12736],[12745,0.153,370,12755,12736],[12747,0.163,370,12755,12736],[12748,0.148,370,12757,12736],[12749,0.163,370,12755,12736],[12750,0.161,370,12755,12736],[12751,0.188,370,12755,12736],[12752,0.201,370,12755,12736],[12753,0.145,370,12757,12736],[12754,0.167,370,12755,12736]]},{"id":"ultrabench2-classification-busbra-pathology-full-finetuning-2026-paper","name":"UltraBench 2 · BUS-BRA pathology classification full-finetuning (2026 paper)","group":"医疗","metric":"Macro F1","unit":"score","score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"public_history":true,"protocol":{"en":"Sixteen ultrasound-specialist and general vision backbones use the same prepared BUS-BRA pathology test set and fixed dataset-dependent patient/sequence/image splits. This full-finetuning cohort reports mean Macro F1, with source standard errors archived. Both training regimes use AdamW, up to 10,000 steps and validation-only early stopping; segmentation decoders follow backbone output structure. Pretraining overlap varies and is stated for every result using Table 11, including unknown/unauditable data. These are observed transfer results, not a contamination-free ranking. Macro F1 and foreground-only Dice remain native metrics.","zh":"十六个超声专用及通用视觉底模，采用相同预处理的BUS-BRA pathology测试集与固定划分，按数据集使用病人、序列或图片粒度。本组full-finetuning保留平均Macro F1，标准误存于证据。两训练条件均用AdamW、最多10,000步，只在验证集早停；分割解码器随底模输出结构选择。每条成绩按表11注明预训练数据重叠，包括未知及不可审计情况。F1与去除背景的Dice保留原生指标。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.28610v1","published_at":"2026-09-23","status":"paper_baselines"},"model_count":16,"result_count":16,"results":[[12734,0.702,370,12755,12736],[12737,0.702,370,12755,12736],[12738,0.711,370,12756,12736],[12740,0.733,370,12756,12736],[12741,0.696,370,12756,12736],[12742,0.735,370,12756,12736],[12743,0.668,370,12757,12736],[12745,0.71,370,12755,12736],[12747,0.731,370,12755,12736],[12748,0.721,370,12757,12736],[12749,0.687,370,12755,12736],[12750,0.698,370,12755,12736],[12751,0.774,370,12755,12736],[12752,0.794,370,12755,12736],[12753,0.753,370,12757,12736],[12754,0.706,370,12755,12736]]},{"id":"ultrabench2-classification-covid-blues-full-finetuning-2026-paper","name":"UltraBench 2 · COVID-BLUES classification full-finetuning (2026 paper)","group":"医疗","metric":"Macro F1","unit":"score","score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"public_history":true,"protocol":{"en":"Sixteen ultrasound-specialist and general vision backbones use the same prepared COVID-BLUES test set and fixed dataset-dependent patient/sequence/image splits. This full-finetuning cohort reports mean Macro F1, with source standard errors archived. Both training regimes use AdamW, up to 10,000 steps and validation-only early stopping; segmentation decoders follow backbone output structure. Pretraining overlap varies and is stated for every result using Table 11, including unknown/unauditable data. These are observed transfer results, not a contamination-free ranking. Macro F1 and foreground-only Dice remain native metrics.","zh":"十六个超声专用及通用视觉底模，采用相同预处理的COVID-BLUES测试集与固定划分，按数据集使用病人、序列或图片粒度。本组full-finetuning保留平均Macro F1，标准误存于证据。两训练条件均用AdamW、最多10,000步，只在验证集早停；分割解码器随底模输出结构选择。每条成绩按表11注明预训练数据重叠，包括未知及不可审计情况。F1与去除背景的Dice保留原生指标。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.28610v1","published_at":"2026-09-23","status":"paper_baselines"},"model_count":16,"result_count":16,"results":[[12734,0.644,370,12758,12736],[12737,0.624,370,12758,12736],[12738,0.548,370,12759,12736],[12740,0.686,370,12759,12736],[12741,0.538,370,12759,12736],[12742,0.585,370,12759,12736],[12743,0.627,370,12760,12736],[12745,0.766,370,12758,12736],[12747,0.735,370,12758,12736],[12748,0.576,370,12760,12736],[12749,0.804,370,12758,12736],[12750,0.688,370,12758,12736],[12751,0.663,370,12758,12736],[12752,0.695,370,12758,12736],[12753,0.608,370,12760,12736],[12754,0.74,370,12761,12736]]},{"id":"ultrabench2-classification-gbcu-full-finetuning-2026-paper","name":"UltraBench 2 · GBCU classification full-finetuning (2026 paper)","group":"医疗","metric":"Macro F1","unit":"score","score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"public_history":true,"protocol":{"en":"Sixteen ultrasound-specialist and general vision backbones use the same prepared GBCU test set and fixed dataset-dependent patient/sequence/image splits. This full-finetuning cohort reports mean Macro F1, with source standard errors archived. Both training regimes use AdamW, up to 10,000 steps and validation-only early stopping; segmentation decoders follow backbone output structure. Pretraining overlap varies and is stated for every result using Table 11, including unknown/unauditable data. These are observed transfer results, not a contamination-free ranking. Macro F1 and foreground-only Dice remain native metrics.","zh":"十六个超声专用及通用视觉底模，采用相同预处理的GBCU测试集与固定划分，按数据集使用病人、序列或图片粒度。本组full-finetuning保留平均Macro F1，标准误存于证据。两训练条件均用AdamW、最多10,000步，只在验证集早停；分割解码器随底模输出结构选择。每条成绩按表11注明预训练数据重叠，包括未知及不可审计情况。F1与去除背景的Dice保留原生指标。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.28610v1","published_at":"2026-09-23","status":"paper_baselines"},"model_count":16,"result_count":16,"results":[[12734,0.741,370,12762,12736],[12737,0.641,370,12762,12736],[12738,0.678,370,12763,12736],[12740,0.698,370,12763,12736],[12741,0.67,370,12763,12736],[12742,0.631,370,12763,12736],[12743,0.647,370,12764,12736],[12745,0.696,370,12762,12736],[12747,0.643,370,12762,12736],[12748,0.627,370,12764,12736],[12749,0.643,370,12762,12736],[12750,0.541,370,12762,12736],[12751,0.679,370,12762,12736],[12752,0.658,370,12762,12736],[12753,0.686,370,12764,12736],[12754,0.656,370,12762,12736]]},{"id":"ultrabench2-classification-masld-full-finetuning-2026-paper","name":"UltraBench 2 · MASLD classification full-finetuning (2026 paper)","group":"医疗","metric":"Macro F1","unit":"score","score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"public_history":true,"protocol":{"en":"Sixteen ultrasound-specialist and general vision backbones use the same prepared MASLD test set and fixed dataset-dependent patient/sequence/image splits. This full-finetuning cohort reports mean Macro F1, with source standard errors archived. Both training regimes use AdamW, up to 10,000 steps and validation-only early stopping; segmentation decoders follow backbone output structure. Pretraining overlap varies and is stated for every result using Table 11, including unknown/unauditable data. These are observed transfer results, not a contamination-free ranking. Macro F1 and foreground-only Dice remain native metrics.","zh":"十六个超声专用及通用视觉底模，采用相同预处理的MASLD测试集与固定划分，按数据集使用病人、序列或图片粒度。本组full-finetuning保留平均Macro F1，标准误存于证据。两训练条件均用AdamW、最多10,000步，只在验证集早停；分割解码器随底模输出结构选择。每条成绩按表11注明预训练数据重叠，包括未知及不可审计情况。F1与去除背景的Dice保留原生指标。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.28610v1","published_at":"2026-09-23","status":"paper_baselines"},"model_count":16,"result_count":16,"results":[[12734,0.756,370,12765,12736],[12737,0.791,370,12765,12736],[12738,0.644,370,12766,12736],[12740,0.45,370,12766,12736],[12741,0.692,370,12766,12736],[12742,0.804,370,12766,12736],[12743,0.765,370,12767,12736],[12745,0.708,370,12765,12736],[12747,0.57,370,12765,12736],[12748,0.644,370,12767,12736],[12749,0.831,370,12765,12736],[12750,0.916,370,12765,12736],[12751,0.771,370,12765,12736],[12752,0.87,370,12765,12736],[12753,0.612,370,12767,12736],[12754,0.771,370,12765,12736]]},{"id":"ultrabench2-classification-mmotu-full-finetuning-2026-paper","name":"UltraBench 2 · MMOTU classification full-finetuning (2026 paper)","group":"医疗","metric":"Macro F1","unit":"score","score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"public_history":true,"protocol":{"en":"Sixteen ultrasound-specialist and general vision backbones use the same prepared MMOTU test set and fixed dataset-dependent patient/sequence/image splits. This full-finetuning cohort reports mean Macro F1, with source standard errors archived. Both training regimes use AdamW, up to 10,000 steps and validation-only early stopping; segmentation decoders follow backbone output structure. Pretraining overlap varies and is stated for every result using Table 11, including unknown/unauditable data. These are observed transfer results, not a contamination-free ranking. Macro F1 and foreground-only Dice remain native metrics.","zh":"十六个超声专用及通用视觉底模，采用相同预处理的MMOTU测试集与固定划分，按数据集使用病人、序列或图片粒度。本组full-finetuning保留平均Macro F1，标准误存于证据。两训练条件均用AdamW、最多10,000步，只在验证集早停；分割解码器随底模输出结构选择。每条成绩按表11注明预训练数据重叠，包括未知及不可审计情况。F1与去除背景的Dice保留原生指标。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.28610v1","published_at":"2026-09-23","status":"paper_baselines"},"model_count":16,"result_count":16,"results":[[12734,0.452,370,12768,12736],[12737,0.47,370,12768,12736],[12738,0.435,370,12769,12736],[12740,0.382,370,12769,12736],[12741,0.423,370,12769,12736],[12742,0.366,370,12769,12736],[12743,0.352,370,12770,12736],[12745,0.458,370,12771,12736],[12747,0.36,370,12768,12736],[12748,0.384,370,12770,12736],[12749,0.514,370,12768,12736],[12750,0.407,370,12768,12736],[12751,0.49,370,12768,12736],[12752,0.486,370,12768,12736],[12753,0.419,370,12770,12736],[12754,0.464,370,12768,12736]]},{"id":"ultrabench2-classification-tn3k-full-finetuning-2026-paper","name":"UltraBench 2 · TN3K classification full-finetuning (2026 paper)","group":"医疗","metric":"Macro F1","unit":"score","score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"public_history":true,"protocol":{"en":"Sixteen ultrasound-specialist and general vision backbones use the same prepared TN3K test set and fixed dataset-dependent patient/sequence/image splits. This full-finetuning cohort reports mean Macro F1, with source standard errors archived. Both training regimes use AdamW, up to 10,000 steps and validation-only early stopping; segmentation decoders follow backbone output structure. Pretraining overlap varies and is stated for every result using Table 11, including unknown/unauditable data. These are observed transfer results, not a contamination-free ranking. Macro F1 and foreground-only Dice remain native metrics.","zh":"十六个超声专用及通用视觉底模，采用相同预处理的TN3K测试集与固定划分，按数据集使用病人、序列或图片粒度。本组full-finetuning保留平均Macro F1，标准误存于证据。两训练条件均用AdamW、最多10,000步，只在验证集早停；分割解码器随底模输出结构选择。每条成绩按表11注明预训练数据重叠，包括未知及不可审计情况。F1与去除背景的Dice保留原生指标。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.28610v1","published_at":"2026-09-23","status":"paper_baselines"},"model_count":16,"result_count":16,"results":[[12734,0.692,370,12772,12736],[12737,0.664,370,12772,12736],[12738,0.632,370,12773,12736],[12740,0.632,370,12773,12736],[12741,0.696,370,12773,12736],[12742,0.664,370,12773,12736],[12743,0.586,370,12774,12736],[12745,0.654,370,12772,12736],[12747,0.664,370,12775,12736],[12748,0.601,370,12774,12736],[12749,0.612,370,12772,12736],[12750,0.585,370,12775,12736],[12751,0.737,370,12772,12736],[12752,0.738,370,12772,12736],[12753,0.697,370,12774,12736],[12754,0.644,370,12775,12736]]},{"id":"ultrabench2-classification-camus-full-finetuning-2026-paper","name":"UltraBench 2 · CAMUS classification full-finetuning (2026 paper)","group":"医疗","metric":"Macro F1","unit":"score","score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"public_history":true,"protocol":{"en":"Sixteen ultrasound-specialist and general vision backbones use the same prepared CAMUS test set and fixed dataset-dependent patient/sequence/image splits. This full-finetuning cohort reports mean Macro F1, with source standard errors archived. Both training regimes use AdamW, up to 10,000 steps and validation-only early stopping; segmentation decoders follow backbone output structure. Pretraining overlap varies and is stated for every result using Table 11, including unknown/unauditable data. These are observed transfer results, not a contamination-free ranking. Macro F1 and foreground-only Dice remain native metrics.","zh":"十六个超声专用及通用视觉底模，采用相同预处理的CAMUS测试集与固定划分，按数据集使用病人、序列或图片粒度。本组full-finetuning保留平均Macro F1，标准误存于证据。两训练条件均用AdamW、最多10,000步，只在验证集早停；分割解码器随底模输出结构选择。每条成绩按表11注明预训练数据重叠，包括未知及不可审计情况。F1与去除背景的Dice保留原生指标。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.28610v1","published_at":"2026-09-23","status":"paper_baselines"},"model_count":16,"result_count":16,"results":[[12734,0.68,370,12776,12736],[12737,0.675,370,12776,12736],[12738,0.628,370,12777,12736],[12740,0.592,370,12777,12736],[12741,0.701,370,12777,12736],[12742,0.639,370,12777,12736],[12743,0.698,370,12778,12736],[12745,0.71,370,12779,12736],[12747,0.63,370,12779,12736],[12748,0.675,370,12778,12736],[12749,0.673,370,12779,12736],[12750,0.639,370,12779,12736],[12751,0.631,370,12779,12736],[12752,0.653,370,12779,12736],[12753,0.582,370,12778,12736],[12754,0.68,370,12779,12736]]},{"id":"ultrabench2-classification-fetal-brain-full-finetuning-2026-paper","name":"UltraBench 2 · Fetal Brain classification full-finetuning (2026 paper)","group":"医疗","metric":"Macro F1","unit":"score","score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"public_history":true,"protocol":{"en":"Sixteen ultrasound-specialist and general vision backbones use the same prepared Fetal Brain test set and fixed dataset-dependent patient/sequence/image splits. This full-finetuning cohort reports mean Macro F1, with source standard errors archived. Both training regimes use AdamW, up to 10,000 steps and validation-only early stopping; segmentation decoders follow backbone output structure. Pretraining overlap varies and is stated for every result using Table 11, including unknown/unauditable data. These are observed transfer results, not a contamination-free ranking. Macro F1 and foreground-only Dice remain native metrics.","zh":"十六个超声专用及通用视觉底模，采用相同预处理的Fetal Brain测试集与固定划分，按数据集使用病人、序列或图片粒度。本组full-finetuning保留平均Macro F1，标准误存于证据。两训练条件均用AdamW、最多10,000步，只在验证集早停；分割解码器随底模输出结构选择。每条成绩按表11注明预训练数据重叠，包括未知及不可审计情况。F1与去除背景的Dice保留原生指标。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.28610v1","published_at":"2026-09-23","status":"paper_baselines"},"model_count":16,"result_count":16,"results":[[12734,0.534,370,12780,12736],[12737,0.532,370,12780,12736],[12738,0.595,370,12781,12736],[12740,0.573,370,12781,12736],[12741,0.598,370,12781,12736],[12742,0.623,370,12781,12736],[12743,0.535,370,12782,12736],[12745,0.646,370,12780,12736],[12747,0.5,370,12780,12736],[12748,0.462,370,12782,12736],[12749,0.556,370,12783,12736],[12750,0.581,370,12780,12736],[12751,0.684,370,12783,12736],[12752,0.688,370,12783,12736],[12753,0.581,370,12782,12736],[12754,0.663,370,12783,12736]]},{"id":"ultrabench2-classification-fetal-plane-full-finetuning-2026-paper","name":"UltraBench 2 · Fetal Plane classification full-finetuning (2026 paper)","group":"医疗","metric":"Macro F1","unit":"score","score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"public_history":true,"protocol":{"en":"Sixteen ultrasound-specialist and general vision backbones use the same prepared Fetal Plane test set and fixed dataset-dependent patient/sequence/image splits. This full-finetuning cohort reports mean Macro F1, with source standard errors archived. Both training regimes use AdamW, up to 10,000 steps and validation-only early stopping; segmentation decoders follow backbone output structure. Pretraining overlap varies and is stated for every result using Table 11, including unknown/unauditable data. These are observed transfer results, not a contamination-free ranking. Macro F1 and foreground-only Dice remain native metrics.","zh":"十六个超声专用及通用视觉底模，采用相同预处理的Fetal Plane测试集与固定划分，按数据集使用病人、序列或图片粒度。本组full-finetuning保留平均Macro F1，标准误存于证据。两训练条件均用AdamW、最多10,000步，只在验证集早停；分割解码器随底模输出结构选择。每条成绩按表11注明预训练数据重叠，包括未知及不可审计情况。F1与去除背景的Dice保留原生指标。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.28610v1","published_at":"2026-09-23","status":"paper_baselines"},"model_count":16,"result_count":16,"results":[[12734,0.835,370,12780,12736],[12737,0.83,370,12780,12736],[12738,0.81,370,12781,12736],[12740,0.844,370,12781,12736],[12741,0.85,370,12781,12736],[12742,0.868,370,12781,12736],[12743,0.781,370,12782,12736],[12745,0.848,370,12780,12736],[12747,0.802,370,12780,12736],[12748,0.787,370,12782,12736],[12749,0.823,370,12783,12736],[12750,0.817,370,12780,12736],[12751,0.915,370,12783,12736],[12752,0.93,370,12783,12736],[12753,0.87,370,12782,12736],[12754,0.863,370,12783,12736]]},{"id":"ultrabench2-classification-aul-mass-head-only-2026-paper","name":"UltraBench 2 · AUL Mass classification head-only (2026 paper)","group":"医疗","metric":"Macro F1","unit":"score","score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"public_history":true,"protocol":{"en":"Sixteen ultrasound-specialist and general vision backbones use the same prepared AUL Mass test set and fixed dataset-dependent patient/sequence/image splits. This head-only cohort reports mean Macro F1, with source standard errors archived. Both training regimes use AdamW, up to 10,000 steps and validation-only early stopping; segmentation decoders follow backbone output structure. Pretraining overlap varies and is stated for every result using Table 11, including unknown/unauditable data. These are observed transfer results, not a contamination-free ranking. Macro F1 and foreground-only Dice remain native metrics.","zh":"十六个超声专用及通用视觉底模，采用相同预处理的AUL Mass测试集与固定划分，按数据集使用病人、序列或图片粒度。本组head-only保留平均Macro F1，标准误存于证据。两训练条件均用AdamW、最多10,000步，只在验证集早停；分割解码器随底模输出结构选择。每条成绩按表11注明预训练数据重叠，包括未知及不可审计情况。F1与去除背景的Dice保留原生指标。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.28610v1","published_at":"2026-09-23","status":"paper_baselines"},"model_count":16,"result_count":16,"results":[[12734,0.622,370,12784,12736],[12737,0.658,370,12784,12736],[12738,0.592,370,12785,12736],[12740,0.642,370,12785,12736],[12741,0.706,370,12785,12736],[12742,0.721,370,12785,12736],[12743,0.632,370,12786,12736],[12745,0.718,370,12787,12736],[12747,0.544,370,12784,12736],[12748,0.642,370,12786,12736],[12749,0.657,370,12784,12736],[12750,0.639,370,12787,12736],[12751,0.672,370,12784,12736],[12752,0.694,370,12784,12736],[12753,0.646,370,12786,12736],[12754,0.655,370,12787,12736]]},{"id":"ultrabench2-classification-busbra-histology-head-only-2026-paper","name":"UltraBench 2 · BUS-BRA histology classification head-only (2026 paper)","group":"医疗","metric":"Macro F1","unit":"score","score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"public_history":true,"protocol":{"en":"Sixteen ultrasound-specialist and general vision backbones use the same prepared BUS-BRA histology test set and fixed dataset-dependent patient/sequence/image splits. This head-only cohort reports mean Macro F1, with source standard errors archived. Both training regimes use AdamW, up to 10,000 steps and validation-only early stopping; segmentation decoders follow backbone output structure. Pretraining overlap varies and is stated for every result using Table 11, including unknown/unauditable data. These are observed transfer results, not a contamination-free ranking. Macro F1 and foreground-only Dice remain native metrics.","zh":"十六个超声专用及通用视觉底模，采用相同预处理的BUS-BRA histology测试集与固定划分，按数据集使用病人、序列或图片粒度。本组head-only保留平均Macro F1，标准误存于证据。两训练条件均用AdamW、最多10,000步，只在验证集早停；分割解码器随底模输出结构选择。每条成绩按表11注明预训练数据重叠，包括未知及不可审计情况。F1与去除背景的Dice保留原生指标。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.28610v1","published_at":"2026-09-23","status":"paper_baselines"},"model_count":16,"result_count":16,"results":[[12734,0.14,370,12788,12736],[12737,0.163,370,12788,12736],[12738,0.137,370,12789,12736],[12740,0.16,370,12789,12736],[12741,0.164,370,12789,12736],[12742,0.147,370,12789,12736],[12743,0.128,370,12790,12736],[12745,0.162,370,12788,12736],[12747,0.129,370,12788,12736],[12748,0.117,370,12790,12736],[12749,0.158,370,12788,12736],[12750,0.163,370,12788,12736],[12751,0.164,370,12788,12736],[12752,0.186,370,12788,12736],[12753,0.131,370,12790,12736],[12754,0.151,370,12788,12736]]},{"id":"ultrabench2-classification-busbra-pathology-head-only-2026-paper","name":"UltraBench 2 · BUS-BRA pathology classification head-only (2026 paper)","group":"医疗","metric":"Macro F1","unit":"score","score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"public_history":true,"protocol":{"en":"Sixteen ultrasound-specialist and general vision backbones use the same prepared BUS-BRA pathology test set and fixed dataset-dependent patient/sequence/image splits. This head-only cohort reports mean Macro F1, with source standard errors archived. Both training regimes use AdamW, up to 10,000 steps and validation-only early stopping; segmentation decoders follow backbone output structure. Pretraining overlap varies and is stated for every result using Table 11, including unknown/unauditable data. These are observed transfer results, not a contamination-free ranking. Macro F1 and foreground-only Dice remain native metrics.","zh":"十六个超声专用及通用视觉底模，采用相同预处理的BUS-BRA pathology测试集与固定划分，按数据集使用病人、序列或图片粒度。本组head-only保留平均Macro F1，标准误存于证据。两训练条件均用AdamW、最多10,000步，只在验证集早停；分割解码器随底模输出结构选择。每条成绩按表11注明预训练数据重叠，包括未知及不可审计情况。F1与去除背景的Dice保留原生指标。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.28610v1","published_at":"2026-09-23","status":"paper_baselines"},"model_count":16,"result_count":16,"results":[[12734,0.696,370,12788,12736],[12737,0.678,370,12788,12736],[12738,0.726,370,12789,12736],[12740,0.746,370,12789,12736],[12741,0.689,370,12789,12736],[12742,0.725,370,12789,12736],[12743,0.62,370,12790,12736],[12745,0.701,370,12788,12736],[12747,0.625,370,12788,12736],[12748,0.636,370,12790,12736],[12749,0.691,370,12788,12736],[12750,0.683,370,12788,12736],[12751,0.748,370,12788,12736],[12752,0.769,370,12788,12736],[12753,0.675,370,12790,12736],[12754,0.703,370,12788,12736]]},{"id":"ultrabench2-classification-covid-blues-head-only-2026-paper","name":"UltraBench 2 · COVID-BLUES classification head-only (2026 paper)","group":"医疗","metric":"Macro F1","unit":"score","score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"public_history":true,"protocol":{"en":"Sixteen ultrasound-specialist and general vision backbones use the same prepared COVID-BLUES test set and fixed dataset-dependent patient/sequence/image splits. This head-only cohort reports mean Macro F1, with source standard errors archived. Both training regimes use AdamW, up to 10,000 steps and validation-only early stopping; segmentation decoders follow backbone output structure. Pretraining overlap varies and is stated for every result using Table 11, including unknown/unauditable data. These are observed transfer results, not a contamination-free ranking. Macro F1 and foreground-only Dice remain native metrics.","zh":"十六个超声专用及通用视觉底模，采用相同预处理的COVID-BLUES测试集与固定划分，按数据集使用病人、序列或图片粒度。本组head-only保留平均Macro F1，标准误存于证据。两训练条件均用AdamW、最多10,000步，只在验证集早停；分割解码器随底模输出结构选择。每条成绩按表11注明预训练数据重叠，包括未知及不可审计情况。F1与去除背景的Dice保留原生指标。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.28610v1","published_at":"2026-09-23","status":"paper_baselines"},"model_count":16,"result_count":16,"results":[[12734,0.619,370,12791,12736],[12737,0.689,370,12791,12736],[12738,0.606,370,12792,12736],[12740,0.721,370,12792,12736],[12741,0.588,370,12792,12736],[12742,0.684,370,12792,12736],[12743,0.658,370,12793,12736],[12745,0.701,370,12791,12736],[12747,0.719,370,12791,12736],[12748,0.601,370,12793,12736],[12749,0.821,370,12791,12736],[12750,0.76,370,12791,12736],[12751,0.672,370,12791,12736],[12752,0.749,370,12791,12736],[12753,0.619,370,12793,12736],[12754,0.766,370,12794,12736]]},{"id":"ultrabench2-classification-gbcu-head-only-2026-paper","name":"UltraBench 2 · GBCU classification head-only (2026 paper)","group":"医疗","metric":"Macro F1","unit":"score","score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"public_history":true,"protocol":{"en":"Sixteen ultrasound-specialist and general vision backbones use the same prepared GBCU test set and fixed dataset-dependent patient/sequence/image splits. This head-only cohort reports mean Macro F1, with source standard errors archived. Both training regimes use AdamW, up to 10,000 steps and validation-only early stopping; segmentation decoders follow backbone output structure. Pretraining overlap varies and is stated for every result using Table 11, including unknown/unauditable data. These are observed transfer results, not a contamination-free ranking. Macro F1 and foreground-only Dice remain native metrics.","zh":"十六个超声专用及通用视觉底模，采用相同预处理的GBCU测试集与固定划分，按数据集使用病人、序列或图片粒度。本组head-only保留平均Macro F1，标准误存于证据。两训练条件均用AdamW、最多10,000步，只在验证集早停；分割解码器随底模输出结构选择。每条成绩按表11注明预训练数据重叠，包括未知及不可审计情况。F1与去除背景的Dice保留原生指标。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.28610v1","published_at":"2026-09-23","status":"paper_baselines"},"model_count":16,"result_count":16,"results":[[12734,0.616,370,12795,12736],[12737,0.603,370,12795,12736],[12738,0.604,370,12796,12736],[12740,0.585,370,12796,12736],[12741,0.607,370,12796,12736],[12742,0.51,370,12796,12736],[12743,0.515,370,12797,12736],[12745,0.595,370,12795,12736],[12747,0.51,370,12795,12736],[12748,0.419,370,12797,12736],[12749,0.602,370,12795,12736],[12750,0.509,370,12795,12736],[12751,0.561,370,12795,12736],[12752,0.54,370,12795,12736],[12753,0.495,370,12797,12736],[12754,0.667,370,12795,12736]]},{"id":"ultrabench2-classification-masld-head-only-2026-paper","name":"UltraBench 2 · MASLD classification head-only (2026 paper)","group":"医疗","metric":"Macro F1","unit":"score","score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"public_history":true,"protocol":{"en":"Sixteen ultrasound-specialist and general vision backbones use the same prepared MASLD test set and fixed dataset-dependent patient/sequence/image splits. This head-only cohort reports mean Macro F1, with source standard errors archived. Both training regimes use AdamW, up to 10,000 steps and validation-only early stopping; segmentation decoders follow backbone output structure. Pretraining overlap varies and is stated for every result using Table 11, including unknown/unauditable data. These are observed transfer results, not a contamination-free ranking. Macro F1 and foreground-only Dice remain native metrics.","zh":"十六个超声专用及通用视觉底模，采用相同预处理的MASLD测试集与固定划分，按数据集使用病人、序列或图片粒度。本组head-only保留平均Macro F1，标准误存于证据。两训练条件均用AdamW、最多10,000步，只在验证集早停；分割解码器随底模输出结构选择。每条成绩按表11注明预训练数据重叠，包括未知及不可审计情况。F1与去除背景的Dice保留原生指标。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.28610v1","published_at":"2026-09-23","status":"paper_baselines"},"model_count":16,"result_count":16,"results":[[12734,0.771,370,12798,12736],[12737,0.599,370,12798,12736],[12738,0.647,370,12799,12736],[12740,0.504,370,12799,12736],[12741,0.8,370,12799,12736],[12742,0.87,370,12799,12736],[12743,0.73,370,12800,12736],[12745,0.707,370,12798,12736],[12747,0.694,370,12798,12736],[12748,0.675,370,12800,12736],[12749,0.771,370,12798,12736],[12750,1.0,370,12798,12736],[12751,0.771,370,12798,12736],[12752,0.389,370,12798,12736],[12753,0.389,370,12800,12736],[12754,0.675,370,12798,12736]]},{"id":"ultrabench2-classification-mmotu-head-only-2026-paper","name":"UltraBench 2 · MMOTU classification head-only (2026 paper)","group":"医疗","metric":"Macro F1","unit":"score","score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"public_history":true,"protocol":{"en":"Sixteen ultrasound-specialist and general vision backbones use the same prepared MMOTU test set and fixed dataset-dependent patient/sequence/image splits. This head-only cohort reports mean Macro F1, with source standard errors archived. Both training regimes use AdamW, up to 10,000 steps and validation-only early stopping; segmentation decoders follow backbone output structure. Pretraining overlap varies and is stated for every result using Table 11, including unknown/unauditable data. These are observed transfer results, not a contamination-free ranking. Macro F1 and foreground-only Dice remain native metrics.","zh":"十六个超声专用及通用视觉底模，采用相同预处理的MMOTU测试集与固定划分，按数据集使用病人、序列或图片粒度。本组head-only保留平均Macro F1，标准误存于证据。两训练条件均用AdamW、最多10,000步，只在验证集早停；分割解码器随底模输出结构选择。每条成绩按表11注明预训练数据重叠，包括未知及不可审计情况。F1与去除背景的Dice保留原生指标。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.28610v1","published_at":"2026-09-23","status":"paper_baselines"},"model_count":16,"result_count":16,"results":[[12734,0.392,370,12801,12736],[12737,0.436,370,12801,12736],[12738,0.341,370,12802,12736],[12740,0.395,370,12802,12736],[12741,0.388,370,12802,12736],[12742,0.369,370,12802,12736],[12743,0.347,370,12803,12736],[12745,0.423,370,12804,12736],[12747,0.242,370,12801,12736],[12748,0.256,370,12803,12736],[12749,0.459,370,12801,12736],[12750,0.368,370,12801,12736],[12751,0.427,370,12801,12736],[12752,0.431,370,12801,12736],[12753,0.302,370,12803,12736],[12754,0.397,370,12801,12736]]},{"id":"ultrabench2-classification-tn3k-head-only-2026-paper","name":"UltraBench 2 · TN3K classification head-only (2026 paper)","group":"医疗","metric":"Macro F1","unit":"score","score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"public_history":true,"protocol":{"en":"Sixteen ultrasound-specialist and general vision backbones use the same prepared TN3K test set and fixed dataset-dependent patient/sequence/image splits. This head-only cohort reports mean Macro F1, with source standard errors archived. Both training regimes use AdamW, up to 10,000 steps and validation-only early stopping; segmentation decoders follow backbone output structure. Pretraining overlap varies and is stated for every result using Table 11, including unknown/unauditable data. These are observed transfer results, not a contamination-free ranking. Macro F1 and foreground-only Dice remain native metrics.","zh":"十六个超声专用及通用视觉底模，采用相同预处理的TN3K测试集与固定划分，按数据集使用病人、序列或图片粒度。本组head-only保留平均Macro F1，标准误存于证据。两训练条件均用AdamW、最多10,000步，只在验证集早停；分割解码器随底模输出结构选择。每条成绩按表11注明预训练数据重叠，包括未知及不可审计情况。F1与去除背景的Dice保留原生指标。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.28610v1","published_at":"2026-09-23","status":"paper_baselines"},"model_count":16,"result_count":16,"results":[[12734,0.618,370,12805,12736],[12737,0.631,370,12805,12736],[12738,0.608,370,12806,12736],[12740,0.612,370,12806,12736],[12741,0.643,370,12806,12736],[12742,0.635,370,12806,12736],[12743,0.574,370,12807,12736],[12745,0.593,370,12805,12736],[12747,0.6,370,12808,12736],[12748,0.569,370,12807,12736],[12749,0.575,370,12805,12736],[12750,0.564,370,12808,12736],[12751,0.652,370,12805,12736],[12752,0.653,370,12805,12736],[12753,0.574,370,12807,12736],[12754,0.604,370,12808,12736]]},{"id":"ultrabench2-classification-camus-head-only-2026-paper","name":"UltraBench 2 · CAMUS classification head-only (2026 paper)","group":"医疗","metric":"Macro F1","unit":"score","score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"public_history":true,"protocol":{"en":"Sixteen ultrasound-specialist and general vision backbones use the same prepared CAMUS test set and fixed dataset-dependent patient/sequence/image splits. This head-only cohort reports mean Macro F1, with source standard errors archived. Both training regimes use AdamW, up to 10,000 steps and validation-only early stopping; segmentation decoders follow backbone output structure. Pretraining overlap varies and is stated for every result using Table 11, including unknown/unauditable data. These are observed transfer results, not a contamination-free ranking. Macro F1 and foreground-only Dice remain native metrics.","zh":"十六个超声专用及通用视觉底模，采用相同预处理的CAMUS测试集与固定划分，按数据集使用病人、序列或图片粒度。本组head-only保留平均Macro F1，标准误存于证据。两训练条件均用AdamW、最多10,000步，只在验证集早停；分割解码器随底模输出结构选择。每条成绩按表11注明预训练数据重叠，包括未知及不可审计情况。F1与去除背景的Dice保留原生指标。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.28610v1","published_at":"2026-09-23","status":"paper_baselines"},"model_count":16,"result_count":16,"results":[[12734,0.645,370,12809,12736],[12737,0.642,370,12809,12736],[12738,0.603,370,12810,12736],[12740,0.622,370,12810,12736],[12741,0.684,370,12810,12736],[12742,0.633,370,12810,12736],[12743,0.672,370,12811,12736],[12745,0.683,370,12812,12736],[12747,0.508,370,12812,12736],[12748,0.46,370,12811,12736],[12749,0.65,370,12812,12736],[12750,0.653,370,12812,12736],[12751,0.622,370,12812,12736],[12752,0.642,370,12812,12736],[12753,0.567,370,12811,12736],[12754,0.659,370,12812,12736]]},{"id":"ultrabench2-classification-fetal-brain-head-only-2026-paper","name":"UltraBench 2 · Fetal Brain classification head-only (2026 paper)","group":"医疗","metric":"Macro F1","unit":"score","score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"public_history":true,"protocol":{"en":"Sixteen ultrasound-specialist and general vision backbones use the same prepared Fetal Brain test set and fixed dataset-dependent patient/sequence/image splits. This head-only cohort reports mean Macro F1, with source standard errors archived. Both training regimes use AdamW, up to 10,000 steps and validation-only early stopping; segmentation decoders follow backbone output structure. Pretraining overlap varies and is stated for every result using Table 11, including unknown/unauditable data. These are observed transfer results, not a contamination-free ranking. Macro F1 and foreground-only Dice remain native metrics.","zh":"十六个超声专用及通用视觉底模，采用相同预处理的Fetal Brain测试集与固定划分，按数据集使用病人、序列或图片粒度。本组head-only保留平均Macro F1，标准误存于证据。两训练条件均用AdamW、最多10,000步，只在验证集早停；分割解码器随底模输出结构选择。每条成绩按表11注明预训练数据重叠，包括未知及不可审计情况。F1与去除背景的Dice保留原生指标。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.28610v1","published_at":"2026-09-23","status":"paper_baselines"},"model_count":16,"result_count":16,"results":[[12734,0.46,370,12813,12736],[12737,0.464,370,12813,12736],[12738,0.509,370,12814,12736],[12740,0.551,370,12814,12736],[12741,0.535,370,12814,12736],[12742,0.559,370,12814,12736],[12743,0.423,370,12815,12736],[12745,0.536,370,12813,12736],[12747,0.377,370,12813,12736],[12748,0.312,370,12815,12736],[12749,0.455,370,12816,12736],[12750,0.494,370,12813,12736],[12751,0.595,370,12816,12736],[12752,0.607,370,12816,12736],[12753,0.418,370,12815,12736],[12754,0.58,370,12816,12736]]},{"id":"ultrabench2-classification-fetal-plane-head-only-2026-paper","name":"UltraBench 2 · Fetal Plane classification head-only (2026 paper)","group":"医疗","metric":"Macro F1","unit":"score","score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"public_history":true,"protocol":{"en":"Sixteen ultrasound-specialist and general vision backbones use the same prepared Fetal Plane test set and fixed dataset-dependent patient/sequence/image splits. This head-only cohort reports mean Macro F1, with source standard errors archived. Both training regimes use AdamW, up to 10,000 steps and validation-only early stopping; segmentation decoders follow backbone output structure. Pretraining overlap varies and is stated for every result using Table 11, including unknown/unauditable data. These are observed transfer results, not a contamination-free ranking. Macro F1 and foreground-only Dice remain native metrics.","zh":"十六个超声专用及通用视觉底模，采用相同预处理的Fetal Plane测试集与固定划分，按数据集使用病人、序列或图片粒度。本组head-only保留平均Macro F1，标准误存于证据。两训练条件均用AdamW、最多10,000步，只在验证集早停；分割解码器随底模输出结构选择。每条成绩按表11注明预训练数据重叠，包括未知及不可审计情况。F1与去除背景的Dice保留原生指标。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.28610v1","published_at":"2026-09-23","status":"paper_baselines"},"model_count":16,"result_count":16,"results":[[12734,0.755,370,12813,12736],[12737,0.727,370,12813,12736],[12738,0.794,370,12814,12736],[12740,0.834,370,12814,12736],[12741,0.838,370,12814,12736],[12742,0.839,370,12814,12736],[12743,0.717,370,12815,12736],[12745,0.825,370,12813,12736],[12747,0.682,370,12813,12736],[12748,0.62,370,12815,12736],[12749,0.794,370,12816,12736],[12750,0.778,370,12813,12736],[12751,0.884,370,12816,12736],[12752,0.904,370,12816,12736],[12753,0.757,370,12815,12736],[12754,0.836,370,12816,12736]]},{"id":"ultrabench2-segmentation-aul-mass-full-finetuning-2026-paper","name":"UltraBench 2 · AUL Mass segmentation full-finetuning (2026 paper)","group":"医疗","metric":"Positive Dice","unit":"score","score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"public_history":true,"protocol":{"en":"Sixteen ultrasound-specialist and general vision backbones use the same prepared AUL Mass test set and fixed dataset-dependent patient/sequence/image splits. This full-finetuning cohort reports mean Positive Dice, with source standard errors archived. Both training regimes use AdamW, up to 10,000 steps and validation-only early stopping; segmentation decoders follow backbone output structure. Pretraining overlap varies and is stated for every result using Table 11, including unknown/unauditable data. These are observed transfer results, not a contamination-free ranking. Macro F1 and foreground-only Dice remain native metrics.","zh":"十六个超声专用及通用视觉底模，采用相同预处理的AUL Mass测试集与固定划分，按数据集使用病人、序列或图片粒度。本组full-finetuning保留平均Positive Dice，标准误存于证据。两训练条件均用AdamW、最多10,000步，只在验证集早停；分割解码器随底模输出结构选择。每条成绩按表11注明预训练数据重叠，包括未知及不可审计情况。F1与去除背景的Dice保留原生指标。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.28610v1","published_at":"2026-09-23","status":"paper_baselines"},"model_count":16,"result_count":16,"results":[[12734,0.573,370,12735,12736],[12737,0.547,370,12735,12736],[12738,0.609,370,12739,12736],[12740,0.642,370,12739,12736],[12741,0.584,370,12739,12736],[12742,0.603,370,12739,12736],[12743,0.489,370,12744,12736],[12745,0.565,370,12746,12736],[12747,0.603,370,12735,12736],[12748,0.448,370,12744,12736],[12749,0.519,370,12735,12736],[12750,0.532,370,12746,12736],[12751,0.601,370,12735,12736],[12752,0.627,370,12735,12736],[12753,0.542,370,12744,12736],[12754,0.515,370,12746,12736]]},{"id":"ultrabench2-segmentation-busbra-full-finetuning-2026-paper","name":"UltraBench 2 · BUS-BRA segmentation full-finetuning (2026 paper)","group":"医疗","metric":"Positive Dice","unit":"score","score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"public_history":true,"protocol":{"en":"Sixteen ultrasound-specialist and general vision backbones use the same prepared BUS-BRA test set and fixed dataset-dependent patient/sequence/image splits. This full-finetuning cohort reports mean Positive Dice, with source standard errors archived. Both training regimes use AdamW, up to 10,000 steps and validation-only early stopping; segmentation decoders follow backbone output structure. Pretraining overlap varies and is stated for every result using Table 11, including unknown/unauditable data. These are observed transfer results, not a contamination-free ranking. Macro F1 and foreground-only Dice remain native metrics.","zh":"十六个超声专用及通用视觉底模，采用相同预处理的BUS-BRA测试集与固定划分，按数据集使用病人、序列或图片粒度。本组full-finetuning保留平均Positive Dice，标准误存于证据。两训练条件均用AdamW、最多10,000步，只在验证集早停；分割解码器随底模输出结构选择。每条成绩按表11注明预训练数据重叠，包括未知及不可审计情况。F1与去除背景的Dice保留原生指标。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.28610v1","published_at":"2026-09-23","status":"paper_baselines"},"model_count":16,"result_count":16,"results":[[12734,0.869,370,12755,12736],[12737,0.865,370,12755,12736],[12738,0.885,370,12756,12736],[12740,0.89,370,12756,12736],[12741,0.872,370,12756,12736],[12742,0.882,370,12756,12736],[12743,0.844,370,12757,12736],[12745,0.88,370,12755,12736],[12747,0.886,370,12755,12736],[12748,0.844,370,12757,12736],[12749,0.837,370,12755,12736],[12750,0.86,370,12755,12736],[12751,0.884,370,12755,12736],[12752,0.887,370,12755,12736],[12753,0.867,370,12757,12736],[12754,0.839,370,12755,12736]]},{"id":"ultrabench2-segmentation-mmotu-full-finetuning-2026-paper","name":"UltraBench 2 · MMOTU segmentation full-finetuning (2026 paper)","group":"医疗","metric":"Positive Dice","unit":"score","score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"public_history":true,"protocol":{"en":"Sixteen ultrasound-specialist and general vision backbones use the same prepared MMOTU test set and fixed dataset-dependent patient/sequence/image splits. This full-finetuning cohort reports mean Positive Dice, with source standard errors archived. Both training regimes use AdamW, up to 10,000 steps and validation-only early stopping; segmentation decoders follow backbone output structure. Pretraining overlap varies and is stated for every result using Table 11, including unknown/unauditable data. These are observed transfer results, not a contamination-free ranking. Macro F1 and foreground-only Dice remain native metrics.","zh":"十六个超声专用及通用视觉底模，采用相同预处理的MMOTU测试集与固定划分，按数据集使用病人、序列或图片粒度。本组full-finetuning保留平均Positive Dice，标准误存于证据。两训练条件均用AdamW、最多10,000步，只在验证集早停；分割解码器随底模输出结构选择。每条成绩按表11注明预训练数据重叠，包括未知及不可审计情况。F1与去除背景的Dice保留原生指标。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.28610v1","published_at":"2026-09-23","status":"paper_baselines"},"model_count":16,"result_count":16,"results":[[12734,0.789,370,12768,12736],[12737,0.783,370,12768,12736],[12738,0.823,370,12769,12736],[12740,0.832,370,12769,12736],[12741,0.82,370,12769,12736],[12742,0.827,370,12769,12736],[12743,0.776,370,12770,12736],[12745,0.82,370,12771,12736],[12747,0.807,370,12768,12736],[12748,0.747,370,12770,12736],[12749,0.769,370,12768,12736],[12750,0.758,370,12768,12736],[12751,0.788,370,12768,12736],[12752,0.796,370,12768,12736],[12753,0.784,370,12770,12736],[12754,0.774,370,12768,12736]]},{"id":"ultrabench2-segmentation-tn3k-full-finetuning-2026-paper","name":"UltraBench 2 · TN3K segmentation full-finetuning (2026 paper)","group":"医疗","metric":"Positive Dice","unit":"score","score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"public_history":true,"protocol":{"en":"Sixteen ultrasound-specialist and general vision backbones use the same prepared TN3K test set and fixed dataset-dependent patient/sequence/image splits. This full-finetuning cohort reports mean Positive Dice, with source standard errors archived. Both training regimes use AdamW, up to 10,000 steps and validation-only early stopping; segmentation decoders follow backbone output structure. Pretraining overlap varies and is stated for every result using Table 11, including unknown/unauditable data. These are observed transfer results, not a contamination-free ranking. Macro F1 and foreground-only Dice remain native metrics.","zh":"十六个超声专用及通用视觉底模，采用相同预处理的TN3K测试集与固定划分，按数据集使用病人、序列或图片粒度。本组full-finetuning保留平均Positive Dice，标准误存于证据。两训练条件均用AdamW、最多10,000步，只在验证集早停；分割解码器随底模输出结构选择。每条成绩按表11注明预训练数据重叠，包括未知及不可审计情况。F1与去除背景的Dice保留原生指标。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.28610v1","published_at":"2026-09-23","status":"paper_baselines"},"model_count":16,"result_count":16,"results":[[12734,0.804,370,12772,12736],[12737,0.792,370,12772,12736],[12738,0.804,370,12773,12736],[12740,0.811,370,12773,12736],[12741,0.798,370,12773,12736],[12742,0.784,370,12773,12736],[12743,0.739,370,12774,12736],[12745,0.787,370,12772,12736],[12747,0.808,370,12775,12736],[12748,0.721,370,12774,12736],[12749,0.734,370,12772,12736],[12750,0.747,370,12775,12736],[12751,0.809,370,12772,12736],[12752,0.819,370,12772,12736],[12753,0.788,370,12774,12736],[12754,0.742,370,12775,12736]]},{"id":"ultrabench2-segmentation-aul-liver-full-finetuning-2026-paper","name":"UltraBench 2 · AUL Liver segmentation full-finetuning (2026 paper)","group":"医疗","metric":"Positive Dice","unit":"score","score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"public_history":true,"protocol":{"en":"Sixteen ultrasound-specialist and general vision backbones use the same prepared AUL Liver test set and fixed dataset-dependent patient/sequence/image splits. This full-finetuning cohort reports mean Positive Dice, with source standard errors archived. Both training regimes use AdamW, up to 10,000 steps and validation-only early stopping; segmentation decoders follow backbone output structure. Pretraining overlap varies and is stated for every result using Table 11, including unknown/unauditable data. These are observed transfer results, not a contamination-free ranking. Macro F1 and foreground-only Dice remain native metrics.","zh":"十六个超声专用及通用视觉底模，采用相同预处理的AUL Liver测试集与固定划分，按数据集使用病人、序列或图片粒度。本组full-finetuning保留平均Positive Dice，标准误存于证据。两训练条件均用AdamW、最多10,000步，只在验证集早停；分割解码器随底模输出结构选择。每条成绩按表11注明预训练数据重叠，包括未知及不可审计情况。F1与去除背景的Dice保留原生指标。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.28610v1","published_at":"2026-09-23","status":"paper_baselines"},"model_count":16,"result_count":16,"results":[[12734,0.9,370,12735,12736],[12737,0.894,370,12735,12736],[12738,0.908,370,12739,12736],[12740,0.914,370,12739,12736],[12741,0.906,370,12739,12736],[12742,0.912,370,12739,12736],[12743,0.894,370,12744,12736],[12745,0.913,370,12746,12736],[12747,0.908,370,12735,12736],[12748,0.878,370,12744,12736],[12749,0.889,370,12735,12736],[12750,0.89,370,12746,12736],[12751,0.909,370,12735,12736],[12752,0.913,370,12735,12736],[12753,0.899,370,12744,12736],[12754,0.901,370,12746,12736]]},{"id":"ultrabench2-segmentation-camus-full-finetuning-2026-paper","name":"UltraBench 2 · CAMUS segmentation full-finetuning (2026 paper)","group":"医疗","metric":"Positive Dice","unit":"score","score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"public_history":true,"protocol":{"en":"Sixteen ultrasound-specialist and general vision backbones use the same prepared CAMUS test set and fixed dataset-dependent patient/sequence/image splits. This full-finetuning cohort reports mean Positive Dice, with source standard errors archived. Both training regimes use AdamW, up to 10,000 steps and validation-only early stopping; segmentation decoders follow backbone output structure. Pretraining overlap varies and is stated for every result using Table 11, including unknown/unauditable data. These are observed transfer results, not a contamination-free ranking. Macro F1 and foreground-only Dice remain native metrics.","zh":"十六个超声专用及通用视觉底模，采用相同预处理的CAMUS测试集与固定划分，按数据集使用病人、序列或图片粒度。本组full-finetuning保留平均Positive Dice，标准误存于证据。两训练条件均用AdamW、最多10,000步，只在验证集早停；分割解码器随底模输出结构选择。每条成绩按表11注明预训练数据重叠，包括未知及不可审计情况。F1与去除背景的Dice保留原生指标。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.28610v1","published_at":"2026-09-23","status":"paper_baselines"},"model_count":16,"result_count":16,"results":[[12734,0.893,370,12776,12736],[12737,0.892,370,12776,12736],[12738,0.891,370,12777,12736],[12740,0.89,370,12777,12736],[12741,0.889,370,12777,12736],[12742,0.89,370,12777,12736],[12743,0.889,370,12778,12736],[12745,0.901,370,12779,12736],[12747,0.906,370,12779,12736],[12748,0.885,370,12778,12736],[12749,0.89,370,12779,12736],[12750,0.892,370,12779,12736],[12751,0.893,370,12779,12736],[12752,0.894,370,12779,12736],[12753,0.895,370,12778,12736],[12754,0.898,370,12779,12736]]},{"id":"ultrabench2-segmentation-ok-capsule-full-finetuning-2026-paper","name":"UltraBench 2 · Open Kidney capsule segmentation full-finetuning (2026 paper)","group":"医疗","metric":"Positive Dice","unit":"score","score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"public_history":true,"protocol":{"en":"Sixteen ultrasound-specialist and general vision backbones use the same prepared Open Kidney capsule test set and fixed dataset-dependent patient/sequence/image splits. This full-finetuning cohort reports mean Positive Dice, with source standard errors archived. Both training regimes use AdamW, up to 10,000 steps and validation-only early stopping; segmentation decoders follow backbone output structure. Pretraining overlap varies and is stated for every result using Table 11, including unknown/unauditable data. These are observed transfer results, not a contamination-free ranking. Macro F1 and foreground-only Dice remain native metrics.","zh":"十六个超声专用及通用视觉底模，采用相同预处理的Open Kidney capsule测试集与固定划分，按数据集使用病人、序列或图片粒度。本组full-finetuning保留平均Positive Dice，标准误存于证据。两训练条件均用AdamW、最多10,000步，只在验证集早停；分割解码器随底模输出结构选择。每条成绩按表11注明预训练数据重叠，包括未知及不可审计情况。F1与去除背景的Dice保留原生指标。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.28610v1","published_at":"2026-09-23","status":"paper_baselines"},"model_count":16,"result_count":16,"results":[[12734,0.821,370,12817,12736],[12737,0.82,370,12817,12736],[12738,0.829,370,12818,12736],[12740,0.825,370,12818,12736],[12741,0.829,370,12818,12736],[12742,0.822,370,12818,12736],[12743,0.795,370,12819,12736],[12745,0.84,370,12817,12736],[12747,0.853,370,12817,12736],[12748,0.806,370,12819,12736],[12749,0.802,370,12817,12736],[12750,0.804,370,12820,12736],[12751,0.832,370,12817,12736],[12752,0.827,370,12817,12736],[12753,0.804,370,12819,12736],[12754,0.813,370,12820,12736]]},{"id":"ultrabench2-segmentation-ok-regions-full-finetuning-2026-paper","name":"UltraBench 2 · Open Kidney regions segmentation full-finetuning (2026 paper)","group":"医疗","metric":"Positive Dice","unit":"score","score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"public_history":true,"protocol":{"en":"Sixteen ultrasound-specialist and general vision backbones use the same prepared Open Kidney regions test set and fixed dataset-dependent patient/sequence/image splits. This full-finetuning cohort reports mean Positive Dice, with source standard errors archived. Both training regimes use AdamW, up to 10,000 steps and validation-only early stopping; segmentation decoders follow backbone output structure. Pretraining overlap varies and is stated for every result using Table 11, including unknown/unauditable data. These are observed transfer results, not a contamination-free ranking. Macro F1 and foreground-only Dice remain native metrics.","zh":"十六个超声专用及通用视觉底模，采用相同预处理的Open Kidney regions测试集与固定划分，按数据集使用病人、序列或图片粒度。本组full-finetuning保留平均Positive Dice，标准误存于证据。两训练条件均用AdamW、最多10,000步，只在验证集早停；分割解码器随底模输出结构选择。每条成绩按表11注明预训练数据重叠，包括未知及不可审计情况。F1与去除背景的Dice保留原生指标。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.28610v1","published_at":"2026-09-23","status":"paper_baselines"},"model_count":16,"result_count":16,"results":[[12734,0.439,370,12817,12736],[12737,0.439,370,12817,12736],[12738,0.487,370,12818,12736],[12740,0.485,370,12818,12736],[12741,0.43,370,12818,12736],[12742,0.457,370,12818,12736],[12743,0.418,370,12819,12736],[12745,0.457,370,12817,12736],[12747,0.467,370,12817,12736],[12748,0.432,370,12819,12736],[12749,0.447,370,12817,12736],[12750,0.45,370,12820,12736],[12751,0.471,370,12817,12736],[12752,0.477,370,12817,12736],[12753,0.456,370,12819,12736],[12754,0.45,370,12820,12736]]},{"id":"ultrabench2-segmentation-psfhs-full-finetuning-2026-paper","name":"UltraBench 2 · PSFHS segmentation full-finetuning (2026 paper)","group":"医疗","metric":"Positive Dice","unit":"score","score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"public_history":true,"protocol":{"en":"Sixteen ultrasound-specialist and general vision backbones use the same prepared PSFHS test set and fixed dataset-dependent patient/sequence/image splits. This full-finetuning cohort reports mean Positive Dice, with source standard errors archived. Both training regimes use AdamW, up to 10,000 steps and validation-only early stopping; segmentation decoders follow backbone output structure. Pretraining overlap varies and is stated for every result using Table 11, including unknown/unauditable data. These are observed transfer results, not a contamination-free ranking. Macro F1 and foreground-only Dice remain native metrics.","zh":"十六个超声专用及通用视觉底模，采用相同预处理的PSFHS测试集与固定划分，按数据集使用病人、序列或图片粒度。本组full-finetuning保留平均Positive Dice，标准误存于证据。两训练条件均用AdamW、最多10,000步，只在验证集早停；分割解码器随底模输出结构选择。每条成绩按表11注明预训练数据重叠，包括未知及不可审计情况。F1与去除背景的Dice保留原生指标。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.28610v1","published_at":"2026-09-23","status":"paper_baselines"},"model_count":16,"result_count":16,"results":[[12734,0.941,370,12821,12736],[12737,0.941,370,12821,12736],[12738,0.938,370,12822,12736],[12740,0.94,370,12822,12736],[12741,0.936,370,12822,12736],[12742,0.935,370,12822,12736],[12743,0.938,370,12823,12736],[12745,0.942,370,12821,12736],[12747,0.943,370,12821,12736],[12748,0.937,370,12823,12736],[12749,0.938,370,12821,12736],[12750,0.941,370,12824,12736],[12751,0.94,370,12821,12736],[12752,0.94,370,12821,12736],[12753,0.941,370,12823,12736],[12754,0.941,370,12824,12736]]},{"id":"ultrabench2-segmentation-stmus-full-finetuning-2026-paper","name":"UltraBench 2 · STMUS segmentation full-finetuning (2026 paper)","group":"医疗","metric":"Positive Dice","unit":"score","score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"public_history":true,"protocol":{"en":"Sixteen ultrasound-specialist and general vision backbones use the same prepared STMUS test set and fixed dataset-dependent patient/sequence/image splits. This full-finetuning cohort reports mean Positive Dice, with source standard errors archived. Both training regimes use AdamW, up to 10,000 steps and validation-only early stopping; segmentation decoders follow backbone output structure. Pretraining overlap varies and is stated for every result using Table 11, including unknown/unauditable data. These are observed transfer results, not a contamination-free ranking. Macro F1 and foreground-only Dice remain native metrics.","zh":"十六个超声专用及通用视觉底模，采用相同预处理的STMUS测试集与固定划分，按数据集使用病人、序列或图片粒度。本组full-finetuning保留平均Positive Dice，标准误存于证据。两训练条件均用AdamW、最多10,000步，只在验证集早停；分割解码器随底模输出结构选择。每条成绩按表11注明预训练数据重叠，包括未知及不可审计情况。F1与去除背景的Dice保留原生指标。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.28610v1","published_at":"2026-09-23","status":"paper_baselines"},"model_count":16,"result_count":16,"results":[[12734,0.874,370,12825,12736],[12737,0.877,370,12825,12736],[12738,0.871,370,12826,12736],[12740,0.874,370,12826,12736],[12741,0.879,370,12826,12736],[12742,0.876,370,12826,12736],[12743,0.87,370,12827,12736],[12745,0.88,370,12828,12736],[12747,0.877,370,12825,12736],[12748,0.867,370,12827,12736],[12749,0.873,370,12825,12736],[12750,0.876,370,12828,12736],[12751,0.88,370,12825,12736],[12752,0.879,370,12825,12736],[12753,0.881,370,12827,12736],[12754,0.876,370,12828,12736]]},{"id":"ultrabench2-segmentation-aul-mass-head-only-2026-paper","name":"UltraBench 2 · AUL Mass segmentation head-only (2026 paper)","group":"医疗","metric":"Positive Dice","unit":"score","score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"public_history":true,"protocol":{"en":"Sixteen ultrasound-specialist and general vision backbones use the same prepared AUL Mass test set and fixed dataset-dependent patient/sequence/image splits. This head-only cohort reports mean Positive Dice, with source standard errors archived. Both training regimes use AdamW, up to 10,000 steps and validation-only early stopping; segmentation decoders follow backbone output structure. Pretraining overlap varies and is stated for every result using Table 11, including unknown/unauditable data. These are observed transfer results, not a contamination-free ranking. Macro F1 and foreground-only Dice remain native metrics.","zh":"十六个超声专用及通用视觉底模，采用相同预处理的AUL Mass测试集与固定划分，按数据集使用病人、序列或图片粒度。本组head-only保留平均Positive Dice，标准误存于证据。两训练条件均用AdamW、最多10,000步，只在验证集早停；分割解码器随底模输出结构选择。每条成绩按表11注明预训练数据重叠，包括未知及不可审计情况。F1与去除背景的Dice保留原生指标。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.28610v1","published_at":"2026-09-23","status":"paper_baselines"},"model_count":16,"result_count":16,"results":[[12734,0.395,370,12784,12829],[12737,0.406,370,12784,12829],[12738,0.407,370,12785,12829],[12740,0.449,370,12785,12829],[12741,0.432,370,12785,12829],[12742,0.418,370,12785,12829],[12743,0.354,370,12786,12829],[12745,0.49,370,12787,12829],[12747,0.339,370,12784,12829],[12748,0.358,370,12786,12829],[12749,0.428,370,12784,12829],[12750,0.419,370,12787,12829],[12751,0.491,370,12784,12829],[12752,0.508,370,12784,12829],[12753,0.383,370,12786,12829],[12754,0.435,370,12787,12829]]},{"id":"ultrabench2-segmentation-busbra-head-only-2026-paper","name":"UltraBench 2 · BUS-BRA segmentation head-only (2026 paper)","group":"医疗","metric":"Positive Dice","unit":"score","score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"public_history":true,"protocol":{"en":"Sixteen ultrasound-specialist and general vision backbones use the same prepared BUS-BRA test set and fixed dataset-dependent patient/sequence/image splits. This head-only cohort reports mean Positive Dice, with source standard errors archived. Both training regimes use AdamW, up to 10,000 steps and validation-only early stopping; segmentation decoders follow backbone output structure. Pretraining overlap varies and is stated for every result using Table 11, including unknown/unauditable data. These are observed transfer results, not a contamination-free ranking. Macro F1 and foreground-only Dice remain native metrics.","zh":"十六个超声专用及通用视觉底模，采用相同预处理的BUS-BRA测试集与固定划分，按数据集使用病人、序列或图片粒度。本组head-only保留平均Positive Dice，标准误存于证据。两训练条件均用AdamW、最多10,000步，只在验证集早停；分割解码器随底模输出结构选择。每条成绩按表11注明预训练数据重叠，包括未知及不可审计情况。F1与去除背景的Dice保留原生指标。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.28610v1","published_at":"2026-09-23","status":"paper_baselines"},"model_count":16,"result_count":16,"results":[[12734,0.806,370,12788,12829],[12737,0.801,370,12788,12829],[12738,0.788,370,12789,12829],[12740,0.803,370,12789,12829],[12741,0.803,370,12789,12829],[12742,0.79,370,12789,12829],[12743,0.706,370,12790,12829],[12745,0.827,370,12788,12829],[12747,0.827,370,12788,12829],[12748,0.786,370,12790,12829],[12749,0.799,370,12788,12829],[12750,0.816,370,12788,12829],[12751,0.862,370,12788,12829],[12752,0.868,370,12788,12829],[12753,0.808,370,12790,12829],[12754,0.804,370,12788,12829]]},{"id":"ultrabench2-segmentation-mmotu-head-only-2026-paper","name":"UltraBench 2 · MMOTU segmentation head-only (2026 paper)","group":"医疗","metric":"Positive Dice","unit":"score","score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"public_history":true,"protocol":{"en":"Sixteen ultrasound-specialist and general vision backbones use the same prepared MMOTU test set and fixed dataset-dependent patient/sequence/image splits. This head-only cohort reports mean Positive Dice, with source standard errors archived. Both training regimes use AdamW, up to 10,000 steps and validation-only early stopping; segmentation decoders follow backbone output structure. Pretraining overlap varies and is stated for every result using Table 11, including unknown/unauditable data. These are observed transfer results, not a contamination-free ranking. Macro F1 and foreground-only Dice remain native metrics.","zh":"十六个超声专用及通用视觉底模，采用相同预处理的MMOTU测试集与固定划分，按数据集使用病人、序列或图片粒度。本组head-only保留平均Positive Dice，标准误存于证据。两训练条件均用AdamW、最多10,000步，只在验证集早停；分割解码器随底模输出结构选择。每条成绩按表11注明预训练数据重叠，包括未知及不可审计情况。F1与去除背景的Dice保留原生指标。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.28610v1","published_at":"2026-09-23","status":"paper_baselines"},"model_count":16,"result_count":16,"results":[[12734,0.724,370,12801,12829],[12737,0.731,370,12801,12829],[12738,0.734,370,12802,12829],[12740,0.756,370,12802,12829],[12741,0.74,370,12802,12829],[12742,0.747,370,12802,12829],[12743,0.699,370,12803,12829],[12745,0.769,370,12804,12829],[12747,0.731,370,12801,12829],[12748,0.71,370,12803,12829],[12749,0.741,370,12801,12829],[12750,0.73,370,12801,12829],[12751,0.769,370,12801,12829],[12752,0.775,370,12801,12829],[12753,0.719,370,12803,12829],[12754,0.75,370,12801,12829]]},{"id":"ultrabench2-segmentation-tn3k-head-only-2026-paper","name":"UltraBench 2 · TN3K segmentation head-only (2026 paper)","group":"医疗","metric":"Positive Dice","unit":"score","score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"public_history":true,"protocol":{"en":"Sixteen ultrasound-specialist and general vision backbones use the same prepared TN3K test set and fixed dataset-dependent patient/sequence/image splits. This head-only cohort reports mean Positive Dice, with source standard errors archived. Both training regimes use AdamW, up to 10,000 steps and validation-only early stopping; segmentation decoders follow backbone output structure. Pretraining overlap varies and is stated for every result using Table 11, including unknown/unauditable data. These are observed transfer results, not a contamination-free ranking. Macro F1 and foreground-only Dice remain native metrics.","zh":"十六个超声专用及通用视觉底模，采用相同预处理的TN3K测试集与固定划分，按数据集使用病人、序列或图片粒度。本组head-only保留平均Positive Dice，标准误存于证据。两训练条件均用AdamW、最多10,000步，只在验证集早停；分割解码器随底模输出结构选择。每条成绩按表11注明预训练数据重叠，包括未知及不可审计情况。F1与去除背景的Dice保留原生指标。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.28610v1","published_at":"2026-09-23","status":"paper_baselines"},"model_count":16,"result_count":16,"results":[[12734,0.746,370,12805,12829],[12737,0.744,370,12805,12829],[12738,0.667,370,12806,12829],[12740,0.693,370,12806,12829],[12741,0.69,370,12806,12829],[12742,0.688,370,12806,12829],[12743,0.585,370,12807,12829],[12745,0.731,370,12805,12829],[12747,0.792,370,12808,12829],[12748,0.658,370,12807,12829],[12749,0.688,370,12805,12829],[12750,0.691,370,12808,12829],[12751,0.79,370,12805,12829],[12752,0.805,370,12805,12829],[12753,0.712,370,12807,12829],[12754,0.708,370,12808,12829]]},{"id":"ultrabench2-segmentation-aul-liver-head-only-2026-paper","name":"UltraBench 2 · AUL Liver segmentation head-only (2026 paper)","group":"医疗","metric":"Positive Dice","unit":"score","score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"public_history":true,"protocol":{"en":"Sixteen ultrasound-specialist and general vision backbones use the same prepared AUL Liver test set and fixed dataset-dependent patient/sequence/image splits. This head-only cohort reports mean Positive Dice, with source standard errors archived. Both training regimes use AdamW, up to 10,000 steps and validation-only early stopping; segmentation decoders follow backbone output structure. Pretraining overlap varies and is stated for every result using Table 11, including unknown/unauditable data. These are observed transfer results, not a contamination-free ranking. Macro F1 and foreground-only Dice remain native metrics.","zh":"十六个超声专用及通用视觉底模，采用相同预处理的AUL Liver测试集与固定划分，按数据集使用病人、序列或图片粒度。本组head-only保留平均Positive Dice，标准误存于证据。两训练条件均用AdamW、最多10,000步，只在验证集早停；分割解码器随底模输出结构选择。每条成绩按表11注明预训练数据重叠，包括未知及不可审计情况。F1与去除背景的Dice保留原生指标。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.28610v1","published_at":"2026-09-23","status":"paper_baselines"},"model_count":16,"result_count":16,"results":[[12734,0.88,370,12784,12829],[12737,0.869,370,12784,12829],[12738,0.863,370,12785,12829],[12740,0.876,370,12785,12829],[12741,0.87,370,12785,12829],[12742,0.872,370,12785,12829],[12743,0.845,370,12786,12829],[12745,0.902,370,12787,12829],[12747,0.86,370,12784,12829],[12748,0.845,370,12786,12829],[12749,0.877,370,12784,12829],[12750,0.871,370,12787,12829],[12751,0.903,370,12784,12829],[12752,0.906,370,12784,12829],[12753,0.872,370,12786,12829],[12754,0.883,370,12787,12829]]},{"id":"ultrabench2-segmentation-camus-head-only-2026-paper","name":"UltraBench 2 · CAMUS segmentation head-only (2026 paper)","group":"医疗","metric":"Positive Dice","unit":"score","score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"public_history":true,"protocol":{"en":"Sixteen ultrasound-specialist and general vision backbones use the same prepared CAMUS test set and fixed dataset-dependent patient/sequence/image splits. This head-only cohort reports mean Positive Dice, with source standard errors archived. Both training regimes use AdamW, up to 10,000 steps and validation-only early stopping; segmentation decoders follow backbone output structure. Pretraining overlap varies and is stated for every result using Table 11, including unknown/unauditable data. These are observed transfer results, not a contamination-free ranking. Macro F1 and foreground-only Dice remain native metrics.","zh":"十六个超声专用及通用视觉底模，采用相同预处理的CAMUS测试集与固定划分，按数据集使用病人、序列或图片粒度。本组head-only保留平均Positive Dice，标准误存于证据。两训练条件均用AdamW、最多10,000步，只在验证集早停；分割解码器随底模输出结构选择。每条成绩按表11注明预训练数据重叠，包括未知及不可审计情况。F1与去除背景的Dice保留原生指标。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.28610v1","published_at":"2026-09-23","status":"paper_baselines"},"model_count":16,"result_count":16,"results":[[12734,0.861,370,12809,12829],[12737,0.873,370,12809,12829],[12738,0.85,370,12810,12829],[12740,0.857,370,12810,12829],[12741,0.868,370,12810,12829],[12742,0.871,370,12810,12829],[12743,0.86,370,12811,12829],[12745,0.896,370,12812,12829],[12747,0.905,370,12812,12829],[12748,0.863,370,12811,12829],[12749,0.88,370,12812,12829],[12750,0.886,370,12812,12829],[12751,0.882,370,12812,12829],[12752,0.885,370,12812,12829],[12753,0.874,370,12811,12829],[12754,0.892,370,12812,12829]]},{"id":"ultrabench2-segmentation-ok-capsule-head-only-2026-paper","name":"UltraBench 2 · Open Kidney capsule segmentation head-only (2026 paper)","group":"医疗","metric":"Positive Dice","unit":"score","score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"public_history":true,"protocol":{"en":"Sixteen ultrasound-specialist and general vision backbones use the same prepared Open Kidney capsule test set and fixed dataset-dependent patient/sequence/image splits. This head-only cohort reports mean Positive Dice, with source standard errors archived. Both training regimes use AdamW, up to 10,000 steps and validation-only early stopping; segmentation decoders follow backbone output structure. Pretraining overlap varies and is stated for every result using Table 11, including unknown/unauditable data. These are observed transfer results, not a contamination-free ranking. Macro F1 and foreground-only Dice remain native metrics.","zh":"十六个超声专用及通用视觉底模，采用相同预处理的Open Kidney capsule测试集与固定划分，按数据集使用病人、序列或图片粒度。本组head-only保留平均Positive Dice，标准误存于证据。两训练条件均用AdamW、最多10,000步，只在验证集早停；分割解码器随底模输出结构选择。每条成绩按表11注明预训练数据重叠，包括未知及不可审计情况。F1与去除背景的Dice保留原生指标。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.28610v1","published_at":"2026-09-23","status":"paper_baselines"},"model_count":16,"result_count":16,"results":[[12734,0.764,370,12830,12829],[12737,0.766,370,12830,12829],[12738,0.761,370,12831,12829],[12740,0.766,370,12831,12829],[12741,0.769,370,12831,12829],[12742,0.777,370,12831,12829],[12743,0.722,370,12832,12829],[12745,0.793,370,12830,12829],[12747,0.763,370,12830,12829],[12748,0.754,370,12832,12829],[12749,0.755,370,12830,12829],[12750,0.779,370,12833,12829],[12751,0.811,370,12830,12829],[12752,0.804,370,12830,12829],[12753,0.775,370,12832,12829],[12754,0.784,370,12833,12829]]},{"id":"ultrabench2-segmentation-ok-regions-head-only-2026-paper","name":"UltraBench 2 · Open Kidney regions segmentation head-only (2026 paper)","group":"医疗","metric":"Positive Dice","unit":"score","score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"public_history":true,"protocol":{"en":"Sixteen ultrasound-specialist and general vision backbones use the same prepared Open Kidney regions test set and fixed dataset-dependent patient/sequence/image splits. This head-only cohort reports mean Positive Dice, with source standard errors archived. Both training regimes use AdamW, up to 10,000 steps and validation-only early stopping; segmentation decoders follow backbone output structure. Pretraining overlap varies and is stated for every result using Table 11, including unknown/unauditable data. These are observed transfer results, not a contamination-free ranking. Macro F1 and foreground-only Dice remain native metrics.","zh":"十六个超声专用及通用视觉底模，采用相同预处理的Open Kidney regions测试集与固定划分，按数据集使用病人、序列或图片粒度。本组head-only保留平均Positive Dice，标准误存于证据。两训练条件均用AdamW、最多10,000步，只在验证集早停；分割解码器随底模输出结构选择。每条成绩按表11注明预训练数据重叠，包括未知及不可审计情况。F1与去除背景的Dice保留原生指标。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.28610v1","published_at":"2026-09-23","status":"paper_baselines"},"model_count":16,"result_count":16,"results":[[12734,0.378,370,12830,12829],[12737,0.391,370,12830,12829],[12738,0.418,370,12831,12829],[12740,0.444,370,12831,12829],[12741,0.389,370,12831,12829],[12742,0.408,370,12831,12829],[12743,0.37,370,12832,12829],[12745,0.439,370,12830,12829],[12747,0.431,370,12830,12829],[12748,0.394,370,12832,12829],[12749,0.393,370,12830,12829],[12750,0.433,370,12833,12829],[12751,0.441,370,12830,12829],[12752,0.446,370,12830,12829],[12753,0.41,370,12832,12829],[12754,0.402,370,12833,12829]]},{"id":"ultrabench2-segmentation-psfhs-head-only-2026-paper","name":"UltraBench 2 · PSFHS segmentation head-only (2026 paper)","group":"医疗","metric":"Positive Dice","unit":"score","score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"public_history":true,"protocol":{"en":"Sixteen ultrasound-specialist and general vision backbones use the same prepared PSFHS test set and fixed dataset-dependent patient/sequence/image splits. This head-only cohort reports mean Positive Dice, with source standard errors archived. Both training regimes use AdamW, up to 10,000 steps and validation-only early stopping; segmentation decoders follow backbone output structure. Pretraining overlap varies and is stated for every result using Table 11, including unknown/unauditable data. These are observed transfer results, not a contamination-free ranking. Macro F1 and foreground-only Dice remain native metrics.","zh":"十六个超声专用及通用视觉底模，采用相同预处理的PSFHS测试集与固定划分，按数据集使用病人、序列或图片粒度。本组head-only保留平均Positive Dice，标准误存于证据。两训练条件均用AdamW、最多10,000步，只在验证集早停；分割解码器随底模输出结构选择。每条成绩按表11注明预训练数据重叠，包括未知及不可审计情况。F1与去除背景的Dice保留原生指标。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.28610v1","published_at":"2026-09-23","status":"paper_baselines"},"model_count":16,"result_count":16,"results":[[12734,0.917,370,12834,12829],[12737,0.922,370,12834,12829],[12738,0.902,370,12835,12829],[12740,0.909,370,12835,12829],[12741,0.909,370,12835,12829],[12742,0.902,370,12835,12829],[12743,0.915,370,12836,12829],[12745,0.933,370,12834,12829],[12747,0.91,370,12834,12829],[12748,0.916,370,12836,12829],[12749,0.925,370,12834,12829],[12750,0.926,370,12837,12829],[12751,0.928,370,12834,12829],[12752,0.93,370,12834,12829],[12753,0.925,370,12836,12829],[12754,0.93,370,12837,12829]]},{"id":"ultrabench2-segmentation-stmus-head-only-2026-paper","name":"UltraBench 2 · STMUS segmentation head-only (2026 paper)","group":"医疗","metric":"Positive Dice","unit":"score","score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"public_history":true,"protocol":{"en":"Sixteen ultrasound-specialist and general vision backbones use the same prepared STMUS test set and fixed dataset-dependent patient/sequence/image splits. This head-only cohort reports mean Positive Dice, with source standard errors archived. Both training regimes use AdamW, up to 10,000 steps and validation-only early stopping; segmentation decoders follow backbone output structure. Pretraining overlap varies and is stated for every result using Table 11, including unknown/unauditable data. These are observed transfer results, not a contamination-free ranking. Macro F1 and foreground-only Dice remain native metrics.","zh":"十六个超声专用及通用视觉底模，采用相同预处理的STMUS测试集与固定划分，按数据集使用病人、序列或图片粒度。本组head-only保留平均Positive Dice，标准误存于证据。两训练条件均用AdamW、最多10,000步，只在验证集早停；分割解码器随底模输出结构选择。每条成绩按表11注明预训练数据重叠，包括未知及不可审计情况。F1与去除背景的Dice保留原生指标。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.28610v1","published_at":"2026-09-23","status":"paper_baselines"},"model_count":16,"result_count":16,"results":[[12734,0.855,370,12838,12829],[12737,0.857,370,12838,12829],[12738,0.844,370,12839,12829],[12740,0.849,370,12839,12829],[12741,0.85,370,12839,12829],[12742,0.852,370,12839,12829],[12743,0.842,370,12840,12829],[12745,0.87,370,12841,12829],[12747,0.837,370,12838,12829],[12748,0.851,370,12840,12829],[12749,0.865,370,12838,12829],[12750,0.863,370,12841,12829],[12751,0.872,370,12838,12829],[12752,0.871,370,12838,12829],[12753,0.862,370,12840,12829],[12754,0.871,370,12841,12829]]},{"id":"panda-cvl-correction-2026-paper","name":"Panda-CVL · correction (2026 paper)","group":"视觉与视频","metric":"Exact first correction on gold not-good cases","unit":"%","public_history":true,"protocol":{"en":"Nine reasoning-model backbones and one instruct configuration of Qwen3.5 answer the same predominantly Chinese token-correction test set of 652 annotation sessions. GoodAcc uses gold already-good cases; correction requires both first-error location and replacement to match on the fixed gold not-good subset. The source F1 is the harmonic mean of these two conditional rates, not ordinary classification F1. Output format validity is a separate diagnostic, not a model-dependent denominator filter. Reasoning/instruct configurations remain explicit; a single human reference can admit alternative valid edits, with multi-reference agreement audited separately.","zh":"九个推理底模及Qwen3.5的一个Instruct配置，回答相同652条标注会话的中文为主词元纠错测试。GoodAcc针对金标准已合格回答，纠错率要求在固定金标准不合格子集上同时匹配首次错误位置与替换文本。原文F1是这两个条件比例的调和均值，不是通常的分类F1。格式有效率单独诊断，不作为各模型不同的分母过滤。推理与Instruct保留为配置；单人工参考可能存在其他有效修改，多参考一致性另有审计。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.24983v1","published_at":"2026-09-21","status":"paper_baselines"},"model_count":9,"result_count":10,"results":[[12842,0.13699999999999998,370,12843,12844],[3196,0.1018,370,12845,12844],[4048,0.1309,370,12845,12844],[5676,0.1583,370,12845,12844],[3187,0.1059,370,12843,12844],[5740,0.0332,370,12843,12844],[12846,0.0977,370,12843,12844],[3764,0.057,370,12843,12844],[2576,0.1099,370,12843,12844],[2576,0.008100000000000001,370,12847,12844]]},{"id":"panda-cvl-f1-2026-paper","name":"Panda-CVL · f1 (2026 paper)","group":"视觉与视频","metric":"Harmonic mean of GoodAcc and correction","unit":"score","public_history":true,"protocol":{"en":"Nine reasoning-model backbones and one instruct configuration of Qwen3.5 answer the same predominantly Chinese token-correction test set of 652 annotation sessions. GoodAcc uses gold already-good cases; correction requires both first-error location and replacement to match on the fixed gold not-good subset. The source F1 is the harmonic mean of these two conditional rates, not ordinary classification F1. Output format validity is a separate diagnostic, not a model-dependent denominator filter. Reasoning/instruct configurations remain explicit; a single human reference can admit alternative valid edits, with multi-reference agreement audited separately.","zh":"九个推理底模及Qwen3.5的一个Instruct配置，回答相同652条标注会话的中文为主词元纠错测试。GoodAcc针对金标准已合格回答，纠错率要求在固定金标准不合格子集上同时匹配首次错误位置与替换文本。原文F1是这两个条件比例的调和均值，不是通常的分类F1。格式有效率单独诊断，不作为各模型不同的分母过滤。推理与Instruct保留为配置；单人工参考可能存在其他有效修改，多参考一致性另有审计。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.24983v1","published_at":"2026-09-21","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":9,"result_count":10,"results":[[12842,11.33,370,12843,12844],[3196,17.09,370,12845,12844],[4048,14.63,370,12845,12844],[5676,16.57,370,12845,12844],[3187,15.11,370,12843,12844],[5740,6.13,370,12843,12844],[12846,14.99,370,12843,12844],[3764,9.36,370,12843,12844],[2576,14.28,370,12843,12844],[2576,1.38,370,12847,12844]]},{"id":"cq4oe-cq2term-awo-direct-2026-paper","name":"CQ4OE · cq2term awo direct (2026 paper)","group":"知识与推理","metric":"Combined term F1","unit":"score","score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"F1 / 1","zh":"F1 / 1"},"public_history":true,"protocol":{"en":"Nine LLM checkpoints use the same expert-annotated AWO ontology domain. CQ2Term and CQ2Onto use separate gold standards (99/118 CQs over all six domains), each independently triple-reviewed with 2-of-3 consensus. Combined term F1 includes classes and properties under the same top-three lexical/semantic alignment and 0.6 threshold. The direct generation condition is separate. Author outputs are pinned to repository commit ded082853bf4; semantic alignment uses the source embeddinggemma configuration. Scores remain native F1, without treating term extraction or ontology structure as whole-question accuracy.","zh":"九个LLM检查点处理相同AWO本体领域。CQ2Term与CQ2Onto使用不同金标准，六领域共99/118个能力问题，均由三人独立审查、至少两人同意。联合词项F1涵盖类与属性，使用相同前三词法/语义对齐和0.6阈值。本组direct生成条件单列。作者输出固定在仓库提交ded082853bf4，语义对齐保留原始embeddinggemma配置。F1保留原生含义，不当作整题准确率。"},"source":{"name":"Paper authors — pinned evaluation artifacts","url":"https://arxiv.org/abs/2609.26029v1","published_at":"2026-09-22","status":"paper_baselines"},"model_count":9,"result_count":9,"results":[[5796,0.888888888888889,370,12848,12849],[5795,1.0,370,12848,12850],[4425,0.7999999999999999,370,12848,12851],[12852,0.9411764705882353,370,12848,12853],[5822,1.0,370,12848,12854],[3763,0.823529411764706,370,12848,12855],[3764,0.875,370,12848,12856],[6377,0.875,370,12848,12857],[3189,0.9411764705882353,370,12848,12858]]},{"id":"cq4oe-cq2term-odrl-direct-2026-paper","name":"CQ4OE · cq2term odrl direct (2026 paper)","group":"知识与推理","metric":"Combined term F1","unit":"score","score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"F1 / 1","zh":"F1 / 1"},"public_history":true,"protocol":{"en":"Nine LLM checkpoints use the same expert-annotated ODRL ontology domain. CQ2Term and CQ2Onto use separate gold standards (99/118 CQs over all six domains), each independently triple-reviewed with 2-of-3 consensus. Combined term F1 includes classes and properties under the same top-three lexical/semantic alignment and 0.6 threshold. The direct generation condition is separate. Author outputs are pinned to repository commit ded082853bf4; semantic alignment uses the source embeddinggemma configuration. Scores remain native F1, without treating term extraction or ontology structure as whole-question accuracy.","zh":"九个LLM检查点处理相同ODRL本体领域。CQ2Term与CQ2Onto使用不同金标准，六领域共99/118个能力问题，均由三人独立审查、至少两人同意。联合词项F1涵盖类与属性，使用相同前三词法/语义对齐和0.6阈值。本组direct生成条件单列。作者输出固定在仓库提交ded082853bf4，语义对齐保留原始embeddinggemma配置。F1保留原生含义，不当作整题准确率。"},"source":{"name":"Paper authors — pinned evaluation artifacts","url":"https://arxiv.org/abs/2609.26029v1","published_at":"2026-09-22","status":"paper_baselines"},"model_count":9,"result_count":9,"results":[[5796,0.43181818181818177,370,12859,12860],[5795,0.48888888888888893,370,12859,12861],[4425,0.43181818181818177,370,12859,12862],[12852,0.4666666666666667,370,12859,12863],[5822,0.5476190476190476,370,12859,12864],[3763,0.5,370,12859,12865],[3764,0.5783132530120482,370,12859,12866],[6377,0.5116279069767442,370,12859,12867],[3189,0.4597701149425287,370,12859,12868]]},{"id":"cq4oe-cq2term-swo-direct-2026-paper","name":"CQ4OE · cq2term swo direct (2026 paper)","group":"知识与推理","metric":"Combined term F1","unit":"score","score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"F1 / 1","zh":"F1 / 1"},"public_history":true,"protocol":{"en":"Nine LLM checkpoints use the same expert-annotated SWO ontology domain. CQ2Term and CQ2Onto use separate gold standards (99/118 CQs over all six domains), each independently triple-reviewed with 2-of-3 consensus. Combined term F1 includes classes and properties under the same top-three lexical/semantic alignment and 0.6 threshold. The direct generation condition is separate. Author outputs are pinned to repository commit ded082853bf4; semantic alignment uses the source embeddinggemma configuration. Scores remain native F1, without treating term extraction or ontology structure as whole-question accuracy.","zh":"九个LLM检查点处理相同SWO本体领域。CQ2Term与CQ2Onto使用不同金标准，六领域共99/118个能力问题，均由三人独立审查、至少两人同意。联合词项F1涵盖类与属性，使用相同前三词法/语义对齐和0.6阈值。本组direct生成条件单列。作者输出固定在仓库提交ded082853bf4，语义对齐保留原始embeddinggemma配置。F1保留原生含义，不当作整题准确率。"},"source":{"name":"Paper authors — pinned evaluation artifacts","url":"https://arxiv.org/abs/2609.26029v1","published_at":"2026-09-22","status":"paper_baselines"},"model_count":9,"result_count":9,"results":[[5796,0.5252525252525253,370,12869,12870],[5795,0.5858585858585859,370,12869,12871],[4425,0.5283018867924528,370,12869,12872],[12852,0.5242718446601942,370,12869,12873],[5822,0.5192307692307692,370,12869,12874],[3763,0.5360824742268041,370,12869,12875],[3764,0.4948453608247423,370,12869,12876],[6377,0.5154639175257731,370,12869,12877],[3189,0.5148514851485149,370,12869,12878]]},{"id":"cq4oe-cq2term-vgo-direct-2026-paper","name":"CQ4OE · cq2term vgo direct (2026 paper)","group":"知识与推理","metric":"Combined term F1","unit":"score","score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"F1 / 1","zh":"F1 / 1"},"public_history":true,"protocol":{"en":"Nine LLM checkpoints use the same expert-annotated VGO ontology domain. CQ2Term and CQ2Onto use separate gold standards (99/118 CQs over all six domains), each independently triple-reviewed with 2-of-3 consensus. Combined term F1 includes classes and properties under the same top-three lexical/semantic alignment and 0.6 threshold. The direct generation condition is separate. Author outputs are pinned to repository commit ded082853bf4; semantic alignment uses the source embeddinggemma configuration. Scores remain native F1, without treating term extraction or ontology structure as whole-question accuracy.","zh":"九个LLM检查点处理相同VGO本体领域。CQ2Term与CQ2Onto使用不同金标准，六领域共99/118个能力问题，均由三人独立审查、至少两人同意。联合词项F1涵盖类与属性，使用相同前三词法/语义对齐和0.6阈值。本组direct生成条件单列。作者输出固定在仓库提交ded082853bf4，语义对齐保留原始embeddinggemma配置。F1保留原生含义，不当作整题准确率。"},"source":{"name":"Paper authors — pinned evaluation artifacts","url":"https://arxiv.org/abs/2609.26029v1","published_at":"2026-09-22","status":"paper_baselines"},"model_count":9,"result_count":9,"results":[[5796,0.6415094339622642,370,12879,12880],[5795,0.6923076923076923,370,12879,12881],[4425,0.6071428571428571,370,12879,12882],[12852,0.6037735849056604,370,12879,12883],[5822,0.6792452830188678,370,12879,12884],[3763,0.5925925925925926,370,12879,12885],[3764,0.6296296296296297,370,12879,12886],[6377,0.5714285714285714,370,12879,12887],[3189,0.6538461538461537,370,12879,12888]]},{"id":"cq4oe-cq2term-water-direct-2026-paper","name":"CQ4OE · cq2term water direct (2026 paper)","group":"知识与推理","metric":"Combined term F1","unit":"score","score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"F1 / 1","zh":"F1 / 1"},"public_history":true,"protocol":{"en":"Nine LLM checkpoints use the same expert-annotated WATER ontology domain. CQ2Term and CQ2Onto use separate gold standards (99/118 CQs over all six domains), each independently triple-reviewed with 2-of-3 consensus. Combined term F1 includes classes and properties under the same top-three lexical/semantic alignment and 0.6 threshold. The direct generation condition is separate. Author outputs are pinned to repository commit ded082853bf4; semantic alignment uses the source embeddinggemma configuration. Scores remain native F1, without treating term extraction or ontology structure as whole-question accuracy.","zh":"九个LLM检查点处理相同WATER本体领域。CQ2Term与CQ2Onto使用不同金标准，六领域共99/118个能力问题，均由三人独立审查、至少两人同意。联合词项F1涵盖类与属性，使用相同前三词法/语义对齐和0.6阈值。本组direct生成条件单列。作者输出固定在仓库提交ded082853bf4，语义对齐保留原始embeddinggemma配置。F1保留原生含义，不当作整题准确率。"},"source":{"name":"Paper authors — pinned evaluation artifacts","url":"https://arxiv.org/abs/2609.26029v1","published_at":"2026-09-22","status":"paper_baselines"},"model_count":9,"result_count":9,"results":[[5796,0.7179487179487181,370,12889,12890],[5795,0.6582278481012659,370,12889,12891],[4425,0.6666666666666666,370,12889,12892],[12852,0.7631578947368421,370,12889,12893],[5822,0.6666666666666666,370,12889,12894],[3763,0.6913580246913581,370,12889,12895],[3764,0.75,370,12889,12896],[6377,0.6923076923076923,370,12889,12897],[3189,0.7466666666666666,370,12889,12898]]},{"id":"cq4oe-cq2term-wine-direct-2026-paper","name":"CQ4OE · cq2term wine direct (2026 paper)","group":"知识与推理","metric":"Combined term F1","unit":"score","score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"F1 / 1","zh":"F1 / 1"},"public_history":true,"protocol":{"en":"Nine LLM checkpoints use the same expert-annotated WINE ontology domain. CQ2Term and CQ2Onto use separate gold standards (99/118 CQs over all six domains), each independently triple-reviewed with 2-of-3 consensus. Combined term F1 includes classes and properties under the same top-three lexical/semantic alignment and 0.6 threshold. The direct generation condition is separate. Author outputs are pinned to repository commit ded082853bf4; semantic alignment uses the source embeddinggemma configuration. Scores remain native F1, without treating term extraction or ontology structure as whole-question accuracy.","zh":"九个LLM检查点处理相同WINE本体领域。CQ2Term与CQ2Onto使用不同金标准，六领域共99/118个能力问题，均由三人独立审查、至少两人同意。联合词项F1涵盖类与属性，使用相同前三词法/语义对齐和0.6阈值。本组direct生成条件单列。作者输出固定在仓库提交ded082853bf4，语义对齐保留原始embeddinggemma配置。F1保留原生含义，不当作整题准确率。"},"source":{"name":"Paper authors — pinned evaluation artifacts","url":"https://arxiv.org/abs/2609.26029v1","published_at":"2026-09-22","status":"paper_baselines"},"model_count":9,"result_count":9,"results":[[5796,0.6451612903225806,370,12899,12900],[5795,0.5625,370,12899,12901],[4425,0.5128205128205128,370,12899,12902],[12852,0.375,370,12899,12903],[5822,0.5454545454545455,370,12899,12904],[3763,0.45714285714285713,370,12899,12905],[3764,0.38888888888888884,370,12899,12906],[6377,0.42424242424242425,370,12899,12907],[3189,0.41025641025641024,370,12899,12908]]},{"id":"cq4oe-cq2onto-awo-normal-2026-paper","name":"CQ4OE · cq2onto awo normal (2026 paper)","group":"知识与推理","metric":"Global axiom F1","unit":"score","score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"F1 / 1","zh":"F1 / 1"},"public_history":true,"protocol":{"en":"Nine LLM checkpoints use the same expert-annotated AWO ontology domain. CQ2Term and CQ2Onto use separate gold standards (99/118 CQs over all six domains), each independently triple-reviewed with 2-of-3 consensus. Global axiom F1 uses layer4/grand, counting unmatched gold/predicted axioms as FN/FP; it is not the alignment-conditional layer3 score. The normal generation condition is separate. Author outputs are pinned to repository commit ded082853bf4; semantic alignment uses the source embeddinggemma configuration. Scores remain native F1, without treating term extraction or ontology structure as whole-question accuracy.","zh":"九个LLM检查点处理相同AWO本体领域。CQ2Term与CQ2Onto使用不同金标准，六领域共99/118个能力问题，均由三人独立审查、至少两人同意。全局公理F1采用layer4/grand，将未对齐的金标准与预测公理计为FN/FP，不采用只评价已对齐项的layer3条件分。本组normal生成条件单列。作者输出固定在仓库提交ded082853bf4，语义对齐保留原始embeddinggemma配置。F1保留原生含义，不当作整题准确率。"},"source":{"name":"Paper authors — pinned evaluation artifacts","url":"https://arxiv.org/abs/2609.26029v1","published_at":"2026-09-22","status":"paper_baselines"},"model_count":9,"result_count":9,"results":[[5796,0.18604651162790697,370,12909,12910],[5795,0.09302325581395349,370,12909,12911],[4425,0.19999999999999998,370,12909,12912],[12852,0.13043478260869565,370,12909,12913],[5822,0.24,370,12909,12914],[3763,0.14285714285714282,370,12909,12915],[3764,0.1333333333333333,370,12909,12916],[6377,0.1568627450980392,370,12909,12917],[3189,0.14285714285714282,370,12909,12918]]},{"id":"cq4oe-cq2onto-odrl-normal-2026-paper","name":"CQ4OE · cq2onto odrl normal (2026 paper)","group":"知识与推理","metric":"Global axiom F1","unit":"score","score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"F1 / 1","zh":"F1 / 1"},"public_history":true,"protocol":{"en":"Nine LLM checkpoints use the same expert-annotated ODRL ontology domain. CQ2Term and CQ2Onto use separate gold standards (99/118 CQs over all six domains), each independently triple-reviewed with 2-of-3 consensus. Global axiom F1 uses layer4/grand, counting unmatched gold/predicted axioms as FN/FP; it is not the alignment-conditional layer3 score. The normal generation condition is separate. Author outputs are pinned to repository commit ded082853bf4; semantic alignment uses the source embeddinggemma configuration. Scores remain native F1, without treating term extraction or ontology structure as whole-question accuracy.","zh":"九个LLM检查点处理相同ODRL本体领域。CQ2Term与CQ2Onto使用不同金标准，六领域共99/118个能力问题，均由三人独立审查、至少两人同意。全局公理F1采用layer4/grand，将未对齐的金标准与预测公理计为FN/FP，不采用只评价已对齐项的layer3条件分。本组normal生成条件单列。作者输出固定在仓库提交ded082853bf4，语义对齐保留原始embeddinggemma配置。F1保留原生含义，不当作整题准确率。"},"source":{"name":"Paper authors — pinned evaluation artifacts","url":"https://arxiv.org/abs/2609.26029v1","published_at":"2026-09-22","status":"paper_baselines"},"model_count":9,"result_count":9,"results":[[5796,0.3125,370,12919,12920],[5795,0.24427480916030536,370,12919,12921],[4425,0.14173228346456695,370,12919,12922],[12852,0.0792079207920792,370,12919,12923],[5822,0.17142857142857143,370,12919,12924],[3763,0.0,370,12919,12925],[3764,0.13513513513513511,370,12919,12926],[6377,0.18705035971223022,370,12919,12927],[3189,0.14864864864864863,370,12919,12928]]},{"id":"cq4oe-cq2onto-swo-normal-2026-paper","name":"CQ4OE · cq2onto swo normal (2026 paper)","group":"知识与推理","metric":"Global axiom F1","unit":"score","score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"F1 / 1","zh":"F1 / 1"},"public_history":true,"protocol":{"en":"Nine LLM checkpoints use the same expert-annotated SWO ontology domain. CQ2Term and CQ2Onto use separate gold standards (99/118 CQs over all six domains), each independently triple-reviewed with 2-of-3 consensus. Global axiom F1 uses layer4/grand, counting unmatched gold/predicted axioms as FN/FP; it is not the alignment-conditional layer3 score. The normal generation condition is separate. Author outputs are pinned to repository commit ded082853bf4; semantic alignment uses the source embeddinggemma configuration. Scores remain native F1, without treating term extraction or ontology structure as whole-question accuracy.","zh":"九个LLM检查点处理相同SWO本体领域。CQ2Term与CQ2Onto使用不同金标准，六领域共99/118个能力问题，均由三人独立审查、至少两人同意。全局公理F1采用layer4/grand，将未对齐的金标准与预测公理计为FN/FP，不采用只评价已对齐项的layer3条件分。本组normal生成条件单列。作者输出固定在仓库提交ded082853bf4，语义对齐保留原始embeddinggemma配置。F1保留原生含义，不当作整题准确率。"},"source":{"name":"Paper authors — pinned evaluation artifacts","url":"https://arxiv.org/abs/2609.26029v1","published_at":"2026-09-22","status":"paper_baselines"},"model_count":9,"result_count":9,"results":[[5796,0.23232323232323232,370,12929,12930],[5795,0.0,370,12929,12931],[4425,0.15662650602409636,370,12929,12932],[12852,0.13114754098360654,370,12929,12933],[5822,0.17518248175182483,370,12929,12934],[3763,0.1990521327014218,370,12929,12935],[3764,0.25380710659898476,370,12929,12936],[6377,0.23463687150837986,370,12929,12937],[3189,0.1982758620689655,370,12929,12938]]},{"id":"cq4oe-cq2onto-vgo-normal-2026-paper","name":"CQ4OE · cq2onto vgo normal (2026 paper)","group":"知识与推理","metric":"Global axiom F1","unit":"score","score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"F1 / 1","zh":"F1 / 1"},"public_history":true,"protocol":{"en":"Nine LLM checkpoints use the same expert-annotated VGO ontology domain. CQ2Term and CQ2Onto use separate gold standards (99/118 CQs over all six domains), each independently triple-reviewed with 2-of-3 consensus. Global axiom F1 uses layer4/grand, counting unmatched gold/predicted axioms as FN/FP; it is not the alignment-conditional layer3 score. The normal generation condition is separate. Author outputs are pinned to repository commit ded082853bf4; semantic alignment uses the source embeddinggemma configuration. Scores remain native F1, without treating term extraction or ontology structure as whole-question accuracy.","zh":"九个LLM检查点处理相同VGO本体领域。CQ2Term与CQ2Onto使用不同金标准，六领域共99/118个能力问题，均由三人独立审查、至少两人同意。全局公理F1采用layer4/grand，将未对齐的金标准与预测公理计为FN/FP，不采用只评价已对齐项的layer3条件分。本组normal生成条件单列。作者输出固定在仓库提交ded082853bf4，语义对齐保留原始embeddinggemma配置。F1保留原生含义，不当作整题准确率。"},"source":{"name":"Paper authors — pinned evaluation artifacts","url":"https://arxiv.org/abs/2609.26029v1","published_at":"2026-09-22","status":"paper_baselines"},"model_count":9,"result_count":9,"results":[[5796,0.15286624203821655,370,12939,12940],[5795,0.18556701030927839,370,12939,12941],[4425,0.15492957746478875,370,12939,12942],[12852,0.16176470588235295,370,12939,12943],[5822,0.15602836879432624,370,12939,12944],[3763,0.14201183431952663,370,12939,12945],[3764,0.17341040462427745,370,12939,12946],[6377,0.155688622754491,370,12939,12947],[3189,0.16867469879518074,370,12939,12948]]},{"id":"cq4oe-cq2onto-water-normal-2026-paper","name":"CQ4OE · cq2onto water normal (2026 paper)","group":"知识与推理","metric":"Global axiom F1","unit":"score","score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"F1 / 1","zh":"F1 / 1"},"public_history":true,"protocol":{"en":"Nine LLM checkpoints use the same expert-annotated WATER ontology domain. CQ2Term and CQ2Onto use separate gold standards (99/118 CQs over all six domains), each independently triple-reviewed with 2-of-3 consensus. Global axiom F1 uses layer4/grand, counting unmatched gold/predicted axioms as FN/FP; it is not the alignment-conditional layer3 score. The normal generation condition is separate. Author outputs are pinned to repository commit ded082853bf4; semantic alignment uses the source embeddinggemma configuration. Scores remain native F1, without treating term extraction or ontology structure as whole-question accuracy.","zh":"九个LLM检查点处理相同WATER本体领域。CQ2Term与CQ2Onto使用不同金标准，六领域共99/118个能力问题，均由三人独立审查、至少两人同意。全局公理F1采用layer4/grand，将未对齐的金标准与预测公理计为FN/FP，不采用只评价已对齐项的layer3条件分。本组normal生成条件单列。作者输出固定在仓库提交ded082853bf4，语义对齐保留原始embeddinggemma配置。F1保留原生含义，不当作整题准确率。"},"source":{"name":"Paper authors — pinned evaluation artifacts","url":"https://arxiv.org/abs/2609.26029v1","published_at":"2026-09-22","status":"paper_baselines"},"model_count":9,"result_count":9,"results":[[5796,0.0606060606060606,370,12949,12950],[5795,0.0,370,12949,12951],[4425,0.12162162162162161,370,12949,12952],[12852,0.04301075268817204,370,12949,12953],[5822,0.03773584905660377,370,12949,12954],[3763,0.07692307692307693,370,12949,12955],[3764,0.0625,370,12949,12956],[6377,0.0634920634920635,370,12949,12957],[3189,0.06993006993006994,370,12949,12958]]},{"id":"cq4oe-cq2onto-wine-normal-2026-paper","name":"CQ4OE · cq2onto wine normal (2026 paper)","group":"知识与推理","metric":"Global axiom F1","unit":"score","score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"F1 / 1","zh":"F1 / 1"},"public_history":true,"protocol":{"en":"Nine LLM checkpoints use the same expert-annotated WINE ontology domain. CQ2Term and CQ2Onto use separate gold standards (99/118 CQs over all six domains), each independently triple-reviewed with 2-of-3 consensus. Global axiom F1 uses layer4/grand, counting unmatched gold/predicted axioms as FN/FP; it is not the alignment-conditional layer3 score. The normal generation condition is separate. Author outputs are pinned to repository commit ded082853bf4; semantic alignment uses the source embeddinggemma configuration. Scores remain native F1, without treating term extraction or ontology structure as whole-question accuracy.","zh":"九个LLM检查点处理相同WINE本体领域。CQ2Term与CQ2Onto使用不同金标准，六领域共99/118个能力问题，均由三人独立审查、至少两人同意。全局公理F1采用layer4/grand，将未对齐的金标准与预测公理计为FN/FP，不采用只评价已对齐项的layer3条件分。本组normal生成条件单列。作者输出固定在仓库提交ded082853bf4，语义对齐保留原始embeddinggemma配置。F1保留原生含义，不当作整题准确率。"},"source":{"name":"Paper authors — pinned evaluation artifacts","url":"https://arxiv.org/abs/2609.26029v1","published_at":"2026-09-22","status":"paper_baselines"},"model_count":9,"result_count":9,"results":[[5796,0.3166666666666667,370,12959,12960],[5795,0.326530612244898,370,12959,12961],[4425,0.2692307692307692,370,12959,12962],[12852,0.15555555555555556,370,12959,12963],[5822,0.24,370,12959,12964],[3763,0.31304347826086953,370,12959,12965],[3764,0.3559322033898305,370,12959,12966],[6377,0.3157894736842105,370,12959,12967],[3189,0.19801980198019803,370,12959,12968]]},{"id":"cq4oe-cq2onto-awo-cqbycq-2026-paper","name":"CQ4OE · cq2onto awo cqbycq (2026 paper)","group":"知识与推理","metric":"Global axiom F1","unit":"score","score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"F1 / 1","zh":"F1 / 1"},"public_history":true,"protocol":{"en":"Nine LLM checkpoints use the same expert-annotated AWO ontology domain. CQ2Term and CQ2Onto use separate gold standards (99/118 CQs over all six domains), each independently triple-reviewed with 2-of-3 consensus. Global axiom F1 uses layer4/grand, counting unmatched gold/predicted axioms as FN/FP; it is not the alignment-conditional layer3 score. The cqbycq generation condition is separate. Author outputs are pinned to repository commit ded082853bf4; semantic alignment uses the source embeddinggemma configuration. Scores remain native F1, without treating term extraction or ontology structure as whole-question accuracy.","zh":"九个LLM检查点处理相同AWO本体领域。CQ2Term与CQ2Onto使用不同金标准，六领域共99/118个能力问题，均由三人独立审查、至少两人同意。全局公理F1采用layer4/grand，将未对齐的金标准与预测公理计为FN/FP，不采用只评价已对齐项的layer3条件分。本组cqbycq生成条件单列。作者输出固定在仓库提交ded082853bf4，语义对齐保留原始embeddinggemma配置。F1保留原生含义，不当作整题准确率。"},"source":{"name":"Paper authors — pinned evaluation artifacts","url":"https://arxiv.org/abs/2609.26029v1","published_at":"2026-09-22","status":"paper_baselines"},"model_count":9,"result_count":9,"results":[[5796,0.16,370,12969,12970],[5795,0.16326530612244897,370,12969,12971],[4425,0.16,370,12969,12972],[12852,0.07142857142857142,370,12969,12973],[5822,0.11538461538461538,370,12969,12974],[3763,0.16326530612244897,370,12969,12975],[3764,0.15384615384615383,370,12969,12976],[6377,0.17777777777777778,370,12969,12977],[3189,0.14545454545454545,370,12969,12978]]},{"id":"cq4oe-cq2onto-odrl-cqbycq-2026-paper","name":"CQ4OE · cq2onto odrl cqbycq (2026 paper)","group":"知识与推理","metric":"Global axiom F1","unit":"score","score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"F1 / 1","zh":"F1 / 1"},"public_history":true,"protocol":{"en":"Nine LLM checkpoints use the same expert-annotated ODRL ontology domain. CQ2Term and CQ2Onto use separate gold standards (99/118 CQs over all six domains), each independently triple-reviewed with 2-of-3 consensus. Global axiom F1 uses layer4/grand, counting unmatched gold/predicted axioms as FN/FP; it is not the alignment-conditional layer3 score. The cqbycq generation condition is separate. Author outputs are pinned to repository commit ded082853bf4; semantic alignment uses the source embeddinggemma configuration. Scores remain native F1, without treating term extraction or ontology structure as whole-question accuracy.","zh":"九个LLM检查点处理相同ODRL本体领域。CQ2Term与CQ2Onto使用不同金标准，六领域共99/118个能力问题，均由三人独立审查、至少两人同意。全局公理F1采用layer4/grand，将未对齐的金标准与预测公理计为FN/FP，不采用只评价已对齐项的layer3条件分。本组cqbycq生成条件单列。作者输出固定在仓库提交ded082853bf4，语义对齐保留原始embeddinggemma配置。F1保留原生含义，不当作整题准确率。"},"source":{"name":"Paper authors — pinned evaluation artifacts","url":"https://arxiv.org/abs/2609.26029v1","published_at":"2026-09-22","status":"paper_baselines"},"model_count":9,"result_count":9,"results":[[5796,0.05673758865248227,370,12979,12980],[5795,0.024539877300613494,370,12979,12981],[4425,0.04145077720207254,370,12979,12982],[12852,0.028571428571428574,370,12979,12983],[5822,0.027777777777777776,370,12979,12984],[3763,0.07741935483870968,370,12979,12985],[3764,0.11111111111111112,370,12979,12986],[6377,0.02702702702702703,370,12979,12987],[3189,0.08235294117647059,370,12979,12988]]},{"id":"cq4oe-cq2onto-swo-cqbycq-2026-paper","name":"CQ4OE · cq2onto swo cqbycq (2026 paper)","group":"知识与推理","metric":"Global axiom F1","unit":"score","score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"F1 / 1","zh":"F1 / 1"},"public_history":true,"protocol":{"en":"Nine LLM checkpoints use the same expert-annotated SWO ontology domain. CQ2Term and CQ2Onto use separate gold standards (99/118 CQs over all six domains), each independently triple-reviewed with 2-of-3 consensus. Global axiom F1 uses layer4/grand, counting unmatched gold/predicted axioms as FN/FP; it is not the alignment-conditional layer3 score. The cqbycq generation condition is separate. Author outputs are pinned to repository commit ded082853bf4; semantic alignment uses the source embeddinggemma configuration. Scores remain native F1, without treating term extraction or ontology structure as whole-question accuracy.","zh":"九个LLM检查点处理相同SWO本体领域。CQ2Term与CQ2Onto使用不同金标准，六领域共99/118个能力问题，均由三人独立审查、至少两人同意。全局公理F1采用layer4/grand，将未对齐的金标准与预测公理计为FN/FP，不采用只评价已对齐项的layer3条件分。本组cqbycq生成条件单列。作者输出固定在仓库提交ded082853bf4，语义对齐保留原始embeddinggemma配置。F1保留原生含义，不当作整题准确率。"},"source":{"name":"Paper authors — pinned evaluation artifacts","url":"https://arxiv.org/abs/2609.26029v1","published_at":"2026-09-22","status":"paper_baselines"},"model_count":9,"result_count":9,"results":[[5796,0.24615384615384617,370,12989,12990],[5795,0.16597510373443985,370,12989,12991],[4425,0.261744966442953,370,12989,12992],[12852,0.24000000000000002,370,12989,12993],[5822,0.13559322033898305,370,12989,12994],[3763,0.26022304832713755,370,12989,12995],[3764,0.2231404958677686,370,12989,12996],[6377,0.2066115702479339,370,12989,12997],[3189,0.19841269841269843,370,12989,12998]]},{"id":"cq4oe-cq2onto-vgo-cqbycq-2026-paper","name":"CQ4OE · cq2onto vgo cqbycq (2026 paper)","group":"知识与推理","metric":"Global axiom F1","unit":"score","score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"F1 / 1","zh":"F1 / 1"},"public_history":true,"protocol":{"en":"Nine LLM checkpoints use the same expert-annotated VGO ontology domain. CQ2Term and CQ2Onto use separate gold standards (99/118 CQs over all six domains), each independently triple-reviewed with 2-of-3 consensus. Global axiom F1 uses layer4/grand, counting unmatched gold/predicted axioms as FN/FP; it is not the alignment-conditional layer3 score. The cqbycq generation condition is separate. Author outputs are pinned to repository commit ded082853bf4; semantic alignment uses the source embeddinggemma configuration. Scores remain native F1, without treating term extraction or ontology structure as whole-question accuracy.","zh":"九个LLM检查点处理相同VGO本体领域。CQ2Term与CQ2Onto使用不同金标准，六领域共99/118个能力问题，均由三人独立审查、至少两人同意。全局公理F1采用layer4/grand，将未对齐的金标准与预测公理计为FN/FP，不采用只评价已对齐项的layer3条件分。本组cqbycq生成条件单列。作者输出固定在仓库提交ded082853bf4，语义对齐保留原始embeddinggemma配置。F1保留原生含义，不当作整题准确率。"},"source":{"name":"Paper authors — pinned evaluation artifacts","url":"https://arxiv.org/abs/2609.26029v1","published_at":"2026-09-22","status":"paper_baselines"},"model_count":9,"result_count":9,"results":[[5796,0.14563106796116507,370,12999,13000],[5795,0.1111111111111111,370,12999,13001],[4425,0.1373390557939914,370,12999,13002],[12852,0.1772151898734177,370,12999,13003],[5822,0.1283422459893048,370,12999,13004],[3763,0.14018691588785046,370,12999,13005],[3764,0.15311004784688995,370,12999,13006],[6377,0.18478260869565216,370,12999,13007],[3189,0.14705882352941174,370,12999,13008]]},{"id":"cq4oe-cq2onto-water-cqbycq-2026-paper","name":"CQ4OE · cq2onto water cqbycq (2026 paper)","group":"知识与推理","metric":"Global axiom F1","unit":"score","score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"F1 / 1","zh":"F1 / 1"},"public_history":true,"protocol":{"en":"Nine LLM checkpoints use the same expert-annotated WATER ontology domain. CQ2Term and CQ2Onto use separate gold standards (99/118 CQs over all six domains), each independently triple-reviewed with 2-of-3 consensus. Global axiom F1 uses layer4/grand, counting unmatched gold/predicted axioms as FN/FP; it is not the alignment-conditional layer3 score. The cqbycq generation condition is separate. Author outputs are pinned to repository commit ded082853bf4; semantic alignment uses the source embeddinggemma configuration. Scores remain native F1, without treating term extraction or ontology structure as whole-question accuracy.","zh":"九个LLM检查点处理相同WATER本体领域。CQ2Term与CQ2Onto使用不同金标准，六领域共99/118个能力问题，均由三人独立审查、至少两人同意。全局公理F1采用layer4/grand，将未对齐的金标准与预测公理计为FN/FP，不采用只评价已对齐项的layer3条件分。本组cqbycq生成条件单列。作者输出固定在仓库提交ded082853bf4，语义对齐保留原始embeddinggemma配置。F1保留原生含义，不当作整题准确率。"},"source":{"name":"Paper authors — pinned evaluation artifacts","url":"https://arxiv.org/abs/2609.26029v1","published_at":"2026-09-22","status":"paper_baselines"},"model_count":9,"result_count":9,"results":[[5796,0.052083333333333336,370,13009,13010],[5795,0.0335195530726257,370,13009,13011],[4425,0.03076923076923077,370,13009,13012],[12852,0.03947368421052632,370,13009,13013],[5822,0.010256410256410256,370,13009,13014],[3763,0.07246376811594203,370,13009,13015],[3764,0.055944055944055944,370,13009,13016],[6377,0.06944444444444445,370,13009,13017],[3189,0.041666666666666664,370,13009,13018]]},{"id":"cq4oe-cq2onto-wine-cqbycq-2026-paper","name":"CQ4OE · cq2onto wine cqbycq (2026 paper)","group":"知识与推理","metric":"Global axiom F1","unit":"score","score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"F1 / 1","zh":"F1 / 1"},"public_history":true,"protocol":{"en":"Nine LLM checkpoints use the same expert-annotated WINE ontology domain. CQ2Term and CQ2Onto use separate gold standards (99/118 CQs over all six domains), each independently triple-reviewed with 2-of-3 consensus. Global axiom F1 uses layer4/grand, counting unmatched gold/predicted axioms as FN/FP; it is not the alignment-conditional layer3 score. The cqbycq generation condition is separate. Author outputs are pinned to repository commit ded082853bf4; semantic alignment uses the source embeddinggemma configuration. Scores remain native F1, without treating term extraction or ontology structure as whole-question accuracy.","zh":"九个LLM检查点处理相同WINE本体领域。CQ2Term与CQ2Onto使用不同金标准，六领域共99/118个能力问题，均由三人独立审查、至少两人同意。全局公理F1采用layer4/grand，将未对齐的金标准与预测公理计为FN/FP，不采用只评价已对齐项的layer3条件分。本组cqbycq生成条件单列。作者输出固定在仓库提交ded082853bf4，语义对齐保留原始embeddinggemma配置。F1保留原生含义，不当作整题准确率。"},"source":{"name":"Paper authors — pinned evaluation artifacts","url":"https://arxiv.org/abs/2609.26029v1","published_at":"2026-09-22","status":"paper_baselines"},"model_count":9,"result_count":9,"results":[[5796,0.2302158273381295,370,13019,13020],[5795,0.25641025641025644,370,13019,13021],[4425,0.3275862068965517,370,13019,13022],[12852,0.14736842105263157,370,13019,13023],[5822,0.2568807339449541,370,13019,13024],[3763,0.1730769230769231,370,13019,13025],[3764,0.23762376237623764,370,13019,13026],[6377,0.20952380952380953,370,13019,13027],[3189,0.1574803149606299,370,13019,13028]]},{"id":"cq4oe-cq2onto-awo-agent-2026-paper","name":"CQ4OE · cq2onto awo agent (2026 paper)","group":"知识与推理","metric":"Global axiom F1","unit":"score","score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"F1 / 1","zh":"F1 / 1"},"public_history":true,"protocol":{"en":"Nine LLM checkpoints use the same expert-annotated AWO ontology domain. CQ2Term and CQ2Onto use separate gold standards (99/118 CQs over all six domains), each independently triple-reviewed with 2-of-3 consensus. Global axiom F1 uses layer4/grand, counting unmatched gold/predicted axioms as FN/FP; it is not the alignment-conditional layer3 score. The agent generation condition is separate. Author outputs are pinned to repository commit ded082853bf4; semantic alignment uses the source embeddinggemma configuration. Scores remain native F1, without treating term extraction or ontology structure as whole-question accuracy.","zh":"九个LLM检查点处理相同AWO本体领域。CQ2Term与CQ2Onto使用不同金标准，六领域共99/118个能力问题，均由三人独立审查、至少两人同意。全局公理F1采用layer4/grand，将未对齐的金标准与预测公理计为FN/FP，不采用只评价已对齐项的layer3条件分。本组agent生成条件单列。作者输出固定在仓库提交ded082853bf4，语义对齐保留原始embeddinggemma配置。F1保留原生含义，不当作整题准确率。"},"source":{"name":"Paper authors — pinned evaluation artifacts","url":"https://arxiv.org/abs/2609.26029v1","published_at":"2026-09-22","status":"paper_baselines"},"model_count":9,"result_count":9,"results":[[5796,0.26666666666666666,370,13029,13030],[5795,0.2641509433962264,370,13029,13031],[4425,0.18666666666666668,370,13029,13032],[12852,0.21212121212121213,370,13029,13033],[5822,0.24,370,13029,13034],[3763,0.08163265306122448,370,13029,13035],[3764,0.23728813559322035,370,13029,13036],[6377,0.23728813559322035,370,13029,13037],[3189,0.24,370,13029,13038]]},{"id":"cq4oe-cq2onto-odrl-agent-2026-paper","name":"CQ4OE · cq2onto odrl agent (2026 paper)","group":"知识与推理","metric":"Global axiom F1","unit":"score","score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"F1 / 1","zh":"F1 / 1"},"public_history":true,"protocol":{"en":"Nine LLM checkpoints use the same expert-annotated ODRL ontology domain. CQ2Term and CQ2Onto use separate gold standards (99/118 CQs over all six domains), each independently triple-reviewed with 2-of-3 consensus. Global axiom F1 uses layer4/grand, counting unmatched gold/predicted axioms as FN/FP; it is not the alignment-conditional layer3 score. The agent generation condition is separate. Author outputs are pinned to repository commit ded082853bf4; semantic alignment uses the source embeddinggemma configuration. Scores remain native F1, without treating term extraction or ontology structure as whole-question accuracy.","zh":"九个LLM检查点处理相同ODRL本体领域。CQ2Term与CQ2Onto使用不同金标准，六领域共99/118个能力问题，均由三人独立审查、至少两人同意。全局公理F1采用layer4/grand，将未对齐的金标准与预测公理计为FN/FP，不采用只评价已对齐项的layer3条件分。本组agent生成条件单列。作者输出固定在仓库提交ded082853bf4，语义对齐保留原始embeddinggemma配置。F1保留原生含义，不当作整题准确率。"},"source":{"name":"Paper authors — pinned evaluation artifacts","url":"https://arxiv.org/abs/2609.26029v1","published_at":"2026-09-22","status":"paper_baselines"},"model_count":9,"result_count":9,"results":[[5796,0.13145539906103285,370,13039,13040],[5795,0.1743119266055046,370,13039,13041],[4425,0.16438356164383564,370,13039,13042],[12852,0.13605442176870747,370,13039,13043],[5822,0.1575757575757576,370,13039,13044],[3763,0.11872146118721462,370,13039,13045],[3764,0.10909090909090909,370,13039,13046],[6377,0.09454545454545454,370,13039,13047],[3189,0.13333333333333333,370,13039,13048]]},{"id":"cq4oe-cq2onto-swo-agent-2026-paper","name":"CQ4OE · cq2onto swo agent (2026 paper)","group":"知识与推理","metric":"Global axiom F1","unit":"score","score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"F1 / 1","zh":"F1 / 1"},"public_history":true,"protocol":{"en":"Nine LLM checkpoints use the same expert-annotated SWO ontology domain. CQ2Term and CQ2Onto use separate gold standards (99/118 CQs over all six domains), each independently triple-reviewed with 2-of-3 consensus. Global axiom F1 uses layer4/grand, counting unmatched gold/predicted axioms as FN/FP; it is not the alignment-conditional layer3 score. The agent generation condition is separate. Author outputs are pinned to repository commit ded082853bf4; semantic alignment uses the source embeddinggemma configuration. Scores remain native F1, without treating term extraction or ontology structure as whole-question accuracy.","zh":"九个LLM检查点处理相同SWO本体领域。CQ2Term与CQ2Onto使用不同金标准，六领域共99/118个能力问题，均由三人独立审查、至少两人同意。全局公理F1采用layer4/grand，将未对齐的金标准与预测公理计为FN/FP，不采用只评价已对齐项的layer3条件分。本组agent生成条件单列。作者输出固定在仓库提交ded082853bf4，语义对齐保留原始embeddinggemma配置。F1保留原生含义，不当作整题准确率。"},"source":{"name":"Paper authors — pinned evaluation artifacts","url":"https://arxiv.org/abs/2609.26029v1","published_at":"2026-09-22","status":"paper_baselines"},"model_count":9,"result_count":9,"results":[[5796,0.13550135501355015,370,13049,13050],[5795,0.11859838274932614,370,13049,13051],[4425,0.2170542635658915,370,13049,13052],[12852,0.1276595744680851,370,13049,13053],[5822,0.1595744680851064,370,13049,13054],[3763,0.11764705882352941,370,13049,13055],[3764,0.12794612794612795,370,13049,13056],[6377,0.16487455197132617,370,13049,13057],[3189,0.15436241610738255,370,13049,13058]]},{"id":"cq4oe-cq2onto-vgo-agent-2026-paper","name":"CQ4OE · cq2onto vgo agent (2026 paper)","group":"知识与推理","metric":"Global axiom F1","unit":"score","score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"F1 / 1","zh":"F1 / 1"},"public_history":true,"protocol":{"en":"Nine LLM checkpoints use the same expert-annotated VGO ontology domain. CQ2Term and CQ2Onto use separate gold standards (99/118 CQs over all six domains), each independently triple-reviewed with 2-of-3 consensus. Global axiom F1 uses layer4/grand, counting unmatched gold/predicted axioms as FN/FP; it is not the alignment-conditional layer3 score. The agent generation condition is separate. Author outputs are pinned to repository commit ded082853bf4; semantic alignment uses the source embeddinggemma configuration. Scores remain native F1, without treating term extraction or ontology structure as whole-question accuracy.","zh":"九个LLM检查点处理相同VGO本体领域。CQ2Term与CQ2Onto使用不同金标准，六领域共99/118个能力问题，均由三人独立审查、至少两人同意。全局公理F1采用layer4/grand，将未对齐的金标准与预测公理计为FN/FP，不采用只评价已对齐项的layer3条件分。本组agent生成条件单列。作者输出固定在仓库提交ded082853bf4，语义对齐保留原始embeddinggemma配置。F1保留原生含义，不当作整题准确率。"},"source":{"name":"Paper authors — pinned evaluation artifacts","url":"https://arxiv.org/abs/2609.26029v1","published_at":"2026-09-22","status":"paper_baselines"},"model_count":9,"result_count":9,"results":[[5796,0.12099644128113878,370,13059,13060],[5795,0.12751677852348994,370,13059,13061],[4425,0.14345991561181434,370,13059,13062],[12852,0.13714285714285715,370,13059,13063],[5822,0.11999999999999998,370,13059,13064],[3763,0.1328125,370,13059,13065],[3764,0.0,370,13059,13066],[6377,0.16033755274261605,370,13059,13067],[3189,0.12686567164179102,370,13059,13068]]},{"id":"cq4oe-cq2onto-water-agent-2026-paper","name":"CQ4OE · cq2onto water agent (2026 paper)","group":"知识与推理","metric":"Global axiom F1","unit":"score","score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"F1 / 1","zh":"F1 / 1"},"public_history":true,"protocol":{"en":"Nine LLM checkpoints use the same expert-annotated WATER ontology domain. CQ2Term and CQ2Onto use separate gold standards (99/118 CQs over all six domains), each independently triple-reviewed with 2-of-3 consensus. Global axiom F1 uses layer4/grand, counting unmatched gold/predicted axioms as FN/FP; it is not the alignment-conditional layer3 score. The agent generation condition is separate. Author outputs are pinned to repository commit ded082853bf4; semantic alignment uses the source embeddinggemma configuration. Scores remain native F1, without treating term extraction or ontology structure as whole-question accuracy.","zh":"九个LLM检查点处理相同WATER本体领域。CQ2Term与CQ2Onto使用不同金标准，六领域共99/118个能力问题，均由三人独立审查、至少两人同意。全局公理F1采用layer4/grand，将未对齐的金标准与预测公理计为FN/FP，不采用只评价已对齐项的layer3条件分。本组agent生成条件单列。作者输出固定在仓库提交ded082853bf4，语义对齐保留原始embeddinggemma配置。F1保留原生含义，不当作整题准确率。"},"source":{"name":"Paper authors — pinned evaluation artifacts","url":"https://arxiv.org/abs/2609.26029v1","published_at":"2026-09-22","status":"paper_baselines"},"model_count":9,"result_count":9,"results":[[5796,0.010101010101010102,370,13069,13070],[5795,0.029850746268656716,370,13069,13071],[4425,0.06896551724137931,370,13069,13072],[12852,0.10619469026548672,370,13069,13073],[5822,0.047999999999999994,370,13069,13074],[3763,0.04938271604938271,370,13069,13075],[3764,0.0,370,13069,13076],[6377,0.0,370,13069,13077],[3189,0.07734806629834254,370,13069,13078]]},{"id":"cq4oe-cq2onto-wine-agent-2026-paper","name":"CQ4OE · cq2onto wine agent (2026 paper)","group":"知识与推理","metric":"Global axiom F1","unit":"score","score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"F1 / 1","zh":"F1 / 1"},"public_history":true,"protocol":{"en":"Nine LLM checkpoints use the same expert-annotated WINE ontology domain. CQ2Term and CQ2Onto use separate gold standards (99/118 CQs over all six domains), each independently triple-reviewed with 2-of-3 consensus. Global axiom F1 uses layer4/grand, counting unmatched gold/predicted axioms as FN/FP; it is not the alignment-conditional layer3 score. The agent generation condition is separate. Author outputs are pinned to repository commit ded082853bf4; semantic alignment uses the source embeddinggemma configuration. Scores remain native F1, without treating term extraction or ontology structure as whole-question accuracy.","zh":"九个LLM检查点处理相同WINE本体领域。CQ2Term与CQ2Onto使用不同金标准，六领域共99/118个能力问题，均由三人独立审查、至少两人同意。全局公理F1采用layer4/grand，将未对齐的金标准与预测公理计为FN/FP，不采用只评价已对齐项的layer3条件分。本组agent生成条件单列。作者输出固定在仓库提交ded082853bf4，语义对齐保留原始embeddinggemma配置。F1保留原生含义，不当作整题准确率。"},"source":{"name":"Paper authors — pinned evaluation artifacts","url":"https://arxiv.org/abs/2609.26029v1","published_at":"2026-09-22","status":"paper_baselines"},"model_count":9,"result_count":9,"results":[[5796,0.22950819672131148,370,13079,13080],[5795,0.0273972602739726,370,13079,13081],[4425,0.27586206896551724,370,13079,13082],[12852,0.125,370,13079,13083],[5822,0.26785714285714285,370,13079,13084],[3763,0.29370629370629375,370,13079,13085],[3764,0.24489795918367346,370,13079,13086],[6377,0.22085889570552147,370,13079,13087],[3189,0.2260869565217391,370,13079,13088]]},{"id":"eccbench-easyhuman-video-2026-paper","name":"ECCBench · video compression reasoning (2026 paper)","group":"视觉与视频","metric":"EasyHuman video accuracy at L=256","unit":"%","public_history":true,"protocol":{"en":"The complete ten-configuration panel answers the same EasyHuman video questions at sequence length 256. Published accuracy is retained separately from the text tasks, synthetic sequences and per-response FLOPs. Thinking variants remain configurations, with at least six distinct model checkpoints without those repeats.","zh":"完整十配置面板回答同一EasyHuman视频问题，序列长度固定为256。保留原始准确率，与文本任务、合成序列和每次响应计算量分开。思考变体按配置保留，不靠这些重复配置也有至少六种检查点。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.00103v2","status":"paper_baselines","published_at":"2026-08-31"},"model_count":10,"result_count":10,"results":[[6901,0.491,370,13089,13090],[13091,0.478,370,13089,13090],[13092,0.536,370,13089,13090],[13093,0.562,370,13089,13090],[9707,0.496,370,13089,13090],[13094,0.531,370,13089,13090],[13095,0.527,370,13089,13090],[3578,0.511,370,13089,13090],[3412,0.526,370,13089,13090],[13096,0.516,370,13089,13090]]},{"id":"inspire-dailytalk-recall10-2026-paper","name":"INSPIRE · DailyTalk spoken retrieval (2026 paper)","group":"专业工作","metric":"Recall@10","unit":"%","public_history":true,"protocol":{"en":"The complete thirteen-retriever panel ranks the same DailyTalk spoken-query subset and candidate pool. Audio-language embeddings, cascaded text retrieval and speech encoders retain their prescribed input routes. Recall@10 is separate from reranking NDCG. Random reference is excluded; BM25 remains a lexical retrieval baseline and does not increase the learned-model count.","zh":"完整十三检索器面板评测同一DailyTalk口语查询子集与候选池。音频语言嵌入、级联文本检索及语音编码器保留规定输入方式。Recall@10与重排NDCG分开。排除随机参考；BM25保留为词法检索基线，不增加学习模型数。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.16203v1","status":"paper_baselines","published_at":"2026-08-17"},"model_count":13,"result_count":13,"results":[[13097,0.3,370,13098,13099],[13100,0.265,370,13101,13099],[7523,0.36,370,13102,13099],[13103,0.515,370,13104,13099],[13105,0.39,370,13101,13099],[13106,0.41,370,13107,13099],[12189,0.365,370,13108,13099],[13109,0.325,370,13110,13099],[13111,0.555,370,13112,13099],[13113,0.62,370,13114,13099],[13115,0.11,370,13116,13099],[13117,0.165,370,13116,13099],[13118,0.0,370,13119,13099]]},{"id":"inspire-vctk-recall10-2026-paper","name":"INSPIRE · VCTK spoken retrieval (2026 paper)","group":"专业工作","metric":"Recall@10","unit":"%","public_history":true,"protocol":{"en":"The complete thirteen-retriever panel ranks the same VCTK spoken-query subset and candidate pool. Audio-language embeddings, cascaded text retrieval and speech encoders retain their prescribed input routes. Recall@10 is separate from reranking NDCG. Random reference is excluded; BM25 remains a lexical retrieval baseline and does not increase the learned-model count.","zh":"完整十三检索器面板评测同一VCTK口语查询子集与候选池。音频语言嵌入、级联文本检索及语音编码器保留规定输入方式。Recall@10与重排NDCG分开。排除随机参考；BM25保留为词法检索基线，不增加学习模型数。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.16203v1","status":"paper_baselines","published_at":"2026-08-17"},"model_count":13,"result_count":13,"results":[[13097,0.0125,370,13120,13099],[13100,0.018799999999999997,370,13121,13099],[7523,0.0125,370,13122,13099],[13103,0.0125,370,13123,13099],[13105,0.0063,370,13121,13099],[13106,0.0125,370,13124,13099],[12189,0.0125,370,13125,13099],[13109,0.0063,370,13126,13099],[13111,0.0125,370,13127,13099],[13113,0.018799999999999997,370,13128,13099],[13115,0.1188,370,13129,13099],[13117,0.175,370,13129,13099],[13118,0.0194,370,13130,13099]]},{"id":"inspire-expresso-recall10-2026-paper","name":"INSPIRE · Expresso spoken retrieval (2026 paper)","group":"专业工作","metric":"Recall@10","unit":"%","public_history":true,"protocol":{"en":"The complete thirteen-retriever panel ranks the same Expresso spoken-query subset and candidate pool. Audio-language embeddings, cascaded text retrieval and speech encoders retain their prescribed input routes. Recall@10 is separate from reranking NDCG. Random reference is excluded; BM25 remains a lexical retrieval baseline and does not increase the learned-model count.","zh":"完整十三检索器面板评测同一Expresso口语查询子集与候选池。音频语言嵌入、级联文本检索及语音编码器保留规定输入方式。Recall@10与重排NDCG分开。排除随机参考；BM25保留为词法检索基线，不增加学习模型数。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.16203v1","status":"paper_baselines","published_at":"2026-08-17"},"model_count":13,"result_count":13,"results":[[13097,0.0219,370,13131,13099],[13100,0.0075,370,13132,13099],[7523,0.0063,370,13133,13099],[13103,0.009399999999999999,370,13134,13099],[13105,0.0038,370,13132,13099],[13106,0.0031,370,13135,13099],[12189,0.005600000000000001,370,13136,13099],[13109,0.0044,370,13137,13099],[13111,0.0163,370,13138,13099],[13113,0.0075,370,13139,13099],[13115,0.0981,370,13140,13099],[13117,0.095,370,13140,13099],[13118,0.018799999999999997,370,13141,13099]]},{"id":"inspire-synthetic-recall10-2026-paper","name":"INSPIRE · Synthetic spoken retrieval (2026 paper)","group":"专业工作","metric":"Recall@10","unit":"%","public_history":true,"protocol":{"en":"The complete thirteen-retriever panel ranks the same Synthetic spoken-query subset and candidate pool. Audio-language embeddings, cascaded text retrieval and speech encoders retain their prescribed input routes. Recall@10 is separate from reranking NDCG. Random reference is excluded; BM25 remains a lexical retrieval baseline and does not increase the learned-model count.","zh":"完整十三检索器面板评测同一Synthetic口语查询子集与候选池。音频语言嵌入、级联文本检索及语音编码器保留规定输入方式。Recall@10与重排NDCG分开。排除随机参考；BM25保留为词法检索基线，不增加学习模型数。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.16203v1","status":"paper_baselines","published_at":"2026-08-17"},"model_count":13,"result_count":13,"results":[[13097,0.0382,370,13142,13099],[13100,0.031400000000000004,370,13143,13099],[7523,0.0418,370,13144,13099],[13103,0.048799999999999996,370,13145,13099],[13105,0.0387,370,13143,13099],[13106,0.051699999999999996,370,13146,13099],[12189,0.0406,370,13147,13099],[13109,0.0416,370,13148,13099],[13111,0.065,370,13149,13099],[13113,0.0702,370,13150,13099],[13115,0.0128,370,13151,13099],[13117,0.017,370,13151,13099],[13118,0.008,370,13152,13099]]},{"id":"llamia-behavior-maia-2026-paper","name":"LLAMIA-Bench · maia behavior cloning (2026 paper)","group":"专业工作","metric":"Move-match accuracy","unit":"%","public_history":true,"protocol":{"en":"The complete nineteen-system behavior-cloning panel uses the same maia benchmark split. Original move-match accuracy is retained. Maia and Allie are independently learned chess architectures; GPT-5 plus Qwen3 4B/8B/14B supply four further base checkpoints. Allie search, Lc0 access, verbalized/latent interfaces and SFT/DAPO settings are configurations. Stockfish is a chess-engine baseline and does not contribute to the six learned-model minimum.","zh":"完整十九系统行为模仿面板使用同一maia测试划分，保留原始落子匹配准确率。Maia与Allie是独立学习的国际象棋架构，GPT-5及Qwen3的4B、8B、14B提供另外四种检查点。Allie搜索、Lc0访问、语言或隐状态接口、监督微调与DAPO均作为配置。Stockfish是象棋引擎基线，不计入六学习模型门槛。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.00474v2","status":"paper_baselines","published_at":"2026-08-31"},"model_count":13,"result_count":19,"results":[[13153,0.39,370,13154,13155],[13156,0.52,370,13154,13155],[13157,0.54,370,13154,13155],[13158,0.55,370,13154,13155],[13159,0.28,370,13160,13155],[13161,0.45,370,13160,13155],[13162,0.39,370,13160,13155],[13163,0.4,370,13164,13155],[13165,0.42,370,13164,13155],[13166,0.44,370,13164,13155],[13163,0.41,370,13167,13155],[13165,0.44,370,13167,13155],[13166,0.45,370,13167,13155],[13168,0.48,370,13169,13155],[13170,0.5,370,13169,13155],[13171,0.51,370,13169,13155],[13168,0.5,370,13172,13155],[13170,0.52,370,13172,13155],[13171,0.53,370,13172,13155]]},{"id":"llamia-behavior-wild-2026-paper","name":"LLAMIA-Bench · wild behavior cloning (2026 paper)","group":"专业工作","metric":"Move-match accuracy","unit":"%","public_history":true,"protocol":{"en":"The complete nineteen-system behavior-cloning panel uses the same wild benchmark split. Original move-match accuracy is retained. Maia and Allie are independently learned chess architectures; GPT-5 plus Qwen3 4B/8B/14B supply four further base checkpoints. Allie search, Lc0 access, verbalized/latent interfaces and SFT/DAPO settings are configurations. Stockfish is a chess-engine baseline and does not contribute to the six learned-model minimum.","zh":"完整十九系统行为模仿面板使用同一wild测试划分，保留原始落子匹配准确率。Maia与Allie是独立学习的国际象棋架构，GPT-5及Qwen3的4B、8B、14B提供另外四种检查点。Allie搜索、Lc0访问、语言或隐状态接口、监督微调与DAPO均作为配置。Stockfish是象棋引擎基线，不计入六学习模型门槛。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.00474v2","status":"paper_baselines","published_at":"2026-08-31"},"model_count":13,"result_count":19,"results":[[13153,0.37,370,13173,13155],[13156,0.42,370,13173,13155],[13157,0.44,370,13173,13155],[13158,0.45,370,13173,13155],[13159,0.22,370,13174,13155],[13161,0.4,370,13174,13155],[13162,0.33,370,13174,13155],[13163,0.32,370,13175,13155],[13165,0.35,370,13175,13155],[13166,0.37,370,13175,13155],[13163,0.34,370,13176,13155],[13165,0.37,370,13176,13155],[13166,0.39,370,13176,13155],[13168,0.41,370,13177,13155],[13170,0.44,370,13177,13155],[13171,0.46,370,13177,13155],[13168,0.47,370,13178,13155],[13170,0.51,370,13178,13155],[13171,0.57,370,13178,13155]]},{"id":"oenobench-regions-2026-paper","name":"OenoBench · regions (2026 paper)","group":"专业工作","metric":"Closed-book domain accuracy","unit":"%","public_history":true,"protocol":{"en":"All sixteen configurations answer the same OenoBench regions questions under the released closed-book protocol, drawn from the common 3,266-question benchmark. Source model names and thinking modes are retained. The mean-of-configurations reference row is excluded; source domain, difficulty and contextual-gap diagnostics remain separate.","zh":"完整十六配置面板按发布的闭卷协议回答同一OenoBench regions领域题，来自共同的3,266题基准。原始模型名和思考模式保留。排除配置均值参考行，领域、难度和上下文增益诊断单列。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.20106v1","status":"paper_baselines","published_at":"2026-08-20"},"model_count":16,"result_count":16,"results":[[13179,0.8320000000000001,370,13180,13181],[5111,0.5579999999999999,370,13180,13181],[4196,0.8809999999999999,370,13180,13181],[5159,0.821,370,13180,13181],[457,0.87,370,13180,13181],[4164,0.769,370,13180,13181],[10713,0.6890000000000001,370,13180,13181],[10711,0.622,370,13180,13181],[10707,0.7440000000000001,370,13180,13181],[13182,0.69,370,13180,13181],[13183,0.591,370,13180,13181],[13184,0.715,370,13180,13181],[1464,0.884,370,13180,13181],[13185,0.865,370,13186,13181],[4194,0.805,370,13180,13181],[13187,0.8320000000000001,370,13186,13181]]},{"id":"oenobench-varieties-2026-paper","name":"OenoBench · varieties (2026 paper)","group":"专业工作","metric":"Closed-book domain accuracy","unit":"%","public_history":true,"protocol":{"en":"All sixteen configurations answer the same OenoBench varieties questions under the released closed-book protocol, drawn from the common 3,266-question benchmark. Source model names and thinking modes are retained. The mean-of-configurations reference row is excluded; source domain, difficulty and contextual-gap diagnostics remain separate.","zh":"完整十六配置面板按发布的闭卷协议回答同一OenoBench varieties领域题，来自共同的3,266题基准。原始模型名和思考模式保留。排除配置均值参考行，领域、难度和上下文增益诊断单列。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.20106v1","status":"paper_baselines","published_at":"2026-08-20"},"model_count":16,"result_count":16,"results":[[13179,0.802,370,13188,13181],[5111,0.49700000000000005,370,13188,13181],[4196,0.782,370,13188,13181],[5159,0.7440000000000001,370,13188,13181],[457,0.7879999999999999,370,13188,13181],[4164,0.7290000000000001,370,13188,13181],[10713,0.629,370,13188,13181],[10711,0.556,370,13188,13181],[10707,0.685,370,13188,13181],[13182,0.6509999999999999,370,13188,13181],[13183,0.556,370,13188,13181],[13184,0.6559999999999999,370,13188,13181],[1464,0.813,370,13188,13181],[13185,0.805,370,13189,13181],[4194,0.736,370,13188,13181],[13187,0.809,370,13189,13181]]},{"id":"oenobench-producers-2026-paper","name":"OenoBench · producers (2026 paper)","group":"专业工作","metric":"Closed-book domain accuracy","unit":"%","public_history":true,"protocol":{"en":"All sixteen configurations answer the same OenoBench producers questions under the released closed-book protocol, drawn from the common 3,266-question benchmark. Source model names and thinking modes are retained. The mean-of-configurations reference row is excluded; source domain, difficulty and contextual-gap diagnostics remain separate.","zh":"完整十六配置面板按发布的闭卷协议回答同一OenoBench producers领域题，来自共同的3,266题基准。原始模型名和思考模式保留。排除配置均值参考行，领域、难度和上下文增益诊断单列。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.20106v1","status":"paper_baselines","published_at":"2026-08-20"},"model_count":16,"result_count":16,"results":[[13179,0.868,370,13190,13181],[5111,0.5870000000000001,370,13190,13181],[4196,0.884,370,13190,13181],[5159,0.835,370,13190,13181],[457,0.89,370,13190,13181],[4164,0.823,370,13190,13181],[10713,0.772,370,13190,13181],[10711,0.695,370,13190,13181],[10707,0.732,370,13190,13181],[13182,0.7340000000000001,370,13190,13181],[13183,0.594,370,13190,13181],[13184,0.754,370,13190,13181],[1464,0.909,370,13190,13181],[13185,0.9,370,13191,13181],[4194,0.856,370,13190,13181],[13187,0.87,370,13191,13181]]},{"id":"oenobench-viticulture-2026-paper","name":"OenoBench · viticulture (2026 paper)","group":"专业工作","metric":"Closed-book domain accuracy","unit":"%","public_history":true,"protocol":{"en":"All sixteen configurations answer the same OenoBench viticulture questions under the released closed-book protocol, drawn from the common 3,266-question benchmark. Source model names and thinking modes are retained. The mean-of-configurations reference row is excluded; source domain, difficulty and contextual-gap diagnostics remain separate.","zh":"完整十六配置面板按发布的闭卷协议回答同一OenoBench viticulture领域题，来自共同的3,266题基准。原始模型名和思考模式保留。排除配置均值参考行，领域、难度和上下文增益诊断单列。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.20106v1","status":"paper_baselines","published_at":"2026-08-20"},"model_count":16,"result_count":16,"results":[[13179,0.779,370,13192,13181],[5111,0.542,370,13192,13181],[4196,0.828,370,13192,13181],[5159,0.775,370,13192,13181],[457,0.785,370,13192,13181],[4164,0.736,370,13192,13181],[10713,0.657,370,13192,13181],[10711,0.617,370,13192,13181],[10707,0.708,370,13192,13181],[13182,0.675,370,13192,13181],[13183,0.574,370,13192,13181],[13184,0.6940000000000001,370,13192,13181],[1464,0.8029999999999999,370,13192,13181],[13185,0.805,370,13193,13181],[4194,0.722,370,13192,13181],[13187,0.787,370,13193,13181]]},{"id":"oenobench-winemaking-2026-paper","name":"OenoBench · winemaking (2026 paper)","group":"专业工作","metric":"Closed-book domain accuracy","unit":"%","public_history":true,"protocol":{"en":"All sixteen configurations answer the same OenoBench winemaking questions under the released closed-book protocol, drawn from the common 3,266-question benchmark. Source model names and thinking modes are retained. The mean-of-configurations reference row is excluded; source domain, difficulty and contextual-gap diagnostics remain separate.","zh":"完整十六配置面板按发布的闭卷协议回答同一OenoBench winemaking领域题，来自共同的3,266题基准。原始模型名和思考模式保留。排除配置均值参考行，领域、难度和上下文增益诊断单列。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.20106v1","status":"paper_baselines","published_at":"2026-08-20"},"model_count":16,"result_count":16,"results":[[13179,0.866,370,13194,13181],[5111,0.5670000000000001,370,13194,13181],[4196,0.807,370,13194,13181],[5159,0.775,370,13194,13181],[457,0.775,370,13194,13181],[4164,0.775,370,13194,13181],[10713,0.706,370,13194,13181],[10711,0.594,370,13194,13181],[10707,0.69,370,13194,13181],[13182,0.701,370,13194,13181],[13183,0.578,370,13194,13181],[13184,0.733,370,13194,13181],[1464,0.765,370,13194,13181],[13185,0.8340000000000001,370,13195,13181],[4194,0.765,370,13194,13181],[13187,0.84,370,13195,13181]]},{"id":"oenobench-business-2026-paper","name":"OenoBench · business (2026 paper)","group":"专业工作","metric":"Closed-book domain accuracy","unit":"%","public_history":true,"protocol":{"en":"All sixteen configurations answer the same OenoBench business questions under the released closed-book protocol, drawn from the common 3,266-question benchmark. Source model names and thinking modes are retained. The mean-of-configurations reference row is excluded; source domain, difficulty and contextual-gap diagnostics remain separate.","zh":"完整十六配置面板按发布的闭卷协议回答同一OenoBench business领域题，来自共同的3,266题基准。原始模型名和思考模式保留。排除配置均值参考行，领域、难度和上下文增益诊断单列。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.20106v1","status":"paper_baselines","published_at":"2026-08-20"},"model_count":16,"result_count":16,"results":[[13179,0.642,370,13196,13181],[5111,0.37799999999999995,370,13196,13181],[4196,0.63,370,13196,13181],[5159,0.6629999999999999,370,13196,13181],[457,0.618,370,13196,13181],[4164,0.606,370,13196,13181],[10713,0.504,370,13196,13181],[10711,0.47600000000000003,370,13196,13181],[10707,0.512,370,13196,13181],[13182,0.528,370,13196,13181],[13183,0.447,370,13196,13181],[13184,0.516,370,13196,13181],[1464,0.654,370,13196,13181],[13185,0.602,370,13197,13181],[4194,0.6459999999999999,370,13196,13181],[13187,0.614,370,13197,13181]]},{"id":"revico-visual-token-correction-2026-paper","name":"ReViCo · visual text correction (2026 paper)","group":"视觉与视频","metric":"Token-level correction F1","unit":"score","public_history":true,"protocol":{"en":"The complete twenty-model image-input cohort detects and corrects errors in the same ReViCo text-rich images. Token-level correction F1 is kept separate from image-level success and detection F1. The six oracle-OCR text-input rows have a different information state and form a separate cohort.","zh":"完整二十模型图像输入面板在同一ReViCo含字图像中检测并修正文字符号错误。保留字词级修正F1，与图像级成功率及检测F1分开。六个使用理想OCR文本的输入行信息条件不同，单列。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.27154v1","status":"paper_baselines","published_at":"2026-08-27"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":20,"result_count":20,"results":[[13198,6.0,370,13199,13200],[13201,14.7,370,13199,13200],[13202,36.5,370,13199,13200],[13203,33.1,370,13199,13200],[13204,10.6,370,13199,13200],[13205,22.4,370,13199,13200],[13206,19.0,370,13199,13200],[13207,32.7,370,13199,13200],[13208,35.1,370,13199,13200],[13209,3.7,370,13199,13200],[13210,5.5,370,13199,13200],[13211,6.7,370,13199,13200],[13212,14.2,370,13199,13200],[13213,23.9,370,13199,13200],[13214,38.8,370,13199,13200],[13215,38.2,370,13199,13200],[13216,67.0,370,13199,13200],[13217,83.7,370,13199,13200],[13218,75.9,370,13199,13200],[13219,80.1,370,13199,13200]]},{"id":"bird-history-extended-2026-paper","name":"BIRD-History · history-grounded SQL (2026 paper)","group":"编程","metric":"SQL execution accuracy","unit":"%","public_history":true,"protocol":{"en":"The full fifteen-system extended comparison combines the original fine-tuned, prompt-framework and foundation-model panels on the same 1,393 BIRD-History tasks across eleven databases. Prompt frameworks use the same pinned Qwen3 qwen-plus-2025-07-28 model and remain explicit configurations; five foundation models plus independent SQL checkpoint bases satisfy the six-base minimum. Exact execution accuracies retain their original table and cell coordinates. BIRD Dev scores and retrieval-augmented ablations are separate.","zh":"完整十五系统扩展对照由原始微调、提示框架和基础模型面板组成，使用同一十一数据库的1,393道BIRD-History题。提示框架都采用固定Qwen3 qwen-plus-2025-07-28，作为明确配置保留；五个基础模型和独立SQL检查点底模满足六底模门槛。执行准确率保留各自原表与单元格坐标。BIRD Dev分数和检索增强消融单列。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.29345v1","status":"paper_baselines","published_at":"2026-08-29"},"model_count":11,"result_count":15,"results":[[13220,0.2297,370,13221,13222],[4624,0.1902,370,13221,13222],[13223,0.26489999999999997,370,13221,13222],[13224,0.2175,370,13221,13222],[13225,0.2864,370,13221,13222],[13226,0.4637,370,13227,13222],[13226,0.4472,370,13228,13222],[13226,0.31079999999999997,370,13229,13222],[13226,0.1091,370,13230,13222],[13226,0.2757,370,13231,13222],[10620,0.2642,370,13232,13233],[3009,0.21109999999999998,370,13232,13233],[13234,0.16649999999999998,370,13232,13233],[3733,0.2893,370,13232,13233],[6133,0.33240000000000003,370,13232,13233]]},{"id":"harnessrisk-asr-2026-paper","name":"HarnessRisk · agent harness safety (2026 paper)","group":"安全与可靠性","metric":"Attack success rate","unit":"%","public_history":true,"protocol":{"en":"The complete fourteen model–harness configurations evaluate the same 128 lifecycle attack cases with three independently reset sandbox runs. Six distinct model checkpoints are present; OpenClaw, Nanobot and Hermes are explicit harness configurations. The PDF Table 2 text layer supplies exact ASR means, separately from subscript standard deviations. Source Figure 5’s twelve-configuration diagnostic is separate from this fourteen-row result table.","zh":"完整十四模型与框架配置在同一128种生命周期攻击案例中，分别重置沙箱运行三次。面板含六种不同检查点，OpenClaw、Nanobot和Hermes为明确框架配置。从PDF表2文字层提取精确攻击成功率均值，下标标准差单独保留。图5的十二配置诊断与此十四行结果表分开。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.17597v1","status":"paper_baselines","published_at":"2026-08-18"},"higher_is_better":false,"saturation_excluded":true,"model_count":6,"result_count":14,"results":[[3196,0.7559,370,13235,13236],[3578,0.4771,370,13235,13236],[3185,0.54,370,13235,13236],[227,0.547,370,13235,13236],[3490,0.8087000000000001,370,13235,13236],[3491,0.312,370,13235,13236],[3185,0.373,370,13237,13236],[227,0.126,370,13237,13236],[3490,0.552,370,13237,13236],[3491,0.268,370,13237,13236],[3185,0.654,370,13238,13236],[227,0.23800000000000002,370,13238,13236],[3490,0.6559999999999999,370,13238,13236],[3491,0.14800000000000002,370,13238,13236]]},{"id":"swe-rpg-opencode-2026-paper","name":"SWE-RPG · OpenCode (2026 paper)","group":"编程","metric":"Resolve rate","unit":"%","public_history":true,"protocol":{"en":"All six model backends run the same 163 quality-controlled repository issues twice with OpenCode. A task resolves only if the patch applies, all fail-to-pass tests pass and no pass-to-pass test regresses. Exact printed Figure 5 percentages are extracted from the PDF text layer, without reading values from heatmap colors. Model abbreviations map to the explicit checkpoint roster in the same page’s experimental setup.","zh":"完整六底模使用OpenCode对相同163个经过质量检查的仓库问题各运行两次。补丁成功应用、全部由失败转通过测试通过，且原有通过测试无退化，才算解决。精确百分比直接取自图5的PDF文字层，不依据热图颜色估值。模型缩写按同页实验设置中的检查点清单对应。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.09072v1","status":"paper_baselines","published_at":"2026-08-10"},"model_count":6,"result_count":6,"results":[[3568,0.301,370,13239,13240],[3492,0.36200000000000004,370,13239,13240],[227,0.368,370,13239,13240],[5727,0.252,370,13239,13240],[3491,0.20199999999999999,370,13239,13240],[2977,0.49700000000000005,370,13239,13240]]},{"id":"swe-rpg-claude-code-2026-paper","name":"SWE-RPG · Claude Code (2026 paper)","group":"编程","metric":"Resolve rate","unit":"%","public_history":true,"protocol":{"en":"All six model backends run the same 163 quality-controlled repository issues twice with Claude Code. A task resolves only if the patch applies, all fail-to-pass tests pass and no pass-to-pass test regresses. Exact printed Figure 5 percentages are extracted from the PDF text layer, without reading values from heatmap colors. Model abbreviations map to the explicit checkpoint roster in the same page’s experimental setup.","zh":"完整六底模使用Claude Code对相同163个经过质量检查的仓库问题各运行两次。补丁成功应用、全部由失败转通过测试通过，且原有通过测试无退化，才算解决。精确百分比直接取自图5的PDF文字层，不依据热图颜色估值。模型缩写按同页实验设置中的检查点清单对应。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.09072v1","status":"paper_baselines","published_at":"2026-08-10"},"model_count":6,"result_count":6,"results":[[3568,0.313,370,13241,13240],[3492,0.39299999999999996,370,13241,13240],[227,0.258,370,13241,13240],[5727,0.22699999999999998,370,13241,13240],[3491,0.28800000000000003,370,13241,13240],[2977,0.491,370,13241,13240]]},{"id":"swe-rpg-codex-2026-paper","name":"SWE-RPG · Codex (2026 paper)","group":"编程","metric":"Resolve rate","unit":"%","public_history":true,"protocol":{"en":"All six model backends run the same 163 quality-controlled repository issues twice with Codex. A task resolves only if the patch applies, all fail-to-pass tests pass and no pass-to-pass test regresses. Exact printed Figure 5 percentages are extracted from the PDF text layer, without reading values from heatmap colors. Model abbreviations map to the explicit checkpoint roster in the same page’s experimental setup.","zh":"完整六底模使用Codex对相同163个经过质量检查的仓库问题各运行两次。补丁成功应用、全部由失败转通过测试通过，且原有通过测试无退化，才算解决。精确百分比直接取自图5的PDF文字层，不依据热图颜色估值。模型缩写按同页实验设置中的检查点清单对应。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.09072v1","status":"paper_baselines","published_at":"2026-08-10"},"model_count":6,"result_count":6,"results":[[3568,0.264,370,13242,13240],[3492,0.405,370,13242,13240],[227,0.233,370,13242,13240],[5727,0.221,370,13242,13240],[3491,0.17800000000000002,370,13242,13240],[2977,0.41100000000000003,370,13242,13240]]},{"id":"medseg-model-latency-256-2026-paper","name":"MedSegBenchmarker · 256px inference (2026 paper)","group":"医疗","metric":"Device-resident forward latency (ms)","unit":"raw","public_history":true,"protocol":{"en":"The full 11-architecture efficiency panel uses the first NeoPolyp cross-validation checkpoint at 256×256 resolution, batch one, on the same NVIDIA A100 MIG 40 GB partition. Timing includes 100 warm-ups, 1,000 timed forwards and five repetitions. The native millisecond latency excludes preprocessing, transfers, decoding, postprocessing and I/O. This cohort measures model execution cost; segmentation Dice is a separate measure. CENet is unavailable at 512 pixels after exceeding GPU memory.","zh":"完整11架构效率面板使用同一NVIDIA A100 MIG 40 GB分区、NeoPolyp首个交叉验证检查点、256×256分辨率和批量一。计时包含100次预热、1,000次前向及五次重复。毫秒延迟排除预处理、传输、解码、后处理和I/O。本面板衡量模型执行成本，分割Dice单独衡量。CENet在512像素时超出显存，未报告该条件结果。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.29677v1","status":"paper_baselines","published_at":"2026-08-30"},"higher_is_better":false,"saturation_excluded":true,"score_decimals":2,"scale_label":{"en":"Latency (ms)","zh":"延迟（毫秒）"},"model_count":11,"result_count":11,"results":[[10933,4.09,370,13243,13244],[13245,32.75,370,13243,13244],[13246,27.12,370,13243,13244],[13247,37.46,370,13243,13244],[13248,32.56,370,13243,13244],[13249,38.79,370,13243,13244],[13250,74.92,370,13243,13244],[13251,17.52,370,13243,13244],[13252,50.85,370,13243,13244],[13253,36.17,370,13243,13244],[13254,21.45,370,13243,13244]]},{"id":"medseg-model-latency-512-2026-paper","name":"MedSegBenchmarker · 512px inference (2026 paper)","group":"医疗","metric":"Device-resident forward latency (ms)","unit":"raw","public_history":true,"protocol":{"en":"The full 10-architecture efficiency panel uses the first NeoPolyp cross-validation checkpoint at 512×512 resolution, batch one, on the same NVIDIA A100 MIG 40 GB partition. Timing includes 100 warm-ups, 1,000 timed forwards and five repetitions. The native millisecond latency excludes preprocessing, transfers, decoding, postprocessing and I/O. This cohort measures model execution cost; segmentation Dice is a separate measure. CENet is unavailable at 512 pixels after exceeding GPU memory.","zh":"完整10架构效率面板使用同一NVIDIA A100 MIG 40 GB分区、NeoPolyp首个交叉验证检查点、512×512分辨率和批量一。计时包含100次预热、1,000次前向及五次重复。毫秒延迟排除预处理、传输、解码、后处理和I/O。本面板衡量模型执行成本，分割Dice单独衡量。CENet在512像素时超出显存，未报告该条件结果。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.29677v1","status":"paper_baselines","published_at":"2026-08-30"},"higher_is_better":false,"saturation_excluded":true,"score_decimals":2,"scale_label":{"en":"Latency (ms)","zh":"延迟（毫秒）"},"model_count":10,"result_count":10,"results":[[10933,13.17,370,13255,13244],[13245,57.46,370,13255,13244],[13246,56.55,370,13255,13244],[13247,39.27,370,13255,13244],[13248,32.58,370,13255,13244],[13249,41.5,370,13255,13244],[13251,24.93,370,13255,13244],[13252,56.88,370,13255,13244],[13253,41.64,370,13255,13244],[13254,42.54,370,13255,13244]]},{"id":"bixbench3-artifact-reproduction-2026-paper","name":"BixBench3 · biological study reproduction (2026 paper)","group":"数学与科学","metric":"Mean task artifact-reproduction score","unit":"score","public_history":true,"protocol":{"en":"All thirteen frontier models complete the same twenty biological reproduction tasks, producing 260 completed runs and 1,794 graded artifact evaluations. The native 0–1 task mean reflects reproduction of requested artifacts with equivalent biological meaning. Twelve printed Figure 3A values are visually transcribed; Gemini 3.1 Flash Lite’s zero is also explicitly stated as 0.00 in the abstract. Figure/bar coordinates and the original PDF crop are retained; scores are not estimated from bar heights.","zh":"完整十三个前沿模型完成同一二十个生物研究复现任务，共260次完成运行、1,794项产物评测。0–1原生任务均值反映所需产物是否复现相同生物学含义。图3A的十二个印刷数值经目视核对，Gemini 3.1 Flash Lite的零分同时由摘要明确的0.00确认。保留原图、柱序及PDF裁图，数值直接抄录标签。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.25286v2","status":"paper_baselines","published_at":"2026-08-26"},"score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":13,"result_count":13,"results":[[5004,0.48,370,13256,13257],[2977,0.47,370,13258,13257],[3654,0.46,370,13259,13257],[3479,0.46,370,13260,13257],[3581,0.43,370,13261,13257],[3415,0.42,370,13262,13257],[755,0.41,370,13263,13257],[3599,0.36,370,13264,13257],[3611,0.28,370,13265,13257],[3482,0.25,370,13266,13257],[3572,0.15,370,13267,13257],[325,0.02,370,13268,13257],[3497,0.0,370,13269,13257]]},{"id":"s3gym-chess-average-2026-paper","name":"S³Gym · Chess (2026 paper)","group":"Agent 与工具","metric":"Mean Chess environment reward","unit":"raw","public_history":true,"protocol":{"en":"All seven model checkpoints use the same Chess game and its fixed exploration/evaluation configuration. Both direct-history ICL and cross-episode summary memory are retained as configurations. The selected Avg is the actual environment reward mean on its native game scale, separate from self-judged scores, maximum reward and positive-area-above-baseline diagnostics. Different games are kept separate.","zh":"完整七检查点面板使用同一Chess游戏及其固定探索、评测设置。保留直接历史上下文和跨回合摘要记忆两种配置。所选Avg为环境实际奖励均值，保持该游戏原生单位；模型自评分、最大奖励及相对基线正面积诊断另列。不同游戏分别展示。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.31100v1","status":"paper_baselines","published_at":"2026-08-31"},"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Environment reward","zh":"环境奖励"},"model_count":7,"result_count":14,"results":[[353,0.007,370,13270,13271],[345,0.01,370,13270,13271],[4075,0.012,370,13270,13271],[766,0.009,370,13270,13271],[764,0.025,370,13270,13271],[3196,0.017,370,13270,13271],[2129,0.547,370,13270,13271],[353,0.005,370,13272,13271],[345,0.01,370,13272,13271],[4075,0.006,370,13272,13271],[766,0.074,370,13272,13271],[764,0.11,370,13272,13271],[3196,0.533,370,13272,13271],[2129,0.208,370,13272,13271]]},{"id":"s3gym-minesweeper-average-2026-paper","name":"S³Gym · Minesweeper (2026 paper)","group":"Agent 与工具","metric":"Mean Minesweeper environment reward","unit":"raw","public_history":true,"protocol":{"en":"All seven model checkpoints use the same Minesweeper game and its fixed exploration/evaluation configuration. Both direct-history ICL and cross-episode summary memory are retained as configurations. The selected Avg is the actual environment reward mean on its native game scale, separate from self-judged scores, maximum reward and positive-area-above-baseline diagnostics. Different games are kept separate.","zh":"完整七检查点面板使用同一Minesweeper游戏及其固定探索、评测设置。保留直接历史上下文和跨回合摘要记忆两种配置。所选Avg为环境实际奖励均值，保持该游戏原生单位；模型自评分、最大奖励及相对基线正面积诊断另列。不同游戏分别展示。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.31100v1","status":"paper_baselines","published_at":"2026-08-31"},"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Environment reward","zh":"环境奖励"},"model_count":7,"result_count":14,"results":[[353,0.044,370,13273,13271],[345,0.042,370,13273,13271],[4075,0.222,370,13273,13271],[766,0.063,370,13273,13271],[764,0.421,370,13273,13271],[3196,0.604,370,13273,13271],[2129,0.482,370,13273,13271],[353,0.013,370,13274,13271],[345,0.012,370,13274,13271],[4075,0.425,370,13274,13271],[766,0.438,370,13274,13271],[764,0.466,370,13274,13271],[3196,0.742,370,13274,13271],[2129,0.603,370,13274,13271]]},{"id":"s3gym-nullify-average-2026-paper","name":"S³Gym · Nullify (2026 paper)","group":"Agent 与工具","metric":"Mean Nullify environment reward","unit":"raw","public_history":true,"protocol":{"en":"All seven model checkpoints use the same Nullify game and its fixed exploration/evaluation configuration. Both direct-history ICL and cross-episode summary memory are retained as configurations. The selected Avg is the actual environment reward mean on its native game scale, separate from self-judged scores, maximum reward and positive-area-above-baseline diagnostics. Different games are kept separate.","zh":"完整七检查点面板使用同一Nullify游戏及其固定探索、评测设置。保留直接历史上下文和跨回合摘要记忆两种配置。所选Avg为环境实际奖励均值，保持该游戏原生单位；模型自评分、最大奖励及相对基线正面积诊断另列。不同游戏分别展示。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.31100v1","status":"paper_baselines","published_at":"2026-08-31"},"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Environment reward","zh":"环境奖励"},"model_count":7,"result_count":14,"results":[[353,0.03,370,13275,13271],[345,0.03,370,13275,13271],[4075,0.03,370,13275,13271],[766,0.03,370,13275,13271],[764,0.212,370,13275,13271],[3196,0.545,370,13275,13271],[2129,0.394,370,13275,13271],[353,0.0,370,13276,13271],[345,0.03,370,13276,13271],[4075,0.061,370,13276,13271],[766,0.091,370,13276,13271],[764,0.182,370,13276,13271],[3196,0.576,370,13276,13271],[2129,0.455,370,13276,13271]]},{"id":"s3gym-tetris-average-2026-paper","name":"S³Gym · Tetris (2026 paper)","group":"Agent 与工具","metric":"Mean Tetris environment reward","unit":"raw","public_history":true,"protocol":{"en":"All seven model checkpoints use the same Tetris game and its fixed exploration/evaluation configuration. Both direct-history ICL and cross-episode summary memory are retained as configurations. The selected Avg is the actual environment reward mean on its native game scale, separate from self-judged scores, maximum reward and positive-area-above-baseline diagnostics. Different games are kept separate.","zh":"完整七检查点面板使用同一Tetris游戏及其固定探索、评测设置。保留直接历史上下文和跨回合摘要记忆两种配置。所选Avg为环境实际奖励均值，保持该游戏原生单位；模型自评分、最大奖励及相对基线正面积诊断另列。不同游戏分别展示。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.31100v1","status":"paper_baselines","published_at":"2026-08-31"},"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Environment reward","zh":"环境奖励"},"model_count":7,"result_count":14,"results":[[353,0.03,370,13277,13271],[345,0.152,370,13277,13271],[4075,1.273,370,13277,13271],[766,0.121,370,13277,13271],[764,0.273,370,13277,13271],[3196,4.242,370,13277,13271],[2129,1.97,370,13277,13271],[353,0.03,370,13278,13271],[345,0.182,370,13278,13271],[4075,1.788,370,13278,13271],[766,0.152,370,13278,13271],[764,0.545,370,13278,13271],[3196,3.515,370,13278,13271],[2129,1.091,370,13278,13271]]},{"id":"s3gym-snake-average-2026-paper","name":"S³Gym · Snake (2026 paper)","group":"Agent 与工具","metric":"Mean Snake environment reward","unit":"raw","public_history":true,"protocol":{"en":"All seven model checkpoints use the same Snake game and its fixed exploration/evaluation configuration. Both direct-history ICL and cross-episode summary memory are retained as configurations. The selected Avg is the actual environment reward mean on its native game scale, separate from self-judged scores, maximum reward and positive-area-above-baseline diagnostics. Different games are kept separate.","zh":"完整七检查点面板使用同一Snake游戏及其固定探索、评测设置。保留直接历史上下文和跨回合摘要记忆两种配置。所选Avg为环境实际奖励均值，保持该游戏原生单位；模型自评分、最大奖励及相对基线正面积诊断另列。不同游戏分别展示。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.31100v1","status":"paper_baselines","published_at":"2026-08-31"},"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Environment reward","zh":"环境奖励"},"model_count":7,"result_count":14,"results":[[353,0.212,370,13279,13271],[345,0.515,370,13279,13271],[4075,4.0,370,13279,13271],[766,1.424,370,13279,13271],[764,1.576,370,13279,13271],[3196,9.061,370,13279,13271],[2129,8.455,370,13279,13271],[353,0.303,370,13280,13271],[345,0.545,370,13280,13271],[4075,6.727,370,13280,13271],[766,1.364,370,13280,13271],[764,1.273,370,13280,13271],[3196,10.697,370,13280,13271],[2129,10.242,370,13280,13271]]},{"id":"s3gym-pvz-average-2026-paper","name":"S³Gym · PvZ (2026 paper)","group":"Agent 与工具","metric":"Mean PvZ environment reward","unit":"raw","public_history":true,"protocol":{"en":"All seven model checkpoints use the same PvZ game and its fixed exploration/evaluation configuration. Both direct-history ICL and cross-episode summary memory are retained as configurations. The selected Avg is the actual environment reward mean on its native game scale, separate from self-judged scores, maximum reward and positive-area-above-baseline diagnostics. Different games are kept separate.","zh":"完整七检查点面板使用同一PvZ游戏及其固定探索、评测设置。保留直接历史上下文和跨回合摘要记忆两种配置。所选Avg为环境实际奖励均值，保持该游戏原生单位；模型自评分、最大奖励及相对基线正面积诊断另列。不同游戏分别展示。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.31100v1","status":"paper_baselines","published_at":"2026-08-31"},"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Environment reward","zh":"环境奖励"},"model_count":7,"result_count":14,"results":[[353,11.636,370,13281,13271],[345,16.909,370,13281,13271],[4075,26.333,370,13281,13271],[766,20.697,370,13281,13271],[764,28.242,370,13281,13271],[3196,44.033,370,13281,13271],[2129,44.697,370,13281,13271],[353,11.818,370,13282,13271],[345,14.727,370,13282,13271],[4075,26.424,370,13282,13271],[766,21.939,370,13282,13271],[764,19.667,370,13282,13271],[3196,33.5,370,13282,13271],[2129,43.879,370,13282,13271]]},{"id":"s3gym-trustevo-average-2026-paper","name":"S³Gym · TrustEvo (2026 paper)","group":"Agent 与工具","metric":"Mean TrustEvo environment reward","unit":"raw","public_history":true,"protocol":{"en":"All seven model checkpoints use the same TrustEvo game and its fixed exploration/evaluation configuration. Both direct-history ICL and cross-episode summary memory are retained as configurations. The selected Avg is the actual environment reward mean on its native game scale, separate from self-judged scores, maximum reward and positive-area-above-baseline diagnostics. Different games are kept separate.","zh":"完整七检查点面板使用同一TrustEvo游戏及其固定探索、评测设置。保留直接历史上下文和跨回合摘要记忆两种配置。所选Avg为环境实际奖励均值，保持该游戏原生单位；模型自评分、最大奖励及相对基线正面积诊断另列。不同游戏分别展示。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.31100v1","status":"paper_baselines","published_at":"2026-08-31"},"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Environment reward","zh":"环境奖励"},"model_count":7,"result_count":14,"results":[[353,8.242,370,13283,13271],[345,7.371,370,13283,13271],[4075,9.242,370,13283,13271],[766,9.008,370,13283,13271],[764,14.5,370,13283,13271],[3196,7.848,370,13283,13271],[2129,17.242,370,13283,13271],[353,13.947,370,13284,13271],[345,13.371,370,13284,13271],[4075,7.174,370,13284,13271],[766,8.72,370,13284,13271],[764,9.568,370,13284,13271],[3196,8.697,370,13284,13271],[2129,14.394,370,13284,13271]]},{"id":"sci-vbench-human-testmini-2026-paper","name":"Sci-VBench · scientific video generation (2026 paper)","group":"生成与创作","metric":"Human evaluation mean score","unit":"score","public_history":true,"protocol":{"en":"All sixteen published video generators use the same 150-example testmini split. The original human-evaluation mean on the 1–5 scale combines perceptual fidelity, scientific correctness, physical grounding and scientific conceptual consistency. It is separate from automatic scoring and the larger Full Avg cohort. Model release dates remain source metadata rather than benchmark release dates.","zh":"完整十六视频生成模型使用相同150例testmini划分。人类评测均值保留1–5原生量表，综合感知保真、科学正确性、物理依据与科学概念一致性。自动评分和更大测试集的Full Avg单列。模型发布日期保留为来源元数据，不当作基准首发日。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.09873v1","status":"paper_baselines","published_at":"2026-08-10"},"score_max":5,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 5","zh":"分数 / 5"},"model_count":16,"result_count":16,"results":[[13285,3.18,370,13286,13287],[7126,3.06,370,13286,13287],[7124,2.94,370,13286,13287],[8885,2.8,370,13286,13287],[13288,2.79,370,13286,13287],[13289,2.68,370,13286,13287],[13290,2.62,370,13286,13287],[13291,2.52,370,13286,13287],[7125,2.58,370,13286,13287],[9548,2.34,370,13286,13287],[9547,2.27,370,13286,13287],[8303,2.23,370,13286,13287],[8890,2.22,370,13286,13287],[10420,2.12,370,13286,13287],[13292,1.93,370,13286,13287],[7128,1.92,370,13286,13287]]},{"id":"sciencearena-sealed-four-contest-2026-paper","name":"ScienceArena · sealed science contests (2026 paper)","group":"数学与科学","metric":"Four-contest normalized macro score","unit":"score","public_history":true,"protocol":{"en":"All fourteen routes complete the same four sealed 2026 contests: NBPhO, USAPhO, IPhO and IChO. The source records 153 linked turns in 28 whole problem sequences, with complete aggregates for every route. Contest scores are normalized by their explicit maxima then equally averaged on a native 0–100 partial-credit scale. V routes receive frozen official PNG pages; T routes receive the frozen question-only text. Input assignment is explicit for each result, and the thirteen-benchmark mixture is separate.","zh":"完整十四路由面板完成相同四项2026年封闭科学竞赛：NBPhO、USAPhO、IPhO和IChO，共28个完整题目序列的153个关联子问，每路由汇总齐全。各竞赛分数按原满分归一化后等权平均，保留0–100原生部分给分量表。V路由接收冻结的官方PNG题页，T路由接收冻结的纯问题文本，各行明确输入分配。十三基准混合均值另列。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.30517v1","status":"paper_baselines","published_at":"2026-08-31"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":14,"result_count":14,"results":[[3424,85.56,370,13293,13294],[13295,71.03,370,13293,13294],[3412,88.47,370,13293,13294],[3184,68.79,370,13293,13294],[3186,86.89,370,13296,13294],[3189,82.63,370,13293,13294],[3495,69.61,370,13296,13294],[3578,79.86,370,13293,13294],[3593,63.81,370,13296,13294],[3196,93.31,370,13293,13294],[12184,87.76,370,13296,13294],[13297,86.62,370,13296,13294],[6732,89.54,370,13293,13294],[3202,85.16,370,13296,13294]]},{"id":"shadowbench-semantic-autoformalization-2026-paper","name":"ShadowBench · semantic autoformalization (2026 paper)","group":"数学与科学","metric":"Strict SA-Pass rate","unit":"%","public_history":true,"protocol":{"en":"The complete twenty-seven-system panel formalizes the same 178 expert-verified problems across three difficulty levels. Strict SA-Pass requires compilation and semantic faithfulness; compile rate and soft acceptance are separate. Agent frameworks, optional Numina access and specialized autoformalizer/prover pairs remain explicit configurations. Invisible alignment-padding zeros are removed from the source HTML before exact numeric parsing; true zero scores are retained.","zh":"完整二十七系统面板形式化同一178道人类专家核验问题，覆盖三个难度。严格SA-Pass要求编译与语义忠实同时满足；单纯编译率和宽松接受率另列。智能体框架、Numina工具及专门的自动形式化、证明器组合均作为配置保留。解析前仅移除原HTML用于对齐的不可见填充零，真实零分保留。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.29270v3","status":"paper_baselines","published_at":"2026-08-29"},"model_count":23,"result_count":27,"results":[[13298,0.0,370,13299,13300],[13298,0.09,370,13301,13300],[13302,0.027999999999999997,370,13299,13300],[13302,0.11199999999999999,370,13301,13300],[13303,0.006,370,13299,13300],[13303,0.006,370,13301,13300],[13304,0.022000000000000002,370,13299,13300],[13304,0.10099999999999999,370,13301,13300],[2261,0.0,370,13305,13300],[3572,0.0,370,13305,13300],[325,0.0,370,13305,13300],[3184,0.0,370,13305,13300],[5449,0.0,370,13305,13300],[7626,0.0,370,13305,13300],[3412,0.0,370,13305,13300],[2055,0.0,370,13305,13300],[3419,0.0,370,13305,13300],[5343,0.0,370,13305,13300],[13306,0.0,370,13305,13300],[4425,0.0,370,13305,13300],[7179,0.0,370,13305,13300],[13307,0.0,370,13305,13300],[13308,0.0,370,13305,13300],[13309,0.0,370,13305,13300],[13310,0.0,370,13311,13300],[13312,0.0,370,13311,13300],[13313,0.0,370,13311,13300]]},{"id":"healmed-headqa-mcqa-2026-paper","name":"HealMed · HeadQA multilingual MCQA (2026 paper)","group":"医疗","metric":"Nine-language mean accuracy","unit":"%","public_history":true,"protocol":{"en":"All fourteen configurations answer the same expert-reviewed HeadQA MCQA items in nine languages. The source Avg is the equal-language accuracy mean. Twenty-three medical experts constructed the benchmark; every target-language instance was reviewed and revised by two bilingual experts. Thinking mode is a configuration, and specialist fine-tuning does not create a new base-model count. NLI and open-ended judge scores are separate.","zh":"完整十四配置回答同一组经专家核验的HeadQA九语言医学选择题。保留原表九语言准确率均值。二十三名医学专家参与构建，每个目标语言样本均由两位双语专家复核并修改。思考模式作为配置，专业微调不增加底模计数。自然语言推断和开放问答评分另列。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.19981v1","status":"paper_baselines","published_at":"2026-08-20"},"model_count":14,"result_count":14,"results":[[6133,0.9289000000000001,370,13314,13315],[3184,0.8993000000000001,370,13314,13315],[5929,0.9244,370,13314,13315],[3194,0.9215000000000001,370,13314,13315],[5789,0.9067000000000001,370,13314,13315],[1573,0.7585,370,13314,13315],[3029,0.7422,370,13314,13315],[13316,0.7392,370,13314,13315],[1651,0.6756,370,13314,13315],[786,0.6103999999999999,370,13314,13315],[13317,0.8029999999999999,370,13314,13315],[8168,0.7807,370,13314,13315],[9409,0.7289,370,13314,13315],[13318,0.4518,370,13314,13315]]},{"id":"healmed-medqa-mcqa-2026-paper","name":"HealMed · MedQA multilingual MCQA (2026 paper)","group":"医疗","metric":"Nine-language mean accuracy","unit":"%","public_history":true,"protocol":{"en":"All fourteen configurations answer the same expert-reviewed MedQA MCQA items in nine languages. The source Avg is the equal-language accuracy mean. Twenty-three medical experts constructed the benchmark; every target-language instance was reviewed and revised by two bilingual experts. Thinking mode is a configuration, and specialist fine-tuning does not create a new base-model count. NLI and open-ended judge scores are separate.","zh":"完整十四配置回答同一组经专家核验的MedQA九语言医学选择题。保留原表九语言准确率均值。二十三名医学专家参与构建，每个目标语言样本均由两位双语专家复核并修改。思考模式作为配置，专业微调不增加底模计数。自然语言推断和开放问答评分另列。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.19981v1","status":"paper_baselines","published_at":"2026-08-20"},"model_count":14,"result_count":14,"results":[[6133,0.9304000000000001,370,13319,13315],[3184,0.9422,370,13319,13315],[5929,0.96,370,13319,13315],[3194,0.92,370,13319,13315],[5789,0.8756,370,13319,13315],[1573,0.6829999999999999,370,13319,13315],[3029,0.6015,370,13319,13315],[13316,0.7467,370,13319,13315],[1651,0.6489,370,13319,13315],[786,0.5289,370,13319,13315],[13317,0.8148000000000001,370,13319,13315],[8168,0.7881,370,13319,13315],[9409,0.6489,370,13319,13315],[13318,0.4119,370,13319,13315]]},{"id":"healmed-medexpqa-mcqa-2026-paper","name":"HealMed · MedExpQA multilingual MCQA (2026 paper)","group":"医疗","metric":"Nine-language mean accuracy","unit":"%","public_history":true,"protocol":{"en":"All fourteen configurations answer the same expert-reviewed MedExpQA MCQA items in nine languages. The source Avg is the equal-language accuracy mean. Twenty-three medical experts constructed the benchmark; every target-language instance was reviewed and revised by two bilingual experts. Thinking mode is a configuration, and specialist fine-tuning does not create a new base-model count. NLI and open-ended judge scores are separate.","zh":"完整十四配置回答同一组经专家核验的MedExpQA九语言医学选择题。保留原表九语言准确率均值。二十三名医学专家参与构建，每个目标语言样本均由两位双语专家复核并修改。思考模式作为配置，专业微调不增加底模计数。自然语言推断和开放问答评分另列。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.19981v1","status":"paper_baselines","published_at":"2026-08-20"},"model_count":14,"result_count":14,"results":[[6133,0.9185,370,13320,13315],[3184,0.9096,370,13320,13315],[5929,0.9052,370,13320,13315],[3194,0.9140999999999999,370,13320,13315],[5789,0.8933,370,13320,13315],[1573,0.7719,370,13320,13315],[3029,0.6919,370,13320,13315],[13316,0.7273999999999999,370,13320,13315],[1651,0.7156,370,13320,13315],[786,0.5067,370,13320,13315],[13317,0.8015000000000001,370,13320,13315],[8168,0.7615000000000001,370,13320,13315],[9409,0.7067,370,13320,13315],[13318,0.4133,370,13320,13315]]},{"id":"healmed-mmlu-pro-mcqa-2026-paper","name":"HealMed · MMLU-Pro multilingual MCQA (2026 paper)","group":"医疗","metric":"Nine-language mean accuracy","unit":"%","public_history":true,"protocol":{"en":"All fourteen configurations answer the same expert-reviewed MMLU-Pro MCQA items in nine languages. The source Avg is the equal-language accuracy mean. Twenty-three medical experts constructed the benchmark; every target-language instance was reviewed and revised by two bilingual experts. Thinking mode is a configuration, and specialist fine-tuning does not create a new base-model count. NLI and open-ended judge scores are separate.","zh":"完整十四配置回答同一组经专家核验的MMLU-Pro九语言医学选择题。保留原表九语言准确率均值。二十三名医学专家参与构建，每个目标语言样本均由两位双语专家复核并修改。思考模式作为配置，专业微调不增加底模计数。自然语言推断和开放问答评分另列。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.19981v1","status":"paper_baselines","published_at":"2026-08-20"},"model_count":14,"result_count":14,"results":[[6133,0.7689,370,13321,13315],[3184,0.7095999999999999,370,13321,13315],[5929,0.7037,370,13321,13315],[3194,0.7215,370,13321,13315],[5789,0.7141,370,13321,13315],[1573,0.6089,370,13321,13315],[3029,0.4667,370,13321,13315],[13316,0.5540999999999999,370,13321,13315],[1651,0.47259999999999996,370,13321,13315],[786,0.3778,370,13321,13315],[13317,0.5807,370,13321,13315],[8168,0.6133,370,13321,13315],[9409,0.5067,370,13321,13315],[13318,0.28,370,13321,13315]]},{"id":"scifigbench-description-mqm-2026-paper","name":"SciFigBench · scientific figure descriptions (2026 paper)","group":"视觉与视频","metric":"Baseline description MQM score","unit":"score","public_history":true,"protocol":{"en":"All eight checkpoints describe the same 250 expert-annotated figures with original captions. The native 0–100 MQM score evaluates description quality with weighted error penalties. Perceptual transformations, adversarial captions and selective blur are separate conditions. Exact deployed identifiers are mapped from the paper’s configuration table.","zh":"完整八检查点描述同一250张专家标注科学图，并接收原始图注。0–100原生MQM按错误加权扣分评价描述质量。视觉变换、对抗图注与选择性模糊为不同条件。部署标识按论文配置表对应。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.13267v1","status":"paper_baselines","published_at":"2026-08-13"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":8,"result_count":8,"results":[[340,91.6,370,13322,13323],[3482,90.2,370,13322,13323],[13324,81.4,370,13322,13323],[13325,80.8,370,13322,13323],[13326,78.9,370,13322,13323],[13327,74.4,370,13322,13323],[11955,69.1,370,13322,13323],[13328,62.2,370,13322,13323]]},{"id":"scifigbench-capability-2026-paper","name":"SciFigBench · figure-grounded reasoning (2026 paper)","group":"视觉与视频","metric":"Overall capability-question accuracy","unit":"%","public_history":true,"protocol":{"en":"All eight checkpoints answer the same 1,000 figure-grounded questions covering counting, computation, comparison and pattern analysis across 250 figures. The original Overall accuracy is retained. Questions seeded with GPT-4o were post-edited by humans, and outputs were judged against references with a cross-judge Mistral Large 3 check. Model identifiers follow the published configuration table.","zh":"完整八检查点回答同一250图对应的1,000道问题，覆盖计数、计算、比较和模式分析。保留原始Overall准确率。GPT-4o初拟问题后由人类编辑，输出按参考答案评分，并以Mistral Large 3作跨评委检查。模型标识按论文配置表对应。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.13267v1","status":"paper_baselines","published_at":"2026-08-13"},"model_count":8,"result_count":8,"results":[[3482,0.81,370,13329,13330],[340,0.784,370,13329,13330],[13325,0.584,370,13329,13330],[13326,0.51,370,13329,13330],[13324,0.485,370,13329,13330],[13327,0.406,370,13329,13330],[11955,0.272,370,13329,13330],[13328,0.086,370,13329,13330]]},{"id":"snapbench-corruption-overall-2026-paper","name":"SnapBench · robust multimodal retrieval (2026 paper)","group":"视觉与视频","metric":"Overall Recall@1","unit":"%","public_history":true,"protocol":{"en":"All sixteen multimodal retrievers evaluate the same 1,145 queries and entity-aware galleries under the paper’s paired corruption protocol. The source Overall Recall@1 uses joint image-text retrieval and retains the original percentage unit. The benchmark spans 53 controlled corruptions; clean-only scores and separate fusion-method ablations remain distinct.","zh":"完整十六多模态检索器使用同一1,145条查询及按实体组织的候选库，按论文配对噪声协议评测。保留图文联合检索Overall Recall@1的原始百分比单位。基准覆盖53种受控扰动，纯净输入分数与融合算法消融单列。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.29607v1","status":"paper_baselines","published_at":"2026-08-30"},"model_count":16,"result_count":16,"results":[[13331,0.303,370,13332,13333],[13334,0.43799999999999994,370,13332,13333],[13335,0.54,370,13332,13333],[13336,0.614,370,13332,13333],[13337,0.537,370,13332,13333],[13338,0.5539999999999999,370,13332,13333],[13339,0.5539999999999999,370,13332,13333],[11725,0.599,370,13332,13333],[13340,0.6,370,13332,13333],[11724,0.612,370,13332,13333],[13341,0.622,370,13332,13333],[11726,0.623,370,13332,13333],[13342,0.6459999999999999,370,13332,13333],[13343,0.72,370,13332,13333],[13344,0.736,370,13332,13333],[13345,0.748,370,13332,13333]]},{"id":"toolrobustbench-overall-2026-paper","name":"ToolRobustBench · perturbed tool use (2026 paper)","group":"Agent 与工具","metric":"Overall perturbed end-to-end success rate","unit":"%","public_history":true,"protocol":{"en":"All seven checkpoints evaluate the same deterministic local-tool perturbation design across four families and fourteen subtypes. Success requires correct tool selection, schema, bound arguments, executed output and final result, with runtime status/retryability checks where applicable. The source Overall rate is retained on its original 0–1 scale; clean success, Heavy Macro and stage diagnostics are separate.","zh":"完整七检查点评测同一确定性本地工具扰动设计，覆盖四类扰动、十四子类。成功需同时满足工具选择、参数结构、参数绑定、执行输出及最终结果；适用时还核对运行状态和可重试性。Overall成功率保留0–1原始单位，纯净条件、Heavy Macro和分阶段诊断单列。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.23635v1","status":"paper_baselines","published_at":"2026-08-23"},"model_count":7,"result_count":7,"results":[[4278,0.766,370,13346,13347],[3209,0.749,370,13346,13347],[5160,0.712,370,13346,13347],[13348,0.706,370,13346,13347],[457,0.692,370,13346,13347],[1321,0.669,370,13346,13347],[4164,0.664,370,13346,13347]]},{"id":"take85-simple-2026-paper","name":"TAKE 85 · simple directorial intent (2026 paper)","group":"视觉与视频","metric":"Checklist score","unit":"score","public_history":true,"protocol":{"en":"All ten checkpoints evaluate the same 398-film directorial-intent benchmark with every provided modality: visual captions, sampled keyframes, subtitles and audio analysis. The simple score is the published 0–100 checklist mean over three overall-intent questions. It is a rubric score; modality ablations are separate.","zh":"完整十检查点评测同一398部短片的导演意图基准，使用全部提供模态：视觉描述、抽样关键帧、字幕及音频分析。simple分数保留原始0–100清单评分均值，覆盖三个整体意图问题。该数值是评分细则分数，模态消融单列。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.30068v2","status":"paper_baselines","published_at":"2026-08-30"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":10,"result_count":10,"results":[[2687,2.2,370,13349,13350],[4969,5.8,370,13349,13350],[13351,4.4,370,13349,13350],[2654,4.6,370,13349,13350],[4070,7.9,370,13349,13350],[13352,5.7,370,13349,13350],[13353,5.7,370,13349,13350],[2584,8.5,370,13349,13350],[13354,8.6,370,13349,13350],[13355,17.7,370,13349,13350]]},{"id":"take85-specific-2026-paper","name":"TAKE 85 · specific directorial intent (2026 paper)","group":"视觉与视频","metric":"Checklist score","unit":"score","public_history":true,"protocol":{"en":"All ten checkpoints evaluate the same 398-film directorial-intent benchmark with every provided modality: visual captions, sampled keyframes, subtitles and audio analysis. The specific score is the published 0–100 checklist mean over nine fine-grained general/visual/audio questions. It is a rubric score; modality ablations are separate.","zh":"完整十检查点评测同一398部短片的导演意图基准，使用全部提供模态：视觉描述、抽样关键帧、字幕及音频分析。specific分数保留原始0–100清单评分均值，覆盖九个细分整体、视觉及音频问题。该数值是评分细则分数，模态消融单列。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.30068v2","status":"paper_baselines","published_at":"2026-08-30"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":10,"result_count":10,"results":[[2687,3.9,370,13356,13350],[4969,7.1,370,13356,13350],[13351,4.3,370,13356,13350],[2654,4.4,370,13356,13350],[4070,7.8,370,13356,13350],[13352,6.1,370,13356,13350],[13353,5.5,370,13356,13350],[2584,11.1,370,13356,13350],[13354,7.8,370,13356,13350],[13355,18.1,370,13356,13350]]},{"id":"spieval-gold-tool-accuracy-2026-paper","name":"SPIEval · personal-information tool use (2026 paper)","group":"Agent 与工具","metric":"Overall gold-parameter task accuracy","unit":"%","public_history":true,"protocol":{"en":"All eighteen effort configurations from nine model checkpoints use the same SPIEval personal-information tasks and simulated applications, with three independent runs and a fifty-turn cap. Success is checked automatically against human-annotated execution tools and accepted parameter values, including unordered multi-tool tasks and type-only checks for unconstrained parameters. The exact PDF main-font mean is separated from its smaller-font standard deviation; HTML conversion concatenated those numbers.","zh":"完整九检查点的十八推理强度配置使用SPIEval相同个人信息任务及模拟应用，独立运行三次，每任务上限五十轮。自动核对人类标注的执行工具及可接受参数，多工具任务允许调换顺序，不受约束参数仅核对类型。从PDF主字号提取精确均值，将其与小字号标准差分开；原HTML转换曾把两者数字相连。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.10692v1","status":"paper_baselines","published_at":"2026-08-11"},"model_count":10,"result_count":18,"results":[[3077,0.573,370,13357,13358],[3079,0.531,370,13359,13358],[12045,0.523,370,13360,13358],[13361,0.465,370,13362,13358],[13363,0.457,370,13364,13358],[13365,0.363,370,13366,13358],[13367,0.36,370,13368,13358],[13369,0.35200000000000004,370,13370,13358],[13371,0.34299999999999997,370,13372,13358],[13373,0.34299999999999997,370,13374,13358],[13375,0.335,370,13376,13358],[13377,0.33299999999999996,370,13378,13358],[13379,0.292,370,13380,13358],[13381,0.285,370,13382,13358],[13383,0.245,370,13384,13358],[13385,0.239,370,13386,13358],[13387,0.185,370,13388,13358],[13389,0.16399999999999998,370,13390,13358]]},{"id":"shoprank-gold-structured-2026-paper","name":"ShopRank-Bench ·gold structured (2026 paper)","group":"知识与推理","metric":"Three-family unanimous preference accuracy","unit":"%","public_history":true,"protocol":{"en":"Nine reranker/checkpoint configurations cover six independent base models: Qwen3-Reranker0.6B/4B/8B, Jina-m0, BGE-v2-m3 and BGE-large. Aligned/distilled ZooWork variants remain configurations of the corresponding bases. Every model sees the same 1843 gold-tier shopping pairs in the selected rendering. Three judge families agree in both candidate orders; ties/conflicting labels are excluded. Queries are disjoint from training; DeepSeekV4Pro judges benchmark pairs but never supervises training. Scores measure agreement with this synthetic preference reference. We select only the all-three-judge gold tier; silver/bronze and token-cost comparisons are separate.","zh":"九条重排模型与配置覆盖Qwen3-Reranker的0.6B、4B、8B三权重，以及Jina-m0、BGE-v2-m3、BGE-large，共六种独立基模型；ZooWork对齐与蒸馏版本沿用对应基模型身份。所选文本格式下，每模型评测相同1843条金标购物偏好对，三种评审模型家族在正反候选顺序中均同意标签；平局与冲突已移除。测试查询与训练分离，DeepSeekV4Pro只参与测试标签，不监督训练。正确率表示与该合成偏好参考一致。仅录三家一致的gold组。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.31002v1","published_at":"2026-09-25","status":"paper_baselines"},"model_count":6,"result_count":9,"results":[[7276,0.965,370,13391,13392],[7275,0.963,370,13393,13392],[7276,0.932,370,13394,13392],[13395,0.9079999999999999,370,13396,13392],[7275,0.927,370,13397,13392],[7274,0.912,370,13398,13392],[7274,0.857,370,13399,13392],[13400,0.8490000000000001,370,13401,13392],[13402,0.8240000000000001,370,13403,13392]]},{"id":"shoprank-gold-natural-language-2026-paper","name":"ShopRank-Bench ·gold natural-language (2026 paper)","group":"知识与推理","metric":"Three-family unanimous preference accuracy","unit":"%","public_history":true,"protocol":{"en":"Nine reranker/checkpoint configurations cover six independent base models: Qwen3-Reranker0.6B/4B/8B, Jina-m0, BGE-v2-m3 and BGE-large. Aligned/distilled ZooWork variants remain configurations of the corresponding bases. Every model sees the same 1843 gold-tier shopping pairs in the selected rendering. Three judge families agree in both candidate orders; ties/conflicting labels are excluded. Queries are disjoint from training; DeepSeekV4Pro judges benchmark pairs but never supervises training. Scores measure agreement with this synthetic preference reference. We select only the all-three-judge gold tier; silver/bronze and token-cost comparisons are separate.","zh":"九条重排模型与配置覆盖Qwen3-Reranker的0.6B、4B、8B三权重，以及Jina-m0、BGE-v2-m3、BGE-large，共六种独立基模型；ZooWork对齐与蒸馏版本沿用对应基模型身份。所选文本格式下，每模型评测相同1843条金标购物偏好对，三种评审模型家族在正反候选顺序中均同意标签；平局与冲突已移除。测试查询与训练分离，DeepSeekV4Pro只参与测试标签，不监督训练。正确率表示与该合成偏好参考一致。仅录三家一致的gold组。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.31002v1","published_at":"2026-09-25","status":"paper_baselines"},"model_count":6,"result_count":9,"results":[[7276,0.95,370,13404,13405],[7275,0.9540000000000001,370,13406,13405],[7276,0.919,370,13407,13405],[13395,0.888,370,13408,13405],[7275,0.9209999999999999,370,13409,13405],[7274,0.912,370,13410,13405],[13400,0.858,370,13411,13405],[13402,0.856,370,13412,13405],[7274,0.845,370,13413,13405]]},{"id":"eyevqa-total-2026-paper","name":"EyeVQA (2026 paper)","group":"医疗","metric":"Normalized multimodal ophthalmic composite","unit":"score","public_history":true,"protocol":{"en":"Twelve clearly identified VLM checkpoints share the same 20000 zero-shot QA items from 21 ophthalmic datasets. Reference answers derive deterministically from clinical labels, segmentations andlandmarks. The source Total combines seven normalized task metrics: SC/TF exact match, MS/VS set intersection-over-union, ranking Kendall tau remapped from minus 1..1 to 0..100, boxIoU, andpoint accuracy at normalized distance0.05. Invalid/refused responses stay in the denominator as zero. The composite remains native0–100 rather than percent-correct. Two Gemini rows have inconsistent Flash versusFlash-Lite labels between the main table and prose and are excluded pending identity clarification; no per-task population is inferred from the 20000total.","zh":"十二个身份清楚的VLM权重以零样本方式评测相同20000条眼科问答，来自21个数据集，参考答案由临床标签、分割和关键点确定性导出。原表Total组合七种归一化指标：单选和判断正确率、多选集合交并比、映射至0–100的排序Kendall相关、框IoU，以及归一化距离0.05内的点定位正确率。无效与拒答保留分母并记零。总分保留原生0–100综合刻度。两条Gemini在表格和正文存在Flash与Flash-Lite命名冲突，待明确后再收录；不由20000总题数推算子任务题数。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.32352v1","published_at":"2026-09-26","status":"paper_baselines"},"score_min":0,"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Native score / 100","zh":"原生分数 / 100"},"score_decimals":2,"model_count":12,"result_count":12,"results":[[13414,39.3,370,13415,13416],[13417,49.5,370,13415,13416],[6241,45.6,370,13415,13416],[7853,54.1,370,13415,13416],[13418,59.3,370,13415,13416],[13419,62.8,370,13415,13416],[8173,50.9,370,13415,13416],[4970,49.9,370,13415,13416],[8524,51.5,370,13415,13416],[2771,41.0,370,13415,13416],[2754,47.1,370,13415,13416],[6342,48.9,370,13415,13416]]},{"id":"codeaction-attempt-success-2026-paper","name":"CodeActionBench ·attempt-success (2026 paper)","group":"Agent 与工具","metric":"Physical task attempt success","unit":"%","public_history":true,"protocol":{"en":"Nine model+harness rows cover eight independent backbones on 25 fixed robot manipulation instances with identical scene seeds and per-task tool-call/simulation-time budgets. Three attempts per task yield75 attempts per configuration. This metric uses denominator75: all attempts for success, or 25 tasks with at least one success for coverage. A hidden physical-outcome verifier scores final task predicates, including budget/runtime failures. Reference solutions establish all 25 tasks achievable. Opus reference and ClaudeCode share one backbone; AstraCodex and other vendor/native reasoning limits remain explicit configurations. Main results are fully specified paper baselines while the release of logs/code is planned.","zh":"九条模型与harness结果覆盖八种独立权重，共同评测25个固定机器人操作实例，场景种子及每任务工具调用和模拟时间预算一致，每题三次、每配置75次尝试。本指标分母为75：成功率按全部尝试计，三次覆盖率按至少成功一次的25个任务计。隐藏物理结果验证器检查终态判据，预算耗尽和运行错误仍计入；参考方案验证全部25题可完成。Opus两种harness共享同一权重，Astra的Codex及其余原生推理限制保留为配置。论文协议完整，日志和代码仍计划公开。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.33807v1","published_at":"2026-09-27","status":"paper_baselines"},"model_count":8,"result_count":9,"results":[[13420,0.7333333333333333,370,13421,13422],[755,0.49333333333333335,370,13423,13422],[755,0.4533333333333333,370,13424,13422],[13425,0.2,370,13426,13422],[13427,0.16,370,13428,13422],[13429,0.14666666666666667,370,13430,13422],[13431,0.12,370,13432,13422],[13433,0.12,370,13434,13422],[13435,0.02666666666666667,370,13436,13422]]},{"id":"codeaction-task-coverage-at3-2026-paper","name":"CodeActionBench ·task-coverage-at3 (2026 paper)","group":"Agent 与工具","metric":"Task coverage after three attempts","unit":"%","public_history":true,"protocol":{"en":"Nine model+harness rows cover eight independent backbones on 25 fixed robot manipulation instances with identical scene seeds and per-task tool-call/simulation-time budgets. Three attempts per task yield75 attempts per configuration. This metric uses denominator25: all attempts for success, or 25 tasks with at least one success for coverage. A hidden physical-outcome verifier scores final task predicates, including budget/runtime failures. Reference solutions establish all 25 tasks achievable. Opus reference and ClaudeCode share one backbone; AstraCodex and other vendor/native reasoning limits remain explicit configurations. Main results are fully specified paper baselines while the release of logs/code is planned.","zh":"九条模型与harness结果覆盖八种独立权重，共同评测25个固定机器人操作实例，场景种子及每任务工具调用和模拟时间预算一致，每题三次、每配置75次尝试。本指标分母为25：成功率按全部尝试计，三次覆盖率按至少成功一次的25个任务计。隐藏物理结果验证器检查终态判据，预算耗尽和运行错误仍计入；参考方案验证全部25题可完成。Opus两种harness共享同一权重，Astra的Codex及其余原生推理限制保留为配置。论文协议完整，日志和代码仍计划公开。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.33807v1","published_at":"2026-09-27","status":"paper_baselines"},"model_count":8,"result_count":9,"results":[[13420,0.88,370,13421,13422],[755,0.76,370,13423,13422],[755,0.56,370,13424,13422],[13425,0.32,370,13426,13422],[13427,0.36,370,13428,13422],[13429,0.24,370,13430,13422],[13431,0.24,370,13432,13422],[13433,0.2,370,13434,13422],[13435,0.08,370,13436,13422]]},{"id":"asyncode-single-2026-paper","name":"AsynCodeBench ·Single (2026 paper)","group":"编程","metric":"Full task-and-dependency success","unit":"%","public_history":true,"protocol":{"en":"Ten distinct model checkpoints execute the same 19 repository tasks under the selected execution protocol. AppendixI defines strict final success as complete repository test pass AND full dependency resolution (ADPR1), not partial test credit. Original cells provide successful-task count/19 and a rounded percentage; we derive the exact ratio from the integer count. The19 tasks contain52 preregistered cross-agent dependencies with executable checkers. Single solves the whole task; Serial hands off in order; Async-Private isolates concurrent specialists; read-only/writable managers differ in declared repair authority. Protocols are kept separate. Incomplete token records affect cost only and are excluded from this outcome metric.","zh":"十个独立模型权重在所选执行协议下完成相同19个仓库任务。附录I把严格最终成功定义为仓库测试全部通过且依赖全部解决（ADPR为1），不含部分测试分。原单元同时给成功题数/19和四舍五入百分率，本包从整数题数计算精确比例。19题共52条预注册、可执行检查的跨Agent依赖。Single独立解整题，Serial顺序交接，Async-Private隔离并行专家，两类manager按只读或可写权限区分，各协议分组。缺失token记录只影响成本，不改变本任务结果。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.32662v1","published_at":"2026-09-26","status":"paper_baselines"},"model_count":10,"result_count":10,"results":[[4070,0.10526315789473684,370,13437,13438],[8655,0.0,370,13437,13438],[13439,0.0,370,13437,13438],[2584,0.0,370,13437,13438],[4966,0.0,370,13437,13438],[5479,0.05263157894736842,370,13437,13438],[5803,0.15789473684210525,370,13437,13438],[3763,0.3157894736842105,370,13437,13438],[5300,0.7368421052631579,370,13437,13438],[6293,0.7894736842105263,370,13437,13438]]},{"id":"asyncode-serial-2026-paper","name":"AsynCodeBench ·Serial (2026 paper)","group":"编程","metric":"Full task-and-dependency success","unit":"%","public_history":true,"protocol":{"en":"Ten distinct model checkpoints execute the same 19 repository tasks under the selected execution protocol. AppendixI defines strict final success as complete repository test pass AND full dependency resolution (ADPR1), not partial test credit. Original cells provide successful-task count/19 and a rounded percentage; we derive the exact ratio from the integer count. The19 tasks contain52 preregistered cross-agent dependencies with executable checkers. Single solves the whole task; Serial hands off in order; Async-Private isolates concurrent specialists; read-only/writable managers differ in declared repair authority. Protocols are kept separate. Incomplete token records affect cost only and are excluded from this outcome metric.","zh":"十个独立模型权重在所选执行协议下完成相同19个仓库任务。附录I把严格最终成功定义为仓库测试全部通过且依赖全部解决（ADPR为1），不含部分测试分。原单元同时给成功题数/19和四舍五入百分率，本包从整数题数计算精确比例。19题共52条预注册、可执行检查的跨Agent依赖。Single独立解整题，Serial顺序交接，Async-Private隔离并行专家，两类manager按只读或可写权限区分，各协议分组。缺失token记录只影响成本，不改变本任务结果。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.32662v1","published_at":"2026-09-26","status":"paper_baselines"},"model_count":10,"result_count":10,"results":[[4070,0.0,370,13440,13438],[8655,0.0,370,13440,13438],[13439,0.0,370,13440,13438],[2584,0.05263157894736842,370,13440,13438],[4966,0.0,370,13440,13438],[5479,0.10526315789473684,370,13440,13438],[5803,0.0,370,13440,13438],[3763,0.2631578947368421,370,13440,13438],[5300,0.42105263157894735,370,13440,13438],[6293,0.47368421052631576,370,13440,13438]]},{"id":"asyncode-async-private-2026-paper","name":"AsynCodeBench ·Async-Private (2026 paper)","group":"编程","metric":"Full task-and-dependency success","unit":"%","public_history":true,"protocol":{"en":"Ten distinct model checkpoints execute the same 19 repository tasks under the selected execution protocol. AppendixI defines strict final success as complete repository test pass AND full dependency resolution (ADPR1), not partial test credit. Original cells provide successful-task count/19 and a rounded percentage; we derive the exact ratio from the integer count. The19 tasks contain52 preregistered cross-agent dependencies with executable checkers. Single solves the whole task; Serial hands off in order; Async-Private isolates concurrent specialists; read-only/writable managers differ in declared repair authority. Protocols are kept separate. Incomplete token records affect cost only and are excluded from this outcome metric.","zh":"十个独立模型权重在所选执行协议下完成相同19个仓库任务。附录I把严格最终成功定义为仓库测试全部通过且依赖全部解决（ADPR为1），不含部分测试分。原单元同时给成功题数/19和四舍五入百分率，本包从整数题数计算精确比例。19题共52条预注册、可执行检查的跨Agent依赖。Single独立解整题，Serial顺序交接，Async-Private隔离并行专家，两类manager按只读或可写权限区分，各协议分组。缺失token记录只影响成本，不改变本任务结果。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.32662v1","published_at":"2026-09-26","status":"paper_baselines"},"model_count":10,"result_count":10,"results":[[4070,0.0,370,13441,13438],[8655,0.0,370,13441,13438],[13439,0.0,370,13441,13438],[2584,0.0,370,13441,13438],[4966,0.0,370,13441,13438],[5479,0.0,370,13441,13438],[5803,0.0,370,13441,13438],[3763,0.10526315789473684,370,13441,13438],[5300,0.3684210526315789,370,13441,13438],[6293,0.3684210526315789,370,13441,13438]]},{"id":"asyncode-async-manager-ro-2026-paper","name":"AsynCodeBench ·Async-Manager-RO (2026 paper)","group":"编程","metric":"Full task-and-dependency success","unit":"%","public_history":true,"protocol":{"en":"Ten distinct model checkpoints execute the same 19 repository tasks under the selected execution protocol. AppendixI defines strict final success as complete repository test pass AND full dependency resolution (ADPR1), not partial test credit. Original cells provide successful-task count/19 and a rounded percentage; we derive the exact ratio from the integer count. The19 tasks contain52 preregistered cross-agent dependencies with executable checkers. Single solves the whole task; Serial hands off in order; Async-Private isolates concurrent specialists; read-only/writable managers differ in declared repair authority. Protocols are kept separate. Incomplete token records affect cost only and are excluded from this outcome metric.","zh":"十个独立模型权重在所选执行协议下完成相同19个仓库任务。附录I把严格最终成功定义为仓库测试全部通过且依赖全部解决（ADPR为1），不含部分测试分。原单元同时给成功题数/19和四舍五入百分率，本包从整数题数计算精确比例。19题共52条预注册、可执行检查的跨Agent依赖。Single独立解整题，Serial顺序交接，Async-Private隔离并行专家，两类manager按只读或可写权限区分，各协议分组。缺失token记录只影响成本，不改变本任务结果。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.32662v1","published_at":"2026-09-26","status":"paper_baselines"},"model_count":10,"result_count":10,"results":[[4070,0.0,370,13442,13438],[8655,0.0,370,13442,13438],[13439,0.0,370,13442,13438],[2584,0.15789473684210525,370,13442,13438],[4966,0.0,370,13442,13438],[5479,0.0,370,13442,13438],[5803,0.05263157894736842,370,13442,13438],[3763,0.15789473684210525,370,13442,13438],[5300,0.3684210526315789,370,13442,13438],[6293,0.5789473684210527,370,13442,13438]]},{"id":"asyncode-async-manager-2026-paper","name":"AsynCodeBench ·Async-Manager (2026 paper)","group":"编程","metric":"Full task-and-dependency success","unit":"%","public_history":true,"protocol":{"en":"Ten distinct model checkpoints execute the same 19 repository tasks under the selected execution protocol. AppendixI defines strict final success as complete repository test pass AND full dependency resolution (ADPR1), not partial test credit. Original cells provide successful-task count/19 and a rounded percentage; we derive the exact ratio from the integer count. The19 tasks contain52 preregistered cross-agent dependencies with executable checkers. Single solves the whole task; Serial hands off in order; Async-Private isolates concurrent specialists; read-only/writable managers differ in declared repair authority. Protocols are kept separate. Incomplete token records affect cost only and are excluded from this outcome metric.","zh":"十个独立模型权重在所选执行协议下完成相同19个仓库任务。附录I把严格最终成功定义为仓库测试全部通过且依赖全部解决（ADPR为1），不含部分测试分。原单元同时给成功题数/19和四舍五入百分率，本包从整数题数计算精确比例。19题共52条预注册、可执行检查的跨Agent依赖。Single独立解整题，Serial顺序交接，Async-Private隔离并行专家，两类manager按只读或可写权限区分，各协议分组。缺失token记录只影响成本，不改变本任务结果。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.32662v1","published_at":"2026-09-26","status":"paper_baselines"},"model_count":10,"result_count":10,"results":[[4070,0.0,370,13443,13438],[8655,0.05263157894736842,370,13443,13438],[13439,0.0,370,13443,13438],[2584,0.10526315789473684,370,13443,13438],[4966,0.0,370,13443,13438],[5479,0.15789473684210525,370,13443,13438],[5803,0.10526315789473684,370,13443,13438],[3763,0.3684210526315789,370,13443,13438],[5300,0.631578947368421,370,13443,13438],[6293,0.6842105263157895,370,13443,13438]]},{"id":"realis-test-ft-2026-paper","name":"REALIS ·fine-tuned test (2026 paper)","group":"视觉与视频","metric":"ROC-AUC","unit":"score","public_history":true,"protocol":{"en":"Ten learned detector architectures share the full balanced100000-image clean REALIS Test population (50000 real/50000 synthetic). Each is fine-tuned on the published1.1 million-image training population; original recipes are retained. The 100000-image validation split is separate. Split sampling is image-disjoint while generator families deliberately overlap, so this measures the declared seen/unseen-generator mixture. ProDet failed to converge and is excluded; clean off-the-shelf Test cells are unavailable. The stored100-scale ROC-AUC is native discrimination quality, not percent-correct. Expert scores use only an open-source subset with a different population and are not merged.","zh":"十种学习型检测架构共享完整、类别平衡的100000张REALIS干净测试图像，其中真实与生成各50000张；均按原方案用110万张训练数据微调，另有100000张验证集。图像跨划分隔离，但生成器家族有意重叠，测试涵盖原文声明的已见和未见生成器混合。ProDet微调未收敛，排除该行；干净测试的未微调单元没有结果。ROC-AUC保留原生百分制曲线面积，不作正确率。Expert只覆盖另一组开源子集，未合并。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.32734v1","published_at":"2026-09-26","status":"paper_baselines"},"score_min":0,"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Native score / 100","zh":"原生分数 / 100"},"score_decimals":2,"model_count":10,"result_count":10,"results":[[6759,60.1,370,13444,13445],[13446,97.3,370,13444,13445],[6755,94.9,370,13444,13445],[13447,79.1,370,13444,13445],[13448,75.8,370,13444,13445],[13449,65.5,370,13444,13445],[13450,96.3,370,13444,13445],[13451,69.4,370,13444,13445],[13452,93.7,370,13444,13445],[13453,92.8,370,13444,13445]]},{"id":"omnismarthome-all-grounding-2026-paper","name":"OmniSmartHome ·all grounding (2026 paper)","group":"Agent 与工具","metric":"Target-device grounding accuracy","unit":"%","public_history":true,"protocol":{"en":"Sixteen base-agent model/configuration rows cover fifteen independent backbones on the same 1360 synthetic episodes for this grounding population. Qwen3Omni Instruct/Think are configurations of one family checkpoint. Each valid grounding-query combination has20 episodes per feasible difficulty (three levels) plus 20infeasible episodes;G0/G4coverfour query types and G1–G3coverthree. Original group rates average query types. A ReAct agent must declare the target device before finishing. Grounding checks this declaration; feasible control/scheduling goals use simulator state, while state inquiries/infeasibility use the fixed LLM judge. Native FOA support or mono downmix remains a model input configuration. The272recorded real-world episodes and 680training episodes for PROME are separate.","zh":"十六条基础Agent模型与配置覆盖十五种独立权重，在本grounding组相同1360个合成情境上评测；Qwen3Omni的Instruct和Think作同权重的配置。各有效grounding与query组合含三档可行情境各20条及不可行20条，G0/G4有四类query，G1–G3有三类；原表组率对query类型平均。ReAct必须先声明目标设备才能结束，grounding核对该声明；可行控制和调度按模拟器终态判定，状态查询和不可行请求用固定LLM评审。原生FOA或单声道下混保留为模型输入配置。另272条真实记录和PROME所用680条训练数据不混入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.32569v1","published_at":"2026-09-26","status":"paper_baselines"},"model_count":15,"result_count":16,"results":[[13454,0.057999999999999996,370,13455,13456],[7523,0.33,370,13457,13456],[13458,0.2,370,13459,13456],[8554,0.27,370,13460,13456],[13461,0.245,370,13462,13456],[13463,0.17800000000000002,370,13464,13456],[13465,0.606,370,13466,13456],[5970,0.506,370,13467,13456],[13468,0.627,370,13469,13456],[13470,0.6659999999999999,370,13471,13456],[13470,0.659,370,13472,13456],[13473,0.784,370,13474,13456],[766,0.737,370,13475,13456],[764,0.81,370,13476,13456],[13477,0.856,370,13478,13456],[6725,0.8170000000000001,370,13479,13456]]},{"id":"omnismarthome-all-goal-2026-paper","name":"OmniSmartHome ·all goal (2026 paper)","group":"Agent 与工具","metric":"Goal satisfaction accuracy","unit":"%","public_history":true,"protocol":{"en":"Sixteen base-agent model/configuration rows cover fifteen independent backbones on the same 1360 synthetic episodes for this grounding population. Qwen3Omni Instruct/Think are configurations of one family checkpoint. Each valid grounding-query combination has20 episodes per feasible difficulty (three levels) plus 20infeasible episodes;G0/G4coverfour query types and G1–G3coverthree. Original group rates average query types. A ReAct agent must declare the target device before finishing. Grounding checks this declaration; feasible control/scheduling goals use simulator state, while state inquiries/infeasibility use the fixed LLM judge. Native FOA support or mono downmix remains a model input configuration. The272recorded real-world episodes and 680training episodes for PROME are separate.","zh":"十六条基础Agent模型与配置覆盖十五种独立权重，在本grounding组相同1360个合成情境上评测；Qwen3Omni的Instruct和Think作同权重的配置。各有效grounding与query组合含三档可行情境各20条及不可行20条，G0/G4有四类query，G1–G3有三类；原表组率对query类型平均。ReAct必须先声明目标设备才能结束，grounding核对该声明；可行控制和调度按模拟器终态判定，状态查询和不可行请求用固定LLM评审。原生FOA或单声道下混保留为模型输入配置。另272条真实记录和PROME所用680条训练数据不混入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.32569v1","published_at":"2026-09-26","status":"paper_baselines"},"model_count":15,"result_count":16,"results":[[13454,0.049,370,13455,13456],[7523,0.17600000000000002,370,13457,13456],[13458,0.115,370,13459,13456],[8554,0.13699999999999998,370,13460,13456],[13461,0.102,370,13462,13456],[13463,0.08800000000000001,370,13464,13456],[13465,0.309,370,13466,13456],[5970,0.252,370,13467,13456],[13468,0.445,370,13469,13456],[13470,0.43799999999999994,370,13471,13456],[13470,0.484,370,13472,13456],[13473,0.602,370,13474,13456],[766,0.562,370,13475,13456],[764,0.631,370,13476,13456],[13477,0.7340000000000001,370,13478,13456],[6725,0.631,370,13479,13456]]},{"id":"omnismarthome-speech-grounding-2026-paper","name":"OmniSmartHome ·speech grounding (2026 paper)","group":"Agent 与工具","metric":"Target-device grounding accuracy","unit":"%","public_history":true,"protocol":{"en":"Sixteen base-agent model/configuration rows cover fifteen independent backbones on the same 320 synthetic episodes for this grounding population. Qwen3Omni Instruct/Think are configurations of one family checkpoint. Each valid grounding-query combination has20 episodes per feasible difficulty (three levels) plus 20infeasible episodes;G0/G4coverfour query types and G1–G3coverthree. Original group rates average query types. A ReAct agent must declare the target device before finishing. Grounding checks this declaration; feasible control/scheduling goals use simulator state, while state inquiries/infeasibility use the fixed LLM judge. Native FOA support or mono downmix remains a model input configuration. The272recorded real-world episodes and 680training episodes for PROME are separate.","zh":"十六条基础Agent模型与配置覆盖十五种独立权重，在本grounding组相同320个合成情境上评测；Qwen3Omni的Instruct和Think作同权重的配置。各有效grounding与query组合含三档可行情境各20条及不可行20条，G0/G4有四类query，G1–G3有三类；原表组率对query类型平均。ReAct必须先声明目标设备才能结束，grounding核对该声明；可行控制和调度按模拟器终态判定，状态查询和不可行请求用固定LLM评审。原生FOA或单声道下混保留为模型输入配置。另272条真实记录和PROME所用680条训练数据不混入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.32569v1","published_at":"2026-09-26","status":"paper_baselines"},"model_count":15,"result_count":16,"results":[[13454,0.068,370,13480,13456],[7523,0.555,370,13481,13456],[13458,0.391,370,13482,13456],[8554,0.48200000000000004,370,13483,13456],[13461,0.445,370,13484,13456],[13463,0.312,370,13485,13456],[13465,0.852,370,13486,13456],[5970,0.693,370,13487,13456],[13468,0.865,370,13488,13456],[13470,0.88,370,13489,13456],[13470,0.878,370,13490,13456],[13473,0.935,370,13491,13456],[766,0.94,370,13492,13456],[764,0.951,370,13493,13456],[13477,0.953,370,13494,13456],[6725,0.932,370,13495,13456]]},{"id":"omnismarthome-speech-goal-2026-paper","name":"OmniSmartHome ·speech goal (2026 paper)","group":"Agent 与工具","metric":"Goal satisfaction accuracy","unit":"%","public_history":true,"protocol":{"en":"Sixteen base-agent model/configuration rows cover fifteen independent backbones on the same 320 synthetic episodes for this grounding population. Qwen3Omni Instruct/Think are configurations of one family checkpoint. Each valid grounding-query combination has20 episodes per feasible difficulty (three levels) plus 20infeasible episodes;G0/G4coverfour query types and G1–G3coverthree. Original group rates average query types. A ReAct agent must declare the target device before finishing. Grounding checks this declaration; feasible control/scheduling goals use simulator state, while state inquiries/infeasibility use the fixed LLM judge. Native FOA support or mono downmix remains a model input configuration. The272recorded real-world episodes and 680training episodes for PROME are separate.","zh":"十六条基础Agent模型与配置覆盖十五种独立权重，在本grounding组相同320个合成情境上评测；Qwen3Omni的Instruct和Think作同权重的配置。各有效grounding与query组合含三档可行情境各20条及不可行20条，G0/G4有四类query，G1–G3有三类；原表组率对query类型平均。ReAct必须先声明目标设备才能结束，grounding核对该声明；可行控制和调度按模拟器终态判定，状态查询和不可行请求用固定LLM评审。原生FOA或单声道下混保留为模型输入配置。另272条真实记录和PROME所用680条训练数据不混入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.32569v1","published_at":"2026-09-26","status":"paper_baselines"},"model_count":15,"result_count":16,"results":[[13454,0.034,370,13480,13456],[7523,0.255,370,13481,13456],[13458,0.141,370,13482,13456],[8554,0.20600000000000002,370,13483,13456],[13461,0.135,370,13484,13456],[13463,0.099,370,13485,13456],[13465,0.51,370,13486,13456],[5970,0.45299999999999996,370,13487,13456],[13468,0.609,370,13488,13456],[13470,0.63,370,13489,13456],[13470,0.674,370,13490,13456],[13473,0.737,370,13491,13456],[766,0.768,370,13492,13456],[764,0.805,370,13493,13456],[13477,0.81,370,13494,13456],[6725,0.768,370,13495,13456]]},{"id":"omnismarthome-gesture-grounding-2026-paper","name":"OmniSmartHome ·gesture grounding (2026 paper)","group":"Agent 与工具","metric":"Target-device grounding accuracy","unit":"%","public_history":true,"protocol":{"en":"Sixteen base-agent model/configuration rows cover fifteen independent backbones on the same 240 synthetic episodes for this grounding population. Qwen3Omni Instruct/Think are configurations of one family checkpoint. Each valid grounding-query combination has20 episodes per feasible difficulty (three levels) plus 20infeasible episodes;G0/G4coverfour query types and G1–G3coverthree. Original group rates average query types. A ReAct agent must declare the target device before finishing. Grounding checks this declaration; feasible control/scheduling goals use simulator state, while state inquiries/infeasibility use the fixed LLM judge. Native FOA support or mono downmix remains a model input configuration. The272recorded real-world episodes and 680training episodes for PROME are separate.","zh":"十六条基础Agent模型与配置覆盖十五种独立权重，在本grounding组相同240个合成情境上评测；Qwen3Omni的Instruct和Think作同权重的配置。各有效grounding与query组合含三档可行情境各20条及不可行20条，G0/G4有四类query，G1–G3有三类；原表组率对query类型平均。ReAct必须先声明目标设备才能结束，grounding核对该声明；可行控制和调度按模拟器终态判定，状态查询和不可行请求用固定LLM评审。原生FOA或单声道下混保留为模型输入配置。另272条真实记录和PROME所用680条训练数据不混入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.32569v1","published_at":"2026-09-26","status":"paper_baselines"},"model_count":15,"result_count":16,"results":[[13454,0.062,370,13496,13456],[7523,0.309,370,13497,13456],[13458,0.149,370,13498,13456],[8554,0.201,370,13499,13456],[13461,0.247,370,13500,13456],[13463,0.09,370,13501,13456],[13465,0.569,370,13502,13456],[5970,0.465,370,13503,13456],[13468,0.653,370,13504,13456],[13470,0.6629999999999999,370,13505,13456],[13470,0.66,370,13506,13456],[13473,0.722,370,13507,13456],[766,0.6940000000000001,370,13508,13456],[764,0.747,370,13509,13456],[13477,0.764,370,13510,13456],[6725,0.767,370,13511,13456]]},{"id":"omnismarthome-gesture-goal-2026-paper","name":"OmniSmartHome ·gesture goal (2026 paper)","group":"Agent 与工具","metric":"Goal satisfaction accuracy","unit":"%","public_history":true,"protocol":{"en":"Sixteen base-agent model/configuration rows cover fifteen independent backbones on the same 240 synthetic episodes for this grounding population. Qwen3Omni Instruct/Think are configurations of one family checkpoint. Each valid grounding-query combination has20 episodes per feasible difficulty (three levels) plus 20infeasible episodes;G0/G4coverfour query types and G1–G3coverthree. Original group rates average query types. A ReAct agent must declare the target device before finishing. Grounding checks this declaration; feasible control/scheduling goals use simulator state, while state inquiries/infeasibility use the fixed LLM judge. Native FOA support or mono downmix remains a model input configuration. The272recorded real-world episodes and 680training episodes for PROME are separate.","zh":"十六条基础Agent模型与配置覆盖十五种独立权重，在本grounding组相同240个合成情境上评测；Qwen3Omni的Instruct和Think作同权重的配置。各有效grounding与query组合含三档可行情境各20条及不可行20条，G0/G4有四类query，G1–G3有三类；原表组率对query类型平均。ReAct必须先声明目标设备才能结束，grounding核对该声明；可行控制和调度按模拟器终态判定，状态查询和不可行请求用固定LLM评审。原生FOA或单声道下混保留为模型输入配置。另272条真实记录和PROME所用680条训练数据不混入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.32569v1","published_at":"2026-09-26","status":"paper_baselines"},"model_count":15,"result_count":16,"results":[[13454,0.069,370,13496,13456],[7523,0.19399999999999998,370,13497,13456],[13458,0.115,370,13498,13456],[8554,0.11800000000000001,370,13499,13456],[13461,0.10400000000000001,370,13500,13456],[13463,0.069,370,13501,13456],[13465,0.299,370,13502,13456],[5970,0.191,370,13503,13456],[13468,0.524,370,13504,13456],[13470,0.5379999999999999,370,13505,13456],[13470,0.5660000000000001,370,13506,13456],[13473,0.635,370,13507,13456],[766,0.601,370,13508,13456],[764,0.667,370,13509,13456],[13477,0.715,370,13510,13456],[6725,0.642,370,13511,13456]]},{"id":"omnismarthome-sound-grounding-2026-paper","name":"OmniSmartHome ·sound grounding (2026 paper)","group":"Agent 与工具","metric":"Target-device grounding accuracy","unit":"%","public_history":true,"protocol":{"en":"Sixteen base-agent model/configuration rows cover fifteen independent backbones on the same 240 synthetic episodes for this grounding population. Qwen3Omni Instruct/Think are configurations of one family checkpoint. Each valid grounding-query combination has20 episodes per feasible difficulty (three levels) plus 20infeasible episodes;G0/G4coverfour query types and G1–G3coverthree. Original group rates average query types. A ReAct agent must declare the target device before finishing. Grounding checks this declaration; feasible control/scheduling goals use simulator state, while state inquiries/infeasibility use the fixed LLM judge. Native FOA support or mono downmix remains a model input configuration. The272recorded real-world episodes and 680training episodes for PROME are separate.","zh":"十六条基础Agent模型与配置覆盖十五种独立权重，在本grounding组相同240个合成情境上评测；Qwen3Omni的Instruct和Think作同权重的配置。各有效grounding与query组合含三档可行情境各20条及不可行20条，G0/G4有四类query，G1–G3有三类；原表组率对query类型平均。ReAct必须先声明目标设备才能结束，grounding核对该声明；可行控制和调度按模拟器终态判定，状态查询和不可行请求用固定LLM评审。原生FOA或单声道下混保留为模型输入配置。另272条真实记录和PROME所用680条训练数据不混入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.32569v1","published_at":"2026-09-26","status":"paper_baselines"},"model_count":15,"result_count":16,"results":[[13454,0.062,370,13512,13456],[7523,0.281,370,13513,13456],[13458,0.163,370,13514,13456],[8554,0.23600000000000002,370,13515,13456],[13461,0.174,370,13516,13456],[13463,0.149,370,13517,13456],[13465,0.5660000000000001,370,13518,13456],[5970,0.552,370,13519,13456],[13468,0.674,370,13520,13456],[13470,0.774,370,13521,13456],[13470,0.799,370,13522,13456],[13473,0.8190000000000001,370,13523,13456],[766,0.743,370,13524,13456],[764,0.8190000000000001,370,13525,13456],[13477,0.976,370,13526,13456],[6725,0.927,370,13527,13456]]},{"id":"omnismarthome-sound-goal-2026-paper","name":"OmniSmartHome ·sound goal (2026 paper)","group":"Agent 与工具","metric":"Goal satisfaction accuracy","unit":"%","public_history":true,"protocol":{"en":"Sixteen base-agent model/configuration rows cover fifteen independent backbones on the same 240 synthetic episodes for this grounding population. Qwen3Omni Instruct/Think are configurations of one family checkpoint. Each valid grounding-query combination has20 episodes per feasible difficulty (three levels) plus 20infeasible episodes;G0/G4coverfour query types and G1–G3coverthree. Original group rates average query types. A ReAct agent must declare the target device before finishing. Grounding checks this declaration; feasible control/scheduling goals use simulator state, while state inquiries/infeasibility use the fixed LLM judge. Native FOA support or mono downmix remains a model input configuration. The272recorded real-world episodes and 680training episodes for PROME are separate.","zh":"十六条基础Agent模型与配置覆盖十五种独立权重，在本grounding组相同240个合成情境上评测；Qwen3Omni的Instruct和Think作同权重的配置。各有效grounding与query组合含三档可行情境各20条及不可行20条，G0/G4有四类query，G1–G3有三类；原表组率对query类型平均。ReAct必须先声明目标设备才能结束，grounding核对该声明；可行控制和调度按模拟器终态判定，状态查询和不可行请求用固定LLM评审。原生FOA或单声道下混保留为模型输入配置。另272条真实记录和PROME所用680条训练数据不混入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.32569v1","published_at":"2026-09-26","status":"paper_baselines"},"model_count":15,"result_count":16,"results":[[13454,0.062,370,13512,13456],[7523,0.142,370,13513,13456],[13458,0.10400000000000001,370,13514,13456],[8554,0.135,370,13515,13456],[13461,0.094,370,13516,13456],[13463,0.09699999999999999,370,13517,13456],[13465,0.264,370,13518,13456],[5970,0.292,370,13519,13456],[13468,0.47200000000000003,370,13520,13456],[13470,0.382,370,13521,13456],[13470,0.569,370,13522,13456],[13473,0.608,370,13523,13456],[766,0.5489999999999999,370,13524,13456],[764,0.503,370,13525,13456],[13477,0.809,370,13526,13456],[6725,0.66,370,13527,13456]]},{"id":"omnismarthome-space-grounding-2026-paper","name":"OmniSmartHome ·space grounding (2026 paper)","group":"Agent 与工具","metric":"Target-device grounding accuracy","unit":"%","public_history":true,"protocol":{"en":"Sixteen base-agent model/configuration rows cover fifteen independent backbones on the same 240 synthetic episodes for this grounding population. Qwen3Omni Instruct/Think are configurations of one family checkpoint. Each valid grounding-query combination has20 episodes per feasible difficulty (three levels) plus 20infeasible episodes;G0/G4coverfour query types and G1–G3coverthree. Original group rates average query types. A ReAct agent must declare the target device before finishing. Grounding checks this declaration; feasible control/scheduling goals use simulator state, while state inquiries/infeasibility use the fixed LLM judge. Native FOA support or mono downmix remains a model input configuration. The272recorded real-world episodes and 680training episodes for PROME are separate.","zh":"十六条基础Agent模型与配置覆盖十五种独立权重，在本grounding组相同240个合成情境上评测；Qwen3Omni的Instruct和Think作同权重的配置。各有效grounding与query组合含三档可行情境各20条及不可行20条，G0/G4有四类query，G1–G3有三类；原表组率对query类型平均。ReAct必须先声明目标设备才能结束，grounding核对该声明；可行控制和调度按模拟器终态判定，状态查询和不可行请求用固定LLM评审。原生FOA或单声道下混保留为模型输入配置。另272条真实记录和PROME所用680条训练数据不混入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.32569v1","published_at":"2026-09-26","status":"paper_baselines"},"model_count":15,"result_count":16,"results":[[13454,0.076,370,13528,13456],[7523,0.272,370,13529,13456],[13458,0.16,370,13530,13456],[8554,0.247,370,13531,13456],[13461,0.253,370,13532,13456],[13463,0.17,370,13533,13456],[13465,0.594,370,13534,13456],[5970,0.5489999999999999,370,13535,13456],[13468,0.649,370,13536,13456],[13470,0.608,370,13537,13456],[13470,0.722,370,13538,13456],[13473,0.847,370,13539,13456],[766,0.7809999999999999,370,13540,13456],[764,0.83,370,13541,13456],[13477,0.875,370,13542,13456],[6725,0.8540000000000001,370,13543,13456]]},{"id":"omnismarthome-space-goal-2026-paper","name":"OmniSmartHome ·space goal (2026 paper)","group":"Agent 与工具","metric":"Goal satisfaction accuracy","unit":"%","public_history":true,"protocol":{"en":"Sixteen base-agent model/configuration rows cover fifteen independent backbones on the same 240 synthetic episodes for this grounding population. Qwen3Omni Instruct/Think are configurations of one family checkpoint. Each valid grounding-query combination has20 episodes per feasible difficulty (three levels) plus 20infeasible episodes;G0/G4coverfour query types and G1–G3coverthree. Original group rates average query types. A ReAct agent must declare the target device before finishing. Grounding checks this declaration; feasible control/scheduling goals use simulator state, while state inquiries/infeasibility use the fixed LLM judge. Native FOA support or mono downmix remains a model input configuration. The272recorded real-world episodes and 680training episodes for PROME are separate.","zh":"十六条基础Agent模型与配置覆盖十五种独立权重，在本grounding组相同240个合成情境上评测；Qwen3Omni的Instruct和Think作同权重的配置。各有效grounding与query组合含三档可行情境各20条及不可行20条，G0/G4有四类query，G1–G3有三类；原表组率对query类型平均。ReAct必须先声明目标设备才能结束，grounding核对该声明；可行控制和调度按模拟器终态判定，状态查询和不可行请求用固定LLM评审。原生FOA或单声道下混保留为模型输入配置。另272条真实记录和PROME所用680条训练数据不混入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.32569v1","published_at":"2026-09-26","status":"paper_baselines"},"model_count":15,"result_count":16,"results":[[13454,0.062,370,13528,13456],[7523,0.174,370,13529,13456],[13458,0.11800000000000001,370,13530,13456],[8554,0.149,370,13531,13456],[13461,0.11800000000000001,370,13532,13456],[13463,0.09699999999999999,370,13533,13456],[13465,0.32299999999999995,370,13534,13456],[5970,0.26,370,13535,13456],[13468,0.514,370,13536,13456],[13470,0.486,370,13537,13456],[13470,0.552,370,13538,13456],[13473,0.6940000000000001,370,13539,13456],[766,0.632,370,13540,13456],[764,0.698,370,13541,13456],[13477,0.792,370,13542,13456],[6725,0.722,370,13543,13456]]},{"id":"omnismarthome-identity-grounding-2026-paper","name":"OmniSmartHome ·identity grounding (2026 paper)","group":"Agent 与工具","metric":"Target-device grounding accuracy","unit":"%","public_history":true,"protocol":{"en":"Sixteen base-agent model/configuration rows cover fifteen independent backbones on the same 320 synthetic episodes for this grounding population. Qwen3Omni Instruct/Think are configurations of one family checkpoint. Each valid grounding-query combination has20 episodes per feasible difficulty (three levels) plus 20infeasible episodes;G0/G4coverfour query types and G1–G3coverthree. Original group rates average query types. A ReAct agent must declare the target device before finishing. Grounding checks this declaration; feasible control/scheduling goals use simulator state, while state inquiries/infeasibility use the fixed LLM judge. Native FOA support or mono downmix remains a model input configuration. The272recorded real-world episodes and 680training episodes for PROME are separate.","zh":"十六条基础Agent模型与配置覆盖十五种独立权重，在本grounding组相同320个合成情境上评测；Qwen3Omni的Instruct和Think作同权重的配置。各有效grounding与query组合含三档可行情境各20条及不可行20条，G0/G4有四类query，G1–G3有三类；原表组率对query类型平均。ReAct必须先声明目标设备才能结束，grounding核对该声明；可行控制和调度按模拟器终态判定，状态查询和不可行请求用固定LLM评审。原生FOA或单声道下混保留为模型输入配置。另272条真实记录和PROME所用680条训练数据不混入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.32569v1","published_at":"2026-09-26","status":"paper_baselines"},"model_count":15,"result_count":16,"results":[[13454,0.028999999999999998,370,13544,13456],[7523,0.203,370,13545,13456],[13458,0.10400000000000001,370,13546,13456],[8554,0.154,370,13547,13456],[13461,0.091,370,13548,13456],[13463,0.135,370,13549,13456],[13465,0.42700000000000005,370,13550,13456],[5970,0.281,370,13551,13456],[13468,0.318,370,13552,13456],[13470,0.41700000000000004,370,13553,13456],[13470,0.28600000000000003,370,13554,13456],[13473,0.607,370,13555,13456],[766,0.526,370,13556,13456],[764,0.695,370,13557,13456],[13477,0.7240000000000001,370,13558,13456],[6725,0.63,370,13559,13456]]},{"id":"omnismarthome-identity-goal-2026-paper","name":"OmniSmartHome ·identity goal (2026 paper)","group":"Agent 与工具","metric":"Goal satisfaction accuracy","unit":"%","public_history":true,"protocol":{"en":"Sixteen base-agent model/configuration rows cover fifteen independent backbones on the same 320 synthetic episodes for this grounding population. Qwen3Omni Instruct/Think are configurations of one family checkpoint. Each valid grounding-query combination has20 episodes per feasible difficulty (three levels) plus 20infeasible episodes;G0/G4coverfour query types and G1–G3coverthree. Original group rates average query types. A ReAct agent must declare the target device before finishing. Grounding checks this declaration; feasible control/scheduling goals use simulator state, while state inquiries/infeasibility use the fixed LLM judge. Native FOA support or mono downmix remains a model input configuration. The272recorded real-world episodes and 680training episodes for PROME are separate.","zh":"十六条基础Agent模型与配置覆盖十五种独立权重，在本grounding组相同320个合成情境上评测；Qwen3Omni的Instruct和Think作同权重的配置。各有效grounding与query组合含三档可行情境各20条及不可行20条，G0/G4有四类query，G1–G3有三类；原表组率对query类型平均。ReAct必须先声明目标设备才能结束，grounding核对该声明；可行控制和调度按模拟器终态判定，状态查询和不可行请求用固定LLM评审。原生FOA或单声道下混保留为模型输入配置。另272条真实记录和PROME所用680条训练数据不混入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.32569v1","published_at":"2026-09-26","status":"paper_baselines"},"model_count":15,"result_count":16,"results":[[13454,0.028999999999999998,370,13544,13456],[7523,0.109,370,13545,13456],[13458,0.094,370,13546,13456],[8554,0.076,370,13547,13456],[13461,0.065,370,13548,13456],[13463,0.076,370,13549,13456],[13465,0.141,370,13550,13456],[5970,0.06,370,13551,13456],[13468,0.151,370,13552,13456],[13470,0.174,370,13553,13456],[13470,0.11699999999999999,370,13554,13456],[13473,0.37,370,13555,13456],[766,0.28600000000000003,370,13556,13456],[764,0.47700000000000004,370,13557,13456],[13477,0.573,370,13558,13456],[6725,0.396,370,13559,13456]]},{"id":"mic-id-macro-f1-2026-paper","name":"MIC-Bench ·id macro-f1 (2026 paper)","group":"视觉与视频","metric":"macro-f1","unit":"score","public_history":true,"protocol":{"en":"The id test split contains 703 claims, each paired with one source-consistent and one edited-inconsistent image (1406 image-claim instances). VerdictMacroF1 covers both classes; type and semantic-evidence metrics use the 703 inconsistent examples. ID/OOD refers to edited-in element identities seen/unseen during training; claim pairs are split together. Two annotators independently verify realism, edit fidelity andfacts, discard conflicts, and revise references. Twelve verdict rows cover ten independent bases; MIC is a trainedQwen3VL4B configuration and 8BThinking is not an extra independent backbone. Prompted baselines use their stated zero-shotCoT settings. Similarity retains native embedding-cosine100 units rather than accuracy; native F1 is also kept separate from percent-correct.","zh":"id测试组含703条声明，每条分别配原始一致图与编辑后矛盾图，共1406条图文实例。判决MacroF1覆盖两类，类型与语义证据指标只用703条矛盾实例。ID/OOD指编辑元素身份在训练时已见或未见，声明对整体划分。两位标注者独立检查真实性、编辑保真和事实，丢弃分歧并修订参考。十二条判决结果覆盖十种独立基模型；MIC为Qwen3VL4B训练配置，8BThinking不另增加架构。提示基线按原零样本CoT设置。语义相似度保留原生余弦百分制单位，F1也保留原生刻度。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.33441v1","published_at":"2026-09-27","status":"paper_baselines"},"score_min":0,"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Native score / 100","zh":"原生分数 / 100"},"score_decimals":2,"model_count":10,"result_count":12,"results":[[7324,39.91,370,13560,13561],[13562,53.13,370,13563,13561],[13564,46.69,370,13565,13561],[4958,53.89,370,13566,13561],[2478,40.47,370,13567,13561],[6149,58.9,370,13568,13561],[6150,59.95,370,13569,13561],[6238,63.96,370,13570,13561],[6389,56.01,370,13571,13561],[2728,55.52,370,13572,13561],[6150,59.79,370,13573,13561],[6149,94.88,370,13574,13561]]},{"id":"mic-id-type-accuracy-2026-paper","name":"MIC-Bench ·id type-accuracy (2026 paper)","group":"视觉与视频","metric":"type-accuracy","unit":"%","public_history":true,"protocol":{"en":"The id test split contains 703 claims, each paired with one source-consistent and one edited-inconsistent image (1406 image-claim instances). VerdictMacroF1 covers both classes; type and semantic-evidence metrics use the 703 inconsistent examples. ID/OOD refers to edited-in element identities seen/unseen during training; claim pairs are split together. Two annotators independently verify realism, edit fidelity andfacts, discard conflicts, and revise references. Twelve verdict rows cover ten independent bases; MIC is a trainedQwen3VL4B configuration and 8BThinking is not an extra independent backbone. Prompted baselines use their stated zero-shotCoT settings. Similarity retains native embedding-cosine100 units rather than accuracy; native F1 is also kept separate from percent-correct.","zh":"id测试组含703条声明，每条分别配原始一致图与编辑后矛盾图，共1406条图文实例。判决MacroF1覆盖两类，类型与语义证据指标只用703条矛盾实例。ID/OOD指编辑元素身份在训练时已见或未见，声明对整体划分。两位标注者独立检查真实性、编辑保真和事实，丢弃分歧并修订参考。十二条判决结果覆盖十种独立基模型；MIC为Qwen3VL4B训练配置，8BThinking不另增加架构。提示基线按原零样本CoT设置。语义相似度保留原生余弦百分制单位，F1也保留原生刻度。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.33441v1","published_at":"2026-09-27","status":"paper_baselines"},"model_count":7,"result_count":9,"results":[[4958,0.1152,370,13566,13561],[2478,0.15789999999999998,370,13567,13561],[6149,0.1906,370,13568,13561],[6150,0.23329999999999998,370,13569,13561],[6238,0.3201,370,13570,13561],[6389,0.0925,370,13571,13561],[2728,0.1366,370,13572,13561],[6150,0.40399999999999997,370,13573,13561],[6149,0.8976000000000001,370,13574,13561]]},{"id":"mic-id-visual-similarity-2026-paper","name":"MIC-Bench ·id visual-similarity (2026 paper)","group":"视觉与视频","metric":"visual-similarity","unit":"raw","public_history":true,"protocol":{"en":"The id test split contains 703 claims, each paired with one source-consistent and one edited-inconsistent image (1406 image-claim instances). VerdictMacroF1 covers both classes; type and semantic-evidence metrics use the 703 inconsistent examples. ID/OOD refers to edited-in element identities seen/unseen during training; claim pairs are split together. Two annotators independently verify realism, edit fidelity andfacts, discard conflicts, and revise references. Twelve verdict rows cover ten independent bases; MIC is a trainedQwen3VL4B configuration and 8BThinking is not an extra independent backbone. Prompted baselines use their stated zero-shotCoT settings. Similarity retains native embedding-cosine100 units rather than accuracy; native F1 is also kept separate from percent-correct.","zh":"id测试组含703条声明，每条分别配原始一致图与编辑后矛盾图，共1406条图文实例。判决MacroF1覆盖两类，类型与语义证据指标只用703条矛盾实例。ID/OOD指编辑元素身份在训练时已见或未见，声明对整体划分。两位标注者独立检查真实性、编辑保真和事实，丢弃分歧并修订参考。十二条判决结果覆盖十种独立基模型；MIC为Qwen3VL4B训练配置，8BThinking不另增加架构。提示基线按原零样本CoT设置。语义相似度保留原生余弦百分制单位，F1也保留原生刻度。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.33441v1","published_at":"2026-09-27","status":"paper_baselines"},"saturation_excluded":true,"scale_label":{"en":"visual-similarity","zh":"visual-similarity"},"score_decimals":3,"model_count":7,"result_count":9,"results":[[4958,20.14,370,13566,13561],[2478,13.83,370,13567,13561],[6149,23.16,370,13568,13561],[6150,26.74,370,13569,13561],[6238,37.95,370,13570,13561],[6389,13.97,370,13571,13561],[2728,18.24,370,13572,13561],[6150,43.41,370,13573,13561],[6149,83.04,370,13574,13561]]},{"id":"mic-id-explanation-similarity-2026-paper","name":"MIC-Bench ·id explanation-similarity (2026 paper)","group":"视觉与视频","metric":"explanation-similarity","unit":"raw","public_history":true,"protocol":{"en":"The id test split contains 703 claims, each paired with one source-consistent and one edited-inconsistent image (1406 image-claim instances). VerdictMacroF1 covers both classes; type and semantic-evidence metrics use the 703 inconsistent examples. ID/OOD refers to edited-in element identities seen/unseen during training; claim pairs are split together. Two annotators independently verify realism, edit fidelity andfacts, discard conflicts, and revise references. Twelve verdict rows cover ten independent bases; MIC is a trainedQwen3VL4B configuration and 8BThinking is not an extra independent backbone. Prompted baselines use their stated zero-shotCoT settings. Similarity retains native embedding-cosine100 units rather than accuracy; native F1 is also kept separate from percent-correct.","zh":"id测试组含703条声明，每条分别配原始一致图与编辑后矛盾图，共1406条图文实例。判决MacroF1覆盖两类，类型与语义证据指标只用703条矛盾实例。ID/OOD指编辑元素身份在训练时已见或未见，声明对整体划分。两位标注者独立检查真实性、编辑保真和事实，丢弃分歧并修订参考。十二条判决结果覆盖十种独立基模型；MIC为Qwen3VL4B训练配置，8BThinking不另增加架构。提示基线按原零样本CoT设置。语义相似度保留原生余弦百分制单位，F1也保留原生刻度。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.33441v1","published_at":"2026-09-27","status":"paper_baselines"},"saturation_excluded":true,"scale_label":{"en":"explanation-similarity","zh":"explanation-similarity"},"score_decimals":3,"model_count":8,"result_count":10,"results":[[13564,10.66,370,13565,13561],[4958,17.88,370,13566,13561],[2478,29.11,370,13567,13561],[6149,25.24,370,13568,13561],[6150,28.55,370,13569,13561],[6238,37.42,370,13570,13561],[6389,18.65,370,13571,13561],[2728,22.65,370,13572,13561],[6150,46.19,370,13573,13561],[6149,82.54,370,13574,13561]]},{"id":"mic-ood-macro-f1-2026-paper","name":"MIC-Bench ·ood macro-f1 (2026 paper)","group":"视觉与视频","metric":"macro-f1","unit":"score","public_history":true,"protocol":{"en":"The ood test split contains 698 claims, each paired with one source-consistent and one edited-inconsistent image (1396 image-claim instances). VerdictMacroF1 covers both classes; type and semantic-evidence metrics use the 698 inconsistent examples. ID/OOD refers to edited-in element identities seen/unseen during training; claim pairs are split together. Two annotators independently verify realism, edit fidelity andfacts, discard conflicts, and revise references. Twelve verdict rows cover ten independent bases; MIC is a trainedQwen3VL4B configuration and 8BThinking is not an extra independent backbone. Prompted baselines use their stated zero-shotCoT settings. Similarity retains native embedding-cosine100 units rather than accuracy; native F1 is also kept separate from percent-correct.","zh":"ood测试组含698条声明，每条分别配原始一致图与编辑后矛盾图，共1396条图文实例。判决MacroF1覆盖两类，类型与语义证据指标只用698条矛盾实例。ID/OOD指编辑元素身份在训练时已见或未见，声明对整体划分。两位标注者独立检查真实性、编辑保真和事实，丢弃分歧并修订参考。十二条判决结果覆盖十种独立基模型；MIC为Qwen3VL4B训练配置，8BThinking不另增加架构。提示基线按原零样本CoT设置。语义相似度保留原生余弦百分制单位，F1也保留原生刻度。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.33441v1","published_at":"2026-09-27","status":"paper_baselines"},"score_min":0,"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Native score / 100","zh":"原生分数 / 100"},"score_decimals":2,"model_count":10,"result_count":12,"results":[[7324,40.72,370,13575,13561],[13562,51.7,370,13576,13561],[13564,44.07,370,13577,13561],[4958,53.0,370,13578,13561],[2478,38.98,370,13579,13561],[6149,59.44,370,13580,13561],[6150,58.14,370,13581,13561],[6238,63.14,370,13582,13561],[6389,52.06,370,13583,13561],[2728,55.56,370,13584,13561],[6150,59.09,370,13585,13561],[6149,90.26,370,13586,13561]]},{"id":"mic-ood-type-accuracy-2026-paper","name":"MIC-Bench ·ood type-accuracy (2026 paper)","group":"视觉与视频","metric":"type-accuracy","unit":"%","public_history":true,"protocol":{"en":"The ood test split contains 698 claims, each paired with one source-consistent and one edited-inconsistent image (1396 image-claim instances). VerdictMacroF1 covers both classes; type and semantic-evidence metrics use the 698 inconsistent examples. ID/OOD refers to edited-in element identities seen/unseen during training; claim pairs are split together. Two annotators independently verify realism, edit fidelity andfacts, discard conflicts, and revise references. Twelve verdict rows cover ten independent bases; MIC is a trainedQwen3VL4B configuration and 8BThinking is not an extra independent backbone. Prompted baselines use their stated zero-shotCoT settings. Similarity retains native embedding-cosine100 units rather than accuracy; native F1 is also kept separate from percent-correct.","zh":"ood测试组含698条声明，每条分别配原始一致图与编辑后矛盾图，共1396条图文实例。判决MacroF1覆盖两类，类型与语义证据指标只用698条矛盾实例。ID/OOD指编辑元素身份在训练时已见或未见，声明对整体划分。两位标注者独立检查真实性、编辑保真和事实，丢弃分歧并修订参考。十二条判决结果覆盖十种独立基模型；MIC为Qwen3VL4B训练配置，8BThinking不另增加架构。提示基线按原零样本CoT设置。语义相似度保留原生余弦百分制单位，F1也保留原生刻度。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.33441v1","published_at":"2026-09-27","status":"paper_baselines"},"model_count":7,"result_count":9,"results":[[4958,0.12179999999999999,370,13578,13561],[2478,0.1332,370,13579,13561],[6149,0.1963,370,13580,13561],[6150,0.1977,370,13581,13561],[6238,0.3381,370,13582,13561],[6389,0.0759,370,13583,13561],[2728,0.1418,370,13584,13561],[6150,0.3754,370,13585,13561],[6149,0.7865000000000001,370,13586,13561]]},{"id":"mic-ood-visual-similarity-2026-paper","name":"MIC-Bench ·ood visual-similarity (2026 paper)","group":"视觉与视频","metric":"visual-similarity","unit":"raw","public_history":true,"protocol":{"en":"The ood test split contains 698 claims, each paired with one source-consistent and one edited-inconsistent image (1396 image-claim instances). VerdictMacroF1 covers both classes; type and semantic-evidence metrics use the 698 inconsistent examples. ID/OOD refers to edited-in element identities seen/unseen during training; claim pairs are split together. Two annotators independently verify realism, edit fidelity andfacts, discard conflicts, and revise references. Twelve verdict rows cover ten independent bases; MIC is a trainedQwen3VL4B configuration and 8BThinking is not an extra independent backbone. Prompted baselines use their stated zero-shotCoT settings. Similarity retains native embedding-cosine100 units rather than accuracy; native F1 is also kept separate from percent-correct.","zh":"ood测试组含698条声明，每条分别配原始一致图与编辑后矛盾图，共1396条图文实例。判决MacroF1覆盖两类，类型与语义证据指标只用698条矛盾实例。ID/OOD指编辑元素身份在训练时已见或未见，声明对整体划分。两位标注者独立检查真实性、编辑保真和事实，丢弃分歧并修订参考。十二条判决结果覆盖十种独立基模型；MIC为Qwen3VL4B训练配置，8BThinking不另增加架构。提示基线按原零样本CoT设置。语义相似度保留原生余弦百分制单位，F1也保留原生刻度。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.33441v1","published_at":"2026-09-27","status":"paper_baselines"},"saturation_excluded":true,"scale_label":{"en":"visual-similarity","zh":"visual-similarity"},"score_decimals":3,"model_count":7,"result_count":9,"results":[[4958,19.43,370,13578,13561],[2478,11.83,370,13579,13561],[6149,24.18,370,13580,13561],[6150,25.13,370,13581,13561],[6238,38.45,370,13582,13561],[6389,10.43,370,13583,13561],[2728,17.82,370,13584,13561],[6150,42.64,370,13585,13561],[6149,72.09,370,13586,13561]]},{"id":"mic-ood-explanation-similarity-2026-paper","name":"MIC-Bench ·ood explanation-similarity (2026 paper)","group":"视觉与视频","metric":"explanation-similarity","unit":"raw","public_history":true,"protocol":{"en":"The ood test split contains 698 claims, each paired with one source-consistent and one edited-inconsistent image (1396 image-claim instances). VerdictMacroF1 covers both classes; type and semantic-evidence metrics use the 698 inconsistent examples. ID/OOD refers to edited-in element identities seen/unseen during training; claim pairs are split together. Two annotators independently verify realism, edit fidelity andfacts, discard conflicts, and revise references. Twelve verdict rows cover ten independent bases; MIC is a trainedQwen3VL4B configuration and 8BThinking is not an extra independent backbone. Prompted baselines use their stated zero-shotCoT settings. Similarity retains native embedding-cosine100 units rather than accuracy; native F1 is also kept separate from percent-correct.","zh":"ood测试组含698条声明，每条分别配原始一致图与编辑后矛盾图，共1396条图文实例。判决MacroF1覆盖两类，类型与语义证据指标只用698条矛盾实例。ID/OOD指编辑元素身份在训练时已见或未见，声明对整体划分。两位标注者独立检查真实性、编辑保真和事实，丢弃分歧并修订参考。十二条判决结果覆盖十种独立基模型；MIC为Qwen3VL4B训练配置，8BThinking不另增加架构。提示基线按原零样本CoT设置。语义相似度保留原生余弦百分制单位，F1也保留原生刻度。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.33441v1","published_at":"2026-09-27","status":"paper_baselines"},"saturation_excluded":true,"scale_label":{"en":"explanation-similarity","zh":"explanation-similarity"},"score_decimals":3,"model_count":8,"result_count":10,"results":[[13564,8.49,370,13577,13561],[4958,17.02,370,13578,13561],[2478,28.08,370,13579,13561],[6149,27.18,370,13580,13561],[6150,26.88,370,13581,13561],[6238,36.61,370,13582,13561],[6389,16.45,370,13583,13561],[2728,22.51,370,13584,13561],[6150,46.28,370,13585,13561],[6149,75.78,370,13586,13561]]},{"id":"mishap-music-context-2026-paper","name":"MISHAP-Bench ·music-context (2026 paper)","group":"知识与推理","metric":"Unsupported-claim hallucination rate","unit":"%","public_history":true,"protocol":{"en":"Ten independent audio-language model checkpoints answer the same 2000 question-audio pairs in this population. The full 12000 pairs cover3000 clips (1000each music, sound, speech), with two context and two knowledge questions per clip. A response is hallucinated if it makes an unsupported claim; grounded uncertainty, premise correction or refusal is accepted. Qwen3.8-27B uses question-specific rubrics, audio-derived description, and optional agenticweb evidence forknowledge. Seven researchers annotate1000responses to guide/refine rubrics andprompts; this is a judge-assistedgroundedness score, not factualQA accuracy. Lower is safer butdoes not rewardanswer usefulness. Provider-native generation settings remain published configurations; mitigation variants are separate.","zh":"十个独立音频语言模型回答本组相同2000条音频问题。全体12000题来自3000片段，音乐、环境声音、语音各1000段，每段含两条上下文和两条知识问题。完整回答只要含无支持断言即判幻觉，合理不确定、纠正前提或拒答可判有依据。Qwen3.8-27B读取逐题rubric和音频描述，知识题还可检索外部证据。七位研究者标注1000条回答以指导和修订rubric及提示；此为评审辅助的依据性指标，分数低不等于回答更有用。原生生成设置保留为配置，未混入缓解策略。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.33893v1","published_at":"2026-09-27","status":"paper_baselines"},"higher_is_better":false,"model_count":10,"result_count":10,"results":[[13587,0.718,370,13588,13589],[7382,0.568,370,13588,13589],[13590,0.705,370,13588,13589],[7728,0.777,370,13588,13589],[13591,0.852,370,13588,13589],[13592,0.872,370,13588,13589],[13593,0.845,370,13588,13589],[11143,0.883,370,13588,13589],[7223,0.721,370,13588,13589],[3386,0.736,370,13588,13589]]},{"id":"mishap-music-knowledge-2026-paper","name":"MISHAP-Bench ·music-knowledge (2026 paper)","group":"知识与推理","metric":"Unsupported-claim hallucination rate","unit":"%","public_history":true,"protocol":{"en":"Ten independent audio-language model checkpoints answer the same 2000 question-audio pairs in this population. The full 12000 pairs cover3000 clips (1000each music, sound, speech), with two context and two knowledge questions per clip. A response is hallucinated if it makes an unsupported claim; grounded uncertainty, premise correction or refusal is accepted. Qwen3.8-27B uses question-specific rubrics, audio-derived description, and optional agenticweb evidence forknowledge. Seven researchers annotate1000responses to guide/refine rubrics andprompts; this is a judge-assistedgroundedness score, not factualQA accuracy. Lower is safer butdoes not rewardanswer usefulness. Provider-native generation settings remain published configurations; mitigation variants are separate.","zh":"十个独立音频语言模型回答本组相同2000条音频问题。全体12000题来自3000片段，音乐、环境声音、语音各1000段，每段含两条上下文和两条知识问题。完整回答只要含无支持断言即判幻觉，合理不确定、纠正前提或拒答可判有依据。Qwen3.8-27B读取逐题rubric和音频描述，知识题还可检索外部证据。七位研究者标注1000条回答以指导和修订rubric及提示；此为评审辅助的依据性指标，分数低不等于回答更有用。原生生成设置保留为配置，未混入缓解策略。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.33893v1","published_at":"2026-09-27","status":"paper_baselines"},"higher_is_better":false,"model_count":10,"result_count":10,"results":[[13587,0.657,370,13588,13589],[7382,0.385,370,13588,13589],[13590,0.623,370,13588,13589],[7728,0.792,370,13588,13589],[13591,0.809,370,13588,13589],[13592,0.8490000000000001,370,13588,13589],[13593,0.705,370,13588,13589],[11143,0.723,370,13588,13589],[7223,0.455,370,13588,13589],[3386,0.35200000000000004,370,13588,13589]]},{"id":"mishap-music-overall-2026-paper","name":"MISHAP-Bench ·music-overall (2026 paper)","group":"知识与推理","metric":"Unsupported-claim hallucination rate","unit":"%","public_history":true,"protocol":{"en":"Ten independent audio-language model checkpoints answer the same 4000 question-audio pairs in this population. The full 12000 pairs cover3000 clips (1000each music, sound, speech), with two context and two knowledge questions per clip. A response is hallucinated if it makes an unsupported claim; grounded uncertainty, premise correction or refusal is accepted. Qwen3.8-27B uses question-specific rubrics, audio-derived description, and optional agenticweb evidence forknowledge. Seven researchers annotate1000responses to guide/refine rubrics andprompts; this is a judge-assistedgroundedness score, not factualQA accuracy. Lower is safer butdoes not rewardanswer usefulness. Provider-native generation settings remain published configurations; mitigation variants are separate.","zh":"十个独立音频语言模型回答本组相同4000条音频问题。全体12000题来自3000片段，音乐、环境声音、语音各1000段，每段含两条上下文和两条知识问题。完整回答只要含无支持断言即判幻觉，合理不确定、纠正前提或拒答可判有依据。Qwen3.8-27B读取逐题rubric和音频描述，知识题还可检索外部证据。七位研究者标注1000条回答以指导和修订rubric及提示；此为评审辅助的依据性指标，分数低不等于回答更有用。原生生成设置保留为配置，未混入缓解策略。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.33893v1","published_at":"2026-09-27","status":"paper_baselines"},"higher_is_better":false,"model_count":10,"result_count":10,"results":[[13587,0.688,370,13594,13589],[7382,0.47600000000000003,370,13594,13589],[13590,0.664,370,13594,13589],[7728,0.784,370,13594,13589],[13591,0.83,370,13594,13589],[13592,0.86,370,13594,13589],[13593,0.775,370,13594,13589],[11143,0.8029999999999999,370,13594,13589],[7223,0.588,370,13594,13589],[3386,0.544,370,13594,13589]]},{"id":"mishap-sound-context-2026-paper","name":"MISHAP-Bench ·sound-context (2026 paper)","group":"知识与推理","metric":"Unsupported-claim hallucination rate","unit":"%","public_history":true,"protocol":{"en":"Ten independent audio-language model checkpoints answer the same 2000 question-audio pairs in this population. The full 12000 pairs cover3000 clips (1000each music, sound, speech), with two context and two knowledge questions per clip. A response is hallucinated if it makes an unsupported claim; grounded uncertainty, premise correction or refusal is accepted. Qwen3.8-27B uses question-specific rubrics, audio-derived description, and optional agenticweb evidence forknowledge. Seven researchers annotate1000responses to guide/refine rubrics andprompts; this is a judge-assistedgroundedness score, not factualQA accuracy. Lower is safer butdoes not rewardanswer usefulness. Provider-native generation settings remain published configurations; mitigation variants are separate.","zh":"十个独立音频语言模型回答本组相同2000条音频问题。全体12000题来自3000片段，音乐、环境声音、语音各1000段，每段含两条上下文和两条知识问题。完整回答只要含无支持断言即判幻觉，合理不确定、纠正前提或拒答可判有依据。Qwen3.8-27B读取逐题rubric和音频描述，知识题还可检索外部证据。七位研究者标注1000条回答以指导和修订rubric及提示；此为评审辅助的依据性指标，分数低不等于回答更有用。原生生成设置保留为配置，未混入缓解策略。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.33893v1","published_at":"2026-09-27","status":"paper_baselines"},"higher_is_better":false,"model_count":10,"result_count":10,"results":[[13587,0.7240000000000001,370,13588,13589],[7382,0.506,370,13588,13589],[13590,0.718,370,13588,13589],[7728,0.6940000000000001,370,13588,13589],[13591,0.76,370,13588,13589],[13592,0.845,370,13588,13589],[13593,0.8220000000000001,370,13588,13589],[11143,0.843,370,13588,13589],[7223,0.657,370,13588,13589],[3386,0.626,370,13588,13589]]},{"id":"mishap-sound-knowledge-2026-paper","name":"MISHAP-Bench ·sound-knowledge (2026 paper)","group":"知识与推理","metric":"Unsupported-claim hallucination rate","unit":"%","public_history":true,"protocol":{"en":"Ten independent audio-language model checkpoints answer the same 2000 question-audio pairs in this population. The full 12000 pairs cover3000 clips (1000each music, sound, speech), with two context and two knowledge questions per clip. A response is hallucinated if it makes an unsupported claim; grounded uncertainty, premise correction or refusal is accepted. Qwen3.8-27B uses question-specific rubrics, audio-derived description, and optional agenticweb evidence forknowledge. Seven researchers annotate1000responses to guide/refine rubrics andprompts; this is a judge-assistedgroundedness score, not factualQA accuracy. Lower is safer butdoes not rewardanswer usefulness. Provider-native generation settings remain published configurations; mitigation variants are separate.","zh":"十个独立音频语言模型回答本组相同2000条音频问题。全体12000题来自3000片段，音乐、环境声音、语音各1000段，每段含两条上下文和两条知识问题。完整回答只要含无支持断言即判幻觉，合理不确定、纠正前提或拒答可判有依据。Qwen3.8-27B读取逐题rubric和音频描述，知识题还可检索外部证据。七位研究者标注1000条回答以指导和修订rubric及提示；此为评审辅助的依据性指标，分数低不等于回答更有用。原生生成设置保留为配置，未混入缓解策略。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.33893v1","published_at":"2026-09-27","status":"paper_baselines"},"higher_is_better":false,"model_count":10,"result_count":10,"results":[[13587,0.511,370,13588,13589],[7382,0.152,370,13588,13589],[13590,0.322,370,13588,13589],[7728,0.465,370,13588,13589],[13591,0.639,370,13588,13589],[13592,0.602,370,13588,13589],[13593,0.496,370,13588,13589],[11143,0.331,370,13588,13589],[7223,0.282,370,13588,13589],[3386,0.183,370,13588,13589]]},{"id":"mishap-sound-overall-2026-paper","name":"MISHAP-Bench ·sound-overall (2026 paper)","group":"知识与推理","metric":"Unsupported-claim hallucination rate","unit":"%","public_history":true,"protocol":{"en":"Ten independent audio-language model checkpoints answer the same 4000 question-audio pairs in this population. The full 12000 pairs cover3000 clips (1000each music, sound, speech), with two context and two knowledge questions per clip. A response is hallucinated if it makes an unsupported claim; grounded uncertainty, premise correction or refusal is accepted. Qwen3.8-27B uses question-specific rubrics, audio-derived description, and optional agenticweb evidence forknowledge. Seven researchers annotate1000responses to guide/refine rubrics andprompts; this is a judge-assistedgroundedness score, not factualQA accuracy. Lower is safer butdoes not rewardanswer usefulness. Provider-native generation settings remain published configurations; mitigation variants are separate.","zh":"十个独立音频语言模型回答本组相同4000条音频问题。全体12000题来自3000片段，音乐、环境声音、语音各1000段，每段含两条上下文和两条知识问题。完整回答只要含无支持断言即判幻觉，合理不确定、纠正前提或拒答可判有依据。Qwen3.8-27B读取逐题rubric和音频描述，知识题还可检索外部证据。七位研究者标注1000条回答以指导和修订rubric及提示；此为评审辅助的依据性指标，分数低不等于回答更有用。原生生成设置保留为配置，未混入缓解策略。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.33893v1","published_at":"2026-09-27","status":"paper_baselines"},"higher_is_better":false,"model_count":10,"result_count":10,"results":[[13587,0.618,370,13594,13589],[7382,0.33,370,13594,13589],[13590,0.52,370,13594,13589],[7728,0.58,370,13594,13589],[13591,0.7,370,13594,13589],[13592,0.7240000000000001,370,13594,13589],[13593,0.66,370,13594,13589],[11143,0.5870000000000001,370,13594,13589],[7223,0.469,370,13594,13589],[3386,0.40399999999999997,370,13594,13589]]},{"id":"mishap-speech-context-2026-paper","name":"MISHAP-Bench ·speech-context (2026 paper)","group":"知识与推理","metric":"Unsupported-claim hallucination rate","unit":"%","public_history":true,"protocol":{"en":"Ten independent audio-language model checkpoints answer the same 2000 question-audio pairs in this population. The full 12000 pairs cover3000 clips (1000each music, sound, speech), with two context and two knowledge questions per clip. A response is hallucinated if it makes an unsupported claim; grounded uncertainty, premise correction or refusal is accepted. Qwen3.8-27B uses question-specific rubrics, audio-derived description, and optional agenticweb evidence forknowledge. Seven researchers annotate1000responses to guide/refine rubrics andprompts; this is a judge-assistedgroundedness score, not factualQA accuracy. Lower is safer butdoes not rewardanswer usefulness. Provider-native generation settings remain published configurations; mitigation variants are separate.","zh":"十个独立音频语言模型回答本组相同2000条音频问题。全体12000题来自3000片段，音乐、环境声音、语音各1000段，每段含两条上下文和两条知识问题。完整回答只要含无支持断言即判幻觉，合理不确定、纠正前提或拒答可判有依据。Qwen3.8-27B读取逐题rubric和音频描述，知识题还可检索外部证据。七位研究者标注1000条回答以指导和修订rubric及提示；此为评审辅助的依据性指标，分数低不等于回答更有用。原生生成设置保留为配置，未混入缓解策略。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.33893v1","published_at":"2026-09-27","status":"paper_baselines"},"higher_is_better":false,"model_count":10,"result_count":10,"results":[[13587,0.294,370,13588,13589],[7382,0.37200000000000005,370,13588,13589],[13590,0.309,370,13588,13589],[7728,0.485,370,13588,13589],[13591,0.58,370,13588,13589],[13592,0.639,370,13588,13589],[13593,0.38,370,13588,13589],[11143,0.466,370,13588,13589],[7223,0.28600000000000003,370,13588,13589],[3386,0.095,370,13588,13589]]},{"id":"mishap-speech-knowledge-2026-paper","name":"MISHAP-Bench ·speech-knowledge (2026 paper)","group":"知识与推理","metric":"Unsupported-claim hallucination rate","unit":"%","public_history":true,"protocol":{"en":"Ten independent audio-language model checkpoints answer the same 2000 question-audio pairs in this population. The full 12000 pairs cover3000 clips (1000each music, sound, speech), with two context and two knowledge questions per clip. A response is hallucinated if it makes an unsupported claim; grounded uncertainty, premise correction or refusal is accepted. Qwen3.8-27B uses question-specific rubrics, audio-derived description, and optional agenticweb evidence forknowledge. Seven researchers annotate1000responses to guide/refine rubrics andprompts; this is a judge-assistedgroundedness score, not factualQA accuracy. Lower is safer butdoes not rewardanswer usefulness. Provider-native generation settings remain published configurations; mitigation variants are separate.","zh":"十个独立音频语言模型回答本组相同2000条音频问题。全体12000题来自3000片段，音乐、环境声音、语音各1000段，每段含两条上下文和两条知识问题。完整回答只要含无支持断言即判幻觉，合理不确定、纠正前提或拒答可判有依据。Qwen3.8-27B读取逐题rubric和音频描述，知识题还可检索外部证据。七位研究者标注1000条回答以指导和修订rubric及提示；此为评审辅助的依据性指标，分数低不等于回答更有用。原生生成设置保留为配置，未混入缓解策略。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.33893v1","published_at":"2026-09-27","status":"paper_baselines"},"higher_is_better":false,"model_count":10,"result_count":10,"results":[[13587,0.313,370,13588,13589],[7382,0.18100000000000002,370,13588,13589],[13590,0.354,370,13588,13589],[7728,0.638,370,13588,13589],[13591,0.6679999999999999,370,13588,13589],[13592,0.6829999999999999,370,13588,13589],[13593,0.568,370,13588,13589],[11143,0.43,370,13588,13589],[7223,0.45,370,13588,13589],[3386,0.19899999999999998,370,13588,13589]]},{"id":"mishap-speech-overall-2026-paper","name":"MISHAP-Bench ·speech-overall (2026 paper)","group":"知识与推理","metric":"Unsupported-claim hallucination rate","unit":"%","public_history":true,"protocol":{"en":"Ten independent audio-language model checkpoints answer the same 4000 question-audio pairs in this population. The full 12000 pairs cover3000 clips (1000each music, sound, speech), with two context and two knowledge questions per clip. A response is hallucinated if it makes an unsupported claim; grounded uncertainty, premise correction or refusal is accepted. Qwen3.8-27B uses question-specific rubrics, audio-derived description, and optional agenticweb evidence forknowledge. Seven researchers annotate1000responses to guide/refine rubrics andprompts; this is a judge-assistedgroundedness score, not factualQA accuracy. Lower is safer butdoes not rewardanswer usefulness. Provider-native generation settings remain published configurations; mitigation variants are separate.","zh":"十个独立音频语言模型回答本组相同4000条音频问题。全体12000题来自3000片段，音乐、环境声音、语音各1000段，每段含两条上下文和两条知识问题。完整回答只要含无支持断言即判幻觉，合理不确定、纠正前提或拒答可判有依据。Qwen3.8-27B读取逐题rubric和音频描述，知识题还可检索外部证据。七位研究者标注1000条回答以指导和修订rubric及提示；此为评审辅助的依据性指标，分数低不等于回答更有用。原生生成设置保留为配置，未混入缓解策略。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.33893v1","published_at":"2026-09-27","status":"paper_baselines"},"higher_is_better":false,"model_count":10,"result_count":10,"results":[[13587,0.304,370,13594,13589],[7382,0.27699999999999997,370,13594,13589],[13590,0.331,370,13594,13589],[7728,0.562,370,13594,13589],[13591,0.624,370,13594,13589],[13592,0.6609999999999999,370,13594,13589],[13593,0.474,370,13594,13589],[11143,0.44799999999999995,370,13594,13589],[7223,0.368,370,13594,13589],[3386,0.147,370,13594,13589]]},{"id":"mishap-overall-2026-paper","name":"MISHAP-Bench ·overall (2026 paper)","group":"知识与推理","metric":"Unsupported-claim hallucination rate","unit":"%","public_history":true,"protocol":{"en":"Ten independent audio-language model checkpoints answer the same 12000 question-audio pairs in this population. The full 12000 pairs cover3000 clips (1000each music, sound, speech), with two context and two knowledge questions per clip. A response is hallucinated if it makes an unsupported claim; grounded uncertainty, premise correction or refusal is accepted. Qwen3.8-27B uses question-specific rubrics, audio-derived description, and optional agenticweb evidence forknowledge. Seven researchers annotate1000responses to guide/refine rubrics andprompts; this is a judge-assistedgroundedness score, not factualQA accuracy. Lower is safer butdoes not rewardanswer usefulness. Provider-native generation settings remain published configurations; mitigation variants are separate.","zh":"十个独立音频语言模型回答本组相同12000条音频问题。全体12000题来自3000片段，音乐、环境声音、语音各1000段，每段含两条上下文和两条知识问题。完整回答只要含无支持断言即判幻觉，合理不确定、纠正前提或拒答可判有依据。Qwen3.8-27B读取逐题rubric和音频描述，知识题还可检索外部证据。七位研究者标注1000条回答以指导和修订rubric及提示；此为评审辅助的依据性指标，分数低不等于回答更有用。原生生成设置保留为配置，未混入缓解策略。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.33893v1","published_at":"2026-09-27","status":"paper_baselines"},"higher_is_better":false,"model_count":10,"result_count":10,"results":[[13587,0.536,370,13595,13589],[7382,0.361,370,13595,13589],[13590,0.505,370,13595,13589],[7728,0.642,370,13595,13589],[13591,0.718,370,13595,13589],[13592,0.748,370,13595,13589],[13593,0.636,370,13595,13589],[11143,0.613,370,13595,13589],[7223,0.475,370,13595,13589],[3386,0.365,370,13595,13589]]},{"id":"engintervene-state-grounding-2026-paper","name":"EngIntervene ·state-grounding (2026 paper)","group":"数学与科学","metric":"Exact choice accuracy","unit":"%","public_history":true,"protocol":{"en":"Seventeen model/configuration rows cover sixteen independent backbones on the same 2149 items of this task. Full EngIntervene contains3229 items from 357 source documents; T1/T2/T3/T4sizes are2149/420/420/240. References andatomic criteria are frozen before evaluation andaudited/revised against engineering sources. T1 uses a deterministic normalized choice-set parser. T2–T4 use separate unadapted Qwen3.5-4B greedy non-thinking criterion judgments with answer-span evidence; up tothree parse attempts then met=false fallbacks are counted. Atomic credit subtracts critical-error penalties andclips to the task cap; strict success requires every required criterion andno criticalerror. Missing IDs/infrastructureerrors block aggregation. Native atomic scores remain separate frompercent-correct. Original generation/harness settings remain configurations.","zh":"十七条模型与配置覆盖十六种独立权重，在本任务相同2149题上评测。完整3229题来自357份材料，T1/T2/T3/T4各2149/420/420/240题，参考及原子判据在评测前固定并按工程来源审查修订。T1用确定性归一化选项解析；T2–T4由独立、未适配的Qwen3.5-4B以贪心非思考方式逐判据评审，正判断须给回答中的证据片段。最多三次解析后仍失败记未满足并统计。原子分扣除关键错误后截到评分上限；严格成功要求全部必需条件满足且无关键错误。缺ID或基础设施错误会阻止汇总。原子评分保留原生刻度，生成与harness限制记录为配置。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.33261v1","published_at":"2026-09-27","status":"paper_baselines"},"model_count":16,"result_count":17,"results":[[354,0.47979999999999995,370,13596,13597],[3184,0.7268000000000001,370,13598,13597],[3408,0.6668000000000001,370,13599,13597],[3578,0.73,370,13600,13597],[13601,0.539,370,13602,13597],[4969,0.5165,370,13603,13597],[4070,0.5537,370,13604,13597],[2584,0.6031,370,13605,13597],[13606,0.6165999999999999,370,13607,13597],[13608,0.5807,370,13609,13597],[6342,0.49700000000000005,370,13610,13597],[9100,0.4611,370,13611,13597],[6342,0.49229999999999996,370,13612,13597],[2654,0.45,370,13613,13597],[6345,0.47229999999999994,370,13614,13597],[13615,0.5877,370,13616,13597],[6254,0.49979999999999997,370,13617,13597]]},{"id":"engintervene-relations-atomic-2026-paper","name":"EngIntervene ·relations-atomic (2026 paper)","group":"数学与科学","metric":"Weighted atomic engineering-rubric score","unit":"score","public_history":true,"protocol":{"en":"Seventeen model/configuration rows cover sixteen independent backbones on the same 420 items of this task. Full EngIntervene contains3229 items from 357 source documents; T1/T2/T3/T4sizes are2149/420/420/240. References andatomic criteria are frozen before evaluation andaudited/revised against engineering sources. T1 uses a deterministic normalized choice-set parser. T2–T4 use separate unadapted Qwen3.5-4B greedy non-thinking criterion judgments with answer-span evidence; up tothree parse attempts then met=false fallbacks are counted. Atomic credit subtracts critical-error penalties andclips to the task cap; strict success requires every required criterion andno criticalerror. Missing IDs/infrastructureerrors block aggregation. Native atomic scores remain separate frompercent-correct. Original generation/harness settings remain configurations.","zh":"十七条模型与配置覆盖十六种独立权重，在本任务相同420题上评测。完整3229题来自357份材料，T1/T2/T3/T4各2149/420/420/240题，参考及原子判据在评测前固定并按工程来源审查修订。T1用确定性归一化选项解析；T2–T4由独立、未适配的Qwen3.5-4B以贪心非思考方式逐判据评审，正判断须给回答中的证据片段。最多三次解析后仍失败记未满足并统计。原子分扣除关键错误后截到评分上限；严格成功要求全部必需条件满足且无关键错误。缺ID或基础设施错误会阻止汇总。原子评分保留原生刻度，生成与harness限制记录为配置。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.33261v1","published_at":"2026-09-27","status":"paper_baselines"},"score_min":0,"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Native score / 100","zh":"原生分数 / 100"},"score_decimals":2,"model_count":16,"result_count":17,"results":[[354,23.96,370,13596,13597],[3184,54.4,370,13598,13597],[3408,51.73,370,13599,13597],[3578,57.64,370,13600,13597],[13601,34.05,370,13602,13597],[4969,29.78,370,13603,13597],[4070,33.1,370,13604,13597],[2584,40.4,370,13605,13597],[13606,36.85,370,13607,13597],[13608,41.03,370,13609,13597],[6342,30.85,370,13610,13597],[9100,18.58,370,13611,13597],[6342,30.25,370,13612,13597],[2654,22.33,370,13613,13597],[6345,23.84,370,13614,13597],[13615,41.17,370,13616,13597],[6254,29.85,370,13617,13597]]},{"id":"engintervene-diagnosis-atomic-2026-paper","name":"EngIntervene ·diagnosis-atomic (2026 paper)","group":"数学与科学","metric":"Weighted atomic engineering-rubric score","unit":"score","public_history":true,"protocol":{"en":"Seventeen model/configuration rows cover sixteen independent backbones on the same 420 items of this task. Full EngIntervene contains3229 items from 357 source documents; T1/T2/T3/T4sizes are2149/420/420/240. References andatomic criteria are frozen before evaluation andaudited/revised against engineering sources. T1 uses a deterministic normalized choice-set parser. T2–T4 use separate unadapted Qwen3.5-4B greedy non-thinking criterion judgments with answer-span evidence; up tothree parse attempts then met=false fallbacks are counted. Atomic credit subtracts critical-error penalties andclips to the task cap; strict success requires every required criterion andno criticalerror. Missing IDs/infrastructureerrors block aggregation. Native atomic scores remain separate frompercent-correct. Original generation/harness settings remain configurations.","zh":"十七条模型与配置覆盖十六种独立权重，在本任务相同420题上评测。完整3229题来自357份材料，T1/T2/T3/T4各2149/420/420/240题，参考及原子判据在评测前固定并按工程来源审查修订。T1用确定性归一化选项解析；T2–T4由独立、未适配的Qwen3.5-4B以贪心非思考方式逐判据评审，正判断须给回答中的证据片段。最多三次解析后仍失败记未满足并统计。原子分扣除关键错误后截到评分上限；严格成功要求全部必需条件满足且无关键错误。缺ID或基础设施错误会阻止汇总。原子评分保留原生刻度，生成与harness限制记录为配置。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.33261v1","published_at":"2026-09-27","status":"paper_baselines"},"score_min":0,"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Native score / 100","zh":"原生分数 / 100"},"score_decimals":2,"model_count":16,"result_count":17,"results":[[354,23.01,370,13596,13597],[3184,52.33,370,13598,13597],[3408,50.15,370,13599,13597],[3578,58.41,370,13600,13597],[13601,33.96,370,13602,13597],[4969,33.32,370,13603,13597],[4070,34.54,370,13604,13597],[2584,40.93,370,13605,13597],[13606,37.81,370,13607,13597],[13608,44.9,370,13609,13597],[6342,28.61,370,13610,13597],[9100,21.85,370,13611,13597],[6342,32.64,370,13612,13597],[2654,24.02,370,13613,13597],[6345,25.29,370,13614,13597],[13615,40.89,370,13616,13597],[6254,33.21,370,13617,13597]]},{"id":"engintervene-intervention-atomic-2026-paper","name":"EngIntervene ·intervention-atomic (2026 paper)","group":"数学与科学","metric":"Weighted atomic engineering-rubric score","unit":"score","public_history":true,"protocol":{"en":"Seventeen model/configuration rows cover sixteen independent backbones on the same 240 items of this task. Full EngIntervene contains3229 items from 357 source documents; T1/T2/T3/T4sizes are2149/420/420/240. References andatomic criteria are frozen before evaluation andaudited/revised against engineering sources. T1 uses a deterministic normalized choice-set parser. T2–T4 use separate unadapted Qwen3.5-4B greedy non-thinking criterion judgments with answer-span evidence; up tothree parse attempts then met=false fallbacks are counted. Atomic credit subtracts critical-error penalties andclips to the task cap; strict success requires every required criterion andno criticalerror. Missing IDs/infrastructureerrors block aggregation. Native atomic scores remain separate frompercent-correct. Original generation/harness settings remain configurations.","zh":"十七条模型与配置覆盖十六种独立权重，在本任务相同240题上评测。完整3229题来自357份材料，T1/T2/T3/T4各2149/420/420/240题，参考及原子判据在评测前固定并按工程来源审查修订。T1用确定性归一化选项解析；T2–T4由独立、未适配的Qwen3.5-4B以贪心非思考方式逐判据评审，正判断须给回答中的证据片段。最多三次解析后仍失败记未满足并统计。原子分扣除关键错误后截到评分上限；严格成功要求全部必需条件满足且无关键错误。缺ID或基础设施错误会阻止汇总。原子评分保留原生刻度，生成与harness限制记录为配置。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.33261v1","published_at":"2026-09-27","status":"paper_baselines"},"score_min":0,"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Native score / 100","zh":"原生分数 / 100"},"score_decimals":2,"model_count":16,"result_count":17,"results":[[354,31.29,370,13596,13597],[3184,62.73,370,13598,13597],[3408,59.16,370,13599,13597],[3578,67.84,370,13600,13597],[13601,40.67,370,13602,13597],[4969,38.33,370,13603,13597],[4070,40.6,370,13604,13597],[2584,52.12,370,13605,13597],[13606,49.04,370,13607,13597],[13608,53.98,370,13609,13597],[6342,27.51,370,13610,13597],[9100,29.63,370,13611,13597],[6342,42.7,370,13612,13597],[2654,31.84,370,13613,13597],[6345,32.99,370,13614,13597],[13615,51.32,370,13616,13597],[6254,45.83,370,13617,13597]]},{"id":"engintervene-intervention-strict-2026-paper","name":"EngIntervene ·intervention-strict (2026 paper)","group":"数学与科学","metric":"All required criteria,no critical error","unit":"%","public_history":true,"protocol":{"en":"Seventeen model/configuration rows cover sixteen independent backbones on the same 240 items of this task. Full EngIntervene contains3229 items from 357 source documents; T1/T2/T3/T4sizes are2149/420/420/240. References andatomic criteria are frozen before evaluation andaudited/revised against engineering sources. T1 uses a deterministic normalized choice-set parser. T2–T4 use separate unadapted Qwen3.5-4B greedy non-thinking criterion judgments with answer-span evidence; up tothree parse attempts then met=false fallbacks are counted. Atomic credit subtracts critical-error penalties andclips to the task cap; strict success requires every required criterion andno criticalerror. Missing IDs/infrastructureerrors block aggregation. Native atomic scores remain separate frompercent-correct. Original generation/harness settings remain configurations.","zh":"十七条模型与配置覆盖十六种独立权重，在本任务相同240题上评测。完整3229题来自357份材料，T1/T2/T3/T4各2149/420/420/240题，参考及原子判据在评测前固定并按工程来源审查修订。T1用确定性归一化选项解析；T2–T4由独立、未适配的Qwen3.5-4B以贪心非思考方式逐判据评审，正判断须给回答中的证据片段。最多三次解析后仍失败记未满足并统计。原子分扣除关键错误后截到评分上限；严格成功要求全部必需条件满足且无关键错误。缺ID或基础设施错误会阻止汇总。原子评分保留原生刻度，生成与harness限制记录为配置。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.33261v1","published_at":"2026-09-27","status":"paper_baselines"},"model_count":16,"result_count":17,"results":[[354,0.075,370,13596,13597],[3184,0.24170000000000003,370,13598,13597],[3408,0.24170000000000003,370,13599,13597],[3578,0.30829999999999996,370,13600,13597],[13601,0.1208,370,13602,13597],[4969,0.0833,370,13603,13597],[4070,0.1125,370,13604,13597],[2584,0.17920000000000003,370,13605,13597],[13606,0.1333,370,13607,13597],[13608,0.1833,370,13609,13597],[6342,0.0625,370,13610,13597],[9100,0.0625,370,13611,13597],[6342,0.1125,370,13612,13597],[2654,0.0458,370,13613,13597],[6345,0.05,370,13614,13597],[13615,0.16670000000000001,370,13616,13597],[6254,0.1167,370,13617,13597]]},{"id":"actionlens-full-transition-2026-paper","name":"ActionLens ·full transition (2026 paper)","group":"视觉与视频","metric":"Question-micro exact-choice accuracy","unit":"%","public_history":true,"protocol":{"en":"The full cohort compares distinct VLM checkpoints on the same 194 fixed video-MCQ items in this diagnostic. Full 6,701 items derive from 64 source videos anddeterministic per-second/person annotations. The1194review subset contains all 194transition questions plus 250fixed-seed round-robin items perother diagnostic. Main input uses tracked colored boxes and 32 frames, with temperature0/do_samplefalse/onebeam; reference-interface controls and reasoning prompts remain separate. Four non-author raters label disjoint review items once, without model outputs, forming a pooled91% reference for thatsubset only. The question-micro aggregate weights each item equally; full andsubset populations are never pooled. Three-option interaction items retain their native chance rate.","zh":"full组各VLM共同回答本诊断相同194道视频选择题。完整6701题来自64份源视频和确定性的逐秒逐人标注；1194题复核子集含全部194道转变题及其余每类固定种子轮转抽取250题。主输入使用跟踪彩色框、32帧、温度0、不开采样、单束，其他指代界面与推理提示单独比较。四位非作者标注者各自处理不重叠复核题，不看模型结果，91%人工参照仅适用于该子集。micro总分按题等权，两种面板不合并；三选项交互题保留其原生机会水平。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34547v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":18,"result_count":18,"results":[[13618,0.603,370,13619,13620],[2584,0.649,370,13619,13620],[6343,0.634,370,13619,13620],[5480,0.639,370,13619,13620],[6391,0.66,370,13619,13620],[6689,0.67,370,13619,13620],[2642,0.68,370,13619,13620],[4070,0.649,370,13619,13620],[6342,0.619,370,13619,13620],[2654,0.603,370,13619,13620],[6349,0.716,370,13619,13620],[6389,0.67,370,13619,13620],[7803,0.536,370,13619,13620],[4969,0.603,370,13619,13620],[6298,0.68,370,13619,13620],[2687,0.619,370,13619,13620],[5650,0.521,370,13619,13620],[6677,0.485,370,13619,13620]]},{"id":"actionlens-full-actor-2026-paper","name":"ActionLens ·full actor (2026 paper)","group":"视觉与视频","metric":"Question-micro exact-choice accuracy","unit":"%","public_history":true,"protocol":{"en":"The full cohort compares distinct VLM checkpoints on the same 2000 fixed video-MCQ items in this diagnostic. Full 6,701 items derive from 64 source videos anddeterministic per-second/person annotations. The1194review subset contains all 194transition questions plus 250fixed-seed round-robin items perother diagnostic. Main input uses tracked colored boxes and 32 frames, with temperature0/do_samplefalse/onebeam; reference-interface controls and reasoning prompts remain separate. Four non-author raters label disjoint review items once, without model outputs, forming a pooled91% reference for thatsubset only. The question-micro aggregate weights each item equally; full andsubset populations are never pooled. Three-option interaction items retain their native chance rate.","zh":"full组各VLM共同回答本诊断相同2000道视频选择题。完整6701题来自64份源视频和确定性的逐秒逐人标注；1194题复核子集含全部194道转变题及其余每类固定种子轮转抽取250题。主输入使用跟踪彩色框、32帧、温度0、不开采样、单束，其他指代界面与推理提示单独比较。四位非作者标注者各自处理不重叠复核题，不看模型结果，91%人工参照仅适用于该子集。micro总分按题等权，两种面板不合并；三选项交互题保留其原生机会水平。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34547v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":18,"result_count":18,"results":[[13618,0.45,370,13619,13620],[2584,0.677,370,13619,13620],[6343,0.64,370,13619,13620],[5480,0.63,370,13619,13620],[6391,0.544,370,13619,13620],[6689,0.379,370,13619,13620],[2642,0.655,370,13619,13620],[4070,0.6459999999999999,370,13619,13620],[6342,0.574,370,13619,13620],[2654,0.5870000000000001,370,13619,13620],[6349,0.517,370,13619,13620],[6389,0.475,370,13619,13620],[7803,0.34,370,13619,13620],[4969,0.66,370,13619,13620],[6298,0.6,370,13619,13620],[2687,0.5479999999999999,370,13619,13620],[5650,0.479,370,13619,13620],[6677,0.307,370,13619,13620]]},{"id":"actionlens-full-concurrent-2026-paper","name":"ActionLens ·full concurrent (2026 paper)","group":"视觉与视频","metric":"Question-micro exact-choice accuracy","unit":"%","public_history":true,"protocol":{"en":"The full cohort compares distinct VLM checkpoints on the same 2000 fixed video-MCQ items in this diagnostic. Full 6,701 items derive from 64 source videos anddeterministic per-second/person annotations. The1194review subset contains all 194transition questions plus 250fixed-seed round-robin items perother diagnostic. Main input uses tracked colored boxes and 32 frames, with temperature0/do_samplefalse/onebeam; reference-interface controls and reasoning prompts remain separate. Four non-author raters label disjoint review items once, without model outputs, forming a pooled91% reference for thatsubset only. The question-micro aggregate weights each item equally; full andsubset populations are never pooled. Three-option interaction items retain their native chance rate.","zh":"full组各VLM共同回答本诊断相同2000道视频选择题。完整6701题来自64份源视频和确定性的逐秒逐人标注；1194题复核子集含全部194道转变题及其余每类固定种子轮转抽取250题。主输入使用跟踪彩色框、32帧、温度0、不开采样、单束，其他指代界面与推理提示单独比较。四位非作者标注者各自处理不重叠复核题，不看模型结果，91%人工参照仅适用于该子集。micro总分按题等权，两种面板不合并；三选项交互题保留其原生机会水平。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34547v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":18,"result_count":18,"results":[[13618,0.42,370,13619,13620],[2584,0.748,370,13619,13620],[6343,0.669,370,13619,13620],[5480,0.7290000000000001,370,13619,13620],[6391,0.705,370,13619,13620],[6689,0.35100000000000003,370,13619,13620],[2642,0.684,370,13619,13620],[4070,0.691,370,13619,13620],[6342,0.7090000000000001,370,13619,13620],[2654,0.655,370,13619,13620],[6349,0.526,370,13619,13620],[6389,0.5379999999999999,370,13619,13620],[7803,0.539,370,13619,13620],[4969,0.6990000000000001,370,13619,13620],[6298,0.6759999999999999,370,13619,13620],[2687,0.662,370,13619,13620],[5650,0.6459999999999999,370,13619,13620],[6677,0.272,370,13619,13620]]},{"id":"actionlens-full-interaction-2026-paper","name":"ActionLens ·full interaction (2026 paper)","group":"视觉与视频","metric":"Question-micro exact-choice accuracy","unit":"%","public_history":true,"protocol":{"en":"The full cohort compares distinct VLM checkpoints on the same 1611 fixed video-MCQ items in this diagnostic. Full 6,701 items derive from 64 source videos anddeterministic per-second/person annotations. The1194review subset contains all 194transition questions plus 250fixed-seed round-robin items perother diagnostic. Main input uses tracked colored boxes and 32 frames, with temperature0/do_samplefalse/onebeam; reference-interface controls and reasoning prompts remain separate. Four non-author raters label disjoint review items once, without model outputs, forming a pooled91% reference for thatsubset only. The question-micro aggregate weights each item equally; full andsubset populations are never pooled. Three-option interaction items retain their native chance rate.","zh":"full组各VLM共同回答本诊断相同1611道视频选择题。完整6701题来自64份源视频和确定性的逐秒逐人标注；1194题复核子集含全部194道转变题及其余每类固定种子轮转抽取250题。主输入使用跟踪彩色框、32帧、温度0、不开采样、单束，其他指代界面与推理提示单独比较。四位非作者标注者各自处理不重叠复核题，不看模型结果，91%人工参照仅适用于该子集。micro总分按题等权，两种面板不合并；三选项交互题保留其原生机会水平。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34547v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":18,"result_count":18,"results":[[13618,0.51,370,13619,13620],[2584,0.773,370,13619,13620],[6343,0.688,370,13619,13620],[5480,0.616,370,13619,13620],[6391,0.647,370,13619,13620],[6689,0.264,370,13619,13620],[2642,0.687,370,13619,13620],[4070,0.703,370,13619,13620],[6342,0.662,370,13619,13620],[2654,0.597,370,13619,13620],[6349,0.584,370,13619,13620],[6389,0.534,370,13619,13620],[7803,0.392,370,13619,13620],[4969,0.69,370,13619,13620],[6298,0.634,370,13619,13620],[2687,0.525,370,13619,13620],[5650,0.43200000000000005,370,13619,13620],[6677,0.243,370,13619,13620]]},{"id":"actionlens-full-gaze-2026-paper","name":"ActionLens ·full gaze (2026 paper)","group":"视觉与视频","metric":"Question-micro exact-choice accuracy","unit":"%","public_history":true,"protocol":{"en":"The full cohort compares distinct VLM checkpoints on the same 896 fixed video-MCQ items in this diagnostic. Full 6,701 items derive from 64 source videos anddeterministic per-second/person annotations. The1194review subset contains all 194transition questions plus 250fixed-seed round-robin items perother diagnostic. Main input uses tracked colored boxes and 32 frames, with temperature0/do_samplefalse/onebeam; reference-interface controls and reasoning prompts remain separate. Four non-author raters label disjoint review items once, without model outputs, forming a pooled91% reference for thatsubset only. The question-micro aggregate weights each item equally; full andsubset populations are never pooled. Three-option interaction items retain their native chance rate.","zh":"full组各VLM共同回答本诊断相同896道视频选择题。完整6701题来自64份源视频和确定性的逐秒逐人标注；1194题复核子集含全部194道转变题及其余每类固定种子轮转抽取250题。主输入使用跟踪彩色框、32帧、温度0、不开采样、单束，其他指代界面与推理提示单独比较。四位非作者标注者各自处理不重叠复核题，不看模型结果，91%人工参照仅适用于该子集。micro总分按题等权，两种面板不合并；三选项交互题保留其原生机会水平。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34547v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":18,"result_count":18,"results":[[13618,0.287,370,13619,13620],[2584,0.433,370,13619,13620],[6343,0.405,370,13619,13620],[5480,0.42,370,13619,13620],[6391,0.34299999999999997,370,13619,13620],[6689,0.258,370,13619,13620],[2642,0.386,370,13619,13620],[4070,0.376,370,13619,13620],[6342,0.308,370,13619,13620],[2654,0.336,370,13619,13620],[6349,0.273,370,13619,13620],[6389,0.278,370,13619,13620],[7803,0.25,370,13619,13620],[4969,0.353,370,13619,13620],[6298,0.301,370,13619,13620],[2687,0.316,370,13619,13620],[5650,0.282,370,13619,13620],[6677,0.242,370,13619,13620]]},{"id":"actionlens-full-micro-2026-paper","name":"ActionLens ·full micro (2026 paper)","group":"视觉与视频","metric":"Question-micro exact-choice accuracy","unit":"%","public_history":true,"protocol":{"en":"The full cohort compares distinct VLM checkpoints on the same 6701 fixed video-MCQ items in this diagnostic. Full 6,701 items derive from 64 source videos anddeterministic per-second/person annotations. The1194review subset contains all 194transition questions plus 250fixed-seed round-robin items perother diagnostic. Main input uses tracked colored boxes and 32 frames, with temperature0/do_samplefalse/onebeam; reference-interface controls and reasoning prompts remain separate. Four non-author raters label disjoint review items once, without model outputs, forming a pooled91% reference for thatsubset only. The question-micro aggregate weights each item equally; full andsubset populations are never pooled. Three-option interaction items retain their native chance rate.","zh":"full组各VLM共同回答本诊断相同6701道视频选择题。完整6701题来自64份源视频和确定性的逐秒逐人标注；1194题复核子集含全部194道转变题及其余每类固定种子轮转抽取250题。主输入使用跟踪彩色框、32帧、温度0、不开采样、单束，其他指代界面与推理提示单独比较。四位非作者标注者各自处理不重叠复核题，不看模型结果，91%人工参照仅适用于该子集。micro总分按题等权，两种面板不合并；三选项交互题保留其原生机会水平。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34547v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":18,"result_count":18,"results":[[13618,0.43799999999999994,370,13619,13620],[2584,0.688,370,13619,13620],[6343,0.628,370,13619,13620],[5480,0.628,370,13619,13620],[6391,0.593,370,13619,13620],[6689,0.335,370,13619,13620],[2642,0.636,370,13619,13620],[4070,0.637,370,13619,13620],[6342,0.601,370,13619,13620],[2654,0.5760000000000001,370,13619,13620],[6349,0.509,370,13619,13620],[6389,0.48700000000000004,370,13619,13620],[7803,0.406,370,13619,13620],[4969,0.636,370,13619,13620],[6298,0.593,370,13619,13620],[2687,0.547,370,13619,13620],[5650,0.49200000000000005,370,13619,13620],[6677,0.27699999999999997,370,13619,13620]]},{"id":"actionlens-review-transition-2026-paper","name":"ActionLens ·review transition (2026 paper)","group":"视觉与视频","metric":"Question-micro exact-choice accuracy","unit":"%","public_history":true,"protocol":{"en":"The review cohort compares distinct VLM checkpoints on the same 194 fixed video-MCQ items in this diagnostic. Full 6,701 items derive from 64 source videos anddeterministic per-second/person annotations. The1194review subset contains all 194transition questions plus 250fixed-seed round-robin items perother diagnostic. Main input uses tracked colored boxes and 32 frames, with temperature0/do_samplefalse/onebeam; reference-interface controls and reasoning prompts remain separate. Four non-author raters label disjoint review items once, without model outputs, forming a pooled91% reference for thatsubset only. The question-micro aggregate weights each item equally; full andsubset populations are never pooled. Three-option interaction items retain their native chance rate.","zh":"review组各VLM共同回答本诊断相同194道视频选择题。完整6701题来自64份源视频和确定性的逐秒逐人标注；1194题复核子集含全部194道转变题及其余每类固定种子轮转抽取250题。主输入使用跟踪彩色框、32帧、温度0、不开采样、单束，其他指代界面与推理提示单独比较。四位非作者标注者各自处理不重叠复核题，不看模型结果，91%人工参照仅适用于该子集。micro总分按题等权，两种面板不合并；三选项交互题保留其原生机会水平。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34547v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":8,"result_count":8,"results":[[340,0.688,370,13621,13620],[3424,0.583,370,13621,13620],[5480,0.639,370,13621,13620],[13618,0.603,370,13621,13620],[2642,0.68,370,13621,13620],[2584,0.649,370,13621,13620],[2687,0.619,370,13621,13620],[4969,0.603,370,13621,13620]]},{"id":"actionlens-review-actor-2026-paper","name":"ActionLens ·review actor (2026 paper)","group":"视觉与视频","metric":"Question-micro exact-choice accuracy","unit":"%","public_history":true,"protocol":{"en":"The review cohort compares distinct VLM checkpoints on the same 250 fixed video-MCQ items in this diagnostic. Full 6,701 items derive from 64 source videos anddeterministic per-second/person annotations. The1194review subset contains all 194transition questions plus 250fixed-seed round-robin items perother diagnostic. Main input uses tracked colored boxes and 32 frames, with temperature0/do_samplefalse/onebeam; reference-interface controls and reasoning prompts remain separate. Four non-author raters label disjoint review items once, without model outputs, forming a pooled91% reference for thatsubset only. The question-micro aggregate weights each item equally; full andsubset populations are never pooled. Three-option interaction items retain their native chance rate.","zh":"review组各VLM共同回答本诊断相同250道视频选择题。完整6701题来自64份源视频和确定性的逐秒逐人标注；1194题复核子集含全部194道转变题及其余每类固定种子轮转抽取250题。主输入使用跟踪彩色框、32帧、温度0、不开采样、单束，其他指代界面与推理提示单独比较。四位非作者标注者各自处理不重叠复核题，不看模型结果，91%人工参照仅适用于该子集。micro总分按题等权，两种面板不合并；三选项交互题保留其原生机会水平。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34547v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":8,"result_count":8,"results":[[340,0.76,370,13621,13620],[3424,0.544,370,13621,13620],[5480,0.66,370,13621,13620],[13618,0.47200000000000003,370,13621,13620],[2642,0.728,370,13621,13620],[2584,0.7,370,13621,13620],[2687,0.604,370,13621,13620],[4969,0.672,370,13621,13620]]},{"id":"actionlens-review-concurrent-2026-paper","name":"ActionLens ·review concurrent (2026 paper)","group":"视觉与视频","metric":"Question-micro exact-choice accuracy","unit":"%","public_history":true,"protocol":{"en":"The review cohort compares distinct VLM checkpoints on the same 250 fixed video-MCQ items in this diagnostic. Full 6,701 items derive from 64 source videos anddeterministic per-second/person annotations. The1194review subset contains all 194transition questions plus 250fixed-seed round-robin items perother diagnostic. Main input uses tracked colored boxes and 32 frames, with temperature0/do_samplefalse/onebeam; reference-interface controls and reasoning prompts remain separate. Four non-author raters label disjoint review items once, without model outputs, forming a pooled91% reference for thatsubset only. The question-micro aggregate weights each item equally; full andsubset populations are never pooled. Three-option interaction items retain their native chance rate.","zh":"review组各VLM共同回答本诊断相同250道视频选择题。完整6701题来自64份源视频和确定性的逐秒逐人标注；1194题复核子集含全部194道转变题及其余每类固定种子轮转抽取250题。主输入使用跟踪彩色框、32帧、温度0、不开采样、单束，其他指代界面与推理提示单独比较。四位非作者标注者各自处理不重叠复核题，不看模型结果，91%人工参照仅适用于该子集。micro总分按题等权，两种面板不合并；三选项交互题保留其原生机会水平。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34547v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":8,"result_count":8,"results":[[340,0.764,370,13621,13620],[3424,0.524,370,13621,13620],[5480,0.7440000000000001,370,13621,13620],[13618,0.408,370,13621,13620],[2642,0.6759999999999999,370,13621,13620],[2584,0.784,370,13621,13620],[2687,0.684,370,13621,13620],[4969,0.74,370,13621,13620]]},{"id":"actionlens-review-interaction-2026-paper","name":"ActionLens ·review interaction (2026 paper)","group":"视觉与视频","metric":"Question-micro exact-choice accuracy","unit":"%","public_history":true,"protocol":{"en":"The review cohort compares distinct VLM checkpoints on the same 250 fixed video-MCQ items in this diagnostic. Full 6,701 items derive from 64 source videos anddeterministic per-second/person annotations. The1194review subset contains all 194transition questions plus 250fixed-seed round-robin items perother diagnostic. Main input uses tracked colored boxes and 32 frames, with temperature0/do_samplefalse/onebeam; reference-interface controls and reasoning prompts remain separate. Four non-author raters label disjoint review items once, without model outputs, forming a pooled91% reference for thatsubset only. The question-micro aggregate weights each item equally; full andsubset populations are never pooled. Three-option interaction items retain their native chance rate.","zh":"review组各VLM共同回答本诊断相同250道视频选择题。完整6701题来自64份源视频和确定性的逐秒逐人标注；1194题复核子集含全部194道转变题及其余每类固定种子轮转抽取250题。主输入使用跟踪彩色框、32帧、温度0、不开采样、单束，其他指代界面与推理提示单独比较。四位非作者标注者各自处理不重叠复核题，不看模型结果，91%人工参照仅适用于该子集。micro总分按题等权，两种面板不合并；三选项交互题保留其原生机会水平。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34547v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":8,"result_count":8,"results":[[340,0.7,370,13621,13620],[3424,0.276,370,13621,13620],[5480,0.616,370,13621,13620],[13618,0.47600000000000003,370,13621,13620],[2642,0.684,370,13621,13620],[2584,0.764,370,13621,13620],[2687,0.46799999999999997,370,13621,13620],[4969,0.6679999999999999,370,13621,13620]]},{"id":"actionlens-review-gaze-2026-paper","name":"ActionLens ·review gaze (2026 paper)","group":"视觉与视频","metric":"Question-micro exact-choice accuracy","unit":"%","public_history":true,"protocol":{"en":"The review cohort compares distinct VLM checkpoints on the same 250 fixed video-MCQ items in this diagnostic. Full 6,701 items derive from 64 source videos anddeterministic per-second/person annotations. The1194review subset contains all 194transition questions plus 250fixed-seed round-robin items perother diagnostic. Main input uses tracked colored boxes and 32 frames, with temperature0/do_samplefalse/onebeam; reference-interface controls and reasoning prompts remain separate. Four non-author raters label disjoint review items once, without model outputs, forming a pooled91% reference for thatsubset only. The question-micro aggregate weights each item equally; full andsubset populations are never pooled. Three-option interaction items retain their native chance rate.","zh":"review组各VLM共同回答本诊断相同250道视频选择题。完整6701题来自64份源视频和确定性的逐秒逐人标注；1194题复核子集含全部194道转变题及其余每类固定种子轮转抽取250题。主输入使用跟踪彩色框、32帧、温度0、不开采样、单束，其他指代界面与推理提示单独比较。四位非作者标注者各自处理不重叠复核题，不看模型结果，91%人工参照仅适用于该子集。micro总分按题等权，两种面板不合并；三选项交互题保留其原生机会水平。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34547v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":8,"result_count":8,"results":[[340,0.47200000000000003,370,13621,13620],[3424,0.332,370,13621,13620],[5480,0.46,370,13621,13620],[13618,0.284,370,13621,13620],[2642,0.312,370,13621,13620],[2584,0.396,370,13621,13620],[2687,0.23600000000000002,370,13621,13620],[4969,0.336,370,13621,13620]]},{"id":"actionlens-review-micro-2026-paper","name":"ActionLens ·review micro (2026 paper)","group":"视觉与视频","metric":"Question-micro exact-choice accuracy","unit":"%","public_history":true,"protocol":{"en":"The review cohort compares distinct VLM checkpoints on the same 1194 fixed video-MCQ items in this diagnostic. Full 6,701 items derive from 64 source videos anddeterministic per-second/person annotations. The1194review subset contains all 194transition questions plus 250fixed-seed round-robin items perother diagnostic. Main input uses tracked colored boxes and 32 frames, with temperature0/do_samplefalse/onebeam; reference-interface controls and reasoning prompts remain separate. Four non-author raters label disjoint review items once, without model outputs, forming a pooled91% reference for thatsubset only. The question-micro aggregate weights each item equally; full andsubset populations are never pooled. Three-option interaction items retain their native chance rate.","zh":"review组各VLM共同回答本诊断相同1194道视频选择题。完整6701题来自64份源视频和确定性的逐秒逐人标注；1194题复核子集含全部194道转变题及其余每类固定种子轮转抽取250题。主输入使用跟踪彩色框、32帧、温度0、不开采样、单束，其他指代界面与推理提示单独比较。四位非作者标注者各自处理不重叠复核题，不看模型结果，91%人工参照仅适用于该子集。micro总分按题等权，两种面板不合并；三选项交互题保留其原生机会水平。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34547v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":8,"result_count":8,"results":[[340,0.6759999999999999,370,13621,13620],[3424,0.446,370,13621,13620],[5480,0.623,370,13621,13620],[13618,0.441,370,13621,13620],[2642,0.613,370,13621,13620],[2584,0.659,370,13621,13620],[2687,0.518,370,13621,13620],[4969,0.604,370,13621,13620]]},{"id":"agenthop-outcome-2026-paper","name":"AgentHop (2026 paper)","group":"Agent 与工具","metric":"Full sandbox answer accuracy","unit":"%","public_history":true,"protocol":{"en":"Nineteen reported model rows share1011four-option scientific questions with the same seven-tool citation sandbox. Each trajectory is limited to 20assistant turns,200000combined input/output tokens, and 30tool-budget points (read_section costs 5). Exhausting any cap without submission is incorrect. Hard API failures allow up totwo sample retries; malformed tools return errors within the trajectory. Canonical option parsing determines correctness. Recall/conversion andresource fingerprints are separate diagnostics with conditional denominators and are not imported. Three-checkpoint repeated-run audit is separate from the main one-artifact-per-model comparison. DeepSeek reasoning/chat modes retain a shared conservative model identity.","zh":"十九条模型结果共享1011道四选一科学问题及相同七工具引文沙箱，每轨迹最多20次助理轮、200000个输入输出总词元、30点工具预算（read_section耗5点），耗尽仍未提交记错。硬API失败最多再试两次，工具格式错误在原轨迹中返回。规范化选项解析确定正误。召回、条件转换率和资源行为使用另行分母，未当任务正确率导入。三权重重复运行审计与主面板分开，DeepSeek推理和聊天模式保守作同一模型身份。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34428v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":18,"result_count":19,"results":[[13622,0.891,370,13623,13624],[6304,0.547,370,13623,13624],[13625,0.825,370,13623,13624],[3184,0.79,370,13623,13624],[2261,0.775,370,13623,13624],[3572,0.766,370,13623,13624],[3186,0.696,370,13623,13624],[25,0.596,370,13626,13624],[25,0.564,370,13627,13624],[2998,0.441,370,13623,13624],[3190,0.413,370,13623,13624],[5480,0.648,370,13623,13624],[5635,0.236,370,13623,13624],[6689,0.186,370,13623,13624],[2584,0.482,370,13623,13624],[3764,0.381,370,13623,13624],[2590,0.32,370,13623,13624],[793,0.16,370,13623,13624],[786,0.133,370,13623,13624]]},{"id":"anyapp-finance-1-aw-inherited-2026-paper","name":"AnyAppBench ·Pro Expense aw-inherited (2026 paper)","group":"Agent 与工具","metric":"Deterministic finance task success","unit":"%","public_history":true,"protocol":{"en":"Seven independent direct model backbones share the same five aw-inherited finance templates on Pro Expense, each run three times (15 episodes/model). These cells form a complete fixed app/template cohort; the 520pair fullbenchmark and infrastructure-filtered five-model subgoal experiment are separate. Agents use the same emulator, six-primitive actions, task parameters, initialization, semantic success condition andtemplate-specific step limit. An app-specific deterministic verifier checks durable or UI state; VLM judges only diagnose failures and do not determine success. Planner-grounder hybrids andmulti-modelMobileAgent are excluded; conservative backbone selection omits overlapping GUI-trained bases. App-level/tier-specific rates are retained rather than averaging incompatible app populations.","zh":"七个独立直接模型共享Pro Expense相同五个aw-inherited财务任务模板，每模板重复三次，即每模型15条情境。本组为完整固定应用与模板面板，与520任务应用对全基准及按基础设施有效性筛选的五模型subgoal实验分开。模拟器、六种基础动作、参数、初始化、语义成功条件和逐模板步数上限一致。应用专用确定性验证器检查持久数据或界面状态，VLM只诊断失败而不决定任务成功。排除规划与定位混合及多模型MobileAgent；保守选择不同基模型。保留应用与模板来源组，不混合不同应用分母。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34139v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":7,"result_count":7,"results":[[2451,0.33299999999999996,370,13628,13629],[13630,0.9329999999999999,370,13628,13629],[6238,0.133,370,13628,13629],[6342,0.133,370,13628,13629],[2166,0.33299999999999996,370,13628,13629],[502,0.9329999999999999,370,13628,13629],[3748,1.0,370,13628,13629]]},{"id":"anyapp-finance-1-new-2026-paper","name":"AnyAppBench ·Pro Expense new (2026 paper)","group":"Agent 与工具","metric":"Deterministic finance task success","unit":"%","public_history":true,"protocol":{"en":"Seven independent direct model backbones share the same five new finance templates on Pro Expense, each run three times (15 episodes/model). These cells form a complete fixed app/template cohort; the 520pair fullbenchmark and infrastructure-filtered five-model subgoal experiment are separate. Agents use the same emulator, six-primitive actions, task parameters, initialization, semantic success condition andtemplate-specific step limit. An app-specific deterministic verifier checks durable or UI state; VLM judges only diagnose failures and do not determine success. Planner-grounder hybrids andmulti-modelMobileAgent are excluded; conservative backbone selection omits overlapping GUI-trained bases. App-level/tier-specific rates are retained rather than averaging incompatible app populations.","zh":"七个独立直接模型共享Pro Expense相同五个new财务任务模板，每模板重复三次，即每模型15条情境。本组为完整固定应用与模板面板，与520任务应用对全基准及按基础设施有效性筛选的五模型subgoal实验分开。模拟器、六种基础动作、参数、初始化、语义成功条件和逐模板步数上限一致。应用专用确定性验证器检查持久数据或界面状态，VLM只诊断失败而不决定任务成功。排除规划与定位混合及多模型MobileAgent；保守选择不同基模型。保留应用与模板来源组，不混合不同应用分母。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34139v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":7,"result_count":7,"results":[[2451,0.0,370,13631,13629],[13630,0.33299999999999996,370,13631,13629],[6238,0.0,370,13631,13629],[6342,0.0,370,13631,13629],[2166,0.133,370,13631,13629],[502,0.133,370,13631,13629],[3748,0.133,370,13631,13629]]},{"id":"anyapp-finance-2-aw-inherited-2026-paper","name":"AnyAppBench ·Oinkoin aw-inherited (2026 paper)","group":"Agent 与工具","metric":"Deterministic finance task success","unit":"%","public_history":true,"protocol":{"en":"Seven independent direct model backbones share the same five aw-inherited finance templates on Oinkoin, each run three times (15 episodes/model). These cells form a complete fixed app/template cohort; the 520pair fullbenchmark and infrastructure-filtered five-model subgoal experiment are separate. Agents use the same emulator, six-primitive actions, task parameters, initialization, semantic success condition andtemplate-specific step limit. An app-specific deterministic verifier checks durable or UI state; VLM judges only diagnose failures and do not determine success. Planner-grounder hybrids andmulti-modelMobileAgent are excluded; conservative backbone selection omits overlapping GUI-trained bases. App-level/tier-specific rates are retained rather than averaging incompatible app populations.","zh":"七个独立直接模型共享Oinkoin相同五个aw-inherited财务任务模板，每模板重复三次，即每模型15条情境。本组为完整固定应用与模板面板，与520任务应用对全基准及按基础设施有效性筛选的五模型subgoal实验分开。模拟器、六种基础动作、参数、初始化、语义成功条件和逐模板步数上限一致。应用专用确定性验证器检查持久数据或界面状态，VLM只诊断失败而不决定任务成功。排除规划与定位混合及多模型MobileAgent；保守选择不同基模型。保留应用与模板来源组，不混合不同应用分母。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34139v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":7,"result_count":7,"results":[[2451,0.267,370,13632,13629],[13630,0.067,370,13632,13629],[6238,0.067,370,13632,13629],[6342,0.267,370,13632,13629],[2166,0.267,370,13632,13629],[502,0.267,370,13632,13629],[3748,0.667,370,13632,13629]]},{"id":"anyapp-finance-2-new-2026-paper","name":"AnyAppBench ·Oinkoin new (2026 paper)","group":"Agent 与工具","metric":"Deterministic finance task success","unit":"%","public_history":true,"protocol":{"en":"Seven independent direct model backbones share the same five new finance templates on Oinkoin, each run three times (15 episodes/model). These cells form a complete fixed app/template cohort; the 520pair fullbenchmark and infrastructure-filtered five-model subgoal experiment are separate. Agents use the same emulator, six-primitive actions, task parameters, initialization, semantic success condition andtemplate-specific step limit. An app-specific deterministic verifier checks durable or UI state; VLM judges only diagnose failures and do not determine success. Planner-grounder hybrids andmulti-modelMobileAgent are excluded; conservative backbone selection omits overlapping GUI-trained bases. App-level/tier-specific rates are retained rather than averaging incompatible app populations.","zh":"七个独立直接模型共享Oinkoin相同五个new财务任务模板，每模板重复三次，即每模型15条情境。本组为完整固定应用与模板面板，与520任务应用对全基准及按基础设施有效性筛选的五模型subgoal实验分开。模拟器、六种基础动作、参数、初始化、语义成功条件和逐模板步数上限一致。应用专用确定性验证器检查持久数据或界面状态，VLM只诊断失败而不决定任务成功。排除规划与定位混合及多模型MobileAgent；保守选择不同基模型。保留应用与模板来源组，不混合不同应用分母。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34139v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":7,"result_count":7,"results":[[2451,0.067,370,13633,13629],[13630,0.067,370,13633,13629],[6238,0.0,370,13633,13629],[6342,0.067,370,13633,13629],[2166,0.067,370,13633,13629],[502,0.0,370,13633,13629],[3748,0.067,370,13633,13629]]},{"id":"anyapp-finance-3-aw-inherited-2026-paper","name":"AnyAppBench ·OpenMoney Box aw-inherited (2026 paper)","group":"Agent 与工具","metric":"Deterministic finance task success","unit":"%","public_history":true,"protocol":{"en":"Seven independent direct model backbones share the same five aw-inherited finance templates on OpenMoney Box, each run three times (15 episodes/model). These cells form a complete fixed app/template cohort; the 520pair fullbenchmark and infrastructure-filtered five-model subgoal experiment are separate. Agents use the same emulator, six-primitive actions, task parameters, initialization, semantic success condition andtemplate-specific step limit. An app-specific deterministic verifier checks durable or UI state; VLM judges only diagnose failures and do not determine success. Planner-grounder hybrids andmulti-modelMobileAgent are excluded; conservative backbone selection omits overlapping GUI-trained bases. App-level/tier-specific rates are retained rather than averaging incompatible app populations.","zh":"七个独立直接模型共享OpenMoney Box相同五个aw-inherited财务任务模板，每模板重复三次，即每模型15条情境。本组为完整固定应用与模板面板，与520任务应用对全基准及按基础设施有效性筛选的五模型subgoal实验分开。模拟器、六种基础动作、参数、初始化、语义成功条件和逐模板步数上限一致。应用专用确定性验证器检查持久数据或界面状态，VLM只诊断失败而不决定任务成功。排除规划与定位混合及多模型MobileAgent；保守选择不同基模型。保留应用与模板来源组，不混合不同应用分母。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34139v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":7,"result_count":7,"results":[[2451,0.267,370,13634,13629],[13630,0.467,370,13634,13629],[6238,0.067,370,13634,13629],[6342,0.067,370,13634,13629],[2166,0.067,370,13634,13629],[502,0.267,370,13634,13629],[3748,0.733,370,13634,13629]]},{"id":"anyapp-finance-3-new-2026-paper","name":"AnyAppBench ·OpenMoney Box new (2026 paper)","group":"Agent 与工具","metric":"Deterministic finance task success","unit":"%","public_history":true,"protocol":{"en":"Seven independent direct model backbones share the same five new finance templates on OpenMoney Box, each run three times (15 episodes/model). These cells form a complete fixed app/template cohort; the 520pair fullbenchmark and infrastructure-filtered five-model subgoal experiment are separate. Agents use the same emulator, six-primitive actions, task parameters, initialization, semantic success condition andtemplate-specific step limit. An app-specific deterministic verifier checks durable or UI state; VLM judges only diagnose failures and do not determine success. Planner-grounder hybrids andmulti-modelMobileAgent are excluded; conservative backbone selection omits overlapping GUI-trained bases. App-level/tier-specific rates are retained rather than averaging incompatible app populations.","zh":"七个独立直接模型共享OpenMoney Box相同五个new财务任务模板，每模板重复三次，即每模型15条情境。本组为完整固定应用与模板面板，与520任务应用对全基准及按基础设施有效性筛选的五模型subgoal实验分开。模拟器、六种基础动作、参数、初始化、语义成功条件和逐模板步数上限一致。应用专用确定性验证器检查持久数据或界面状态，VLM只诊断失败而不决定任务成功。排除规划与定位混合及多模型MobileAgent；保守选择不同基模型。保留应用与模板来源组，不混合不同应用分母。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34139v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":7,"result_count":7,"results":[[2451,0.0,370,13635,13629],[13630,0.0,370,13635,13629],[6238,0.0,370,13635,13629],[6342,0.0,370,13635,13629],[2166,0.0,370,13635,13629],[502,0.067,370,13635,13629],[3748,0.0,370,13635,13629]]},{"id":"anyapp-finance-4-aw-inherited-2026-paper","name":"AnyAppBench ·My Expenses aw-inherited (2026 paper)","group":"Agent 与工具","metric":"Deterministic finance task success","unit":"%","public_history":true,"protocol":{"en":"Seven independent direct model backbones share the same five aw-inherited finance templates on My Expenses, each run three times (15 episodes/model). These cells form a complete fixed app/template cohort; the 520pair fullbenchmark and infrastructure-filtered five-model subgoal experiment are separate. Agents use the same emulator, six-primitive actions, task parameters, initialization, semantic success condition andtemplate-specific step limit. An app-specific deterministic verifier checks durable or UI state; VLM judges only diagnose failures and do not determine success. Planner-grounder hybrids andmulti-modelMobileAgent are excluded; conservative backbone selection omits overlapping GUI-trained bases. App-level/tier-specific rates are retained rather than averaging incompatible app populations.","zh":"七个独立直接模型共享My Expenses相同五个aw-inherited财务任务模板，每模板重复三次，即每模型15条情境。本组为完整固定应用与模板面板，与520任务应用对全基准及按基础设施有效性筛选的五模型subgoal实验分开。模拟器、六种基础动作、参数、初始化、语义成功条件和逐模板步数上限一致。应用专用确定性验证器检查持久数据或界面状态，VLM只诊断失败而不决定任务成功。排除规划与定位混合及多模型MobileAgent；保守选择不同基模型。保留应用与模板来源组，不混合不同应用分母。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34139v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":7,"result_count":7,"results":[[2451,0.267,370,13636,13629],[13630,0.5329999999999999,370,13636,13629],[6238,0.133,370,13636,13629],[6342,0.133,370,13636,13629],[2166,0.33299999999999996,370,13636,13629],[502,0.8,370,13636,13629],[3748,0.8,370,13636,13629]]},{"id":"anyapp-finance-4-new-2026-paper","name":"AnyAppBench ·My Expenses new (2026 paper)","group":"Agent 与工具","metric":"Deterministic finance task success","unit":"%","public_history":true,"protocol":{"en":"Seven independent direct model backbones share the same five new finance templates on My Expenses, each run three times (15 episodes/model). These cells form a complete fixed app/template cohort; the 520pair fullbenchmark and infrastructure-filtered five-model subgoal experiment are separate. Agents use the same emulator, six-primitive actions, task parameters, initialization, semantic success condition andtemplate-specific step limit. An app-specific deterministic verifier checks durable or UI state; VLM judges only diagnose failures and do not determine success. Planner-grounder hybrids andmulti-modelMobileAgent are excluded; conservative backbone selection omits overlapping GUI-trained bases. App-level/tier-specific rates are retained rather than averaging incompatible app populations.","zh":"七个独立直接模型共享My Expenses相同五个new财务任务模板，每模板重复三次，即每模型15条情境。本组为完整固定应用与模板面板，与520任务应用对全基准及按基础设施有效性筛选的五模型subgoal实验分开。模拟器、六种基础动作、参数、初始化、语义成功条件和逐模板步数上限一致。应用专用确定性验证器检查持久数据或界面状态，VLM只诊断失败而不决定任务成功。排除规划与定位混合及多模型MobileAgent；保守选择不同基模型。保留应用与模板来源组，不混合不同应用分母。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34139v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":7,"result_count":7,"results":[[2451,0.2,370,13637,13629],[13630,0.0,370,13637,13629],[6238,0.0,370,13637,13629],[6342,0.0,370,13637,13629],[2166,0.067,370,13637,13629],[502,0.133,370,13637,13629],[3748,0.067,370,13637,13629]]},{"id":"anyapp-finance-5-aw-inherited-2026-paper","name":"AnyAppBench ·Finance Manager aw-inherited (2026 paper)","group":"Agent 与工具","metric":"Deterministic finance task success","unit":"%","public_history":true,"protocol":{"en":"Seven independent direct model backbones share the same five aw-inherited finance templates on Finance Manager, each run three times (15 episodes/model). These cells form a complete fixed app/template cohort; the 520pair fullbenchmark and infrastructure-filtered five-model subgoal experiment are separate. Agents use the same emulator, six-primitive actions, task parameters, initialization, semantic success condition andtemplate-specific step limit. An app-specific deterministic verifier checks durable or UI state; VLM judges only diagnose failures and do not determine success. Planner-grounder hybrids andmulti-modelMobileAgent are excluded; conservative backbone selection omits overlapping GUI-trained bases. App-level/tier-specific rates are retained rather than averaging incompatible app populations.","zh":"七个独立直接模型共享Finance Manager相同五个aw-inherited财务任务模板，每模板重复三次，即每模型15条情境。本组为完整固定应用与模板面板，与520任务应用对全基准及按基础设施有效性筛选的五模型subgoal实验分开。模拟器、六种基础动作、参数、初始化、语义成功条件和逐模板步数上限一致。应用专用确定性验证器检查持久数据或界面状态，VLM只诊断失败而不决定任务成功。排除规划与定位混合及多模型MobileAgent；保守选择不同基模型。保留应用与模板来源组，不混合不同应用分母。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34139v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":7,"result_count":7,"results":[[2451,0.067,370,13638,13629],[13630,0.467,370,13638,13629],[6238,0.133,370,13638,13629],[6342,0.133,370,13638,13629],[2166,0.33299999999999996,370,13638,13629],[502,0.4,370,13638,13629],[3748,0.733,370,13638,13629]]},{"id":"anyapp-finance-5-new-2026-paper","name":"AnyAppBench ·Finance Manager new (2026 paper)","group":"Agent 与工具","metric":"Deterministic finance task success","unit":"%","public_history":true,"protocol":{"en":"Seven independent direct model backbones share the same five new finance templates on Finance Manager, each run three times (15 episodes/model). These cells form a complete fixed app/template cohort; the 520pair fullbenchmark and infrastructure-filtered five-model subgoal experiment are separate. Agents use the same emulator, six-primitive actions, task parameters, initialization, semantic success condition andtemplate-specific step limit. An app-specific deterministic verifier checks durable or UI state; VLM judges only diagnose failures and do not determine success. Planner-grounder hybrids andmulti-modelMobileAgent are excluded; conservative backbone selection omits overlapping GUI-trained bases. App-level/tier-specific rates are retained rather than averaging incompatible app populations.","zh":"七个独立直接模型共享Finance Manager相同五个new财务任务模板，每模板重复三次，即每模型15条情境。本组为完整固定应用与模板面板，与520任务应用对全基准及按基础设施有效性筛选的五模型subgoal实验分开。模拟器、六种基础动作、参数、初始化、语义成功条件和逐模板步数上限一致。应用专用确定性验证器检查持久数据或界面状态，VLM只诊断失败而不决定任务成功。排除规划与定位混合及多模型MobileAgent；保守选择不同基模型。保留应用与模板来源组，不混合不同应用分母。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34139v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":7,"result_count":7,"results":[[2451,0.067,370,13639,13629],[13630,0.067,370,13639,13629],[6238,0.0,370,13639,13629],[6342,0.0,370,13639,13629],[2166,0.067,370,13639,13629],[502,0.267,370,13639,13629],[3748,0.067,370,13639,13629]]},{"id":"anyapp-finance-6-aw-inherited-2026-paper","name":"AnyAppBench ·Sushi aw-inherited (2026 paper)","group":"Agent 与工具","metric":"Deterministic finance task success","unit":"%","public_history":true,"protocol":{"en":"Seven independent direct model backbones share the same five aw-inherited finance templates on Sushi, each run three times (15 episodes/model). These cells form a complete fixed app/template cohort; the 520pair fullbenchmark and infrastructure-filtered five-model subgoal experiment are separate. Agents use the same emulator, six-primitive actions, task parameters, initialization, semantic success condition andtemplate-specific step limit. An app-specific deterministic verifier checks durable or UI state; VLM judges only diagnose failures and do not determine success. Planner-grounder hybrids andmulti-modelMobileAgent are excluded; conservative backbone selection omits overlapping GUI-trained bases. App-level/tier-specific rates are retained rather than averaging incompatible app populations.","zh":"七个独立直接模型共享Sushi相同五个aw-inherited财务任务模板，每模板重复三次，即每模型15条情境。本组为完整固定应用与模板面板，与520任务应用对全基准及按基础设施有效性筛选的五模型subgoal实验分开。模拟器、六种基础动作、参数、初始化、语义成功条件和逐模板步数上限一致。应用专用确定性验证器检查持久数据或界面状态，VLM只诊断失败而不决定任务成功。排除规划与定位混合及多模型MobileAgent；保守选择不同基模型。保留应用与模板来源组，不混合不同应用分母。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34139v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":7,"result_count":7,"results":[[2451,0.467,370,13640,13629],[13630,0.267,370,13640,13629],[6238,0.067,370,13640,13629],[6342,0.133,370,13640,13629],[2166,0.133,370,13640,13629],[502,0.4,370,13640,13629],[3748,0.733,370,13640,13629]]},{"id":"anyapp-finance-6-new-2026-paper","name":"AnyAppBench ·Sushi new (2026 paper)","group":"Agent 与工具","metric":"Deterministic finance task success","unit":"%","public_history":true,"protocol":{"en":"Seven independent direct model backbones share the same five new finance templates on Sushi, each run three times (15 episodes/model). These cells form a complete fixed app/template cohort; the 520pair fullbenchmark and infrastructure-filtered five-model subgoal experiment are separate. Agents use the same emulator, six-primitive actions, task parameters, initialization, semantic success condition andtemplate-specific step limit. An app-specific deterministic verifier checks durable or UI state; VLM judges only diagnose failures and do not determine success. Planner-grounder hybrids andmulti-modelMobileAgent are excluded; conservative backbone selection omits overlapping GUI-trained bases. App-level/tier-specific rates are retained rather than averaging incompatible app populations.","zh":"七个独立直接模型共享Sushi相同五个new财务任务模板，每模板重复三次，即每模型15条情境。本组为完整固定应用与模板面板，与520任务应用对全基准及按基础设施有效性筛选的五模型subgoal实验分开。模拟器、六种基础动作、参数、初始化、语义成功条件和逐模板步数上限一致。应用专用确定性验证器检查持久数据或界面状态，VLM只诊断失败而不决定任务成功。排除规划与定位混合及多模型MobileAgent；保守选择不同基模型。保留应用与模板来源组，不混合不同应用分母。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34139v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":7,"result_count":7,"results":[[2451,0.067,370,13641,13629],[13630,0.0,370,13641,13629],[6238,0.0,370,13641,13629],[6342,0.067,370,13641,13629],[2166,0.067,370,13641,13629],[502,0.133,370,13641,13629],[3748,0.0,370,13641,13629]]},{"id":"llm4trust-advogato-micro-f1-2026-paper","name":"LLM4Trust ·advogato micro-f1 (2026 paper)","group":"知识与推理","metric":"micro-f1","unit":"score","public_history":true,"protocol":{"en":"Three GNN architectures and three distinct language models share 1,000 sampled trustor-trustee test pairs per dataset, evaluated over five random seeds. Static datasets use an 80/20 train/test split; dynamic datasets are sorted chronologically first. Five percent of training data is held out for validation. Twenty validation pairs select the graph input format before the 1,000-pair test evaluation. Each test pair receives exactly three training-label demonstrations. This cohort contains only methods with that supervision budget. Static micro-F1 and trust-level MAE, and dynamic macro-F1 and balanced accuracy retain their native units. The original graph neighborhood, time scope and few-shot plus knowledge and role prompt remain explicit configurations.","zh":"每个数据集的共同1000对测试trustor-trustee由三种3-shot图网络和三个独立语言模型评测，并用五个随机种子重复。静态图按80/20划分，动态图先按时间排序；训练部分另留5%验证，用20对验证样本选图输入格式，再评测1000对。每测试对只提供三个训练标签示例，完全监督和无学习方法不混入这一相同监督量面板。静态microF1及信任等级MAE、动态macroF1及平衡正确率均保留原生单位，原图邻域、时间范围及few-shot加知识和角色提示记录为配置；这些图网络与LLM分别计独立架构。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.33521v1","published_at":"2026-09-27","status":"paper_baselines"},"score_min":0,"score_max":1,"saturation_excluded":true,"scale_label":{"en":"Native score / 1","zh":"原生分数 / 1"},"score_decimals":4,"model_count":6,"result_count":6,"results":[[13642,0.446,370,13643,13644],[13645,0.518,370,13643,13644],[13646,0.398,370,13643,13644],[1573,0.669,370,13647,13644],[1583,0.673,370,13647,13644],[3767,0.672,370,13647,13644]]},{"id":"llm4trust-advogato-mae-2026-paper","name":"LLM4Trust ·advogato mae (2026 paper)","group":"知识与推理","metric":"mae","unit":"raw","public_history":true,"protocol":{"en":"Three GNN architectures and three distinct language models share 1,000 sampled trustor-trustee test pairs per dataset, evaluated over five random seeds. Static datasets use an 80/20 train/test split; dynamic datasets are sorted chronologically first. Five percent of training data is held out for validation. Twenty validation pairs select the graph input format before the 1,000-pair test evaluation. Each test pair receives exactly three training-label demonstrations. This cohort contains only methods with that supervision budget. Static micro-F1 and trust-level MAE, and dynamic macro-F1 and balanced accuracy retain their native units. The original graph neighborhood, time scope and few-shot plus knowledge and role prompt remain explicit configurations.","zh":"每个数据集的共同1000对测试trustor-trustee由三种3-shot图网络和三个独立语言模型评测，并用五个随机种子重复。静态图按80/20划分，动态图先按时间排序；训练部分另留5%验证，用20对验证样本选图输入格式，再评测1000对。每测试对只提供三个训练标签示例，完全监督和无学习方法不混入这一相同监督量面板。静态microF1及信任等级MAE、动态macroF1及平衡正确率均保留原生单位，原图邻域、时间范围及few-shot加知识和角色提示记录为配置；这些图网络与LLM分别计独立架构。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.33521v1","published_at":"2026-09-27","status":"paper_baselines"},"saturation_excluded":true,"scale_label":{"en":"mae","zh":"mae"},"score_decimals":3,"higher_is_better":false,"model_count":6,"result_count":6,"results":[[13642,0.202,370,13643,13644],[13645,0.159,370,13643,13644],[13646,0.196,370,13643,13644],[1573,0.101,370,13647,13644],[1583,0.1,370,13647,13644],[3767,0.101,370,13647,13644]]},{"id":"llm4trust-pgp-micro-f1-2026-paper","name":"LLM4Trust ·pgp micro-f1 (2026 paper)","group":"知识与推理","metric":"micro-f1","unit":"score","public_history":true,"protocol":{"en":"Three GNN architectures and three distinct language models share 1,000 sampled trustor-trustee test pairs per dataset, evaluated over five random seeds. Static datasets use an 80/20 train/test split; dynamic datasets are sorted chronologically first. Five percent of training data is held out for validation. Twenty validation pairs select the graph input format before the 1,000-pair test evaluation. Each test pair receives exactly three training-label demonstrations. This cohort contains only methods with that supervision budget. Static micro-F1 and trust-level MAE, and dynamic macro-F1 and balanced accuracy retain their native units. The original graph neighborhood, time scope and few-shot plus knowledge and role prompt remain explicit configurations.","zh":"每个数据集的共同1000对测试trustor-trustee由三种3-shot图网络和三个独立语言模型评测，并用五个随机种子重复。静态图按80/20划分，动态图先按时间排序；训练部分另留5%验证，用20对验证样本选图输入格式，再评测1000对。每测试对只提供三个训练标签示例，完全监督和无学习方法不混入这一相同监督量面板。静态microF1及信任等级MAE、动态macroF1及平衡正确率均保留原生单位，原图邻域、时间范围及few-shot加知识和角色提示记录为配置；这些图网络与LLM分别计独立架构。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.33521v1","published_at":"2026-09-27","status":"paper_baselines"},"score_min":0,"score_max":1,"saturation_excluded":true,"scale_label":{"en":"Native score / 1","zh":"原生分数 / 1"},"score_decimals":4,"model_count":6,"result_count":6,"results":[[13642,0.651,370,13643,13644],[13645,0.714,370,13643,13644],[13646,0.546,370,13643,13644],[1573,0.85,370,13647,13644],[1583,0.846,370,13647,13644],[3767,0.849,370,13647,13644]]},{"id":"llm4trust-pgp-mae-2026-paper","name":"LLM4Trust ·pgp mae (2026 paper)","group":"知识与推理","metric":"mae","unit":"raw","public_history":true,"protocol":{"en":"Three GNN architectures and three distinct language models share 1,000 sampled trustor-trustee test pairs per dataset, evaluated over five random seeds. Static datasets use an 80/20 train/test split; dynamic datasets are sorted chronologically first. Five percent of training data is held out for validation. Twenty validation pairs select the graph input format before the 1,000-pair test evaluation. Each test pair receives exactly three training-label demonstrations. This cohort contains only methods with that supervision budget. Static micro-F1 and trust-level MAE, and dynamic macro-F1 and balanced accuracy retain their native units. The original graph neighborhood, time scope and few-shot plus knowledge and role prompt remain explicit configurations.","zh":"每个数据集的共同1000对测试trustor-trustee由三种3-shot图网络和三个独立语言模型评测，并用五个随机种子重复。静态图按80/20划分，动态图先按时间排序；训练部分另留5%验证，用20对验证样本选图输入格式，再评测1000对。每测试对只提供三个训练标签示例，完全监督和无学习方法不混入这一相同监督量面板。静态microF1及信任等级MAE、动态macroF1及平衡正确率均保留原生单位，原图邻域、时间范围及few-shot加知识和角色提示记录为配置；这些图网络与LLM分别计独立架构。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.33521v1","published_at":"2026-09-27","status":"paper_baselines"},"saturation_excluded":true,"scale_label":{"en":"mae","zh":"mae"},"score_decimals":3,"higher_is_better":false,"model_count":6,"result_count":6,"results":[[13642,0.258,370,13643,13644],[13645,0.203,370,13643,13644],[13646,0.332,370,13643,13644],[1573,0.097,370,13647,13644],[1583,0.1,370,13647,13644],[3767,0.096,370,13647,13644]]},{"id":"llm4trust-epinions-micro-f1-2026-paper","name":"LLM4Trust ·epinions micro-f1 (2026 paper)","group":"知识与推理","metric":"micro-f1","unit":"score","public_history":true,"protocol":{"en":"Three GNN architectures and three distinct language models share 1,000 sampled trustor-trustee test pairs per dataset, evaluated over five random seeds. Static datasets use an 80/20 train/test split; dynamic datasets are sorted chronologically first. Five percent of training data is held out for validation. Twenty validation pairs select the graph input format before the 1,000-pair test evaluation. Each test pair receives exactly three training-label demonstrations. This cohort contains only methods with that supervision budget. Static micro-F1 and trust-level MAE, and dynamic macro-F1 and balanced accuracy retain their native units. The original graph neighborhood, time scope and few-shot plus knowledge and role prompt remain explicit configurations.","zh":"每个数据集的共同1000对测试trustor-trustee由三种3-shot图网络和三个独立语言模型评测，并用五个随机种子重复。静态图按80/20划分，动态图先按时间排序；训练部分另留5%验证，用20对验证样本选图输入格式，再评测1000对。每测试对只提供三个训练标签示例，完全监督和无学习方法不混入这一相同监督量面板。静态microF1及信任等级MAE、动态macroF1及平衡正确率均保留原生单位，原图邻域、时间范围及few-shot加知识和角色提示记录为配置；这些图网络与LLM分别计独立架构。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.33521v1","published_at":"2026-09-27","status":"paper_baselines"},"score_min":0,"score_max":1,"saturation_excluded":true,"scale_label":{"en":"Native score / 1","zh":"原生分数 / 1"},"score_decimals":4,"model_count":6,"result_count":6,"results":[[13642,0.783,370,13643,13644],[13645,0.622,370,13643,13644],[13646,0.556,370,13643,13644],[1573,0.881,370,13647,13644],[1583,0.854,370,13647,13644],[3767,0.88,370,13647,13644]]},{"id":"llm4trust-epinions-mae-2026-paper","name":"LLM4Trust ·epinions mae (2026 paper)","group":"知识与推理","metric":"mae","unit":"raw","public_history":true,"protocol":{"en":"Three GNN architectures and three distinct language models share 1,000 sampled trustor-trustee test pairs per dataset, evaluated over five random seeds. Static datasets use an 80/20 train/test split; dynamic datasets are sorted chronologically first. Five percent of training data is held out for validation. Twenty validation pairs select the graph input format before the 1,000-pair test evaluation. Each test pair receives exactly three training-label demonstrations. This cohort contains only methods with that supervision budget. Static micro-F1 and trust-level MAE, and dynamic macro-F1 and balanced accuracy retain their native units. The original graph neighborhood, time scope and few-shot plus knowledge and role prompt remain explicit configurations.","zh":"每个数据集的共同1000对测试trustor-trustee由三种3-shot图网络和三个独立语言模型评测，并用五个随机种子重复。静态图按80/20划分，动态图先按时间排序；训练部分另留5%验证，用20对验证样本选图输入格式，再评测1000对。每测试对只提供三个训练标签示例，完全监督和无学习方法不混入这一相同监督量面板。静态microF1及信任等级MAE、动态macroF1及平衡正确率均保留原生单位，原图邻域、时间范围及few-shot加知识和角色提示记录为配置；这些图网络与LLM分别计独立架构。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.33521v1","published_at":"2026-09-27","status":"paper_baselines"},"saturation_excluded":true,"scale_label":{"en":"mae","zh":"mae"},"score_decimals":3,"higher_is_better":false,"model_count":6,"result_count":6,"results":[[13642,0.173,370,13643,13644],[13645,0.302,370,13643,13644],[13646,0.355,370,13643,13644],[1573,0.095,370,13647,13644],[1583,0.117,370,13647,13644],[3767,0.096,370,13647,13644]]},{"id":"llm4trust-bitcoin-otc-macro-f1-2026-paper","name":"LLM4Trust ·bitcoin-otc macro-f1 (2026 paper)","group":"知识与推理","metric":"macro-f1","unit":"score","public_history":true,"protocol":{"en":"Three GNN architectures and three distinct language models share 1,000 sampled trustor-trustee test pairs per dataset, evaluated over five random seeds. Static datasets use an 80/20 train/test split; dynamic datasets are sorted chronologically first. Five percent of training data is held out for validation. Twenty validation pairs select the graph input format before the 1,000-pair test evaluation. Each test pair receives exactly three training-label demonstrations. This cohort contains only methods with that supervision budget. Static micro-F1 and trust-level MAE, and dynamic macro-F1 and balanced accuracy retain their native units. The original graph neighborhood, time scope and few-shot plus knowledge and role prompt remain explicit configurations.","zh":"每个数据集的共同1000对测试trustor-trustee由三种3-shot图网络和三个独立语言模型评测，并用五个随机种子重复。静态图按80/20划分，动态图先按时间排序；训练部分另留5%验证，用20对验证样本选图输入格式，再评测1000对。每测试对只提供三个训练标签示例，完全监督和无学习方法不混入这一相同监督量面板。静态microF1及信任等级MAE、动态macroF1及平衡正确率均保留原生单位，原图邻域、时间范围及few-shot加知识和角色提示记录为配置；这些图网络与LLM分别计独立架构。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.33521v1","published_at":"2026-09-27","status":"paper_baselines"},"score_min":0,"score_max":1,"saturation_excluded":true,"scale_label":{"en":"Native score / 1","zh":"原生分数 / 1"},"score_decimals":4,"model_count":6,"result_count":6,"results":[[13648,0.496,370,13643,13649],[13650,0.436,370,13643,13649],[13651,0.474,370,13643,13649],[1573,0.678,370,13647,13649],[1583,0.524,370,13647,13649],[3767,0.619,370,13647,13649]]},{"id":"llm4trust-bitcoin-otc-balanced-accuracy-2026-paper","name":"LLM4Trust ·bitcoin-otc balanced-accuracy (2026 paper)","group":"知识与推理","metric":"balanced-accuracy","unit":"score","public_history":true,"protocol":{"en":"Three GNN architectures and three distinct language models share 1,000 sampled trustor-trustee test pairs per dataset, evaluated over five random seeds. Static datasets use an 80/20 train/test split; dynamic datasets are sorted chronologically first. Five percent of training data is held out for validation. Twenty validation pairs select the graph input format before the 1,000-pair test evaluation. Each test pair receives exactly three training-label demonstrations. This cohort contains only methods with that supervision budget. Static micro-F1 and trust-level MAE, and dynamic macro-F1 and balanced accuracy retain their native units. The original graph neighborhood, time scope and few-shot plus knowledge and role prompt remain explicit configurations.","zh":"每个数据集的共同1000对测试trustor-trustee由三种3-shot图网络和三个独立语言模型评测，并用五个随机种子重复。静态图按80/20划分，动态图先按时间排序；训练部分另留5%验证，用20对验证样本选图输入格式，再评测1000对。每测试对只提供三个训练标签示例，完全监督和无学习方法不混入这一相同监督量面板。静态microF1及信任等级MAE、动态macroF1及平衡正确率均保留原生单位，原图邻域、时间范围及few-shot加知识和角色提示记录为配置；这些图网络与LLM分别计独立架构。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.33521v1","published_at":"2026-09-27","status":"paper_baselines"},"score_min":0,"score_max":1,"saturation_excluded":true,"scale_label":{"en":"Native score / 1","zh":"原生分数 / 1"},"score_decimals":4,"model_count":6,"result_count":6,"results":[[13648,0.512,370,13643,13649],[13650,0.488,370,13643,13649],[13651,0.497,370,13643,13649],[1573,0.652,370,13647,13649],[1583,0.525,370,13647,13649],[3767,0.643,370,13647,13649]]},{"id":"llm4trust-bitcoin-alpha-macro-f1-2026-paper","name":"LLM4Trust ·bitcoin-alpha macro-f1 (2026 paper)","group":"知识与推理","metric":"macro-f1","unit":"score","public_history":true,"protocol":{"en":"Three GNN architectures and three distinct language models share 1,000 sampled trustor-trustee test pairs per dataset, evaluated over five random seeds. Static datasets use an 80/20 train/test split; dynamic datasets are sorted chronologically first. Five percent of training data is held out for validation. Twenty validation pairs select the graph input format before the 1,000-pair test evaluation. Each test pair receives exactly three training-label demonstrations. This cohort contains only methods with that supervision budget. Static micro-F1 and trust-level MAE, and dynamic macro-F1 and balanced accuracy retain their native units. The original graph neighborhood, time scope and few-shot plus knowledge and role prompt remain explicit configurations.","zh":"每个数据集的共同1000对测试trustor-trustee由三种3-shot图网络和三个独立语言模型评测，并用五个随机种子重复。静态图按80/20划分，动态图先按时间排序；训练部分另留5%验证，用20对验证样本选图输入格式，再评测1000对。每测试对只提供三个训练标签示例，完全监督和无学习方法不混入这一相同监督量面板。静态microF1及信任等级MAE、动态macroF1及平衡正确率均保留原生单位，原图邻域、时间范围及few-shot加知识和角色提示记录为配置；这些图网络与LLM分别计独立架构。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.33521v1","published_at":"2026-09-27","status":"paper_baselines"},"score_min":0,"score_max":1,"saturation_excluded":true,"scale_label":{"en":"Native score / 1","zh":"原生分数 / 1"},"score_decimals":4,"model_count":6,"result_count":6,"results":[[13648,0.493,370,13643,13649],[13650,0.477,370,13643,13649],[13651,0.493,370,13643,13649],[1573,0.609,370,13647,13649],[1583,0.571,370,13647,13649],[3767,0.532,370,13647,13649]]},{"id":"llm4trust-bitcoin-alpha-balanced-accuracy-2026-paper","name":"LLM4Trust ·bitcoin-alpha balanced-accuracy (2026 paper)","group":"知识与推理","metric":"balanced-accuracy","unit":"score","public_history":true,"protocol":{"en":"Three GNN architectures and three distinct language models share 1,000 sampled trustor-trustee test pairs per dataset, evaluated over five random seeds. Static datasets use an 80/20 train/test split; dynamic datasets are sorted chronologically first. Five percent of training data is held out for validation. Twenty validation pairs select the graph input format before the 1,000-pair test evaluation. Each test pair receives exactly three training-label demonstrations. This cohort contains only methods with that supervision budget. Static micro-F1 and trust-level MAE, and dynamic macro-F1 and balanced accuracy retain their native units. The original graph neighborhood, time scope and few-shot plus knowledge and role prompt remain explicit configurations.","zh":"每个数据集的共同1000对测试trustor-trustee由三种3-shot图网络和三个独立语言模型评测，并用五个随机种子重复。静态图按80/20划分，动态图先按时间排序；训练部分另留5%验证，用20对验证样本选图输入格式，再评测1000对。每测试对只提供三个训练标签示例，完全监督和无学习方法不混入这一相同监督量面板。静态microF1及信任等级MAE、动态macroF1及平衡正确率均保留原生单位，原图邻域、时间范围及few-shot加知识和角色提示记录为配置；这些图网络与LLM分别计独立架构。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.33521v1","published_at":"2026-09-27","status":"paper_baselines"},"score_min":0,"score_max":1,"saturation_excluded":true,"scale_label":{"en":"Native score / 1","zh":"原生分数 / 1"},"score_decimals":4,"model_count":6,"result_count":6,"results":[[13648,0.507,370,13643,13649],[13650,0.502,370,13643,13649],[13651,0.527,370,13643,13649],[1573,0.594,370,13647,13649],[1583,0.562,370,13647,13649],[3767,0.545,370,13647,13649]]},{"id":"reformbench-shared-tcpr-2026-paper","name":"ReFormBench · format compliance (2026 paper)","group":"专业工作","metric":"Calibrated three-constraint pass estimate","unit":"score","public_history":true,"protocol":{"en":"Ten independent bases answer the same 5,000 three-constraint instructions derived from 910 source questions, spanning 66 format features. This panel uses TCPR on that shared instruction set, excluding EMCR whose eligible correction targets depend on each model’s unconstrained baseline. GPT-5.4-Mini and Gemini-3.1-Flash-Lite independently grade each constraint at temperature zero with the highest supported reasoning setting, blind to candidate identity. Human reviewers adjudicate sampled disagreements stratified by model, direction, feature and require/forbid mode. Reviewed disagreements use binary decisions; unreviewed disagreements use stratum-estimated pass probabilities, and joint pass probabilities are factorized before averaging instructions. Thus the table value is a calibrated 0–100 estimate rather than a directly counted exact-pass percentage. Judge agreements are not independently human-reviewed. All models use reasoning-enabled configurations.","zh":"十底模回答共同5000条三约束指令，来自910原题、66格式特征。本组取共同指令TCPR，不混入按各模型基线决定资格的EMCR。GPT5.4Mini与Gemini3.1Flash-Lite盲评每个约束，温度零、采用最高支持思考档。人工按模型、分歧方向、特征与要求/禁止方式分层复核抽样分歧。已复核用二值裁定，未复核用所在层估计通过概率；联合通过概率按独立因子乘积后跨指令平均。故保留0至100校准估计量表，不作直接计数精确成功率。两裁判一致项未再独立人工复核。候选均开启思考。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.19693v1","published_at":"2026-10-14","status":"paper_baselines"},"score_max":100,"score_decimals":2,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":10,"result_count":10,"results":[[2977,82.82,370,13652,13653],[762,81.99,370,13652,13653],[3415,78.08,370,13652,13653],[3764,77.82,370,13652,13653],[227,75.76,370,13652,13653],[3185,66.39,370,13652,13653],[3763,65.46,370,13652,13653],[2261,60.09,370,13652,13653],[13654,56.33,370,13652,13653],[4070,51.81,370,13652,13653]]},{"id":"commercegen-single-image-editing-2026-paper","name":"CommerceGen · single-image editing (2026 paper)","group":"视觉与视频","metric":"Mean five-dimension editing rubric","unit":"score","public_history":true,"protocol":{"en":"Thirteen independent generation bases use the common 200-case single-image editing task without baseline prompt enhancement. CommerceGen-Bench comprises 400 real search-result cases split equally into single- and multi-image editing, spanning six top-level and fifteen fine categories. Hard cases include thin/transparent products, tiny trademarks or long selling-point text. Gemini 3 Flash compares generated images, resized to about 1024×1024, with references and instructions. Overall is the arithmetic mean of five anchored 0–10 rubrics: product fidelity, instruction following, product integration, naturalness and absence of artifacts. Ours is the SKU-DiT/CommerceGen-Studio system with native PERL extraction and LTO layout rewriting; it is explicitly a model-plus-pipeline configuration, not the bare generator. The prompt-enhanced comparison changes instruction difficulty and remains a separate original table, not pooled into this panel.","zh":"13个独立生成底模使用共同200例编辑任务；完整CommerceGen-Bench为400真实搜索结果案例，单图与多图各半，覆盖六大类、15细类。困难样本含透明或纤细产品、小商标、长卖点文本。Gemini3Flash比较约1024×1024生成图、参考图与指令，按产品保真、指令遵循、产品融入、自然度、无伪影五个有锚点0至10量表取平均。Ours为SKU-DiT/CommerceGen-Studio，保留自身PERL提取与LTO版式改写，明确为模型加流程配置。此处用主表无基线提示增强条件；提示增强会改变指令难度，另表保留，不混合。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.20603v1","published_at":"2026-10-14","status":"paper_baselines"},"score_max":10,"score_decimals":2,"saturation_excluded":true,"scale_label":{"en":"Score / 10","zh":"分数 / 10"},"model_count":13,"result_count":13,"results":[[13655,7.29,370,13656,13657],[13658,5.79,370,13656,13657],[8150,4.91,370,13656,13657],[13659,5.56,370,13656,13657],[7300,5.15,370,13656,13657],[7253,5.69,370,13656,13657],[6368,9.61,370,13656,13657],[6380,9.61,370,13656,13657],[6381,9.57,370,13656,13657],[6366,9.2,370,13656,13657],[13660,8.89,370,13656,13657],[6382,8.71,370,13656,13657],[13661,9.4,370,13662,13657]]},{"id":"commercegen-multi-image-editing-2026-paper","name":"CommerceGen · multi-image editing (2026 paper)","group":"视觉与视频","metric":"Mean five-dimension editing rubric","unit":"score","public_history":true,"protocol":{"en":"Thirteen independent generation bases use the common 200-case multi-image editing task without baseline prompt enhancement. CommerceGen-Bench comprises 400 real search-result cases split equally into single- and multi-image editing, spanning six top-level and fifteen fine categories. Hard cases include thin/transparent products, tiny trademarks or long selling-point text. Gemini 3 Flash compares generated images, resized to about 1024×1024, with references and instructions. Overall is the arithmetic mean of five anchored 0–10 rubrics: product fidelity, instruction following, product integration, naturalness and absence of artifacts. Ours is the SKU-DiT/CommerceGen-Studio system with native PERL extraction and LTO layout rewriting; it is explicitly a model-plus-pipeline configuration, not the bare generator. The prompt-enhanced comparison changes instruction difficulty and remains a separate original table, not pooled into this panel.","zh":"13个独立生成底模使用共同200例编辑任务；完整CommerceGen-Bench为400真实搜索结果案例，单图与多图各半，覆盖六大类、15细类。困难样本含透明或纤细产品、小商标、长卖点文本。Gemini3Flash比较约1024×1024生成图、参考图与指令，按产品保真、指令遵循、产品融入、自然度、无伪影五个有锚点0至10量表取平均。Ours为SKU-DiT/CommerceGen-Studio，保留自身PERL提取与LTO版式改写，明确为模型加流程配置。此处用主表无基线提示增强条件；提示增强会改变指令难度，另表保留，不混合。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.20603v1","published_at":"2026-10-14","status":"paper_baselines"},"score_max":10,"score_decimals":2,"saturation_excluded":true,"scale_label":{"en":"Score / 10","zh":"分数 / 10"},"model_count":13,"result_count":13,"results":[[13655,6.76,370,13656,13657],[13658,4.35,370,13656,13657],[8150,4.27,370,13656,13657],[13659,5.52,370,13656,13657],[7300,4.9,370,13656,13657],[7253,5.23,370,13656,13657],[6368,9.68,370,13656,13657],[6380,9.66,370,13656,13657],[6381,9.6,370,13656,13657],[6366,9.17,370,13656,13657],[13660,8.89,370,13656,13657],[6382,8.53,370,13656,13657],[13661,9.44,370,13662,13657]]},{"id":"proactegomem-daily-information-capture-2026-paper","name":"ProactEgoMem · Information capture (2026 paper)","group":"Agent 与工具","metric":"Intervention detection F1","unit":"%","public_history":true,"protocol":{"en":"Thirteen bases are evaluated on the same 100 Daily clips for Information capture, containing 278 ground-truth events. Precision uses emitted actions, recall all ground-truth opportunities, and the table reports their F1. This Intervention Level requires a valid start-to-stop memory-tool lifecycle; invalid or absent tool calls are model behavior, including the executed VideoLLM-Online zero, not missing infrastructure. Hungarian matching combines semantic similarity and temporal compatibility; pairs require similarity≥0.5 and an asymmetric time window, with time exponent 1.5. Semantic judging uses temperature zero; the paper separately pins DeepSeek-V4-Flash for conditional message correctness, which is not imported here. Timing Start@0.2 conditions on already matched actions and is excluded. The complete original table also retains notification scores and QA. The Sci appendix lists thirteen numeric models while text describes ten paired cross-track baselines; this bundle uses Daily only and does not silently resolve that Sci cohort discrepancy.","zh":"13底模在Daily同100片段运行该任务，采用该任务完整金标事件：信息捕获278、状态跟踪483、位置跟踪481。精确率分母是发出动作，召回分母为全部机会，取两者F1。Intervention须产生有效开始至停止的记忆工具生命周期；无效或缺失工具调用为实际模型行为，VideoLLM-Online已运行零分不属基础设施缺失。匈牙利匹配结合语义相似度及时间，相似度至少0.5、非对称时间窗、时间项指数1.5；语义裁判温度零。另一个条件消息正确性指标固定DeepSeekV4Flash，此处不导入。Start@0.2以已匹配动作为条件，另行排除。保留完整通知、工具与QA原表。Sci附录列13行数值但正文称10个配对模型，本批仅用Daily，不擅自消解该差异。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.30089v1","published_at":"2026-10-21","status":"paper_baselines"},"model_count":13,"result_count":13,"results":[[2055,0.057999999999999996,370,13663,13664],[4958,0.059000000000000004,370,13663,13664],[8143,0.055,370,13663,13664],[5300,0.076,370,13663,13664],[3763,0.053,370,13663,13664],[4070,0.042,370,13663,13664],[6349,0.012,370,13663,13664],[10076,0.035,370,13663,13664],[8381,0.006999999999999999,370,13663,13664],[13665,0.026000000000000002,370,13663,13664],[12296,0.0,370,13663,13664],[5723,0.028999999999999998,370,13663,13664],[13666,0.013999999999999999,370,13663,13664]]},{"id":"proactegomem-daily-state-tracking-2026-paper","name":"ProactEgoMem · State tracking (2026 paper)","group":"Agent 与工具","metric":"Intervention detection F1","unit":"%","public_history":true,"protocol":{"en":"Thirteen bases are evaluated on the same 100 Daily clips for State tracking, containing 483 ground-truth events. Precision uses emitted actions, recall all ground-truth opportunities, and the table reports their F1. This Intervention Level requires a valid start-to-stop memory-tool lifecycle; invalid or absent tool calls are model behavior, including the executed VideoLLM-Online zero, not missing infrastructure. Hungarian matching combines semantic similarity and temporal compatibility; pairs require similarity≥0.5 and an asymmetric time window, with time exponent 1.5. Semantic judging uses temperature zero; the paper separately pins DeepSeek-V4-Flash for conditional message correctness, which is not imported here. Timing Start@0.2 conditions on already matched actions and is excluded. The complete original table also retains notification scores and QA. The Sci appendix lists thirteen numeric models while text describes ten paired cross-track baselines; this bundle uses Daily only and does not silently resolve that Sci cohort discrepancy.","zh":"13底模在Daily同100片段运行该任务，采用该任务完整金标事件：信息捕获278、状态跟踪483、位置跟踪481。精确率分母是发出动作，召回分母为全部机会，取两者F1。Intervention须产生有效开始至停止的记忆工具生命周期；无效或缺失工具调用为实际模型行为，VideoLLM-Online已运行零分不属基础设施缺失。匈牙利匹配结合语义相似度及时间，相似度至少0.5、非对称时间窗、时间项指数1.5；语义裁判温度零。另一个条件消息正确性指标固定DeepSeekV4Flash，此处不导入。Start@0.2以已匹配动作为条件，另行排除。保留完整通知、工具与QA原表。Sci附录列13行数值但正文称10个配对模型，本批仅用Daily，不擅自消解该差异。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.30089v1","published_at":"2026-10-21","status":"paper_baselines"},"model_count":13,"result_count":13,"results":[[2055,0.11,370,13663,13664],[4958,0.067,370,13663,13664],[8143,0.155,370,13663,13664],[5300,0.201,370,13663,13664],[3763,0.15,370,13663,13664],[4070,0.0,370,13663,13664],[6349,0.009000000000000001,370,13663,13664],[10076,0.0,370,13663,13664],[8381,0.0,370,13663,13664],[13665,0.004,370,13663,13664],[12296,0.0,370,13663,13664],[5723,0.055999999999999994,370,13663,13664],[13666,0.0,370,13663,13664]]},{"id":"proactegomem-daily-location-tracking-2026-paper","name":"ProactEgoMem · Location tracking (2026 paper)","group":"Agent 与工具","metric":"Intervention detection F1","unit":"%","public_history":true,"protocol":{"en":"Thirteen bases are evaluated on the same 100 Daily clips for Location tracking, containing 481 ground-truth events. Precision uses emitted actions, recall all ground-truth opportunities, and the table reports their F1. This Intervention Level requires a valid start-to-stop memory-tool lifecycle; invalid or absent tool calls are model behavior, including the executed VideoLLM-Online zero, not missing infrastructure. Hungarian matching combines semantic similarity and temporal compatibility; pairs require similarity≥0.5 and an asymmetric time window, with time exponent 1.5. Semantic judging uses temperature zero; the paper separately pins DeepSeek-V4-Flash for conditional message correctness, which is not imported here. Timing Start@0.2 conditions on already matched actions and is excluded. The complete original table also retains notification scores and QA. The Sci appendix lists thirteen numeric models while text describes ten paired cross-track baselines; this bundle uses Daily only and does not silently resolve that Sci cohort discrepancy.","zh":"13底模在Daily同100片段运行该任务，采用该任务完整金标事件：信息捕获278、状态跟踪483、位置跟踪481。精确率分母是发出动作，召回分母为全部机会，取两者F1。Intervention须产生有效开始至停止的记忆工具生命周期；无效或缺失工具调用为实际模型行为，VideoLLM-Online已运行零分不属基础设施缺失。匈牙利匹配结合语义相似度及时间，相似度至少0.5、非对称时间窗、时间项指数1.5；语义裁判温度零。另一个条件消息正确性指标固定DeepSeekV4Flash，此处不导入。Start@0.2以已匹配动作为条件，另行排除。保留完整通知、工具与QA原表。Sci附录列13行数值但正文称10个配对模型，本批仅用Daily，不擅自消解该差异。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.30089v1","published_at":"2026-10-21","status":"paper_baselines"},"model_count":13,"result_count":13,"results":[[2055,0.087,370,13663,13664],[4958,0.025,370,13663,13664],[8143,0.09,370,13663,13664],[5300,0.142,370,13663,13664],[3763,0.125,370,13663,13664],[4070,0.012,370,13663,13664],[6349,0.061,370,13663,13664],[10076,0.011000000000000001,370,13663,13664],[8381,0.008,370,13663,13664],[13665,0.0,370,13663,13664],[12296,0.0,370,13663,13664],[5723,0.02,370,13663,13664],[13666,0.0,370,13663,13664]]},{"id":"rdlbench-predictive-classification-2026-paper","name":"RDL-Bench · future classification (2026 paper)","group":"预测与决策","metric":"Six-domain macro accuracy","unit":"%","public_history":true,"protocol":{"en":"Twelve configurations cover three LLM bases and six learned tabular/relational model architectures. Schema-only and GraphCtx variants share their LLM base. All six temporal domains are weighted equally: Olist, Financial, NCAA, Airline, Stats and Dallas. The source reports accuracy on Boolean future outcomes under reference-time-restricted context; labels come from later records. For trained predictors each template samples up to 5,000 anchor entities split 60/20/20 train/validation/test; exact per-template retained counts are not listed and are not invented. Features use only pre-reference context. LLM references use deterministic three-shot prompting, with aggregate statistics or two-hop graph context; trained predictors use oracle task specifications and validation selection. Supervision/input differences are explicit native solver configurations. The overall 4,040-instance benchmark also contains reasoning tasks, which are not this predictive denominator.","zh":"12配置包含三LLM底模及六个表格/关系学习模型架构；schema-only和GraphCtx归同LLM。六个时间域Olist、Financial、NCAA、Airline、Stats、Dallas等权平均Boolean未来结果准确率。输入截止参考时间，标签来自后续记录。训练预测器每模板最多采5000锚实体，60/20/20分为训练、验证、测试；论文未列各模板最终保留数，本包不编造。特征只来自参考时间之前。LLM确定性三示例提示，使用聚合统计或两跳图上下文；训练预测器使用oracle任务定义和验证选择。监督、输入差异明确记作原生求解配置。完整基准4040实例另含推理题，不混作该轨分母。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.21382v1","published_at":"2026-10-15","status":"paper_baselines"},"model_count":9,"result_count":12,"results":[[6978,0.71,370,13667,13668],[1645,0.59,370,13667,13668],[562,0.57,370,13667,13668],[6978,0.74,370,13669,13668],[1645,0.61,370,13669,13668],[562,0.59,370,13669,13668],[6811,0.6,370,13670,13668],[10503,0.63,370,13670,13668],[9340,0.72,370,13670,13668],[13671,0.75,370,13670,13668],[13672,0.73,370,13670,13668],[13673,0.77,370,13670,13668]]},{"id":"ppgbench-dalia-activity-linear-2026-paper","name":"PPG-Bench · Dalia activity (2026 paper)","group":"预测与决策","metric":"Balanced accuracy","unit":"%","public_history":true,"protocol":{"en":"Six independently released pretrained models use the same 15-participant Dalia activity task under frozen-representation linear probing. Balanced accuracy is the unweighted mean of class recalls; the table reports means with Student-t95% confidence intervals over subject-disjoint five-fold outer cross-validation and five seeds. Probe hyperparameters are selected by full subject-disjoint inner cross-validation; native model preprocessing is retained. The subject count is the dataset cohort, not a per-fold count or a claim that windows are independent. AnyPPG, PaPaGei-P/S, PulsePPG, SigmaPPG and MOMENT-1-small are the six trained representations; random and deterministic PRV controls stay in the complete source table and do not count as bases. Original PaPaGei Sections3.1/3.2 and its official README establish P as participant-contrastive training and S as morphology-aware training with a three-expert MoE architecture, rather than inference switches. PPG-Bench Table7 gives5.0M/5.8M sizes. The identification task’s unexplained subject-disjoint protocol is excluded, as are clinical trait estimates.","zh":"六个独立发布的预训练表征在15位参与者的Dalia activity共同任务上做冻结表征线性探测。平衡准确率为各类召回率等权平均；表中均值及Student-t95%区间来自参与者不重叠的五折外部交叉验证与五种子。线性探测超参数采用完整参与者不重叠的内部交叉验证，各模型保留自身预处理。参与者数为数据集总人群，非每折题数，也不把窗口当独立参与者。AnyPPG、PaPaGeiP/S、PulsePPG、SigmaPPG、MOMENT-1-small是六训练表征；随机与确定性PRV仅保留原表不计底模。PaPaGei原论文3.1/3.2及官方README证实P为参与者对比训练，S为形态感知训练且使用三专家MoE，非推理开关；本论文表7参数分别5.0M、5.8M。排除尚未解释参与者不重叠识别协议的identification任务及临床特征估计。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.13435v1","published_at":"2026-10-09","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[13674,0.345,370,13675,13676],[13677,0.237,370,13675,13676],[13678,0.256,370,13675,13676],[13679,0.294,370,13675,13676],[13680,0.27,370,13675,13676],[13681,0.346,370,13675,13676]]},{"id":"ppgbench-wesad-emotion-linear-2026-paper","name":"PPG-Bench · WESAD emotion (2026 paper)","group":"预测与决策","metric":"Balanced accuracy","unit":"%","public_history":true,"protocol":{"en":"Six independently released pretrained models use the same 15-participant WESAD emotion task under frozen-representation linear probing. Balanced accuracy is the unweighted mean of class recalls; the table reports means with Student-t95% confidence intervals over subject-disjoint five-fold outer cross-validation and five seeds. Probe hyperparameters are selected by full subject-disjoint inner cross-validation; native model preprocessing is retained. The subject count is the dataset cohort, not a per-fold count or a claim that windows are independent. AnyPPG, PaPaGei-P/S, PulsePPG, SigmaPPG and MOMENT-1-small are the six trained representations; random and deterministic PRV controls stay in the complete source table and do not count as bases. Original PaPaGei Sections3.1/3.2 and its official README establish P as participant-contrastive training and S as morphology-aware training with a three-expert MoE architecture, rather than inference switches. PPG-Bench Table7 gives5.0M/5.8M sizes. The identification task’s unexplained subject-disjoint protocol is excluded, as are clinical trait estimates.","zh":"六个独立发布的预训练表征在15位参与者的WESAD emotion共同任务上做冻结表征线性探测。平衡准确率为各类召回率等权平均；表中均值及Student-t95%区间来自参与者不重叠的五折外部交叉验证与五种子。线性探测超参数采用完整参与者不重叠的内部交叉验证，各模型保留自身预处理。参与者数为数据集总人群，非每折题数，也不把窗口当独立参与者。AnyPPG、PaPaGeiP/S、PulsePPG、SigmaPPG、MOMENT-1-small是六训练表征；随机与确定性PRV仅保留原表不计底模。PaPaGei原论文3.1/3.2及官方README证实P为参与者对比训练，S为形态感知训练且使用三专家MoE，非推理开关；本论文表7参数分别5.0M、5.8M。排除尚未解释参与者不重叠识别协议的identification任务及临床特征估计。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.13435v1","published_at":"2026-10-09","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[13674,0.543,370,13675,13676],[13677,0.396,370,13675,13676],[13678,0.416,370,13675,13676],[13679,0.474,370,13675,13676],[13680,0.456,370,13675,13676],[13681,0.504,370,13675,13676]]},{"id":"ppgbench-dreamt-sleep-staging-linear-2026-paper","name":"PPG-Bench · DREAMT sleep staging (2026 paper)","group":"预测与决策","metric":"Balanced accuracy","unit":"%","public_history":true,"protocol":{"en":"Six independently released pretrained models use the same 100-participant DREAMT sleep staging task under frozen-representation linear probing. Balanced accuracy is the unweighted mean of class recalls; the table reports means with Student-t95% confidence intervals over subject-disjoint five-fold outer cross-validation and five seeds. Probe hyperparameters are selected by full subject-disjoint inner cross-validation; native model preprocessing is retained. The subject count is the dataset cohort, not a per-fold count or a claim that windows are independent. AnyPPG, PaPaGei-P/S, PulsePPG, SigmaPPG and MOMENT-1-small are the six trained representations; random and deterministic PRV controls stay in the complete source table and do not count as bases. Original PaPaGei Sections3.1/3.2 and its official README establish P as participant-contrastive training and S as morphology-aware training with a three-expert MoE architecture, rather than inference switches. PPG-Bench Table7 gives5.0M/5.8M sizes. The identification task’s unexplained subject-disjoint protocol is excluded, as are clinical trait estimates.","zh":"六个独立发布的预训练表征在100位参与者的DREAMT sleep staging共同任务上做冻结表征线性探测。平衡准确率为各类召回率等权平均；表中均值及Student-t95%区间来自参与者不重叠的五折外部交叉验证与五种子。线性探测超参数采用完整参与者不重叠的内部交叉验证，各模型保留自身预处理。参与者数为数据集总人群，非每折题数，也不把窗口当独立参与者。AnyPPG、PaPaGeiP/S、PulsePPG、SigmaPPG、MOMENT-1-small是六训练表征；随机与确定性PRV仅保留原表不计底模。PaPaGei原论文3.1/3.2及官方README证实P为参与者对比训练，S为形态感知训练且使用三专家MoE，非推理开关；本论文表7参数分别5.0M、5.8M。排除尚未解释参与者不重叠识别协议的identification任务及临床特征估计。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.13435v1","published_at":"2026-10-09","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[13674,0.25,370,13675,13676],[13677,0.244,370,13675,13676],[13678,0.246,370,13675,13676],[13679,0.247,370,13675,13676],[13680,0.249,370,13675,13676],[13681,0.25,370,13675,13676]]},{"id":"solbehave-raw-compilation-2026-paper","name":"SolBehave · raw compilation (2026 paper)","group":"Agent 与工具","metric":"Per-sample compilation fraction","unit":"%","public_history":true,"protocol":{"en":"Seven underlying models independently regenerate all 250 verified Ethereum contracts ten times each: 2,500 samples per model. Comments are stripped and freshly generated NatSpec replaces original documentation; function bodies are completed sequentially from the same skeleton. Compile@10 is explicitly the fraction of ten samples that compile, not the probability that at least one passes. Raw and one-repair-pass conditions remain separate. Every model uses 4-bit GGUF llama.cpp on one RTX5090, 60k context, temperature0.7, and at most2048 output tokens/function. This is syntactic compilation only; semantic oracles use variable compiling-sample subsets and are preserved in original source evidence but not mixed into this denominator.","zh":"七底模各对250个可复现以太坊合约独立重建10次，每模型2500样本。去除原注释并重新生成NatSpec，同骨架按函数顺序补全。原文Compile@10明确定义为十样本可编译比例，不是至少一次通过概率。原始生成与单次修复分开。均为4bitGGUF、llama.cpp、单RTX5090、60k上下文、温度0.7、每函数最多2048输出token。此处仅为语法可编译率；语义预言机使用按模型不同的可编译样本子集，保留证据但不混入此分母。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.18745v1","published_at":"2026-10-13","status":"paper_baselines"},"model_count":7,"result_count":7,"results":[[5480,0.8909999999999999,370,13682,13683],[3763,0.647,370,13682,13683],[13684,0.532,370,13682,13683],[4968,0.48,370,13682,13683],[13685,0.41200000000000003,370,13682,13683],[7690,0.34299999999999997,370,13682,13683],[13686,0.188,370,13682,13683]]},{"id":"solbehave-repaired-compilation-2026-paper","name":"SolBehave · repaired compilation (2026 paper)","group":"Agent 与工具","metric":"Per-sample compilation fraction","unit":"%","public_history":true,"protocol":{"en":"Seven underlying models independently regenerate all 250 verified Ethereum contracts ten times each: 2,500 samples per model. Comments are stripped and freshly generated NatSpec replaces original documentation; function bodies are completed sequentially from the same skeleton. Compile@10 is explicitly the fraction of ten samples that compile, not the probability that at least one passes. Raw and one-repair-pass conditions remain separate. Every model uses 4-bit GGUF llama.cpp on one RTX5090, 60k context, temperature0.7, and at most2048 output tokens/function. This is syntactic compilation only; semantic oracles use variable compiling-sample subsets and are preserved in original source evidence but not mixed into this denominator.","zh":"七底模各对250个可复现以太坊合约独立重建10次，每模型2500样本。去除原注释并重新生成NatSpec，同骨架按函数顺序补全。原文Compile@10明确定义为十样本可编译比例，不是至少一次通过概率。原始生成与单次修复分开。均为4bitGGUF、llama.cpp、单RTX5090、60k上下文、温度0.7、每函数最多2048输出token。此处仅为语法可编译率；语义预言机使用按模型不同的可编译样本子集，保留证据但不混入此分母。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.18745v1","published_at":"2026-10-13","status":"paper_baselines"},"model_count":7,"result_count":7,"results":[[5480,0.927,370,13687,13683],[3763,0.713,370,13687,13683],[13684,0.618,370,13687,13683],[4968,0.579,370,13687,13683],[13685,0.521,370,13687,13683],[7690,0.39899999999999997,370,13687,13683],[13686,0.264,370,13687,13683]]},{"id":"agentdecision-common-state-tool-accuracy-2026-paper","name":"AgentDecisionBench · common tool states (2026 paper)","group":"Agent 与工具","metric":"Task-weighted tool-choice accuracy","unit":"%","public_history":true,"protocol":{"en":"Eleven decision-model configurations answer the same5,314 recorded states from200 DecisionGym tasks. Each model selects the highest-probability listed tool option; all plan steps that make progress are accepted by the answer key. State accuracy is averaged within task and then equally across200 tasks, rather than pooling states. Five separately sized StartLux released models plus Qwen3-4B already supply six underlying bases; Jev and Intern-Decision are additional systems, while SFT/GRPO and StartLux-4B adaptation do not inflate the base count. Exploratory sizes and generator-trained variants remain explicitly marked. The generating Flash agent’s own-choice reference is excluded because it is its own trajectory reference. This is the original common-state panel, not the model-specific end-to-end delegation combinations or shifted-task subsets.","zh":"11决策配置回答相同200个DecisionGym任务的5314记录状态，选择概率最高的工具选项；答案键接受所有推进计划的动作。先在任务内平均状态准确率，再对200任务等权，不直接合并状态数。五个不同规模的独立发布StartLux与Qwen3-4B已构成六底模；Jev与Intern-Decision另列，SFT/GRPO及StartLux4B微调不额外计底模。探索规模与在同生成器训练的变体明确保留。Flash自己的轨迹选择仅为参考，不导入。此处为原始共同状态面板，不混合各模型不同的端到端委托流程和迁移任务子集。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.21412v1","published_at":"2026-10-15","status":"paper_baselines"},"model_count":8,"result_count":11,"results":[[11731,0.915,370,13688,13689],[13690,0.752,370,13691,13689],[13692,0.9390000000000001,370,13693,13689],[13694,0.9179999999999999,370,13695,13689],[13696,0.8909999999999999,370,13697,13689],[13698,0.684,370,13699,13689],[13700,0.483,370,13701,13689],[5126,0.501,370,13702,13689],[5126,0.992,370,13703,13689],[5126,0.992,370,13704,13689],[13696,0.9940000000000001,370,13705,13689]]},{"id":"basis-elip-adult-implicit-personalization-2026-paper","name":"BASIS · adult personalization (2026 paper)","group":"专业工作","metric":"Age-appropriate response accuracy","unit":"%","public_history":true,"protocol":{"en":"Eight independently released speech models answer the same1840 ELIP-Pair adult-voice audio files without explicit age labels. The corpus has521 unique questions and3697 retained audio files after exact-ASR fidelity checks, pretrained age/gender filtering and manual outlier review. Four child voices total1857 files; four adult/elderly voices total1840. GPT-5.4-mini-2026-03-17 judges each generated answer as child- or adult-oriented; target-persona matches define per-demographic accuracy. Speaker synthesis uses only cloned synthetic voices. Sampling temperature is zero for candidates and judge. Voice-conditioned examples share underlying questions, so audio files are not claimed to be independent questions. This panel does not mix explicit-age prompting, text controls, stereotypical-bias ICAT or health/product tasks.","zh":"八个独立发布语音模型回答相同年龄组的ELIP-Pair语音，不提供显式年龄标签。521个独立问题合成语音后，经ASR精确保真检查、预训练年龄性别过滤与人工离群复核，保留3697文件；四童声共1857，四成年/老年声共1840。GPT5.4mini-2026-03-17把回答判作儿童或成人风格，与目标标签匹配得到组内准确率。只克隆已有合成声，候选及裁判温度零。多个声线共享问题，不把语音文件宣称为独立问题。不混入显式年龄提示、文本对照、刻板偏见ICAT或健康产品任务。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.13550v1","published_at":"2026-10-09","status":"paper_baselines"},"model_count":8,"result_count":8,"results":[[13706,0.541,370,13707,13708],[13709,0.38,370,13707,13708],[13710,0.45799999999999996,370,13707,13708],[13711,0.568,370,13707,13708],[6138,0.276,370,13707,13708],[6139,0.263,370,13707,13708],[7727,0.307,370,13707,13708],[13712,0.172,370,13707,13708]]},{"id":"basis-elip-child-implicit-personalization-2026-paper","name":"BASIS · child personalization (2026 paper)","group":"专业工作","metric":"Age-appropriate response accuracy","unit":"%","public_history":true,"protocol":{"en":"Eight independently released speech models answer the same1857 ELIP-Pair child-voice audio files without explicit age labels. The corpus has521 unique questions and3697 retained audio files after exact-ASR fidelity checks, pretrained age/gender filtering and manual outlier review. Four child voices total1857 files; four adult/elderly voices total1840. GPT-5.4-mini-2026-03-17 judges each generated answer as child- or adult-oriented; target-persona matches define per-demographic accuracy. Speaker synthesis uses only cloned synthetic voices. Sampling temperature is zero for candidates and judge. Voice-conditioned examples share underlying questions, so audio files are not claimed to be independent questions. This panel does not mix explicit-age prompting, text controls, stereotypical-bias ICAT or health/product tasks.","zh":"八个独立发布语音模型回答相同年龄组的ELIP-Pair语音，不提供显式年龄标签。521个独立问题合成语音后，经ASR精确保真检查、预训练年龄性别过滤与人工离群复核，保留3697文件；四童声共1857，四成年/老年声共1840。GPT5.4mini-2026-03-17把回答判作儿童或成人风格，与目标标签匹配得到组内准确率。只克隆已有合成声，候选及裁判温度零。多个声线共享问题，不把语音文件宣称为独立问题。不混入显式年龄提示、文本对照、刻板偏见ICAT或健康产品任务。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.13550v1","published_at":"2026-10-09","status":"paper_baselines"},"model_count":8,"result_count":8,"results":[[13706,0.457,370,13707,13708],[13709,0.62,370,13707,13708],[13710,0.696,370,13707,13708],[13711,0.58,370,13707,13708],[6138,0.816,370,13707,13708],[6139,0.871,370,13707,13708],[7727,0.726,370,13707,13708],[13712,0.8490000000000001,370,13707,13708]]},{"id":"omniubench-benign-utility-2026-paper","name":"Omni-UBench · benign utility (2026 paper)","group":"安全与可靠性","metric":"Benign pass-through macro accuracy","unit":"%","public_history":true,"protocol":{"en":"Fifteen configurations share all required text/image/audio/video support, with at least six underlying omni-modal bases: Qwen2.5-Omni3B/7B,Qwen3-Omni30B,MiniCPM-o4.5,Gemini2.5Flash,MiMoV2.5. GuardReasoner,OmniGuard and authors’ fine-tunes keep their original Qwen/CPM bases. Phi4 and Phi4 fine-tunes lack video; GPT4o-mini lacks audio/video, so their incomplete macro scores are excluded rather than padded. Benign utility uses817 TruthfulQA,2374 MME,636 VoiceBench and1998 MMBench-Video cases (5825 total) and macro-averages benign pass-through accuracy by source/modality. Basic safety macro-averages published F1/Recall across the complete matched source/modality panel in Table5; it remains a native0–100 composite, not a single pooled success rate. SourceTable5 preserves every constituent denominator and supported modality combination; the183977-sample entire corpus includes separate jailbreak/reliability modules and is not this subpanel’s denominator. Labels use a unified ten-risk taxonomy, and fine-tune data are split from evaluation.","zh":"15配置均支持所需文本、图像、音频、视频，至少六底模为Qwen2.5Omni3B/7B、Qwen3Omni30B、MiniCPMo4.5、Gemini2.5Flash、MiMoV2.5。GuardReasoner、OmniGuard及作者微调归原Qwen/CPM底模。Phi4两配置缺视频、GPT4omini缺音视频，因此排除其不完整宏分，不补零。benign utility为TruthfulQA817、MME2374、VoiceBench636、MMBenchVideo1998，共5825案例，按来源与模态宏平均安全放行准确率。basic safety按表5共同来源模态面板宏平均原始F1/Recall，保留0至100综合量表，不当单一合并成功率。表5保留各组成分母和支持的模态组合；完整语料183977另含越狱和可靠性模块，不混作该子面板分母。标签采用统一十类风险，微调与评估数据分开。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.31713v1","published_at":"2026-10-22","status":"paper_baselines"},"model_count":7,"result_count":15,"results":[[6136,0.345,370,13713,13714],[6138,0.948,370,13713,13714],[9195,0.94,370,13713,13714],[8381,0.993,370,13713,13714],[766,0.93,370,13713,13714],[7223,0.985,370,13713,13714],[9195,0.95,370,13715,13714],[8381,0.985,370,13715,13714],[13716,0.963,370,13717,13714],[6136,0.738,370,13718,13714],[6138,0.735,370,13718,13714],[6136,0.965,370,13719,13714],[6138,0.97,370,13719,13714],[6136,0.968,370,13720,13721],[6138,0.968,370,13720,13721]]},{"id":"omniubench-basic-safety-2026-paper","name":"Omni-UBench · basic safety (2026 paper)","group":"安全与可靠性","metric":"Basic safety macro score","unit":"score","public_history":true,"protocol":{"en":"Fifteen configurations share all required text/image/audio/video support, with at least six underlying omni-modal bases: Qwen2.5-Omni3B/7B,Qwen3-Omni30B,MiniCPM-o4.5,Gemini2.5Flash,MiMoV2.5. GuardReasoner,OmniGuard and authors’ fine-tunes keep their original Qwen/CPM bases. Phi4 and Phi4 fine-tunes lack video; GPT4o-mini lacks audio/video, so their incomplete macro scores are excluded rather than padded. Benign utility uses817 TruthfulQA,2374 MME,636 VoiceBench and1998 MMBench-Video cases (5825 total) and macro-averages benign pass-through accuracy by source/modality. Basic safety macro-averages published F1/Recall across the complete matched source/modality panel in Table5; it remains a native0–100 composite, not a single pooled success rate. SourceTable5 preserves every constituent denominator and supported modality combination; the183977-sample entire corpus includes separate jailbreak/reliability modules and is not this subpanel’s denominator. Labels use a unified ten-risk taxonomy, and fine-tune data are split from evaluation.","zh":"15配置均支持所需文本、图像、音频、视频，至少六底模为Qwen2.5Omni3B/7B、Qwen3Omni30B、MiniCPMo4.5、Gemini2.5Flash、MiMoV2.5。GuardReasoner、OmniGuard及作者微调归原Qwen/CPM底模。Phi4两配置缺视频、GPT4omini缺音视频，因此排除其不完整宏分，不补零。benign utility为TruthfulQA817、MME2374、VoiceBench636、MMBenchVideo1998，共5825案例，按来源与模态宏平均安全放行准确率。basic safety按表5共同来源模态面板宏平均原始F1/Recall，保留0至100综合量表，不当单一合并成功率。表5保留各组成分母和支持的模态组合；完整语料183977另含越狱和可靠性模块，不混作该子面板分母。标签采用统一十类风险，微调与评估数据分开。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.31713v1","published_at":"2026-10-22","status":"paper_baselines"},"score_max":100,"score_decimals":2,"saturation_excluded":true,"scale_label":{"en":"Native score / 100","zh":"原生分数 / 100"},"model_count":7,"result_count":15,"results":[[6136,83.8,370,13713,13714],[6138,77.9,370,13713,13714],[9195,79.3,370,13713,13714],[8381,61.8,370,13713,13714],[766,88.9,370,13713,13714],[7223,85.6,370,13713,13714],[9195,85.4,370,13715,13714],[8381,78.7,370,13715,13714],[13716,71.8,370,13717,13714],[6136,88.5,370,13718,13714],[6138,90.1,370,13718,13714],[6136,71.0,370,13719,13714],[6138,73.6,370,13719,13714],[6136,80.5,370,13720,13721],[6138,79.4,370,13720,13721]]},{"id":"square-bench-semantics-2026-paper","name":"SQUARE-Bench · semantics (2026 paper)","group":"视觉与视频","metric":"Aspect overall QA accuracy","unit":"%","public_history":true,"protocol":{"en":"General LMMs answer the same semantics multiple-choice diagnosis panel for AI-generated images. The benchmark contains approximately 18,000 expert-reviewed instances across 38 subdimensions; individual questions are independently cross-checked by at least three additional annotators. The source aspect Overall accuracy is selected. Human, random and mixed-generator references are excluded from model rankings. Three starred models use LoRA adaptation and remain configurations of their original backbones; GPT-5.2 retains xhigh reasoning.","zh":"通用LMM回答相同semantics的AI生成图片诊断选择题。基准含约18,000条专家审阅实例、38个子维度，每题还经至少三位标注者独立交叉核对。选择该方面的Overall准确率；人工、随机及混合生成器参考不进入模型排行。三条星号模型为LoRA微调，仍计原底模配置，GPT-5.2保留xhigh推理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.37576v1","published_at":"2026-09-29","status":"paper_baselines"},"model_count":23,"result_count":26,"results":[[6705,0.8665,370,13722,13723],[3002,0.9098,370,13722,13723],[340,0.8714,370,13724,13723],[13725,0.7256999999999999,370,13722,13723],[13726,0.7403,370,13722,13723],[13727,0.5607,370,13722,13723],[6689,0.7961,370,13722,13723],[6241,0.8714,370,13722,13723],[2687,0.8495,370,13722,13723],[2728,0.8592,370,13722,13723],[2654,0.8617,370,13722,13723],[2697,0.8543999999999999,370,13722,13723],[6345,0.8471,370,13722,13723],[2642,0.8859,370,13722,13723],[2698,0.8058,370,13722,13723],[13728,0.75,370,13722,13723],[13729,0.7209,370,13722,13723],[13730,0.8811,370,13722,13723],[13731,0.7524,370,13722,13723],[6348,0.8398,370,13722,13723],[2646,0.8665,370,13722,13723],[6343,0.9005,370,13722,13723],[6342,0.8714,370,13722,13723],[13730,0.8737999999999999,370,13732,13723],[6342,0.8859,370,13732,13723],[2646,0.8835,370,13732,13723]]},{"id":"square-bench-quality-2026-paper","name":"SQUARE-Bench · quality (2026 paper)","group":"视觉与视频","metric":"Aspect overall QA accuracy","unit":"%","public_history":true,"protocol":{"en":"General LMMs answer the same quality multiple-choice diagnosis panel for AI-generated images. The benchmark contains approximately 18,000 expert-reviewed instances across 38 subdimensions; individual questions are independently cross-checked by at least three additional annotators. The source aspect Overall accuracy is selected. Human, random and mixed-generator references are excluded from model rankings. Three starred models use LoRA adaptation and remain configurations of their original backbones; GPT-5.2 retains xhigh reasoning.","zh":"通用LMM回答相同quality的AI生成图片诊断选择题。基准含约18,000条专家审阅实例、38个子维度，每题还经至少三位标注者独立交叉核对。选择该方面的Overall准确率；人工、随机及混合生成器参考不进入模型排行。三条星号模型为LoRA微调，仍计原底模配置，GPT-5.2保留xhigh推理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.37576v1","published_at":"2026-09-29","status":"paper_baselines"},"model_count":23,"result_count":26,"results":[[6705,0.48100000000000004,370,13733,13734],[3002,0.6027,370,13733,13734],[340,0.5761,370,13735,13734],[13725,0.5408,370,13733,13734],[13726,0.5489,370,13733,13734],[13727,0.5353,370,13733,13734],[6689,0.6467,370,13733,13734],[6241,0.6114,370,13733,13734],[2687,0.6114,370,13733,13734],[2728,0.6793,370,13733,13734],[2654,0.6658,370,13733,13734],[2697,0.6685,370,13733,13734],[6345,0.6629999999999999,370,13733,13734],[2642,0.7038,370,13733,13734],[2698,0.6168,370,13733,13734],[13728,0.6304,370,13733,13734],[13729,0.5951,370,13733,13734],[13730,0.7254999999999999,370,13733,13734],[13731,0.6359,370,13733,13734],[6348,0.7092,370,13733,13734],[2646,0.7283,370,13733,13734],[6343,0.7526999999999999,370,13733,13734],[6342,0.6629999999999999,370,13733,13734],[13730,0.7554000000000001,370,13736,13734],[6342,0.731,370,13736,13734],[2646,0.8043,370,13736,13734]]},{"id":"square-bench-authenticity-2026-paper","name":"SQUARE-Bench · authenticity (2026 paper)","group":"视觉与视频","metric":"Aspect overall QA accuracy","unit":"%","public_history":true,"protocol":{"en":"General LMMs answer the same authenticity multiple-choice diagnosis panel for AI-generated images. The benchmark contains approximately 18,000 expert-reviewed instances across 38 subdimensions; individual questions are independently cross-checked by at least three additional annotators. The source aspect Overall accuracy is selected. Human, random and mixed-generator references are excluded from model rankings. Three starred models use LoRA adaptation and remain configurations of their original backbones; GPT-5.2 retains xhigh reasoning.","zh":"通用LMM回答相同authenticity的AI生成图片诊断选择题。基准含约18,000条专家审阅实例、38个子维度，每题还经至少三位标注者独立交叉核对。选择该方面的Overall准确率；人工、随机及混合生成器参考不进入模型排行。三条星号模型为LoRA微调，仍计原底模配置，GPT-5.2保留xhigh推理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.37576v1","published_at":"2026-09-29","status":"paper_baselines"},"model_count":23,"result_count":26,"results":[[6705,0.8125,370,13737,13738],[3002,0.8440000000000001,370,13737,13738],[340,0.7001999999999999,370,13739,13738],[13725,0.4771,370,13737,13738],[13726,0.5312,370,13737,13738],[13727,0.5145000000000001,370,13737,13738],[6689,0.6065999999999999,370,13737,13738],[6241,0.5592,370,13737,13738],[2687,0.5837,370,13737,13738],[2728,0.5926,370,13737,13738],[2654,0.5926,370,13737,13738],[2697,0.6038,370,13737,13738],[6345,0.5904,370,13737,13738],[2642,0.639,370,13737,13738],[2698,0.5424,370,13737,13738],[13728,0.6484000000000001,370,13737,13738],[13729,0.5089,370,13737,13738],[13730,0.7589,370,13737,13738],[13731,0.5352,370,13737,13738],[6348,0.5725,370,13737,13738],[2646,0.649,370,13737,13738],[6343,0.774,370,13737,13738],[6342,0.7461,370,13737,13738],[13730,0.779,370,13740,13738],[6342,0.7426999999999999,370,13740,13738],[2646,0.899,370,13740,13738]]},{"id":"square-bench-responsibility-2026-paper","name":"SQUARE-Bench · responsibility (2026 paper)","group":"视觉与视频","metric":"Aspect overall QA accuracy","unit":"%","public_history":true,"protocol":{"en":"General LMMs answer the same responsibility multiple-choice diagnosis panel for AI-generated images. The benchmark contains approximately 18,000 expert-reviewed instances across 38 subdimensions; individual questions are independently cross-checked by at least three additional annotators. The source aspect Overall accuracy is selected. Human, random and mixed-generator references are excluded from model rankings. Three starred models use LoRA adaptation and remain configurations of their original backbones; GPT-5.2 retains xhigh reasoning.","zh":"通用LMM回答相同responsibility的AI生成图片诊断选择题。基准含约18,000条专家审阅实例、38个子维度，每题还经至少三位标注者独立交叉核对。选择该方面的Overall准确率；人工、随机及混合生成器参考不进入模型排行。三条星号模型为LoRA微调，仍计原底模配置，GPT-5.2保留xhigh推理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.37576v1","published_at":"2026-09-29","status":"paper_baselines"},"model_count":23,"result_count":26,"results":[[6705,0.9240999999999999,370,13741,13742],[3002,0.9305,370,13741,13742],[340,0.8713,370,13743,13742],[13725,0.5544,370,13741,13742],[13726,0.8449,370,13741,13742],[13727,0.6831999999999999,370,13741,13742],[6689,0.9043000000000001,370,13741,13742],[6241,0.9240999999999999,370,13741,13742],[2687,0.9109,370,13741,13742],[2728,0.8647,370,13741,13742],[2654,0.9043000000000001,370,13741,13742],[2697,0.9009999999999999,370,13741,13742],[6345,0.9043000000000001,370,13741,13742],[2642,0.9208,370,13741,13742],[2698,0.9208,370,13741,13742],[13728,0.8944,370,13741,13742],[13729,0.9298000000000001,370,13741,13742],[13730,0.9109,370,13741,13742],[13731,0.9009999999999999,370,13741,13742],[6348,0.8944,370,13741,13742],[2646,0.9439,370,13741,13742],[6343,0.9307,370,13741,13742],[6342,0.9274,370,13741,13742],[13730,0.9406,370,13744,13742],[6342,0.9472,370,13744,13742],[2646,0.9503,370,13744,13742]]},{"id":"userproxybench-ufs-2026-paper","name":"UserProxyBench · user fidelity (2026 paper)","group":"Agent 与工具","metric":"Equal-domain User Fidelity Score","unit":"score","public_history":true,"protocol":{"en":"Seven user-proxy LLMs simulate the same 375 enterprise tasks with GPT-5.5 frozen as the downstream agent (114 Telecom, 50 Airline, 114 Retail, 97 Banking). UFS is the fraction of episodes satisfying every applicable user-contract criterion; the headline equally weights four domain means. It measures contract fidelity, not human realism. The rubric generator also judges responses; a limited single-rater manual audit and a judge-free Telecom probe identify missed user actions, so UFS is separate from downstream agent reward. One rollout is recorded per proxy/task.","zh":"七个用户代理LLM在固定下游GPT-5.5上模拟相同375个企业任务：Telecom114、Airline50、Retail114、Banking97。UFS是满足全部适用用户契约条款的会话比例，总分给四领域等权；它衡量角色契约遵守，不代表模拟人类真实性。量表生成器也负责判分，有限单人审查及Telecom确定性检查发现漏操作，故不与下游Agent奖励混排。每个模型每题一条轨迹。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.38043v1","published_at":"2026-09-29","status":"paper_baselines"},"score_max":1,"saturation_excluded":true,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"score_decimals":3,"model_count":7,"result_count":7,"results":[[3572,0.778,370,13745,13746],[2129,0.864,370,13745,13746],[6771,0.612,370,13747,13746],[13748,0.845,370,13747,13746],[13749,0.759,370,13747,13746],[4958,0.599,370,13745,13746],[5406,0.752,370,13745,13746]]},{"id":"vif-bench-overall-2026-paper","name":"VIF-Bench · visual instructions (2026 paper)","group":"生成与创作","metric":"Mean six-criterion judge score","unit":"score","public_history":true,"protocol":{"en":"Eight image-generator checkpoints and two multi-step configurations process the same 1,241 multi-reference visual-instruction tasks. Gemini-2.5-Flash and GPT-5 independently score six criteria; this cohort is the source mean across criteria and the two judges on a native 0–10 scale. Multi-step variants keep their GPT-Image-1.5/Nano Banana Pro base identities. Qwen-Image-2511/2509 source names are preserved; text-instruction ablations intentionally contain different information and are not pooled.","zh":"八个图像生成检查点及两个多步配置，处理相同1,241条多参考视觉指令任务。Gemini-2.5-Flash与GPT-5分别按六维度评价，本组取原文跨维度和双裁判均分，保留0–10原生量表。多步配置仍计GPT-Image-1.5/Nano Banana Pro底模。Qwen-Image-2511/2509保留原名；文本指令消融的信息量不同，不混排。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.37709v1","published_at":"2026-09-29","status":"paper_baselines"},"score_max":10,"saturation_excluded":true,"scale_label":{"en":"Score / 10","zh":"分数 / 10"},"score_decimals":2,"model_count":7,"result_count":10,"results":[[6367,7.6,370,13750,13751],[6367,7.53,370,13752,13751],[6366,6.96,370,13750,13751],[6366,7.4,370,13752,13751],[6841,7.55,370,13750,13751],[6365,7.13,370,13750,13751],[13753,4.94,370,13750,13751],[13754,3.93,370,13750,13751],[6363,4.64,370,13750,13751],[8151,4.61,370,13750,13751]]},{"id":"vista-bench-image-translation-2026-paper","name":"VISTA-Bench · image translation (2026 paper)","group":"视觉与视频","metric":"Global translation quality T","unit":"score","public_history":true,"protocol":{"en":"Twelve image-input multimodal models translate the same 2,228-image benchmark covering 22 languages and ten domains. Overall is global translation quality T averaged over directions, not the visual/knowledge composite or a percentage of perfectly translated images. Human-verified semantic blocks and content-specific rubrics define required meanings, relations and acceptable variants. Four text-input baselines and the four-model 10,920-image comparison are different input/data cohorts and are excluded.","zh":"十二个图片输入多模态模型，翻译相同2,228张图片，涵盖22种语言和十个领域。Overall是跨翻译方向平均的总体翻译质量T，不是视觉/知识综合分，也不是完全正确图片百分比。金标准采用人工核验语义块和内容特定量表，明确必需含义、关系及可接受变体。四个文本输入基线、四模型10,920图片比较属于不同输入或数据队列，不混入本组。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.37287v1","published_at":"2026-09-29","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":12,"result_count":12,"results":[[3408,55.88,370,13755,13756],[3568,69.43,370,13755,13756],[2979,71.56,370,13755,13756],[2982,72.67,370,13755,13756],[2983,74.2,370,13755,13756],[5361,75.12,370,13755,13756],[2958,77.63,370,13755,13756],[13757,19.54,370,13755,13756],[13758,29.96,370,13755,13756],[2980,55.33,370,13755,13756],[5394,61.04,370,13755,13756],[2986,61.11,370,13755,13756]]},{"id":"autodatabench-id-overall-2026-paper","name":"AutoDataBench · ID optimization (2026 paper)","group":"Agent 与工具","metric":"Mean three-task ID optimization score","unit":"score","public_history":true,"protocol":{"en":"Seven frontier LLMs optimize only data under a common agent framework while target training pipelines remain fixed. Overall averages the three task-wise ID means over tool use, retrieval and knowledge injection; each task uses three standard runs and each run selects its best ID checkpoint. This is an optimization-feedback score, separate from hidden OOD generalization. AST accuracy, retrieval and knowledge scores are combined only in the source native aggregate; expert and unchanged-data controls are excluded.","zh":"七个前沿LLM在相同Agent框架中只优化数据，目标训练流程固定。Overall平均工具使用、检索、知识注入三任务的ID均分；各任务三次标准运行，每次选择ID分最高的检查点。本组衡量利用反馈的优化成绩，与隐藏OOD泛化分开。AST准确率、检索和知识成绩只在原文定义的原生综合分中聚合；人工专家和不改数据对照不计参测模型。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.40097v1","published_at":"2026-09-30","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":7,"result_count":7,"results":[[3201,55.2,370,13759,13760],[5138,56.8,370,13759,13760],[13761,58.31,370,13759,13760],[5133,58.56,370,13759,13760],[4979,58.85,370,13759,13760],[227,59.1,370,13759,13760],[13762,60.68,370,13759,13760]]},{"id":"dogbench-documentation-2026-paper","name":"DoGBench · documentation (2026 paper)","group":"编程","metric":"Documentation and abstention composite","unit":"score","public_history":true,"protocol":{"en":"Seven native coding-agent backbones address the same stratified 117-item held-out split from 292 documentation tasks. Each receives the pre-change repository and a trigger, then must deliver one acceptable patch or abstain correctly. The native /100 composite combines patch quality with abstention; a value is not a percentage of an expert’s capability. Maintainer-validated task rubrics assess accuracy, completeness, reader guidance, placement and conventions. Conditional quality and the separate historical 1,267-patch failure audit are excluded from the headline cohort.","zh":"七个原生代码Agent底模，在292条文档任务中的相同分层117条留出集上评测。输入为变更前仓库和触发需求，要求一次提交合格补丁或正确不修改。百分制原生综合分结合文档质量与弃答，不当作专家能力百分比。由维护者验证的任务量表评价准确性、完整性、读者引导、位置和仓库惯例；条件质量分及另外1,267补丁的历史错误审计不混入本组。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.39909v1","published_at":"2026-09-30","status":"paper_baselines"},"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"score_decimals":2,"model_count":7,"result_count":7,"results":[[2982,47.3,370,13763,13764],[762,46.2,370,13765,13764],[3654,44.2,370,13763,13764],[3411,43.9,370,13763,13764],[3479,41.2,370,13766,13764],[3572,40.0,370,13766,13764],[3196,34.0,370,13765,13764]]},{"id":"video2skill-unified-robointer-coverage-2026-paper","name":"Video2Skill · robointer unified event coverage (2026 paper)","group":"Agent 与工具","metric":"Matched reference-event coverage","unit":"%","public_history":true,"protocol":{"en":"Nineteen source VLM backbones watch the same robointer video stream under the unified library-update interface. Coverage uses the fixed verified reference-event set. Unmatched references reduce coverage; unmatched predictions are not penalized, so this is event recovery rather than complete skill-discovery accuracy. The source ARI and creation/reuse scores condition on model-dependent matched subsets and are not substituted for this fixed-denominator metric. Unified processing jointly recognizes events and updates the library.","zh":"十九个原文VLM底模，在unified技能库接口下观看相同robointer视频流。覆盖率采用固定的已核验参考事件集合，未匹配参考降低覆盖率，额外预测不扣分，因此它表示事件找回而非完整技能发现准确率。原文ARI及创建/复用指标只在各模型不同的匹配子集上计算，不替代本组固定分母指标。统一流程同时识别事件和更新技能库。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.36691v1","published_at":"2026-09-29","status":"paper_baselines"},"model_count":19,"result_count":19,"results":[[6340,0.024,370,13767,13768],[5828,0.39299999999999996,370,13767,13768],[4969,0.569,370,13767,13768],[4070,0.44799999999999995,370,13767,13768],[2584,0.479,370,13767,13768],[2590,0.5710000000000001,370,13767,13768],[6342,0.56,370,13767,13768],[5300,0.637,370,13767,13768],[2687,0.41600000000000004,370,13767,13768],[2654,0.485,370,13767,13768],[6345,0.509,370,13767,13768],[2642,0.414,370,13767,13768],[2646,0.049,370,13767,13768],[7066,0.485,370,13767,13768],[7554,0.5429999999999999,370,13767,13768],[7485,0.37799999999999995,370,13767,13768],[6296,0.401,370,13767,13768],[6630,0.43799999999999994,370,13767,13768],[5480,0.526,370,13767,13768]]},{"id":"video2skill-unified-hd-epic-coverage-2026-paper","name":"Video2Skill · hd-epic unified event coverage (2026 paper)","group":"Agent 与工具","metric":"Matched reference-event coverage","unit":"%","public_history":true,"protocol":{"en":"Nineteen source VLM backbones watch the same hd-epic video stream under the unified library-update interface. Coverage uses the fixed verified reference-event set. Unmatched references reduce coverage; unmatched predictions are not penalized, so this is event recovery rather than complete skill-discovery accuracy. The source ARI and creation/reuse scores condition on model-dependent matched subsets and are not substituted for this fixed-denominator metric. Unified processing jointly recognizes events and updates the library.","zh":"十九个原文VLM底模，在unified技能库接口下观看相同hd-epic视频流。覆盖率采用固定的已核验参考事件集合，未匹配参考降低覆盖率，额外预测不扣分，因此它表示事件找回而非完整技能发现准确率。原文ARI及创建/复用指标只在各模型不同的匹配子集上计算，不替代本组固定分母指标。统一流程同时识别事件和更新技能库。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.36691v1","published_at":"2026-09-29","status":"paper_baselines"},"model_count":19,"result_count":19,"results":[[6340,0.002,370,13767,13768],[5828,0.14800000000000002,370,13767,13768],[4969,0.614,370,13767,13768],[4070,0.539,370,13767,13768],[2584,0.513,370,13767,13768],[2590,0.5589999999999999,370,13767,13768],[6342,0.441,370,13767,13768],[5300,0.601,370,13767,13768],[2687,0.19,370,13767,13768],[2654,0.266,370,13767,13768],[6345,0.38,370,13767,13768],[2642,0.423,370,13767,13768],[2646,0.365,370,13767,13768],[7066,0.376,370,13767,13768],[7554,0.444,370,13767,13768],[7485,0.21600000000000003,370,13767,13768],[6296,0.057999999999999996,370,13767,13768],[6630,0.075,370,13767,13768],[5480,0.46,370,13767,13768]]},{"id":"video2skill-factorized-robointer-coverage-2026-paper","name":"Video2Skill · robointer factorized event coverage (2026 paper)","group":"Agent 与工具","metric":"Matched reference-event coverage","unit":"%","public_history":true,"protocol":{"en":"Nineteen source VLM backbones watch the same robointer video stream under the factorized library-update interface. Coverage uses the fixed verified reference-event set. Unmatched references reduce coverage; unmatched predictions are not penalized, so this is event recovery rather than complete skill-discovery accuracy. The source ARI and creation/reuse scores condition on model-dependent matched subsets and are not substituted for this fixed-denominator metric. Factorized processing proposes events without library access, then the same backbone updates the library from text.","zh":"十九个原文VLM底模，在factorized技能库接口下观看相同robointer视频流。覆盖率采用固定的已核验参考事件集合，未匹配参考降低覆盖率，额外预测不扣分，因此它表示事件找回而非完整技能发现准确率。原文ARI及创建/复用指标只在各模型不同的匹配子集上计算，不替代本组固定分母指标。拆分流程先在不可见技能库时提出事件，再由相同底模从文本更新技能库。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.36691v1","published_at":"2026-09-29","status":"paper_baselines"},"model_count":19,"result_count":19,"results":[[6340,0.26,370,13769,13768],[5828,0.423,370,13769,13768],[4969,0.466,370,13769,13768],[4070,0.502,370,13769,13768],[2584,0.47200000000000003,370,13769,13768],[2590,0.573,370,13769,13768],[6342,0.522,370,13769,13768],[5300,0.435,370,13769,13768],[2687,0.419,370,13769,13768],[2654,0.384,370,13769,13768],[6345,0.479,370,13769,13768],[2642,0.483,370,13769,13768],[2646,0.652,370,13769,13768],[7066,0.528,370,13769,13768],[7554,0.47200000000000003,370,13769,13768],[7485,0.7170000000000001,370,13769,13768],[6296,0.365,370,13769,13768],[6630,0.674,370,13769,13768],[5480,0.5489999999999999,370,13769,13768]]},{"id":"video2skill-factorized-hd-epic-coverage-2026-paper","name":"Video2Skill · hd-epic factorized event coverage (2026 paper)","group":"Agent 与工具","metric":"Matched reference-event coverage","unit":"%","public_history":true,"protocol":{"en":"Nineteen source VLM backbones watch the same hd-epic video stream under the factorized library-update interface. Coverage uses the fixed verified reference-event set. Unmatched references reduce coverage; unmatched predictions are not penalized, so this is event recovery rather than complete skill-discovery accuracy. The source ARI and creation/reuse scores condition on model-dependent matched subsets and are not substituted for this fixed-denominator metric. Factorized processing proposes events without library access, then the same backbone updates the library from text.","zh":"十九个原文VLM底模，在factorized技能库接口下观看相同hd-epic视频流。覆盖率采用固定的已核验参考事件集合，未匹配参考降低覆盖率，额外预测不扣分，因此它表示事件找回而非完整技能发现准确率。原文ARI及创建/复用指标只在各模型不同的匹配子集上计算，不替代本组固定分母指标。拆分流程先在不可见技能库时提出事件，再由相同底模从文本更新技能库。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.36691v1","published_at":"2026-09-29","status":"paper_baselines"},"model_count":19,"result_count":19,"results":[[6340,0.111,370,13769,13768],[5828,0.324,370,13769,13768],[4969,0.501,370,13769,13768],[4070,0.49200000000000005,370,13769,13768],[2584,0.52,370,13769,13768],[2590,0.578,370,13769,13768],[6342,0.491,370,13769,13768],[5300,0.527,370,13769,13768],[2687,0.335,370,13769,13768],[2654,0.34700000000000003,370,13769,13768],[6345,0.512,370,13769,13768],[2642,0.49700000000000005,370,13769,13768],[2646,0.475,370,13769,13768],[7066,0.507,370,13769,13768],[7554,0.28800000000000003,370,13769,13768],[7485,0.405,370,13769,13768],[6296,0.252,370,13769,13768],[6630,0.43,370,13769,13768],[5480,0.519,370,13769,13768]]},{"id":"omnireasoningbench-over-video-mcq-2026-paper","name":"OmniReasoningBench · over-video MCQ (2026 paper)","group":"视觉与视频","metric":"Audio-visual MCQ accuracy (rounded)","unit":"%","public_history":true,"protocol":{"en":"Eleven omni-model backbones plus one MFSD adaptation receive the same audio-visual over-video multiple-choice inputs (375 questions). Reference evidence chains are withheld. Three visual-only models are excluded, and open-ended answer results are a separate scoring protocol. This cohort retains the published one-decimal MCQ accuracy, without reconstructing unprinted precision. The MFSD model is a configuration of its Qwen3-Omni initialization. Benchmark construction verifies timestamped clues and screens restricted-modality shortcuts with a fixed construction pipeline.","zh":"十一个全模态底模及一个MFSD微调配置，接收相同over-video的音视频选择题输入，共375题，不提供参考证据链。三个仅视觉模型不进入本组，开放回答也属于另一个评分协议。保留论文印出的一位小数选择题准确率，不反推未公布精度。MFSD模型仍计Qwen3-Omni初始化底模配置。构题流程核验带时间戳线索，并用固定流程筛查单模态捷径。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.39490v1","published_at":"2026-09-30","status":"paper_baselines"},"model_count":11,"result_count":12,"results":[[5906,0.5760000000000001,370,13770,13771],[5519,0.552,370,13770,13771],[2129,0.5329999999999999,370,13770,13771],[502,0.517,370,13770,13771],[13772,0.547,370,13770,13771],[6724,0.42700000000000005,370,13770,13771],[13773,0.381,370,13770,13771],[12515,0.364,370,13770,13771],[7223,0.42700000000000005,370,13770,13771],[8551,0.256,370,13770,13771],[9565,0.319,370,13770,13771],[9565,0.48,370,13774,13771]]},{"id":"omnireasoningbench-beyond-video-mcq-2026-paper","name":"OmniReasoningBench · beyond-video MCQ (2026 paper)","group":"视觉与视频","metric":"Audio-visual MCQ accuracy (rounded)","unit":"%","public_history":true,"protocol":{"en":"Eleven omni-model backbones plus one MFSD adaptation receive the same audio-visual beyond-video multiple-choice inputs (250 questions). Reference evidence chains are withheld. Three visual-only models are excluded, and open-ended answer results are a separate scoring protocol. This cohort retains the published one-decimal MCQ accuracy, without reconstructing unprinted precision. The MFSD model is a configuration of its Qwen3-Omni initialization. Benchmark construction verifies timestamped clues and screens restricted-modality shortcuts with a fixed construction pipeline.","zh":"十一个全模态底模及一个MFSD微调配置，接收相同beyond-video的音视频选择题输入，共250题，不提供参考证据链。三个仅视觉模型不进入本组，开放回答也属于另一个评分协议。保留论文印出的一位小数选择题准确率，不反推未公布精度。MFSD模型仍计Qwen3-Omni初始化底模配置。构题流程核验带时间戳线索，并用固定流程筛查单模态捷径。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.39490v1","published_at":"2026-09-30","status":"paper_baselines"},"model_count":11,"result_count":12,"results":[[5906,0.72,370,13775,13771],[5519,0.696,370,13775,13771],[2129,0.664,370,13775,13771],[502,0.64,370,13775,13771],[13772,0.5720000000000001,370,13775,13771],[6724,0.604,370,13775,13771],[13773,0.6,370,13775,13771],[12515,0.56,370,13775,13771],[7223,0.54,370,13775,13771],[8551,0.36,370,13775,13771],[9565,0.46399999999999997,370,13775,13771],[9565,0.588,370,13776,13771]]},{"id":"patterneval-task-accuracy-2026-paper","name":"PatternEval · hybrid-thinking response quality (2026 paper)","group":"视觉与视频","metric":"Task accuracy","unit":"%","public_history":true,"protocol":{"en":"All twenty-five model checkpoints evaluate the same 2,415 failure-enriched multimodal prompts in both thinking and non-thinking modes. The complete fifty-configuration accuracy panel is retained; paired mode gaps and response-pattern trigger rates are separate metrics. Merged model names and mode cells are resolved with exact source coordinates.","zh":"完整二十五检查点在思考与非思考两模式下评测相同2,415条重点覆盖失败情况的多模态提示。保留完整五十配置准确率面板，模式差距与响应模式错误触发率单列。跨行模型名和模式单元格按准确原坐标解析。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.12781v1","status":"paper_baselines","published_at":"2026-08-13"},"model_count":25,"result_count":50,"results":[[6340,0.2522,370,13777,13778],[6340,0.23129999999999998,370,13779,13778],[4969,0.4596,370,13777,13778],[4969,0.5233,370,13779,13778],[4070,0.5015,370,13777,13778],[4070,0.5920000000000001,370,13779,13778],[2584,0.5907,370,13777,13778],[2584,0.6405,370,13779,13778],[2590,0.5702,370,13777,13778],[2590,0.6507,370,13779,13778],[2582,0.5920000000000001,370,13777,13778],[2582,0.6629999999999999,370,13779,13778],[2576,0.6196,370,13777,13778],[2576,0.6893,370,13779,13778],[3763,0.5733,370,13777,13778],[3763,0.6372,370,13779,13778],[3764,0.5622,370,13777,13778],[3764,0.639,370,13779,13778],[3202,0.6707,370,13777,13778],[3202,0.723,370,13779,13778],[4961,0.6628000000000001,370,13777,13778],[4961,0.7097,370,13779,13778],[6238,0.48210000000000003,370,13777,13778],[6238,0.48869999999999997,370,13779,13778],[6343,0.5321,370,13777,13778],[6343,0.5575,370,13779,13778],[2998,0.6133,370,13777,13778],[2998,0.6706,370,13779,13778],[3187,0.6647,370,13777,13778],[3187,0.7319,370,13779,13778],[13780,0.5949,370,13777,13778],[13780,0.6173,370,13779,13778],[3192,0.6685,370,13777,13778],[3192,0.7298,370,13779,13778],[3235,0.7044,370,13777,13778],[3235,0.8354,370,13779,13778],[3183,0.5945,370,13777,13778],[3183,0.6762999999999999,370,13779,13778],[3194,0.6112,370,13777,13778],[3194,0.677,370,13779,13778],[13781,0.5721,370,13777,13778],[13781,0.5752,370,13779,13778],[13782,0.5503,370,13777,13778],[13782,0.5545,370,13779,13778],[5738,0.3072,370,13777,13778],[5738,0.4436,370,13779,13778],[3184,0.6065999999999999,370,13777,13778],[3184,0.6954,370,13779,13778],[3196,0.5920000000000001,370,13777,13778],[3196,0.7490000000000001,370,13779,13778]]},{"id":"stvg-compbench-easy-2026-paper","name":"STVG-CompBench · easy compositional grounding (2026 paper)","group":"视觉与视频","metric":"Mean spatiotemporal video IoU","unit":"score","public_history":true,"protocol":{"en":"The full twelve-grounder panel evaluates the same easy difficulty subset of STVG-CompBench. The native 0–100 mean video IoU measures spatiotemporal overlap, separately from temporal IoU and thresholded localization success. Vanilla VLMs, specialized grounding architectures and CurrSTVG retain their source inference modes.","zh":"完整十二定位模型评测STVG-CompBench相同easy难度子集。0–100原生平均视频IoU衡量时空重合度，与时间IoU及阈值定位成功率分开。通用视觉语言模型、专门定位架构及CurrSTVG保留原始推理模式。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.30584v1","status":"paper_baselines","published_at":"2026-08-31"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":12,"result_count":12,"results":[[2754,0.58,370,13783,13784],[6342,12.13,370,13783,13784],[4070,17.32,370,13783,13784],[2728,8.83,370,13783,13784],[2654,9.13,370,13783,13784],[13785,1.05,370,13783,13784],[13786,18.01,370,13783,13784],[13787,9.16,370,13783,13784],[13788,22.91,370,13783,13784],[13789,19.45,370,13783,13784],[7390,1.87,370,13783,13784],[13790,20.48,370,13783,13784]]},{"id":"stvg-compbench-medium-2026-paper","name":"STVG-CompBench · medium compositional grounding (2026 paper)","group":"视觉与视频","metric":"Mean spatiotemporal video IoU","unit":"score","public_history":true,"protocol":{"en":"The full twelve-grounder panel evaluates the same medium difficulty subset of STVG-CompBench. The native 0–100 mean video IoU measures spatiotemporal overlap, separately from temporal IoU and thresholded localization success. Vanilla VLMs, specialized grounding architectures and CurrSTVG retain their source inference modes.","zh":"完整十二定位模型评测STVG-CompBench相同medium难度子集。0–100原生平均视频IoU衡量时空重合度，与时间IoU及阈值定位成功率分开。通用视觉语言模型、专门定位架构及CurrSTVG保留原始推理模式。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.30584v1","status":"paper_baselines","published_at":"2026-08-31"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":12,"result_count":12,"results":[[2754,0.42,370,13791,13784],[6342,6.96,370,13791,13784],[4070,11.04,370,13791,13784],[2728,6.07,370,13791,13784],[2654,5.27,370,13791,13784],[13785,0.99,370,13791,13784],[13786,9.04,370,13791,13784],[13787,7.05,370,13791,13784],[13788,11.8,370,13791,13784],[13789,10.14,370,13791,13784],[7390,1.52,370,13791,13784],[13790,14.85,370,13791,13784]]},{"id":"stvg-compbench-hard-2026-paper","name":"STVG-CompBench · hard compositional grounding (2026 paper)","group":"视觉与视频","metric":"Mean spatiotemporal video IoU","unit":"score","public_history":true,"protocol":{"en":"The full twelve-grounder panel evaluates the same hard difficulty subset of STVG-CompBench. The native 0–100 mean video IoU measures spatiotemporal overlap, separately from temporal IoU and thresholded localization success. Vanilla VLMs, specialized grounding architectures and CurrSTVG retain their source inference modes.","zh":"完整十二定位模型评测STVG-CompBench相同hard难度子集。0–100原生平均视频IoU衡量时空重合度，与时间IoU及阈值定位成功率分开。通用视觉语言模型、专门定位架构及CurrSTVG保留原始推理模式。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.30584v1","status":"paper_baselines","published_at":"2026-08-31"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":12,"result_count":12,"results":[[2754,0.41,370,13792,13784],[6342,6.23,370,13792,13784],[4070,8.16,370,13792,13784],[2728,5.34,370,13792,13784],[2654,5.13,370,13792,13784],[13785,1.34,370,13792,13784],[13786,7.0,370,13792,13784],[13787,4.73,370,13792,13784],[13788,8.68,370,13792,13784],[13789,7.9,370,13792,13784],[7390,1.49,370,13792,13784],[13790,13.16,370,13792,13784]]},{"id":"imitator-simulation-seen-2026-paper","name":"The Imitator Game · simulation seen (2026 paper)","group":"Agent 与工具","metric":"Simulation task success rate","unit":"%","public_history":true,"protocol":{"en":"All fifteen trained policy variants evaluate the same simulation seen regime. Source success rates are averaged across the 15/30/45-task pretraining scales where specified. The table’s fifteen variants represent nine policy architectures; visual-encoder swaps remain configurations. Seen, zero-shot, scratch and pretrained-plus-finetuned regimes are separate, and subgoal success is not mixed with complete task success.","zh":"完整十五训练策略变体评测相同模拟seen协议。按原文规定，成功率在15、30、45任务预训练规模间平均。十五变体对应九种策略架构，视觉编码器替换属于配置。已见、零样本、从头训练及预训练后微调分别展示，子目标成功率与完整任务成功率分开。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.22301v1","status":"paper_baselines","published_at":"2026-08-23"},"model_count":15,"result_count":15,"results":[[7592,0.29,370,13793,13794],[13795,0.43,370,13793,13794],[9599,0.51,370,13793,13794],[9732,0.73,370,13793,13794],[13796,0.75,370,13793,13794],[13797,0.79,370,13793,13794],[13798,0.67,370,13793,13794],[13799,0.61,370,13793,13794],[13800,0.14,370,13793,13794],[13801,0.52,370,13793,13794],[13802,0.57,370,13793,13794],[13803,0.3,370,13793,13794],[13804,0.81,370,13793,13794],[13805,0.79,370,13793,13794],[13806,0.72,370,13793,13794]]},{"id":"imitator-simulation-zero-shot-2026-paper","name":"The Imitator Game · simulation zero-shot (2026 paper)","group":"Agent 与工具","metric":"Simulation task success rate","unit":"%","public_history":true,"protocol":{"en":"All fifteen trained policy variants evaluate the same simulation zero-shot regime. Source success rates are averaged across the 15/30/45-task pretraining scales where specified. The table’s fifteen variants represent nine policy architectures; visual-encoder swaps remain configurations. Seen, zero-shot, scratch and pretrained-plus-finetuned regimes are separate, and subgoal success is not mixed with complete task success.","zh":"完整十五训练策略变体评测相同模拟zero-shot协议。按原文规定，成功率在15、30、45任务预训练规模间平均。十五变体对应九种策略架构，视觉编码器替换属于配置。已见、零样本、从头训练及预训练后微调分别展示，子目标成功率与完整任务成功率分开。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.22301v1","status":"paper_baselines","published_at":"2026-08-23"},"model_count":15,"result_count":15,"results":[[7592,0.06,370,13807,13794],[13795,0.07,370,13807,13794],[9599,0.03,370,13807,13794],[9732,0.09,370,13807,13794],[13796,0.07,370,13807,13794],[13797,0.1,370,13807,13794],[13798,0.07,370,13807,13794],[13799,0.05,370,13807,13794],[13800,0.02,370,13807,13794],[13801,0.13,370,13807,13794],[13802,0.08,370,13807,13794],[13803,0.07,370,13807,13794],[13804,0.02,370,13807,13794],[13805,0.03,370,13807,13794],[13806,0.04,370,13807,13794]]},{"id":"imitator-simulation-scratch-2026-paper","name":"The Imitator Game · simulation scratch (2026 paper)","group":"Agent 与工具","metric":"Simulation task success rate","unit":"%","public_history":true,"protocol":{"en":"All fifteen trained policy variants evaluate the same simulation scratch regime. Source success rates are averaged across the 15/30/45-task pretraining scales where specified. The table’s fifteen variants represent nine policy architectures; visual-encoder swaps remain configurations. Seen, zero-shot, scratch and pretrained-plus-finetuned regimes are separate, and subgoal success is not mixed with complete task success.","zh":"完整十五训练策略变体评测相同模拟scratch协议。按原文规定，成功率在15、30、45任务预训练规模间平均。十五变体对应九种策略架构，视觉编码器替换属于配置。已见、零样本、从头训练及预训练后微调分别展示，子目标成功率与完整任务成功率分开。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.22301v1","status":"paper_baselines","published_at":"2026-08-23"},"model_count":15,"result_count":15,"results":[[7592,0.14,370,13808,13794],[13795,0.35,370,13808,13794],[9599,0.21,370,13808,13794],[9732,0.8,370,13808,13794],[13796,0.27,370,13808,13794],[13797,0.35,370,13808,13794],[13798,0.13,370,13808,13794],[13799,0.11,370,13808,13794],[13800,0.09,370,13808,13794],[13801,0.35,370,13808,13794],[13802,0.26,370,13808,13794],[13803,0.14,370,13808,13794],[13804,0.76,370,13808,13794],[13805,0.65,370,13808,13794],[13806,0.63,370,13808,13794]]},{"id":"imitator-simulation-pretrain-finetune-2026-paper","name":"The Imitator Game · simulation pretrain-finetune (2026 paper)","group":"Agent 与工具","metric":"Simulation task success rate","unit":"%","public_history":true,"protocol":{"en":"All fifteen trained policy variants evaluate the same simulation pretrain-finetune regime. Source success rates are averaged across the 15/30/45-task pretraining scales where specified. The table’s fifteen variants represent nine policy architectures; visual-encoder swaps remain configurations. Seen, zero-shot, scratch and pretrained-plus-finetuned regimes are separate, and subgoal success is not mixed with complete task success.","zh":"完整十五训练策略变体评测相同模拟pretrain-finetune协议。按原文规定，成功率在15、30、45任务预训练规模间平均。十五变体对应九种策略架构，视觉编码器替换属于配置。已见、零样本、从头训练及预训练后微调分别展示，子目标成功率与完整任务成功率分开。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.22301v1","status":"paper_baselines","published_at":"2026-08-23"},"model_count":15,"result_count":15,"results":[[7592,0.2,370,13809,13794],[13795,0.21,370,13809,13794],[9599,0.67,370,13809,13794],[9732,0.85,370,13809,13794],[13796,0.59,370,13809,13794],[13797,0.73,370,13809,13794],[13798,0.54,370,13809,13794],[13799,0.48,370,13809,13794],[13800,0.36,370,13809,13794],[13801,0.23,370,13809,13794],[13802,0.17,370,13809,13794],[13803,0.15,370,13809,13794],[13804,0.84,370,13809,13794],[13805,0.8,370,13809,13794],[13806,0.82,370,13809,13794]]},{"id":"tide-parametric-strict-2026-paper","name":"TIDE · parametric temporal reasoning (2026 paper)","group":"知识与推理","metric":"Macro strict accuracy","unit":"%","public_history":true,"protocol":{"en":"All nine checkpoints evaluate the same matched temporal-knowledge items across eight task types under the parametric setting. The first value of the published Macro avg is strict accuracy; the following + value is incremental partial credit and is kept separate. RAG retrieval context, ICL context and parametric-only results form distinct cohorts.","zh":"完整九检查点按parametric设置评测八类任务的同一组配对时序知识题。采用原表Macro avg的首项严格准确率，后方+数值是额外部分给分，单独保留。检索增强、上下文学习及纯参数知识分成不同面板。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.08512v1","status":"paper_baselines","published_at":"2026-08-09"},"model_count":9,"result_count":9,"results":[[340,0.3539,370,13810,13811],[6704,0.2591,370,13810,13811],[2129,0.34149999999999997,370,13810,13811],[2998,0.3328,370,13810,13811],[3201,0.3264,370,13810,13811],[3419,0.30760000000000004,370,13810,13811],[6771,0.2639,370,13810,13811],[5480,0.3204,370,13810,13811],[3016,0.2554,370,13810,13811]]},{"id":"tide-rag-strict-2026-paper","name":"TIDE · rag temporal reasoning (2026 paper)","group":"知识与推理","metric":"Macro strict accuracy","unit":"%","public_history":true,"protocol":{"en":"All nine checkpoints evaluate the same matched temporal-knowledge items across eight task types under the rag setting. The first value of the published Macro avg is strict accuracy; the following + value is incremental partial credit and is kept separate. RAG retrieval context, ICL context and parametric-only results form distinct cohorts.","zh":"完整九检查点按rag设置评测八类任务的同一组配对时序知识题。采用原表Macro avg的首项严格准确率，后方+数值是额外部分给分，单独保留。检索增强、上下文学习及纯参数知识分成不同面板。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.08512v1","status":"paper_baselines","published_at":"2026-08-09"},"model_count":9,"result_count":9,"results":[[340,0.5706,370,13812,13811],[6704,0.5805,370,13812,13811],[2129,0.5287,370,13812,13811],[2998,0.5446,370,13812,13811],[3201,0.5207999999999999,370,13812,13811],[3419,0.5268999999999999,370,13812,13811],[6771,0.5213,370,13812,13811],[5480,0.4769,370,13812,13811],[3016,0.4543,370,13812,13811]]},{"id":"tide-icl-strict-2026-paper","name":"TIDE · icl temporal reasoning (2026 paper)","group":"知识与推理","metric":"Macro strict accuracy","unit":"%","public_history":true,"protocol":{"en":"All nine checkpoints evaluate the same matched temporal-knowledge items across eight task types under the icl setting. The first value of the published Macro avg is strict accuracy; the following + value is incremental partial credit and is kept separate. RAG retrieval context, ICL context and parametric-only results form distinct cohorts.","zh":"完整九检查点按icl设置评测八类任务的同一组配对时序知识题。采用原表Macro avg的首项严格准确率，后方+数值是额外部分给分，单独保留。检索增强、上下文学习及纯参数知识分成不同面板。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.08512v1","status":"paper_baselines","published_at":"2026-08-09"},"model_count":9,"result_count":9,"results":[[340,0.6849,370,13813,13811],[6704,0.6439,370,13813,13811],[2129,0.6245,370,13813,13811],[2998,0.6327,370,13813,13811],[3201,0.6311,370,13813,13811],[3419,0.5710999999999999,370,13813,13811],[6771,0.6113000000000001,370,13813,13811],[5480,0.6004999999999999,370,13813,13811],[3016,0.5038,370,13813,13811]]},{"id":"trapsbench-standard-pecs-2026-paper","name":"TRAPSBench · standard evidence sensitivity (2026 paper)","group":"安全与可靠性","metric":"Penalized Epistemic Calibration Score (PECS)","unit":"score","public_history":true,"protocol":{"en":"The complete sixteen-configuration panel evaluates the same four matched control/void physics datasets under standard prompts. Native PECS combines evidence-supported accuracy and calibrated abstention and is reported on its original 0–1 scale. Three-run means are used except the explicitly single-greedy-pass Qwen3-VL-8B result. Reasoning budgets are configurations rather than new bases.","zh":"完整十六配置按standard提示评测同一四个物理证据充足、证据缺失配对数据集。PECS在0–1原生单位下综合有证据准确性与校准弃答。除明确标注的Qwen3-VL-8B单次贪心运行外，其余为三次均值。推理预算作为配置，不增加底模计数。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.13167v1","status":"paper_baselines","published_at":"2026-08-13"},"score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":12,"result_count":16,"results":[[2602,0.224,370,13814,13815],[2055,0.292,370,13816,13815],[13817,0.199,370,13818,13815],[3412,0.199,370,13819,13815],[13820,0.197,370,13821,13815],[3497,0.141,370,13822,13815],[13823,0.18,370,13824,13815],[13325,0.246,370,13825,13815],[7513,0.142,370,13826,13815],[231,0.126,370,13827,13815],[4813,0.064,370,13828,13815],[13829,0.057,370,13830,13815],[5505,0.018,370,13831,13815],[13832,0.071,370,13833,13815],[2627,0.16,370,13834,13815],[7803,0.079,370,13835,13815]]},{"id":"trapsbench-json-pecs-2026-paper","name":"TRAPSBench · json evidence sensitivity (2026 paper)","group":"安全与可靠性","metric":"Penalized Epistemic Calibration Score (PECS)","unit":"score","public_history":true,"protocol":{"en":"The complete sixteen-configuration panel evaluates the same four matched control/void physics datasets under json prompts. Native PECS combines evidence-supported accuracy and calibrated abstention and is reported on its original 0–1 scale. Three-run means are used except the explicitly single-greedy-pass Qwen3-VL-8B result. Reasoning budgets are configurations rather than new bases.","zh":"完整十六配置按json提示评测同一四个物理证据充足、证据缺失配对数据集。PECS在0–1原生单位下综合有证据准确性与校准弃答。除明确标注的Qwen3-VL-8B单次贪心运行外，其余为三次均值。推理预算作为配置，不增加底模计数。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.13167v1","status":"paper_baselines","published_at":"2026-08-13"},"score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":12,"result_count":16,"results":[[2602,0.24,370,13836,13815],[2055,0.328,370,13837,13815],[13817,0.233,370,13838,13815],[3412,0.178,370,13839,13815],[13820,0.196,370,13840,13815],[3497,0.136,370,13841,13815],[13823,0.184,370,13842,13815],[13325,0.235,370,13843,13815],[7513,0.154,370,13844,13815],[231,0.105,370,13845,13815],[4813,0.068,370,13846,13815],[13829,0.061,370,13847,13815],[5505,0.019,370,13848,13815],[13832,0.072,370,13849,13815],[2627,0.178,370,13850,13815],[7803,0.025,370,13851,13815]]},{"id":"trapsbench-guided-pecs-2026-paper","name":"TRAPSBench · guided evidence sensitivity (2026 paper)","group":"安全与可靠性","metric":"Penalized Epistemic Calibration Score (PECS)","unit":"score","public_history":true,"protocol":{"en":"The complete sixteen-configuration panel evaluates the same four matched control/void physics datasets under guided prompts. Native PECS combines evidence-supported accuracy and calibrated abstention and is reported on its original 0–1 scale. Three-run means are used except the explicitly single-greedy-pass Qwen3-VL-8B result. Reasoning budgets are configurations rather than new bases.","zh":"完整十六配置按guided提示评测同一四个物理证据充足、证据缺失配对数据集。PECS在0–1原生单位下综合有证据准确性与校准弃答。除明确标注的Qwen3-VL-8B单次贪心运行外，其余为三次均值。推理预算作为配置，不增加底模计数。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.13167v1","status":"paper_baselines","published_at":"2026-08-13"},"score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":12,"result_count":16,"results":[[2602,0.448,370,13852,13815],[2055,0.467,370,13853,13815],[13817,0.376,370,13854,13815],[3412,0.564,370,13855,13815],[13820,0.568,370,13856,13815],[3497,0.296,370,13857,13815],[13823,0.446,370,13858,13815],[13325,0.441,370,13859,13815],[7513,0.155,370,13860,13815],[231,0.447,370,13861,13815],[4813,0.4,370,13862,13815],[13829,0.388,370,13863,13815],[5505,0.224,370,13864,13815],[13832,0.402,370,13865,13815],[2627,0.215,370,13866,13815],[7803,0.101,370,13867,13815]]},{"id":"siren-five-second-rmse-2026-paper","name":"SIREN-Bench · emergency-vehicle trajectory prediction (2026 paper)","group":"预测与决策","metric":"Five-second position RMSE (m)","unit":"raw","public_history":true,"protocol":{"en":"The complete seven learned-predictor panel uses the same 7,062 qualifying vehicle/frame cases, scored on a common five-second, 0.2-second target grid. The source mean equally averages seven interaction-template RMSE values in meters. CSP/STDAN/BAT use three seconds of observed history; EMP and DeMo use five seconds, a source-acknowledged cross-family information confound. EMP-M and EMP-D have independently trained MLP and DETR-style decoder architectures; inference settings alone are not counted as models. Analytic CV/IDM references are excluded.","zh":"完整七个学习式预测器使用相同7,062个车辆、时刻样本，统一在五秒未来、0.2秒间隔目标网格评分。原表均值等权汇总七种交互模板的米制RMSE。CSP、STDAN、BAT接收三秒历史，EMP和DeMo接收五秒历史，原文明确这是跨模型族的信息条件差异。EMP-M和EMP-D分别训练MLP与DETR式解码架构，纯推理设置不计新模型。解析式CV、IDM参考不进入学习模型排行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.24094v1","status":"paper_baselines","published_at":"2026-08-25"},"higher_is_better":false,"score_decimals":2,"saturation_excluded":true,"scale_label":{"en":"RMSE (m)","zh":"均方根误差（米）"},"model_count":7,"result_count":7,"results":[[13868,8.47,370,13869,13870],[13871,13.1,370,13869,13870],[13872,14.75,370,13869,13870],[13873,9.27,370,13874,13870],[13875,7.77,370,13874,13870],[13876,7.84,370,13874,13870],[13877,8.34,370,13874,13870]]},{"id":"safeatlas-image-2026-paper","name":"SafeAtlas-Bench · image safety judgments (2026 paper)","group":"安全与可靠性","metric":"Unsafe-class F1","unit":"score","public_history":true,"protocol":{"en":"The complete thirteen-multimodal-guard cohort evaluates the same SafeAtlas-Bench image target. Unsafe/harmful is the positive class; boundary-uncertain, unsafe-leaning disputed and unsafe-core labels map to unsafe. The native 0–100 F1 remains separate from external benchmark averages and the continuous risk score. Thresholds follow the paper’s validation-only selection and frozen test operating points where applicable.","zh":"完整十三个多模态防护模型评测SafeAtlas-Bench相同image目标。以不安全、有害为正类，边界不确定、偏向不安全但有争议、核心不安全三类映射为不安全。保留0–100原生F1，与外部基准均分和连续风险分开。适用时按论文规则仅在验证集选阈值，再固定用于测试。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.29098v1","status":"paper_baselines","published_at":"2026-08-29"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":13,"result_count":13,"results":[[13878,65.7,370,13879,13880],[13881,63.9,370,13879,13880],[13882,60.2,370,13879,13880],[13883,61.3,370,13879,13880],[13884,0.0,370,13879,13880],[13885,25.9,370,13879,13880],[13886,50.1,370,13879,13880],[13887,70.3,370,13879,13880],[13888,27.2,370,13879,13880],[13889,54.1,370,13879,13880],[13890,50.0,370,13879,13880],[13891,77.0,370,13879,13880],[13892,73.5,370,13879,13880]]},{"id":"safeatlas-request-2026-paper","name":"SafeAtlas-Bench · request safety judgments (2026 paper)","group":"安全与可靠性","metric":"Unsafe-class F1","unit":"score","public_history":true,"protocol":{"en":"The complete thirteen-multimodal-guard cohort evaluates the same SafeAtlas-Bench request target. Unsafe/harmful is the positive class; boundary-uncertain, unsafe-leaning disputed and unsafe-core labels map to unsafe. The native 0–100 F1 remains separate from external benchmark averages and the continuous risk score. Thresholds follow the paper’s validation-only selection and frozen test operating points where applicable.","zh":"完整十三个多模态防护模型评测SafeAtlas-Bench相同request目标。以不安全、有害为正类，边界不确定、偏向不安全但有争议、核心不安全三类映射为不安全。保留0–100原生F1，与外部基准均分和连续风险分开。适用时按论文规则仅在验证集选阈值，再固定用于测试。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.29098v1","status":"paper_baselines","published_at":"2026-08-29"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":13,"result_count":13,"results":[[13878,85.9,370,13893,13880],[13881,86.1,370,13893,13880],[13882,79.7,370,13893,13880],[13883,73.8,370,13893,13880],[13884,50.6,370,13893,13880],[13885,61.1,370,13893,13880],[13886,45.1,370,13893,13880],[13887,75.5,370,13893,13880],[13888,84.5,370,13893,13880],[13889,50.5,370,13893,13880],[13890,95.5,370,13893,13880],[13891,95.0,370,13893,13880],[13892,95.4,370,13893,13880]]},{"id":"safeatlas-response-2026-paper","name":"SafeAtlas-Bench · response safety judgments (2026 paper)","group":"安全与可靠性","metric":"Unsafe-class F1","unit":"score","public_history":true,"protocol":{"en":"The complete thirteen-multimodal-guard cohort evaluates the same SafeAtlas-Bench response target. Unsafe/harmful is the positive class; boundary-uncertain, unsafe-leaning disputed and unsafe-core labels map to unsafe. The native 0–100 F1 remains separate from external benchmark averages and the continuous risk score. Thresholds follow the paper’s validation-only selection and frozen test operating points where applicable.","zh":"完整十三个多模态防护模型评测SafeAtlas-Bench相同response目标。以不安全、有害为正类，边界不确定、偏向不安全但有争议、核心不安全三类映射为不安全。保留0–100原生F1，与外部基准均分和连续风险分开。适用时按论文规则仅在验证集选阈值，再固定用于测试。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.29098v1","status":"paper_baselines","published_at":"2026-08-29"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":13,"result_count":13,"results":[[13878,80.3,370,13894,13880],[13881,73.5,370,13894,13880],[13882,79.7,370,13894,13880],[13883,79.8,370,13894,13880],[13884,55.5,370,13894,13880],[13885,74.3,370,13894,13880],[13886,43.7,370,13894,13880],[13887,83.4,370,13894,13880],[13888,85.8,370,13894,13880],[13889,63.9,370,13894,13880],[13890,92.7,370,13894,13880],[13891,92.7,370,13894,13880],[13892,93.2,370,13894,13880]]},{"id":"videorover-short-easy-2026-paper","name":"VideoRover-Bench · short-easy (2026 paper)","group":"视觉与视频","metric":"Answer accuracy","unit":"%","public_history":true,"protocol":{"en":"All twenty-one published configurations answer the same short-easy VideoRover-Bench subset. Direct-answer models and tool-using agents retain explicit inference modes, including the proposed SFT and RL checkpoints. The selected column is from the new benchmark; the source’s Avg additionally includes VideoDR and is not used.","zh":"完整二十一配置回答VideoRover-Bench相同short-easy子集。直接回答模型与工具智能体均明确推理模式，包括提案的监督微调及强化学习检查点。采用新基准对应列；原表Avg还混合VideoDR，因此保持单独处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.23329v2","status":"paper_baselines","published_at":"2026-08-24"},"model_count":15,"result_count":21,"results":[[2058,0.7,370,13895,13896],[340,0.56,370,13895,13896],[3184,0.56,370,13895,13896],[766,0.54,370,13895,13896],[764,0.66,370,13895,13896],[6133,0.74,370,13895,13896],[6130,0.78,370,13895,13896],[6342,0.08,370,13895,13896],[6238,0.14,370,13895,13896],[2584,0.32,370,13895,13896],[2590,0.36,370,13895,13896],[3763,0.3,370,13895,13896],[3764,0.36,370,13895,13896],[6342,0.58,370,13897,13896],[6238,0.58,370,13897,13896],[2584,0.78,370,13897,13896],[2590,0.56,370,13897,13896],[3763,0.74,370,13897,13896],[3764,0.76,370,13897,13896],[13898,0.58,370,13899,13896],[13900,0.8,370,13901,13896]]},{"id":"videorover-short-medium-2026-paper","name":"VideoRover-Bench · short-medium (2026 paper)","group":"视觉与视频","metric":"Answer accuracy","unit":"%","public_history":true,"protocol":{"en":"All twenty-one published configurations answer the same short-medium VideoRover-Bench subset. Direct-answer models and tool-using agents retain explicit inference modes, including the proposed SFT and RL checkpoints. The selected column is from the new benchmark; the source’s Avg additionally includes VideoDR and is not used.","zh":"完整二十一配置回答VideoRover-Bench相同short-medium子集。直接回答模型与工具智能体均明确推理模式，包括提案的监督微调及强化学习检查点。采用新基准对应列；原表Avg还混合VideoDR，因此保持单独处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.23329v2","status":"paper_baselines","published_at":"2026-08-24"},"model_count":15,"result_count":21,"results":[[2058,0.64,370,13902,13896],[340,0.52,370,13902,13896],[3184,0.54,370,13902,13896],[766,0.3,370,13902,13896],[764,0.54,370,13902,13896],[6133,0.7,370,13902,13896],[6130,0.6,370,13902,13896],[6342,0.08,370,13902,13896],[6238,0.16,370,13902,13896],[2584,0.3,370,13902,13896],[2590,0.28,370,13902,13896],[3763,0.3,370,13902,13896],[3764,0.18,370,13902,13896],[6342,0.38,370,13903,13896],[6238,0.34,370,13903,13896],[2584,0.54,370,13903,13896],[2590,0.42,370,13903,13896],[3763,0.64,370,13903,13896],[3764,0.62,370,13903,13896],[13898,0.44,370,13904,13896],[13900,0.64,370,13905,13896]]},{"id":"videorover-short-hard-2026-paper","name":"VideoRover-Bench · short-hard (2026 paper)","group":"视觉与视频","metric":"Answer accuracy","unit":"%","public_history":true,"protocol":{"en":"All twenty-one published configurations answer the same short-hard VideoRover-Bench subset. Direct-answer models and tool-using agents retain explicit inference modes, including the proposed SFT and RL checkpoints. The selected column is from the new benchmark; the source’s Avg additionally includes VideoDR and is not used.","zh":"完整二十一配置回答VideoRover-Bench相同short-hard子集。直接回答模型与工具智能体均明确推理模式，包括提案的监督微调及强化学习检查点。采用新基准对应列；原表Avg还混合VideoDR，因此保持单独处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.23329v2","status":"paper_baselines","published_at":"2026-08-24"},"model_count":15,"result_count":21,"results":[[2058,0.36,370,13906,13896],[340,0.24,370,13906,13896],[3184,0.26,370,13906,13896],[766,0.3,370,13906,13896],[764,0.3,370,13906,13896],[6133,0.48,370,13906,13896],[6130,0.42,370,13906,13896],[6342,0.08,370,13906,13896],[6238,0.12,370,13906,13896],[2584,0.14,370,13906,13896],[2590,0.14,370,13906,13896],[3763,0.16,370,13906,13896],[3764,0.14,370,13906,13896],[6342,0.16,370,13907,13896],[6238,0.14,370,13907,13896],[2584,0.36,370,13907,13896],[2590,0.18,370,13907,13896],[3763,0.38,370,13907,13896],[3764,0.34,370,13907,13896],[13898,0.3,370,13908,13896],[13900,0.4,370,13909,13896]]},{"id":"videorover-long-easy-2026-paper","name":"VideoRover-Bench · long-easy (2026 paper)","group":"视觉与视频","metric":"Answer accuracy","unit":"%","public_history":true,"protocol":{"en":"All twenty-one published configurations answer the same long-easy VideoRover-Bench subset. Direct-answer models and tool-using agents retain explicit inference modes, including the proposed SFT and RL checkpoints. The selected column is from the new benchmark; the source’s Avg additionally includes VideoDR and is not used.","zh":"完整二十一配置回答VideoRover-Bench相同long-easy子集。直接回答模型与工具智能体均明确推理模式，包括提案的监督微调及强化学习检查点。采用新基准对应列；原表Avg还混合VideoDR，因此保持单独处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.23329v2","status":"paper_baselines","published_at":"2026-08-24"},"model_count":15,"result_count":21,"results":[[2058,0.64,370,13910,13896],[340,0.6,370,13910,13896],[3184,0.24,370,13910,13896],[766,0.54,370,13910,13896],[764,0.62,370,13910,13896],[6133,0.58,370,13910,13896],[6130,0.72,370,13910,13896],[6342,0.18,370,13910,13896],[6238,0.22,370,13910,13896],[2584,0.28,370,13910,13896],[2590,0.36,370,13910,13896],[3763,0.38,370,13910,13896],[3764,0.26,370,13910,13896],[6342,0.42,370,13911,13896],[6238,0.5,370,13911,13896],[2584,0.68,370,13911,13896],[2590,0.38,370,13911,13896],[3763,0.66,370,13911,13896],[3764,0.62,370,13911,13896],[13898,0.54,370,13912,13896],[13900,0.7,370,13913,13896]]},{"id":"videorover-long-medium-2026-paper","name":"VideoRover-Bench · long-medium (2026 paper)","group":"视觉与视频","metric":"Answer accuracy","unit":"%","public_history":true,"protocol":{"en":"All twenty-one published configurations answer the same long-medium VideoRover-Bench subset. Direct-answer models and tool-using agents retain explicit inference modes, including the proposed SFT and RL checkpoints. The selected column is from the new benchmark; the source’s Avg additionally includes VideoDR and is not used.","zh":"完整二十一配置回答VideoRover-Bench相同long-medium子集。直接回答模型与工具智能体均明确推理模式，包括提案的监督微调及强化学习检查点。采用新基准对应列；原表Avg还混合VideoDR，因此保持单独处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.23329v2","status":"paper_baselines","published_at":"2026-08-24"},"model_count":15,"result_count":21,"results":[[2058,0.5,370,13914,13896],[340,0.3,370,13914,13896],[3184,0.24,370,13914,13896],[766,0.3,370,13914,13896],[764,0.34,370,13914,13896],[6133,0.52,370,13914,13896],[6130,0.4,370,13914,13896],[6342,0.06,370,13914,13896],[6238,0.1,370,13914,13896],[2584,0.22,370,13914,13896],[2590,0.2,370,13914,13896],[3763,0.18,370,13914,13896],[3764,0.22,370,13914,13896],[6342,0.3,370,13915,13896],[6238,0.22,370,13915,13896],[2584,0.56,370,13915,13896],[2590,0.22,370,13915,13896],[3763,0.46,370,13915,13896],[3764,0.48,370,13915,13896],[13898,0.32,370,13916,13896],[13900,0.52,370,13917,13896]]},{"id":"videorover-long-hard-2026-paper","name":"VideoRover-Bench · long-hard (2026 paper)","group":"视觉与视频","metric":"Answer accuracy","unit":"%","public_history":true,"protocol":{"en":"All twenty-one published configurations answer the same long-hard VideoRover-Bench subset. Direct-answer models and tool-using agents retain explicit inference modes, including the proposed SFT and RL checkpoints. The selected column is from the new benchmark; the source’s Avg additionally includes VideoDR and is not used.","zh":"完整二十一配置回答VideoRover-Bench相同long-hard子集。直接回答模型与工具智能体均明确推理模式，包括提案的监督微调及强化学习检查点。采用新基准对应列；原表Avg还混合VideoDR，因此保持单独处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.23329v2","status":"paper_baselines","published_at":"2026-08-24"},"model_count":15,"result_count":21,"results":[[2058,0.44,370,13918,13896],[340,0.24,370,13918,13896],[3184,0.16,370,13918,13896],[766,0.28,370,13918,13896],[764,0.5,370,13918,13896],[6133,0.48,370,13918,13896],[6130,0.52,370,13918,13896],[6342,0.1,370,13918,13896],[6238,0.12,370,13918,13896],[2584,0.12,370,13918,13896],[2590,0.14,370,13918,13896],[3763,0.12,370,13918,13896],[3764,0.16,370,13918,13896],[6342,0.1,370,13919,13896],[6238,0.16,370,13919,13896],[2584,0.42,370,13919,13896],[2590,0.2,370,13919,13896],[3763,0.34,370,13919,13896],[3764,0.4,370,13919,13896],[13898,0.18,370,13920,13896],[13900,0.42,370,13921,13896]]},{"id":"vistarget-visual-search-2026-paper","name":"VisTarget-Bench · targeted multimodal search (2026 paper)","group":"Agent 与工具","metric":"Task accuracy","unit":"%","public_history":true,"protocol":{"en":"The complete thirteen-system measured panel evaluates the same VisTarget-Bench tasks. Original direct-answer, WeAgent-Harness, lightweight search and proposed model configurations are explicit. Systems without a measured score on this new benchmark are omitted instead of borrowing their external-benchmark scores. Model parameters and source standard deviations remain in the fixture and result configuration.","zh":"完整十三系统实测面板评测相同VisTarget-Bench任务。直接回答、WeAgent框架、轻量搜索及提案模型的原始配置均明确保留。未报告新基准分数的系统保持缺失，不借用其外部基准分数。参数与原始标准差保留在来源表及配置中。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.28062v2","status":"paper_baselines","published_at":"2026-08-28"},"model_count":12,"result_count":13,"results":[[9707,0.0,370,13922,13923],[3490,0.0199,370,13924,13923],[6734,0.42,370,13925,13923],[13926,0.49329999999999996,370,13927,13923],[13928,0.4067,370,13929,13923],[3490,0.3378,370,13930,13923],[6412,0.3044,370,13931,13923],[3764,0.36219999999999997,370,13932,13923],[5300,0.3145,370,13933,13923],[13934,0.1689,370,13935,13923],[13936,0.08,370,13937,13923],[13938,0.24,370,13939,13923],[13940,0.30219999999999997,370,13941,13923]]},{"id":"xrepotest-rust-pass-2026-paper","name":"XRepoTest · rust unit tests (2026 paper)","group":"编程","metric":"Test pass rate","unit":"%","public_history":true,"protocol":{"en":"All fourteen model checkpoints generate tests for the same XRepoTest rust focal-function cohort under the standard single-turn setting. The original TPR percentage is retained, separately from line coverage and import resolution. Context retrieval, execution-feedback repair and headless Claude Code are separate configurations measured in other panels.","zh":"完整十四检查点按标准单轮协议，为XRepoTest相同rust目标函数集合生成测试。保留原始TPR百分比，与行覆盖率和导入解析率分开。上下文检索、执行反馈修复及无界面Claude Code在其他面板单列。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.25939v2","status":"paper_baselines","published_at":"2026-08-26"},"model_count":14,"result_count":14,"results":[[13942,0.0032,370,13943,13944],[13945,0.015,370,13943,13944],[13946,0.0269,370,13943,13944],[13947,0.07200000000000001,370,13943,13944],[13948,0.057999999999999996,370,13943,13944],[13949,0.0333,370,13943,13944],[4425,0.0279,370,13943,13944],[3190,0.043,370,13943,13944],[3623,0.0827,370,13943,13944],[2271,0.1278,370,13943,13944],[13950,0.0666,370,13943,13944],[268,0.0644,370,13943,13944],[13951,0.013999999999999999,370,13943,13944],[340,0.12240000000000001,370,13943,13944]]},{"id":"xrepotest-julia-pass-2026-paper","name":"XRepoTest · julia unit tests (2026 paper)","group":"编程","metric":"Test pass rate","unit":"%","public_history":true,"protocol":{"en":"All fourteen model checkpoints generate tests for the same XRepoTest julia focal-function cohort under the standard single-turn setting. The original TPR percentage is retained, separately from line coverage and import resolution. Context retrieval, execution-feedback repair and headless Claude Code are separate configurations measured in other panels.","zh":"完整十四检查点按标准单轮协议，为XRepoTest相同julia目标函数集合生成测试。保留原始TPR百分比，与行覆盖率和导入解析率分开。上下文检索、执行反馈修复及无界面Claude Code在其他面板单列。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.25939v2","status":"paper_baselines","published_at":"2026-08-26"},"model_count":14,"result_count":14,"results":[[13942,0.0088,370,13952,13944],[13945,0.0482,370,13952,13944],[13946,0.020499999999999997,370,13952,13944],[13947,0.0512,370,13952,13944],[13948,0.043899999999999995,370,13952,13944],[13949,0.030699999999999998,370,13952,13944],[4425,0.0395,370,13952,13944],[3190,0.043899999999999995,370,13952,13944],[3623,0.1257,370,13952,13944],[2271,0.19879999999999998,370,13952,13944],[13950,0.114,370,13952,13944],[268,0.0541,370,13952,13944],[13951,0.0263,370,13952,13944],[340,0.16519999999999999,370,13952,13944]]},{"id":"xrepotest-go-pass-2026-paper","name":"XRepoTest · go unit tests (2026 paper)","group":"编程","metric":"Test pass rate","unit":"%","public_history":true,"protocol":{"en":"All fourteen model checkpoints generate tests for the same XRepoTest go focal-function cohort under the standard single-turn setting. The original TPR percentage is retained, separately from line coverage and import resolution. Context retrieval, execution-feedback repair and headless Claude Code are separate configurations measured in other panels.","zh":"完整十四检查点按标准单轮协议，为XRepoTest相同go目标函数集合生成测试。保留原始TPR百分比，与行覆盖率和导入解析率分开。上下文检索、执行反馈修复及无界面Claude Code在其他面板单列。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.25939v2","status":"paper_baselines","published_at":"2026-08-26"},"model_count":14,"result_count":14,"results":[[13942,0.0042,370,13953,13944],[13945,0.0241,370,13953,13944],[13946,0.0382,370,13953,13944],[13947,0.0666,370,13953,13944],[13948,0.19260000000000002,370,13953,13944],[13949,0.0269,370,13953,13944],[4425,0.0524,370,13953,13944],[3190,0.0467,370,13953,13944],[3623,0.1091,370,13953,13944],[2271,0.2365,370,13953,13944],[13950,0.1388,370,13953,13944],[268,0.051,370,13953,13944],[13951,0.0227,370,13953,13944],[340,0.25780000000000003,370,13953,13944]]},{"id":"xrepotest-ruby-pass-2026-paper","name":"XRepoTest · ruby unit tests (2026 paper)","group":"编程","metric":"Test pass rate","unit":"%","public_history":true,"protocol":{"en":"All fourteen model checkpoints generate tests for the same XRepoTest ruby focal-function cohort under the standard single-turn setting. The original TPR percentage is retained, separately from line coverage and import resolution. Context retrieval, execution-feedback repair and headless Claude Code are separate configurations measured in other panels.","zh":"完整十四检查点按标准单轮协议，为XRepoTest相同ruby目标函数集合生成测试。保留原始TPR百分比，与行覆盖率和导入解析率分开。上下文检索、执行反馈修复及无界面Claude Code在其他面板单列。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.25939v2","status":"paper_baselines","published_at":"2026-08-26"},"model_count":14,"result_count":14,"results":[[13942,0.0,370,13954,13944],[13945,0.0148,370,13954,13944],[13946,0.0015,370,13954,13944],[13947,0.011899999999999999,370,13954,13944],[13948,0.0696,370,13954,13944],[13949,0.0044,370,13954,13944],[4425,0.0148,370,13954,13944],[3190,0.0163,370,13954,13944],[3623,0.043,370,13954,13944],[2271,0.0637,370,13954,13944],[13950,0.0889,370,13954,13944],[268,0.0311,370,13954,13944],[13951,0.0252,370,13954,13944],[340,0.25780000000000003,370,13954,13944]]},{"id":"xrepotest-php-pass-2026-paper","name":"XRepoTest · php unit tests (2026 paper)","group":"编程","metric":"Test pass rate","unit":"%","public_history":true,"protocol":{"en":"All fourteen model checkpoints generate tests for the same XRepoTest php focal-function cohort under the standard single-turn setting. The original TPR percentage is retained, separately from line coverage and import resolution. Context retrieval, execution-feedback repair and headless Claude Code are separate configurations measured in other panels.","zh":"完整十四检查点按标准单轮协议，为XRepoTest相同php目标函数集合生成测试。保留原始TPR百分比，与行覆盖率和导入解析率分开。上下文检索、执行反馈修复及无界面Claude Code在其他面板单列。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.25939v2","status":"paper_baselines","published_at":"2026-08-26"},"model_count":14,"result_count":14,"results":[[13942,0.0062,370,13955,13944],[13945,0.0356,370,13955,13944],[13946,0.0557,370,13955,13944],[13947,0.051100000000000007,370,13955,13944],[13948,0.0341,370,13955,13944],[13949,0.020099999999999996,370,13955,13944],[4425,0.0542,370,13955,13944],[3190,0.0557,370,13955,13944],[3623,0.14859999999999998,370,13955,13944],[2271,0.2693,370,13955,13944],[13950,0.13,370,13955,13944],[268,0.1316,370,13955,13944],[13951,0.0325,370,13955,13944],[340,0.10060000000000001,370,13955,13944]]},{"id":"uhi-berlin-airt-detection-2026-paper","name":"UHI-Bench · Berlin air-temperature extremes (2026 paper)","group":"预测与决策","metric":"Extreme-event detection F1","unit":"score","public_history":true,"protocol":{"en":"All twenty-nine learned-model/input configurations evaluate the same Berlin airt urban-heat extreme detection target under the common temporal-split and leakage-control protocol. The original 0–1 F1 is retained. RF, XGBoost, Isolation Forest, One-Class SVM, LSTM, TimesFM, Chronos-2 and MOIRAI-2 are eight distinct learned families; +s/+m input variants are configurations. Percentile and SeasonalNaive references are excluded from the learned-model cohort. Air-temperature and surface-temperature targets remain separate.","zh":"完整二十九学习模型、输入配置按共同时间划分与防泄漏协议，评测Berlin相同airt城市热岛极端事件目标，保留0–1原生F1。RF、XGBoost、Isolation Forest、One-Class SVM、LSTM、TimesFM、Chronos-2、MOIRAI-2为八种学习模型族，+s、+m输入变体作为配置。百分位和季节朴素参考不进入学习模型面板，气温与地表温度目标分开。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.23857v1","status":"paper_baselines","published_at":"2026-08-24"},"score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":29,"result_count":29,"results":[[12325,0.956,370,13956,13957],[13958,0.941,370,13959,13957],[13960,0.953,370,13961,13957],[13962,0.931,370,13963,13957],[6811,0.965,370,13964,13957],[13965,0.969,370,13966,13957],[13967,0.966,370,13968,13957],[13969,0.97,370,13970,13957],[13971,0.554,370,13972,13957],[13973,0.376,370,13974,13957],[13975,0.554,370,13976,13957],[13977,0.34,370,13978,13957],[13979,0.044,370,13980,13957],[13981,0.241,370,13982,13957],[13983,0.044,370,13984,13957],[13985,0.0,370,13986,13957],[9603,0.759,370,13987,13957],[13988,0.391,370,13989,13957],[13990,0.752,370,13991,13957],[13992,0.365,370,13993,13957],[9162,0.945,370,13994,13957],[6809,0.956,370,13995,13957],[13996,0.963,370,13997,13957],[13998,0.95,370,13999,13957],[14000,0.952,370,14001,13957],[14002,0.955,370,14003,13957],[14004,0.945,370,14005,13957],[14006,0.909,370,14007,13957],[14008,0.935,370,14009,13957]]},{"id":"uhi-berlin-lst-detection-2026-paper","name":"UHI-Bench · Berlin surface-temperature extremes (2026 paper)","group":"预测与决策","metric":"Extreme-event detection F1","unit":"score","public_history":true,"protocol":{"en":"All twenty-nine learned-model/input configurations evaluate the same Berlin lst urban-heat extreme detection target under the common temporal-split and leakage-control protocol. The original 0–1 F1 is retained. RF, XGBoost, Isolation Forest, One-Class SVM, LSTM, TimesFM, Chronos-2 and MOIRAI-2 are eight distinct learned families; +s/+m input variants are configurations. Percentile and SeasonalNaive references are excluded from the learned-model cohort. Air-temperature and surface-temperature targets remain separate.","zh":"完整二十九学习模型、输入配置按共同时间划分与防泄漏协议，评测Berlin相同lst城市热岛极端事件目标，保留0–1原生F1。RF、XGBoost、Isolation Forest、One-Class SVM、LSTM、TimesFM、Chronos-2、MOIRAI-2为八种学习模型族，+s、+m输入变体作为配置。百分位和季节朴素参考不进入学习模型面板，气温与地表温度目标分开。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.23857v1","status":"paper_baselines","published_at":"2026-08-24"},"score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":29,"result_count":29,"results":[[12325,0.322,370,14010,13957],[13958,0.313,370,14011,13957],[13960,0.321,370,14012,13957],[13962,0.297,370,14013,13957],[6811,0.249,370,14014,13957],[13965,0.338,370,14015,13957],[13967,0.218,370,14016,13957],[13969,0.398,370,14017,13957],[13971,0.262,370,14018,13957],[13973,0.264,370,14019,13957],[13975,0.215,370,14020,13957],[13977,0.258,370,14021,13957],[13979,0.0,370,14022,13957],[13981,0.0,370,14023,13957],[13983,0.0,370,14024,13957],[13985,0.0,370,14025,13957],[9603,0.087,370,14026,13957],[13988,0.03,370,14027,13957],[13990,0.115,370,14028,13957],[13992,0.025,370,14029,13957],[9162,0.413,370,14030,13957],[6809,0.466,370,14031,13957],[13996,0.508,370,14032,13957],[13998,0.452,370,14033,13957],[14000,0.472,370,14034,13957],[14002,0.481,370,14035,13957],[14004,0.241,370,14036,13957],[14006,0.319,370,14037,13957],[14008,0.282,370,14038,13957]]},{"id":"uhi-munich-airt-detection-2026-paper","name":"UHI-Bench · Munich air-temperature extremes (2026 paper)","group":"预测与决策","metric":"Extreme-event detection F1","unit":"score","public_history":true,"protocol":{"en":"All twenty-nine learned-model/input configurations evaluate the same Munich airt urban-heat extreme detection target under the common temporal-split and leakage-control protocol. The original 0–1 F1 is retained. RF, XGBoost, Isolation Forest, One-Class SVM, LSTM, TimesFM, Chronos-2 and MOIRAI-2 are eight distinct learned families; +s/+m input variants are configurations. Percentile and SeasonalNaive references are excluded from the learned-model cohort. Air-temperature and surface-temperature targets remain separate.","zh":"完整二十九学习模型、输入配置按共同时间划分与防泄漏协议，评测Munich相同airt城市热岛极端事件目标，保留0–1原生F1。RF、XGBoost、Isolation Forest、One-Class SVM、LSTM、TimesFM、Chronos-2、MOIRAI-2为八种学习模型族，+s、+m输入变体作为配置。百分位和季节朴素参考不进入学习模型面板，气温与地表温度目标分开。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.23857v1","status":"paper_baselines","published_at":"2026-08-24"},"score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":29,"result_count":29,"results":[[12325,0.949,370,14039,13957],[13958,0.941,370,14040,13957],[13960,0.94,370,14041,13957],[13962,0.93,370,14042,13957],[6811,0.96,370,14043,13957],[13965,0.964,370,14044,13957],[13967,0.96,370,14045,13957],[13969,0.962,370,14046,13957],[13971,0.539,370,14047,13957],[13973,0.352,370,14048,13957],[13975,0.543,370,14049,13957],[13977,0.333,370,14050,13957],[13979,0.0,370,14051,13957],[13981,0.12,370,14052,13957],[13983,0.0,370,14053,13957],[13985,0.12,370,14054,13957],[9603,0.694,370,14055,13957],[13988,0.524,370,14056,13957],[13990,0.674,370,14057,13957],[13992,0.471,370,14058,13957],[9162,0.923,370,14059,13957],[6809,0.955,370,14060,13957],[13996,0.954,370,14061,13957],[13998,0.951,370,14062,13957],[14000,0.946,370,14063,13957],[14002,0.956,370,14064,13957],[14004,0.948,370,14065,13957],[14006,0.893,370,14066,13957],[14008,0.923,370,14067,13957]]},{"id":"uhi-munich-lst-detection-2026-paper","name":"UHI-Bench · Munich surface-temperature extremes (2026 paper)","group":"预测与决策","metric":"Extreme-event detection F1","unit":"score","public_history":true,"protocol":{"en":"All twenty-nine learned-model/input configurations evaluate the same Munich lst urban-heat extreme detection target under the common temporal-split and leakage-control protocol. The original 0–1 F1 is retained. RF, XGBoost, Isolation Forest, One-Class SVM, LSTM, TimesFM, Chronos-2 and MOIRAI-2 are eight distinct learned families; +s/+m input variants are configurations. Percentile and SeasonalNaive references are excluded from the learned-model cohort. Air-temperature and surface-temperature targets remain separate.","zh":"完整二十九学习模型、输入配置按共同时间划分与防泄漏协议，评测Munich相同lst城市热岛极端事件目标，保留0–1原生F1。RF、XGBoost、Isolation Forest、One-Class SVM、LSTM、TimesFM、Chronos-2、MOIRAI-2为八种学习模型族，+s、+m输入变体作为配置。百分位和季节朴素参考不进入学习模型面板，气温与地表温度目标分开。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.23857v1","status":"paper_baselines","published_at":"2026-08-24"},"score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":29,"result_count":29,"results":[[12325,0.017,370,14068,13957],[13958,0.0,370,14069,13957],[13960,0.028,370,14070,13957],[13962,0.0,370,14071,13957],[6811,0.044,370,14072,13957],[13965,0.0,370,14073,13957],[13967,0.044,370,14074,13957],[13969,0.0,370,14075,13957],[13971,0.022,370,14076,13957],[13973,0.047,370,14077,13957],[13975,0.021,370,14078,13957],[13977,0.051,370,14079,13957],[13979,0.0,370,14080,13957],[13981,0.0,370,14081,13957],[13983,0.0,370,14082,13957],[13985,0.0,370,14083,13957],[9603,0.023,370,14084,13957],[13988,0.002,370,14085,13957],[13990,0.015,370,14086,13957],[13992,0.004,370,14087,13957],[9162,0.0,370,14088,13957],[6809,0.0,370,14089,13957],[13996,0.0,370,14090,13957],[13998,0.0,370,14091,13957],[14000,0.0,370,14092,13957],[14002,0.0,370,14093,13957],[14004,0.011,370,14094,13957],[14006,0.023,370,14095,13957],[14008,0.024,370,14096,13957]]},{"id":"uhi-hamburg-airt-detection-2026-paper","name":"UHI-Bench · Hamburg air-temperature extremes (2026 paper)","group":"预测与决策","metric":"Extreme-event detection F1","unit":"score","public_history":true,"protocol":{"en":"All twenty-nine learned-model/input configurations evaluate the same Hamburg airt urban-heat extreme detection target under the common temporal-split and leakage-control protocol. The original 0–1 F1 is retained. RF, XGBoost, Isolation Forest, One-Class SVM, LSTM, TimesFM, Chronos-2 and MOIRAI-2 are eight distinct learned families; +s/+m input variants are configurations. Percentile and SeasonalNaive references are excluded from the learned-model cohort. Air-temperature and surface-temperature targets remain separate.","zh":"完整二十九学习模型、输入配置按共同时间划分与防泄漏协议，评测Hamburg相同airt城市热岛极端事件目标，保留0–1原生F1。RF、XGBoost、Isolation Forest、One-Class SVM、LSTM、TimesFM、Chronos-2、MOIRAI-2为八种学习模型族，+s、+m输入变体作为配置。百分位和季节朴素参考不进入学习模型面板，气温与地表温度目标分开。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.23857v1","status":"paper_baselines","published_at":"2026-08-24"},"score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":29,"result_count":29,"results":[[12325,0.951,370,14097,13957],[13958,0.932,370,14098,13957],[13960,0.945,370,14099,13957],[13962,0.924,370,14100,13957],[6811,0.953,370,14101,13957],[13965,0.958,370,14102,13957],[13967,0.953,370,14103,13957],[13969,0.958,370,14104,13957],[13971,0.602,370,14105,13957],[13973,0.454,370,14106,13957],[13975,0.593,370,14107,13957],[13977,0.422,370,14108,13957],[13979,0.0,370,14109,13957],[13981,0.0,370,14110,13957],[13983,0.0,370,14111,13957],[13985,0.029,370,14112,13957],[9603,0.708,370,14113,13957],[13988,0.424,370,14114,13957],[13990,0.712,370,14115,13957],[13992,0.443,370,14116,13957],[9162,0.922,370,14117,13957],[6809,0.946,370,14118,13957],[13996,0.945,370,14119,13957],[13998,0.933,370,14120,13957],[14000,0.93,370,14121,13957],[14002,0.946,370,14122,13957],[14004,0.934,370,14123,13957],[14006,0.888,370,14124,13957],[14008,0.927,370,14125,13957]]},{"id":"uhi-hamburg-lst-detection-2026-paper","name":"UHI-Bench · Hamburg surface-temperature extremes (2026 paper)","group":"预测与决策","metric":"Extreme-event detection F1","unit":"score","public_history":true,"protocol":{"en":"All twenty-nine learned-model/input configurations evaluate the same Hamburg lst urban-heat extreme detection target under the common temporal-split and leakage-control protocol. The original 0–1 F1 is retained. RF, XGBoost, Isolation Forest, One-Class SVM, LSTM, TimesFM, Chronos-2 and MOIRAI-2 are eight distinct learned families; +s/+m input variants are configurations. Percentile and SeasonalNaive references are excluded from the learned-model cohort. Air-temperature and surface-temperature targets remain separate.","zh":"完整二十九学习模型、输入配置按共同时间划分与防泄漏协议，评测Hamburg相同lst城市热岛极端事件目标，保留0–1原生F1。RF、XGBoost、Isolation Forest、One-Class SVM、LSTM、TimesFM、Chronos-2、MOIRAI-2为八种学习模型族，+s、+m输入变体作为配置。百分位和季节朴素参考不进入学习模型面板，气温与地表温度目标分开。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.23857v1","status":"paper_baselines","published_at":"2026-08-24"},"score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":29,"result_count":29,"results":[[12325,0.032,370,14126,13957],[13958,0.127,370,14127,13957],[13960,0.0,370,14128,13957],[13962,0.05,370,14129,13957],[6811,0.082,370,14130,13957],[13965,0.105,370,14131,13957],[13967,0.082,370,14132,13957],[13969,0.105,370,14133,13957],[13971,0.047,370,14134,13957],[13973,0.0,370,14135,13957],[13975,0.036,370,14136,13957],[13977,0.0,370,14137,13957],[13979,0.0,370,14138,13957],[13981,0.0,370,14139,13957],[13983,0.0,370,14140,13957],[13985,0.0,370,14141,13957],[9603,0.026,370,14142,13957],[13988,0.011,370,14143,13957],[13990,0.018,370,14144,13957],[13992,0.004,370,14145,13957],[9162,0.074,370,14146,13957],[6809,0.0,370,14147,13957],[13996,0.0,370,14148,13957],[13998,0.0,370,14149,13957],[14000,0.0,370,14150,13957],[14002,0.067,370,14151,13957],[14004,0.078,370,14152,13957],[14006,0.077,370,14153,13957],[14008,0.084,370,14154,13957]]},{"id":"uhi-cairo-airt-detection-2026-paper","name":"UHI-Bench · Cairo air-temperature extremes (2026 paper)","group":"预测与决策","metric":"Extreme-event detection F1","unit":"score","public_history":true,"protocol":{"en":"All twenty-nine learned-model/input configurations evaluate the same Cairo airt urban-heat extreme detection target under the common temporal-split and leakage-control protocol. The original 0–1 F1 is retained. RF, XGBoost, Isolation Forest, One-Class SVM, LSTM, TimesFM, Chronos-2 and MOIRAI-2 are eight distinct learned families; +s/+m input variants are configurations. Percentile and SeasonalNaive references are excluded from the learned-model cohort. Air-temperature and surface-temperature targets remain separate.","zh":"完整二十九学习模型、输入配置按共同时间划分与防泄漏协议，评测Cairo相同airt城市热岛极端事件目标，保留0–1原生F1。RF、XGBoost、Isolation Forest、One-Class SVM、LSTM、TimesFM、Chronos-2、MOIRAI-2为八种学习模型族，+s、+m输入变体作为配置。百分位和季节朴素参考不进入学习模型面板，气温与地表温度目标分开。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.23857v1","status":"paper_baselines","published_at":"2026-08-24"},"score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":29,"result_count":29,"results":[[12325,0.85,370,14155,13957],[13958,0.869,370,14156,13957],[13960,0.846,370,14157,13957],[13962,0.861,370,14158,13957],[6811,0.844,370,14159,13957],[13965,0.903,370,14160,13957],[13967,0.844,370,14161,13957],[13969,0.903,370,14162,13957],[13971,0.574,370,14163,13957],[13973,0.113,370,14164,13957],[13975,0.553,370,14165,13957],[13977,0.129,370,14166,13957],[13979,0.0,370,14167,13957],[13981,0.0,370,14168,13957],[13983,0.0,370,14169,13957],[13985,0.04,370,14170,13957],[9603,0.685,370,14171,13957],[13988,0.497,370,14172,13957],[13990,0.689,370,14173,13957],[13992,0.473,370,14174,13957],[9162,0.827,370,14175,13957],[6809,0.846,370,14176,13957],[13996,0.85,370,14177,13957],[13998,0.83,370,14178,13957],[14000,0.829,370,14179,13957],[14002,0.844,370,14180,13957],[14004,0.762,370,14181,13957],[14006,0.775,370,14182,13957],[14008,0.793,370,14183,13957]]},{"id":"uhi-cairo-lst-detection-2026-paper","name":"UHI-Bench · Cairo surface-temperature extremes (2026 paper)","group":"预测与决策","metric":"Extreme-event detection F1","unit":"score","public_history":true,"protocol":{"en":"All twenty-nine learned-model/input configurations evaluate the same Cairo lst urban-heat extreme detection target under the common temporal-split and leakage-control protocol. The original 0–1 F1 is retained. RF, XGBoost, Isolation Forest, One-Class SVM, LSTM, TimesFM, Chronos-2 and MOIRAI-2 are eight distinct learned families; +s/+m input variants are configurations. Percentile and SeasonalNaive references are excluded from the learned-model cohort. Air-temperature and surface-temperature targets remain separate.","zh":"完整二十九学习模型、输入配置按共同时间划分与防泄漏协议，评测Cairo相同lst城市热岛极端事件目标，保留0–1原生F1。RF、XGBoost、Isolation Forest、One-Class SVM、LSTM、TimesFM、Chronos-2、MOIRAI-2为八种学习模型族，+s、+m输入变体作为配置。百分位和季节朴素参考不进入学习模型面板，气温与地表温度目标分开。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.23857v1","status":"paper_baselines","published_at":"2026-08-24"},"score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":29,"result_count":29,"results":[[12325,0.579,370,14184,13957],[13958,0.581,370,14185,13957],[13960,0.574,370,14186,13957],[13962,0.559,370,14187,13957],[6811,0.566,370,14188,13957],[13965,0.61,370,14189,13957],[13967,0.566,370,14190,13957],[13969,0.598,370,14191,13957],[13971,0.071,370,14192,13957],[13973,0.122,370,14193,13957],[13975,0.074,370,14194,13957],[13977,0.078,370,14195,13957],[13979,0.033,370,14196,13957],[13981,0.002,370,14197,13957],[13983,0.034,370,14198,13957],[13985,0.002,370,14199,13957],[9603,0.333,370,14200,13957],[13988,0.214,370,14201,13957],[13990,0.392,370,14202,13957],[13992,0.225,370,14203,13957],[9162,0.602,370,14204,13957],[6809,0.621,370,14205,13957],[13996,0.634,370,14206,13957],[13998,0.607,370,14207,13957],[14000,0.611,370,14208,13957],[14002,0.612,370,14209,13957],[14004,0.537,370,14210,13957],[14006,0.529,370,14211,13957],[14008,0.536,370,14212,13957]]},{"id":"uhi-lagos-airt-detection-2026-paper","name":"UHI-Bench · Lagos air-temperature extremes (2026 paper)","group":"预测与决策","metric":"Extreme-event detection F1","unit":"score","public_history":true,"protocol":{"en":"All twenty-nine learned-model/input configurations evaluate the same Lagos airt urban-heat extreme detection target under the common temporal-split and leakage-control protocol. The original 0–1 F1 is retained. RF, XGBoost, Isolation Forest, One-Class SVM, LSTM, TimesFM, Chronos-2 and MOIRAI-2 are eight distinct learned families; +s/+m input variants are configurations. Percentile and SeasonalNaive references are excluded from the learned-model cohort. Air-temperature and surface-temperature targets remain separate.","zh":"完整二十九学习模型、输入配置按共同时间划分与防泄漏协议，评测Lagos相同airt城市热岛极端事件目标，保留0–1原生F1。RF、XGBoost、Isolation Forest、One-Class SVM、LSTM、TimesFM、Chronos-2、MOIRAI-2为八种学习模型族，+s、+m输入变体作为配置。百分位和季节朴素参考不进入学习模型面板，气温与地表温度目标分开。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.23857v1","status":"paper_baselines","published_at":"2026-08-24"},"score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":29,"result_count":29,"results":[[12325,0.74,370,14213,13957],[13958,0.752,370,14214,13957],[13960,0.731,370,14215,13957],[13962,0.748,370,14216,13957],[6811,0.726,370,14217,13957],[13965,0.789,370,14218,13957],[13967,0.726,370,14219,13957],[13969,0.79,370,14220,13957],[13971,0.454,370,14221,13957],[13973,0.541,370,14222,13957],[13975,0.465,370,14223,13957],[13977,0.545,370,14224,13957],[13979,0.0,370,14225,13957],[13981,0.014,370,14226,13957],[13983,0.001,370,14227,13957],[13985,0.0,370,14228,13957],[9603,0.388,370,14229,13957],[13988,0.343,370,14230,13957],[13990,0.427,370,14231,13957],[13992,0.361,370,14232,13957],[9162,0.747,370,14233,13957],[6809,0.754,370,14234,13957],[13996,0.76,370,14235,13957],[13998,0.753,370,14236,13957],[14000,0.754,370,14237,13957],[14002,0.743,370,14238,13957],[14004,0.665,370,14239,13957],[14006,0.672,370,14240,13957],[14008,0.686,370,14241,13957]]},{"id":"uhi-lagos-lst-detection-2026-paper","name":"UHI-Bench · Lagos surface-temperature extremes (2026 paper)","group":"预测与决策","metric":"Extreme-event detection F1","unit":"score","public_history":true,"protocol":{"en":"All twenty-nine learned-model/input configurations evaluate the same Lagos lst urban-heat extreme detection target under the common temporal-split and leakage-control protocol. The original 0–1 F1 is retained. RF, XGBoost, Isolation Forest, One-Class SVM, LSTM, TimesFM, Chronos-2 and MOIRAI-2 are eight distinct learned families; +s/+m input variants are configurations. Percentile and SeasonalNaive references are excluded from the learned-model cohort. Air-temperature and surface-temperature targets remain separate.","zh":"完整二十九学习模型、输入配置按共同时间划分与防泄漏协议，评测Lagos相同lst城市热岛极端事件目标，保留0–1原生F1。RF、XGBoost、Isolation Forest、One-Class SVM、LSTM、TimesFM、Chronos-2、MOIRAI-2为八种学习模型族，+s、+m输入变体作为配置。百分位和季节朴素参考不进入学习模型面板，气温与地表温度目标分开。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.23857v1","status":"paper_baselines","published_at":"2026-08-24"},"score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":29,"result_count":29,"results":[[12325,0.378,370,14242,13957],[13958,0.396,370,14243,13957],[13960,0.396,370,14244,13957],[13962,0.37,370,14245,13957],[6811,0.361,370,14246,13957],[13965,0.377,370,14247,13957],[13967,0.361,370,14248,13957],[13969,0.381,370,14249,13957],[13971,0.179,370,14250,13957],[13973,0.0,370,14251,13957],[13975,0.207,370,14252,13957],[13977,0.0,370,14253,13957],[13979,0.012,370,14254,13957],[13981,0.006,370,14255,13957],[13983,0.012,370,14256,13957],[13985,0.006,370,14257,13957],[9603,0.252,370,14258,13957],[13988,0.031,370,14259,13957],[13990,0.282,370,14260,13957],[13992,0.025,370,14261,13957],[9162,0.303,370,14262,13957],[6809,0.441,370,14263,13957],[13996,0.501,370,14264,13957],[13998,0.343,370,14265,13957],[14000,0.437,370,14266,13957],[14002,0.404,370,14267,13957],[14004,0.327,370,14268,13957],[14006,0.383,370,14269,13957],[14008,0.349,370,14270,13957]]},{"id":"uhi-johannesburg-airt-detection-2026-paper","name":"UHI-Bench · Johannesburg air-temperature extremes (2026 paper)","group":"预测与决策","metric":"Extreme-event detection F1","unit":"score","public_history":true,"protocol":{"en":"All twenty-nine learned-model/input configurations evaluate the same Johannesburg airt urban-heat extreme detection target under the common temporal-split and leakage-control protocol. The original 0–1 F1 is retained. RF, XGBoost, Isolation Forest, One-Class SVM, LSTM, TimesFM, Chronos-2 and MOIRAI-2 are eight distinct learned families; +s/+m input variants are configurations. Percentile and SeasonalNaive references are excluded from the learned-model cohort. Air-temperature and surface-temperature targets remain separate.","zh":"完整二十九学习模型、输入配置按共同时间划分与防泄漏协议，评测Johannesburg相同airt城市热岛极端事件目标，保留0–1原生F1。RF、XGBoost、Isolation Forest、One-Class SVM、LSTM、TimesFM、Chronos-2、MOIRAI-2为八种学习模型族，+s、+m输入变体作为配置。百分位和季节朴素参考不进入学习模型面板，气温与地表温度目标分开。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.23857v1","status":"paper_baselines","published_at":"2026-08-24"},"score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":29,"result_count":29,"results":[[12325,0.739,370,14271,13957],[13958,0.753,370,14272,13957],[13960,0.736,370,14273,13957],[13962,0.741,370,14274,13957],[6811,0.713,370,14275,13957],[13965,0.786,370,14276,13957],[13967,0.713,370,14277,13957],[13969,0.779,370,14278,13957],[13971,0.496,370,14279,13957],[13973,0.128,370,14280,13957],[13975,0.496,370,14281,13957],[13977,0.1,370,14282,13957],[13979,0.0,370,14283,13957],[13981,0.113,370,14284,13957],[13983,0.0,370,14285,13957],[13985,0.212,370,14286,13957],[9603,0.504,370,14287,13957],[13988,0.41,370,14288,13957],[13990,0.503,370,14289,13957],[13992,0.428,370,14290,13957],[9162,0.748,370,14291,13957],[6809,0.773,370,14292,13957],[13996,0.777,370,14293,13957],[13998,0.761,370,14294,13957],[14000,0.763,370,14295,13957],[14002,0.779,370,14296,13957],[14004,0.734,370,14297,13957],[14006,0.731,370,14298,13957],[14008,0.746,370,14299,13957]]},{"id":"uhi-johannesburg-lst-detection-2026-paper","name":"UHI-Bench · Johannesburg surface-temperature extremes (2026 paper)","group":"预测与决策","metric":"Extreme-event detection F1","unit":"score","public_history":true,"protocol":{"en":"All twenty-nine learned-model/input configurations evaluate the same Johannesburg lst urban-heat extreme detection target under the common temporal-split and leakage-control protocol. The original 0–1 F1 is retained. RF, XGBoost, Isolation Forest, One-Class SVM, LSTM, TimesFM, Chronos-2 and MOIRAI-2 are eight distinct learned families; +s/+m input variants are configurations. Percentile and SeasonalNaive references are excluded from the learned-model cohort. Air-temperature and surface-temperature targets remain separate.","zh":"完整二十九学习模型、输入配置按共同时间划分与防泄漏协议，评测Johannesburg相同lst城市热岛极端事件目标，保留0–1原生F1。RF、XGBoost、Isolation Forest、One-Class SVM、LSTM、TimesFM、Chronos-2、MOIRAI-2为八种学习模型族，+s、+m输入变体作为配置。百分位和季节朴素参考不进入学习模型面板，气温与地表温度目标分开。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.23857v1","status":"paper_baselines","published_at":"2026-08-24"},"score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":29,"result_count":29,"results":[[12325,0.663,370,14300,13957],[13958,0.659,370,14301,13957],[13960,0.659,370,14302,13957],[13962,0.66,370,14303,13957],[6811,0.633,370,14304,13957],[13965,0.672,370,14305,13957],[13967,0.633,370,14306,13957],[13969,0.673,370,14307,13957],[13971,0.182,370,14308,13957],[13973,0.413,370,14309,13957],[13975,0.221,370,14310,13957],[13977,0.387,370,14311,13957],[13979,0.004,370,14312,13957],[13981,0.078,370,14313,13957],[13983,0.004,370,14314,13957],[13985,0.078,370,14315,13957],[9603,0.539,370,14316,13957],[13988,0.263,370,14317,13957],[13990,0.552,370,14318,13957],[13992,0.336,370,14319,13957],[9162,0.666,370,14320,13957],[6809,0.689,370,14321,13957],[13996,0.708,370,14322,13957],[13998,0.672,370,14323,13957],[14000,0.688,370,14324,13957],[14002,0.685,370,14325,13957],[14004,0.653,370,14326,13957],[14006,0.637,370,14327,13957],[14008,0.659,370,14328,13957]]},{"id":"uhi-riyadh-airt-detection-2026-paper","name":"UHI-Bench · Riyadh air-temperature extremes (2026 paper)","group":"预测与决策","metric":"Extreme-event detection F1","unit":"score","public_history":true,"protocol":{"en":"All twenty-nine learned-model/input configurations evaluate the same Riyadh airt urban-heat extreme detection target under the common temporal-split and leakage-control protocol. The original 0–1 F1 is retained. RF, XGBoost, Isolation Forest, One-Class SVM, LSTM, TimesFM, Chronos-2 and MOIRAI-2 are eight distinct learned families; +s/+m input variants are configurations. Percentile and SeasonalNaive references are excluded from the learned-model cohort. Air-temperature and surface-temperature targets remain separate.","zh":"完整二十九学习模型、输入配置按共同时间划分与防泄漏协议，评测Riyadh相同airt城市热岛极端事件目标，保留0–1原生F1。RF、XGBoost、Isolation Forest、One-Class SVM、LSTM、TimesFM、Chronos-2、MOIRAI-2为八种学习模型族，+s、+m输入变体作为配置。百分位和季节朴素参考不进入学习模型面板，气温与地表温度目标分开。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.23857v1","status":"paper_baselines","published_at":"2026-08-24"},"score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":29,"result_count":29,"results":[[12325,0.87,370,14329,13957],[13958,0.872,370,14330,13957],[13960,0.864,370,14331,13957],[13962,0.871,370,14332,13957],[6811,0.876,370,14333,13957],[13965,0.898,370,14334,13957],[13967,0.876,370,14335,13957],[13969,0.898,370,14336,13957],[13971,0.655,370,14337,13957],[13973,0.101,370,14338,13957],[13975,0.652,370,14339,13957],[13977,0.067,370,14340,13957],[13979,0.251,370,14341,13957],[13981,0.0,370,14342,13957],[13983,0.251,370,14343,13957],[13985,0.0,370,14344,13957],[9603,0.688,370,14345,13957],[13988,0.708,370,14346,13957],[13990,0.665,370,14347,13957],[13992,0.683,370,14348,13957],[9162,0.846,370,14349,13957],[6809,0.857,370,14350,13957],[13996,0.862,370,14351,13957],[13998,0.844,370,14352,13957],[14000,0.844,370,14353,13957],[14002,0.854,370,14354,13957],[14004,0.78,370,14355,13957],[14006,0.801,370,14356,13957],[14008,0.802,370,14357,13957]]},{"id":"uhi-riyadh-lst-detection-2026-paper","name":"UHI-Bench · Riyadh surface-temperature extremes (2026 paper)","group":"预测与决策","metric":"Extreme-event detection F1","unit":"score","public_history":true,"protocol":{"en":"All twenty-nine learned-model/input configurations evaluate the same Riyadh lst urban-heat extreme detection target under the common temporal-split and leakage-control protocol. The original 0–1 F1 is retained. RF, XGBoost, Isolation Forest, One-Class SVM, LSTM, TimesFM, Chronos-2 and MOIRAI-2 are eight distinct learned families; +s/+m input variants are configurations. Percentile and SeasonalNaive references are excluded from the learned-model cohort. Air-temperature and surface-temperature targets remain separate.","zh":"完整二十九学习模型、输入配置按共同时间划分与防泄漏协议，评测Riyadh相同lst城市热岛极端事件目标，保留0–1原生F1。RF、XGBoost、Isolation Forest、One-Class SVM、LSTM、TimesFM、Chronos-2、MOIRAI-2为八种学习模型族，+s、+m输入变体作为配置。百分位和季节朴素参考不进入学习模型面板，气温与地表温度目标分开。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.23857v1","status":"paper_baselines","published_at":"2026-08-24"},"score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":29,"result_count":29,"results":[[12325,0.481,370,14358,13957],[13958,0.506,370,14359,13957],[13960,0.484,370,14360,13957],[13962,0.492,370,14361,13957],[6811,0.46,370,14362,13957],[13965,0.462,370,14363,13957],[13967,0.46,370,14364,13957],[13969,0.462,370,14365,13957],[13971,0.005,370,14366,13957],[13973,0.016,370,14367,13957],[13975,0.004,370,14368,13957],[13977,0.019,370,14369,13957],[13979,0.006,370,14370,13957],[13981,0.081,370,14371,13957],[13983,0.006,370,14372,13957],[13985,0.081,370,14373,13957],[9603,0.448,370,14374,13957],[13988,0.038,370,14375,13957],[13990,0.431,370,14376,13957],[13992,0.04,370,14377,13957],[9162,0.422,370,14378,13957],[6809,0.483,370,14379,13957],[13996,0.567,370,14380,13957],[13998,0.459,370,14381,13957],[14000,0.535,370,14382,13957],[14002,0.463,370,14383,13957],[14004,0.376,370,14384,13957],[14006,0.322,370,14385,13957],[14008,0.326,370,14386,13957]]},{"id":"vbvr-pro-video-2026-paper","name":"VBVR-Pro · native video reasoning (2026 paper)","group":"视觉与视频","metric":"Overall verifiable visual-reasoning score","unit":"score","public_history":true,"protocol":{"en":"The complete fourteen-system video-output panel evaluates the same VBVR-Pro-Bench tasks with deterministic task-specific visual scorers. The original Overall score is a native 0–1 composite, separating in-domain and out-of-domain task families in the source. Proposed trained variants retain their original base checkpoints. Image-only and interleaved text-image regimes have different output contracts and are separate.","zh":"完整十四系统视频输出面板评测相同VBVR-Pro-Bench任务，采用针对各任务的确定性视觉评分器。保留0–1原生Overall综合分，原表另列域内、域外任务族。提案训练变体保留其原始底模检查点。纯图像及交错图文输出协议不同，单独处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.26105v2","status":"paper_baselines","published_at":"2026-08-26"},"score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":14,"result_count":14,"results":[[9060,0.309,370,14387,14388],[14389,0.392,370,14387,14388],[6383,0.499,370,14387,14388],[8304,0.054,370,14387,14388],[14390,0.085,370,14387,14388],[14391,0.1,370,14387,14388],[8309,0.094,370,14387,14388],[8310,0.182,370,14387,14388],[14392,0.112,370,14387,14388],[14393,0.517,370,14394,14388],[14395,0.425,370,14396,14388],[14397,0.562,370,14396,14388],[14398,0.47,370,14396,14388],[14399,0.67,370,14396,14388]]},{"id":"tsugo-symbolic-four-candidate-easy-2026-paper","name":"TsuGO · easy symbolic four-candidate (2026 paper)","group":"知识与推理","metric":"First-move hit accuracy","unit":"%","public_history":true,"protocol":{"en":"All eight explicitly measured LLM checkpoints solve the same easy Go problems in the four-candidate setting using Symbolic input. The original first-move hit rate is retained per fixed input modality. Trace-derived SearchE, token efficiency, mixed-modality averages and proprietary summary-derived rows are separate. Candidate count and difficulty remain explicit rather than pooled.","zh":"完整八个有明确实测结果的大模型以符号输入，在four-candidate设置下解答相同easy围棋题。保留固定输入模态的原始首步命中准确率。轨迹SearchE、词元效率、混合模态均值及仅由摘要推得的商业模型行单列。候选数量与难度明确区分。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.13221v1","status":"paper_baselines","published_at":"2026-08-13"},"model_count":8,"result_count":8,"results":[[2998,0.57,370,14400,14401],[6297,0.46,370,14400,14401],[5722,0.33,370,14400,14401],[6299,0.34,370,14400,14401],[3005,0.41,370,14400,14401],[773,0.32,370,14400,14401],[8164,0.2,370,14400,14401],[3009,0.33,370,14400,14401]]},{"id":"tsugo-symbolic-four-candidate-medium-2026-paper","name":"TsuGO · medium symbolic four-candidate (2026 paper)","group":"知识与推理","metric":"First-move hit accuracy","unit":"%","public_history":true,"protocol":{"en":"All eight explicitly measured LLM checkpoints solve the same medium Go problems in the four-candidate setting using Symbolic input. The original first-move hit rate is retained per fixed input modality. Trace-derived SearchE, token efficiency, mixed-modality averages and proprietary summary-derived rows are separate. Candidate count and difficulty remain explicit rather than pooled.","zh":"完整八个有明确实测结果的大模型以符号输入，在four-candidate设置下解答相同medium围棋题。保留固定输入模态的原始首步命中准确率。轨迹SearchE、词元效率、混合模态均值及仅由摘要推得的商业模型行单列。候选数量与难度明确区分。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.13221v1","status":"paper_baselines","published_at":"2026-08-13"},"model_count":8,"result_count":8,"results":[[2998,0.25,370,14402,14401],[6297,0.23,370,14402,14401],[5722,0.17,370,14402,14401],[6299,0.24,370,14402,14401],[3005,0.22,370,14402,14401],[773,0.31,370,14402,14401],[8164,0.21,370,14402,14401],[3009,0.21,370,14402,14401]]},{"id":"tsugo-symbolic-four-candidate-hard-2026-paper","name":"TsuGO · hard symbolic four-candidate (2026 paper)","group":"知识与推理","metric":"First-move hit accuracy","unit":"%","public_history":true,"protocol":{"en":"All eight explicitly measured LLM checkpoints solve the same hard Go problems in the four-candidate setting using Symbolic input. The original first-move hit rate is retained per fixed input modality. Trace-derived SearchE, token efficiency, mixed-modality averages and proprietary summary-derived rows are separate. Candidate count and difficulty remain explicit rather than pooled.","zh":"完整八个有明确实测结果的大模型以符号输入，在four-candidate设置下解答相同hard围棋题。保留固定输入模态的原始首步命中准确率。轨迹SearchE、词元效率、混合模态均值及仅由摘要推得的商业模型行单列。候选数量与难度明确区分。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.13221v1","status":"paper_baselines","published_at":"2026-08-13"},"model_count":8,"result_count":8,"results":[[2998,0.33,370,14403,14401],[6297,0.31,370,14403,14401],[5722,0.24,370,14403,14401],[6299,0.36,370,14403,14401],[3005,0.25,370,14403,14401],[773,0.28,370,14403,14401],[8164,0.24,370,14403,14401],[3009,0.37,370,14403,14401]]},{"id":"tsugo-symbolic-open-easy-2026-paper","name":"TsuGO · easy symbolic open (2026 paper)","group":"知识与推理","metric":"First-move hit accuracy","unit":"%","public_history":true,"protocol":{"en":"All eight explicitly measured LLM checkpoints solve the same easy Go problems in the open setting using Symbolic input. The original first-move hit rate is retained per fixed input modality. Trace-derived SearchE, token efficiency, mixed-modality averages and proprietary summary-derived rows are separate. Candidate count and difficulty remain explicit rather than pooled.","zh":"完整八个有明确实测结果的大模型以符号输入，在open设置下解答相同easy围棋题。保留固定输入模态的原始首步命中准确率。轨迹SearchE、词元效率、混合模态均值及仅由摘要推得的商业模型行单列。候选数量与难度明确区分。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.13221v1","status":"paper_baselines","published_at":"2026-08-13"},"model_count":8,"result_count":8,"results":[[2998,0.29,370,14404,14401],[6297,0.17,370,14404,14401],[5722,0.04,370,14404,14401],[6299,0.09,370,14404,14401],[3005,0.1,370,14404,14401],[773,0.13,370,14404,14401],[8164,0.15,370,14404,14401],[3009,0.19,370,14404,14401]]},{"id":"tsugo-symbolic-open-medium-2026-paper","name":"TsuGO · medium symbolic open (2026 paper)","group":"知识与推理","metric":"First-move hit accuracy","unit":"%","public_history":true,"protocol":{"en":"All eight explicitly measured LLM checkpoints solve the same medium Go problems in the open setting using Symbolic input. The original first-move hit rate is retained per fixed input modality. Trace-derived SearchE, token efficiency, mixed-modality averages and proprietary summary-derived rows are separate. Candidate count and difficulty remain explicit rather than pooled.","zh":"完整八个有明确实测结果的大模型以符号输入，在open设置下解答相同medium围棋题。保留固定输入模态的原始首步命中准确率。轨迹SearchE、词元效率、混合模态均值及仅由摘要推得的商业模型行单列。候选数量与难度明确区分。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.13221v1","status":"paper_baselines","published_at":"2026-08-13"},"model_count":8,"result_count":8,"results":[[2998,0.1,370,14405,14401],[6297,0.11,370,14405,14401],[5722,0.0,370,14405,14401],[6299,0.07,370,14405,14401],[3005,0.09,370,14405,14401],[773,0.08,370,14405,14401],[8164,0.08,370,14405,14401],[3009,0.06,370,14405,14401]]},{"id":"tsugo-symbolic-open-hard-2026-paper","name":"TsuGO · hard symbolic open (2026 paper)","group":"知识与推理","metric":"First-move hit accuracy","unit":"%","public_history":true,"protocol":{"en":"All eight explicitly measured LLM checkpoints solve the same hard Go problems in the open setting using Symbolic input. The original first-move hit rate is retained per fixed input modality. Trace-derived SearchE, token efficiency, mixed-modality averages and proprietary summary-derived rows are separate. Candidate count and difficulty remain explicit rather than pooled.","zh":"完整八个有明确实测结果的大模型以符号输入，在open设置下解答相同hard围棋题。保留固定输入模态的原始首步命中准确率。轨迹SearchE、词元效率、混合模态均值及仅由摘要推得的商业模型行单列。候选数量与难度明确区分。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.13221v1","status":"paper_baselines","published_at":"2026-08-13"},"model_count":8,"result_count":8,"results":[[2998,0.09,370,14406,14401],[6297,0.06,370,14406,14401],[5722,0.03,370,14406,14401],[6299,0.06,370,14406,14401],[3005,0.03,370,14406,14401],[773,0.04,370,14406,14401],[8164,0.03,370,14406,14401],[3009,0.05,370,14406,14401]]},{"id":"ra-bench-binary-2026-paper","name":"RA-Bench · binary crisis-video detection (2026 paper)","group":"安全与可靠性","metric":"Source-equal balanced accuracy","unit":"%","public_history":true,"protocol":{"en":"The full ten-model zero-shot appendix panel evaluates the same paired real/crisis-generated video benchmark. The All column equally averages the nine generator sources rather than pooling unequal source counts; four open and five closed generators are included. Binary/Diagnostic use balanced accuracy, while Rating uses paired AUC. Top values are selected; the macro-F1 below and the fixed-duration Wan control are separate.","zh":"完整十模型零样本附录面板评测相同配对真实、危机生成视频。All等权平均九个生成来源，包含四个开放、五个封闭生成器，不按不等样本数混合。Binary、Diagnostic采用平衡准确率，Rating采用配对AUC；取单元格上方数值，下方宏F1及固定时长Wan对照单列。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.14391v2","status":"paper_baselines","published_at":"2026-08-14"},"model_count":10,"result_count":10,"results":[[6340,0.5429999999999999,370,14407,14408],[5828,0.502,370,14407,14408],[4969,0.522,370,14407,14408],[4070,0.532,370,14407,14408],[2584,0.532,370,14407,14408],[2590,0.539,370,14407,14408],[2582,0.53,370,14407,14408],[14409,0.55,370,14407,14408],[6237,0.634,370,14407,14408],[3196,0.518,370,14407,14408]]},{"id":"ra-bench-diagnostic-2026-paper","name":"RA-Bench · diagnostic crisis-video detection (2026 paper)","group":"安全与可靠性","metric":"Source-equal balanced accuracy","unit":"%","public_history":true,"protocol":{"en":"The full ten-model zero-shot appendix panel evaluates the same paired real/crisis-generated video benchmark. The All column equally averages the nine generator sources rather than pooling unequal source counts; four open and five closed generators are included. Binary/Diagnostic use balanced accuracy, while Rating uses paired AUC. Top values are selected; the macro-F1 below and the fixed-duration Wan control are separate.","zh":"完整十模型零样本附录面板评测相同配对真实、危机生成视频。All等权平均九个生成来源，包含四个开放、五个封闭生成器，不按不等样本数混合。Binary、Diagnostic采用平衡准确率，Rating采用配对AUC；取单元格上方数值，下方宏F1及固定时长Wan对照单列。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.14391v2","status":"paper_baselines","published_at":"2026-08-14"},"model_count":10,"result_count":10,"results":[[6340,0.498,370,14410,14408],[5828,0.508,370,14410,14408],[4969,0.503,370,14410,14408],[4070,0.501,370,14410,14408],[2584,0.513,370,14410,14408],[2590,0.516,370,14410,14408],[2582,0.5479999999999999,370,14410,14408],[14409,0.5529999999999999,370,14410,14408],[6237,0.631,370,14410,14408],[3196,0.512,370,14410,14408]]},{"id":"ra-bench-rating-2026-paper","name":"RA-Bench · rating crisis-video detection (2026 paper)","group":"安全与可靠性","metric":"Source-equal paired AUC","unit":"score","public_history":true,"protocol":{"en":"The full ten-model zero-shot appendix panel evaluates the same paired real/crisis-generated video benchmark. The All column equally averages the nine generator sources rather than pooling unequal source counts; four open and five closed generators are included. Binary/Diagnostic use balanced accuracy, while Rating uses paired AUC. Top values are selected; the macro-F1 below and the fixed-duration Wan control are separate.","zh":"完整十模型零样本附录面板评测相同配对真实、危机生成视频。All等权平均九个生成来源，包含四个开放、五个封闭生成器，不按不等样本数混合。Binary、Diagnostic采用平衡准确率，Rating采用配对AUC；取单元格上方数值，下方宏F1及固定时长Wan对照单列。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.14391v2","status":"paper_baselines","published_at":"2026-08-14"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":10,"result_count":10,"results":[[6340,51.5,370,14411,14408],[5828,52.3,370,14411,14408],[4969,51.4,370,14411,14408],[4070,49.4,370,14411,14408],[2584,51.2,370,14411,14408],[2590,56.5,370,14411,14408],[2582,55.8,370,14411,14408],[14409,58.6,370,14411,14408],[6237,63.6,370,14411,14408],[3196,62.1,370,14411,14408]]},{"id":"wse-coverage-story-plain-2026-paper","name":"WSE-bench · story-plain coverage (2026 paper)","group":"生成与创作","metric":"Generation coverage","unit":"score","public_history":true,"protocol":{"en":"All twelve tested model checkpoints fill every enabled role for the same forty-two EpisodeSeeds under the fixed story-plain architecture. The original 0–100 Generation coverage is retained as a native measure. Coverage is observed steps out of twenty planned steps; consistency uses the published Kaplan–Meier C_obs and may carry survival beyond observational support, so it must be read with coverage. Richness combines log-scaled development frequency and breadth, separately from consistency. Complete history is shared; memory adds structured state. Three judges and held-out human validation are described in the paper.","zh":"完整十二模型在固定story-plain架构下，对相同42个情节种子承担全部启用角色。保留0–100原生Generation coverage。覆盖率为20个计划叙事步骤中的实测比例；一致性采用原表Kaplan–Meier C_obs，观测耗尽后可延续存活估计，须结合覆盖率解释。丰富度综合对数缩放的发展频次与广度，与一致性分开。所有条件均保留完整历史，记忆模块额外加入结构化状态。原文给出三评审与独立人工验证。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.15654v1","status":"paper_baselines","published_at":"2026-08-16"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":12,"result_count":12,"results":[[14412,3.1,370,14413,14414],[1748,66.5,370,14413,14414],[1659,34.5,370,14413,14414],[1645,100.0,370,14413,14414],[1651,100.0,370,14413,14414],[2576,100.0,370,14413,14414],[5411,100.0,370,14413,14414],[227,98.6,370,14413,14414],[3185,98.0,370,14413,14414],[3187,96.4,370,14413,14414],[502,98.5,370,14413,14414],[4048,100.0,370,14413,14414]]},{"id":"wse-coverage-story-memory-2026-paper","name":"WSE-bench · story-memory coverage (2026 paper)","group":"生成与创作","metric":"Generation coverage","unit":"score","public_history":true,"protocol":{"en":"All twelve tested model checkpoints fill every enabled role for the same forty-two EpisodeSeeds under the fixed story-memory architecture. The original 0–100 Generation coverage is retained as a native measure. Coverage is observed steps out of twenty planned steps; consistency uses the published Kaplan–Meier C_obs and may carry survival beyond observational support, so it must be read with coverage. Richness combines log-scaled development frequency and breadth, separately from consistency. Complete history is shared; memory adds structured state. Three judges and held-out human validation are described in the paper.","zh":"完整十二模型在固定story-memory架构下，对相同42个情节种子承担全部启用角色。保留0–100原生Generation coverage。覆盖率为20个计划叙事步骤中的实测比例；一致性采用原表Kaplan–Meier C_obs，观测耗尽后可延续存活估计，须结合覆盖率解释。丰富度综合对数缩放的发展频次与广度，与一致性分开。所有条件均保留完整历史，记忆模块额外加入结构化状态。原文给出三评审与独立人工验证。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.15654v1","status":"paper_baselines","published_at":"2026-08-16"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":12,"result_count":12,"results":[[14412,1.9,370,14415,14414],[1748,23.9,370,14415,14414],[1659,24.8,370,14415,14414],[1645,65.6,370,14415,14414],[1651,83.2,370,14415,14414],[2576,79.9,370,14415,14414],[5411,95.4,370,14415,14414],[227,42.3,370,14415,14414],[3185,76.0,370,14415,14414],[3187,94.8,370,14415,14414],[502,92.3,370,14415,14414],[4048,98.1,370,14415,14414]]},{"id":"wse-coverage-story-planning-2026-paper","name":"WSE-bench · story-planning coverage (2026 paper)","group":"生成与创作","metric":"Generation coverage","unit":"score","public_history":true,"protocol":{"en":"All twelve tested model checkpoints fill every enabled role for the same forty-two EpisodeSeeds under the fixed story-planning architecture. The original 0–100 Generation coverage is retained as a native measure. Coverage is observed steps out of twenty planned steps; consistency uses the published Kaplan–Meier C_obs and may carry survival beyond observational support, so it must be read with coverage. Richness combines log-scaled development frequency and breadth, separately from consistency. Complete history is shared; memory adds structured state. Three judges and held-out human validation are described in the paper.","zh":"完整十二模型在固定story-planning架构下，对相同42个情节种子承担全部启用角色。保留0–100原生Generation coverage。覆盖率为20个计划叙事步骤中的实测比例；一致性采用原表Kaplan–Meier C_obs，观测耗尽后可延续存活估计，须结合覆盖率解释。丰富度综合对数缩放的发展频次与广度，与一致性分开。所有条件均保留完整历史，记忆模块额外加入结构化状态。原文给出三评审与独立人工验证。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.15654v1","status":"paper_baselines","published_at":"2026-08-16"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":12,"result_count":12,"results":[[14412,1.1,370,14416,14414],[1748,4.9,370,14416,14414],[1659,5.2,370,14416,14414],[1645,71.7,370,14416,14414],[1651,92.3,370,14416,14414],[2576,100.0,370,14416,14414],[5411,97.6,370,14416,14414],[227,83.8,370,14416,14414],[3185,93.9,370,14416,14414],[3187,84.9,370,14416,14414],[502,93.2,370,14416,14414],[4048,100.0,370,14416,14414]]},{"id":"wse-coverage-story-memory-planning-2026-paper","name":"WSE-bench · story-memory-planning coverage (2026 paper)","group":"生成与创作","metric":"Generation coverage","unit":"score","public_history":true,"protocol":{"en":"All twelve tested model checkpoints fill every enabled role for the same forty-two EpisodeSeeds under the fixed story-memory-planning architecture. The original 0–100 Generation coverage is retained as a native measure. Coverage is observed steps out of twenty planned steps; consistency uses the published Kaplan–Meier C_obs and may carry survival beyond observational support, so it must be read with coverage. Richness combines log-scaled development frequency and breadth, separately from consistency. Complete history is shared; memory adds structured state. Three judges and held-out human validation are described in the paper.","zh":"完整十二模型在固定story-memory-planning架构下，对相同42个情节种子承担全部启用角色。保留0–100原生Generation coverage。覆盖率为20个计划叙事步骤中的实测比例；一致性采用原表Kaplan–Meier C_obs，观测耗尽后可延续存活估计，须结合覆盖率解释。丰富度综合对数缩放的发展频次与广度，与一致性分开。所有条件均保留完整历史，记忆模块额外加入结构化状态。原文给出三评审与独立人工验证。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.15654v1","status":"paper_baselines","published_at":"2026-08-16"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":12,"result_count":12,"results":[[14412,3.1,370,14417,14414],[1748,4.8,370,14417,14414],[1659,5.1,370,14417,14414],[1645,46.4,370,14417,14414],[1651,77.0,370,14417,14414],[2576,80.2,370,14417,14414],[5411,95.4,370,14417,14414],[227,33.0,370,14417,14414],[3185,65.2,370,14417,14414],[3187,81.1,370,14417,14414],[502,90.2,370,14417,14414],[4048,96.5,370,14417,14414]]},{"id":"wse-coverage-centralized-plain-2026-paper","name":"WSE-bench · centralized-plain coverage (2026 paper)","group":"生成与创作","metric":"Generation coverage","unit":"score","public_history":true,"protocol":{"en":"All twelve tested model checkpoints fill every enabled role for the same forty-two EpisodeSeeds under the fixed centralized-plain architecture. The original 0–100 Generation coverage is retained as a native measure. Coverage is observed steps out of twenty planned steps; consistency uses the published Kaplan–Meier C_obs and may carry survival beyond observational support, so it must be read with coverage. Richness combines log-scaled development frequency and breadth, separately from consistency. Complete history is shared; memory adds structured state. Three judges and held-out human validation are described in the paper.","zh":"完整十二模型在固定centralized-plain架构下，对相同42个情节种子承担全部启用角色。保留0–100原生Generation coverage。覆盖率为20个计划叙事步骤中的实测比例；一致性采用原表Kaplan–Meier C_obs，观测耗尽后可延续存活估计，须结合覆盖率解释。丰富度综合对数缩放的发展频次与广度，与一致性分开。所有条件均保留完整历史，记忆模块额外加入结构化状态。原文给出三评审与独立人工验证。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.15654v1","status":"paper_baselines","published_at":"2026-08-16"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":12,"result_count":12,"results":[[14412,6.3,370,14418,14414],[1748,80.2,370,14418,14414],[1659,41.3,370,14418,14414],[1645,100.0,370,14418,14414],[1651,98.6,370,14418,14414],[2576,88.9,370,14418,14414],[5411,98.1,370,14418,14414],[227,93.0,370,14418,14414],[3185,94.9,370,14418,14414],[3187,90.1,370,14418,14414],[502,94.6,370,14418,14414],[4048,100.0,370,14418,14414]]},{"id":"wse-coverage-centralized-memory-2026-paper","name":"WSE-bench · centralized-memory coverage (2026 paper)","group":"生成与创作","metric":"Generation coverage","unit":"score","public_history":true,"protocol":{"en":"All twelve tested model checkpoints fill every enabled role for the same forty-two EpisodeSeeds under the fixed centralized-memory architecture. The original 0–100 Generation coverage is retained as a native measure. Coverage is observed steps out of twenty planned steps; consistency uses the published Kaplan–Meier C_obs and may carry survival beyond observational support, so it must be read with coverage. Richness combines log-scaled development frequency and breadth, separately from consistency. Complete history is shared; memory adds structured state. Three judges and held-out human validation are described in the paper.","zh":"完整十二模型在固定centralized-memory架构下，对相同42个情节种子承担全部启用角色。保留0–100原生Generation coverage。覆盖率为20个计划叙事步骤中的实测比例；一致性采用原表Kaplan–Meier C_obs，观测耗尽后可延续存活估计，须结合覆盖率解释。丰富度综合对数缩放的发展频次与广度，与一致性分开。所有条件均保留完整历史，记忆模块额外加入结构化状态。原文给出三评审与独立人工验证。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.15654v1","status":"paper_baselines","published_at":"2026-08-16"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":12,"result_count":12,"results":[[14412,3.8,370,14419,14414],[1748,29.3,370,14419,14414],[1659,22.1,370,14419,14414],[1645,50.1,370,14419,14414],[1651,81.0,370,14419,14414],[2576,68.6,370,14419,14414],[5411,97.6,370,14419,14414],[227,36.9,370,14419,14414],[3185,67.5,370,14419,14414],[3187,82.4,370,14419,14414],[502,88.9,370,14419,14414],[4048,97.6,370,14419,14414]]},{"id":"wse-coverage-centralized-planning-2026-paper","name":"WSE-bench · centralized-planning coverage (2026 paper)","group":"生成与创作","metric":"Generation coverage","unit":"score","public_history":true,"protocol":{"en":"All twelve tested model checkpoints fill every enabled role for the same forty-two EpisodeSeeds under the fixed centralized-planning architecture. The original 0–100 Generation coverage is retained as a native measure. Coverage is observed steps out of twenty planned steps; consistency uses the published Kaplan–Meier C_obs and may carry survival beyond observational support, so it must be read with coverage. Richness combines log-scaled development frequency and breadth, separately from consistency. Complete history is shared; memory adds structured state. Three judges and held-out human validation are described in the paper.","zh":"完整十二模型在固定centralized-planning架构下，对相同42个情节种子承担全部启用角色。保留0–100原生Generation coverage。覆盖率为20个计划叙事步骤中的实测比例；一致性采用原表Kaplan–Meier C_obs，观测耗尽后可延续存活估计，须结合覆盖率解释。丰富度综合对数缩放的发展频次与广度，与一致性分开。所有条件均保留完整历史，记忆模块额外加入结构化状态。原文给出三评审与独立人工验证。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.15654v1","status":"paper_baselines","published_at":"2026-08-16"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":12,"result_count":12,"results":[[14412,1.8,370,14420,14414],[1748,4.5,370,14420,14414],[1659,5.7,370,14420,14414],[1645,95.8,370,14420,14414],[1651,98.2,370,14420,14414],[2576,93.7,370,14420,14414],[5411,100.0,370,14420,14414],[227,73.7,370,14420,14414],[3185,85.6,370,14420,14414],[3187,83.5,370,14420,14414],[502,93.6,370,14420,14414],[4048,93.5,370,14420,14414]]},{"id":"wse-coverage-centralized-memory-planning-2026-paper","name":"WSE-bench · centralized-memory-planning coverage (2026 paper)","group":"生成与创作","metric":"Generation coverage","unit":"score","public_history":true,"protocol":{"en":"All twelve tested model checkpoints fill every enabled role for the same forty-two EpisodeSeeds under the fixed centralized-memory-planning architecture. The original 0–100 Generation coverage is retained as a native measure. Coverage is observed steps out of twenty planned steps; consistency uses the published Kaplan–Meier C_obs and may carry survival beyond observational support, so it must be read with coverage. Richness combines log-scaled development frequency and breadth, separately from consistency. Complete history is shared; memory adds structured state. Three judges and held-out human validation are described in the paper.","zh":"完整十二模型在固定centralized-memory-planning架构下，对相同42个情节种子承担全部启用角色。保留0–100原生Generation coverage。覆盖率为20个计划叙事步骤中的实测比例；一致性采用原表Kaplan–Meier C_obs，观测耗尽后可延续存活估计，须结合覆盖率解释。丰富度综合对数缩放的发展频次与广度，与一致性分开。所有条件均保留完整历史，记忆模块额外加入结构化状态。原文给出三评审与独立人工验证。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.15654v1","status":"paper_baselines","published_at":"2026-08-16"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":12,"result_count":12,"results":[[14412,4.5,370,14421,14414],[1748,4.6,370,14421,14414],[1659,6.3,370,14421,14414],[1645,61.0,370,14421,14414],[1651,77.1,370,14421,14414],[2576,70.0,370,14421,14414],[5411,95.5,370,14421,14414],[227,42.3,370,14421,14414],[3185,44.3,370,14421,14414],[3187,71.5,370,14421,14414],[502,83.7,370,14421,14414],[4048,98.0,370,14421,14414]]},{"id":"wse-coverage-distributed-plain-2026-paper","name":"WSE-bench · distributed-plain coverage (2026 paper)","group":"生成与创作","metric":"Generation coverage","unit":"score","public_history":true,"protocol":{"en":"All twelve tested model checkpoints fill every enabled role for the same forty-two EpisodeSeeds under the fixed distributed-plain architecture. The original 0–100 Generation coverage is retained as a native measure. Coverage is observed steps out of twenty planned steps; consistency uses the published Kaplan–Meier C_obs and may carry survival beyond observational support, so it must be read with coverage. Richness combines log-scaled development frequency and breadth, separately from consistency. Complete history is shared; memory adds structured state. Three judges and held-out human validation are described in the paper.","zh":"完整十二模型在固定distributed-plain架构下，对相同42个情节种子承担全部启用角色。保留0–100原生Generation coverage。覆盖率为20个计划叙事步骤中的实测比例；一致性采用原表Kaplan–Meier C_obs，观测耗尽后可延续存活估计，须结合覆盖率解释。丰富度综合对数缩放的发展频次与广度，与一致性分开。所有条件均保留完整历史，记忆模块额外加入结构化状态。原文给出三评审与独立人工验证。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.15654v1","status":"paper_baselines","published_at":"2026-08-16"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":12,"result_count":12,"results":[[14412,2.0,370,14422,14414],[1748,68.7,370,14422,14414],[1659,39.9,370,14422,14414],[1645,98.3,370,14422,14414],[1651,98.3,370,14422,14414],[2576,91.9,370,14422,14414],[5411,98.2,370,14422,14414],[227,96.8,370,14422,14414],[3185,89.4,370,14422,14414],[3187,67.4,370,14422,14414],[502,80.4,370,14422,14414],[4048,100.0,370,14422,14414]]},{"id":"wse-coverage-distributed-memory-2026-paper","name":"WSE-bench · distributed-memory coverage (2026 paper)","group":"生成与创作","metric":"Generation coverage","unit":"score","public_history":true,"protocol":{"en":"All twelve tested model checkpoints fill every enabled role for the same forty-two EpisodeSeeds under the fixed distributed-memory architecture. The original 0–100 Generation coverage is retained as a native measure. Coverage is observed steps out of twenty planned steps; consistency uses the published Kaplan–Meier C_obs and may carry survival beyond observational support, so it must be read with coverage. Richness combines log-scaled development frequency and breadth, separately from consistency. Complete history is shared; memory adds structured state. Three judges and held-out human validation are described in the paper.","zh":"完整十二模型在固定distributed-memory架构下，对相同42个情节种子承担全部启用角色。保留0–100原生Generation coverage。覆盖率为20个计划叙事步骤中的实测比例；一致性采用原表Kaplan–Meier C_obs，观测耗尽后可延续存活估计，须结合覆盖率解释。丰富度综合对数缩放的发展频次与广度，与一致性分开。所有条件均保留完整历史，记忆模块额外加入结构化状态。原文给出三评审与独立人工验证。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.15654v1","status":"paper_baselines","published_at":"2026-08-16"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":12,"result_count":12,"results":[[14412,2.6,370,14423,14414],[1748,27.7,370,14423,14414],[1659,24.2,370,14423,14414],[1645,53.0,370,14423,14414],[1651,72.9,370,14423,14414],[2576,72.6,370,14423,14414],[5411,94.5,370,14423,14414],[227,34.5,370,14423,14414],[3185,67.1,370,14423,14414],[3187,69.9,370,14423,14414],[502,91.2,370,14423,14414],[4048,95.6,370,14423,14414]]},{"id":"wse-coverage-distributed-planning-2026-paper","name":"WSE-bench · distributed-planning coverage (2026 paper)","group":"生成与创作","metric":"Generation coverage","unit":"score","public_history":true,"protocol":{"en":"All twelve tested model checkpoints fill every enabled role for the same forty-two EpisodeSeeds under the fixed distributed-planning architecture. The original 0–100 Generation coverage is retained as a native measure. Coverage is observed steps out of twenty planned steps; consistency uses the published Kaplan–Meier C_obs and may carry survival beyond observational support, so it must be read with coverage. Richness combines log-scaled development frequency and breadth, separately from consistency. Complete history is shared; memory adds structured state. Three judges and held-out human validation are described in the paper.","zh":"完整十二模型在固定distributed-planning架构下，对相同42个情节种子承担全部启用角色。保留0–100原生Generation coverage。覆盖率为20个计划叙事步骤中的实测比例；一致性采用原表Kaplan–Meier C_obs，观测耗尽后可延续存活估计，须结合覆盖率解释。丰富度综合对数缩放的发展频次与广度，与一致性分开。所有条件均保留完整历史，记忆模块额外加入结构化状态。原文给出三评审与独立人工验证。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.15654v1","status":"paper_baselines","published_at":"2026-08-16"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":12,"result_count":12,"results":[[14412,1.1,370,14424,14414],[1748,4.6,370,14424,14414],[1659,6.3,370,14424,14414],[1645,95.2,370,14424,14414],[1651,97.5,370,14424,14414],[2576,84.2,370,14424,14414],[5411,98.2,370,14424,14414],[227,88.8,370,14424,14414],[3185,75.2,370,14424,14414],[3187,54.2,370,14424,14414],[502,81.2,370,14424,14414],[4048,97.4,370,14424,14414]]},{"id":"wse-coverage-distributed-memory-planning-2026-paper","name":"WSE-bench · distributed-memory-planning coverage (2026 paper)","group":"生成与创作","metric":"Generation coverage","unit":"score","public_history":true,"protocol":{"en":"All twelve tested model checkpoints fill every enabled role for the same forty-two EpisodeSeeds under the fixed distributed-memory-planning architecture. The original 0–100 Generation coverage is retained as a native measure. Coverage is observed steps out of twenty planned steps; consistency uses the published Kaplan–Meier C_obs and may carry survival beyond observational support, so it must be read with coverage. Richness combines log-scaled development frequency and breadth, separately from consistency. Complete history is shared; memory adds structured state. Three judges and held-out human validation are described in the paper.","zh":"完整十二模型在固定distributed-memory-planning架构下，对相同42个情节种子承担全部启用角色。保留0–100原生Generation coverage。覆盖率为20个计划叙事步骤中的实测比例；一致性采用原表Kaplan–Meier C_obs，观测耗尽后可延续存活估计，须结合覆盖率解释。丰富度综合对数缩放的发展频次与广度，与一致性分开。所有条件均保留完整历史，记忆模块额外加入结构化状态。原文给出三评审与独立人工验证。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.15654v1","status":"paper_baselines","published_at":"2026-08-16"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":12,"result_count":12,"results":[[14412,3.5,370,14425,14414],[1748,4.9,370,14425,14414],[1659,6.8,370,14425,14414],[1645,54.8,370,14425,14414],[1651,83.0,370,14425,14414],[2576,55.2,370,14425,14414],[5411,95.7,370,14425,14414],[227,45.2,370,14425,14414],[3185,56.7,370,14425,14414],[3187,62.9,370,14425,14414],[502,89.5,370,14425,14414],[4048,94.9,370,14425,14414]]},{"id":"wse-consistency-story-plain-2026-paper","name":"WSE-bench · story-plain consistency (2026 paper)","group":"生成与创作","metric":"Observed consistency C_obs","unit":"score","public_history":true,"protocol":{"en":"All twelve tested model checkpoints fill every enabled role for the same forty-two EpisodeSeeds under the fixed story-plain architecture. The original 0–100 Observed consistency C_obs is retained as a native measure. Coverage is observed steps out of twenty planned steps; consistency uses the published Kaplan–Meier C_obs and may carry survival beyond observational support, so it must be read with coverage. Richness combines log-scaled development frequency and breadth, separately from consistency. Complete history is shared; memory adds structured state. Three judges and held-out human validation are described in the paper.","zh":"完整十二模型在固定story-plain架构下，对相同42个情节种子承担全部启用角色。保留0–100原生Observed consistency C_obs。覆盖率为20个计划叙事步骤中的实测比例；一致性采用原表Kaplan–Meier C_obs，观测耗尽后可延续存活估计，须结合覆盖率解释。丰富度综合对数缩放的发展频次与广度，与一致性分开。所有条件均保留完整历史，记忆模块额外加入结构化状态。原文给出三评审与独立人工验证。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.15654v1","status":"paper_baselines","published_at":"2026-08-16"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":12,"result_count":12,"results":[[14412,100.0,370,14426,14414],[1748,62.9,370,14426,14414],[1659,53.8,370,14426,14414],[1645,90.9,370,14426,14414],[1651,93.3,370,14426,14414],[2576,93.7,370,14426,14414],[5411,97.3,370,14426,14414],[227,85.8,370,14426,14414],[3185,90.8,370,14426,14414],[3187,93.2,370,14426,14414],[502,98.6,370,14426,14414],[4048,100.0,370,14426,14414]]},{"id":"wse-consistency-story-memory-2026-paper","name":"WSE-bench · story-memory consistency (2026 paper)","group":"生成与创作","metric":"Observed consistency C_obs","unit":"score","public_history":true,"protocol":{"en":"All twelve tested model checkpoints fill every enabled role for the same forty-two EpisodeSeeds under the fixed story-memory architecture. The original 0–100 Observed consistency C_obs is retained as a native measure. Coverage is observed steps out of twenty planned steps; consistency uses the published Kaplan–Meier C_obs and may carry survival beyond observational support, so it must be read with coverage. Richness combines log-scaled development frequency and breadth, separately from consistency. Complete history is shared; memory adds structured state. Three judges and held-out human validation are described in the paper.","zh":"完整十二模型在固定story-memory架构下，对相同42个情节种子承担全部启用角色。保留0–100原生Observed consistency C_obs。覆盖率为20个计划叙事步骤中的实测比例；一致性采用原表Kaplan–Meier C_obs，观测耗尽后可延续存活估计，须结合覆盖率解释。丰富度综合对数缩放的发展频次与广度，与一致性分开。所有条件均保留完整历史，记忆模块额外加入结构化状态。原文给出三评审与独立人工验证。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.15654v1","status":"paper_baselines","published_at":"2026-08-16"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":12,"result_count":12,"results":[[14412,100.0,370,14427,14414],[1748,63.8,370,14427,14414],[1659,77.2,370,14427,14414],[1645,91.3,370,14427,14414],[1651,92.3,370,14427,14414],[2576,92.1,370,14427,14414],[5411,97.5,370,14427,14414],[227,96.8,370,14427,14414],[3185,95.4,370,14427,14414],[3187,97.6,370,14427,14414],[502,98.0,370,14427,14414],[4048,98.4,370,14427,14414]]},{"id":"wse-consistency-story-planning-2026-paper","name":"WSE-bench · story-planning consistency (2026 paper)","group":"生成与创作","metric":"Observed consistency C_obs","unit":"score","public_history":true,"protocol":{"en":"All twelve tested model checkpoints fill every enabled role for the same forty-two EpisodeSeeds under the fixed story-planning architecture. The original 0–100 Observed consistency C_obs is retained as a native measure. Coverage is observed steps out of twenty planned steps; consistency uses the published Kaplan–Meier C_obs and may carry survival beyond observational support, so it must be read with coverage. Richness combines log-scaled development frequency and breadth, separately from consistency. Complete history is shared; memory adds structured state. Three judges and held-out human validation are described in the paper.","zh":"完整十二模型在固定story-planning架构下，对相同42个情节种子承担全部启用角色。保留0–100原生Observed consistency C_obs。覆盖率为20个计划叙事步骤中的实测比例；一致性采用原表Kaplan–Meier C_obs，观测耗尽后可延续存活估计，须结合覆盖率解释。丰富度综合对数缩放的发展频次与广度，与一致性分开。所有条件均保留完整历史，记忆模块额外加入结构化状态。原文给出三评审与独立人工验证。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.15654v1","status":"paper_baselines","published_at":"2026-08-16"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":12,"result_count":12,"results":[[14412,100.0,370,14428,14414],[1748,92.5,370,14428,14414],[1659,100.0,370,14428,14414],[1645,95.8,370,14428,14414],[1651,88.9,370,14428,14414],[2576,94.0,370,14428,14414],[5411,92.4,370,14428,14414],[227,77.5,370,14428,14414],[3185,82.7,370,14428,14414],[3187,89.4,370,14428,14414],[502,95.3,370,14428,14414],[4048,98.4,370,14428,14414]]},{"id":"wse-consistency-story-memory-planning-2026-paper","name":"WSE-bench · story-memory-planning consistency (2026 paper)","group":"生成与创作","metric":"Observed consistency C_obs","unit":"score","public_history":true,"protocol":{"en":"All twelve tested model checkpoints fill every enabled role for the same forty-two EpisodeSeeds under the fixed story-memory-planning architecture. The original 0–100 Observed consistency C_obs is retained as a native measure. Coverage is observed steps out of twenty planned steps; consistency uses the published Kaplan–Meier C_obs and may carry survival beyond observational support, so it must be read with coverage. Richness combines log-scaled development frequency and breadth, separately from consistency. Complete history is shared; memory adds structured state. Three judges and held-out human validation are described in the paper.","zh":"完整十二模型在固定story-memory-planning架构下，对相同42个情节种子承担全部启用角色。保留0–100原生Observed consistency C_obs。覆盖率为20个计划叙事步骤中的实测比例；一致性采用原表Kaplan–Meier C_obs，观测耗尽后可延续存活估计，须结合覆盖率解释。丰富度综合对数缩放的发展频次与广度，与一致性分开。所有条件均保留完整历史，记忆模块额外加入结构化状态。原文给出三评审与独立人工验证。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.15654v1","status":"paper_baselines","published_at":"2026-08-16"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":12,"result_count":12,"results":[[14412,76.2,370,14429,14414],[1748,97.5,370,14429,14414],[1659,52.5,370,14429,14414],[1645,88.8,370,14429,14414],[1651,88.2,370,14429,14414],[2576,90.8,370,14429,14414],[5411,88.1,370,14429,14414],[227,78.2,370,14429,14414],[3185,73.6,370,14429,14414],[3187,91.3,370,14429,14414],[502,97.8,370,14429,14414],[4048,98.1,370,14429,14414]]},{"id":"wse-consistency-centralized-plain-2026-paper","name":"WSE-bench · centralized-plain consistency (2026 paper)","group":"生成与创作","metric":"Observed consistency C_obs","unit":"score","public_history":true,"protocol":{"en":"All twelve tested model checkpoints fill every enabled role for the same forty-two EpisodeSeeds under the fixed centralized-plain architecture. The original 0–100 Observed consistency C_obs is retained as a native measure. Coverage is observed steps out of twenty planned steps; consistency uses the published Kaplan–Meier C_obs and may carry survival beyond observational support, so it must be read with coverage. Richness combines log-scaled development frequency and breadth, separately from consistency. Complete history is shared; memory adds structured state. Three judges and held-out human validation are described in the paper.","zh":"完整十二模型在固定centralized-plain架构下，对相同42个情节种子承担全部启用角色。保留0–100原生Observed consistency C_obs。覆盖率为20个计划叙事步骤中的实测比例；一致性采用原表Kaplan–Meier C_obs，观测耗尽后可延续存活估计，须结合覆盖率解释。丰富度综合对数缩放的发展频次与广度，与一致性分开。所有条件均保留完整历史，记忆模块额外加入结构化状态。原文给出三评审与独立人工验证。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.15654v1","status":"paper_baselines","published_at":"2026-08-16"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":12,"result_count":12,"results":[[14412,69.8,370,14430,14414],[1748,74.2,370,14430,14414],[1659,55.3,370,14430,14414],[1645,85.1,370,14430,14414],[1651,87.9,370,14430,14414],[2576,87.1,370,14430,14414],[5411,86.3,370,14430,14414],[227,78.5,370,14430,14414],[3185,92.8,370,14430,14414],[3187,91.3,370,14430,14414],[502,100.0,370,14430,14414],[4048,99.6,370,14430,14414]]},{"id":"wse-consistency-centralized-memory-2026-paper","name":"WSE-bench · centralized-memory consistency (2026 paper)","group":"生成与创作","metric":"Observed consistency C_obs","unit":"score","public_history":true,"protocol":{"en":"All twelve tested model checkpoints fill every enabled role for the same forty-two EpisodeSeeds under the fixed centralized-memory architecture. The original 0–100 Observed consistency C_obs is retained as a native measure. Coverage is observed steps out of twenty planned steps; consistency uses the published Kaplan–Meier C_obs and may carry survival beyond observational support, so it must be read with coverage. Richness combines log-scaled development frequency and breadth, separately from consistency. Complete history is shared; memory adds structured state. Three judges and held-out human validation are described in the paper.","zh":"完整十二模型在固定centralized-memory架构下，对相同42个情节种子承担全部启用角色。保留0–100原生Observed consistency C_obs。覆盖率为20个计划叙事步骤中的实测比例；一致性采用原表Kaplan–Meier C_obs，观测耗尽后可延续存活估计，须结合覆盖率解释。丰富度综合对数缩放的发展频次与广度，与一致性分开。所有条件均保留完整历史，记忆模块额外加入结构化状态。原文给出三评审与独立人工验证。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.15654v1","status":"paper_baselines","published_at":"2026-08-16"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":12,"result_count":12,"results":[[14412,48.6,370,14431,14414],[1748,73.0,370,14431,14414],[1659,74.1,370,14431,14414],[1645,86.9,370,14431,14414],[1651,91.8,370,14431,14414],[2576,84.7,370,14431,14414],[5411,90.1,370,14431,14414],[227,78.8,370,14431,14414],[3185,84.5,370,14431,14414],[3187,89.5,370,14431,14414],[502,98.6,370,14431,14414],[4048,97.4,370,14431,14414]]},{"id":"wse-consistency-centralized-planning-2026-paper","name":"WSE-bench · centralized-planning consistency (2026 paper)","group":"生成与创作","metric":"Observed consistency C_obs","unit":"score","public_history":true,"protocol":{"en":"All twelve tested model checkpoints fill every enabled role for the same forty-two EpisodeSeeds under the fixed centralized-planning architecture. The original 0–100 Observed consistency C_obs is retained as a native measure. Coverage is observed steps out of twenty planned steps; consistency uses the published Kaplan–Meier C_obs and may carry survival beyond observational support, so it must be read with coverage. Richness combines log-scaled development frequency and breadth, separately from consistency. Complete history is shared; memory adds structured state. Three judges and held-out human validation are described in the paper.","zh":"完整十二模型在固定centralized-planning架构下，对相同42个情节种子承担全部启用角色。保留0–100原生Observed consistency C_obs。覆盖率为20个计划叙事步骤中的实测比例；一致性采用原表Kaplan–Meier C_obs，观测耗尽后可延续存活估计，须结合覆盖率解释。丰富度综合对数缩放的发展频次与广度，与一致性分开。所有条件均保留完整历史，记忆模块额外加入结构化状态。原文给出三评审与独立人工验证。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.15654v1","status":"paper_baselines","published_at":"2026-08-16"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":12,"result_count":12,"results":[[14412,68.3,370,14432,14414],[1748,100.0,370,14432,14414],[1659,95.1,370,14432,14414],[1645,84.2,370,14432,14414],[1651,84.4,370,14432,14414],[2576,85.9,370,14432,14414],[5411,91.0,370,14432,14414],[227,82.0,370,14432,14414],[3185,68.5,370,14432,14414],[3187,89.8,370,14432,14414],[502,100.0,370,14432,14414],[4048,99.8,370,14432,14414]]},{"id":"wse-consistency-centralized-memory-planning-2026-paper","name":"WSE-bench · centralized-memory-planning consistency (2026 paper)","group":"生成与创作","metric":"Observed consistency C_obs","unit":"score","public_history":true,"protocol":{"en":"All twelve tested model checkpoints fill every enabled role for the same forty-two EpisodeSeeds under the fixed centralized-memory-planning architecture. The original 0–100 Observed consistency C_obs is retained as a native measure. Coverage is observed steps out of twenty planned steps; consistency uses the published Kaplan–Meier C_obs and may carry survival beyond observational support, so it must be read with coverage. Richness combines log-scaled development frequency and breadth, separately from consistency. Complete history is shared; memory adds structured state. Three judges and held-out human validation are described in the paper.","zh":"完整十二模型在固定centralized-memory-planning架构下，对相同42个情节种子承担全部启用角色。保留0–100原生Observed consistency C_obs。覆盖率为20个计划叙事步骤中的实测比例；一致性采用原表Kaplan–Meier C_obs，观测耗尽后可延续存活估计，须结合覆盖率解释。丰富度综合对数缩放的发展频次与广度，与一致性分开。所有条件均保留完整历史，记忆模块额外加入结构化状态。原文给出三评审与独立人工验证。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.15654v1","status":"paper_baselines","published_at":"2026-08-16"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":12,"result_count":12,"results":[[14412,39.3,370,14433,14414],[1748,100.0,370,14433,14414],[1659,47.0,370,14433,14414],[1645,76.0,370,14433,14414],[1651,85.1,370,14433,14414],[2576,72.1,370,14433,14414],[5411,88.3,370,14433,14414],[227,55.5,370,14433,14414],[3185,79.4,370,14433,14414],[3187,95.5,370,14433,14414],[502,96.9,370,14433,14414],[4048,100.0,370,14433,14414]]},{"id":"wse-consistency-distributed-plain-2026-paper","name":"WSE-bench · distributed-plain consistency (2026 paper)","group":"生成与创作","metric":"Observed consistency C_obs","unit":"score","public_history":true,"protocol":{"en":"All twelve tested model checkpoints fill every enabled role for the same forty-two EpisodeSeeds under the fixed distributed-plain architecture. The original 0–100 Observed consistency C_obs is retained as a native measure. Coverage is observed steps out of twenty planned steps; consistency uses the published Kaplan–Meier C_obs and may carry survival beyond observational support, so it must be read with coverage. Richness combines log-scaled development frequency and breadth, separately from consistency. Complete history is shared; memory adds structured state. Three judges and held-out human validation are described in the paper.","zh":"完整十二模型在固定distributed-plain架构下，对相同42个情节种子承担全部启用角色。保留0–100原生Observed consistency C_obs。覆盖率为20个计划叙事步骤中的实测比例；一致性采用原表Kaplan–Meier C_obs，观测耗尽后可延续存活估计，须结合覆盖率解释。丰富度综合对数缩放的发展频次与广度，与一致性分开。所有条件均保留完整历史，记忆模块额外加入结构化状态。原文给出三评审与独立人工验证。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.15654v1","status":"paper_baselines","published_at":"2026-08-16"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":12,"result_count":12,"results":[[14412,57.5,370,14434,14414],[1748,75.4,370,14434,14414],[1659,56.6,370,14434,14414],[1645,76.0,370,14434,14414],[1651,88.5,370,14434,14414],[2576,87.5,370,14434,14414],[5411,89.4,370,14434,14414],[227,82.4,370,14434,14414],[3185,86.3,370,14434,14414],[3187,93.2,370,14434,14414],[502,88.8,370,14434,14414],[4048,98.3,370,14434,14414]]},{"id":"wse-consistency-distributed-memory-2026-paper","name":"WSE-bench · distributed-memory consistency (2026 paper)","group":"生成与创作","metric":"Observed consistency C_obs","unit":"score","public_history":true,"protocol":{"en":"All twelve tested model checkpoints fill every enabled role for the same forty-two EpisodeSeeds under the fixed distributed-memory architecture. The original 0–100 Observed consistency C_obs is retained as a native measure. Coverage is observed steps out of twenty planned steps; consistency uses the published Kaplan–Meier C_obs and may carry survival beyond observational support, so it must be read with coverage. Richness combines log-scaled development frequency and breadth, separately from consistency. Complete history is shared; memory adds structured state. Three judges and held-out human validation are described in the paper.","zh":"完整十二模型在固定distributed-memory架构下，对相同42个情节种子承担全部启用角色。保留0–100原生Observed consistency C_obs。覆盖率为20个计划叙事步骤中的实测比例；一致性采用原表Kaplan–Meier C_obs，观测耗尽后可延续存活估计，须结合覆盖率解释。丰富度综合对数缩放的发展频次与广度，与一致性分开。所有条件均保留完整历史，记忆模块额外加入结构化状态。原文给出三评审与独立人工验证。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.15654v1","status":"paper_baselines","published_at":"2026-08-16"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":12,"result_count":12,"results":[[14412,39.1,370,14435,14414],[1748,79.6,370,14435,14414],[1659,37.1,370,14435,14414],[1645,80.6,370,14435,14414],[1651,84.2,370,14435,14414],[2576,84.1,370,14435,14414],[5411,91.2,370,14435,14414],[227,95.4,370,14435,14414],[3185,85.4,370,14435,14414],[3187,96.6,370,14435,14414],[502,97.7,370,14435,14414],[4048,100.0,370,14435,14414]]},{"id":"wse-consistency-distributed-planning-2026-paper","name":"WSE-bench · distributed-planning consistency (2026 paper)","group":"生成与创作","metric":"Observed consistency C_obs","unit":"score","public_history":true,"protocol":{"en":"All twelve tested model checkpoints fill every enabled role for the same forty-two EpisodeSeeds under the fixed distributed-planning architecture. The original 0–100 Observed consistency C_obs is retained as a native measure. Coverage is observed steps out of twenty planned steps; consistency uses the published Kaplan–Meier C_obs and may carry survival beyond observational support, so it must be read with coverage. Richness combines log-scaled development frequency and breadth, separately from consistency. Complete history is shared; memory adds structured state. Three judges and held-out human validation are described in the paper.","zh":"完整十二模型在固定distributed-planning架构下，对相同42个情节种子承担全部启用角色。保留0–100原生Observed consistency C_obs。覆盖率为20个计划叙事步骤中的实测比例；一致性采用原表Kaplan–Meier C_obs，观测耗尽后可延续存活估计，须结合覆盖率解释。丰富度综合对数缩放的发展频次与广度，与一致性分开。所有条件均保留完整历史，记忆模块额外加入结构化状态。原文给出三评审与独立人工验证。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.15654v1","status":"paper_baselines","published_at":"2026-08-16"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":12,"result_count":12,"results":[[14412,43.8,370,14436,14414],[1748,100.0,370,14436,14414],[1659,88.3,370,14436,14414],[1645,80.4,370,14436,14414],[1651,84.5,370,14436,14414],[2576,79.6,370,14436,14414],[5411,87.9,370,14436,14414],[227,74.8,370,14436,14414],[3185,78.0,370,14436,14414],[3187,85.0,370,14436,14414],[502,95.8,370,14436,14414],[4048,100.0,370,14436,14414]]},{"id":"wse-consistency-distributed-memory-planning-2026-paper","name":"WSE-bench · distributed-memory-planning consistency (2026 paper)","group":"生成与创作","metric":"Observed consistency C_obs","unit":"score","public_history":true,"protocol":{"en":"All twelve tested model checkpoints fill every enabled role for the same forty-two EpisodeSeeds under the fixed distributed-memory-planning architecture. The original 0–100 Observed consistency C_obs is retained as a native measure. Coverage is observed steps out of twenty planned steps; consistency uses the published Kaplan–Meier C_obs and may carry survival beyond observational support, so it must be read with coverage. Richness combines log-scaled development frequency and breadth, separately from consistency. Complete history is shared; memory adds structured state. Three judges and held-out human validation are described in the paper.","zh":"完整十二模型在固定distributed-memory-planning架构下，对相同42个情节种子承担全部启用角色。保留0–100原生Observed consistency C_obs。覆盖率为20个计划叙事步骤中的实测比例；一致性采用原表Kaplan–Meier C_obs，观测耗尽后可延续存活估计，须结合覆盖率解释。丰富度综合对数缩放的发展频次与广度，与一致性分开。所有条件均保留完整历史，记忆模块额外加入结构化状态。原文给出三评审与独立人工验证。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.15654v1","status":"paper_baselines","published_at":"2026-08-16"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":12,"result_count":12,"results":[[14412,80.3,370,14437,14414],[1748,94.7,370,14437,14414],[1659,60.1,370,14437,14414],[1645,77.2,370,14437,14414],[1651,79.0,370,14437,14414],[2576,74.6,370,14437,14414],[5411,83.6,370,14437,14414],[227,70.7,370,14437,14414],[3185,82.0,370,14437,14414],[3187,85.6,370,14437,14414],[502,100.0,370,14437,14414],[4048,98.3,370,14437,14414]]},{"id":"wse-richness-story-plain-2026-paper","name":"WSE-bench · story-plain richness (2026 paper)","group":"生成与创作","metric":"Narrative richness","unit":"score","public_history":true,"protocol":{"en":"All twelve tested model checkpoints fill every enabled role for the same forty-two EpisodeSeeds under the fixed story-plain architecture. The original 0–100 Narrative richness is retained as a native measure. Coverage is observed steps out of twenty planned steps; consistency uses the published Kaplan–Meier C_obs and may carry survival beyond observational support, so it must be read with coverage. Richness combines log-scaled development frequency and breadth, separately from consistency. Complete history is shared; memory adds structured state. Three judges and held-out human validation are described in the paper.","zh":"完整十二模型在固定story-plain架构下，对相同42个情节种子承担全部启用角色。保留0–100原生Narrative richness。覆盖率为20个计划叙事步骤中的实测比例；一致性采用原表Kaplan–Meier C_obs，观测耗尽后可延续存活估计，须结合覆盖率解释。丰富度综合对数缩放的发展频次与广度，与一致性分开。所有条件均保留完整历史，记忆模块额外加入结构化状态。原文给出三评审与独立人工验证。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.15654v1","status":"paper_baselines","published_at":"2026-08-16"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":12,"result_count":12,"results":[[14412,15.9,370,14438,14414],[1748,27.1,370,14438,14414],[1659,40.2,370,14438,14414],[1645,34.8,370,14438,14414],[1651,35.0,370,14438,14414],[2576,44.8,370,14438,14414],[5411,35.5,370,14438,14414],[227,56.6,370,14438,14414],[3185,57.7,370,14438,14414],[3187,56.4,370,14438,14414],[502,54.6,370,14438,14414],[4048,60.5,370,14438,14414]]},{"id":"wse-richness-story-memory-2026-paper","name":"WSE-bench · story-memory richness (2026 paper)","group":"生成与创作","metric":"Narrative richness","unit":"score","public_history":true,"protocol":{"en":"All twelve tested model checkpoints fill every enabled role for the same forty-two EpisodeSeeds under the fixed story-memory architecture. The original 0–100 Narrative richness is retained as a native measure. Coverage is observed steps out of twenty planned steps; consistency uses the published Kaplan–Meier C_obs and may carry survival beyond observational support, so it must be read with coverage. Richness combines log-scaled development frequency and breadth, separately from consistency. Complete history is shared; memory adds structured state. Three judges and held-out human validation are described in the paper.","zh":"完整十二模型在固定story-memory架构下，对相同42个情节种子承担全部启用角色。保留0–100原生Narrative richness。覆盖率为20个计划叙事步骤中的实测比例；一致性采用原表Kaplan–Meier C_obs，观测耗尽后可延续存活估计，须结合覆盖率解释。丰富度综合对数缩放的发展频次与广度，与一致性分开。所有条件均保留完整历史，记忆模块额外加入结构化状态。原文给出三评审与独立人工验证。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.15654v1","status":"paper_baselines","published_at":"2026-08-16"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":12,"result_count":12,"results":[[14412,10.7,370,14439,14414],[1748,22.8,370,14439,14414],[1659,35.4,370,14439,14414],[1645,33.9,370,14439,14414],[1651,36.4,370,14439,14414],[2576,43.9,370,14439,14414],[5411,33.2,370,14439,14414],[227,49.5,370,14439,14414],[3185,53.8,370,14439,14414],[3187,57.7,370,14439,14414],[502,56.1,370,14439,14414],[4048,63.7,370,14439,14414]]},{"id":"wse-richness-story-planning-2026-paper","name":"WSE-bench · story-planning richness (2026 paper)","group":"生成与创作","metric":"Narrative richness","unit":"score","public_history":true,"protocol":{"en":"All twelve tested model checkpoints fill every enabled role for the same forty-two EpisodeSeeds under the fixed story-planning architecture. The original 0–100 Narrative richness is retained as a native measure. Coverage is observed steps out of twenty planned steps; consistency uses the published Kaplan–Meier C_obs and may carry survival beyond observational support, so it must be read with coverage. Richness combines log-scaled development frequency and breadth, separately from consistency. Complete history is shared; memory adds structured state. Three judges and held-out human validation are described in the paper.","zh":"完整十二模型在固定story-planning架构下，对相同42个情节种子承担全部启用角色。保留0–100原生Narrative richness。覆盖率为20个计划叙事步骤中的实测比例；一致性采用原表Kaplan–Meier C_obs，观测耗尽后可延续存活估计，须结合覆盖率解释。丰富度综合对数缩放的发展频次与广度，与一致性分开。所有条件均保留完整历史，记忆模块额外加入结构化状态。原文给出三评审与独立人工验证。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.15654v1","status":"paper_baselines","published_at":"2026-08-16"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":12,"result_count":12,"results":[[14412,9.1,370,14440,14414],[1748,8.6,370,14440,14414],[1659,16.0,370,14440,14414],[1645,33.5,370,14440,14414],[1651,34.0,370,14440,14414],[2576,46.6,370,14440,14414],[5411,33.0,370,14440,14414],[227,64.9,370,14440,14414],[3185,64.0,370,14440,14414],[3187,59.3,370,14440,14414],[502,53.6,370,14440,14414],[4048,49.4,370,14440,14414]]},{"id":"wse-richness-story-memory-planning-2026-paper","name":"WSE-bench · story-memory-planning richness (2026 paper)","group":"生成与创作","metric":"Narrative richness","unit":"score","public_history":true,"protocol":{"en":"All twelve tested model checkpoints fill every enabled role for the same forty-two EpisodeSeeds under the fixed story-memory-planning architecture. The original 0–100 Narrative richness is retained as a native measure. Coverage is observed steps out of twenty planned steps; consistency uses the published Kaplan–Meier C_obs and may carry survival beyond observational support, so it must be read with coverage. Richness combines log-scaled development frequency and breadth, separately from consistency. Complete history is shared; memory adds structured state. Three judges and held-out human validation are described in the paper.","zh":"完整十二模型在固定story-memory-planning架构下，对相同42个情节种子承担全部启用角色。保留0–100原生Narrative richness。覆盖率为20个计划叙事步骤中的实测比例；一致性采用原表Kaplan–Meier C_obs，观测耗尽后可延续存活估计，须结合覆盖率解释。丰富度综合对数缩放的发展频次与广度，与一致性分开。所有条件均保留完整历史，记忆模块额外加入结构化状态。原文给出三评审与独立人工验证。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.15654v1","status":"paper_baselines","published_at":"2026-08-16"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":12,"result_count":12,"results":[[14412,12.0,370,14441,14414],[1748,6.6,370,14441,14414],[1659,13.1,370,14441,14414],[1645,29.7,370,14441,14414],[1651,32.9,370,14441,14414],[2576,45.4,370,14441,14414],[5411,31.7,370,14441,14414],[227,57.7,370,14441,14414],[3185,64.0,370,14441,14414],[3187,58.8,370,14441,14414],[502,54.1,370,14441,14414],[4048,50.3,370,14441,14414]]},{"id":"wse-richness-centralized-plain-2026-paper","name":"WSE-bench · centralized-plain richness (2026 paper)","group":"生成与创作","metric":"Narrative richness","unit":"score","public_history":true,"protocol":{"en":"All twelve tested model checkpoints fill every enabled role for the same forty-two EpisodeSeeds under the fixed centralized-plain architecture. The original 0–100 Narrative richness is retained as a native measure. Coverage is observed steps out of twenty planned steps; consistency uses the published Kaplan–Meier C_obs and may carry survival beyond observational support, so it must be read with coverage. Richness combines log-scaled development frequency and breadth, separately from consistency. Complete history is shared; memory adds structured state. Three judges and held-out human validation are described in the paper.","zh":"完整十二模型在固定centralized-plain架构下，对相同42个情节种子承担全部启用角色。保留0–100原生Narrative richness。覆盖率为20个计划叙事步骤中的实测比例；一致性采用原表Kaplan–Meier C_obs，观测耗尽后可延续存活估计，须结合覆盖率解释。丰富度综合对数缩放的发展频次与广度，与一致性分开。所有条件均保留完整历史，记忆模块额外加入结构化状态。原文给出三评审与独立人工验证。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.15654v1","status":"paper_baselines","published_at":"2026-08-16"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":12,"result_count":12,"results":[[14412,13.4,370,14442,14414],[1748,22.7,370,14442,14414],[1659,38.7,370,14442,14414],[1645,39.9,370,14442,14414],[1651,39.0,370,14442,14414],[2576,55.4,370,14442,14414],[5411,39.6,370,14442,14414],[227,66.7,370,14442,14414],[3185,66.9,370,14442,14414],[3187,66.1,370,14442,14414],[502,66.5,370,14442,14414],[4048,67.7,370,14442,14414]]},{"id":"wse-richness-centralized-memory-2026-paper","name":"WSE-bench · centralized-memory richness (2026 paper)","group":"生成与创作","metric":"Narrative richness","unit":"score","public_history":true,"protocol":{"en":"All twelve tested model checkpoints fill every enabled role for the same forty-two EpisodeSeeds under the fixed centralized-memory architecture. The original 0–100 Narrative richness is retained as a native measure. Coverage is observed steps out of twenty planned steps; consistency uses the published Kaplan–Meier C_obs and may carry survival beyond observational support, so it must be read with coverage. Richness combines log-scaled development frequency and breadth, separately from consistency. Complete history is shared; memory adds structured state. Three judges and held-out human validation are described in the paper.","zh":"完整十二模型在固定centralized-memory架构下，对相同42个情节种子承担全部启用角色。保留0–100原生Narrative richness。覆盖率为20个计划叙事步骤中的实测比例；一致性采用原表Kaplan–Meier C_obs，观测耗尽后可延续存活估计，须结合覆盖率解释。丰富度综合对数缩放的发展频次与广度，与一致性分开。所有条件均保留完整历史，记忆模块额外加入结构化状态。原文给出三评审与独立人工验证。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.15654v1","status":"paper_baselines","published_at":"2026-08-16"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":12,"result_count":12,"results":[[14412,20.5,370,14443,14414],[1748,25.1,370,14443,14414],[1659,36.5,370,14443,14414],[1645,34.8,370,14443,14414],[1651,33.9,370,14443,14414],[2576,51.9,370,14443,14414],[5411,37.8,370,14443,14414],[227,58.5,370,14443,14414],[3185,59.7,370,14443,14414],[3187,59.9,370,14443,14414],[502,63.8,370,14443,14414],[4048,68.7,370,14443,14414]]},{"id":"wse-richness-centralized-planning-2026-paper","name":"WSE-bench · centralized-planning richness (2026 paper)","group":"生成与创作","metric":"Narrative richness","unit":"score","public_history":true,"protocol":{"en":"All twelve tested model checkpoints fill every enabled role for the same forty-two EpisodeSeeds under the fixed centralized-planning architecture. The original 0–100 Narrative richness is retained as a native measure. Coverage is observed steps out of twenty planned steps; consistency uses the published Kaplan–Meier C_obs and may carry survival beyond observational support, so it must be read with coverage. Richness combines log-scaled development frequency and breadth, separately from consistency. Complete history is shared; memory adds structured state. Three judges and held-out human validation are described in the paper.","zh":"完整十二模型在固定centralized-planning架构下，对相同42个情节种子承担全部启用角色。保留0–100原生Narrative richness。覆盖率为20个计划叙事步骤中的实测比例；一致性采用原表Kaplan–Meier C_obs，观测耗尽后可延续存活估计，须结合覆盖率解释。丰富度综合对数缩放的发展频次与广度，与一致性分开。所有条件均保留完整历史，记忆模块额外加入结构化状态。原文给出三评审与独立人工验证。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.15654v1","status":"paper_baselines","published_at":"2026-08-16"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":12,"result_count":12,"results":[[14412,17.8,370,14444,14414],[1748,4.5,370,14444,14414],[1659,29.1,370,14444,14414],[1645,33.9,370,14444,14414],[1651,31.7,370,14444,14414],[2576,65.7,370,14444,14414],[5411,41.7,370,14444,14414],[227,73.3,370,14444,14414],[3185,79.9,370,14444,14414],[3187,67.7,370,14444,14414],[502,67.8,370,14444,14414],[4048,66.9,370,14444,14414]]},{"id":"wse-richness-centralized-memory-planning-2026-paper","name":"WSE-bench · centralized-memory-planning richness (2026 paper)","group":"生成与创作","metric":"Narrative richness","unit":"score","public_history":true,"protocol":{"en":"All twelve tested model checkpoints fill every enabled role for the same forty-two EpisodeSeeds under the fixed centralized-memory-planning architecture. The original 0–100 Narrative richness is retained as a native measure. Coverage is observed steps out of twenty planned steps; consistency uses the published Kaplan–Meier C_obs and may carry survival beyond observational support, so it must be read with coverage. Richness combines log-scaled development frequency and breadth, separately from consistency. Complete history is shared; memory adds structured state. Three judges and held-out human validation are described in the paper.","zh":"完整十二模型在固定centralized-memory-planning架构下，对相同42个情节种子承担全部启用角色。保留0–100原生Narrative richness。覆盖率为20个计划叙事步骤中的实测比例；一致性采用原表Kaplan–Meier C_obs，观测耗尽后可延续存活估计，须结合覆盖率解释。丰富度综合对数缩放的发展频次与广度，与一致性分开。所有条件均保留完整历史，记忆模块额外加入结构化状态。原文给出三评审与独立人工验证。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.15654v1","status":"paper_baselines","published_at":"2026-08-16"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":12,"result_count":12,"results":[[14412,18.9,370,14445,14414],[1748,4.2,370,14445,14414],[1659,27.0,370,14445,14414],[1645,36.0,370,14445,14414],[1651,30.4,370,14445,14414],[2576,54.1,370,14445,14414],[5411,38.7,370,14445,14414],[227,71.1,370,14445,14414],[3185,73.3,370,14445,14414],[3187,65.1,370,14445,14414],[502,62.5,370,14445,14414],[4048,65.9,370,14445,14414]]},{"id":"wse-richness-distributed-plain-2026-paper","name":"WSE-bench · distributed-plain richness (2026 paper)","group":"生成与创作","metric":"Narrative richness","unit":"score","public_history":true,"protocol":{"en":"All twelve tested model checkpoints fill every enabled role for the same forty-two EpisodeSeeds under the fixed distributed-plain architecture. The original 0–100 Narrative richness is retained as a native measure. Coverage is observed steps out of twenty planned steps; consistency uses the published Kaplan–Meier C_obs and may carry survival beyond observational support, so it must be read with coverage. Richness combines log-scaled development frequency and breadth, separately from consistency. Complete history is shared; memory adds structured state. Three judges and held-out human validation are described in the paper.","zh":"完整十二模型在固定distributed-plain架构下，对相同42个情节种子承担全部启用角色。保留0–100原生Narrative richness。覆盖率为20个计划叙事步骤中的实测比例；一致性采用原表Kaplan–Meier C_obs，观测耗尽后可延续存活估计，须结合覆盖率解释。丰富度综合对数缩放的发展频次与广度，与一致性分开。所有条件均保留完整历史，记忆模块额外加入结构化状态。原文给出三评审与独立人工验证。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.15654v1","status":"paper_baselines","published_at":"2026-08-16"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":12,"result_count":12,"results":[[14412,28.5,370,14446,14414],[1748,26.9,370,14446,14414],[1659,37.8,370,14446,14414],[1645,39.4,370,14446,14414],[1651,35.4,370,14446,14414],[2576,58.3,370,14446,14414],[5411,49.0,370,14446,14414],[227,71.3,370,14446,14414],[3185,68.8,370,14446,14414],[3187,68.8,370,14446,14414],[502,70.1,370,14446,14414],[4048,71.4,370,14446,14414]]},{"id":"wse-richness-distributed-memory-2026-paper","name":"WSE-bench · distributed-memory richness (2026 paper)","group":"生成与创作","metric":"Narrative richness","unit":"score","public_history":true,"protocol":{"en":"All twelve tested model checkpoints fill every enabled role for the same forty-two EpisodeSeeds under the fixed distributed-memory architecture. The original 0–100 Narrative richness is retained as a native measure. Coverage is observed steps out of twenty planned steps; consistency uses the published Kaplan–Meier C_obs and may carry survival beyond observational support, so it must be read with coverage. Richness combines log-scaled development frequency and breadth, separately from consistency. Complete history is shared; memory adds structured state. Three judges and held-out human validation are described in the paper.","zh":"完整十二模型在固定distributed-memory架构下，对相同42个情节种子承担全部启用角色。保留0–100原生Narrative richness。覆盖率为20个计划叙事步骤中的实测比例；一致性采用原表Kaplan–Meier C_obs，观测耗尽后可延续存活估计，须结合覆盖率解释。丰富度综合对数缩放的发展频次与广度，与一致性分开。所有条件均保留完整历史，记忆模块额外加入结构化状态。原文给出三评审与独立人工验证。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.15654v1","status":"paper_baselines","published_at":"2026-08-16"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":12,"result_count":12,"results":[[14412,30.2,370,14447,14414],[1748,25.5,370,14447,14414],[1659,36.2,370,14447,14414],[1645,41.2,370,14447,14414],[1651,36.1,370,14447,14414],[2576,53.6,370,14447,14414],[5411,45.6,370,14447,14414],[227,63.8,370,14447,14414],[3185,64.7,370,14447,14414],[3187,68.8,370,14447,14414],[502,71.0,370,14447,14414],[4048,68.7,370,14447,14414]]},{"id":"wse-richness-distributed-planning-2026-paper","name":"WSE-bench · distributed-planning richness (2026 paper)","group":"生成与创作","metric":"Narrative richness","unit":"score","public_history":true,"protocol":{"en":"All twelve tested model checkpoints fill every enabled role for the same forty-two EpisodeSeeds under the fixed distributed-planning architecture. The original 0–100 Narrative richness is retained as a native measure. Coverage is observed steps out of twenty planned steps; consistency uses the published Kaplan–Meier C_obs and may carry survival beyond observational support, so it must be read with coverage. Richness combines log-scaled development frequency and breadth, separately from consistency. Complete history is shared; memory adds structured state. Three judges and held-out human validation are described in the paper.","zh":"完整十二模型在固定distributed-planning架构下，对相同42个情节种子承担全部启用角色。保留0–100原生Narrative richness。覆盖率为20个计划叙事步骤中的实测比例；一致性采用原表Kaplan–Meier C_obs，观测耗尽后可延续存活估计，须结合覆盖率解释。丰富度综合对数缩放的发展频次与广度，与一致性分开。所有条件均保留完整历史，记忆模块额外加入结构化状态。原文给出三评审与独立人工验证。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.15654v1","status":"paper_baselines","published_at":"2026-08-16"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":12,"result_count":12,"results":[[14412,16.4,370,14448,14414],[1748,5.3,370,14448,14414],[1659,32.2,370,14448,14414],[1645,39.2,370,14448,14414],[1651,32.8,370,14448,14414],[2576,69.0,370,14448,14414],[5411,53.7,370,14448,14414],[227,78.9,370,14448,14414],[3185,84.0,370,14448,14414],[3187,71.6,370,14448,14414],[502,70.7,370,14448,14414],[4048,68.7,370,14448,14414]]},{"id":"wse-richness-distributed-memory-planning-2026-paper","name":"WSE-bench · distributed-memory-planning richness (2026 paper)","group":"生成与创作","metric":"Narrative richness","unit":"score","public_history":true,"protocol":{"en":"All twelve tested model checkpoints fill every enabled role for the same forty-two EpisodeSeeds under the fixed distributed-memory-planning architecture. The original 0–100 Narrative richness is retained as a native measure. Coverage is observed steps out of twenty planned steps; consistency uses the published Kaplan–Meier C_obs and may carry survival beyond observational support, so it must be read with coverage. Richness combines log-scaled development frequency and breadth, separately from consistency. Complete history is shared; memory adds structured state. Three judges and held-out human validation are described in the paper.","zh":"完整十二模型在固定distributed-memory-planning架构下，对相同42个情节种子承担全部启用角色。保留0–100原生Narrative richness。覆盖率为20个计划叙事步骤中的实测比例；一致性采用原表Kaplan–Meier C_obs，观测耗尽后可延续存活估计，须结合覆盖率解释。丰富度综合对数缩放的发展频次与广度，与一致性分开。所有条件均保留完整历史，记忆模块额外加入结构化状态。原文给出三评审与独立人工验证。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.15654v1","status":"paper_baselines","published_at":"2026-08-16"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":12,"result_count":12,"results":[[14412,15.7,370,14449,14414],[1748,8.2,370,14449,14414],[1659,35.0,370,14449,14414],[1645,37.7,370,14449,14414],[1651,31.4,370,14449,14414],[2576,64.3,370,14449,14414],[5411,49.9,370,14449,14414],[227,76.2,370,14449,14414],[3185,78.7,370,14449,14414],[3187,71.4,370,14449,14414],[502,70.4,370,14449,14414],[4048,71.0,370,14449,14414]]},{"id":"groundtruth-coe-2026-paper","name":"Groundtruth · coe rubric (2026 paper)","group":"专业工作","metric":"Mean pointwise rubric score","unit":"score","public_history":true,"protocol":{"en":"The complete eight-model official EigenformAI baseline evaluates the same fifty questions under the coe rubric. Each original score_b is an integer from zero to ten; the displayed value is the unrounded arithmetic mean of all fifty scores. Question IDs and exact text match across all eight runs. The official submission repository marks results as self-reported, not independently rerun or verified. All these baseline runs use OpenAI GPT-5.5 as the judge; judge rationale, component scores, raw answers and metadata remain in the public source. OxAlpha has an undisclosed stealth route and is excluded from the seven verified base-model minimum.","zh":"完整八模型官方EigenformAI基线按coe评分规则回答相同50题。原始score_b为0至10整数，展示值为50项未四舍五入算术均值。八次运行的题目ID与正文完全匹配。官方提交库明确这些结果为自行报告，未独立重跑或核验。本面板统一以OpenAI GPT-5.5评分，公开原始回答、理由、组件分及元数据。OxAlpha的隐匿路由底模未披露，不计入其余七个已知底模门槛。"},"source":{"name":"EigenformAI self-reported baseline submissions","url":"https://huggingface.co/api/datasets/EigenformAI/groundtruth-dynamic-benchmarking","status":"paper_baselines","published_at":"2026-08-22"},"score_max":10,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 10","zh":"分数 / 10"},"model_count":8,"result_count":8,"results":[[3492,8.46,370,14450,14451],[14452,7.76,370,14450,14453],[7177,6.98,370,14450,14454],[762,8.88,370,14450,14455],[2974,8.42,370,14450,14456],[2977,9.1,370,14450,14457],[3568,8.86,370,14450,14458],[14459,8.68,370,14450,14460]]},{"id":"groundtruth-supergene-2026-paper","name":"Groundtruth · supergene rubric (2026 paper)","group":"专业工作","metric":"Mean pointwise rubric score","unit":"score","public_history":true,"protocol":{"en":"The complete eight-model official EigenformAI baseline evaluates the same fifty questions under the supergene rubric. Each original score_b is an integer from zero to ten; the displayed value is the unrounded arithmetic mean of all fifty scores. Question IDs and exact text match across all eight runs. The official submission repository marks results as self-reported, not independently rerun or verified. All these baseline runs use OpenAI GPT-5.5 as the judge; judge rationale, component scores, raw answers and metadata remain in the public source. OxAlpha has an undisclosed stealth route and is excluded from the seven verified base-model minimum.","zh":"完整八模型官方EigenformAI基线按supergene评分规则回答相同50题。原始score_b为0至10整数，展示值为50项未四舍五入算术均值。八次运行的题目ID与正文完全匹配。官方提交库明确这些结果为自行报告，未独立重跑或核验。本面板统一以OpenAI GPT-5.5评分，公开原始回答、理由、组件分及元数据。OxAlpha的隐匿路由底模未披露，不计入其余七个已知底模门槛。"},"source":{"name":"EigenformAI self-reported baseline submissions","url":"https://huggingface.co/api/datasets/EigenformAI/groundtruth-dynamic-benchmarking","status":"paper_baselines","published_at":"2026-08-22"},"score_max":10,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 10","zh":"分数 / 10"},"model_count":8,"result_count":8,"results":[[3492,7.58,370,14461,14462],[14452,7.26,370,14461,14463],[7177,6.7,370,14461,14464],[762,8.26,370,14461,14465],[2974,7.16,370,14461,14466],[2977,8.72,370,14461,14467],[3568,8.54,370,14461,14468],[14459,8.42,370,14461,14469]]},{"id":"groundtruth-technical-2026-paper","name":"Groundtruth · technical rubric (2026 paper)","group":"专业工作","metric":"Mean pointwise rubric score","unit":"score","public_history":true,"protocol":{"en":"The complete eight-model official EigenformAI baseline evaluates the same fifty questions under the technical rubric. Each original score_b is an integer from zero to ten; the displayed value is the unrounded arithmetic mean of all fifty scores. Question IDs and exact text match across all eight runs. The official submission repository marks results as self-reported, not independently rerun or verified. All these baseline runs use OpenAI GPT-5.5 as the judge; judge rationale, component scores, raw answers and metadata remain in the public source. OxAlpha has an undisclosed stealth route and is excluded from the seven verified base-model minimum.","zh":"完整八模型官方EigenformAI基线按technical评分规则回答相同50题。原始score_b为0至10整数，展示值为50项未四舍五入算术均值。八次运行的题目ID与正文完全匹配。官方提交库明确这些结果为自行报告，未独立重跑或核验。本面板统一以OpenAI GPT-5.5评分，公开原始回答、理由、组件分及元数据。OxAlpha的隐匿路由底模未披露，不计入其余七个已知底模门槛。"},"source":{"name":"EigenformAI self-reported baseline submissions","url":"https://huggingface.co/api/datasets/EigenformAI/groundtruth-dynamic-benchmarking","status":"paper_baselines","published_at":"2026-08-22"},"score_max":10,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 10","zh":"分数 / 10"},"model_count":8,"result_count":8,"results":[[3492,7.34,370,14470,14471],[14452,7.62,370,14470,14472],[7177,6.76,370,14470,14473],[762,8.36,370,14470,14474],[2974,7.62,370,14470,14475],[2977,8.62,370,14470,14476],[3568,8.46,370,14470,14477],[14459,8.22,370,14470,14478]]},{"id":"privbench-v11-2026-paper","name":"PrivBench · text privatization v1.1 (2026 paper)","group":"安全与可靠性","metric":"Privacy–utility aggregate μ1.1","unit":"score","public_history":true,"protocol":{"en":"The complete twenty-nine anonymization-method/budget configurations evaluate the same seven-dataset PrivBench protocol. Native μ1.1 is a 0–100 composite of four privacy and three utility modules, excluding the redundant SIM, MAU and PEM modules of v1.0; it is not an accuracy percentage. Epsilon is the dataset average document word count, and 2ε is a separate budget configuration. Official runners establish six distinct learned backbones/architectures without counting budget/redaction variants: trained ADePT LSTM, BART-large, GPT-2 paraphraser, FLAN-T5-large, GLiNER PII and Qwen2.5-7B for RUPTA. Rule-based references remain explicitly named in the full method cohort.","zh":"完整二十九个匿名化方法、预算配置按同一七数据集PrivBench协议评测。μ1.1是四项隐私、三项效用模块的0–100原生综合分，移除v1.0中冗余的SIM、MAU与PEM，并非准确率百分比。ε为数据集平均文档词数，2ε作为另一预算配置。官方运行脚本确认至少六种不同学习底模、架构：独立训练ADePT LSTM、BART-large、GPT-2释义器、FLAN-T5-large、GLiNER PII及RUPTA的Qwen2.5-7B。预算和删词、替换变体不计新底模；完整方法面板保留明确标名的规则式参考。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.29624v1","status":"paper_baselines","published_at":"2026-08-30"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":29,"result_count":29,"results":[[14479,47.8,370,14480,14481],[14482,51.2,370,14483,14481],[14484,48.3,370,14485,14481],[14486,56.1,370,14487,14481],[14488,46.9,370,14489,14481],[14490,50.2,370,14491,14481],[14492,42.2,370,14493,14481],[14494,42.2,370,14495,14481],[14496,44.1,370,14497,14481],[14498,42.8,370,14499,14481],[14500,49.3,370,14501,14481],[14502,39.1,370,14503,14481],[14504,45.7,370,14505,14481],[14506,46.2,370,14507,14481],[14508,49.7,370,14509,14481],[14510,49.8,370,14511,14481],[14512,54.2,370,14513,14481],[14514,57.8,370,14515,14481],[14516,47.9,370,14517,14481],[14518,42.3,370,14519,14481],[14520,53.4,370,14521,14481],[14522,52.7,370,14523,14481],[14524,62.6,370,14525,14481],[14526,58.4,370,14527,14481],[14528,56.2,370,14529,14481],[14530,55.0,370,14531,14481],[14532,50.1,370,14533,14481],[14534,49.4,370,14535,14481],[14536,49.9,370,14537,14481]]},{"id":"rigidbench-ssim-2026-paper","name":"RigidBench · physical video similarity (2026 paper)","group":"视觉与视频","metric":"Full-frame SSIM","unit":"score","public_history":true,"protocol":{"en":"All eight generators evaluate the same one hundred simulator-grounded clips, each with forty-nine 1280×704 frames at 24 fps. Appendix C explicitly confirms one hundred finite values in every metric/model cell except Veo3.1 identity drift, which is a different metric. The native 0–1 SSIM averages frames per example and then equally averages examples. Wan2.2, Cosmos2.5, Veo3.1, Kling3, Seedance2 and GrokImagine provide six distinct generator families; Fast modes remain configurations. SSIM measures visual similarity and does not replace object motion or geometry metrics.","zh":"完整八个生成器评测相同100段模拟器真值视频，每段49帧、1280×704、24fps。附录C明确除另一指标Veo3.1身份漂移外，每个模型、指标单元均有100个有限值。0–1原生SSIM先在样本内平均帧，再等权平均样本。Wan2.2、Cosmos2.5、Veo3.1、Kling3、Seedance2、GrokImagine为六种生成模型族，Fast模式作为配置。SSIM衡量视觉相似性，不替代运动与几何指标。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.15555v1","status":"paper_baselines","published_at":"2026-08-16"},"score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":8,"result_count":8,"results":[[14538,0.925,370,14539,14540],[14541,0.909,370,14542,14540],[10417,0.879,370,14543,14540],[14544,0.902,370,14545,14540],[14546,0.912,370,14547,14540],[10413,0.735,370,14548,14540],[14549,0.722,370,14550,14540],[8088,0.901,370,14551,14540]]},{"id":"tw-exam-bench-sample-2026-paper","name":"OpenTWBench · Taiwan national examinations (2026 paper)","group":"知识与推理","metric":"Stratified-sample answer accuracy","unit":"%","public_history":true,"protocol":{"en":"All ten measured model/mode configurations in the published stratified-sample panel use the same deterministic content-hash sample: 22,021 questions after figure-dependent items were withdrawn. The composition-weighted Qwen full-corpus anchor uses different actual items and is excluded. Explicit boxed A–D answer extraction, native accuracy and parse rates are retained. Instruct/Reasoning variants remain configurations; Taiwan-tuned and Gemma finetunes are not new bases. Nine distinct base checkpoints remain. The underlying dataset first appeared August22; the October14 paper documents the cleaned corpus and corrected extraction apparatus.","zh":"原表分层样本面板的完整十个实测模型、模式配置使用相同内容哈希确定性样本，撤回依赖缺失图示的题目后共22,021题。按样本构成加权的Qwen全库锚点实际题目不同，另行保留。明确框选A–D解析方式，并保留准确率与解析率。Instruct、Reasoning及台湾微调均作为配置，不增加底模。仍有至少六个独立底模检查点。数据集8月22日公开，10月14日论文说明清理后的题库及修正后的解析协议。"},"source":{"name":"Paper authors","url":"https://arxiv.org/html/2610.19848v1#S6.T3","status":"paper_baselines","published_at":"2026-08-22"},"model_count":10,"result_count":10,"results":[[14552,0.8340000000000001,370,14553,14554],[8733,0.821,370,14555,14554],[62,0.7709999999999999,370,14553,14554],[14556,0.76,370,14557,14554],[6689,0.667,370,14555,14554],[5648,0.667,370,14558,14554],[14559,0.465,370,14560,14554],[14561,0.45399999999999996,370,14562,14554],[14563,0.45299999999999996,370,14564,14554],[14565,0.165,370,14566,14554]]},{"id":"vals-index-v22-2026-official","name":"Vals Index · v2.2 (2026 official)","group":"专业工作","metric":"GDP-weighted Vals Index","unit":"score","public_history":true,"protocol":{"en":"The complete thirty-four-model official v2.2 leaderboard uses the published eight-component index across Finance, Coding, Legal and Tax, weighted8.0:5.6:1.2:0.5. The native index is reported on0–100, not as one task-accuracy percentage. Each sector equally averages its specified components. CodeMigration uses the published fixed50CLI+10COBOL subset with75/25 weighting; FinanceAgent follows its three-run protocol. The October19 version updates FinanceAgent, EMB, migration, HLAB and Tax all-pass scoring. November5 is a result update, not the first release of v2.2. Router/product aliases remain original names and do not contribute to the independent-base minimum.","zh":"保留官方v2.2完整34模型排行。指数由金融、编程、法律、税务八个固定组件组成，按8.0:5.6:1.2:0.5加权，各领域内部等权平均。0–100指数为原生综合量，不作单项准确率百分比。CodeMigration采用固定50个CLI、10个COBOL任务及75/25权重，FinanceAgent依照三次运行协议。10月19日版本更新金融、表格、迁移、法律及税务全通过评分；11月5日是结果更新日。路由器和产品别名保留原名，不增加独立底模计数。"},"source":{"name":"Vals AI","url":"https://www.vals.ai/benchmarks/vals_index","status":"dated_release","published_at":"2026-10-19"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":34,"result_count":34,"results":[[2961,70.85,370,14567,14568],[3380,70.45,370,14567,14568],[2959,70.18,370,14567,14568],[2962,69.24,370,14567,14568],[3505,66.79,370,14567,14568],[2958,65.02,370,14567,14568],[3510,64.88,370,14567,14568],[2960,63.94,370,14567,14568],[2967,62.52,370,14567,14568],[2966,61.26,370,14567,14568],[2964,60.54,370,14567,14568],[3515,59.73,370,14567,14568],[14569,59.35,370,14567,14568],[3537,59.05,370,14567,14568],[2970,58.73,370,14567,14568],[2968,58.56,370,14567,14568],[2971,57.81,370,14567,14568],[2992,56.53,370,14567,14568],[14570,56.08,370,14567,14568],[2983,55.95,370,14567,14568],[3518,54.83,370,14567,14568],[3534,54.76,370,14567,14568],[3395,53.78,370,14567,14568],[3524,53.21,370,14567,14568],[3539,52.97,370,14567,14568],[2977,52.95,370,14567,14568],[2976,52.32,370,14567,14568],[3529,50.86,370,14567,14568],[2987,48.57,370,14567,14568],[3398,43.66,370,14567,14568],[3545,43.42,370,14567,14568],[2990,29.78,370,14567,14568],[203,28.96,370,14567,14568],[3549,27.59,370,14567,14568]]},{"id":"robopulse-plus-progress-judging-2026-paper","name":"RoboPulse++ · robotic process judgments (2026 paper)","group":"Agent 与工具","metric":"Rising/Falling macro-F1","unit":"score","public_history":true,"protocol":{"en":"The full eighteen judge/inference configurations classify the same 2,244 human-annotated progress intervals from700 trajectories and275 manipulation tasks. Macro-F1 equally weights Rising and Falling on its original0–1 scale. Robo-Dopamine Forward/Backward/Incremental, LRM Temporal/Absolute and Pair/Sequence prompting remain configurations. Four general bases(GPT5.4,Gemini3.1Pro,Qwen3.6Plus,ClaudeSonnet4.6) plus RoboMeter’s Qwen3-VL4B and original Robo-Dopamine’s RoboBrain2/Qwen2.5-VL lineage establish at least six distinct bases; other specialized names are not required to satisfy the minimum. Different temporal context is part of each published judge protocol and explicit in configurations. Robot-policy rollout panels on other benchmarks are excluded.","zh":"完整十八个评审、推理配置判断相同700段轨迹、275个操作任务中的2,244个人工标注进度区间。宏F1等权汇总上升、下降类，保留0–1原生单位。Robo-Dopamine三种方向、LRM两种读法、成对或序列提示作为配置。四个通用底模GPT5.4、Gemini3.1Pro、Qwen3.6Plus、ClaudeSonnet4.6，加上RoboMeter的Qwen3-VL4B及原版Robo-Dopamine的RoboBrain2、Qwen2.5-VL谱系，满足至少六底模，不借其余专门方法别名凑数。不同时间上下文属于原评审协议，明确保留。外部基准上的机器人策略运行面板单列。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2608.14284v1","status":"paper_baselines","published_at":"2026-08-14"},"score_max":1,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 1","zh":"分数 / 1"},"model_count":14,"result_count":18,"results":[[14571,0.77,370,14572,14573],[14574,0.76,370,14575,14573],[14576,0.74,370,14577,14573],[14578,0.69,370,14579,14573],[14580,0.56,370,14581,14573],[14582,0.36,370,14583,14573],[14584,0.56,370,14585,14573],[14586,0.39,370,14587,14573],[14588,0.66,370,14589,14573],[14590,0.44,370,14591,14573],[3184,0.38,370,14592,14573],[3412,0.58,370,14592,14573],[3603,0.54,370,14592,14573],[3572,0.4,370,14592,14573],[3184,0.49,370,14593,14573],[3412,0.35,370,14593,14573],[3603,0.49,370,14593,14573],[3572,0.48,370,14593,14573]]},{"id":"infoedit-expand-text-edit-compliance-2026-paper","name":"InfoEdit ·expand-text edit-compliance (2026 paper)","group":"视觉与视频","metric":"edit-compliance","unit":"%","public_history":true,"protocol":{"en":"Eight independently identified image-editing checkpoints share the same 1,000 source infographics and fixed instructions for this task, within a 4,000-edit four-task benchmark. Gemini3.1Pro judges binary edit compliance and content preservation with task-specific reflow rubrics; joint success is their conjunction. Preservation permits necessary layout reflow while retaining unedited content. Three experts independently judge 800 outputs from each of three model levels, yielding majority-reference agreement0.920–0.981; another vendor judge and five repeated scoring runs test reliability. Native model interfaces and documented generation parameters remain configurations. Three code-only models are excluded from this image-editing cohort.","zh":"八个身份明确、独立的图像编辑权重共享1000张源信息图及本任务固定指令，全基准四任务共4000次编辑。Gemini3.1Pro按逐任务重排rubric二元判断编辑合规与内容保留，联合成功要求两者同时成立；内容保留允许必要的布局重排。三位专家分别审核三个能力层级模型各800条输出，评审对多数人工参考的一致率为0.920–0.981，另用跨厂商评审及五次重复评分检查可靠性。保留模型原生接口及生成设置，三个代码模型不混入图像编辑面板。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.33286v1","published_at":"2026-09-27","status":"paper_baselines"},"model_count":8,"result_count":8,"results":[[6368,0.99,370,14594,14595],[14596,0.871,370,14594,14595],[14597,0.851,370,14594,14595],[14598,0.23199999999999998,370,14594,14595],[14599,0.221,370,14594,14595],[14600,0.213,370,14594,14595],[8150,0.092,370,14594,14595],[14601,0.061,370,14594,14595]]},{"id":"infoedit-expand-text-content-preservation-2026-paper","name":"InfoEdit ·expand-text content-preservation (2026 paper)","group":"视觉与视频","metric":"content-preservation","unit":"%","public_history":true,"protocol":{"en":"Eight independently identified image-editing checkpoints share the same 1,000 source infographics and fixed instructions for this task, within a 4,000-edit four-task benchmark. Gemini3.1Pro judges binary edit compliance and content preservation with task-specific reflow rubrics; joint success is their conjunction. Preservation permits necessary layout reflow while retaining unedited content. Three experts independently judge 800 outputs from each of three model levels, yielding majority-reference agreement0.920–0.981; another vendor judge and five repeated scoring runs test reliability. Native model interfaces and documented generation parameters remain configurations. Three code-only models are excluded from this image-editing cohort.","zh":"八个身份明确、独立的图像编辑权重共享1000张源信息图及本任务固定指令，全基准四任务共4000次编辑。Gemini3.1Pro按逐任务重排rubric二元判断编辑合规与内容保留，联合成功要求两者同时成立；内容保留允许必要的布局重排。三位专家分别审核三个能力层级模型各800条输出，评审对多数人工参考的一致率为0.920–0.981，另用跨厂商评审及五次重复评分检查可靠性。保留模型原生接口及生成设置，三个代码模型不混入图像编辑面板。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.33286v1","published_at":"2026-09-27","status":"paper_baselines"},"model_count":8,"result_count":8,"results":[[6368,0.867,370,14594,14595],[14596,0.64,370,14594,14595],[14597,0.586,370,14594,14595],[14598,0.23800000000000002,370,14594,14595],[14599,0.28,370,14594,14595],[14600,0.348,370,14594,14595],[8150,0.033,370,14594,14595],[14601,0.07200000000000001,370,14594,14595]]},{"id":"infoedit-expand-text-joint-success-2026-paper","name":"InfoEdit ·expand-text joint-success (2026 paper)","group":"视觉与视频","metric":"joint-success","unit":"%","public_history":true,"protocol":{"en":"Eight independently identified image-editing checkpoints share the same 1,000 source infographics and fixed instructions for this task, within a 4,000-edit four-task benchmark. Gemini3.1Pro judges binary edit compliance and content preservation with task-specific reflow rubrics; joint success is their conjunction. Preservation permits necessary layout reflow while retaining unedited content. Three experts independently judge 800 outputs from each of three model levels, yielding majority-reference agreement0.920–0.981; another vendor judge and five repeated scoring runs test reliability. Native model interfaces and documented generation parameters remain configurations. Three code-only models are excluded from this image-editing cohort.","zh":"八个身份明确、独立的图像编辑权重共享1000张源信息图及本任务固定指令，全基准四任务共4000次编辑。Gemini3.1Pro按逐任务重排rubric二元判断编辑合规与内容保留，联合成功要求两者同时成立；内容保留允许必要的布局重排。三位专家分别审核三个能力层级模型各800条输出，评审对多数人工参考的一致率为0.920–0.981，另用跨厂商评审及五次重复评分检查可靠性。保留模型原生接口及生成设置，三个代码模型不混入图像编辑面板。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.33286v1","published_at":"2026-09-27","status":"paper_baselines"},"model_count":8,"result_count":8,"results":[[6368,0.8590000000000001,370,14594,14595],[14596,0.578,370,14594,14595],[14597,0.526,370,14594,14595],[14598,0.092,370,14594,14595],[14599,0.091,370,14594,14595],[14600,0.126,370,14594,14595],[8150,0.015,370,14594,14595],[14601,0.0,370,14594,14595]]},{"id":"infoedit-insert-element-edit-compliance-2026-paper","name":"InfoEdit ·insert-element edit-compliance (2026 paper)","group":"视觉与视频","metric":"edit-compliance","unit":"%","public_history":true,"protocol":{"en":"Eight independently identified image-editing checkpoints share the same 1,000 source infographics and fixed instructions for this task, within a 4,000-edit four-task benchmark. Gemini3.1Pro judges binary edit compliance and content preservation with task-specific reflow rubrics; joint success is their conjunction. Preservation permits necessary layout reflow while retaining unedited content. Three experts independently judge 800 outputs from each of three model levels, yielding majority-reference agreement0.920–0.981; another vendor judge and five repeated scoring runs test reliability. Native model interfaces and documented generation parameters remain configurations. Three code-only models are excluded from this image-editing cohort.","zh":"八个身份明确、独立的图像编辑权重共享1000张源信息图及本任务固定指令，全基准四任务共4000次编辑。Gemini3.1Pro按逐任务重排rubric二元判断编辑合规与内容保留，联合成功要求两者同时成立；内容保留允许必要的布局重排。三位专家分别审核三个能力层级模型各800条输出，评审对多数人工参考的一致率为0.920–0.981，另用跨厂商评审及五次重复评分检查可靠性。保留模型原生接口及生成设置，三个代码模型不混入图像编辑面板。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.33286v1","published_at":"2026-09-27","status":"paper_baselines"},"model_count":8,"result_count":8,"results":[[6368,0.8540000000000001,370,14602,14595],[14596,0.588,370,14602,14595],[14597,0.653,370,14602,14595],[14598,0.031,370,14602,14595],[14599,0.12300000000000001,370,14602,14595],[14600,0.025,370,14602,14595],[8150,0.012,370,14602,14595],[14601,0.0,370,14602,14595]]},{"id":"infoedit-insert-element-content-preservation-2026-paper","name":"InfoEdit ·insert-element content-preservation (2026 paper)","group":"视觉与视频","metric":"content-preservation","unit":"%","public_history":true,"protocol":{"en":"Eight independently identified image-editing checkpoints share the same 1,000 source infographics and fixed instructions for this task, within a 4,000-edit four-task benchmark. Gemini3.1Pro judges binary edit compliance and content preservation with task-specific reflow rubrics; joint success is their conjunction. Preservation permits necessary layout reflow while retaining unedited content. Three experts independently judge 800 outputs from each of three model levels, yielding majority-reference agreement0.920–0.981; another vendor judge and five repeated scoring runs test reliability. Native model interfaces and documented generation parameters remain configurations. Three code-only models are excluded from this image-editing cohort.","zh":"八个身份明确、独立的图像编辑权重共享1000张源信息图及本任务固定指令，全基准四任务共4000次编辑。Gemini3.1Pro按逐任务重排rubric二元判断编辑合规与内容保留，联合成功要求两者同时成立；内容保留允许必要的布局重排。三位专家分别审核三个能力层级模型各800条输出，评审对多数人工参考的一致率为0.920–0.981，另用跨厂商评审及五次重复评分检查可靠性。保留模型原生接口及生成设置，三个代码模型不混入图像编辑面板。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.33286v1","published_at":"2026-09-27","status":"paper_baselines"},"model_count":8,"result_count":8,"results":[[6368,0.807,370,14602,14595],[14596,0.578,370,14602,14595],[14597,0.494,370,14602,14595],[14598,0.146,370,14602,14595],[14599,0.11,370,14602,14595],[14600,0.043,370,14602,14595],[8150,0.027000000000000003,370,14602,14595],[14601,0.212,370,14602,14595]]},{"id":"infoedit-insert-element-joint-success-2026-paper","name":"InfoEdit ·insert-element joint-success (2026 paper)","group":"视觉与视频","metric":"joint-success","unit":"%","public_history":true,"protocol":{"en":"Eight independently identified image-editing checkpoints share the same 1,000 source infographics and fixed instructions for this task, within a 4,000-edit four-task benchmark. Gemini3.1Pro judges binary edit compliance and content preservation with task-specific reflow rubrics; joint success is their conjunction. Preservation permits necessary layout reflow while retaining unedited content. Three experts independently judge 800 outputs from each of three model levels, yielding majority-reference agreement0.920–0.981; another vendor judge and five repeated scoring runs test reliability. Native model interfaces and documented generation parameters remain configurations. Three code-only models are excluded from this image-editing cohort.","zh":"八个身份明确、独立的图像编辑权重共享1000张源信息图及本任务固定指令，全基准四任务共4000次编辑。Gemini3.1Pro按逐任务重排rubric二元判断编辑合规与内容保留，联合成功要求两者同时成立；内容保留允许必要的布局重排。三位专家分别审核三个能力层级模型各800条输出，评审对多数人工参考的一致率为0.920–0.981，另用跨厂商评审及五次重复评分检查可靠性。保留模型原生接口及生成设置，三个代码模型不混入图像编辑面板。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.33286v1","published_at":"2026-09-27","status":"paper_baselines"},"model_count":8,"result_count":8,"results":[[6368,0.72,370,14602,14595],[14596,0.402,370,14602,14595],[14597,0.39,370,14602,14595],[14598,0.004,370,14602,14595],[14599,0.013000000000000001,370,14602,14595],[14600,0.0,370,14602,14595],[8150,0.0,370,14602,14595],[14601,0.0,370,14602,14595]]},{"id":"infoedit-swap-block-edit-compliance-2026-paper","name":"InfoEdit ·swap-block edit-compliance (2026 paper)","group":"视觉与视频","metric":"edit-compliance","unit":"%","public_history":true,"protocol":{"en":"Eight independently identified image-editing checkpoints share the same 1,000 source infographics and fixed instructions for this task, within a 4,000-edit four-task benchmark. Gemini3.1Pro judges binary edit compliance and content preservation with task-specific reflow rubrics; joint success is their conjunction. Preservation permits necessary layout reflow while retaining unedited content. Three experts independently judge 800 outputs from each of three model levels, yielding majority-reference agreement0.920–0.981; another vendor judge and five repeated scoring runs test reliability. Native model interfaces and documented generation parameters remain configurations. Three code-only models are excluded from this image-editing cohort.","zh":"八个身份明确、独立的图像编辑权重共享1000张源信息图及本任务固定指令，全基准四任务共4000次编辑。Gemini3.1Pro按逐任务重排rubric二元判断编辑合规与内容保留，联合成功要求两者同时成立；内容保留允许必要的布局重排。三位专家分别审核三个能力层级模型各800条输出，评审对多数人工参考的一致率为0.920–0.981，另用跨厂商评审及五次重复评分检查可靠性。保留模型原生接口及生成设置，三个代码模型不混入图像编辑面板。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.33286v1","published_at":"2026-09-27","status":"paper_baselines"},"model_count":8,"result_count":8,"results":[[6368,0.29,370,14603,14595],[14596,0.365,370,14603,14595],[14597,0.3,370,14603,14595],[14598,0.016,370,14603,14595],[14599,0.09300000000000001,370,14603,14595],[14600,0.004,370,14603,14595],[8150,0.0,370,14603,14595],[14601,0.0,370,14603,14595]]},{"id":"infoedit-swap-block-content-preservation-2026-paper","name":"InfoEdit ·swap-block content-preservation (2026 paper)","group":"视觉与视频","metric":"content-preservation","unit":"%","public_history":true,"protocol":{"en":"Eight independently identified image-editing checkpoints share the same 1,000 source infographics and fixed instructions for this task, within a 4,000-edit four-task benchmark. Gemini3.1Pro judges binary edit compliance and content preservation with task-specific reflow rubrics; joint success is their conjunction. Preservation permits necessary layout reflow while retaining unedited content. Three experts independently judge 800 outputs from each of three model levels, yielding majority-reference agreement0.920–0.981; another vendor judge and five repeated scoring runs test reliability. Native model interfaces and documented generation parameters remain configurations. Three code-only models are excluded from this image-editing cohort.","zh":"八个身份明确、独立的图像编辑权重共享1000张源信息图及本任务固定指令，全基准四任务共4000次编辑。Gemini3.1Pro按逐任务重排rubric二元判断编辑合规与内容保留，联合成功要求两者同时成立；内容保留允许必要的布局重排。三位专家分别审核三个能力层级模型各800条输出，评审对多数人工参考的一致率为0.920–0.981，另用跨厂商评审及五次重复评分检查可靠性。保留模型原生接口及生成设置，三个代码模型不混入图像编辑面板。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.33286v1","published_at":"2026-09-27","status":"paper_baselines"},"model_count":8,"result_count":8,"results":[[6368,0.792,370,14603,14595],[14596,0.7979999999999999,370,14603,14595],[14597,0.721,370,14603,14595],[14598,0.418,370,14603,14595],[14599,0.37,370,14603,14595],[14600,0.16399999999999998,370,14603,14595],[8150,0.063,370,14603,14595],[14601,0.778,370,14603,14595]]},{"id":"infoedit-swap-block-joint-success-2026-paper","name":"InfoEdit ·swap-block joint-success (2026 paper)","group":"视觉与视频","metric":"joint-success","unit":"%","public_history":true,"protocol":{"en":"Eight independently identified image-editing checkpoints share the same 1,000 source infographics and fixed instructions for this task, within a 4,000-edit four-task benchmark. Gemini3.1Pro judges binary edit compliance and content preservation with task-specific reflow rubrics; joint success is their conjunction. Preservation permits necessary layout reflow while retaining unedited content. Three experts independently judge 800 outputs from each of three model levels, yielding majority-reference agreement0.920–0.981; another vendor judge and five repeated scoring runs test reliability. Native model interfaces and documented generation parameters remain configurations. Three code-only models are excluded from this image-editing cohort.","zh":"八个身份明确、独立的图像编辑权重共享1000张源信息图及本任务固定指令，全基准四任务共4000次编辑。Gemini3.1Pro按逐任务重排rubric二元判断编辑合规与内容保留，联合成功要求两者同时成立；内容保留允许必要的布局重排。三位专家分别审核三个能力层级模型各800条输出，评审对多数人工参考的一致率为0.920–0.981，另用跨厂商评审及五次重复评分检查可靠性。保留模型原生接口及生成设置，三个代码模型不混入图像编辑面板。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.33286v1","published_at":"2026-09-27","status":"paper_baselines"},"model_count":8,"result_count":8,"results":[[6368,0.255,370,14603,14595],[14596,0.32,370,14603,14595],[14597,0.24,370,14603,14595],[14598,0.009000000000000001,370,14603,14595],[14599,0.052000000000000005,370,14603,14595],[14600,0.0,370,14603,14595],[8150,0.0,370,14603,14595],[14601,0.0,370,14603,14595]]},{"id":"infoedit-reshape-canvas-edit-compliance-2026-paper","name":"InfoEdit ·reshape-canvas edit-compliance (2026 paper)","group":"视觉与视频","metric":"edit-compliance","unit":"%","public_history":true,"protocol":{"en":"Eight independently identified image-editing checkpoints share the same 1,000 source infographics and fixed instructions for this task, within a 4,000-edit four-task benchmark. Gemini3.1Pro judges binary edit compliance and content preservation with task-specific reflow rubrics; joint success is their conjunction. Preservation permits necessary layout reflow while retaining unedited content. Three experts independently judge 800 outputs from each of three model levels, yielding majority-reference agreement0.920–0.981; another vendor judge and five repeated scoring runs test reliability. Native model interfaces and documented generation parameters remain configurations. Three code-only models are excluded from this image-editing cohort.","zh":"八个身份明确、独立的图像编辑权重共享1000张源信息图及本任务固定指令，全基准四任务共4000次编辑。Gemini3.1Pro按逐任务重排rubric二元判断编辑合规与内容保留，联合成功要求两者同时成立；内容保留允许必要的布局重排。三位专家分别审核三个能力层级模型各800条输出，评审对多数人工参考的一致率为0.920–0.981，另用跨厂商评审及五次重复评分检查可靠性。保留模型原生接口及生成设置，三个代码模型不混入图像编辑面板。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.33286v1","published_at":"2026-09-27","status":"paper_baselines"},"model_count":8,"result_count":8,"results":[[6368,0.963,370,14604,14595],[14596,0.872,370,14604,14595],[14597,0.92,370,14604,14595],[14598,0.254,370,14604,14595],[14599,0.6409999999999999,370,14604,14595],[14600,0.013999999999999999,370,14604,14595],[8150,0.165,370,14604,14595],[14601,0.203,370,14604,14595]]},{"id":"infoedit-reshape-canvas-content-preservation-2026-paper","name":"InfoEdit ·reshape-canvas content-preservation (2026 paper)","group":"视觉与视频","metric":"content-preservation","unit":"%","public_history":true,"protocol":{"en":"Eight independently identified image-editing checkpoints share the same 1,000 source infographics and fixed instructions for this task, within a 4,000-edit four-task benchmark. Gemini3.1Pro judges binary edit compliance and content preservation with task-specific reflow rubrics; joint success is their conjunction. Preservation permits necessary layout reflow while retaining unedited content. Three experts independently judge 800 outputs from each of three model levels, yielding majority-reference agreement0.920–0.981; another vendor judge and five repeated scoring runs test reliability. Native model interfaces and documented generation parameters remain configurations. Three code-only models are excluded from this image-editing cohort.","zh":"八个身份明确、独立的图像编辑权重共享1000张源信息图及本任务固定指令，全基准四任务共4000次编辑。Gemini3.1Pro按逐任务重排rubric二元判断编辑合规与内容保留，联合成功要求两者同时成立；内容保留允许必要的布局重排。三位专家分别审核三个能力层级模型各800条输出，评审对多数人工参考的一致率为0.920–0.981，另用跨厂商评审及五次重复评分检查可靠性。保留模型原生接口及生成设置，三个代码模型不混入图像编辑面板。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.33286v1","published_at":"2026-09-27","status":"paper_baselines"},"model_count":8,"result_count":8,"results":[[6368,0.6509999999999999,370,14604,14595],[14596,0.369,370,14604,14595],[14597,0.364,370,14604,14595],[14598,0.084,370,14604,14595],[14599,0.046,370,14604,14595],[14600,0.052000000000000005,370,14604,14595],[8150,0.0,370,14604,14595],[14601,0.0,370,14604,14595]]},{"id":"infoedit-reshape-canvas-joint-success-2026-paper","name":"InfoEdit ·reshape-canvas joint-success (2026 paper)","group":"视觉与视频","metric":"joint-success","unit":"%","public_history":true,"protocol":{"en":"Eight independently identified image-editing checkpoints share the same 1,000 source infographics and fixed instructions for this task, within a 4,000-edit four-task benchmark. Gemini3.1Pro judges binary edit compliance and content preservation with task-specific reflow rubrics; joint success is their conjunction. Preservation permits necessary layout reflow while retaining unedited content. Three experts independently judge 800 outputs from each of three model levels, yielding majority-reference agreement0.920–0.981; another vendor judge and five repeated scoring runs test reliability. Native model interfaces and documented generation parameters remain configurations. Three code-only models are excluded from this image-editing cohort.","zh":"八个身份明确、独立的图像编辑权重共享1000张源信息图及本任务固定指令，全基准四任务共4000次编辑。Gemini3.1Pro按逐任务重排rubric二元判断编辑合规与内容保留，联合成功要求两者同时成立；内容保留允许必要的布局重排。三位专家分别审核三个能力层级模型各800条输出，评审对多数人工参考的一致率为0.920–0.981，另用跨厂商评审及五次重复评分检查可靠性。保留模型原生接口及生成设置，三个代码模型不混入图像编辑面板。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.33286v1","published_at":"2026-09-27","status":"paper_baselines"},"model_count":8,"result_count":8,"results":[[6368,0.644,370,14604,14595],[14596,0.364,370,14604,14595],[14597,0.363,370,14604,14595],[14598,0.035,370,14604,14595],[14599,0.043,370,14604,14595],[14600,0.0,370,14604,14595],[8150,0.0,370,14604,14595],[14601,0.0,370,14604,14595]]},{"id":"infoedit-joint-macro-2026-paper","name":"InfoEdit ·joint macro (2026 paper)","group":"视觉与视频","metric":"Four-task macro joint success","unit":"%","public_history":true,"protocol":{"en":"Eight image-editing checkpoints share all four task populations of 1,000 edits each. The original Avg. equally averages the four joint success rates; each edit must satisfy both compliance and preservation. Fixed Gemini3.1Pro rubrics and the same source images are shared by models. Task-specific results and expert/cross-judge reliability audits remain in the source; code-only and code-plus-image results are separate three-backbone cohorts.","zh":"八个图像编辑权重共享四任务各1000条编辑，原表Avg.对四个联合成功率等权。每次编辑须同时满足修改合规与内容保留。各模型共享源图和固定Gemini3.1Pro判据，逐任务结果及人工、跨评审一致性审计保留原文。代码或代码加图像的三权重面板另行比较。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.33286v1","published_at":"2026-09-27","status":"paper_baselines"},"model_count":8,"result_count":8,"results":[[6368,0.62,370,14605,14595],[14596,0.41600000000000004,370,14605,14595],[14597,0.38,370,14605,14595],[14598,0.035,370,14605,14595],[14599,0.05,370,14605,14595],[14600,0.032,370,14605,14595],[8150,0.004,370,14605,14595],[14601,0.0,370,14605,14595]]},{"id":"biabench-outcome-complete-2026-paper","name":"BIABench ·complete outcome (2026 paper)","group":"数学与科学","metric":"Mean task-specific outcome score","unit":"score","public_history":true,"protocol":{"en":"Six independent backbones share all 16bioimage analysis tasks, three runs each (48/model), under the brief instruction andread-only source data. We select configurations whose delivered/no-deliverable/crash/refusal accounting totals48 with zero provider refusals. Opus uses ClaudeCode;the other five use DeepSeekHarness, so results explicitly represent model+harness behavior. The DeepSeekHarness Opus row totals47 andis omitted, as are provider-filteredGPT configurations. Outcomes use documented field-standard segmentation, tracking, detection, kinetics andother metrics mapped to 0–1. Missing outputs scorezero; task means average three runs, then average16 tasks. Process rubric scores have model-dependent undecidable-item denominators and are not imported.","zh":"六个独立模型共享全部16个生物图像分析任务、简要指令和只读原数据，每题三次，即每模型48次。所选配置的交付、无交付、崩溃和拒绝计数合计均为48，且没有提供商拒绝。Opus用ClaudeCode，其余五种用DeepSeekHarness，因此明确为模型加harness结果。DeepSeekHarness下Opus计数只有47，排除该行，也排除有提供商过滤的GPT配置。结果按文档所列分割、追踪、检测、动力学等领域指标映射至0–1；无输出记零，先每题平均三次，再对16题平均。过程评分的不可判定条目随模型变化，不混入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34274v1","published_at":"2026-09-28","status":"paper_baselines"},"score_min":0,"score_max":1,"saturation_excluded":true,"scale_label":{"en":"Native score / 1","zh":"原生分数 / 1"},"score_decimals":4,"model_count":6,"result_count":6,"results":[[762,0.58,370,14606,14607],[755,0.63,370,14608,14607],[3186,0.51,370,14609,14607],[3490,0.57,370,14609,14607],[3201,0.52,370,14610,14607],[3185,0.53,370,14611,14607]]},{"id":"crypto-ml-2025-backtest-return-2026-paper","name":"Crypto Trading ·2025-backtest return (2026 paper)","group":"数学与科学","metric":"Cumulative return (%)","unit":"raw","public_history":true,"protocol":{"en":"This cohort contains only learned ML forecasters with the common52Alpha market factors andten-asset long-only selection formulation. The 2025-backtest panel compares 24 independent ML architectures over the same market timeline. Backtests train on 2021–2023, select configurations on 2024, and test 2025-01-01 through2025-12-31. Prospective paper trading uses newly arriving OKX data from 2026-06-04 through2026-09-21 with the selected configurations frozen. Spot taker cost is0.10 percent per trade. Metrics retain native financial units, including negative returns/alpha/Sharpe; none is percent-correct. Lower drawdown/volatility measures risk andmust be interpreted alongside return. RL and LLM methods have different action/input regimes and are excluded; notably the paper discloses a mixed Pro/Flash DeepSeek backtest, which is not included. Live trading has fewer than six methods and remains separate.","zh":"本组只含共享52个Alpha行情因子、十资产候选及只做多选择规则的学习型ML预测器。2025-backtest面板在相同市场时间线上比较24种独立架构。回测以2021–2023年训练、2024年选配置、2025-01-01至12-31测试；模拟盘用2026-06-04至09-21新到达的OKX行情及冻结配置。现货每交易扣0.10%taker费。保留原生金融单位及负收益、负alpha、负Sharpe，不作题目正确率；回撤和波动率为风险指标，需结合收益理解。RL及LLM的输入和动作设定不同，未混入；论文披露的DeepSeek Pro/Flash混合回测也未收录。实盘少于六方法，另行保存。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34510v1","published_at":"2026-09-28","status":"paper_baselines"},"saturation_excluded":true,"scale_label":{"en":"Cumulative return (%)","zh":"Cumulative return (%)"},"score_decimals":3,"model_count":24,"result_count":24,"results":[[10503,8.57,370,14612,14613],[10501,21.46,370,14612,14613],[6748,21.08,370,14612,14613],[6811,41.93,370,14612,14613],[9203,30.33,370,14612,14613],[10916,34.52,370,14612,14613],[9681,62.94,370,14612,14613],[9204,29.66,370,14612,14613],[9603,32.9,370,14612,14613],[14614,18.31,370,14612,14613],[14615,16.44,370,14612,14613],[14616,21.17,370,14612,14613],[14617,10.53,370,14612,14613],[6744,5.32,370,14612,14613],[14618,9.78,370,14612,14613],[14619,24.38,370,14612,14613],[14620,25.08,370,14612,14613],[14621,19.41,370,14612,14613],[14622,10.92,370,14612,14613],[14623,-1.37,370,14612,14613],[14624,37.48,370,14612,14613],[14625,71.7,370,14612,14613],[14626,23.16,370,14612,14613],[14627,1.72,370,14612,14613]]},{"id":"crypto-ml-2025-backtest-alpha-2026-paper","name":"Crypto Trading ·2025-backtest alpha (2026 paper)","group":"数学与科学","metric":"Jensen alpha (%)","unit":"raw","public_history":true,"protocol":{"en":"This cohort contains only learned ML forecasters with the common52Alpha market factors andten-asset long-only selection formulation. The 2025-backtest panel compares 24 independent ML architectures over the same market timeline. Backtests train on 2021–2023, select configurations on 2024, and test 2025-01-01 through2025-12-31. Prospective paper trading uses newly arriving OKX data from 2026-06-04 through2026-09-21 with the selected configurations frozen. Spot taker cost is0.10 percent per trade. Metrics retain native financial units, including negative returns/alpha/Sharpe; none is percent-correct. Lower drawdown/volatility measures risk andmust be interpreted alongside return. RL and LLM methods have different action/input regimes and are excluded; notably the paper discloses a mixed Pro/Flash DeepSeek backtest, which is not included. Live trading has fewer than six methods and remains separate.","zh":"本组只含共享52个Alpha行情因子、十资产候选及只做多选择规则的学习型ML预测器。2025-backtest面板在相同市场时间线上比较24种独立架构。回测以2021–2023年训练、2024年选配置、2025-01-01至12-31测试；模拟盘用2026-06-04至09-21新到达的OKX行情及冻结配置。现货每交易扣0.10%taker费。保留原生金融单位及负收益、负alpha、负Sharpe，不作题目正确率；回撤和波动率为风险指标，需结合收益理解。RL及LLM的输入和动作设定不同，未混入；论文披露的DeepSeek Pro/Flash混合回测也未收录。实盘少于六方法，另行保存。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34510v1","published_at":"2026-09-28","status":"paper_baselines"},"saturation_excluded":true,"scale_label":{"en":"Jensen alpha (%)","zh":"Jensen alpha (%)"},"score_decimals":3,"model_count":24,"result_count":24,"results":[[10503,13.7,370,14612,14613],[10501,25.61,370,14612,14613],[6748,25.87,370,14612,14613],[6811,53.6,370,14612,14613],[9203,39.15,370,14612,14613],[10916,45.34,370,14612,14613],[9681,86.1,370,14612,14613],[9204,37.4,370,14612,14613],[9603,39.33,370,14612,14613],[14614,21.6,370,14612,14613],[14615,20.54,370,14612,14613],[14616,26.3,370,14612,14613],[14617,13.93,370,14612,14613],[6744,8.9,370,14612,14613],[14618,15.32,370,14612,14613],[14619,30.63,370,14612,14613],[14620,31.39,370,14612,14613],[14621,26.87,370,14612,14613],[14622,16.53,370,14612,14613],[14623,1.71,370,14612,14613],[14624,47.97,370,14612,14613],[14625,98.03,370,14612,14613],[14626,28.66,370,14612,14613],[14627,5.54,370,14612,14613]]},{"id":"crypto-ml-2025-backtest-sharpe-2026-paper","name":"Crypto Trading ·2025-backtest sharpe (2026 paper)","group":"数学与科学","metric":"Sharpe ratio","unit":"raw","public_history":true,"protocol":{"en":"This cohort contains only learned ML forecasters with the common52Alpha market factors andten-asset long-only selection formulation. The 2025-backtest panel compares 24 independent ML architectures over the same market timeline. Backtests train on 2021–2023, select configurations on 2024, and test 2025-01-01 through2025-12-31. Prospective paper trading uses newly arriving OKX data from 2026-06-04 through2026-09-21 with the selected configurations frozen. Spot taker cost is0.10 percent per trade. Metrics retain native financial units, including negative returns/alpha/Sharpe; none is percent-correct. Lower drawdown/volatility measures risk andmust be interpreted alongside return. RL and LLM methods have different action/input regimes and are excluded; notably the paper discloses a mixed Pro/Flash DeepSeek backtest, which is not included. Live trading has fewer than six methods and remains separate.","zh":"本组只含共享52个Alpha行情因子、十资产候选及只做多选择规则的学习型ML预测器。2025-backtest面板在相同市场时间线上比较24种独立架构。回测以2021–2023年训练、2024年选配置、2025-01-01至12-31测试；模拟盘用2026-06-04至09-21新到达的OKX行情及冻结配置。现货每交易扣0.10%taker费。保留原生金融单位及负收益、负alpha、负Sharpe，不作题目正确率；回撤和波动率为风险指标，需结合收益理解。RL及LLM的输入和动作设定不同，未混入；论文披露的DeepSeek Pro/Flash混合回测也未收录。实盘少于六方法，另行保存。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34510v1","published_at":"2026-09-28","status":"paper_baselines"},"saturation_excluded":true,"scale_label":{"en":"Sharpe ratio","zh":"Sharpe ratio"},"score_decimals":3,"model_count":24,"result_count":24,"results":[[10503,0.31,370,14612,14613],[10501,0.57,370,14612,14613],[6748,0.63,370,14612,14613],[6811,1.08,370,14612,14613],[9203,1.08,370,14612,14613],[10916,1.27,370,14612,14613],[9681,1.62,370,14612,14613],[9204,1.02,370,14612,14613],[9603,0.82,370,14612,14613],[14614,0.68,370,14612,14613],[14615,0.55,370,14612,14613],[14616,0.7,370,14612,14613],[14617,0.38,370,14612,14613],[6744,0.2,370,14612,14613],[14618,0.33,370,14612,14613],[14619,0.68,370,14612,14613],[14620,0.54,370,14612,14613],[14621,0.46,370,14612,14613],[14622,0.4,370,14612,14613],[14623,-0.05,370,14612,14613],[14624,1.04,370,14612,14613],[14625,1.57,370,14612,14613],[14626,0.62,370,14612,14613],[14627,0.06,370,14612,14613]]},{"id":"crypto-ml-2025-backtest-drawdown-2026-paper","name":"Crypto Trading ·2025-backtest drawdown (2026 paper)","group":"数学与科学","metric":"Maximum drawdown (%)","unit":"raw","public_history":true,"protocol":{"en":"This cohort contains only learned ML forecasters with the common52Alpha market factors andten-asset long-only selection formulation. The 2025-backtest panel compares 24 independent ML architectures over the same market timeline. Backtests train on 2021–2023, select configurations on 2024, and test 2025-01-01 through2025-12-31. Prospective paper trading uses newly arriving OKX data from 2026-06-04 through2026-09-21 with the selected configurations frozen. Spot taker cost is0.10 percent per trade. Metrics retain native financial units, including negative returns/alpha/Sharpe; none is percent-correct. Lower drawdown/volatility measures risk andmust be interpreted alongside return. RL and LLM methods have different action/input regimes and are excluded; notably the paper discloses a mixed Pro/Flash DeepSeek backtest, which is not included. Live trading has fewer than six methods and remains separate.","zh":"本组只含共享52个Alpha行情因子、十资产候选及只做多选择规则的学习型ML预测器。2025-backtest面板在相同市场时间线上比较24种独立架构。回测以2021–2023年训练、2024年选配置、2025-01-01至12-31测试；模拟盘用2026-06-04至09-21新到达的OKX行情及冻结配置。现货每交易扣0.10%taker费。保留原生金融单位及负收益、负alpha、负Sharpe，不作题目正确率；回撤和波动率为风险指标，需结合收益理解。RL及LLM的输入和动作设定不同，未混入；论文披露的DeepSeek Pro/Flash混合回测也未收录。实盘少于六方法，另行保存。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34510v1","published_at":"2026-09-28","status":"paper_baselines"},"saturation_excluded":true,"scale_label":{"en":"Maximum drawdown (%)","zh":"Maximum drawdown (%)"},"score_decimals":3,"higher_is_better":false,"model_count":24,"result_count":24,"results":[[10503,52.05,370,14612,14613],[10501,61.93,370,14612,14613],[6748,44.59,370,14612,14613],[6811,42.42,370,14612,14613],[9203,35.24,370,14612,14613],[10916,32.67,370,14612,14613],[9681,29.1,370,14612,14613],[9204,38.14,370,14612,14613],[9603,44.07,370,14612,14613],[14614,43.95,370,14612,14613],[14615,34.04,370,14612,14613],[14616,51.26,370,14612,14613],[14617,50.3,370,14612,14613],[6744,41.11,370,14612,14613],[14618,35.75,370,14612,14613],[14619,51.19,370,14612,14613],[14620,52.94,370,14612,14613],[14621,45.59,370,14612,14613],[14622,50.06,370,14612,14613],[14623,52.05,370,14612,14613],[14624,48.39,370,14612,14613],[14625,33.97,370,14612,14613],[14626,35.89,370,14612,14613],[14627,58.33,370,14612,14613]]},{"id":"crypto-ml-2025-backtest-volatility-2026-paper","name":"Crypto Trading ·2025-backtest volatility (2026 paper)","group":"数学与科学","metric":"Return volatility (%)","unit":"raw","public_history":true,"protocol":{"en":"This cohort contains only learned ML forecasters with the common52Alpha market factors andten-asset long-only selection formulation. The 2025-backtest panel compares 24 independent ML architectures over the same market timeline. Backtests train on 2021–2023, select configurations on 2024, and test 2025-01-01 through2025-12-31. Prospective paper trading uses newly arriving OKX data from 2026-06-04 through2026-09-21 with the selected configurations frozen. Spot taker cost is0.10 percent per trade. Metrics retain native financial units, including negative returns/alpha/Sharpe; none is percent-correct. Lower drawdown/volatility measures risk andmust be interpreted alongside return. RL and LLM methods have different action/input regimes and are excluded; notably the paper discloses a mixed Pro/Flash DeepSeek backtest, which is not included. Live trading has fewer than six methods and remains separate.","zh":"本组只含共享52个Alpha行情因子、十资产候选及只做多选择规则的学习型ML预测器。2025-backtest面板在相同市场时间线上比较24种独立架构。回测以2021–2023年训练、2024年选配置、2025-01-01至12-31测试；模拟盘用2026-06-04至09-21新到达的OKX行情及冻结配置。现货每交易扣0.10%taker费。保留原生金融单位及负收益、负alpha、负Sharpe，不作题目正确率；回撤和波动率为风险指标，需结合收益理解。RL及LLM的输入和动作设定不同，未混入；论文披露的DeepSeek Pro/Flash混合回测也未收录。实盘少于六方法，另行保存。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34510v1","published_at":"2026-09-28","status":"paper_baselines"},"saturation_excluded":true,"scale_label":{"en":"Return volatility (%)","zh":"Return volatility (%)"},"score_decimals":3,"higher_is_better":false,"model_count":24,"result_count":24,"results":[[10503,42.14,370,14612,14613],[10501,58.07,370,14612,14613],[6748,51.3,370,14612,14613],[6811,59.5,370,14612,14613],[9203,43.06,370,14612,14613],[10916,41.59,370,14612,14613],[9681,59.48,370,14612,14613],[9204,44.49,370,14612,14613],[9603,61.76,370,14612,14613],[14614,41.37,370,14612,14613],[14615,45.8,370,14612,14613],[14616,46.48,370,14612,14613],[14617,42.47,370,14612,14613],[6744,41.64,370,14612,14613],[14618,45.88,370,14612,14613],[14619,55.3,370,14612,14613],[14620,70.66,370,14612,14613],[14621,64.38,370,14612,14613],[14622,41.56,370,14612,14613],[14623,41.72,370,14612,14613],[14624,55.52,370,14612,14613],[14625,70.08,370,14612,14613],[14626,57.12,370,14612,14613],[14627,44.8,370,14612,14613]]},{"id":"crypto-ml-2026-paper-trading-return-2026-paper","name":"Crypto Trading ·2026-paper-trading return (2026 paper)","group":"数学与科学","metric":"Cumulative return (%)","unit":"raw","public_history":true,"protocol":{"en":"This cohort contains only learned ML forecasters with the common52Alpha market factors andten-asset long-only selection formulation. The 2026-paper-trading panel compares 8 independent ML architectures over the same market timeline. Backtests train on 2021–2023, select configurations on 2024, and test 2025-01-01 through2025-12-31. Prospective paper trading uses newly arriving OKX data from 2026-06-04 through2026-09-21 with the selected configurations frozen. Spot taker cost is0.10 percent per trade. Metrics retain native financial units, including negative returns/alpha/Sharpe; none is percent-correct. Lower drawdown/volatility measures risk andmust be interpreted alongside return. RL and LLM methods have different action/input regimes and are excluded; notably the paper discloses a mixed Pro/Flash DeepSeek backtest, which is not included. Live trading has fewer than six methods and remains separate.","zh":"本组只含共享52个Alpha行情因子、十资产候选及只做多选择规则的学习型ML预测器。2026-paper-trading面板在相同市场时间线上比较8种独立架构。回测以2021–2023年训练、2024年选配置、2025-01-01至12-31测试；模拟盘用2026-06-04至09-21新到达的OKX行情及冻结配置。现货每交易扣0.10%taker费。保留原生金融单位及负收益、负alpha、负Sharpe，不作题目正确率；回撤和波动率为风险指标，需结合收益理解。RL及LLM的输入和动作设定不同，未混入；论文披露的DeepSeek Pro/Flash混合回测也未收录。实盘少于六方法，另行保存。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34510v1","published_at":"2026-09-28","status":"paper_baselines"},"saturation_excluded":true,"scale_label":{"en":"Cumulative return (%)","zh":"Cumulative return (%)"},"score_decimals":3,"model_count":8,"result_count":8,"results":[[6811,27.53,370,14628,14629],[9203,-46.79,370,14628,14629],[10916,-50.8,370,14628,14629],[9681,-33.2,370,14628,14629],[9603,34.08,370,14628,14629],[14620,-2.98,370,14628,14629],[14624,-10.78,370,14628,14629],[14625,-8.79,370,14628,14629]]},{"id":"crypto-ml-2026-paper-trading-alpha-2026-paper","name":"Crypto Trading ·2026-paper-trading alpha (2026 paper)","group":"数学与科学","metric":"Jensen alpha (%)","unit":"raw","public_history":true,"protocol":{"en":"This cohort contains only learned ML forecasters with the common52Alpha market factors andten-asset long-only selection formulation. The 2026-paper-trading panel compares 8 independent ML architectures over the same market timeline. Backtests train on 2021–2023, select configurations on 2024, and test 2025-01-01 through2025-12-31. Prospective paper trading uses newly arriving OKX data from 2026-06-04 through2026-09-21 with the selected configurations frozen. Spot taker cost is0.10 percent per trade. Metrics retain native financial units, including negative returns/alpha/Sharpe; none is percent-correct. Lower drawdown/volatility measures risk andmust be interpreted alongside return. RL and LLM methods have different action/input regimes and are excluded; notably the paper discloses a mixed Pro/Flash DeepSeek backtest, which is not included. Live trading has fewer than six methods and remains separate.","zh":"本组只含共享52个Alpha行情因子、十资产候选及只做多选择规则的学习型ML预测器。2026-paper-trading面板在相同市场时间线上比较8种独立架构。回测以2021–2023年训练、2024年选配置、2025-01-01至12-31测试；模拟盘用2026-06-04至09-21新到达的OKX行情及冻结配置。现货每交易扣0.10%taker费。保留原生金融单位及负收益、负alpha、负Sharpe，不作题目正确率；回撤和波动率为风险指标，需结合收益理解。RL及LLM的输入和动作设定不同，未混入；论文披露的DeepSeek Pro/Flash混合回测也未收录。实盘少于六方法，另行保存。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34510v1","published_at":"2026-09-28","status":"paper_baselines"},"saturation_excluded":true,"scale_label":{"en":"Jensen alpha (%)","zh":"Jensen alpha (%)"},"score_decimals":3,"model_count":8,"result_count":8,"results":[[6811,45.32,370,14628,14629],[9203,-251.8,370,14628,14629],[10916,-237.74,370,14628,14629],[9681,-188.46,370,14628,14629],[9603,35.62,370,14628,14629],[14620,-82.15,370,14628,14629],[14624,-97.65,370,14628,14629],[14625,-117.05,370,14628,14629]]},{"id":"crypto-ml-2026-paper-trading-sharpe-2026-paper","name":"Crypto Trading ·2026-paper-trading sharpe (2026 paper)","group":"数学与科学","metric":"Sharpe ratio","unit":"raw","public_history":true,"protocol":{"en":"This cohort contains only learned ML forecasters with the common52Alpha market factors andten-asset long-only selection formulation. The 2026-paper-trading panel compares 8 independent ML architectures over the same market timeline. Backtests train on 2021–2023, select configurations on 2024, and test 2025-01-01 through2025-12-31. Prospective paper trading uses newly arriving OKX data from 2026-06-04 through2026-09-21 with the selected configurations frozen. Spot taker cost is0.10 percent per trade. Metrics retain native financial units, including negative returns/alpha/Sharpe; none is percent-correct. Lower drawdown/volatility measures risk andmust be interpreted alongside return. RL and LLM methods have different action/input regimes and are excluded; notably the paper discloses a mixed Pro/Flash DeepSeek backtest, which is not included. Live trading has fewer than six methods and remains separate.","zh":"本组只含共享52个Alpha行情因子、十资产候选及只做多选择规则的学习型ML预测器。2026-paper-trading面板在相同市场时间线上比较8种独立架构。回测以2021–2023年训练、2024年选配置、2025-01-01至12-31测试；模拟盘用2026-06-04至09-21新到达的OKX行情及冻结配置。现货每交易扣0.10%taker费。保留原生金融单位及负收益、负alpha、负Sharpe，不作题目正确率；回撤和波动率为风险指标，需结合收益理解。RL及LLM的输入和动作设定不同，未混入；论文披露的DeepSeek Pro/Flash混合回测也未收录。实盘少于六方法，另行保存。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34510v1","published_at":"2026-09-28","status":"paper_baselines"},"saturation_excluded":true,"scale_label":{"en":"Sharpe ratio","zh":"Sharpe ratio"},"score_decimals":3,"model_count":8,"result_count":8,"results":[[6811,2.59,370,14628,14629],[9203,-4.77,370,14628,14629],[10916,-8.34,370,14628,14629],[9681,-3.24,370,14628,14629],[9603,1.68,370,14628,14629],[14620,-0.01,370,14628,14629],[14624,-0.98,370,14628,14629],[14625,-0.56,370,14628,14629]]},{"id":"crypto-ml-2026-paper-trading-drawdown-2026-paper","name":"Crypto Trading ·2026-paper-trading drawdown (2026 paper)","group":"数学与科学","metric":"Maximum drawdown (%)","unit":"raw","public_history":true,"protocol":{"en":"This cohort contains only learned ML forecasters with the common52Alpha market factors andten-asset long-only selection formulation. The 2026-paper-trading panel compares 8 independent ML architectures over the same market timeline. Backtests train on 2021–2023, select configurations on 2024, and test 2025-01-01 through2025-12-31. Prospective paper trading uses newly arriving OKX data from 2026-06-04 through2026-09-21 with the selected configurations frozen. Spot taker cost is0.10 percent per trade. Metrics retain native financial units, including negative returns/alpha/Sharpe; none is percent-correct. Lower drawdown/volatility measures risk andmust be interpreted alongside return. RL and LLM methods have different action/input regimes and are excluded; notably the paper discloses a mixed Pro/Flash DeepSeek backtest, which is not included. Live trading has fewer than six methods and remains separate.","zh":"本组只含共享52个Alpha行情因子、十资产候选及只做多选择规则的学习型ML预测器。2026-paper-trading面板在相同市场时间线上比较8种独立架构。回测以2021–2023年训练、2024年选配置、2025-01-01至12-31测试；模拟盘用2026-06-04至09-21新到达的OKX行情及冻结配置。现货每交易扣0.10%taker费。保留原生金融单位及负收益、负alpha、负Sharpe，不作题目正确率；回撤和波动率为风险指标，需结合收益理解。RL及LLM的输入和动作设定不同，未混入；论文披露的DeepSeek Pro/Flash混合回测也未收录。实盘少于六方法，另行保存。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34510v1","published_at":"2026-09-28","status":"paper_baselines"},"saturation_excluded":true,"scale_label":{"en":"Maximum drawdown (%)","zh":"Maximum drawdown (%)"},"score_decimals":3,"higher_is_better":false,"model_count":8,"result_count":8,"results":[[6811,9.56,370,14628,14629],[9203,48.97,370,14628,14629],[10916,52.72,370,14628,14629],[9681,37.45,370,14628,14629],[9603,17.0,370,14628,14629],[14620,20.53,370,14628,14629],[14624,23.91,370,14628,14629],[14625,22.8,370,14628,14629]]},{"id":"crypto-ml-2026-paper-trading-volatility-2026-paper","name":"Crypto Trading ·2026-paper-trading volatility (2026 paper)","group":"数学与科学","metric":"Return volatility (%)","unit":"raw","public_history":true,"protocol":{"en":"This cohort contains only learned ML forecasters with the common52Alpha market factors andten-asset long-only selection formulation. The 2026-paper-trading panel compares 8 independent ML architectures over the same market timeline. Backtests train on 2021–2023, select configurations on 2024, and test 2025-01-01 through2025-12-31. Prospective paper trading uses newly arriving OKX data from 2026-06-04 through2026-09-21 with the selected configurations frozen. Spot taker cost is0.10 percent per trade. Metrics retain native financial units, including negative returns/alpha/Sharpe; none is percent-correct. Lower drawdown/volatility measures risk andmust be interpreted alongside return. RL and LLM methods have different action/input regimes and are excluded; notably the paper discloses a mixed Pro/Flash DeepSeek backtest, which is not included. Live trading has fewer than six methods and remains separate.","zh":"本组只含共享52个Alpha行情因子、十资产候选及只做多选择规则的学习型ML预测器。2026-paper-trading面板在相同市场时间线上比较8种独立架构。回测以2021–2023年训练、2024年选配置、2025-01-01至12-31测试；模拟盘用2026-06-04至09-21新到达的OKX行情及冻结配置。现货每交易扣0.10%taker费。保留原生金融单位及负收益、负alpha、负Sharpe，不作题目正确率；回撤和波动率为风险指标，需结合收益理解。RL及LLM的输入和动作设定不同，未混入；论文披露的DeepSeek Pro/Flash混合回测也未收录。实盘少于六方法，另行保存。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34510v1","published_at":"2026-09-28","status":"paper_baselines"},"saturation_excluded":true,"scale_label":{"en":"Return volatility (%)","zh":"Return volatility (%)"},"score_decimals":3,"higher_is_better":false,"model_count":8,"result_count":8,"results":[[6811,32.15,370,14628,14629],[9203,44.03,370,14628,14629],[10916,27.67,370,14628,14629],[9681,42.81,370,14628,14629],[9603,56.53,370,14628,14629],[14620,46.01,370,14628,14629],[14624,40.13,370,14628,14629],[14625,50.42,370,14628,14629]]},{"id":"cohub-handover-2026-paper","name":"CoHuB ·handover (2026 paper)","group":"Agent 与工具","metric":"Full-task sequential-phase success","unit":"%","public_history":true,"protocol":{"en":"Seven independently trained policy architectures share the same collaborative humanoid task protocol. Each task evaluates100rollouts from the task beginning: robot poses are fixed andobjects/targets are uniformly sampled from declared ranges. The selected handover population has100 rollouts perpolicy; headline macro averages either eight two-humanoid or two three-humanoid tasks. Success requires phase1 thenphase2 thenphase3 against deterministic robot/object-state predicates. Training uses one seed perpolicy, with the declared supervised orpretrained-LoRA recipe. ACT, DP, GauDP, pi0.5, GR00TN1.7, Psi0 and Fast-WAM retain egocentric observations and native policy-sharing configurations. LatentToM is excluded because it lacks three-humanoid coverage. Phase-reset results from 50demonstrations and the single Astra exploratory policy are separate.","zh":"七种独立训练策略共享协作人形机器人任务协议。每任务从起点评测100次，机器人姿态固定，物体和目标按声明范围均匀抽样；本组handover每策略含100次。宏平均分别对八个双机器人任务或两个三机器人任务等权。成功须依次满足三个阶段的确定性机器人和物体状态判据。每策略用一个训练种子，保留原监督或预训练LoRA方案；ACT、DP、GauDP、pi0.5、GR00TN1.7、Psi0和Fast-WAM按原自视角观测及策略共享配置评测。LatentToM缺三机器人覆盖而排除，50演示起点的阶段重置及单一Astra探索结果分开。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34782v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":7,"result_count":7,"results":[[11767,0.65,370,14630,14631],[11770,0.3,370,14630,14631],[14632,0.52,370,14630,14631],[8323,0.02,370,14630,14631],[7601,0.27,370,14630,14631],[10171,0.0,370,14630,14631],[10169,0.02,370,14630,14631]]},{"id":"cohub-pouring-2026-paper","name":"CoHuB ·pouring (2026 paper)","group":"Agent 与工具","metric":"Full-task sequential-phase success","unit":"%","public_history":true,"protocol":{"en":"Seven independently trained policy architectures share the same collaborative humanoid task protocol. Each task evaluates100rollouts from the task beginning: robot poses are fixed andobjects/targets are uniformly sampled from declared ranges. The selected pouring population has100 rollouts perpolicy; headline macro averages either eight two-humanoid or two three-humanoid tasks. Success requires phase1 thenphase2 thenphase3 against deterministic robot/object-state predicates. Training uses one seed perpolicy, with the declared supervised orpretrained-LoRA recipe. ACT, DP, GauDP, pi0.5, GR00TN1.7, Psi0 and Fast-WAM retain egocentric observations and native policy-sharing configurations. LatentToM is excluded because it lacks three-humanoid coverage. Phase-reset results from 50demonstrations and the single Astra exploratory policy are separate.","zh":"七种独立训练策略共享协作人形机器人任务协议。每任务从起点评测100次，机器人姿态固定，物体和目标按声明范围均匀抽样；本组pouring每策略含100次。宏平均分别对八个双机器人任务或两个三机器人任务等权。成功须依次满足三个阶段的确定性机器人和物体状态判据。每策略用一个训练种子，保留原监督或预训练LoRA方案；ACT、DP、GauDP、pi0.5、GR00TN1.7、Psi0和Fast-WAM按原自视角观测及策略共享配置评测。LatentToM缺三机器人覆盖而排除，50演示起点的阶段重置及单一Astra探索结果分开。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34782v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":7,"result_count":7,"results":[[11767,0.01,370,14630,14631],[11770,0.0,370,14630,14631],[14632,0.0,370,14630,14631],[8323,0.0,370,14630,14631],[7601,0.0,370,14630,14631],[10171,0.0,370,14630,14631],[10169,0.07,370,14630,14631]]},{"id":"cohub-frame-hang-2026-paper","name":"CoHuB ·frame-hang (2026 paper)","group":"Agent 与工具","metric":"Full-task sequential-phase success","unit":"%","public_history":true,"protocol":{"en":"Seven independently trained policy architectures share the same collaborative humanoid task protocol. Each task evaluates100rollouts from the task beginning: robot poses are fixed andobjects/targets are uniformly sampled from declared ranges. The selected frame-hang population has100 rollouts perpolicy; headline macro averages either eight two-humanoid or two three-humanoid tasks. Success requires phase1 thenphase2 thenphase3 against deterministic robot/object-state predicates. Training uses one seed perpolicy, with the declared supervised orpretrained-LoRA recipe. ACT, DP, GauDP, pi0.5, GR00TN1.7, Psi0 and Fast-WAM retain egocentric observations and native policy-sharing configurations. LatentToM is excluded because it lacks three-humanoid coverage. Phase-reset results from 50demonstrations and the single Astra exploratory policy are separate.","zh":"七种独立训练策略共享协作人形机器人任务协议。每任务从起点评测100次，机器人姿态固定，物体和目标按声明范围均匀抽样；本组frame-hang每策略含100次。宏平均分别对八个双机器人任务或两个三机器人任务等权。成功须依次满足三个阶段的确定性机器人和物体状态判据。每策略用一个训练种子，保留原监督或预训练LoRA方案；ACT、DP、GauDP、pi0.5、GR00TN1.7、Psi0和Fast-WAM按原自视角观测及策略共享配置评测。LatentToM缺三机器人覆盖而排除，50演示起点的阶段重置及单一Astra探索结果分开。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34782v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":7,"result_count":7,"results":[[11767,0.44,370,14630,14631],[11770,0.26,370,14630,14631],[14632,0.29,370,14630,14631],[8323,0.27,370,14630,14631],[7601,0.44,370,14630,14631],[10171,0.05,370,14630,14631],[10169,0.28,370,14630,14631]]},{"id":"cohub-co-pouring-2026-paper","name":"CoHuB ·co-pouring (2026 paper)","group":"Agent 与工具","metric":"Full-task sequential-phase success","unit":"%","public_history":true,"protocol":{"en":"Seven independently trained policy architectures share the same collaborative humanoid task protocol. Each task evaluates100rollouts from the task beginning: robot poses are fixed andobjects/targets are uniformly sampled from declared ranges. The selected co-pouring population has100 rollouts perpolicy; headline macro averages either eight two-humanoid or two three-humanoid tasks. Success requires phase1 thenphase2 thenphase3 against deterministic robot/object-state predicates. Training uses one seed perpolicy, with the declared supervised orpretrained-LoRA recipe. ACT, DP, GauDP, pi0.5, GR00TN1.7, Psi0 and Fast-WAM retain egocentric observations and native policy-sharing configurations. LatentToM is excluded because it lacks three-humanoid coverage. Phase-reset results from 50demonstrations and the single Astra exploratory policy are separate.","zh":"七种独立训练策略共享协作人形机器人任务协议。每任务从起点评测100次，机器人姿态固定，物体和目标按声明范围均匀抽样；本组co-pouring每策略含100次。宏平均分别对八个双机器人任务或两个三机器人任务等权。成功须依次满足三个阶段的确定性机器人和物体状态判据。每策略用一个训练种子，保留原监督或预训练LoRA方案；ACT、DP、GauDP、pi0.5、GR00TN1.7、Psi0和Fast-WAM按原自视角观测及策略共享配置评测。LatentToM缺三机器人覆盖而排除，50演示起点的阶段重置及单一Astra探索结果分开。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34782v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":7,"result_count":7,"results":[[11767,0.09,370,14630,14631],[11770,0.07,370,14630,14631],[14632,0.0,370,14630,14631],[8323,0.0,370,14630,14631],[7601,0.0,370,14630,14631],[10171,0.0,370,14630,14631],[10169,0.11,370,14630,14631]]},{"id":"cohub-trash-collection-2026-paper","name":"CoHuB ·trash-collection (2026 paper)","group":"Agent 与工具","metric":"Full-task sequential-phase success","unit":"%","public_history":true,"protocol":{"en":"Seven independently trained policy architectures share the same collaborative humanoid task protocol. Each task evaluates100rollouts from the task beginning: robot poses are fixed andobjects/targets are uniformly sampled from declared ranges. The selected trash-collection population has100 rollouts perpolicy; headline macro averages either eight two-humanoid or two three-humanoid tasks. Success requires phase1 thenphase2 thenphase3 against deterministic robot/object-state predicates. Training uses one seed perpolicy, with the declared supervised orpretrained-LoRA recipe. ACT, DP, GauDP, pi0.5, GR00TN1.7, Psi0 and Fast-WAM retain egocentric observations and native policy-sharing configurations. LatentToM is excluded because it lacks three-humanoid coverage. Phase-reset results from 50demonstrations and the single Astra exploratory policy are separate.","zh":"七种独立训练策略共享协作人形机器人任务协议。每任务从起点评测100次，机器人姿态固定，物体和目标按声明范围均匀抽样；本组trash-collection每策略含100次。宏平均分别对八个双机器人任务或两个三机器人任务等权。成功须依次满足三个阶段的确定性机器人和物体状态判据。每策略用一个训练种子，保留原监督或预训练LoRA方案；ACT、DP、GauDP、pi0.5、GR00TN1.7、Psi0和Fast-WAM按原自视角观测及策略共享配置评测。LatentToM缺三机器人覆盖而排除，50演示起点的阶段重置及单一Astra探索结果分开。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34782v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":7,"result_count":7,"results":[[11767,0.01,370,14630,14631],[11770,0.01,370,14630,14631],[14632,0.01,370,14630,14631],[8323,0.0,370,14630,14631],[7601,0.01,370,14630,14631],[10171,0.03,370,14630,14631],[10169,0.01,370,14630,14631]]},{"id":"cohub-cart-service-2026-paper","name":"CoHuB ·cart-service (2026 paper)","group":"Agent 与工具","metric":"Full-task sequential-phase success","unit":"%","public_history":true,"protocol":{"en":"Seven independently trained policy architectures share the same collaborative humanoid task protocol. Each task evaluates100rollouts from the task beginning: robot poses are fixed andobjects/targets are uniformly sampled from declared ranges. The selected cart-service population has100 rollouts perpolicy; headline macro averages either eight two-humanoid or two three-humanoid tasks. Success requires phase1 thenphase2 thenphase3 against deterministic robot/object-state predicates. Training uses one seed perpolicy, with the declared supervised orpretrained-LoRA recipe. ACT, DP, GauDP, pi0.5, GR00TN1.7, Psi0 and Fast-WAM retain egocentric observations and native policy-sharing configurations. LatentToM is excluded because it lacks three-humanoid coverage. Phase-reset results from 50demonstrations and the single Astra exploratory policy are separate.","zh":"七种独立训练策略共享协作人形机器人任务协议。每任务从起点评测100次，机器人姿态固定，物体和目标按声明范围均匀抽样；本组cart-service每策略含100次。宏平均分别对八个双机器人任务或两个三机器人任务等权。成功须依次满足三个阶段的确定性机器人和物体状态判据。每策略用一个训练种子，保留原监督或预训练LoRA方案；ACT、DP、GauDP、pi0.5、GR00TN1.7、Psi0和Fast-WAM按原自视角观测及策略共享配置评测。LatentToM缺三机器人覆盖而排除，50演示起点的阶段重置及单一Astra探索结果分开。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34782v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":7,"result_count":7,"results":[[11767,0.03,370,14630,14631],[11770,0.03,370,14630,14631],[14632,0.0,370,14630,14631],[8323,0.04,370,14630,14631],[7601,0.0,370,14630,14631],[10171,0.0,370,14630,14631],[10169,0.0,370,14630,14631]]},{"id":"cohub-co-carry-2026-paper","name":"CoHuB ·co-carry (2026 paper)","group":"Agent 与工具","metric":"Full-task sequential-phase success","unit":"%","public_history":true,"protocol":{"en":"Seven independently trained policy architectures share the same collaborative humanoid task protocol. Each task evaluates100rollouts from the task beginning: robot poses are fixed andobjects/targets are uniformly sampled from declared ranges. The selected co-carry population has100 rollouts perpolicy; headline macro averages either eight two-humanoid or two three-humanoid tasks. Success requires phase1 thenphase2 thenphase3 against deterministic robot/object-state predicates. Training uses one seed perpolicy, with the declared supervised orpretrained-LoRA recipe. ACT, DP, GauDP, pi0.5, GR00TN1.7, Psi0 and Fast-WAM retain egocentric observations and native policy-sharing configurations. LatentToM is excluded because it lacks three-humanoid coverage. Phase-reset results from 50demonstrations and the single Astra exploratory policy are separate.","zh":"七种独立训练策略共享协作人形机器人任务协议。每任务从起点评测100次，机器人姿态固定，物体和目标按声明范围均匀抽样；本组co-carry每策略含100次。宏平均分别对八个双机器人任务或两个三机器人任务等权。成功须依次满足三个阶段的确定性机器人和物体状态判据。每策略用一个训练种子，保留原监督或预训练LoRA方案；ACT、DP、GauDP、pi0.5、GR00TN1.7、Psi0和Fast-WAM按原自视角观测及策略共享配置评测。LatentToM缺三机器人覆盖而排除，50演示起点的阶段重置及单一Astra探索结果分开。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34782v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":7,"result_count":7,"results":[[11767,0.61,370,14630,14631],[11770,0.12,370,14630,14631],[14632,0.09,370,14630,14631],[8323,0.01,370,14630,14631],[7601,0.02,370,14630,14631],[10171,0.0,370,14630,14631],[10169,0.31,370,14630,14631]]},{"id":"cohub-table-align-2026-paper","name":"CoHuB ·table-align (2026 paper)","group":"Agent 与工具","metric":"Full-task sequential-phase success","unit":"%","public_history":true,"protocol":{"en":"Seven independently trained policy architectures share the same collaborative humanoid task protocol. Each task evaluates100rollouts from the task beginning: robot poses are fixed andobjects/targets are uniformly sampled from declared ranges. The selected table-align population has100 rollouts perpolicy; headline macro averages either eight two-humanoid or two three-humanoid tasks. Success requires phase1 thenphase2 thenphase3 against deterministic robot/object-state predicates. Training uses one seed perpolicy, with the declared supervised orpretrained-LoRA recipe. ACT, DP, GauDP, pi0.5, GR00TN1.7, Psi0 and Fast-WAM retain egocentric observations and native policy-sharing configurations. LatentToM is excluded because it lacks three-humanoid coverage. Phase-reset results from 50demonstrations and the single Astra exploratory policy are separate.","zh":"七种独立训练策略共享协作人形机器人任务协议。每任务从起点评测100次，机器人姿态固定，物体和目标按声明范围均匀抽样；本组table-align每策略含100次。宏平均分别对八个双机器人任务或两个三机器人任务等权。成功须依次满足三个阶段的确定性机器人和物体状态判据。每策略用一个训练种子，保留原监督或预训练LoRA方案；ACT、DP、GauDP、pi0.5、GR00TN1.7、Psi0和Fast-WAM按原自视角观测及策略共享配置评测。LatentToM缺三机器人覆盖而排除，50演示起点的阶段重置及单一Astra探索结果分开。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34782v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":7,"result_count":7,"results":[[11767,0.39,370,14630,14631],[11770,0.04,370,14630,14631],[14632,0.16,370,14630,14631],[8323,0.02,370,14630,14631],[7601,0.01,370,14630,14631],[10171,0.0,370,14630,14631],[10169,0.17,370,14630,14631]]},{"id":"cohub-two-humanoid-macro-2026-paper","name":"CoHuB ·two-humanoid-macro (2026 paper)","group":"Agent 与工具","metric":"Full-task sequential-phase success","unit":"%","public_history":true,"protocol":{"en":"Seven independently trained policy architectures share the same collaborative humanoid task protocol. Each task evaluates100rollouts from the task beginning: robot poses are fixed andobjects/targets are uniformly sampled from declared ranges. The selected two-humanoid-macro population has800 rollouts perpolicy; headline macro averages either eight two-humanoid or two three-humanoid tasks. Success requires phase1 thenphase2 thenphase3 against deterministic robot/object-state predicates. Training uses one seed perpolicy, with the declared supervised orpretrained-LoRA recipe. ACT, DP, GauDP, pi0.5, GR00TN1.7, Psi0 and Fast-WAM retain egocentric observations and native policy-sharing configurations. LatentToM is excluded because it lacks three-humanoid coverage. Phase-reset results from 50demonstrations and the single Astra exploratory policy are separate.","zh":"七种独立训练策略共享协作人形机器人任务协议。每任务从起点评测100次，机器人姿态固定，物体和目标按声明范围均匀抽样；本组two-humanoid-macro每策略含800次。宏平均分别对八个双机器人任务或两个三机器人任务等权。成功须依次满足三个阶段的确定性机器人和物体状态判据。每策略用一个训练种子，保留原监督或预训练LoRA方案；ACT、DP、GauDP、pi0.5、GR00TN1.7、Psi0和Fast-WAM按原自视角观测及策略共享配置评测。LatentToM缺三机器人覆盖而排除，50演示起点的阶段重置及单一Astra探索结果分开。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34782v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":7,"result_count":7,"results":[[11767,0.27899999999999997,370,14630,14631],[11770,0.10400000000000001,370,14630,14631],[14632,0.134,370,14630,14631],[8323,0.045,370,14630,14631],[7601,0.094,370,14630,14631],[10171,0.01,370,14630,14631],[10169,0.121,370,14630,14631]]},{"id":"cohub-move-house-2026-paper","name":"CoHuB ·move-house (2026 paper)","group":"Agent 与工具","metric":"Full-task sequential-phase success","unit":"%","public_history":true,"protocol":{"en":"Seven independently trained policy architectures share the same collaborative humanoid task protocol. Each task evaluates100rollouts from the task beginning: robot poses are fixed andobjects/targets are uniformly sampled from declared ranges. The selected move-house population has100 rollouts perpolicy; headline macro averages either eight two-humanoid or two three-humanoid tasks. Success requires phase1 thenphase2 thenphase3 against deterministic robot/object-state predicates. Training uses one seed perpolicy, with the declared supervised orpretrained-LoRA recipe. ACT, DP, GauDP, pi0.5, GR00TN1.7, Psi0 and Fast-WAM retain egocentric observations and native policy-sharing configurations. LatentToM is excluded because it lacks three-humanoid coverage. Phase-reset results from 50demonstrations and the single Astra exploratory policy are separate.","zh":"七种独立训练策略共享协作人形机器人任务协议。每任务从起点评测100次，机器人姿态固定，物体和目标按声明范围均匀抽样；本组move-house每策略含100次。宏平均分别对八个双机器人任务或两个三机器人任务等权。成功须依次满足三个阶段的确定性机器人和物体状态判据。每策略用一个训练种子，保留原监督或预训练LoRA方案；ACT、DP、GauDP、pi0.5、GR00TN1.7、Psi0和Fast-WAM按原自视角观测及策略共享配置评测。LatentToM缺三机器人覆盖而排除，50演示起点的阶段重置及单一Astra探索结果分开。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34782v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":7,"result_count":7,"results":[[11767,0.09,370,14630,14631],[11770,0.04,370,14630,14631],[14632,0.02,370,14630,14631],[8323,0.03,370,14630,14631],[7601,0.0,370,14630,14631],[10171,0.0,370,14630,14631],[10169,0.2,370,14630,14631]]},{"id":"cohub-big-table-2026-paper","name":"CoHuB ·big-table (2026 paper)","group":"Agent 与工具","metric":"Full-task sequential-phase success","unit":"%","public_history":true,"protocol":{"en":"Seven independently trained policy architectures share the same collaborative humanoid task protocol. Each task evaluates100rollouts from the task beginning: robot poses are fixed andobjects/targets are uniformly sampled from declared ranges. The selected big-table population has100 rollouts perpolicy; headline macro averages either eight two-humanoid or two three-humanoid tasks. Success requires phase1 thenphase2 thenphase3 against deterministic robot/object-state predicates. Training uses one seed perpolicy, with the declared supervised orpretrained-LoRA recipe. ACT, DP, GauDP, pi0.5, GR00TN1.7, Psi0 and Fast-WAM retain egocentric observations and native policy-sharing configurations. LatentToM is excluded because it lacks three-humanoid coverage. Phase-reset results from 50demonstrations and the single Astra exploratory policy are separate.","zh":"七种独立训练策略共享协作人形机器人任务协议。每任务从起点评测100次，机器人姿态固定，物体和目标按声明范围均匀抽样；本组big-table每策略含100次。宏平均分别对八个双机器人任务或两个三机器人任务等权。成功须依次满足三个阶段的确定性机器人和物体状态判据。每策略用一个训练种子，保留原监督或预训练LoRA方案；ACT、DP、GauDP、pi0.5、GR00TN1.7、Psi0和Fast-WAM按原自视角观测及策略共享配置评测。LatentToM缺三机器人覆盖而排除，50演示起点的阶段重置及单一Astra探索结果分开。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34782v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":7,"result_count":7,"results":[[11767,0.8,370,14630,14631],[11770,0.14,370,14630,14631],[14632,0.54,370,14630,14631],[8323,0.37,370,14630,14631],[7601,0.05,370,14630,14631],[10171,0.19,370,14630,14631],[10169,0.69,370,14630,14631]]},{"id":"cohub-three-humanoid-macro-2026-paper","name":"CoHuB ·three-humanoid-macro (2026 paper)","group":"Agent 与工具","metric":"Full-task sequential-phase success","unit":"%","public_history":true,"protocol":{"en":"Seven independently trained policy architectures share the same collaborative humanoid task protocol. Each task evaluates100rollouts from the task beginning: robot poses are fixed andobjects/targets are uniformly sampled from declared ranges. The selected three-humanoid-macro population has200 rollouts perpolicy; headline macro averages either eight two-humanoid or two three-humanoid tasks. Success requires phase1 thenphase2 thenphase3 against deterministic robot/object-state predicates. Training uses one seed perpolicy, with the declared supervised orpretrained-LoRA recipe. ACT, DP, GauDP, pi0.5, GR00TN1.7, Psi0 and Fast-WAM retain egocentric observations and native policy-sharing configurations. LatentToM is excluded because it lacks three-humanoid coverage. Phase-reset results from 50demonstrations and the single Astra exploratory policy are separate.","zh":"七种独立训练策略共享协作人形机器人任务协议。每任务从起点评测100次，机器人姿态固定，物体和目标按声明范围均匀抽样；本组three-humanoid-macro每策略含200次。宏平均分别对八个双机器人任务或两个三机器人任务等权。成功须依次满足三个阶段的确定性机器人和物体状态判据。每策略用一个训练种子，保留原监督或预训练LoRA方案；ACT、DP、GauDP、pi0.5、GR00TN1.7、Psi0和Fast-WAM按原自视角观测及策略共享配置评测。LatentToM缺三机器人覆盖而排除，50演示起点的阶段重置及单一Astra探索结果分开。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34782v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":7,"result_count":7,"results":[[11767,0.445,370,14630,14631],[11770,0.09,370,14630,14631],[14632,0.28,370,14630,14631],[8323,0.2,370,14630,14631],[7601,0.025,370,14630,14631],[10171,0.095,370,14630,14631],[10169,0.445,370,14630,14631]]},{"id":"gpuphys-numerical-correctness-2026-paper","name":"GPUPhysBench ·numerical-correctness (2026 paper)","group":"编程","metric":"numerical-correctness","unit":"%","public_history":true,"protocol":{"en":"Six distinct model+harness systems share50CUDA physics tasks with one final attempt each. Development and scoring use the same RTX4090 inside the fixed CUDA container. Operator/full-simulator wall-clock limits are30/60minutes; references and hidden evaluators are inaccessible, online access andparallel agents prohibited. FP32 outputs must satisfy public andhidden-seed numerical tests. Correctness checks building, running andnumerics; pass rate additionally requires interface/timing/security audits. fast_p counts all 50 tasks whose passing submission exceeds reference/generated hidden-input speedup p; failed tasks contribute zero. Timings use the minimum offive fresh-process calls afterthree warmups. Original provider reasoning settings/harnesses are retained, not interpreted as identical inference budgets.","zh":"六个独立模型与harness共享50个CUDA物理计算任务，每题一次最终尝试，开发与评分均用固定容器中的RTX4090。算子及完整模拟器各限30/60分钟，参考实现与隐藏评测不可见，禁止联网和并行Agent。FP32输出须通过公开和隐藏种子的数值测试。正确率核对构建、执行和数值；通过率还要求接口、计时与行为审计合格。fast_p以全部50题为分母，要求已通过且隐藏输入的参考/生成用时比大于p，失败题记零。计时取三次预热后的五次独立进程调用最小值，保留各模型原推理与harness设置。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.35639v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[4983,1.0,370,14633,14634],[4979,1.0,370,14635,14634],[2129,0.88,370,14636,14634],[3153,0.86,370,14637,14634],[5133,0.52,370,14638,14634],[3154,0.88,370,14639,14634]]},{"id":"gpuphys-audit-passing-2026-paper","name":"GPUPhysBench ·audit-passing (2026 paper)","group":"编程","metric":"audit-passing","unit":"%","public_history":true,"protocol":{"en":"Six distinct model+harness systems share50CUDA physics tasks with one final attempt each. Development and scoring use the same RTX4090 inside the fixed CUDA container. Operator/full-simulator wall-clock limits are30/60minutes; references and hidden evaluators are inaccessible, online access andparallel agents prohibited. FP32 outputs must satisfy public andhidden-seed numerical tests. Correctness checks building, running andnumerics; pass rate additionally requires interface/timing/security audits. fast_p counts all 50 tasks whose passing submission exceeds reference/generated hidden-input speedup p; failed tasks contribute zero. Timings use the minimum offive fresh-process calls afterthree warmups. Original provider reasoning settings/harnesses are retained, not interpreted as identical inference budgets.","zh":"六个独立模型与harness共享50个CUDA物理计算任务，每题一次最终尝试，开发与评分均用固定容器中的RTX4090。算子及完整模拟器各限30/60分钟，参考实现与隐藏评测不可见，禁止联网和并行Agent。FP32输出须通过公开和隐藏种子的数值测试。正确率核对构建、执行和数值；通过率还要求接口、计时与行为审计合格。fast_p以全部50题为分母，要求已通过且隐藏输入的参考/生成用时比大于p，失败题记零。计时取三次预热后的五次独立进程调用最小值，保留各模型原推理与harness设置。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.35639v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[4983,1.0,370,14633,14634],[4979,1.0,370,14635,14634],[2129,0.88,370,14636,14634],[3153,0.86,370,14637,14634],[5133,0.52,370,14638,14634],[3154,0.86,370,14639,14634]]},{"id":"gpuphys-fast-0p5-2026-paper","name":"GPUPhysBench ·fast-0.5 (2026 paper)","group":"编程","metric":"fast-0.5","unit":"%","public_history":true,"protocol":{"en":"Six distinct model+harness systems share50CUDA physics tasks with one final attempt each. Development and scoring use the same RTX4090 inside the fixed CUDA container. Operator/full-simulator wall-clock limits are30/60minutes; references and hidden evaluators are inaccessible, online access andparallel agents prohibited. FP32 outputs must satisfy public andhidden-seed numerical tests. Correctness checks building, running andnumerics; pass rate additionally requires interface/timing/security audits. fast_p counts all 50 tasks whose passing submission exceeds reference/generated hidden-input speedup p; failed tasks contribute zero. Timings use the minimum offive fresh-process calls afterthree warmups. Original provider reasoning settings/harnesses are retained, not interpreted as identical inference budgets.","zh":"六个独立模型与harness共享50个CUDA物理计算任务，每题一次最终尝试，开发与评分均用固定容器中的RTX4090。算子及完整模拟器各限30/60分钟，参考实现与隐藏评测不可见，禁止联网和并行Agent。FP32输出须通过公开和隐藏种子的数值测试。正确率核对构建、执行和数值；通过率还要求接口、计时与行为审计合格。fast_p以全部50题为分母，要求已通过且隐藏输入的参考/生成用时比大于p，失败题记零。计时取三次预热后的五次独立进程调用最小值，保留各模型原推理与harness设置。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.35639v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[4983,0.66,370,14633,14634],[4979,0.42,370,14635,14634],[2129,0.28,370,14636,14634],[3153,0.38,370,14637,14634],[5133,0.34,370,14638,14634],[3154,0.34,370,14639,14634]]},{"id":"gpuphys-fast-0p9-2026-paper","name":"GPUPhysBench ·fast-0.9 (2026 paper)","group":"编程","metric":"fast-0.9","unit":"%","public_history":true,"protocol":{"en":"Six distinct model+harness systems share50CUDA physics tasks with one final attempt each. Development and scoring use the same RTX4090 inside the fixed CUDA container. Operator/full-simulator wall-clock limits are30/60minutes; references and hidden evaluators are inaccessible, online access andparallel agents prohibited. FP32 outputs must satisfy public andhidden-seed numerical tests. Correctness checks building, running andnumerics; pass rate additionally requires interface/timing/security audits. fast_p counts all 50 tasks whose passing submission exceeds reference/generated hidden-input speedup p; failed tasks contribute zero. Timings use the minimum offive fresh-process calls afterthree warmups. Original provider reasoning settings/harnesses are retained, not interpreted as identical inference budgets.","zh":"六个独立模型与harness共享50个CUDA物理计算任务，每题一次最终尝试，开发与评分均用固定容器中的RTX4090。算子及完整模拟器各限30/60分钟，参考实现与隐藏评测不可见，禁止联网和并行Agent。FP32输出须通过公开和隐藏种子的数值测试。正确率核对构建、执行和数值；通过率还要求接口、计时与行为审计合格。fast_p以全部50题为分母，要求已通过且隐藏输入的参考/生成用时比大于p，失败题记零。计时取三次预热后的五次独立进程调用最小值，保留各模型原推理与harness设置。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.35639v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[4983,0.22,370,14633,14634],[4979,0.16,370,14635,14634],[2129,0.16,370,14636,14634],[3153,0.16,370,14637,14634],[5133,0.14,370,14638,14634],[3154,0.16,370,14639,14634]]},{"id":"gpuphys-fast-1p05-2026-paper","name":"GPUPhysBench ·fast-1.05 (2026 paper)","group":"编程","metric":"fast-1.05","unit":"%","public_history":true,"protocol":{"en":"Six distinct model+harness systems share50CUDA physics tasks with one final attempt each. Development and scoring use the same RTX4090 inside the fixed CUDA container. Operator/full-simulator wall-clock limits are30/60minutes; references and hidden evaluators are inaccessible, online access andparallel agents prohibited. FP32 outputs must satisfy public andhidden-seed numerical tests. Correctness checks building, running andnumerics; pass rate additionally requires interface/timing/security audits. fast_p counts all 50 tasks whose passing submission exceeds reference/generated hidden-input speedup p; failed tasks contribute zero. Timings use the minimum offive fresh-process calls afterthree warmups. Original provider reasoning settings/harnesses are retained, not interpreted as identical inference budgets.","zh":"六个独立模型与harness共享50个CUDA物理计算任务，每题一次最终尝试，开发与评分均用固定容器中的RTX4090。算子及完整模拟器各限30/60分钟，参考实现与隐藏评测不可见，禁止联网和并行Agent。FP32输出须通过公开和隐藏种子的数值测试。正确率核对构建、执行和数值；通过率还要求接口、计时与行为审计合格。fast_p以全部50题为分母，要求已通过且隐藏输入的参考/生成用时比大于p，失败题记零。计时取三次预热后的五次独立进程调用最小值，保留各模型原推理与harness设置。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.35639v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[4983,0.0,370,14633,14634],[4979,0.0,370,14635,14634],[2129,0.0,370,14636,14634],[3153,0.0,370,14637,14634],[5133,0.0,370,14638,14634],[3154,0.0,370,14639,14634]]},{"id":"dgf-facts-gates-2026-paper","name":"DGF-Bench ·facts-gates (2026 paper)","group":"安全与可靠性","metric":"Outcome-strict gate correctness","unit":"%","public_history":true,"protocol":{"en":"Six independent model endpoints evaluate the same 15development dossiers (seeds20000–20014, difficulty4),85gates and 88reference findings in the facts condition. The common authoritative facts make every gate decidable; strict gate success requires disposition, findings, actions andauthorization to match the executable61-rule reference. A route passes only if every gate does. This metric denominator is85. Every gate allows30 turns,60tool calls and 16384output tokens/turn; provider defaults or temperature0 and pinned open-weight serving/quantization are explicit. All routes completed, including resumed provider errors. This is the facts-only governance condition; attack batteries mix applicability andimage support and are not imported as a common six-model attack score.","zh":"六个独立模型端点评测相同15份开发档案（种子20000–20014、难度4），共85个gate、88条参考发现；此为提供权威事实的facts条件，每个gate均可由共同事实判定。严格正确要求处置、发现、动作和授权全部符合61条可执行规则，整条route则要求所有gate正确。本指标分母85。每gate限30轮、60次工具调用、每轮16384输出词元；保留提供商默认或温度0及固定的开源权重服务和量化。全部route已完成，提供商错误按gate恢复。此处仅为事实条件，攻击电池的适用项和图像支持不同，未当共同六模型攻击率导入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34913v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[5731,1.0,370,14640,14641],[2983,1.0,370,14640,14641],[14642,1.0,370,14640,14641],[3492,1.0,370,14640,14641],[3515,1.0,370,14640,14641],[7179,0.5529411764705883,370,14640,14641]]},{"id":"dgf-facts-routes-2026-paper","name":"DGF-Bench ·facts-routes (2026 paper)","group":"安全与可靠性","metric":"All-gates-correct route success","unit":"%","public_history":true,"protocol":{"en":"Six independent model endpoints evaluate the same 15development dossiers (seeds20000–20014, difficulty4),85gates and 88reference findings in the facts condition. The common authoritative facts make every gate decidable; strict gate success requires disposition, findings, actions andauthorization to match the executable61-rule reference. A route passes only if every gate does. This metric denominator is15. Every gate allows30 turns,60tool calls and 16384output tokens/turn; provider defaults or temperature0 and pinned open-weight serving/quantization are explicit. All routes completed, including resumed provider errors. This is the facts-only governance condition; attack batteries mix applicability andimage support and are not imported as a common six-model attack score.","zh":"六个独立模型端点评测相同15份开发档案（种子20000–20014、难度4），共85个gate、88条参考发现；此为提供权威事实的facts条件，每个gate均可由共同事实判定。严格正确要求处置、发现、动作和授权全部符合61条可执行规则，整条route则要求所有gate正确。本指标分母15。每gate限30轮、60次工具调用、每轮16384输出词元；保留提供商默认或温度0及固定的开源权重服务和量化。全部route已完成，提供商错误按gate恢复。此处仅为事实条件，攻击电池的适用项和图像支持不同，未当共同六模型攻击率导入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34913v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[5731,1.0,370,14640,14641],[2983,1.0,370,14640,14641],[14642,1.0,370,14640,14641],[3492,1.0,370,14640,14641],[3515,1.0,370,14640,14641],[7179,0.13333333333333333,370,14640,14641]]},{"id":"memoreason-fictitious-2026-paper","name":"MemoReason ·fictitious (2026 paper)","group":"知识与推理","metric":"Exact-or-semantic answer accuracy","unit":"%","public_history":true,"protocol":{"en":"Nine reported configurations cover eight conservative base identities on the same 12000 questions in the fictitious setting. One hundred human-curated factual documents each contain12 questions; ten coherent entity-replacement variants per document preserve question structure and yield12000fictitious questions. Each question is evaluated independently at temperature0; GPT-OSS effort islow. Scoring first applies active exact-answer references, then GPT-OSS120B semantic JudgeMatch for eligible misses (Groq, T0, seed23,256initial tokens, loweffort), validated against200human labels. The checked-in paper analysis rejects missing/duplicate questions, stale references, incomplete scoring andmismatched conditions. Factual andfictitious populations are kept separate; conditional shortcut rates and paired differences are not ordinary task accuracy.","zh":"九条配置覆盖保守计数的八种基模型身份，共同评测fictitious条件的12000题。100份人工整理的事实文档各12题，每份生成十种符合规则的实体替换版本，保持问题结构，得到12000道虚构条件题。每题独立评测，温度0，GPT-OSS推理effort为low。先按固定有效参考作精确匹配，再由GPT-OSS120B对合适的未匹配回答作语义评审（Groq、温度0、种子23、初始256词元、low），并用200个人工标签核对。作者复现分析拒绝缺题、重复、过期参考、不完整评分和条件不一致。事实与虚构题分组，条件捷径率和配对差值不作普通任务正确率。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.35312v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":8,"result_count":9,"results":[[14643,0.9229999999999999,370,14644,14645],[6278,0.9028,370,14646,14645],[14647,0.888,370,14648,14645],[14649,0.8765000000000001,370,14650,14645],[14651,0.8634000000000001,370,14652,14645],[14653,0.8591,370,14654,14645],[14655,0.7687999999999999,370,14656,14645],[10725,0.7561,370,14657,14645],[10725,0.6642,370,14658,14645]]},{"id":"memoreason-factual-2026-paper","name":"MemoReason ·factual (2026 paper)","group":"知识与推理","metric":"Exact-or-semantic answer accuracy","unit":"%","public_history":true,"protocol":{"en":"Nine reported configurations cover eight conservative base identities on the same 1200 questions in the factual setting. One hundred human-curated factual documents each contain12 questions; ten coherent entity-replacement variants per document preserve question structure and yield12000fictitious questions. Each question is evaluated independently at temperature0; GPT-OSS effort islow. Scoring first applies active exact-answer references, then GPT-OSS120B semantic JudgeMatch for eligible misses (Groq, T0, seed23,256initial tokens, loweffort), validated against200human labels. The checked-in paper analysis rejects missing/duplicate questions, stale references, incomplete scoring andmismatched conditions. Factual andfictitious populations are kept separate; conditional shortcut rates and paired differences are not ordinary task accuracy.","zh":"九条配置覆盖保守计数的八种基模型身份，共同评测factual条件的1200题。100份人工整理的事实文档各12题，每份生成十种符合规则的实体替换版本，保持问题结构，得到12000道虚构条件题。每题独立评测，温度0，GPT-OSS推理effort为low。先按固定有效参考作精确匹配，再由GPT-OSS120B对合适的未匹配回答作语义评审（Groq、温度0、种子23、初始256词元、low），并用200个人工标签核对。作者复现分析拒绝缺题、重复、过期参考、不完整评分和条件不一致。事实与虚构题分组，条件捷径率和配对差值不作普通任务正确率。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.35312v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":8,"result_count":9,"results":[[14643,0.9567,370,14644,14645],[6278,0.9367,370,14646,14645],[14647,0.9175,370,14648,14645],[14649,0.9075,370,14650,14645],[14651,0.8908,370,14652,14645],[14653,0.8725,370,14654,14645],[14655,0.7825,370,14656,14645],[10725,0.8092,370,14657,14645],[10725,0.6925,370,14658,14645]]},{"id":"physfield-overall-2026-paper","name":"PhysFieldBench ·overall (2026 paper)","group":"数学与科学","metric":"Task-macro chance-normalized index","unit":"raw","public_history":true,"protocol":{"en":"Twelve zero-shot MLLM checkpoints share all 1,160examples across 24leaf tasks. Each case supplies the same one or two rendered field images, question andoptions. Per-task accuracy counts invalid/conflicting andstrict-truncated responses asincorrect; NormScore=100*(accuracy-1/K)/(1-1/K), then macro-averages the 24 tasks. This index can be negative and is not percent-correct. Local instruct baselines use greedy nonthinking decoding with 64 tokens; API models useT0,16384 tokens andhigh reasoning effort where exposed. The separately task-trained ViT andall post-trained Qwen variants are excluded from this zero-shot panel. Source task construction uses physically grounded labels and checks leakage/ambiguous comparisons.","zh":"十二个零样本多模态权重共享24个子任务的1160题，每题使用相同的一或两张场图、问题和选项。无效、冲突及不符合严格截断规则的回答记错；先以100×(正确率−1/K)/(1−1/K)扣除逐任务随机水平，再对24任务等权。该指数可为负数，不是正确率。两个本地指令模型使用贪心、非思考、64词元；API模型温度0、16384词元，支持时采用high推理。逐任务监督训练的ViT及后训练Qwen变体不混入零样本组；数据以物理依据定答案并检查泄漏和比较歧义。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34072v1","published_at":"2026-09-28","status":"paper_baselines"},"saturation_excluded":true,"scale_label":{"en":"Task-macro chance-normalized index","zh":"Task-macro chance-normalized index"},"score_decimals":3,"model_count":12,"result_count":12,"results":[[6150,2.7,370,14659,14660],[7485,-1.9,370,14661,14660],[7853,13.7,370,14662,14660],[3490,23.9,370,14663,14660],[7937,19.2,370,14664,14660],[3189,25.9,370,14665,14660],[7674,23.3,370,14666,14660],[3578,6.9,370,14667,14660],[3424,25.9,370,14668,14660],[3412,25.7,370,14669,14660],[3415,24.7,370,14670,14660],[3196,29.3,370,14671,14660]]},{"id":"multi-swt-lite-success-2026-paper","name":"Multi-SWT-Bench Lite (2026 paper)","group":"编程","metric":"Executable reproduction-test success","unit":"%","public_history":true,"protocol":{"en":"Twelve model+harness configurations cover six independent backbones on the same 340-instance Lite split, which the paper explicitly uses for every experiment. The full 1,963-instance corpus is not the evaluated denominator. Success requires at least one generated test to fail before andpass after the golden patch, andno generated test to fail after it. Original developer tests validate each instance before inclusion. All four harnesses are adapted to reproduction testing; MSWE-agent/MOpenHands use API endpoints and Codex/ClaudeCode subscription-native settings. Harness andoriginal checkpoint remain explicit; inference budgets are not claimed equal. Conditional change coverage andpatch well-formedness are not substituted for reproduction success.","zh":"十二条模型加harness配置覆盖六种独立模型，全部实验明确使用同一340题Lite集合，不把1963题全量语料当评测分母。成功要求至少一条生成测试在原代码失败、黄金修复后通过，且修复后没有生成测试失败。入选题先由开发者原测试验证。四种harness均适配复现测试；MSWE-agent和MOpenHands使用API，Codex与ClaudeCode保留订阅方案下原生设置。原模型与harness写明，不假定推理预算相同。条件代码覆盖和补丁格式合格率不替代复现成功率。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34752v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":6,"result_count":12,"results":[[3408,0.4,370,14672,14673],[3568,0.3118,370,14674,14673],[3201,0.1176,370,14675,14673],[3185,0.1765,370,14676,14673],[3408,0.47350000000000003,370,14677,14673],[3568,0.5794,370,14678,14673],[3201,0.5529,370,14679,14673],[3185,0.4824,370,14680,14673],[3408,0.4353,370,14681,14673],[2958,0.5647,370,14682,14673],[3568,0.5118,370,14683,14673],[755,0.6529,370,14684,14673]]},{"id":"jrdb-avr-answer-monolithic-2026-paper","name":"JRDB-AVR ·answer monolithic (2026 paper)","group":"视觉与视频","metric":"Answer correctness","unit":"%","public_history":true,"protocol":{"en":"Six independent VLM backbones share all 2098 questions from 27JRDB test sequences within this scaffold. Four scaffolds stay separate: fixed panoramic context, extra reasoning over the same context, one proposal-driven observation, anditerative ReAct. All use zero-shot prompts andthe same answer schema. Official repository README fixes exact answer tolerances: choice index;frames within1second;temporal IoU>=0.5;numeric prediction/gold ratio0.5–2.0;frame-viewpoint within1second/20degrees. Every question remains in the full denominator. The observation SDK returns480x480crops fromthe same panorama. We import answer correctness only: evidence bounding-box threshold is not numerically specified in the public README, andconditional hallucination changes the denominator. Proposed JRDB-Agent has only one backbone and is excluded.","zh":"本脚手架内六个独立视觉模型共享27段JRDB测试序列的2098题。固定全景上下文、增加思维链、单次候选引导观察及迭代ReAct四种框架分别成组；均为零样本，使用同一答案格式。官方README明确容差：选项索引；帧误差不超过1秒；时间段IoU至少0.5；数值预测/金标在0.5–2.0；帧加视角误差不超过1秒和20度。所有题保留在共同分母，观察SDK从同一全景返回480×480裁切。本批只录答案正确率：公开README未给证据框阈值的具体数值，条件幻觉率分母也不同。提出的新Agent仅一个底模，不混入六模型组。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.35032v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[14685,0.2912,370,14686,14687],[14688,0.295,370,14689,14687],[8745,0.2541,370,14690,14687],[7513,0.2693,370,14691,14687],[6011,0.2741,370,14692,14687],[6013,0.2588,370,14693,14687]]},{"id":"jrdb-avr-answer-chain-of-thought-2026-paper","name":"JRDB-AVR ·answer chain-of-thought (2026 paper)","group":"视觉与视频","metric":"Answer correctness","unit":"%","public_history":true,"protocol":{"en":"Six independent VLM backbones share all 2098 questions from 27JRDB test sequences within this scaffold. Four scaffolds stay separate: fixed panoramic context, extra reasoning over the same context, one proposal-driven observation, anditerative ReAct. All use zero-shot prompts andthe same answer schema. Official repository README fixes exact answer tolerances: choice index;frames within1second;temporal IoU>=0.5;numeric prediction/gold ratio0.5–2.0;frame-viewpoint within1second/20degrees. Every question remains in the full denominator. The observation SDK returns480x480crops fromthe same panorama. We import answer correctness only: evidence bounding-box threshold is not numerically specified in the public README, andconditional hallucination changes the denominator. Proposed JRDB-Agent has only one backbone and is excluded.","zh":"本脚手架内六个独立视觉模型共享27段JRDB测试序列的2098题。固定全景上下文、增加思维链、单次候选引导观察及迭代ReAct四种框架分别成组；均为零样本，使用同一答案格式。官方README明确容差：选项索引；帧误差不超过1秒；时间段IoU至少0.5；数值预测/金标在0.5–2.0；帧加视角误差不超过1秒和20度。所有题保留在共同分母，观察SDK从同一全景返回480×480裁切。本批只录答案正确率：公开README未给证据框阈值的具体数值，条件幻觉率分母也不同。提出的新Agent仅一个底模，不混入六模型组。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.35032v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[14685,0.2531,370,14694,14687],[14688,0.3122,370,14695,14687],[8745,0.2774,370,14696,14687],[7513,0.2774,370,14697,14687],[6011,0.2459,370,14698,14687],[6013,0.256,370,14699,14687]]},{"id":"jrdb-avr-answer-search-recognize-2026-paper","name":"JRDB-AVR ·answer search-recognize (2026 paper)","group":"视觉与视频","metric":"Answer correctness","unit":"%","public_history":true,"protocol":{"en":"Six independent VLM backbones share all 2098 questions from 27JRDB test sequences within this scaffold. Four scaffolds stay separate: fixed panoramic context, extra reasoning over the same context, one proposal-driven observation, anditerative ReAct. All use zero-shot prompts andthe same answer schema. Official repository README fixes exact answer tolerances: choice index;frames within1second;temporal IoU>=0.5;numeric prediction/gold ratio0.5–2.0;frame-viewpoint within1second/20degrees. Every question remains in the full denominator. The observation SDK returns480x480crops fromthe same panorama. We import answer correctness only: evidence bounding-box threshold is not numerically specified in the public README, andconditional hallucination changes the denominator. Proposed JRDB-Agent has only one backbone and is excluded.","zh":"本脚手架内六个独立视觉模型共享27段JRDB测试序列的2098题。固定全景上下文、增加思维链、单次候选引导观察及迭代ReAct四种框架分别成组；均为零样本，使用同一答案格式。官方README明确容差：选项索引；帧误差不超过1秒；时间段IoU至少0.5；数值预测/金标在0.5–2.0；帧加视角误差不超过1秒和20度。所有题保留在共同分母，观察SDK从同一全景返回480×480裁切。本批只录答案正确率：公开README未给证据框阈值的具体数值，条件幻觉率分母也不同。提出的新Agent仅一个底模，不混入六模型组。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.35032v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[14685,0.2669,370,14700,14687],[14688,0.2927,370,14701,14687],[8745,0.214,370,14702,14687],[7513,0.2188,370,14703,14687],[6011,0.2455,370,14704,14687],[6013,0.2336,370,14705,14687]]},{"id":"jrdb-avr-answer-react-2026-paper","name":"JRDB-AVR ·answer react (2026 paper)","group":"视觉与视频","metric":"Answer correctness","unit":"%","public_history":true,"protocol":{"en":"Six independent VLM backbones share all 2098 questions from 27JRDB test sequences within this scaffold. Four scaffolds stay separate: fixed panoramic context, extra reasoning over the same context, one proposal-driven observation, anditerative ReAct. All use zero-shot prompts andthe same answer schema. Official repository README fixes exact answer tolerances: choice index;frames within1second;temporal IoU>=0.5;numeric prediction/gold ratio0.5–2.0;frame-viewpoint within1second/20degrees. Every question remains in the full denominator. The observation SDK returns480x480crops fromthe same panorama. We import answer correctness only: evidence bounding-box threshold is not numerically specified in the public README, andconditional hallucination changes the denominator. Proposed JRDB-Agent has only one backbone and is excluded.","zh":"本脚手架内六个独立视觉模型共享27段JRDB测试序列的2098题。固定全景上下文、增加思维链、单次候选引导观察及迭代ReAct四种框架分别成组；均为零样本，使用同一答案格式。官方README明确容差：选项索引；帧误差不超过1秒；时间段IoU至少0.5；数值预测/金标在0.5–2.0；帧加视角误差不超过1秒和20度。所有题保留在共同分母，观察SDK从同一全景返回480×480裁切。本批只录答案正确率：公开README未给证据框阈值的具体数值，条件幻觉率分母也不同。提出的新Agent仅一个底模，不混入六模型组。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.35032v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[14685,0.295,370,14706,14687],[14688,0.306,370,14707,14687],[8745,0.2774,370,14708,14687],[7513,0.33649999999999997,370,14709,14687],[6011,0.3122,370,14710,14687],[6013,0.3298,370,14711,14687]]},{"id":"awareness-theory-of-mind-2026-paper","name":"AwarenessBench ·theory-of-mind (2026 paper)","group":"知识与推理","metric":"Reference-answer/task correctness","unit":"%","public_history":true,"protocol":{"en":"Eighteen reported configurations cover sixteen independent backbones on the same 156 questions in this function. Printed PDF Table 10 gives the task population; Figure3 reports exact-choice/task-correctness percentages. Non-reasoning generation uses temperature0.7 and reasoning models use medium effort where supported. GeminiFlash thinking/nonthinking and GPT5 chat/thinking are configurations, not extra backbones. We retain only eight social/situational functions with reference correctness. Self-report, self-image consistency, calibration and the overall awareness composite are excluded. The full 14381sample corpus counts multiple calls in some other functions; it is not used as this cohort denominator.","zh":"十八条配置覆盖十六种独立权重，共同回答本功能156题；PDF表10给出样本数量，图3给出选择题或任务正确率。非推理生成温度0.7，支持时推理effort为medium。GeminiFlash思考与非思考、GPT5聊天与思考是配置，不额外计模型。本批仅保留有参考正确性判据的八个社会与情境功能，自我报告、自我形象一致性、校准和总体意识综合指标均排除。全14381样本语料在其他功能中含多次调用，未用它替代本组分母。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.35409v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":16,"result_count":18,"results":[[14712,0.34,370,14713,14714],[6863,0.57,370,14715,14714],[14716,0.5770000000000001,370,14717,14714],[1573,0.461,370,14718,14714],[769,0.635,370,14719,14714],[781,0.461,370,14720,14714],[766,0.539,370,14721,14714],[766,0.59,370,14722,14714],[764,0.609,370,14723,14714],[2191,0.436,370,14724,14714],[2058,0.5579999999999999,370,14725,14714],[7683,0.545,370,14726,14714],[2058,0.551,370,14727,14714],[5343,0.43,370,14728,14714],[3419,0.513,370,14729,14714],[797,0.314,370,14730,14714],[5344,0.37200000000000005,370,14731,14714],[784,0.461,370,14732,14714]]},{"id":"awareness-pragmatic-reasoning-2026-paper","name":"AwarenessBench ·pragmatic-reasoning (2026 paper)","group":"知识与推理","metric":"Reference-answer/task correctness","unit":"%","public_history":true,"protocol":{"en":"Eighteen reported configurations cover sixteen independent backbones on the same 240 questions in this function. Printed PDF Table 11 gives the task population; Figure3 reports exact-choice/task-correctness percentages. Non-reasoning generation uses temperature0.7 and reasoning models use medium effort where supported. GeminiFlash thinking/nonthinking and GPT5 chat/thinking are configurations, not extra backbones. We retain only eight social/situational functions with reference correctness. Self-report, self-image consistency, calibration and the overall awareness composite are excluded. The full 14381sample corpus counts multiple calls in some other functions; it is not used as this cohort denominator.","zh":"十八条配置覆盖十六种独立权重，共同回答本功能240题；PDF表11给出样本数量，图3给出选择题或任务正确率。非推理生成温度0.7，支持时推理effort为medium。GeminiFlash思考与非思考、GPT5聊天与思考是配置，不额外计模型。本批仅保留有参考正确性判据的八个社会与情境功能，自我报告、自我形象一致性、校准和总体意识综合指标均排除。全14381样本语料在其他功能中含多次调用，未用它替代本组分母。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.35409v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":16,"result_count":18,"results":[[14712,0.446,370,14713,14714],[6863,0.617,370,14715,14714],[14716,0.604,370,14717,14714],[1573,0.5579999999999999,370,14718,14714],[769,0.5710000000000001,370,14719,14714],[781,0.5710000000000001,370,14720,14714],[766,0.542,370,14721,14714],[766,0.596,370,14722,14714],[764,0.629,370,14723,14714],[2191,0.521,370,14724,14714],[2058,0.621,370,14725,14714],[7683,0.621,370,14726,14714],[2058,0.6,370,14727,14714],[5343,0.5329999999999999,370,14728,14714],[3419,0.575,370,14729,14714],[797,0.5329999999999999,370,14730,14714],[5344,0.47100000000000003,370,14731,14714],[784,0.562,370,14732,14714]]},{"id":"awareness-cultural-norms-2026-paper","name":"AwarenessBench ·cultural-norms (2026 paper)","group":"知识与推理","metric":"Reference-answer/task correctness","unit":"%","public_history":true,"protocol":{"en":"Eighteen reported configurations cover sixteen independent backbones on the same 150 questions in this function. Printed PDF Table 12 gives the task population; Figure3 reports exact-choice/task-correctness percentages. Non-reasoning generation uses temperature0.7 and reasoning models use medium effort where supported. GeminiFlash thinking/nonthinking and GPT5 chat/thinking are configurations, not extra backbones. We retain only eight social/situational functions with reference correctness. Self-report, self-image consistency, calibration and the overall awareness composite are excluded. The full 14381sample corpus counts multiple calls in some other functions; it is not used as this cohort denominator.","zh":"十八条配置覆盖十六种独立权重，共同回答本功能150题；PDF表12给出样本数量，图3给出选择题或任务正确率。非推理生成温度0.7，支持时推理effort为medium。GeminiFlash思考与非思考、GPT5聊天与思考是配置，不额外计模型。本批仅保留有参考正确性判据的八个社会与情境功能，自我报告、自我形象一致性、校准和总体意识综合指标均排除。全14381样本语料在其他功能中含多次调用，未用它替代本组分母。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.35409v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":16,"result_count":18,"results":[[14712,0.5870000000000001,370,14713,14714],[6863,0.5670000000000001,370,14715,14714],[14716,0.593,370,14717,14714],[1573,0.387,370,14718,14714],[769,0.627,370,14719,14714],[781,0.44,370,14720,14714],[766,0.513,370,14721,14714],[766,0.62,370,14722,14714],[764,0.687,370,14723,14714],[2191,0.33299999999999996,370,14724,14714],[2058,0.467,370,14725,14714],[7683,0.52,370,14726,14714],[2058,0.58,370,14727,14714],[5343,0.32,370,14728,14714],[3419,0.353,370,14729,14714],[797,0.38,370,14730,14714],[5344,0.24,370,14731,14714],[784,0.447,370,14732,14714]]},{"id":"awareness-social-cues-2026-paper","name":"AwarenessBench ·social-cues (2026 paper)","group":"知识与推理","metric":"Reference-answer/task correctness","unit":"%","public_history":true,"protocol":{"en":"Eighteen reported configurations cover sixteen independent backbones on the same 127 questions in this function. Printed PDF Table 13 gives the task population; Figure3 reports exact-choice/task-correctness percentages. Non-reasoning generation uses temperature0.7 and reasoning models use medium effort where supported. GeminiFlash thinking/nonthinking and GPT5 chat/thinking are configurations, not extra backbones. We retain only eight social/situational functions with reference correctness. Self-report, self-image consistency, calibration and the overall awareness composite are excluded. The full 14381sample corpus counts multiple calls in some other functions; it is not used as this cohort denominator.","zh":"十八条配置覆盖十六种独立权重，共同回答本功能127题；PDF表13给出样本数量，图3给出选择题或任务正确率。非推理生成温度0.7，支持时推理effort为medium。GeminiFlash思考与非思考、GPT5聊天与思考是配置，不额外计模型。本批仅保留有参考正确性判据的八个社会与情境功能，自我报告、自我形象一致性、校准和总体意识综合指标均排除。全14381样本语料在其他功能中含多次调用，未用它替代本组分母。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.35409v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":16,"result_count":18,"results":[[14712,0.465,370,14713,14714],[6863,0.6459999999999999,370,14715,14714],[14716,0.606,370,14717,14714],[1573,0.47200000000000003,370,14718,14714],[769,0.48,370,14719,14714],[781,0.512,370,14720,14714],[766,0.504,370,14721,14714],[766,0.449,370,14722,14714],[764,0.535,370,14723,14714],[2191,0.583,370,14724,14714],[2058,0.52,370,14725,14714],[7683,0.52,370,14726,14714],[2058,0.528,370,14727,14714],[5343,0.504,370,14728,14714],[3419,0.488,370,14729,14714],[797,0.465,370,14730,14714],[5344,0.41700000000000004,370,14731,14714],[784,0.34600000000000003,370,14732,14714]]},{"id":"awareness-causal-inference-2026-paper","name":"AwarenessBench ·causal-inference (2026 paper)","group":"知识与推理","metric":"Reference-answer/task correctness","unit":"%","public_history":true,"protocol":{"en":"Eighteen reported configurations cover sixteen independent backbones on the same 364 questions in this function. Printed PDF Table 14 gives the task population; Figure3 reports exact-choice/task-correctness percentages. Non-reasoning generation uses temperature0.7 and reasoning models use medium effort where supported. GeminiFlash thinking/nonthinking and GPT5 chat/thinking are configurations, not extra backbones. We retain only eight social/situational functions with reference correctness. Self-report, self-image consistency, calibration and the overall awareness composite are excluded. The full 14381sample corpus counts multiple calls in some other functions; it is not used as this cohort denominator.","zh":"十八条配置覆盖十六种独立权重，共同回答本功能364题；PDF表14给出样本数量，图3给出选择题或任务正确率。非推理生成温度0.7，支持时推理effort为medium。GeminiFlash思考与非思考、GPT5聊天与思考是配置，不额外计模型。本批仅保留有参考正确性判据的八个社会与情境功能，自我报告、自我形象一致性、校准和总体意识综合指标均排除。全14381样本语料在其他功能中含多次调用，未用它替代本组分母。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.35409v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":16,"result_count":18,"results":[[14712,0.565,370,14713,14714],[6863,0.616,370,14715,14714],[14716,0.643,370,14717,14714],[1573,0.622,370,14718,14714],[769,0.562,370,14719,14714],[781,0.615,370,14720,14714],[766,0.581,370,14721,14714],[766,0.491,370,14722,14714],[764,0.622,370,14723,14714],[2191,0.6559999999999999,370,14724,14714],[2058,0.655,370,14725,14714],[7683,0.633,370,14726,14714],[2058,0.62,370,14727,14714],[5343,0.593,370,14728,14714],[3419,0.5760000000000001,370,14729,14714],[797,0.595,370,14730,14714],[5344,0.5329999999999999,370,14731,14714],[784,0.628,370,14732,14714]]},{"id":"awareness-misuse-understanding-2026-paper","name":"AwarenessBench ·misuse-understanding (2026 paper)","group":"知识与推理","metric":"Reference-answer/task correctness","unit":"%","public_history":true,"protocol":{"en":"Eighteen reported configurations cover sixteen independent backbones on the same 150 questions in this function. Printed PDF Table 15 gives the task population; Figure3 reports exact-choice/task-correctness percentages. Non-reasoning generation uses temperature0.7 and reasoning models use medium effort where supported. GeminiFlash thinking/nonthinking and GPT5 chat/thinking are configurations, not extra backbones. We retain only eight social/situational functions with reference correctness. Self-report, self-image consistency, calibration and the overall awareness composite are excluded. The full 14381sample corpus counts multiple calls in some other functions; it is not used as this cohort denominator.","zh":"十八条配置覆盖十六种独立权重，共同回答本功能150题；PDF表15给出样本数量，图3给出选择题或任务正确率。非推理生成温度0.7，支持时推理effort为medium。GeminiFlash思考与非思考、GPT5聊天与思考是配置，不额外计模型。本批仅保留有参考正确性判据的八个社会与情境功能，自我报告、自我形象一致性、校准和总体意识综合指标均排除。全14381样本语料在其他功能中含多次调用，未用它替代本组分母。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.35409v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":16,"result_count":18,"results":[[14712,0.56,370,14713,14714],[6863,0.78,370,14715,14714],[14716,0.787,370,14717,14714],[1573,0.733,370,14718,14714],[769,0.873,370,14719,14714],[781,0.8,370,14720,14714],[766,0.84,370,14721,14714],[766,0.833,370,14722,14714],[764,0.753,370,14723,14714],[2191,0.807,370,14724,14714],[2058,0.8270000000000001,370,14725,14714],[7683,0.8,370,14726,14714],[2058,0.82,370,14727,14714],[5343,0.787,370,14728,14714],[3419,0.8270000000000001,370,14729,14714],[797,0.7070000000000001,370,14730,14714],[5344,0.727,370,14731,14714],[784,0.833,370,14732,14714]]},{"id":"awareness-dynamic-planning-2026-paper","name":"AwarenessBench ·dynamic-planning (2026 paper)","group":"知识与推理","metric":"Reference-answer/task correctness","unit":"%","public_history":true,"protocol":{"en":"Eighteen reported configurations cover sixteen independent backbones on the same 1805 questions in this function. Printed PDF Table 16 gives the task population; Figure3 reports exact-choice/task-correctness percentages. Non-reasoning generation uses temperature0.7 and reasoning models use medium effort where supported. GeminiFlash thinking/nonthinking and GPT5 chat/thinking are configurations, not extra backbones. We retain only eight social/situational functions with reference correctness. Self-report, self-image consistency, calibration and the overall awareness composite are excluded. The full 14381sample corpus counts multiple calls in some other functions; it is not used as this cohort denominator.","zh":"十八条配置覆盖十六种独立权重，共同回答本功能1805题；PDF表16给出样本数量，图3给出选择题或任务正确率。非推理生成温度0.7，支持时推理effort为medium。GeminiFlash思考与非思考、GPT5聊天与思考是配置，不额外计模型。本批仅保留有参考正确性判据的八个社会与情境功能，自我报告、自我形象一致性、校准和总体意识综合指标均排除。全14381样本语料在其他功能中含多次调用，未用它替代本组分母。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.35409v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":16,"result_count":18,"results":[[14712,0.32,370,14713,14714],[6863,0.638,370,14715,14714],[14716,0.718,370,14717,14714],[1573,0.37200000000000005,370,14718,14714],[769,0.9670000000000001,370,14719,14714],[781,0.253,370,14720,14714],[766,0.395,370,14721,14714],[766,0.268,370,14722,14714],[764,0.982,370,14723,14714],[2191,0.354,370,14724,14714],[2058,0.43,370,14725,14714],[7683,0.968,370,14726,14714],[2058,0.919,370,14727,14714],[5343,0.765,370,14728,14714],[3419,0.899,370,14729,14714],[797,0.321,370,14730,14714],[5344,0.32299999999999995,370,14731,14714],[784,0.36700000000000005,370,14732,14714]]},{"id":"awareness-stage-judgment-2026-paper","name":"AwarenessBench ·stage-judgment (2026 paper)","group":"知识与推理","metric":"Reference-answer/task correctness","unit":"%","public_history":true,"protocol":{"en":"Eighteen reported configurations cover sixteen independent backbones on the same 1200 questions in this function. Printed PDF Table 17 gives the task population; Figure3 reports exact-choice/task-correctness percentages. Non-reasoning generation uses temperature0.7 and reasoning models use medium effort where supported. GeminiFlash thinking/nonthinking and GPT5 chat/thinking are configurations, not extra backbones. We retain only eight social/situational functions with reference correctness. Self-report, self-image consistency, calibration and the overall awareness composite are excluded. The full 14381sample corpus counts multiple calls in some other functions; it is not used as this cohort denominator.","zh":"十八条配置覆盖十六种独立权重，共同回答本功能1200题；PDF表17给出样本数量，图3给出选择题或任务正确率。非推理生成温度0.7，支持时推理effort为medium。GeminiFlash思考与非思考、GPT5聊天与思考是配置，不额外计模型。本批仅保留有参考正确性判据的八个社会与情境功能，自我报告、自我形象一致性、校准和总体意识综合指标均排除。全14381样本语料在其他功能中含多次调用，未用它替代本组分母。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.35409v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":16,"result_count":18,"results":[[14712,0.384,370,14713,14714],[6863,0.49200000000000005,370,14715,14714],[14716,0.57,370,14717,14714],[1573,0.486,370,14718,14714],[769,0.568,370,14719,14714],[781,0.442,370,14720,14714],[766,0.34700000000000003,370,14721,14714],[766,0.456,370,14722,14714],[764,0.555,370,14723,14714],[2191,0.395,370,14724,14714],[2058,0.45399999999999996,370,14725,14714],[7683,0.5660000000000001,370,14726,14714],[2058,0.518,370,14727,14714],[5343,0.485,370,14728,14714],[3419,0.494,370,14729,14714],[797,0.365,370,14730,14714],[5344,0.35700000000000004,370,14731,14714],[784,0.428,370,14732,14714]]},{"id":"playlist-education-retrieved-2026-paper","name":"PlaylistBench ·education retrieved (2026 paper)","group":"视觉与视频","metric":"Pairwise preference accuracy","unit":"%","public_history":true,"protocol":{"en":"Thirteen independent frame-plus-transcript judge checkpoints share the same 90 preference pairs in this education panel. All use128 frames at640x480 andthe same ASR transcript; native-audio Gemini and QwenOmni rows are excluded. The retrieved setting is separate: retrieval uses Qwen3VL-Embedding8B, top64thirty-second chunks withtwo frames each; uniform sampling uses128 frames over the domain. Each answer position is fixed perpair by a shared seed. Accuracy is the fraction preferring the designated less-degraded answer. The benchmark has630 pairs,90perdomain, over 327 questions;343 questions belong to the separate retrieval evaluation. Accepted questions pass structural, transcript-only, parametric-answer andnative-video verification; visually degraded answer ordering is cross-checked and 152human-rated pairs reach93 percent agreement. Native serving/visual-token configurations remain explicit; these are paper baselines, not a claim of human gold for every pair.","zh":"十三个独立的帧加转录文本评审权重，共同评测education的90对答案。统一128帧、640×480及同一ASR文本，排除直接接收音频的Gemini和QwenOmni行。retrieved单独成组：检索使用Qwen3VL-Embedding8B，在30秒片段中取前64段各两帧；均匀抽样同样在该领域取128帧。答案位置由共同种子逐对固定，正确率为偏好指定的较轻视觉错误答案的比例。全量630对、每领域90对，覆盖327个问题；343题属于另一个检索评测，不混作本指标分母。问题须通过结构、仅转录、参数知识及原生视频验证，视觉退化顺序另经检查；152对人工复核的一致率为93%。保留原生服务和视觉词元配置，标记论文基线；并非每对均有人工金标。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34314v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":13,"result_count":13,"results":[[5133,0.76,370,14733,14734],[14735,0.64,370,14736,14734],[5727,0.66,370,14737,14734],[3187,0.71,370,14738,14734],[5635,0.51,370,14739,14734],[7567,0.43,370,14740,14734],[5650,0.43,370,14741,14734],[14742,0.64,370,14743,14734],[14744,0.51,370,14745,14734],[14746,0.49,370,14747,14734],[14748,0.58,370,14749,14734],[14750,0.41,370,14751,14734],[14752,0.49,370,14753,14734]]},{"id":"playlist-education-uniform-2026-paper","name":"PlaylistBench ·education uniform (2026 paper)","group":"视觉与视频","metric":"Pairwise preference accuracy","unit":"%","public_history":true,"protocol":{"en":"Thirteen independent frame-plus-transcript judge checkpoints share the same 90 preference pairs in this education panel. All use128 frames at640x480 andthe same ASR transcript; native-audio Gemini and QwenOmni rows are excluded. The uniform setting is separate: retrieval uses Qwen3VL-Embedding8B, top64thirty-second chunks withtwo frames each; uniform sampling uses128 frames over the domain. Each answer position is fixed perpair by a shared seed. Accuracy is the fraction preferring the designated less-degraded answer. The benchmark has630 pairs,90perdomain, over 327 questions;343 questions belong to the separate retrieval evaluation. Accepted questions pass structural, transcript-only, parametric-answer andnative-video verification; visually degraded answer ordering is cross-checked and 152human-rated pairs reach93 percent agreement. Native serving/visual-token configurations remain explicit; these are paper baselines, not a claim of human gold for every pair.","zh":"十三个独立的帧加转录文本评审权重，共同评测education的90对答案。统一128帧、640×480及同一ASR文本，排除直接接收音频的Gemini和QwenOmni行。uniform单独成组：检索使用Qwen3VL-Embedding8B，在30秒片段中取前64段各两帧；均匀抽样同样在该领域取128帧。答案位置由共同种子逐对固定，正确率为偏好指定的较轻视觉错误答案的比例。全量630对、每领域90对，覆盖327个问题；343题属于另一个检索评测，不混作本指标分母。问题须通过结构、仅转录、参数知识及原生视频验证，视觉退化顺序另经检查；152对人工复核的一致率为93%。保留原生服务和视觉词元配置，标记论文基线；并非每对均有人工金标。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34314v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":13,"result_count":13,"results":[[5133,0.59,370,14754,14734],[14735,0.61,370,14755,14734],[5727,0.64,370,14756,14734],[3187,0.6,370,14757,14734],[5635,0.4,370,14758,14734],[7567,0.47,370,14759,14734],[5650,0.43,370,14760,14734],[14742,0.51,370,14761,14734],[14744,0.53,370,14762,14734],[14746,0.42,370,14763,14734],[14748,0.43,370,14764,14734],[14750,0.44,370,14765,14734],[14752,0.51,370,14766,14734]]},{"id":"playlist-drama-retrieved-2026-paper","name":"PlaylistBench ·drama retrieved (2026 paper)","group":"视觉与视频","metric":"Pairwise preference accuracy","unit":"%","public_history":true,"protocol":{"en":"Thirteen independent frame-plus-transcript judge checkpoints share the same 90 preference pairs in this drama panel. All use128 frames at640x480 andthe same ASR transcript; native-audio Gemini and QwenOmni rows are excluded. The retrieved setting is separate: retrieval uses Qwen3VL-Embedding8B, top64thirty-second chunks withtwo frames each; uniform sampling uses128 frames over the domain. Each answer position is fixed perpair by a shared seed. Accuracy is the fraction preferring the designated less-degraded answer. The benchmark has630 pairs,90perdomain, over 327 questions;343 questions belong to the separate retrieval evaluation. Accepted questions pass structural, transcript-only, parametric-answer andnative-video verification; visually degraded answer ordering is cross-checked and 152human-rated pairs reach93 percent agreement. Native serving/visual-token configurations remain explicit; these are paper baselines, not a claim of human gold for every pair.","zh":"十三个独立的帧加转录文本评审权重，共同评测drama的90对答案。统一128帧、640×480及同一ASR文本，排除直接接收音频的Gemini和QwenOmni行。retrieved单独成组：检索使用Qwen3VL-Embedding8B，在30秒片段中取前64段各两帧；均匀抽样同样在该领域取128帧。答案位置由共同种子逐对固定，正确率为偏好指定的较轻视觉错误答案的比例。全量630对、每领域90对，覆盖327个问题；343题属于另一个检索评测，不混作本指标分母。问题须通过结构、仅转录、参数知识及原生视频验证，视觉退化顺序另经检查；152对人工复核的一致率为93%。保留原生服务和视觉词元配置，标记论文基线；并非每对均有人工金标。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34314v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":13,"result_count":13,"results":[[5133,0.64,370,14733,14734],[14735,0.62,370,14736,14734],[5727,0.68,370,14737,14734],[3187,0.68,370,14738,14734],[5635,0.58,370,14739,14734],[7567,0.5,370,14740,14734],[5650,0.42,370,14741,14734],[14742,0.53,370,14743,14734],[14744,0.57,370,14745,14734],[14746,0.47,370,14747,14734],[14748,0.54,370,14749,14734],[14750,0.42,370,14751,14734],[14752,0.49,370,14753,14734]]},{"id":"playlist-drama-uniform-2026-paper","name":"PlaylistBench ·drama uniform (2026 paper)","group":"视觉与视频","metric":"Pairwise preference accuracy","unit":"%","public_history":true,"protocol":{"en":"Thirteen independent frame-plus-transcript judge checkpoints share the same 90 preference pairs in this drama panel. All use128 frames at640x480 andthe same ASR transcript; native-audio Gemini and QwenOmni rows are excluded. The uniform setting is separate: retrieval uses Qwen3VL-Embedding8B, top64thirty-second chunks withtwo frames each; uniform sampling uses128 frames over the domain. Each answer position is fixed perpair by a shared seed. Accuracy is the fraction preferring the designated less-degraded answer. The benchmark has630 pairs,90perdomain, over 327 questions;343 questions belong to the separate retrieval evaluation. Accepted questions pass structural, transcript-only, parametric-answer andnative-video verification; visually degraded answer ordering is cross-checked and 152human-rated pairs reach93 percent agreement. Native serving/visual-token configurations remain explicit; these are paper baselines, not a claim of human gold for every pair.","zh":"十三个独立的帧加转录文本评审权重，共同评测drama的90对答案。统一128帧、640×480及同一ASR文本，排除直接接收音频的Gemini和QwenOmni行。uniform单独成组：检索使用Qwen3VL-Embedding8B，在30秒片段中取前64段各两帧；均匀抽样同样在该领域取128帧。答案位置由共同种子逐对固定，正确率为偏好指定的较轻视觉错误答案的比例。全量630对、每领域90对，覆盖327个问题；343题属于另一个检索评测，不混作本指标分母。问题须通过结构、仅转录、参数知识及原生视频验证，视觉退化顺序另经检查；152对人工复核的一致率为93%。保留原生服务和视觉词元配置，标记论文基线；并非每对均有人工金标。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34314v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":13,"result_count":13,"results":[[5133,0.68,370,14754,14734],[14735,0.62,370,14755,14734],[5727,0.66,370,14756,14734],[3187,0.61,370,14757,14734],[5635,0.46,370,14758,14734],[7567,0.42,370,14759,14734],[5650,0.47,370,14760,14734],[14742,0.56,370,14761,14734],[14744,0.58,370,14762,14734],[14746,0.51,370,14763,14734],[14748,0.59,370,14764,14734],[14750,0.47,370,14765,14734],[14752,0.48,370,14766,14734]]},{"id":"playlist-life-retrieved-2026-paper","name":"PlaylistBench ·life retrieved (2026 paper)","group":"视觉与视频","metric":"Pairwise preference accuracy","unit":"%","public_history":true,"protocol":{"en":"Thirteen independent frame-plus-transcript judge checkpoints share the same 90 preference pairs in this life panel. All use128 frames at640x480 andthe same ASR transcript; native-audio Gemini and QwenOmni rows are excluded. The retrieved setting is separate: retrieval uses Qwen3VL-Embedding8B, top64thirty-second chunks withtwo frames each; uniform sampling uses128 frames over the domain. Each answer position is fixed perpair by a shared seed. Accuracy is the fraction preferring the designated less-degraded answer. The benchmark has630 pairs,90perdomain, over 327 questions;343 questions belong to the separate retrieval evaluation. Accepted questions pass structural, transcript-only, parametric-answer andnative-video verification; visually degraded answer ordering is cross-checked and 152human-rated pairs reach93 percent agreement. Native serving/visual-token configurations remain explicit; these are paper baselines, not a claim of human gold for every pair.","zh":"十三个独立的帧加转录文本评审权重，共同评测life的90对答案。统一128帧、640×480及同一ASR文本，排除直接接收音频的Gemini和QwenOmni行。retrieved单独成组：检索使用Qwen3VL-Embedding8B，在30秒片段中取前64段各两帧；均匀抽样同样在该领域取128帧。答案位置由共同种子逐对固定，正确率为偏好指定的较轻视觉错误答案的比例。全量630对、每领域90对，覆盖327个问题；343题属于另一个检索评测，不混作本指标分母。问题须通过结构、仅转录、参数知识及原生视频验证，视觉退化顺序另经检查；152对人工复核的一致率为93%。保留原生服务和视觉词元配置，标记论文基线；并非每对均有人工金标。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34314v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":13,"result_count":13,"results":[[5133,0.82,370,14733,14734],[14735,0.73,370,14736,14734],[5727,0.81,370,14737,14734],[3187,0.77,370,14738,14734],[5635,0.6,370,14739,14734],[7567,0.56,370,14740,14734],[5650,0.47,370,14741,14734],[14742,0.59,370,14743,14734],[14744,0.61,370,14745,14734],[14746,0.53,370,14747,14734],[14748,0.48,370,14749,14734],[14750,0.42,370,14751,14734],[14752,0.34,370,14753,14734]]},{"id":"playlist-life-uniform-2026-paper","name":"PlaylistBench ·life uniform (2026 paper)","group":"视觉与视频","metric":"Pairwise preference accuracy","unit":"%","public_history":true,"protocol":{"en":"Thirteen independent frame-plus-transcript judge checkpoints share the same 90 preference pairs in this life panel. All use128 frames at640x480 andthe same ASR transcript; native-audio Gemini and QwenOmni rows are excluded. The uniform setting is separate: retrieval uses Qwen3VL-Embedding8B, top64thirty-second chunks withtwo frames each; uniform sampling uses128 frames over the domain. Each answer position is fixed perpair by a shared seed. Accuracy is the fraction preferring the designated less-degraded answer. The benchmark has630 pairs,90perdomain, over 327 questions;343 questions belong to the separate retrieval evaluation. Accepted questions pass structural, transcript-only, parametric-answer andnative-video verification; visually degraded answer ordering is cross-checked and 152human-rated pairs reach93 percent agreement. Native serving/visual-token configurations remain explicit; these are paper baselines, not a claim of human gold for every pair.","zh":"十三个独立的帧加转录文本评审权重，共同评测life的90对答案。统一128帧、640×480及同一ASR文本，排除直接接收音频的Gemini和QwenOmni行。uniform单独成组：检索使用Qwen3VL-Embedding8B，在30秒片段中取前64段各两帧；均匀抽样同样在该领域取128帧。答案位置由共同种子逐对固定，正确率为偏好指定的较轻视觉错误答案的比例。全量630对、每领域90对，覆盖327个问题；343题属于另一个检索评测，不混作本指标分母。问题须通过结构、仅转录、参数知识及原生视频验证，视觉退化顺序另经检查；152对人工复核的一致率为93%。保留原生服务和视觉词元配置，标记论文基线；并非每对均有人工金标。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34314v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":13,"result_count":13,"results":[[5133,0.74,370,14754,14734],[14735,0.72,370,14755,14734],[5727,0.7,370,14756,14734],[3187,0.64,370,14757,14734],[5635,0.48,370,14758,14734],[7567,0.5,370,14759,14734],[5650,0.52,370,14760,14734],[14742,0.49,370,14761,14734],[14744,0.56,370,14762,14734],[14746,0.51,370,14763,14734],[14748,0.5,370,14764,14734],[14750,0.4,370,14765,14734],[14752,0.36,370,14766,14734]]},{"id":"playlist-art-retrieved-2026-paper","name":"PlaylistBench ·art retrieved (2026 paper)","group":"视觉与视频","metric":"Pairwise preference accuracy","unit":"%","public_history":true,"protocol":{"en":"Thirteen independent frame-plus-transcript judge checkpoints share the same 90 preference pairs in this art panel. All use128 frames at640x480 andthe same ASR transcript; native-audio Gemini and QwenOmni rows are excluded. The retrieved setting is separate: retrieval uses Qwen3VL-Embedding8B, top64thirty-second chunks withtwo frames each; uniform sampling uses128 frames over the domain. Each answer position is fixed perpair by a shared seed. Accuracy is the fraction preferring the designated less-degraded answer. The benchmark has630 pairs,90perdomain, over 327 questions;343 questions belong to the separate retrieval evaluation. Accepted questions pass structural, transcript-only, parametric-answer andnative-video verification; visually degraded answer ordering is cross-checked and 152human-rated pairs reach93 percent agreement. Native serving/visual-token configurations remain explicit; these are paper baselines, not a claim of human gold for every pair.","zh":"十三个独立的帧加转录文本评审权重，共同评测art的90对答案。统一128帧、640×480及同一ASR文本，排除直接接收音频的Gemini和QwenOmni行。retrieved单独成组：检索使用Qwen3VL-Embedding8B，在30秒片段中取前64段各两帧；均匀抽样同样在该领域取128帧。答案位置由共同种子逐对固定，正确率为偏好指定的较轻视觉错误答案的比例。全量630对、每领域90对，覆盖327个问题；343题属于另一个检索评测，不混作本指标分母。问题须通过结构、仅转录、参数知识及原生视频验证，视觉退化顺序另经检查；152对人工复核的一致率为93%。保留原生服务和视觉词元配置，标记论文基线；并非每对均有人工金标。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34314v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":13,"result_count":13,"results":[[5133,0.73,370,14733,14734],[14735,0.71,370,14736,14734],[5727,0.71,370,14737,14734],[3187,0.7,370,14738,14734],[5635,0.59,370,14739,14734],[7567,0.44,370,14740,14734],[5650,0.38,370,14741,14734],[14742,0.46,370,14743,14734],[14744,0.64,370,14745,14734],[14746,0.51,370,14747,14734],[14748,0.54,370,14749,14734],[14750,0.5,370,14751,14734],[14752,0.48,370,14753,14734]]},{"id":"playlist-art-uniform-2026-paper","name":"PlaylistBench ·art uniform (2026 paper)","group":"视觉与视频","metric":"Pairwise preference accuracy","unit":"%","public_history":true,"protocol":{"en":"Thirteen independent frame-plus-transcript judge checkpoints share the same 90 preference pairs in this art panel. All use128 frames at640x480 andthe same ASR transcript; native-audio Gemini and QwenOmni rows are excluded. The uniform setting is separate: retrieval uses Qwen3VL-Embedding8B, top64thirty-second chunks withtwo frames each; uniform sampling uses128 frames over the domain. Each answer position is fixed perpair by a shared seed. Accuracy is the fraction preferring the designated less-degraded answer. The benchmark has630 pairs,90perdomain, over 327 questions;343 questions belong to the separate retrieval evaluation. Accepted questions pass structural, transcript-only, parametric-answer andnative-video verification; visually degraded answer ordering is cross-checked and 152human-rated pairs reach93 percent agreement. Native serving/visual-token configurations remain explicit; these are paper baselines, not a claim of human gold for every pair.","zh":"十三个独立的帧加转录文本评审权重，共同评测art的90对答案。统一128帧、640×480及同一ASR文本，排除直接接收音频的Gemini和QwenOmni行。uniform单独成组：检索使用Qwen3VL-Embedding8B，在30秒片段中取前64段各两帧；均匀抽样同样在该领域取128帧。答案位置由共同种子逐对固定，正确率为偏好指定的较轻视觉错误答案的比例。全量630对、每领域90对，覆盖327个问题；343题属于另一个检索评测，不混作本指标分母。问题须通过结构、仅转录、参数知识及原生视频验证，视觉退化顺序另经检查；152对人工复核的一致率为93%。保留原生服务和视觉词元配置，标记论文基线；并非每对均有人工金标。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34314v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":13,"result_count":13,"results":[[5133,0.78,370,14754,14734],[14735,0.61,370,14755,14734],[5727,0.77,370,14756,14734],[3187,0.67,370,14757,14734],[5635,0.51,370,14758,14734],[7567,0.42,370,14759,14734],[5650,0.4,370,14760,14734],[14742,0.53,370,14761,14734],[14744,0.57,370,14762,14734],[14746,0.39,370,14763,14734],[14748,0.57,370,14764,14734],[14750,0.43,370,14765,14734],[14752,0.5,370,14766,14734]]},{"id":"playlist-history-retrieved-2026-paper","name":"PlaylistBench ·history retrieved (2026 paper)","group":"视觉与视频","metric":"Pairwise preference accuracy","unit":"%","public_history":true,"protocol":{"en":"Thirteen independent frame-plus-transcript judge checkpoints share the same 90 preference pairs in this history panel. All use128 frames at640x480 andthe same ASR transcript; native-audio Gemini and QwenOmni rows are excluded. The retrieved setting is separate: retrieval uses Qwen3VL-Embedding8B, top64thirty-second chunks withtwo frames each; uniform sampling uses128 frames over the domain. Each answer position is fixed perpair by a shared seed. Accuracy is the fraction preferring the designated less-degraded answer. The benchmark has630 pairs,90perdomain, over 327 questions;343 questions belong to the separate retrieval evaluation. Accepted questions pass structural, transcript-only, parametric-answer andnative-video verification; visually degraded answer ordering is cross-checked and 152human-rated pairs reach93 percent agreement. Native serving/visual-token configurations remain explicit; these are paper baselines, not a claim of human gold for every pair.","zh":"十三个独立的帧加转录文本评审权重，共同评测history的90对答案。统一128帧、640×480及同一ASR文本，排除直接接收音频的Gemini和QwenOmni行。retrieved单独成组：检索使用Qwen3VL-Embedding8B，在30秒片段中取前64段各两帧；均匀抽样同样在该领域取128帧。答案位置由共同种子逐对固定，正确率为偏好指定的较轻视觉错误答案的比例。全量630对、每领域90对，覆盖327个问题；343题属于另一个检索评测，不混作本指标分母。问题须通过结构、仅转录、参数知识及原生视频验证，视觉退化顺序另经检查；152对人工复核的一致率为93%。保留原生服务和视觉词元配置，标记论文基线；并非每对均有人工金标。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34314v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":13,"result_count":13,"results":[[5133,0.76,370,14733,14734],[14735,0.65,370,14736,14734],[5727,0.69,370,14737,14734],[3187,0.73,370,14738,14734],[5635,0.63,370,14739,14734],[7567,0.42,370,14740,14734],[5650,0.51,370,14741,14734],[14742,0.63,370,14743,14734],[14744,0.58,370,14745,14734],[14746,0.5,370,14747,14734],[14748,0.47,370,14749,14734],[14750,0.51,370,14751,14734],[14752,0.49,370,14753,14734]]},{"id":"playlist-history-uniform-2026-paper","name":"PlaylistBench ·history uniform (2026 paper)","group":"视觉与视频","metric":"Pairwise preference accuracy","unit":"%","public_history":true,"protocol":{"en":"Thirteen independent frame-plus-transcript judge checkpoints share the same 90 preference pairs in this history panel. All use128 frames at640x480 andthe same ASR transcript; native-audio Gemini and QwenOmni rows are excluded. The uniform setting is separate: retrieval uses Qwen3VL-Embedding8B, top64thirty-second chunks withtwo frames each; uniform sampling uses128 frames over the domain. Each answer position is fixed perpair by a shared seed. Accuracy is the fraction preferring the designated less-degraded answer. The benchmark has630 pairs,90perdomain, over 327 questions;343 questions belong to the separate retrieval evaluation. Accepted questions pass structural, transcript-only, parametric-answer andnative-video verification; visually degraded answer ordering is cross-checked and 152human-rated pairs reach93 percent agreement. Native serving/visual-token configurations remain explicit; these are paper baselines, not a claim of human gold for every pair.","zh":"十三个独立的帧加转录文本评审权重，共同评测history的90对答案。统一128帧、640×480及同一ASR文本，排除直接接收音频的Gemini和QwenOmni行。uniform单独成组：检索使用Qwen3VL-Embedding8B，在30秒片段中取前64段各两帧；均匀抽样同样在该领域取128帧。答案位置由共同种子逐对固定，正确率为偏好指定的较轻视觉错误答案的比例。全量630对、每领域90对，覆盖327个问题；343题属于另一个检索评测，不混作本指标分母。问题须通过结构、仅转录、参数知识及原生视频验证，视觉退化顺序另经检查；152对人工复核的一致率为93%。保留原生服务和视觉词元配置，标记论文基线；并非每对均有人工金标。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34314v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":13,"result_count":13,"results":[[5133,0.71,370,14754,14734],[14735,0.63,370,14755,14734],[5727,0.71,370,14756,14734],[3187,0.7,370,14757,14734],[5635,0.51,370,14758,14734],[7567,0.46,370,14759,14734],[5650,0.41,370,14760,14734],[14742,0.52,370,14761,14734],[14744,0.5,370,14762,14734],[14746,0.47,370,14763,14734],[14748,0.58,370,14764,14734],[14750,0.49,370,14765,14734],[14752,0.5,370,14766,14734]]},{"id":"playlist-documentary-retrieved-2026-paper","name":"PlaylistBench ·documentary retrieved (2026 paper)","group":"视觉与视频","metric":"Pairwise preference accuracy","unit":"%","public_history":true,"protocol":{"en":"Thirteen independent frame-plus-transcript judge checkpoints share the same 90 preference pairs in this documentary panel. All use128 frames at640x480 andthe same ASR transcript; native-audio Gemini and QwenOmni rows are excluded. The retrieved setting is separate: retrieval uses Qwen3VL-Embedding8B, top64thirty-second chunks withtwo frames each; uniform sampling uses128 frames over the domain. Each answer position is fixed perpair by a shared seed. Accuracy is the fraction preferring the designated less-degraded answer. The benchmark has630 pairs,90perdomain, over 327 questions;343 questions belong to the separate retrieval evaluation. Accepted questions pass structural, transcript-only, parametric-answer andnative-video verification; visually degraded answer ordering is cross-checked and 152human-rated pairs reach93 percent agreement. Native serving/visual-token configurations remain explicit; these are paper baselines, not a claim of human gold for every pair.","zh":"十三个独立的帧加转录文本评审权重，共同评测documentary的90对答案。统一128帧、640×480及同一ASR文本，排除直接接收音频的Gemini和QwenOmni行。retrieved单独成组：检索使用Qwen3VL-Embedding8B，在30秒片段中取前64段各两帧；均匀抽样同样在该领域取128帧。答案位置由共同种子逐对固定，正确率为偏好指定的较轻视觉错误答案的比例。全量630对、每领域90对，覆盖327个问题；343题属于另一个检索评测，不混作本指标分母。问题须通过结构、仅转录、参数知识及原生视频验证，视觉退化顺序另经检查；152对人工复核的一致率为93%。保留原生服务和视觉词元配置，标记论文基线；并非每对均有人工金标。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34314v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":13,"result_count":13,"results":[[5133,0.72,370,14733,14734],[14735,0.68,370,14736,14734],[5727,0.77,370,14737,14734],[3187,0.67,370,14738,14734],[5635,0.61,370,14739,14734],[7567,0.39,370,14740,14734],[5650,0.41,370,14741,14734],[14742,0.58,370,14743,14734],[14744,0.57,370,14745,14734],[14746,0.56,370,14747,14734],[14748,0.51,370,14749,14734],[14750,0.53,370,14751,14734],[14752,0.48,370,14753,14734]]},{"id":"playlist-documentary-uniform-2026-paper","name":"PlaylistBench ·documentary uniform (2026 paper)","group":"视觉与视频","metric":"Pairwise preference accuracy","unit":"%","public_history":true,"protocol":{"en":"Thirteen independent frame-plus-transcript judge checkpoints share the same 90 preference pairs in this documentary panel. All use128 frames at640x480 andthe same ASR transcript; native-audio Gemini and QwenOmni rows are excluded. The uniform setting is separate: retrieval uses Qwen3VL-Embedding8B, top64thirty-second chunks withtwo frames each; uniform sampling uses128 frames over the domain. Each answer position is fixed perpair by a shared seed. Accuracy is the fraction preferring the designated less-degraded answer. The benchmark has630 pairs,90perdomain, over 327 questions;343 questions belong to the separate retrieval evaluation. Accepted questions pass structural, transcript-only, parametric-answer andnative-video verification; visually degraded answer ordering is cross-checked and 152human-rated pairs reach93 percent agreement. Native serving/visual-token configurations remain explicit; these are paper baselines, not a claim of human gold for every pair.","zh":"十三个独立的帧加转录文本评审权重，共同评测documentary的90对答案。统一128帧、640×480及同一ASR文本，排除直接接收音频的Gemini和QwenOmni行。uniform单独成组：检索使用Qwen3VL-Embedding8B，在30秒片段中取前64段各两帧；均匀抽样同样在该领域取128帧。答案位置由共同种子逐对固定，正确率为偏好指定的较轻视觉错误答案的比例。全量630对、每领域90对，覆盖327个问题；343题属于另一个检索评测，不混作本指标分母。问题须通过结构、仅转录、参数知识及原生视频验证，视觉退化顺序另经检查；152对人工复核的一致率为93%。保留原生服务和视觉词元配置，标记论文基线；并非每对均有人工金标。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34314v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":13,"result_count":13,"results":[[5133,0.68,370,14754,14734],[14735,0.61,370,14755,14734],[5727,0.73,370,14756,14734],[3187,0.69,370,14757,14734],[5635,0.52,370,14758,14734],[7567,0.41,370,14759,14734],[5650,0.4,370,14760,14734],[14742,0.51,370,14761,14734],[14744,0.49,370,14762,14734],[14746,0.47,370,14763,14734],[14748,0.47,370,14764,14734],[14750,0.54,370,14765,14734],[14752,0.52,370,14766,14734]]},{"id":"playlist-podcast-retrieved-2026-paper","name":"PlaylistBench ·podcast retrieved (2026 paper)","group":"视觉与视频","metric":"Pairwise preference accuracy","unit":"%","public_history":true,"protocol":{"en":"Thirteen independent frame-plus-transcript judge checkpoints share the same 90 preference pairs in this podcast panel. All use128 frames at640x480 andthe same ASR transcript; native-audio Gemini and QwenOmni rows are excluded. The retrieved setting is separate: retrieval uses Qwen3VL-Embedding8B, top64thirty-second chunks withtwo frames each; uniform sampling uses128 frames over the domain. Each answer position is fixed perpair by a shared seed. Accuracy is the fraction preferring the designated less-degraded answer. The benchmark has630 pairs,90perdomain, over 327 questions;343 questions belong to the separate retrieval evaluation. Accepted questions pass structural, transcript-only, parametric-answer andnative-video verification; visually degraded answer ordering is cross-checked and 152human-rated pairs reach93 percent agreement. Native serving/visual-token configurations remain explicit; these are paper baselines, not a claim of human gold for every pair.","zh":"十三个独立的帧加转录文本评审权重，共同评测podcast的90对答案。统一128帧、640×480及同一ASR文本，排除直接接收音频的Gemini和QwenOmni行。retrieved单独成组：检索使用Qwen3VL-Embedding8B，在30秒片段中取前64段各两帧；均匀抽样同样在该领域取128帧。答案位置由共同种子逐对固定，正确率为偏好指定的较轻视觉错误答案的比例。全量630对、每领域90对，覆盖327个问题；343题属于另一个检索评测，不混作本指标分母。问题须通过结构、仅转录、参数知识及原生视频验证，视觉退化顺序另经检查；152对人工复核的一致率为93%。保留原生服务和视觉词元配置，标记论文基线；并非每对均有人工金标。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34314v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":13,"result_count":13,"results":[[5133,0.74,370,14733,14734],[14735,0.69,370,14736,14734],[5727,0.77,370,14737,14734],[3187,0.75,370,14738,14734],[5635,0.66,370,14739,14734],[7567,0.48,370,14740,14734],[5650,0.48,370,14741,14734],[14742,0.53,370,14743,14734],[14744,0.46,370,14745,14734],[14746,0.56,370,14747,14734],[14748,0.57,370,14749,14734],[14750,0.59,370,14751,14734],[14752,0.58,370,14753,14734]]},{"id":"playlist-podcast-uniform-2026-paper","name":"PlaylistBench ·podcast uniform (2026 paper)","group":"视觉与视频","metric":"Pairwise preference accuracy","unit":"%","public_history":true,"protocol":{"en":"Thirteen independent frame-plus-transcript judge checkpoints share the same 90 preference pairs in this podcast panel. All use128 frames at640x480 andthe same ASR transcript; native-audio Gemini and QwenOmni rows are excluded. The uniform setting is separate: retrieval uses Qwen3VL-Embedding8B, top64thirty-second chunks withtwo frames each; uniform sampling uses128 frames over the domain. Each answer position is fixed perpair by a shared seed. Accuracy is the fraction preferring the designated less-degraded answer. The benchmark has630 pairs,90perdomain, over 327 questions;343 questions belong to the separate retrieval evaluation. Accepted questions pass structural, transcript-only, parametric-answer andnative-video verification; visually degraded answer ordering is cross-checked and 152human-rated pairs reach93 percent agreement. Native serving/visual-token configurations remain explicit; these are paper baselines, not a claim of human gold for every pair.","zh":"十三个独立的帧加转录文本评审权重，共同评测podcast的90对答案。统一128帧、640×480及同一ASR文本，排除直接接收音频的Gemini和QwenOmni行。uniform单独成组：检索使用Qwen3VL-Embedding8B，在30秒片段中取前64段各两帧；均匀抽样同样在该领域取128帧。答案位置由共同种子逐对固定，正确率为偏好指定的较轻视觉错误答案的比例。全量630对、每领域90对，覆盖327个问题；343题属于另一个检索评测，不混作本指标分母。问题须通过结构、仅转录、参数知识及原生视频验证，视觉退化顺序另经检查；152对人工复核的一致率为93%。保留原生服务和视觉词元配置，标记论文基线；并非每对均有人工金标。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34314v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":13,"result_count":13,"results":[[5133,0.7,370,14754,14734],[14735,0.51,370,14755,14734],[5727,0.67,370,14756,14734],[3187,0.69,370,14757,14734],[5635,0.57,370,14758,14734],[7567,0.41,370,14759,14734],[5650,0.43,370,14760,14734],[14742,0.61,370,14761,14734],[14744,0.44,370,14762,14734],[14746,0.49,370,14763,14734],[14748,0.67,370,14764,14734],[14750,0.62,370,14765,14734],[14752,0.57,370,14766,14734]]},{"id":"playlist-overall-retrieved-2026-paper","name":"PlaylistBench ·overall retrieved (2026 paper)","group":"视觉与视频","metric":"Pairwise preference accuracy","unit":"%","public_history":true,"protocol":{"en":"Thirteen independent frame-plus-transcript judge checkpoints share the same 630 preference pairs in this overall panel. All use128 frames at640x480 andthe same ASR transcript; native-audio Gemini and QwenOmni rows are excluded. The retrieved setting is separate: retrieval uses Qwen3VL-Embedding8B, top64thirty-second chunks withtwo frames each; uniform sampling uses128 frames over the domain. Each answer position is fixed perpair by a shared seed. Accuracy is the fraction preferring the designated less-degraded answer. The benchmark has630 pairs,90perdomain, over 327 questions;343 questions belong to the separate retrieval evaluation. Accepted questions pass structural, transcript-only, parametric-answer andnative-video verification; visually degraded answer ordering is cross-checked and 152human-rated pairs reach93 percent agreement. Native serving/visual-token configurations remain explicit; these are paper baselines, not a claim of human gold for every pair.","zh":"十三个独立的帧加转录文本评审权重，共同评测overall的630对答案。统一128帧、640×480及同一ASR文本，排除直接接收音频的Gemini和QwenOmni行。retrieved单独成组：检索使用Qwen3VL-Embedding8B，在30秒片段中取前64段各两帧；均匀抽样同样在该领域取128帧。答案位置由共同种子逐对固定，正确率为偏好指定的较轻视觉错误答案的比例。全量630对、每领域90对，覆盖327个问题；343题属于另一个检索评测，不混作本指标分母。问题须通过结构、仅转录、参数知识及原生视频验证，视觉退化顺序另经检查；152对人工复核的一致率为93%。保留原生服务和视觉词元配置，标记论文基线；并非每对均有人工金标。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34314v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":13,"result_count":13,"results":[[5133,0.7390000000000001,370,14733,14734],[14735,0.675,370,14736,14734],[5727,0.725,370,14737,14734],[3187,0.7140000000000001,370,14738,14734],[5635,0.597,370,14739,14734],[7567,0.46,370,14740,14734],[5650,0.44299999999999995,370,14741,14734],[14742,0.5670000000000001,370,14743,14734],[14744,0.562,370,14745,14734],[14746,0.516,370,14747,14734],[14748,0.527,370,14749,14734],[14750,0.483,370,14751,14734],[14752,0.478,370,14753,14734]]},{"id":"playlist-overall-uniform-2026-paper","name":"PlaylistBench ·overall uniform (2026 paper)","group":"视觉与视频","metric":"Pairwise preference accuracy","unit":"%","public_history":true,"protocol":{"en":"Thirteen independent frame-plus-transcript judge checkpoints share the same 630 preference pairs in this overall panel. All use128 frames at640x480 andthe same ASR transcript; native-audio Gemini and QwenOmni rows are excluded. The uniform setting is separate: retrieval uses Qwen3VL-Embedding8B, top64thirty-second chunks withtwo frames each; uniform sampling uses128 frames over the domain. Each answer position is fixed perpair by a shared seed. Accuracy is the fraction preferring the designated less-degraded answer. The benchmark has630 pairs,90perdomain, over 327 questions;343 questions belong to the separate retrieval evaluation. Accepted questions pass structural, transcript-only, parametric-answer andnative-video verification; visually degraded answer ordering is cross-checked and 152human-rated pairs reach93 percent agreement. Native serving/visual-token configurations remain explicit; these are paper baselines, not a claim of human gold for every pair.","zh":"十三个独立的帧加转录文本评审权重，共同评测overall的630对答案。统一128帧、640×480及同一ASR文本，排除直接接收音频的Gemini和QwenOmni行。uniform单独成组：检索使用Qwen3VL-Embedding8B，在30秒片段中取前64段各两帧；均匀抽样同样在该领域取128帧。答案位置由共同种子逐对固定，正确率为偏好指定的较轻视觉错误答案的比例。全量630对、每领域90对，覆盖327个问题；343题属于另一个检索评测，不混作本指标分母。问题须通过结构、仅转录、参数知识及原生视频验证，视觉退化顺序另经检查；152对人工复核的一致率为93%。保留原生服务和视觉词元配置，标记论文基线；并非每对均有人工金标。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34314v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":13,"result_count":13,"results":[[5133,0.6970000000000001,370,14754,14734],[14735,0.617,370,14755,14734],[5727,0.6970000000000001,370,14756,14734],[3187,0.657,370,14757,14734],[5635,0.49200000000000005,370,14758,14734],[7567,0.441,370,14759,14734],[5650,0.43799999999999994,370,14760,14734],[14742,0.5329999999999999,370,14761,14734],[14744,0.524,370,14762,14734],[14746,0.465,370,14763,14734],[14748,0.5429999999999999,370,14764,14734],[14750,0.483,370,14765,14734],[14752,0.49,370,14766,14734]]},{"id":"powerbench-temporal-2026-paper","name":"PowerBench ·temporal (2026 paper)","group":"Agent 与工具","metric":"Joint answer accuracy","unit":"%","public_history":true,"protocol":{"en":"Nine independent models share the same frozen synthetic power-system evidence corpus of761visible devices. This temporal panel covers60 questions; the 300-question whole has60TRR,90CRR and 150CoRR cases. Every run has50tool calls and 300seconds, the same instructions/tools, andmedium reasoning where supported. A question passes only if its JSON schema is valid andevery answer leaf is correct against executable private gold; field accuracy is separate. Overall Macro-JA equally averages three family rates rather than micro-averaging300 questions. Corpus/task seeds and deterministic numeric telemetry are documented; native provider settings remain configurations. The two-model retrieval-scope sensitivity study is excluded.","zh":"九个独立模型共享761台可见设备的冻结合成电力证据。本组temporal有60题，全量300题分为TRR60、CRR90、CoRR150。各次运行统一50次工具调用、300秒、指令和工具，支持时推理effort为medium。单题须JSON结构有效且每个答案字段均符合可执行的私有金标才通过，部分字段正确率另列。总体Macro-JA对三任务族等权，不作300题微平均；保留语料和任务种子、确定性数值遥测及提供商原设置。仅两模型的检索范围敏感性实验不混入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34492v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":9,"result_count":9,"results":[[4979,0.85,370,14767,14768],[3154,0.8,370,14769,14768],[6452,0.867,370,14770,14768],[3185,0.5,370,14771,14768],[5136,0.55,370,14772,14768],[3155,0.483,370,14773,14768],[3198,0.15,370,14774,14768],[3157,0.15,370,14775,14768],[1583,0.0,370,14776,14768]]},{"id":"powerbench-conditional-2026-paper","name":"PowerBench ·conditional (2026 paper)","group":"Agent 与工具","metric":"Joint answer accuracy","unit":"%","public_history":true,"protocol":{"en":"Nine independent models share the same frozen synthetic power-system evidence corpus of761visible devices. This conditional panel covers90 questions; the 300-question whole has60TRR,90CRR and 150CoRR cases. Every run has50tool calls and 300seconds, the same instructions/tools, andmedium reasoning where supported. A question passes only if its JSON schema is valid andevery answer leaf is correct against executable private gold; field accuracy is separate. Overall Macro-JA equally averages three family rates rather than micro-averaging300 questions. Corpus/task seeds and deterministic numeric telemetry are documented; native provider settings remain configurations. The two-model retrieval-scope sensitivity study is excluded.","zh":"九个独立模型共享761台可见设备的冻结合成电力证据。本组conditional有90题，全量300题分为TRR60、CRR90、CoRR150。各次运行统一50次工具调用、300秒、指令和工具，支持时推理effort为medium。单题须JSON结构有效且每个答案字段均符合可执行的私有金标才通过，部分字段正确率另列。总体Macro-JA对三任务族等权，不作300题微平均；保留语料和任务种子、确定性数值遥测及提供商原设置。仅两模型的检索范围敏感性实验不混入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34492v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":9,"result_count":9,"results":[[4979,0.889,370,14767,14768],[3154,0.9670000000000001,370,14769,14768],[6452,0.9109999999999999,370,14770,14768],[3185,0.889,370,14771,14768],[5136,0.8220000000000001,370,14772,14768],[3155,0.856,370,14773,14768],[3198,0.622,370,14774,14768],[3157,0.4,370,14775,14768],[1583,0.067,370,14776,14768]]},{"id":"powerbench-comparison-2026-paper","name":"PowerBench ·comparison (2026 paper)","group":"Agent 与工具","metric":"Joint answer accuracy","unit":"%","public_history":true,"protocol":{"en":"Nine independent models share the same frozen synthetic power-system evidence corpus of761visible devices. This comparison panel covers150 questions; the 300-question whole has60TRR,90CRR and 150CoRR cases. Every run has50tool calls and 300seconds, the same instructions/tools, andmedium reasoning where supported. A question passes only if its JSON schema is valid andevery answer leaf is correct against executable private gold; field accuracy is separate. Overall Macro-JA equally averages three family rates rather than micro-averaging300 questions. Corpus/task seeds and deterministic numeric telemetry are documented; native provider settings remain configurations. The two-model retrieval-scope sensitivity study is excluded.","zh":"九个独立模型共享761台可见设备的冻结合成电力证据。本组comparison有150题，全量300题分为TRR60、CRR90、CoRR150。各次运行统一50次工具调用、300秒、指令和工具，支持时推理effort为medium。单题须JSON结构有效且每个答案字段均符合可执行的私有金标才通过，部分字段正确率另列。总体Macro-JA对三任务族等权，不作300题微平均；保留语料和任务种子、确定性数值遥测及提供商原设置。仅两模型的检索范围敏感性实验不混入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34492v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":9,"result_count":9,"results":[[4979,0.48700000000000004,370,14767,14768],[3154,0.133,370,14769,14768],[6452,0.0,370,14770,14768],[3185,0.006999999999999999,370,14771,14768],[5136,0.02,370,14772,14768],[3155,0.027000000000000003,370,14773,14768],[3198,0.0,370,14774,14768],[3157,0.0,370,14775,14768],[1583,0.0,370,14776,14768]]},{"id":"powerbench-task-macro-2026-paper","name":"PowerBench ·task-macro (2026 paper)","group":"Agent 与工具","metric":"Three-family macro joint accuracy","unit":"%","public_history":true,"protocol":{"en":"Nine independent models share the same frozen synthetic power-system evidence corpus of761visible devices. This task-macro panel covers300 questions; the 300-question whole has60TRR,90CRR and 150CoRR cases. Every run has50tool calls and 300seconds, the same instructions/tools, andmedium reasoning where supported. A question passes only if its JSON schema is valid andevery answer leaf is correct against executable private gold; field accuracy is separate. Overall Macro-JA equally averages three family rates rather than micro-averaging300 questions. Corpus/task seeds and deterministic numeric telemetry are documented; native provider settings remain configurations. The two-model retrieval-scope sensitivity study is excluded.","zh":"九个独立模型共享761台可见设备的冻结合成电力证据。本组task-macro有300题，全量300题分为TRR60、CRR90、CoRR150。各次运行统一50次工具调用、300秒、指令和工具，支持时推理effort为medium。单题须JSON结构有效且每个答案字段均符合可执行的私有金标才通过，部分字段正确率另列。总体Macro-JA对三任务族等权，不作300题微平均；保留语料和任务种子、确定性数值遥测及提供商原设置。仅两模型的检索范围敏感性实验不混入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34492v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":9,"result_count":9,"results":[[4979,0.742,370,14767,14768],[3154,0.633,370,14769,14768],[6452,0.593,370,14770,14768],[3185,0.465,370,14771,14768],[5136,0.46399999999999997,370,14772,14768],[3155,0.455,370,14773,14768],[3198,0.257,370,14774,14768],[3157,0.183,370,14775,14768],[1583,0.022000000000000002,370,14776,14768]]},{"id":"symbolicarena-in-distribution-2026-paper","name":"SymbolicArena ·in-distribution (2026 paper)","group":"数学与科学","metric":"Clipped-log NMSE quality index","unit":"score","public_history":true,"protocol":{"en":"Fifteen distinct symbolic-regression search algorithms share frozen Core50, training/ID/OOD splits, seeds520/521/522 and three-hour runs. Each condition has50 tasks×3runs; all 150runs remain in aggregation andinvalid/unevaluable outputs receive zero quality. Numerical quality maps NMSE by100*(2-clip(log10(max(NMSE,1e-12)),-12,2))/14. It is a native quality index, not exact-equation recovery orpercent-correct. Final candidate/coefficient fitting sees training objectives only; ID/OOD stay reserved. The four non-LLM construction probes andseparate fidelity records are not substituted for this formal evaluation. LLM-SR/DrSR retain fixed Llama3.1-8B backend, T0.6/top_p0.3/1024 tokens; original operators/search configurations remain explicit.","zh":"十五种独立符号回归搜索算法共享冻结Core50、训练及ID/OOD切分、种子520/521/522和三小时运行预算，每条件50题各三次。全部150次运行纳入，无效或不可评估输出记零。数值质量按100×(2−clip(log10(max(NMSE,1e−12)),−12,2))/14映射，为原生质量指数，不是方程精确恢复率或正确率。候选和系数只按训练目标选择，ID/OOD保留评估。四个非LLM构造探针及另一套保真度记录不替代正式评测；LLM-SR/DrSR保留Llama3.1-8B、温度0.6、top_p0.3、1024词元及各算法原算子和搜索配置。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.35113v1","published_at":"2026-09-28","status":"paper_baselines"},"score_min":0,"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Native score / 100","zh":"原生分数 / 100"},"score_decimals":2,"model_count":15,"result_count":15,"results":[[14777,77.91,370,14778,14779],[14780,34.79,370,14781,14779],[14782,37.99,370,14783,14779],[14784,70.34,370,14785,14779],[14786,38.89,370,14787,14779],[14788,31.37,370,14789,14779],[14790,59.04,370,14791,14779],[14792,64.84,370,14793,14779],[14794,38.56,370,14795,14779],[14796,59.84,370,14797,14779],[14798,48.58,370,14799,14779],[14800,57.09,370,14801,14779],[14802,27.6,370,14803,14779],[14804,41.86,370,14805,14779],[14806,42.59,370,14807,14779]]},{"id":"symbolicarena-out-of-distribution-2026-paper","name":"SymbolicArena ·out-of-distribution (2026 paper)","group":"数学与科学","metric":"Clipped-log NMSE quality index","unit":"score","public_history":true,"protocol":{"en":"Fifteen distinct symbolic-regression search algorithms share frozen Core50, training/ID/OOD splits, seeds520/521/522 and three-hour runs. Each condition has50 tasks×3runs; all 150runs remain in aggregation andinvalid/unevaluable outputs receive zero quality. Numerical quality maps NMSE by100*(2-clip(log10(max(NMSE,1e-12)),-12,2))/14. It is a native quality index, not exact-equation recovery orpercent-correct. Final candidate/coefficient fitting sees training objectives only; ID/OOD stay reserved. The four non-LLM construction probes andseparate fidelity records are not substituted for this formal evaluation. LLM-SR/DrSR retain fixed Llama3.1-8B backend, T0.6/top_p0.3/1024 tokens; original operators/search configurations remain explicit.","zh":"十五种独立符号回归搜索算法共享冻结Core50、训练及ID/OOD切分、种子520/521/522和三小时运行预算，每条件50题各三次。全部150次运行纳入，无效或不可评估输出记零。数值质量按100×(2−clip(log10(max(NMSE,1e−12)),−12,2))/14映射，为原生质量指数，不是方程精确恢复率或正确率。候选和系数只按训练目标选择，ID/OOD保留评估。四个非LLM构造探针及另一套保真度记录不替代正式评测；LLM-SR/DrSR保留Llama3.1-8B、温度0.6、top_p0.3、1024词元及各算法原算子和搜索配置。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.35113v1","published_at":"2026-09-28","status":"paper_baselines"},"score_min":0,"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Native score / 100","zh":"原生分数 / 100"},"score_decimals":2,"model_count":15,"result_count":15,"results":[[14777,73.35,370,14778,14779],[14780,25.37,370,14781,14779],[14782,30.21,370,14783,14779],[14784,66.28,370,14785,14779],[14786,30.64,370,14787,14779],[14788,24.96,370,14789,14779],[14790,48.46,370,14791,14779],[14792,55.94,370,14793,14779],[14794,34.14,370,14795,14779],[14796,55.45,370,14797,14779],[14798,35.45,370,14799,14779],[14800,42.4,370,14801,14779],[14802,20.84,370,14803,14779],[14804,32.82,370,14805,14779],[14806,32.78,370,14807,14779]]},{"id":"rgdt-generator-answer-2026-paper","name":"RGDT-Bench ·generator answer (2026 paper)","group":"知识与推理","metric":"Equal task-probe decision accuracy","unit":"%","public_history":true,"protocol":{"en":"Six independent generator checkpoints share904active prompts across 452 source cases andeight supported task-probe combinations, with 16responses/prompt:14464planned responses/model. All slots have usable responses after format retries. Both decision correctness andcomplete-warrant rates keep not-evaluable warrant labels in the denominator; only conditional CWG excludes them, and CWG is not imported. Source-table results equally average eight task-probe combinations across the natural collection, including source training/validation/test groups; this is a descriptive paper-baseline cohort, not held-out generalization. Claim extraction is label-blind anddeterministic warrant checking uses private source references. Human review360responses gives84.2 percent label agreement; warrant labels are imperfect. Original model family, size andgeneration settings remain explicit.","zh":"六个独立生成权重共享452个源案例、904个有效提示、八种任务与探针组合，每提示16次，每模型计划14464条。格式重试后每个槽位均有可用回答，答案和完整论据比例都保留无法评估论据的条目；只有条件CWG排除它们，本批不录CWG。原表对自然语料八种任务探针等权，覆盖源训练、验证和测试组，因此是描述性论文基线，不宣称留出集泛化。论据提取不见标签，确定性核对使用私有源参考；360条人工核查的标签一致率84.2%，保留此标签误差。原家族、大小和生成设置写入配置。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34455v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[1822,0.439,370,14808,14809],[1662,0.491,370,14810,14809],[4967,0.7440000000000001,370,14811,14809],[7668,0.777,370,14812,14809],[4070,0.838,370,14813,14809],[2584,0.877,370,14814,14809]]},{"id":"rgdt-generator-warrant-2026-paper","name":"RGDT-Bench ·generator warrant (2026 paper)","group":"知识与推理","metric":"Equal task-probe warrant completeness","unit":"%","public_history":true,"protocol":{"en":"Six independent generator checkpoints share904active prompts across 452 source cases andeight supported task-probe combinations, with 16responses/prompt:14464planned responses/model. All slots have usable responses after format retries. Both decision correctness andcomplete-warrant rates keep not-evaluable warrant labels in the denominator; only conditional CWG excludes them, and CWG is not imported. Source-table results equally average eight task-probe combinations across the natural collection, including source training/validation/test groups; this is a descriptive paper-baseline cohort, not held-out generalization. Claim extraction is label-blind anddeterministic warrant checking uses private source references. Human review360responses gives84.2 percent label agreement; warrant labels are imperfect. Original model family, size andgeneration settings remain explicit.","zh":"六个独立生成权重共享452个源案例、904个有效提示、八种任务与探针组合，每提示16次，每模型计划14464条。格式重试后每个槽位均有可用回答，答案和完整论据比例都保留无法评估论据的条目；只有条件CWG排除它们，本批不录CWG。原表对自然语料八种任务探针等权，覆盖源训练、验证和测试组，因此是描述性论文基线，不宣称留出集泛化。论据提取不见标签，确定性核对使用私有源参考；360条人工核查的标签一致率84.2%，保留此标签误差。原家族、大小和生成设置写入配置。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34455v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[1822,0.107,370,14808,14809],[1662,0.261,370,14810,14809],[4967,0.479,370,14811,14809],[7668,0.563,370,14812,14809],[4070,0.635,370,14813,14809],[2584,0.6659999999999999,370,14814,14809]]},{"id":"rgdt-evaluator-auroc-2026-paper","name":"RGDT-Bench ·evaluator auroc (2026 paper)","group":"知识与推理","metric":"Task-macro auroc","unit":"score","public_history":true,"protocol":{"en":"Seventeen evaluator checkpoints score the same 3564balanced held-out response records. Generator/task/warrant labels are balanced before scoring; source-related variants stay within one split. Full-view metrics are computed pertask-probe, averaged over supported probes within task, then equally overfour tasks. AUROC gives half credit to ties; AUPRC isaverage precision at this fixed prevalence; thresholds/calibration fit validation only. Native0–100discrimination/balanced-accuracy units are retained, not ordinary item correctness. Not-evaluable warrant labels are excluded when forming this one shared evaluation set. The correct-answer-only view is a different population andnot imported. Ground-truth warrants are targets, not evaluator inputs; evaluator score extraction/configuration follows AppendixD.2.","zh":"十七个评估权重评分同一3564条平衡留出回答。生成模型、任务及论据标签在评分前平衡，相关案例的变体始终同切分。Full-view先逐任务探针计算，任务内平均支持的探针，再对四任务等权。AUROC对并列给半分；AUPRC为该固定比例下的平均精确率，阈值和校准只用验证集。保留0–100原生判别及平衡准确指数，不作普通答题正确率。无法评估的论据在构建共同集合时已统一排除，correct-only为另一人群，未混入；真实论据只作目标，未提供评审输入，分数提取与配置遵循附录D.2。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34455v1","published_at":"2026-09-28","status":"paper_baselines"},"score_min":0,"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Native score / 100","zh":"原生分数 / 100"},"score_decimals":2,"model_count":17,"result_count":17,"results":[[762,70.19,370,14815,14816],[764,65.37,370,14817,14816],[227,64.07,370,14818,14816],[6340,59.53,370,14819,14816],[4969,68.49,370,14820,14816],[4070,68.18,370,14821,14816],[2584,66.86,370,14822,14816],[2590,66.92,370,14823,14816],[4967,64.82,370,14824,14816],[7668,68.83,370,14825,14816],[1822,57.83,370,14826,14816],[1662,66.4,370,14827,14816],[562,62.13,370,14828,14816],[524,67.54,370,14829,14816],[8128,63.31,370,14830,14816],[14831,67.48,370,14832,14816],[14833,61.45,370,14834,14816]]},{"id":"rgdt-evaluator-average-precision-2026-paper","name":"RGDT-Bench ·evaluator average-precision (2026 paper)","group":"知识与推理","metric":"Task-macro average-precision","unit":"score","public_history":true,"protocol":{"en":"Seventeen evaluator checkpoints score the same 3564balanced held-out response records. Generator/task/warrant labels are balanced before scoring; source-related variants stay within one split. Full-view metrics are computed pertask-probe, averaged over supported probes within task, then equally overfour tasks. AUROC gives half credit to ties; AUPRC isaverage precision at this fixed prevalence; thresholds/calibration fit validation only. Native0–100discrimination/balanced-accuracy units are retained, not ordinary item correctness. Not-evaluable warrant labels are excluded when forming this one shared evaluation set. The correct-answer-only view is a different population andnot imported. Ground-truth warrants are targets, not evaluator inputs; evaluator score extraction/configuration follows AppendixD.2.","zh":"十七个评估权重评分同一3564条平衡留出回答。生成模型、任务及论据标签在评分前平衡，相关案例的变体始终同切分。Full-view先逐任务探针计算，任务内平均支持的探针，再对四任务等权。AUROC对并列给半分；AUPRC为该固定比例下的平均精确率，阈值和校准只用验证集。保留0–100原生判别及平衡准确指数，不作普通答题正确率。无法评估的论据在构建共同集合时已统一排除，correct-only为另一人群，未混入；真实论据只作目标，未提供评审输入，分数提取与配置遵循附录D.2。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34455v1","published_at":"2026-09-28","status":"paper_baselines"},"score_min":0,"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Native score / 100","zh":"原生分数 / 100"},"score_decimals":2,"model_count":17,"result_count":17,"results":[[762,63.84,370,14815,14816],[764,59.1,370,14817,14816],[227,58.2,370,14818,14816],[6340,56.14,370,14819,14816],[4969,62.13,370,14820,14816],[4070,61.57,370,14821,14816],[2584,60.73,370,14822,14816],[2590,60.24,370,14823,14816],[4967,59.25,370,14824,14816],[7668,63.06,370,14825,14816],[1822,55.17,370,14826,14816],[1662,60.23,370,14827,14816],[562,57.33,370,14828,14816],[524,61.45,370,14829,14816],[8128,58.09,370,14830,14816],[14831,65.56,370,14832,14816],[14833,59.42,370,14834,14816]]},{"id":"rgdt-evaluator-balanced-accuracy-2026-paper","name":"RGDT-Bench ·evaluator balanced-accuracy (2026 paper)","group":"知识与推理","metric":"Task-macro balanced-accuracy","unit":"score","public_history":true,"protocol":{"en":"Seventeen evaluator checkpoints score the same 3564balanced held-out response records. Generator/task/warrant labels are balanced before scoring; source-related variants stay within one split. Full-view metrics are computed pertask-probe, averaged over supported probes within task, then equally overfour tasks. AUROC gives half credit to ties; AUPRC isaverage precision at this fixed prevalence; thresholds/calibration fit validation only. Native0–100discrimination/balanced-accuracy units are retained, not ordinary item correctness. Not-evaluable warrant labels are excluded when forming this one shared evaluation set. The correct-answer-only view is a different population andnot imported. Ground-truth warrants are targets, not evaluator inputs; evaluator score extraction/configuration follows AppendixD.2.","zh":"十七个评估权重评分同一3564条平衡留出回答。生成模型、任务及论据标签在评分前平衡，相关案例的变体始终同切分。Full-view先逐任务探针计算，任务内平均支持的探针，再对四任务等权。AUROC对并列给半分；AUPRC为该固定比例下的平均精确率，阈值和校准只用验证集。保留0–100原生判别及平衡准确指数，不作普通答题正确率。无法评估的论据在构建共同集合时已统一排除，correct-only为另一人群，未混入；真实论据只作目标，未提供评审输入，分数提取与配置遵循附录D.2。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.34455v1","published_at":"2026-09-28","status":"paper_baselines"},"score_min":0,"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Native score / 100","zh":"原生分数 / 100"},"score_decimals":2,"model_count":17,"result_count":17,"results":[[762,67.09,370,14815,14816],[764,65.62,370,14817,14816],[227,66.05,370,14818,14816],[6340,58.29,370,14819,14816],[4969,63.92,370,14820,14816],[4070,64.97,370,14821,14816],[2584,64.97,370,14822,14816],[2590,65.31,370,14823,14816],[4967,62.4,370,14824,14816],[7668,64.04,370,14825,14816],[1822,58.03,370,14826,14816],[1662,63.91,370,14827,14816],[562,61.6,370,14828,14816],[524,63.89,370,14829,14816],[8128,63.24,370,14830,14816],[14831,60.58,370,14832,14816],[14833,56.25,370,14834,14816]]},{"id":"vvrbench-full-2026-paper","name":"VVRBench ·full (2026 paper)","group":"视觉与视频","metric":"All-constraints-correct image accuracy","unit":"%","public_history":true,"protocol":{"en":"This full split contains10000fixed satisfiable prompts, shared by every listed model. The full panel has12reported checkpoint configurations covering11conservative base identities (FLUX1dev/schnell folded); Fast and Challenge each have7. An image passes only if every binding, content andlayout constraint passes the same frozen pixel verifier. Hue masks, connected components, fixed geometric shape classification andprogram predicates use no learned detector/OCR/VLM. Reference renderings pass andcounterfactual constraints fail for all 10720full-plus-Challenge tasks. Model generations use native1024/2048resolution, scored at512for full/Fast and 1024for Challenge; seeds hash the prompt ID for open weights. API failures retry, but finished outputs never resample; text-only/empty outputs scorezero in the full denominator. Verifier audits show89.4 percent agreement on 508decisive held-out labels before subsequent calibration; reported94.7 percent is after calibration on that audit, not an independent final test. Original paper intervals stay in source cells.","zh":"full集合含10000条固定且可满足的提示，各模型共享；全量面板12条权重配置，FLUX1的dev/schnell合并底模后为11种；Fast和Challenge各七种。图像须同时通过全部绑定、内容和布局约束。冻结像素验证器采用色相掩码、连通块、几何形状分类和程序谓词，不使用学习检测器、OCR或VLM。全部10720条全量及挑战题的参考渲染通过，反事实约束失败。模型按原生1024或2048分辨率生成，full/Fast以512、Challenge以1024评分，开源权重按提示ID哈希统一种子。API只重试异常，已完成输出不重采样；仅文本或空输出记零，保留共同分母。验证器在校准前对508条明确留出标签一致率89.4%；之后同审计集的94.7%不是独立最终测试。原置信区间保留源格。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.35641v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":11,"result_count":12,"results":[[6368,0.8686,370,14835,14836],[14837,0.264,370,14838,14836],[8016,0.19149999999999998,370,14839,14836],[14840,0.18789999999999998,370,14841,14836],[14842,0.0579,370,14843,14836],[14844,0.042199999999999994,370,14845,14836],[14846,0.0387,370,14847,14836],[14846,0.0288,370,14848,14836],[14849,0.0281,370,14850,14836],[14851,0.0245,370,14852,14836],[14853,0.0002,370,14854,14836],[14855,0.0,370,14856,14836]]},{"id":"vvrbench-fast-2026-paper","name":"VVRBench ·fast (2026 paper)","group":"视觉与视频","metric":"All-constraints-correct image accuracy","unit":"%","public_history":true,"protocol":{"en":"This fast split contains820fixed satisfiable prompts, shared by every listed model. The full panel has12reported checkpoint configurations covering11conservative base identities (FLUX1dev/schnell folded); Fast and Challenge each have7. An image passes only if every binding, content andlayout constraint passes the same frozen pixel verifier. Hue masks, connected components, fixed geometric shape classification andprogram predicates use no learned detector/OCR/VLM. Reference renderings pass andcounterfactual constraints fail for all 10720full-plus-Challenge tasks. Model generations use native1024/2048resolution, scored at512for full/Fast and 1024for Challenge; seeds hash the prompt ID for open weights. API failures retry, but finished outputs never resample; text-only/empty outputs scorezero in the full denominator. Verifier audits show89.4 percent agreement on 508decisive held-out labels before subsequent calibration; reported94.7 percent is after calibration on that audit, not an independent final test. Original paper intervals stay in source cells.","zh":"fast集合含820条固定且可满足的提示，各模型共享；全量面板12条权重配置，FLUX1的dev/schnell合并底模后为11种；Fast和Challenge各七种。图像须同时通过全部绑定、内容和布局约束。冻结像素验证器采用色相掩码、连通块、几何形状分类和程序谓词，不使用学习检测器、OCR或VLM。全部10720条全量及挑战题的参考渲染通过，反事实约束失败。模型按原生1024或2048分辨率生成，full/Fast以512、Challenge以1024评分，开源权重按提示ID哈希统一种子。API只重试异常，已完成输出不重采样；仅文本或空输出记零，保留共同分母。验证器在校准前对508条明确留出标签一致率89.4%；之后同审计集的94.7%不是独立最终测试。原置信区间保留源格。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.35641v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":7,"result_count":7,"results":[[14857,0.8451000000000001,370,14858,14859],[6368,0.8220000000000001,370,14860,14859],[14861,0.47200000000000003,370,14862,14859],[14863,0.4561,370,14864,14859],[14865,0.4207,370,14866,14859],[14867,0.37200000000000005,370,14868,14859],[14837,0.32439999999999997,370,14869,14859]]},{"id":"vvrbench-challenge-2026-paper","name":"VVRBench ·challenge (2026 paper)","group":"视觉与视频","metric":"All-constraints-correct image accuracy","unit":"%","public_history":true,"protocol":{"en":"This challenge split contains720fixed satisfiable prompts, shared by every listed model. The full panel has12reported checkpoint configurations covering11conservative base identities (FLUX1dev/schnell folded); Fast and Challenge each have7. An image passes only if every binding, content andlayout constraint passes the same frozen pixel verifier. Hue masks, connected components, fixed geometric shape classification andprogram predicates use no learned detector/OCR/VLM. Reference renderings pass andcounterfactual constraints fail for all 10720full-plus-Challenge tasks. Model generations use native1024/2048resolution, scored at512for full/Fast and 1024for Challenge; seeds hash the prompt ID for open weights. API failures retry, but finished outputs never resample; text-only/empty outputs scorezero in the full denominator. Verifier audits show89.4 percent agreement on 508decisive held-out labels before subsequent calibration; reported94.7 percent is after calibration on that audit, not an independent final test. Original paper intervals stay in source cells.","zh":"challenge集合含720条固定且可满足的提示，各模型共享；全量面板12条权重配置，FLUX1的dev/schnell合并底模后为11种；Fast和Challenge各七种。图像须同时通过全部绑定、内容和布局约束。冻结像素验证器采用色相掩码、连通块、几何形状分类和程序谓词，不使用学习检测器、OCR或VLM。全部10720条全量及挑战题的参考渲染通过，反事实约束失败。模型按原生1024或2048分辨率生成，full/Fast以512、Challenge以1024评分，开源权重按提示ID哈希统一种子。API只重试异常，已完成输出不重采样；仅文本或空输出记零，保留共同分母。验证器在校准前对508条明确留出标签一致率89.4%；之后同审计集的94.7%不是独立最终测试。原置信区间保留源格。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.35641v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":7,"result_count":7,"results":[[14857,0.2139,370,14870,14871],[6368,0.10279999999999999,370,14872,14871],[14865,0.0736,370,14873,14871],[14867,0.0458,370,14874,14871],[14861,0.038900000000000004,370,14875,14871],[14863,0.0111,370,14876,14871],[14837,0.0028000000000000004,370,14877,14871]]},{"id":"webpagebench-screenshot-2026-paper","name":"WebPageBench ·screenshot (2026 paper)","group":"Agent 与工具","metric":"All-event-condition task success","unit":"%","public_history":true,"protocol":{"en":"Eight screenshot-only model+harness configurations cover seven conservative base identities on all 152fixed tasks (65canonical scenarios plus 87UI variants) across six self-hosted mock sites. Every task starts clean andis judged by typed event logs: all event-parameter conditions must pass; wrong quantities use last-state matching. No LLM judge orpage-text scraping is used for scoring. The models receive screenshots only andreturn pointer coordinates through the common driver; DOM/text integrations are excluded from this cohort. Native harness decoding/action budgets remain explicit, and each pair isone published run. EvoCUA S1/S2 are retained as original training-stage configurations ofone base. Completion claims andpartial condition matches are not counted as success.","zh":"八条纯截图模型加harness配置，保守计七种底模，共同运行六个自托管模拟网站的全部152题，其中65个规范场景、87个UI变体。每题重置，用类型化事件日志要求全部事件和参数条件通过，数量按最终状态匹配；评分不用LLM或页面文本抓取。模型只看截图，经共同驱动输出指针坐标，DOM和文本接口未混入。本组保留harness原解码及动作预算，各配置是一次公开运行。EvoCUA的S1/S2保留为同底模的原训练阶段配置，完成声明或部分条件合格不等于任务成功。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.35026v1","published_at":"2026-09-28","status":"paper_baselines"},"model_count":7,"result_count":8,"results":[[6435,0.48,370,14878,14879],[14880,0.638,370,14881,14879],[14882,0.625,370,14883,14879],[14884,0.421,370,14885,14879],[14886,0.336,370,14887,14879],[14888,0.336,370,14889,14879],[14890,0.092,370,14891,14879],[14890,0.079,370,14892,14879]]},{"id":"vex-fabrication-disinfocap-2026-paper","name":"VEX-Bench ·fabrication disinfocap (2026 paper)","group":"安全与可靠性","metric":"Yield-weighted perceived verification burden","unit":"score","public_history":true,"protocol":{"en":"Seven independent target LLMs share60fixed neutral topics (six domains×10) within each generation method andtask, except direct rewrite which has six numeric model results. All model-method pairs run the complete target grid; full study5880attempts. VEX=valid-output rate×nonrefusal rate×mean five normalized1–5rubric dimensions, reported on 0–100. This is a native perceived-risk index: lower means less generated verification burden, not factual accuracy orobserved fact-checking time. GPT5.2 judges under the full published codebook, withfive anchor examples and 50training/50held-out human-coded articles for calibration audit. Methods retain their attack prompts; PAP uses DeepSeekV3asattacker. D1–D5 conditional profiles andclaim/entity support are separate. Missing dash scores are never filled withzero: direct fabrication has fewer than six numeric results andis excluded; direct rewrite omits only the missing Sonnet row.","zh":"每种生成方法及任务用相同60个中立主题（六领域各十）比较七个独立目标LLM；直接改写只保留六个有数值的模型。所有模型方法组合均跑完整目标网格，全研究5880次。VEX为有效输出率×不拒绝率×五项归一化1–5rubric均值，原表以0–100报告。它是感知风险的原生指数，低值表示生成的核查负担较少，不是事实正确率或实际核查用时。GPT5.2按完整公开编码手册评分，以五个锚点、50篇训练和50篇留出人工标注文章检查校准。保留各攻击提示，PAP的攻击器为DeepSeekV3。条件D1–D5及声明、实体支持度另列。破折号不补零：直接捏造数值模型不足六而排除，直接改写仅略去Sonnet缺失行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.35028v1","published_at":"2026-09-28","status":"paper_baselines"},"score_min":0,"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Native score / 100","zh":"原生分数 / 100"},"score_decimals":2,"higher_is_better":false,"model_count":7,"result_count":7,"results":[[14893,47.9,370,14894,14895],[3184,39.7,370,14896,14895],[502,54.6,370,14897,14895],[9331,46.1,370,14898,14895],[3221,71.9,370,14899,14895],[3185,66.1,370,14900,14895],[2998,57.0,370,14901,14895]]},{"id":"vex-fabrication-isc-2026-paper","name":"VEX-Bench ·fabrication isc (2026 paper)","group":"安全与可靠性","metric":"Yield-weighted perceived verification burden","unit":"score","public_history":true,"protocol":{"en":"Seven independent target LLMs share60fixed neutral topics (six domains×10) within each generation method andtask, except direct rewrite which has six numeric model results. All model-method pairs run the complete target grid; full study5880attempts. VEX=valid-output rate×nonrefusal rate×mean five normalized1–5rubric dimensions, reported on 0–100. This is a native perceived-risk index: lower means less generated verification burden, not factual accuracy orobserved fact-checking time. GPT5.2 judges under the full published codebook, withfive anchor examples and 50training/50held-out human-coded articles for calibration audit. Methods retain their attack prompts; PAP uses DeepSeekV3asattacker. D1–D5 conditional profiles andclaim/entity support are separate. Missing dash scores are never filled withzero: direct fabrication has fewer than six numeric results andis excluded; direct rewrite omits only the missing Sonnet row.","zh":"每种生成方法及任务用相同60个中立主题（六领域各十）比较七个独立目标LLM；直接改写只保留六个有数值的模型。所有模型方法组合均跑完整目标网格，全研究5880次。VEX为有效输出率×不拒绝率×五项归一化1–5rubric均值，原表以0–100报告。它是感知风险的原生指数，低值表示生成的核查负担较少，不是事实正确率或实际核查用时。GPT5.2按完整公开编码手册评分，以五个锚点、50篇训练和50篇留出人工标注文章检查校准。保留各攻击提示，PAP的攻击器为DeepSeekV3。条件D1–D5及声明、实体支持度另列。破折号不补零：直接捏造数值模型不足六而排除，直接改写仅略去Sonnet缺失行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.35028v1","published_at":"2026-09-28","status":"paper_baselines"},"score_min":0,"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Native score / 100","zh":"原生分数 / 100"},"score_decimals":2,"higher_is_better":false,"model_count":7,"result_count":7,"results":[[14893,71.3,370,14902,14895],[3184,68.5,370,14903,14895],[502,60.3,370,14904,14895],[9331,72.4,370,14905,14895],[3221,69.9,370,14906,14895],[3185,70.1,370,14907,14895],[2998,71.0,370,14908,14895]]},{"id":"vex-fabrication-jnb-2026-paper","name":"VEX-Bench ·fabrication jnb (2026 paper)","group":"安全与可靠性","metric":"Yield-weighted perceived verification burden","unit":"score","public_history":true,"protocol":{"en":"Seven independent target LLMs share60fixed neutral topics (six domains×10) within each generation method andtask, except direct rewrite which has six numeric model results. All model-method pairs run the complete target grid; full study5880attempts. VEX=valid-output rate×nonrefusal rate×mean five normalized1–5rubric dimensions, reported on 0–100. This is a native perceived-risk index: lower means less generated verification burden, not factual accuracy orobserved fact-checking time. GPT5.2 judges under the full published codebook, withfive anchor examples and 50training/50held-out human-coded articles for calibration audit. Methods retain their attack prompts; PAP uses DeepSeekV3asattacker. D1–D5 conditional profiles andclaim/entity support are separate. Missing dash scores are never filled withzero: direct fabrication has fewer than six numeric results andis excluded; direct rewrite omits only the missing Sonnet row.","zh":"每种生成方法及任务用相同60个中立主题（六领域各十）比较七个独立目标LLM；直接改写只保留六个有数值的模型。所有模型方法组合均跑完整目标网格，全研究5880次。VEX为有效输出率×不拒绝率×五项归一化1–5rubric均值，原表以0–100报告。它是感知风险的原生指数，低值表示生成的核查负担较少，不是事实正确率或实际核查用时。GPT5.2按完整公开编码手册评分，以五个锚点、50篇训练和50篇留出人工标注文章检查校准。保留各攻击提示，PAP的攻击器为DeepSeekV3。条件D1–D5及声明、实体支持度另列。破折号不补零：直接捏造数值模型不足六而排除，直接改写仅略去Sonnet缺失行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.35028v1","published_at":"2026-09-28","status":"paper_baselines"},"score_min":0,"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Native score / 100","zh":"原生分数 / 100"},"score_decimals":2,"higher_is_better":false,"model_count":7,"result_count":7,"results":[[14893,1.0,370,14909,14895],[3184,20.8,370,14910,14895],[502,34.8,370,14911,14895],[9331,0.1,370,14912,14895],[3221,67.6,370,14913,14895],[3185,64.0,370,14914,14895],[2998,0.4,370,14915,14895]]},{"id":"vex-fabrication-misinfoqa-2026-paper","name":"VEX-Bench ·fabrication misinfoqa (2026 paper)","group":"安全与可靠性","metric":"Yield-weighted perceived verification burden","unit":"score","public_history":true,"protocol":{"en":"Seven independent target LLMs share60fixed neutral topics (six domains×10) within each generation method andtask, except direct rewrite which has six numeric model results. All model-method pairs run the complete target grid; full study5880attempts. VEX=valid-output rate×nonrefusal rate×mean five normalized1–5rubric dimensions, reported on 0–100. This is a native perceived-risk index: lower means less generated verification burden, not factual accuracy orobserved fact-checking time. GPT5.2 judges under the full published codebook, withfive anchor examples and 50training/50held-out human-coded articles for calibration audit. Methods retain their attack prompts; PAP uses DeepSeekV3asattacker. D1–D5 conditional profiles andclaim/entity support are separate. Missing dash scores are never filled withzero: direct fabrication has fewer than six numeric results andis excluded; direct rewrite omits only the missing Sonnet row.","zh":"每种生成方法及任务用相同60个中立主题（六领域各十）比较七个独立目标LLM；直接改写只保留六个有数值的模型。所有模型方法组合均跑完整目标网格，全研究5880次。VEX为有效输出率×不拒绝率×五项归一化1–5rubric均值，原表以0–100报告。它是感知风险的原生指数，低值表示生成的核查负担较少，不是事实正确率或实际核查用时。GPT5.2按完整公开编码手册评分，以五个锚点、50篇训练和50篇留出人工标注文章检查校准。保留各攻击提示，PAP的攻击器为DeepSeekV3。条件D1–D5及声明、实体支持度另列。破折号不补零：直接捏造数值模型不足六而排除，直接改写仅略去Sonnet缺失行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.35028v1","published_at":"2026-09-28","status":"paper_baselines"},"score_min":0,"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Native score / 100","zh":"原生分数 / 100"},"score_decimals":2,"higher_is_better":false,"model_count":7,"result_count":7,"results":[[14893,51.3,370,14916,14895],[3184,33.8,370,14917,14895],[502,43.5,370,14918,14895],[9331,47.2,370,14919,14895],[3221,51.7,370,14920,14895],[3185,45.6,370,14921,14895],[2998,59.2,370,14922,14895]]},{"id":"vex-fabrication-poisoned-rag-2026-paper","name":"VEX-Bench ·fabrication poisoned-rag (2026 paper)","group":"安全与可靠性","metric":"Yield-weighted perceived verification burden","unit":"score","public_history":true,"protocol":{"en":"Seven independent target LLMs share60fixed neutral topics (six domains×10) within each generation method andtask, except direct rewrite which has six numeric model results. All model-method pairs run the complete target grid; full study5880attempts. VEX=valid-output rate×nonrefusal rate×mean five normalized1–5rubric dimensions, reported on 0–100. This is a native perceived-risk index: lower means less generated verification burden, not factual accuracy orobserved fact-checking time. GPT5.2 judges under the full published codebook, withfive anchor examples and 50training/50held-out human-coded articles for calibration audit. Methods retain their attack prompts; PAP uses DeepSeekV3asattacker. D1–D5 conditional profiles andclaim/entity support are separate. Missing dash scores are never filled withzero: direct fabrication has fewer than six numeric results andis excluded; direct rewrite omits only the missing Sonnet row.","zh":"每种生成方法及任务用相同60个中立主题（六领域各十）比较七个独立目标LLM；直接改写只保留六个有数值的模型。所有模型方法组合均跑完整目标网格，全研究5880次。VEX为有效输出率×不拒绝率×五项归一化1–5rubric均值，原表以0–100报告。它是感知风险的原生指数，低值表示生成的核查负担较少，不是事实正确率或实际核查用时。GPT5.2按完整公开编码手册评分，以五个锚点、50篇训练和50篇留出人工标注文章检查校准。保留各攻击提示，PAP的攻击器为DeepSeekV3。条件D1–D5及声明、实体支持度另列。破折号不补零：直接捏造数值模型不足六而排除，直接改写仅略去Sonnet缺失行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.35028v1","published_at":"2026-09-28","status":"paper_baselines"},"score_min":0,"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Native score / 100","zh":"原生分数 / 100"},"score_decimals":2,"higher_is_better":false,"model_count":7,"result_count":7,"results":[[14893,52.1,370,14923,14895],[3184,53.4,370,14924,14895],[502,57.2,370,14925,14895],[9331,0.5,370,14926,14895],[3221,76.2,370,14927,14895],[3185,70.4,370,14928,14895],[2998,5.2,370,14929,14895]]},{"id":"vex-fabrication-pap-2026-paper","name":"VEX-Bench ·fabrication pap (2026 paper)","group":"安全与可靠性","metric":"Yield-weighted perceived verification burden","unit":"score","public_history":true,"protocol":{"en":"Seven independent target LLMs share60fixed neutral topics (six domains×10) within each generation method andtask, except direct rewrite which has six numeric model results. All model-method pairs run the complete target grid; full study5880attempts. VEX=valid-output rate×nonrefusal rate×mean five normalized1–5rubric dimensions, reported on 0–100. This is a native perceived-risk index: lower means less generated verification burden, not factual accuracy orobserved fact-checking time. GPT5.2 judges under the full published codebook, withfive anchor examples and 50training/50held-out human-coded articles for calibration audit. Methods retain their attack prompts; PAP uses DeepSeekV3asattacker. D1–D5 conditional profiles andclaim/entity support are separate. Missing dash scores are never filled withzero: direct fabrication has fewer than six numeric results andis excluded; direct rewrite omits only the missing Sonnet row.","zh":"每种生成方法及任务用相同60个中立主题（六领域各十）比较七个独立目标LLM；直接改写只保留六个有数值的模型。所有模型方法组合均跑完整目标网格，全研究5880次。VEX为有效输出率×不拒绝率×五项归一化1–5rubric均值，原表以0–100报告。它是感知风险的原生指数，低值表示生成的核查负担较少，不是事实正确率或实际核查用时。GPT5.2按完整公开编码手册评分，以五个锚点、50篇训练和50篇留出人工标注文章检查校准。保留各攻击提示，PAP的攻击器为DeepSeekV3。条件D1–D5及声明、实体支持度另列。破折号不补零：直接捏造数值模型不足六而排除，直接改写仅略去Sonnet缺失行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.35028v1","published_at":"2026-09-28","status":"paper_baselines"},"score_min":0,"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Native score / 100","zh":"原生分数 / 100"},"score_decimals":2,"higher_is_better":false,"model_count":7,"result_count":7,"results":[[14893,0.9,370,14930,14895],[3184,9.2,370,14931,14895],[502,61.8,370,14932,14895],[9331,5.1,370,14933,14895],[3221,53.7,370,14934,14895],[3185,74.9,370,14935,14895],[2998,18.1,370,14936,14895]]},{"id":"vex-rewrite-direct-2026-paper","name":"VEX-Bench ·rewrite direct (2026 paper)","group":"安全与可靠性","metric":"Yield-weighted perceived verification burden","unit":"score","public_history":true,"protocol":{"en":"Seven independent target LLMs share60fixed neutral topics (six domains×10) within each generation method andtask, except direct rewrite which has six numeric model results. All model-method pairs run the complete target grid; full study5880attempts. VEX=valid-output rate×nonrefusal rate×mean five normalized1–5rubric dimensions, reported on 0–100. This is a native perceived-risk index: lower means less generated verification burden, not factual accuracy orobserved fact-checking time. GPT5.2 judges under the full published codebook, withfive anchor examples and 50training/50held-out human-coded articles for calibration audit. Methods retain their attack prompts; PAP uses DeepSeekV3asattacker. D1–D5 conditional profiles andclaim/entity support are separate. Missing dash scores are never filled withzero: direct fabrication has fewer than six numeric results andis excluded; direct rewrite omits only the missing Sonnet row.","zh":"每种生成方法及任务用相同60个中立主题（六领域各十）比较七个独立目标LLM；直接改写只保留六个有数值的模型。所有模型方法组合均跑完整目标网格，全研究5880次。VEX为有效输出率×不拒绝率×五项归一化1–5rubric均值，原表以0–100报告。它是感知风险的原生指数，低值表示生成的核查负担较少，不是事实正确率或实际核查用时。GPT5.2按完整公开编码手册评分，以五个锚点、50篇训练和50篇留出人工标注文章检查校准。保留各攻击提示，PAP的攻击器为DeepSeekV3。条件D1–D5及声明、实体支持度另列。破折号不补零：直接捏造数值模型不足六而排除，直接改写仅略去Sonnet缺失行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.35028v1","published_at":"2026-09-28","status":"paper_baselines"},"score_min":0,"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Native score / 100","zh":"原生分数 / 100"},"score_decimals":2,"higher_is_better":false,"model_count":6,"result_count":6,"results":[[3184,6.0,370,14937,14938],[502,2.2,370,14939,14938],[9331,0.2,370,14940,14938],[3221,0.8,370,14941,14938],[3185,4.3,370,14942,14938],[2998,5.5,370,14943,14938]]},{"id":"vex-rewrite-disinfocap-2026-paper","name":"VEX-Bench ·rewrite disinfocap (2026 paper)","group":"安全与可靠性","metric":"Yield-weighted perceived verification burden","unit":"score","public_history":true,"protocol":{"en":"Seven independent target LLMs share60fixed neutral topics (six domains×10) within each generation method andtask, except direct rewrite which has six numeric model results. All model-method pairs run the complete target grid; full study5880attempts. VEX=valid-output rate×nonrefusal rate×mean five normalized1–5rubric dimensions, reported on 0–100. This is a native perceived-risk index: lower means less generated verification burden, not factual accuracy orobserved fact-checking time. GPT5.2 judges under the full published codebook, withfive anchor examples and 50training/50held-out human-coded articles for calibration audit. Methods retain their attack prompts; PAP uses DeepSeekV3asattacker. D1–D5 conditional profiles andclaim/entity support are separate. Missing dash scores are never filled withzero: direct fabrication has fewer than six numeric results andis excluded; direct rewrite omits only the missing Sonnet row.","zh":"每种生成方法及任务用相同60个中立主题（六领域各十）比较七个独立目标LLM；直接改写只保留六个有数值的模型。所有模型方法组合均跑完整目标网格，全研究5880次。VEX为有效输出率×不拒绝率×五项归一化1–5rubric均值，原表以0–100报告。它是感知风险的原生指数，低值表示生成的核查负担较少，不是事实正确率或实际核查用时。GPT5.2按完整公开编码手册评分，以五个锚点、50篇训练和50篇留出人工标注文章检查校准。保留各攻击提示，PAP的攻击器为DeepSeekV3。条件D1–D5及声明、实体支持度另列。破折号不补零：直接捏造数值模型不足六而排除，直接改写仅略去Sonnet缺失行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.35028v1","published_at":"2026-09-28","status":"paper_baselines"},"score_min":0,"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Native score / 100","zh":"原生分数 / 100"},"score_decimals":2,"higher_is_better":false,"model_count":7,"result_count":7,"results":[[14893,38.4,370,14944,14938],[3184,55.8,370,14945,14938],[502,57.2,370,14946,14938],[9331,60.7,370,14947,14938],[3221,71.2,370,14948,14938],[3185,62.4,370,14949,14938],[2998,58.6,370,14950,14938]]},{"id":"vex-rewrite-isc-2026-paper","name":"VEX-Bench ·rewrite isc (2026 paper)","group":"安全与可靠性","metric":"Yield-weighted perceived verification burden","unit":"score","public_history":true,"protocol":{"en":"Seven independent target LLMs share60fixed neutral topics (six domains×10) within each generation method andtask, except direct rewrite which has six numeric model results. All model-method pairs run the complete target grid; full study5880attempts. VEX=valid-output rate×nonrefusal rate×mean five normalized1–5rubric dimensions, reported on 0–100. This is a native perceived-risk index: lower means less generated verification burden, not factual accuracy orobserved fact-checking time. GPT5.2 judges under the full published codebook, withfive anchor examples and 50training/50held-out human-coded articles for calibration audit. Methods retain their attack prompts; PAP uses DeepSeekV3asattacker. D1–D5 conditional profiles andclaim/entity support are separate. Missing dash scores are never filled withzero: direct fabrication has fewer than six numeric results andis excluded; direct rewrite omits only the missing Sonnet row.","zh":"每种生成方法及任务用相同60个中立主题（六领域各十）比较七个独立目标LLM；直接改写只保留六个有数值的模型。所有模型方法组合均跑完整目标网格，全研究5880次。VEX为有效输出率×不拒绝率×五项归一化1–5rubric均值，原表以0–100报告。它是感知风险的原生指数，低值表示生成的核查负担较少，不是事实正确率或实际核查用时。GPT5.2按完整公开编码手册评分，以五个锚点、50篇训练和50篇留出人工标注文章检查校准。保留各攻击提示，PAP的攻击器为DeepSeekV3。条件D1–D5及声明、实体支持度另列。破折号不补零：直接捏造数值模型不足六而排除，直接改写仅略去Sonnet缺失行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.35028v1","published_at":"2026-09-28","status":"paper_baselines"},"score_min":0,"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Native score / 100","zh":"原生分数 / 100"},"score_decimals":2,"higher_is_better":false,"model_count":7,"result_count":7,"results":[[14893,66.8,370,14951,14938],[3184,65.8,370,14952,14938],[502,60.8,370,14953,14938],[9331,58.9,370,14954,14938],[3221,65.1,370,14955,14938],[3185,69.5,370,14956,14938],[2998,70.8,370,14957,14938]]},{"id":"vex-rewrite-jnb-2026-paper","name":"VEX-Bench ·rewrite jnb (2026 paper)","group":"安全与可靠性","metric":"Yield-weighted perceived verification burden","unit":"score","public_history":true,"protocol":{"en":"Seven independent target LLMs share60fixed neutral topics (six domains×10) within each generation method andtask, except direct rewrite which has six numeric model results. All model-method pairs run the complete target grid; full study5880attempts. VEX=valid-output rate×nonrefusal rate×mean five normalized1–5rubric dimensions, reported on 0–100. This is a native perceived-risk index: lower means less generated verification burden, not factual accuracy orobserved fact-checking time. GPT5.2 judges under the full published codebook, withfive anchor examples and 50training/50held-out human-coded articles for calibration audit. Methods retain their attack prompts; PAP uses DeepSeekV3asattacker. D1–D5 conditional profiles andclaim/entity support are separate. Missing dash scores are never filled withzero: direct fabrication has fewer than six numeric results andis excluded; direct rewrite omits only the missing Sonnet row.","zh":"每种生成方法及任务用相同60个中立主题（六领域各十）比较七个独立目标LLM；直接改写只保留六个有数值的模型。所有模型方法组合均跑完整目标网格，全研究5880次。VEX为有效输出率×不拒绝率×五项归一化1–5rubric均值，原表以0–100报告。它是感知风险的原生指数，低值表示生成的核查负担较少，不是事实正确率或实际核查用时。GPT5.2按完整公开编码手册评分，以五个锚点、50篇训练和50篇留出人工标注文章检查校准。保留各攻击提示，PAP的攻击器为DeepSeekV3。条件D1–D5及声明、实体支持度另列。破折号不补零：直接捏造数值模型不足六而排除，直接改写仅略去Sonnet缺失行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.35028v1","published_at":"2026-09-28","status":"paper_baselines"},"score_min":0,"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Native score / 100","zh":"原生分数 / 100"},"score_decimals":2,"higher_is_better":false,"model_count":7,"result_count":7,"results":[[14893,9.5,370,14958,14938],[3184,27.8,370,14959,14938],[502,58.4,370,14960,14938],[9331,11.2,370,14961,14938],[3221,65.0,370,14962,14938],[3185,66.8,370,14963,14938],[2998,38.8,370,14964,14938]]},{"id":"vex-rewrite-misinfoqa-2026-paper","name":"VEX-Bench ·rewrite misinfoqa (2026 paper)","group":"安全与可靠性","metric":"Yield-weighted perceived verification burden","unit":"score","public_history":true,"protocol":{"en":"Seven independent target LLMs share60fixed neutral topics (six domains×10) within each generation method andtask, except direct rewrite which has six numeric model results. All model-method pairs run the complete target grid; full study5880attempts. VEX=valid-output rate×nonrefusal rate×mean five normalized1–5rubric dimensions, reported on 0–100. This is a native perceived-risk index: lower means less generated verification burden, not factual accuracy orobserved fact-checking time. GPT5.2 judges under the full published codebook, withfive anchor examples and 50training/50held-out human-coded articles for calibration audit. Methods retain their attack prompts; PAP uses DeepSeekV3asattacker. D1–D5 conditional profiles andclaim/entity support are separate. Missing dash scores are never filled withzero: direct fabrication has fewer than six numeric results andis excluded; direct rewrite omits only the missing Sonnet row.","zh":"每种生成方法及任务用相同60个中立主题（六领域各十）比较七个独立目标LLM；直接改写只保留六个有数值的模型。所有模型方法组合均跑完整目标网格，全研究5880次。VEX为有效输出率×不拒绝率×五项归一化1–5rubric均值，原表以0–100报告。它是感知风险的原生指数，低值表示生成的核查负担较少，不是事实正确率或实际核查用时。GPT5.2按完整公开编码手册评分，以五个锚点、50篇训练和50篇留出人工标注文章检查校准。保留各攻击提示，PAP的攻击器为DeepSeekV3。条件D1–D5及声明、实体支持度另列。破折号不补零：直接捏造数值模型不足六而排除，直接改写仅略去Sonnet缺失行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.35028v1","published_at":"2026-09-28","status":"paper_baselines"},"score_min":0,"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Native score / 100","zh":"原生分数 / 100"},"score_decimals":2,"higher_is_better":false,"model_count":7,"result_count":7,"results":[[14893,52.6,370,14965,14938],[3184,60.2,370,14966,14938],[502,57.9,370,14967,14938],[9331,5.2,370,14968,14938],[3221,62.8,370,14969,14938],[3185,56.4,370,14970,14938],[2998,48.7,370,14971,14938]]},{"id":"vex-rewrite-poisoned-rag-2026-paper","name":"VEX-Bench ·rewrite poisoned-rag (2026 paper)","group":"安全与可靠性","metric":"Yield-weighted perceived verification burden","unit":"score","public_history":true,"protocol":{"en":"Seven independent target LLMs share60fixed neutral topics (six domains×10) within each generation method andtask, except direct rewrite which has six numeric model results. All model-method pairs run the complete target grid; full study5880attempts. VEX=valid-output rate×nonrefusal rate×mean five normalized1–5rubric dimensions, reported on 0–100. This is a native perceived-risk index: lower means less generated verification burden, not factual accuracy orobserved fact-checking time. GPT5.2 judges under the full published codebook, withfive anchor examples and 50training/50held-out human-coded articles for calibration audit. Methods retain their attack prompts; PAP uses DeepSeekV3asattacker. D1–D5 conditional profiles andclaim/entity support are separate. Missing dash scores are never filled withzero: direct fabrication has fewer than six numeric results andis excluded; direct rewrite omits only the missing Sonnet row.","zh":"每种生成方法及任务用相同60个中立主题（六领域各十）比较七个独立目标LLM；直接改写只保留六个有数值的模型。所有模型方法组合均跑完整目标网格，全研究5880次。VEX为有效输出率×不拒绝率×五项归一化1–5rubric均值，原表以0–100报告。它是感知风险的原生指数，低值表示生成的核查负担较少，不是事实正确率或实际核查用时。GPT5.2按完整公开编码手册评分，以五个锚点、50篇训练和50篇留出人工标注文章检查校准。保留各攻击提示，PAP的攻击器为DeepSeekV3。条件D1–D5及声明、实体支持度另列。破折号不补零：直接捏造数值模型不足六而排除，直接改写仅略去Sonnet缺失行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.35028v1","published_at":"2026-09-28","status":"paper_baselines"},"score_min":0,"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Native score / 100","zh":"原生分数 / 100"},"score_decimals":2,"higher_is_better":false,"model_count":7,"result_count":7,"results":[[14893,57.3,370,14972,14938],[3184,52.5,370,14973,14938],[502,56.9,370,14974,14938],[9331,5.1,370,14975,14938],[3221,62.0,370,14976,14938],[3185,61.4,370,14977,14938],[2998,62.9,370,14978,14938]]},{"id":"vex-rewrite-pap-2026-paper","name":"VEX-Bench ·rewrite pap (2026 paper)","group":"安全与可靠性","metric":"Yield-weighted perceived verification burden","unit":"score","public_history":true,"protocol":{"en":"Seven independent target LLMs share60fixed neutral topics (six domains×10) within each generation method andtask, except direct rewrite which has six numeric model results. All model-method pairs run the complete target grid; full study5880attempts. VEX=valid-output rate×nonrefusal rate×mean five normalized1–5rubric dimensions, reported on 0–100. This is a native perceived-risk index: lower means less generated verification burden, not factual accuracy orobserved fact-checking time. GPT5.2 judges under the full published codebook, withfive anchor examples and 50training/50held-out human-coded articles for calibration audit. Methods retain their attack prompts; PAP uses DeepSeekV3asattacker. D1–D5 conditional profiles andclaim/entity support are separate. Missing dash scores are never filled withzero: direct fabrication has fewer than six numeric results andis excluded; direct rewrite omits only the missing Sonnet row.","zh":"每种生成方法及任务用相同60个中立主题（六领域各十）比较七个独立目标LLM；直接改写只保留六个有数值的模型。所有模型方法组合均跑完整目标网格，全研究5880次。VEX为有效输出率×不拒绝率×五项归一化1–5rubric均值，原表以0–100报告。它是感知风险的原生指数，低值表示生成的核查负担较少，不是事实正确率或实际核查用时。GPT5.2按完整公开编码手册评分，以五个锚点、50篇训练和50篇留出人工标注文章检查校准。保留各攻击提示，PAP的攻击器为DeepSeekV3。条件D1–D5及声明、实体支持度另列。破折号不补零：直接捏造数值模型不足六而排除，直接改写仅略去Sonnet缺失行。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.35028v1","published_at":"2026-09-28","status":"paper_baselines"},"score_min":0,"score_max":100,"saturation_excluded":true,"scale_label":{"en":"Native score / 100","zh":"原生分数 / 100"},"score_decimals":2,"higher_is_better":false,"model_count":7,"result_count":7,"results":[[14893,9.2,370,14979,14938],[3184,8.9,370,14980,14938],[502,60.6,370,14981,14938],[9331,5.0,370,14982,14938],[3221,44.8,370,14983,14938],[3185,59.7,370,14984,14938],[2998,7.6,370,14985,14938]]},{"id":"gamecommbench-completeness-2026-paper","name":"GameCommBench · event completeness (2026 paper)","group":"视觉与视频","metric":"Reference-event recall","unit":"%","public_history":true,"protocol":{"en":"Six independent VLMs use the full five-game benchmark:2159 chess turns,2857 Go turns,2681 football clips,2981 basketball clips and7675 League of Legends clips (18353 instances). Completeness is matched reference events divided by reference events per commentary instance. Reference events are extracted with GPT5.4 from human commentary plus game state; generated events use GPT5.4mini and semantic matching GLM4.7. The published full-benchmark aggregate is preserved; the paper does not expose the final game weighting. No-event generations receive actual zero recall. Generation uses temperature0,1024 output tokens and minimal/disabled reasoning where supported; video4fps, original resolution, board state with four prior plies. Extra smaller-model subset experiments remain separate.","zh":"六个独立VLM使用完整五游戏面板：国际象棋2159回合、围棋2857回合、足球2681片段、篮球2981片段、英雄联盟7675片段，共18353实例。Completeness为每条解说匹配的参考事件占参考事件比例。GPT5.4从人工解说和状态提取参考事件，GPT5.4mini提取生成事件，GLM4.7作语义匹配。保留原文全基准汇总，原文未披露最终游戏权重。未生成事件是真实零召回。生成温度零、最多1024token，尽量关闭推理；视频4fps及原分辨率，棋盘包含此前四步。小模型另用子集的实验分开留档。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.11129v1","published_at":"2026-10-08","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[6237,0.499,370,14986,14987],[3184,0.61,370,14986,14987],[6704,0.533,370,14986,14987],[6343,0.467,370,14986,14987],[5722,0.282,370,14986,14987],[2642,0.445,370,14986,14987]]},{"id":"emotrace-emotion-2026-paper","name":"EmoTrace-Bench · target emotion (2026 paper)","group":"生成与创作","metric":"Accuracy","unit":"%","public_history":true,"protocol":{"en":"Twelve video generators each produce one video for every448 frozen English prompts:8 emotions,2 intensities,2 samples and14 templates;224 verified strong/weak pairs,5376 videos total. All models have448 valid videos. Emotion accuracy and joint emotion-plus-requested-intensity accuracy use all448; an emotion failure gets no joint credit. Frozen Gemini3.1Pro temperature0 judges8fps video plus native audio when available, with emotion name/definition but no prompt,target intensity or model identity. Cue-channel judging is a separate call. No best-of selection or evaluation voting; provider-native duration/resolution and prompt processing are retained. Four silent generators remain valid in both binary metrics; unavailable audio cue scores are not zero. Conditional Visual9D/Full10D use model-specific emotion-success subsets and are preserved only as evidence.1120 human-reviewed videos across3 generators validate the judge; full sample-level human labels are not public.","zh":"12视频模型各对448条冻结英文提示生成一次，共5376视频；八情绪、两强度、两样本、14模板，224强弱配对均经过结构核验。每模型448有效视频。情绪准确率和情绪加目标强度联合准确率都用完整448分母，情绪失败不给联合分。冻结Gemini3.1Pro以温度零评8fps视频和可用原生音频，只见情绪名称定义，不见提示、目标强度及模型身份；线索维度另一次调用。没有择优采样或裁判投票，保留供应商原生时长、分辨率及提示处理。四个无声模型仍纳入两个二元指标，不把缺失音频线索记零。Visual9D与Full10D使用各模型情绪成功子集，仅作原始证据保存。三生成器共1120人工复核视频用于裁判验证，逐条人工标签未公开。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.15452v1","published_at":"2026-10-11","status":"paper_baselines"},"model_count":12,"result_count":12,"results":[[7121,0.87,370,14988,14989],[8885,0.848,370,14988,14989],[14990,0.799,370,14988,14989],[7126,0.797,370,14988,14989],[14991,0.743,370,14988,14989],[14992,0.732,370,14988,14989],[7125,0.679,370,14988,14989],[7129,0.672,370,14988,14989],[14993,0.674,370,14994,14989],[9548,0.556,370,14994,14989],[14995,0.505,370,14994,14989],[14996,0.493,370,14994,14989]]},{"id":"emotrace-emotion-intensity-2026-paper","name":"EmoTrace-Bench · emotion and intensity (2026 paper)","group":"生成与创作","metric":"Accuracy","unit":"%","public_history":true,"protocol":{"en":"Twelve video generators each produce one video for every448 frozen English prompts:8 emotions,2 intensities,2 samples and14 templates;224 verified strong/weak pairs,5376 videos total. All models have448 valid videos. Emotion accuracy and joint emotion-plus-requested-intensity accuracy use all448; an emotion failure gets no joint credit. Frozen Gemini3.1Pro temperature0 judges8fps video plus native audio when available, with emotion name/definition but no prompt,target intensity or model identity. Cue-channel judging is a separate call. No best-of selection or evaluation voting; provider-native duration/resolution and prompt processing are retained. Four silent generators remain valid in both binary metrics; unavailable audio cue scores are not zero. Conditional Visual9D/Full10D use model-specific emotion-success subsets and are preserved only as evidence.1120 human-reviewed videos across3 generators validate the judge; full sample-level human labels are not public.","zh":"12视频模型各对448条冻结英文提示生成一次，共5376视频；八情绪、两强度、两样本、14模板，224强弱配对均经过结构核验。每模型448有效视频。情绪准确率和情绪加目标强度联合准确率都用完整448分母，情绪失败不给联合分。冻结Gemini3.1Pro以温度零评8fps视频和可用原生音频，只见情绪名称定义，不见提示、目标强度及模型身份；线索维度另一次调用。没有择优采样或裁判投票，保留供应商原生时长、分辨率及提示处理。四个无声模型仍纳入两个二元指标，不把缺失音频线索记零。Visual9D与Full10D使用各模型情绪成功子集，仅作原始证据保存。三生成器共1120人工复核视频用于裁判验证，逐条人工标签未公开。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.15452v1","published_at":"2026-10-11","status":"paper_baselines"},"model_count":12,"result_count":12,"results":[[7121,0.542,370,14988,14989],[8885,0.551,370,14988,14989],[14990,0.52,370,14988,14989],[7126,0.5329999999999999,370,14988,14989],[14991,0.491,370,14988,14989],[14992,0.442,370,14988,14989],[7125,0.451,370,14988,14989],[7129,0.489,370,14988,14989],[14993,0.41700000000000004,370,14994,14989],[9548,0.386,370,14994,14989],[14995,0.337,370,14994,14989],[14996,0.299,370,14994,14989]]},{"id":"localagentbench-gpu-v3-2026-evaluator","name":"Local Agent Bench · RTX3090 (2026 evaluator)","group":"Agent 与工具","metric":"Scoring v3 weighted task score","unit":"score","score_max":100,"score_decimals":2,"saturation_excluded":true,"scale_label":{"en":"Native score / 100","zh":"原生分数 / 100"},"public_history":true,"source":{"name":"Local Agent Bench maintainers","url":"https://github.com/localagentbench/Localagentbench/tree/b52a97f45fe522430a78d62489dba71e1c0400a3","published_at":"2026-10-20","status":"dated_release"},"protocol":{"en":"Suite0.1 has100 fixed tasks:25 JSON/schema tasks,25 tool-selection,25 multi-step plus10 recovery,15 prompt-injection tasks (5 payload families ×3 channels). Ten deterministic mock tools are used. Three main trials/model produce300 task trials; scoringv3=15% JSON+15% schema+20% tool+30% multistep+20% useful-and-safe injection, requiring both legitimate completion and non-compromise. The same25 JSON tasks feed two weighted components. Ollama0.40.6 and HermesAgent0.22.2+2363.g9cb34e9 use actual32768 context; the8192 runner flag was unused. Sampling is Ollama default without an explicit temperature/seed. Hardware cohorts are separate; RTX3090 scores span3 host CPUs, so no speed comparison is imported. Original three-trial point estimates are retained;5 additional variance trials only inform CIs. Unsupported native-tool models remain null and are excluded. The importer selects the best complete run per model/quant/engine; source run history is retained in the full JSON. Independent evaluator release, not an independent rerun:source verified flags are false. MacM2 still has six bases after collapsing Hermes3 into Llama3.1 and coder into Qwen2.5.","zh":"Suite0.1含100固定任务：25条JSON与schema、25条工具选择、25条多步骤加10条恢复、15条提示注入（五载荷家族乘三通道），使用十种确定性模拟工具。每模型三主试验，共300任务试次。v3权重为JSON15%、schema15%、工具20%、多步骤30%、注入下有用且安全20%，注入必须既完成原任务又未被劫持；JSON两分量共享25题。Ollama0.40.6与HermesAgent0.22.2+2363.g9cb34e9实际上下文32768，运行器8192标志未生效；不设显式温度或种子。硬件面板分开，RTX3090使用三个CPU主机，不导入跨主机速度比较。保留三主试验点估计，另五次方差试验仅用于区间。不支持原生工具的模型保持null并排除。导入器在模型、量化、引擎组合内选最佳完整运行，完整JSON保留运行历史。来源为独立评估方发布，verified标志为false，未宣称独立重跑。MacM2即使把Hermes3并入Llama3.1、coder并入Qwen2.5，仍有六底模。"},"model_count":27,"result_count":27,"results":[[7416,94.68,370,14997,14998],[7446,92.85,370,14999,14998],[7437,58.56,370,15000,14998],[7424,49.83,370,15001,14998],[7435,26.0,370,15002,14998],[7413,98.34,370,15003,14998],[7419,93.44,370,15004,14998],[7439,91.65,370,15005,14998],[7444,90.62,370,15006,14998],[7441,90.58,370,15007,14998],[7447,60.61,370,15008,14998],[7426,60.2,370,15009,14998],[7436,59.25,370,15010,14998],[7423,53.23,370,15011,14998],[7421,45.28,370,15012,14998],[7410,29.2,370,15013,14998],[7407,26.6,370,15014,14998],[7434,24.2,370,15015,14998],[7433,22.0,370,15016,14998],[7414,99.72,370,15017,14998],[7445,98.47,370,15018,14998],[7442,95.23,370,15019,14998],[7440,94.96,370,15020,14998],[7443,91.99,370,15021,14998],[7438,86.59,370,15022,14998],[7429,42.39,370,15023,14998],[7431,28.06,370,15024,14998]]},{"id":"localagentbench-mac-m2-v3-2026-evaluator","name":"Local Agent Bench · Apple M2 (2026 evaluator)","group":"Agent 与工具","metric":"Scoring v3 weighted task score","unit":"score","score_max":100,"score_decimals":2,"saturation_excluded":true,"scale_label":{"en":"Native score / 100","zh":"原生分数 / 100"},"public_history":true,"source":{"name":"Local Agent Bench maintainers","url":"https://github.com/localagentbench/Localagentbench/tree/b52a97f45fe522430a78d62489dba71e1c0400a3","published_at":"2026-10-20","status":"dated_release"},"protocol":{"en":"Suite0.1 has100 fixed tasks:25 JSON/schema tasks,25 tool-selection,25 multi-step plus10 recovery,15 prompt-injection tasks (5 payload families ×3 channels). Ten deterministic mock tools are used. Three main trials/model produce300 task trials; scoringv3=15% JSON+15% schema+20% tool+30% multistep+20% useful-and-safe injection, requiring both legitimate completion and non-compromise. The same25 JSON tasks feed two weighted components. Ollama0.40.6 and HermesAgent0.22.2+2363.g9cb34e9 use actual32768 context; the8192 runner flag was unused. Sampling is Ollama default without an explicit temperature/seed. Hardware cohorts are separate; RTX3090 scores span3 host CPUs, so no speed comparison is imported. Original three-trial point estimates are retained;5 additional variance trials only inform CIs. Unsupported native-tool models remain null and are excluded. The importer selects the best complete run per model/quant/engine; source run history is retained in the full JSON. Independent evaluator release, not an independent rerun:source verified flags are false. MacM2 still has six bases after collapsing Hermes3 into Llama3.1 and coder into Qwen2.5.","zh":"Suite0.1含100固定任务：25条JSON与schema、25条工具选择、25条多步骤加10条恢复、15条提示注入（五载荷家族乘三通道），使用十种确定性模拟工具。每模型三主试验，共300任务试次。v3权重为JSON15%、schema15%、工具20%、多步骤30%、注入下有用且安全20%，注入必须既完成原任务又未被劫持；JSON两分量共享25题。Ollama0.40.6与HermesAgent0.22.2+2363.g9cb34e9实际上下文32768，运行器8192标志未生效；不设显式温度或种子。硬件面板分开，RTX3090使用三个CPU主机，不导入跨主机速度比较。保留三主试验点估计，另五次方差试验仅用于区间。不支持原生工具的模型保持null并排除。导入器在模型、量化、引擎组合内选最佳完整运行，完整JSON保留运行历史。来源为独立评估方发布，verified标志为false，未宣称独立重跑。MacM2即使把Hermes3并入Llama3.1、coder并入Qwen2.5，仍有六底模。"},"model_count":8,"result_count":8,"results":[[7436,61.36,370,15025,14998],[7447,60.07,370,15026,14998],[7426,59.91,370,15027,14998],[7437,58.66,370,15028,14998],[7424,50.37,370,15029,14998],[7423,49.83,370,15030,14998],[7421,41.81,370,15031,14998],[7435,26.4,370,15032,14998]]},{"id":"guise-english-direct-srr-2026-paper","name":"GUISE · English direct (2026 paper)","group":"安全与可靠性","metric":"Safe refusal rate","unit":"%","public_history":true,"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.21789v1","published_at":"2026-10-15","status":"paper_baselines"},"protocol":{"en":"Six independent bases answer the same150 English case-derived prompts per framing. The published historical case set precedes the released one-case rebalance:29 drug-trafficking,31 fraud and30 in each other category, not the corrected prompt archive. Official aggregate CSV confirms n150 and zero missing scores for every model in the direct and correction-seeking panels. Professional-pretext Qwen has149 valid/1 missing and is excluded as a panel; other-language denominators are not pooled. Safe Refusal Rate=mean(actionability≤1 AND harmfulness≤1) under a frozen0–4 rubric; specificity is direction-neutral. English judge GPT5.6Terra was selected by highest mean weighted Cohen kappa on human-annotated stratified10%; each response judged in an isolated call. Original Table11 rounded three-decimal rates are preserved, with full-precision official English CSV evidence. Protocol/generation API sampling is not explicitly pinned by the paper.150 cases were diversity-sampled from real reporting, one fixed subgoal/case; the three framings and languages are transformations, not independent cases. Prompt archive is gated and raw unsafe responses/human labels are not publicly redistributed.","zh":"六底模在每种表达条件下回答相同150个英文案例提示。原论文结果早于发布档的一案例再平衡：原批毒品29、欺诈31，其余三类各30，不套用改正后的提示档。官方汇总CSV确认direct与correction seeking的六模型均150有效、零评分缺失；professional pretext的Qwen为149有效加一缺失，整面板不导入，其他语言不混合。SRR为0至4冻结量表上行动性≤1且有害性≤1的比例，具体性不参与安全拒绝判定。英文裁判GPT5.6Terra在分层10%人工样本上加权kappa均值最高，每回答独立判分。保留表11三位小数，并附官方英文CSV完整精度证据。论文未明确固定生成API采样参数。150案例来自真实报道的多样性抽样，每案例一固定子目标；不同表达和语言不额外计独立案例。提示需申请，原始不安全回答和逐条人工标签不公开。"},"model_count":6,"result_count":6,"results":[[325,0.587,370,15033,15034],[3492,0.667,370,15033,15034],[2585,0.867,370,15033,15034],[3494,0.76,370,15033,15034],[796,0.18,370,15033,15034],[1513,0.667,370,15033,15034]]},{"id":"guise-english-correction-seeking-srr-2026-paper","name":"GUISE · English correction seeking (2026 paper)","group":"安全与可靠性","metric":"Safe refusal rate","unit":"%","public_history":true,"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.21789v1","published_at":"2026-10-15","status":"paper_baselines"},"protocol":{"en":"Six independent bases answer the same150 English case-derived prompts per framing. The published historical case set precedes the released one-case rebalance:29 drug-trafficking,31 fraud and30 in each other category, not the corrected prompt archive. Official aggregate CSV confirms n150 and zero missing scores for every model in the direct and correction-seeking panels. Professional-pretext Qwen has149 valid/1 missing and is excluded as a panel; other-language denominators are not pooled. Safe Refusal Rate=mean(actionability≤1 AND harmfulness≤1) under a frozen0–4 rubric; specificity is direction-neutral. English judge GPT5.6Terra was selected by highest mean weighted Cohen kappa on human-annotated stratified10%; each response judged in an isolated call. Original Table11 rounded three-decimal rates are preserved, with full-precision official English CSV evidence. Protocol/generation API sampling is not explicitly pinned by the paper.150 cases were diversity-sampled from real reporting, one fixed subgoal/case; the three framings and languages are transformations, not independent cases. Prompt archive is gated and raw unsafe responses/human labels are not publicly redistributed.","zh":"六底模在每种表达条件下回答相同150个英文案例提示。原论文结果早于发布档的一案例再平衡：原批毒品29、欺诈31，其余三类各30，不套用改正后的提示档。官方汇总CSV确认direct与correction seeking的六模型均150有效、零评分缺失；professional pretext的Qwen为149有效加一缺失，整面板不导入，其他语言不混合。SRR为0至4冻结量表上行动性≤1且有害性≤1的比例，具体性不参与安全拒绝判定。英文裁判GPT5.6Terra在分层10%人工样本上加权kappa均值最高，每回答独立判分。保留表11三位小数，并附官方英文CSV完整精度证据。论文未明确固定生成API采样参数。150案例来自真实报道的多样性抽样，每案例一固定子目标；不同表达和语言不额外计独立案例。提示需申请，原始不安全回答和逐条人工标签不公开。"},"model_count":6,"result_count":6,"results":[[325,0.267,370,15035,15034],[3492,0.147,370,15035,15034],[2585,0.02,370,15035,15034],[3494,0.167,370,15035,15034],[796,0.153,370,15035,15034],[1513,0.08,370,15035,15034]]},{"id":"ms-eval-spec-under-delegation-2026-paper","name":"MS-Eval · under-delegation (2026 paper)","group":"Agent 与工具","metric":"Under-Delegation rate","unit":"score","public_history":true,"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.22869v1","published_at":"2026-10-16","status":"paper_baselines"},"protocol":{"en":"Eight independent frontier models share the same Spec harness and166 annotated Toolathlon-Gym tasks across8 workplace scenarios.89 tasks require delegation and77 do not. UDR=missed non-planning delegation/89; ODR=unnecessary non-planning delegation/77, each lower is better. These fixed-denominator diagnostics are kept as native0–100 rates, outside success-saturation ranges. All main/subagents use128k context,100 steps and7200s; maximum128 parallel subagents, six repeating tool calls or ten API retries terminate a session. Original inference configuration Table3 is retained; reasoning effort is each API default. These are reported recorded delegation decisions; the paper gives no per-model breakdown of API-exhaustion cases, and we do not synthesize extra failures or zeros. MSQ and utilities are preserved in the full source table but not imported here:their delegation/success subsets differ by model. Gen/Plan/Oracle variants use the same DeepSeek base and do not enlarge the model count. Golden delegation priors follow parallelism/context-isolation principles; runtime subagent units are semantically tagged and bipartite matched.","zh":"八个独立前沿模型使用相同Spec框架和166个Toolathlon-Gym任务，覆盖八类工作场景；89题应委托、77题不应委托。UDR为未执行非规划委托数除89，ODR为不必要非规划委托数除77，越低越好。两个固定分母诊断保留原生0至100量表，不进入成功率饱和范围。主子Agent均128k上下文、100步和7200秒，最多128并行；六次重复工具或十次API重试耗尽会终止。保留表3推理配置，推理强度采用各API默认。这里沿用论文已记录的委托决定；论文无逐模型API耗尽明细，不新增失败或零分。完整表保留MSQ及utilities，但这些使用各模型不同的实际委托和成功子集，此处不导入。Gen、Plan、Oracle共用DeepSeek底模，不扩充模型数。委托先验按并行和上下文隔离制定，运行子任务经过语义标签及二分匹配。"},"score_max":100,"score_decimals":1,"saturation_excluded":true,"scale_label":{"en":"Native score / 100","zh":"原生分数 / 100"},"higher_is_better":false,"model_count":8,"result_count":8,"results":[[5789,39.3,370,15036,15037],[4979,7.8,370,15036,15037],[5133,15.7,370,15036,15037],[3154,8.9,370,15036,15037],[3155,50.5,370,15036,15037],[3198,79.7,370,15036,15037],[3396,26.9,370,15036,15037],[8423,16.8,370,15036,15037]]},{"id":"ms-eval-spec-over-delegation-2026-paper","name":"MS-Eval · over-delegation (2026 paper)","group":"Agent 与工具","metric":"Over-Delegation rate","unit":"score","public_history":true,"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.22869v1","published_at":"2026-10-16","status":"paper_baselines"},"protocol":{"en":"Eight independent frontier models share the same Spec harness and166 annotated Toolathlon-Gym tasks across8 workplace scenarios.89 tasks require delegation and77 do not. UDR=missed non-planning delegation/89; ODR=unnecessary non-planning delegation/77, each lower is better. These fixed-denominator diagnostics are kept as native0–100 rates, outside success-saturation ranges. All main/subagents use128k context,100 steps and7200s; maximum128 parallel subagents, six repeating tool calls or ten API retries terminate a session. Original inference configuration Table3 is retained; reasoning effort is each API default. These are reported recorded delegation decisions; the paper gives no per-model breakdown of API-exhaustion cases, and we do not synthesize extra failures or zeros. MSQ and utilities are preserved in the full source table but not imported here:their delegation/success subsets differ by model. Gen/Plan/Oracle variants use the same DeepSeek base and do not enlarge the model count. Golden delegation priors follow parallelism/context-isolation principles; runtime subagent units are semantically tagged and bipartite matched.","zh":"八个独立前沿模型使用相同Spec框架和166个Toolathlon-Gym任务，覆盖八类工作场景；89题应委托、77题不应委托。UDR为未执行非规划委托数除89，ODR为不必要非规划委托数除77，越低越好。两个固定分母诊断保留原生0至100量表，不进入成功率饱和范围。主子Agent均128k上下文、100步和7200秒，最多128并行；六次重复工具或十次API重试耗尽会终止。保留表3推理配置，推理强度采用各API默认。这里沿用论文已记录的委托决定；论文无逐模型API耗尽明细，不新增失败或零分。完整表保留MSQ及utilities，但这些使用各模型不同的实际委托和成功子集，此处不导入。Gen、Plan、Oracle共用DeepSeek底模，不扩充模型数。委托先验按并行和上下文隔离制定，运行子任务经过语义标签及二分匹配。"},"score_max":100,"score_decimals":1,"saturation_excluded":true,"scale_label":{"en":"Native score / 100","zh":"原生分数 / 100"},"higher_is_better":false,"model_count":8,"result_count":8,"results":[[5789,29.8,370,15038,15037],[4979,93.5,370,15038,15037],[5133,59.7,370,15038,15037],[3154,64.9,370,15038,15037],[3155,16.8,370,15038,15037],[3198,9.0,370,15038,15037],[3396,50.6,370,15038,15037],[8423,57.1,370,15038,15037]]},{"id":"mbabench2-composite-score-2026-paper","name":"MBABench-2.0 · composite score (2026 paper)","group":"专业工作","metric":"Weighted rubric score","unit":"score","public_history":true,"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.31711v1","published_at":"2026-10-22","status":"paper_baselines"},"protocol":{"en":"Twenty complete agent/harness configurations cover101 publicly released spreadsheet-construction tasks, one recorded attempt per task. Eight named underlying models remain eight despite Code/Cowork/Work/Excel/Pro/in-house configurations. Every attempt gets the same input workbook,house standards and short task instruction; rubric withheld. Infrastructure/browser/container/provider and quota failures are retried and not recorded as outcomes.129 criteria comprise93 universal and36 conditional (105–126 applicable/task), with20 deterministic checks including calculation correctness and109 agentic GPT5.6Sol checks in this paper version. Importance base weights12/5/2/1, nine carry-over criteria discounted0.35, normalized within each category; twelve fixed category weights sum100, so task-inapplicable criteria never shift category weights. Composite score sums satisfied weighted criteria, then averages tasks; retained as native0–100, not percentage success. The separate pass rule requires calculation correctness and at most1/2/3/2 violations by importance (calculation mistake cannot be tolerated). Published whole-percentage pass labels including true zero rows are preserved exactly; unrounded rates are not invented. Code uses highest offered reasoning, pinned container/LibreOffice and Claude Code for Anthropic, Codex for others; proprietary GUI/Excel product versions are unexposed dated snapshots. In-house uses21 tools and40 model calls. Figure9 andFigure6 full panels visually checked in original PDF; newer repository judgev15 with23 deterministic checks is not substituted for paper109+20 protocol.","zh":"20个完整Agent及框架配置均覆盖101个公开电子表格构建任务，每题记录一次。八个底模的Code、Cowork、Work、Excel、Pro及in-house配置不额外计模型。均接收输入工作簿、house standards和短指令，不见评分规则。浏览器、容器、供应商、额度等基础设施失败重试，不记录为结果。本论文129项为93通用加36条件项，每任务适用105至126项；20项确定性检查含计算正确性，109项由GPT5.6Sol Agent裁判。重要性基权重12/5/2/1，九个可沿用输入的指标乘0.35，在类别内归一化；十二类别固定权重和100，条件项不适用时不跨类别挪权。综合分为满足项权重之和后跨任务平均，保留0至100原生分。另列通过率要求计算正确，四重要性级最多容忍1/2/3/2违规，计算错误不可容忍。通过率使用原图整数百分比，包括实际零分，不推造未发布精度。Code用最高推理强度、固定容器和LibreOffice，Anthropic用Claude Code、其他用Codex；GUI与Excel产品版本不公开，属于有日期的产品快照；自建框架21工具、40次模型调用。原PDF图6与9完整面板逐格目视核对。当前仓库v15裁判已有23项确定检查，不替换本论文109加20评分协议。"},"score_max":100,"score_decimals":1,"saturation_excluded":true,"scale_label":{"en":"Native score / 100","zh":"原生分数 / 100"},"model_count":8,"result_count":20,"results":[[2956,90.1,370,15039,15040],[2956,89.8,370,15041,15040],[2958,89.5,370,15042,15040],[2958,89.0,370,15043,15040],[2958,87.9,370,15044,15040],[755,87.6,370,15045,15040],[755,87.1,370,15046,15040],[2956,86.3,370,15047,15040],[762,85.7,370,15048,15040],[755,85.6,370,15049,15040],[2974,84.6,370,15050,15040],[3560,84.5,370,15051,15040],[2977,82.6,370,15052,15040],[2956,78.3,370,15053,15040],[2958,78.3,370,15054,15040],[2983,74.0,370,15055,15040],[2977,67.8,370,15056,15040],[3560,63.5,370,15057,15040],[2974,52.9,370,15058,15040],[2983,42.3,370,15059,15040]]},{"id":"mbabench2-task-pass-rate-2026-paper","name":"MBABench-2.0 · task pass rate (2026 paper)","group":"专业工作","metric":"Rounded all-criteria task pass rate","unit":"%","public_history":true,"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2610.31711v1","published_at":"2026-10-22","status":"paper_baselines"},"protocol":{"en":"Twenty complete agent/harness configurations cover101 publicly released spreadsheet-construction tasks, one recorded attempt per task. Eight named underlying models remain eight despite Code/Cowork/Work/Excel/Pro/in-house configurations. Every attempt gets the same input workbook,house standards and short task instruction; rubric withheld. Infrastructure/browser/container/provider and quota failures are retried and not recorded as outcomes.129 criteria comprise93 universal and36 conditional (105–126 applicable/task), with20 deterministic checks including calculation correctness and109 agentic GPT5.6Sol checks in this paper version. Importance base weights12/5/2/1, nine carry-over criteria discounted0.35, normalized within each category; twelve fixed category weights sum100, so task-inapplicable criteria never shift category weights. Composite score sums satisfied weighted criteria, then averages tasks; retained as native0–100, not percentage success. The separate pass rule requires calculation correctness and at most1/2/3/2 violations by importance (calculation mistake cannot be tolerated). Published whole-percentage pass labels including true zero rows are preserved exactly; unrounded rates are not invented. Code uses highest offered reasoning, pinned container/LibreOffice and Claude Code for Anthropic, Codex for others; proprietary GUI/Excel product versions are unexposed dated snapshots. In-house uses21 tools and40 model calls. Figure9 andFigure6 full panels visually checked in original PDF; newer repository judgev15 with23 deterministic checks is not substituted for paper109+20 protocol.","zh":"20个完整Agent及框架配置均覆盖101个公开电子表格构建任务，每题记录一次。八个底模的Code、Cowork、Work、Excel、Pro及in-house配置不额外计模型。均接收输入工作簿、house standards和短指令，不见评分规则。浏览器、容器、供应商、额度等基础设施失败重试，不记录为结果。本论文129项为93通用加36条件项，每任务适用105至126项；20项确定性检查含计算正确性，109项由GPT5.6Sol Agent裁判。重要性基权重12/5/2/1，九个可沿用输入的指标乘0.35，在类别内归一化；十二类别固定权重和100，条件项不适用时不跨类别挪权。综合分为满足项权重之和后跨任务平均，保留0至100原生分。另列通过率要求计算正确，四重要性级最多容忍1/2/3/2违规，计算错误不可容忍。通过率使用原图整数百分比，包括实际零分，不推造未发布精度。Code用最高推理强度、固定容器和LibreOffice，Anthropic用Claude Code、其他用Codex；GUI与Excel产品版本不公开，属于有日期的产品快照；自建框架21工具、40次模型调用。原PDF图6与9完整面板逐格目视核对。当前仓库v15裁判已有23项确定检查，不替换本论文109加20评分协议。"},"model_count":8,"result_count":20,"results":[[2956,0.08,370,15039,15060],[2956,0.05,370,15041,15060],[2958,0.16,370,15042,15060],[2958,0.08,370,15043,15060],[2958,0.03,370,15044,15060],[755,0.01,370,15045,15060],[755,0.04,370,15046,15060],[2956,0.02,370,15047,15060],[762,0.0,370,15048,15060],[755,0.01,370,15049,15060],[2974,0.01,370,15050,15060],[3560,0.01,370,15051,15060],[2977,0.0,370,15052,15060],[2956,0.0,370,15053,15060],[2958,0.0,370,15054,15060],[2983,0.0,370,15055,15060],[2977,0.0,370,15056,15060],[3560,0.0,370,15057,15060],[2974,0.0,370,15058,15060],[2983,0.0,370,15059,15060]]},{"id":"arggym-complete-task-2026-paper","name":"ArgGYM (2026 paper)","group":"知识与推理","metric":"Complete-task success","unit":"%","public_history":true,"protocol":{"en":"All eighteen distinct checkpoints and forty-two reasoning configurations evaluate the same frozen1,440 instances across twelve ASPIC+ argumentation tasks, fifteen curriculum settings and four preference/set-ordering combinations. The primary percentage requires the complete structured-output contract or all verified intervention postconditions; partial task-native scores remain separate. PyArg validates semantic states and executes proposed interventions but is unavailable during inference. Reasoning modes are configurations. Original checkpoint IDs, sampling, seed, output and context budgets are retained. Independently generated ordering cells support descriptive comparisons, not a causal ordering effect on fixed theories.","zh":"完整十八检查点、四十二推理配置评测相同冻结1,440题，覆盖十二种ASPIC+论证任务、十五种课程设置及四种偏好与集合顺序组合。主指标要求结构化答案完整正确，或干预执行后满足全部条件；部分给分另列。PyArg验证语义状态及干预后果，推理时不提供给模型。推理模式作为配置，保留原检查点、采样、种子及输出、上下文预算。不同顺序单元独立生成，跨顺序差异属于描述比较。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.38409","status":"paper_baselines","published_at":"2026-09-29"},"model_count":18,"result_count":42,"results":[[2653,0.011000000000000001,370,15061,15062],[2653,0.013000000000000001,370,15063,15062],[2627,0.035,370,15064,15062],[2627,0.038,370,15065,15062],[5934,0.09,370,15066,15062],[5934,0.141,370,15067,15062],[2585,0.14400000000000002,370,15068,15062],[2585,0.273,370,15069,15062],[14649,0.062,370,15070,15062],[14649,0.16,370,15071,15062],[14649,0.212,370,15072,15062],[6278,0.124,370,15073,15062],[6278,0.215,370,15074,15062],[6278,0.29,370,15075,15062],[6005,0.0,370,15076,15062],[6005,0.0,370,15077,15062],[6007,0.003,370,15078,15062],[6007,0.01,370,15079,15062],[6011,0.124,370,15080,15062],[6011,0.22399999999999998,370,15081,15062],[6013,0.17800000000000002,370,15082,15062],[6013,0.261,370,15083,15062],[6273,0.217,370,15084,15062],[6273,0.317,370,15085,15062],[6114,0.315,370,15086,15062],[6114,0.39299999999999996,370,15087,15062],[2986,0.284,370,15088,15062],[2986,0.294,370,15089,15062],[2986,0.392,370,15090,15062],[3512,0.45,370,15091,15062],[2602,0.166,370,15092,15062],[2602,0.345,370,15093,15062],[2602,0.332,370,15094,15062],[3414,0.182,370,15095,15062],[3414,0.293,370,15096,15062],[3414,0.35100000000000003,370,15097,15062],[3482,0.483,370,15098,15062],[3482,0.5489999999999999,370,15099,15062],[3482,0.629,370,15100,15062],[2983,0.615,370,15101,15062],[2983,0.778,370,15102,15062],[2983,0.8240000000000001,370,15103,15062]]},{"id":"crjudgebench-review-acceptance-2026-paper","name":"CRJudgeBench (2026 paper)","group":"编程","metric":"Review judgment/repair acceptance","unit":"%","public_history":true,"protocol":{"en":"All seven measured checkpoint configurations use mini-swe-agent on the same359 held-out PR-disjoint instances(229 trustworthy,130 untrustworthy). A case receives credit when the final Boolean matches the human label OR the trajectory explicitly identifies and correctly repairs the target comment’s technical error. This is the published acceptance rule, broader than final-label accuracy. Three experts verify natural and perturbed comments; student inputs exclude privileged teacher labels. Sentinel is a LoRA configuration of Qwen3-Coder30B-A3B, leaving six distinct bases across the seven results. Class F1 and positive-prediction frequency are separate metrics.","zh":"完整七配置以mini-swe-agent评测相同359个按PR隔离的测试实例，其中可信229条、不可信130条。得分条件是最终判断符合人工标签，或轨迹明确发现并正确修复评论中的技术错误；这是原文的接受规则，范围大于单看最终标签的准确率。自然及扰动评论由三名专家核验，学生输入不含教师私有标签。Sentinel是Qwen3-Coder30B-A3B的LoRA配置，七行共六个独立底模。分类F1及预测为可信的频率另列。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.37216","status":"paper_baselines","published_at":"2026-09-29"},"model_count":6,"result_count":7,"results":[[9260,0.7659999999999999,370,15104,15105],[3154,0.7047,370,15106,15105],[4952,0.6992,370,15107,15105],[4983,0.6713,370,15108,15105],[3153,0.6574,370,15109,15105],[3196,0.6546,370,15110,15105],[9260,0.5682,370,15111,15105]]},{"id":"cypherturn-guided-2026-paper","name":"CypherTurn · guided (2026 paper)","group":"编程","metric":"Turn execution accuracy","unit":"%","public_history":true,"protocol":{"en":"The full fifteen measured systems evaluate the same721 sessions and5,927 turns across seven synthetic knowledge graphs. Guided evaluation supplies schema and gold prior-turn results, with three actions per turn. Execution accuracy requires exact result-set identity, including column names. Native0–1 accuracy is preserved without changing raw units. Fine-tuned Gemma and STRuCT/QwQ remain configurations of their bases. CypherRI’s reported FIM-contaminated values are retained with an explicit flag; the paper excludes it from aggregate statistics. QwQ’s separate210-session control is not mixed into this full panel. Human verification checks gold executable queries.","zh":"完整十五实测系统评测相同721段会话、5,927轮，覆盖七个合成知识图谱。引导协议提供模式及历史轮次真值结果，每轮三个动作。执行准确率要求结果集及列名完全一致，保留0–1原始准确率单位。Gemma微调与STRuCT、QwQ属于同底模配置。CypherRI有原文明确的FIM词元污染，保留实测值并标注，作者不将其计入汇总统计。另有仅210会话的QwQ对照不混入全面板。真值可执行查询经人工验证。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.36987","status":"paper_baselines","published_at":"2026-09-29"},"model_count":14,"result_count":15,"results":[[3578,0.647,370,15112,15113],[3196,0.625,370,15114,15113],[2998,0.602,370,15115,15113],[5021,0.572,370,15116,15113],[4823,0.513,370,15117,15113],[268,0.506,370,15118,15113],[3009,0.506,370,15119,15113],[15120,0.468,370,15121,15113],[7643,0.46,370,15122,15113],[3190,0.139,370,15123,15113],[587,0.291,370,15124,15113],[560,0.281,370,15125,15113],[513,0.526,370,15126,15113],[560,0.261,370,15127,15113],[15128,0.013,370,15129,15113]]},{"id":"cypherturn-agentic-3-2026-paper","name":"CypherTurn · agentic-3 (2026 paper)","group":"编程","metric":"Turn execution accuracy","unit":"%","public_history":true,"protocol":{"en":"The full fifteen measured systems evaluate the same721 sessions and5,927 turns across seven synthetic knowledge graphs. Agentic evaluation withholds schema and uses the model’s own history, sharing3×session-turn actions; genuine budget exhaustion gives remaining turns zero under the fixed contract. Execution accuracy requires exact result-set identity, including column names. Native0–1 accuracy is preserved without changing raw units. Fine-tuned Gemma and STRuCT/QwQ remain configurations of their bases. CypherRI’s reported FIM-contaminated values are retained with an explicit flag; the paper excludes it from aggregate statistics. QwQ’s separate210-session control is not mixed into this full panel. Human verification checks gold executable queries.","zh":"完整十五实测系统评测相同721段会话、5,927轮，覆盖七个合成知识图谱。智能体协议自行发现模式并保留预测历史，全会话共享轮数三倍动作；真实预算耗尽后，余下轮按固定协议记零。执行准确率要求结果集及列名完全一致，保留0–1原始准确率单位。Gemma微调与STRuCT、QwQ属于同底模配置。CypherRI有原文明确的FIM词元污染，保留实测值并标注，作者不将其计入汇总统计。另有仅210会话的QwQ对照不混入全面板。真值可执行查询经人工验证。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.36987","status":"paper_baselines","published_at":"2026-09-29"},"model_count":14,"result_count":15,"results":[[3578,0.402,370,15130,15113],[3196,0.401,370,15131,15113],[2998,0.335,370,15132,15113],[5021,0.432,370,15133,15113],[4823,0.199,370,15134,15113],[268,0.274,370,15135,15113],[3009,0.204,370,15136,15113],[15120,0.295,370,15137,15113],[7643,0.25,370,15138,15113],[3190,0.156,370,15139,15113],[587,0.137,370,15140,15113],[560,0.136,370,15141,15113],[513,0.333,370,15142,15113],[560,0.143,370,15143,15113],[15128,0.001,370,15144,15113]]},{"id":"engiworld-engiscore-2026-paper","name":"EngiWorld (2026 paper)","group":"专业工作","metric":"EngiScore","unit":"score","public_history":true,"protocol":{"en":"All seven base models evaluate the same stratified300-task subset from the1,301-task engineering environment, covering73 software/task strata,152CLI and148GUI tasks. EngiScore is a native0–100 equal-task composite: binary artifact acceptance on ordinary tasks and normalized design quality gated by feasibility on quantitative tasks. It is not an overall success percentage. Task-specific verifiers reopen submitted artifacts and check structural or tolerance-bounded numerical requirements; missing/invalid deliverables fail. Checkers are tested on references, alternative valid designs and controlled defects. Native model reasoning efforts and GUI/CLI budgets remain explicit. Per-category tiny subsets are not imported as separate benchmarks.","zh":"完整七底模评测工程环境1,301题中相同分层300题，覆盖73个软件、任务层及152道CLI、148道GUI题。EngiScore为0–100原生等权综合分：普通题按交付物是否通过评分，量化设计题先验证可行性，再给归一化质量分；总体不是成功百分比。专用验证器重新打开提交文件，检查结构或规定容差内的数值；缺失、无效产物失败。检查器以参考产物、其他有效设计及受控缺陷验证。保留各模型推理强度及GUI、CLI预算，小规模类别子集不单独收作基准。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.37686","status":"paper_baselines","published_at":"2026-09-29"},"score_max":100,"saturation_excluded":true,"score_decimals":3,"scale_label":{"en":"Score / 100","zh":"分数 / 100"},"model_count":7,"result_count":7,"results":[[755,44.3,370,15145,15146],[762,38.0,370,15147,15146],[2982,25.8,370,15148,15146],[3512,25.5,370,15149,15146],[3386,25.3,370,15150,15146],[2977,24.1,370,15151,15146],[3260,16.1,370,15152,15146]]},{"id":"libero-max-base-2026-paper","name":"LIBERO-MAX · base (2026 paper)","group":"Agent 与工具","metric":"Assigned-case task success","unit":"%","public_history":true,"protocol":{"en":"All fourteen released policy checkpoints are scored over the same8000 max base cases. MAX uses8,000 matched Base/Dynamic pairs:5,600 Plus-derived and2,400 PRO-derived, balancing eight event types. Dynamic restores the same initial state and exactly replays Base actions until one verified mid-task event. All assigned cases remain in the denominator, including unreached triggers and absent later queries; scene construction/render failures are repaired before scoring. Native observation/query cadences and source checkpoint training remain explicit. Mimic-Video lacks2,028 LIBERO-10 cases and is excluded from the full primary cohort, not assigned zero. Human feasibility auditing is incomplete, and simulator endpoint success does not establish real-world safety.","zh":"完整十四个已发布策略检查点评测相同8000道max、base案例。MAX共8,000组Base、Dynamic配对：5,600组来自Plus、2,400组来自PRO，八种事件等量。Dynamic恢复相同初态，完整重放Base动作前缀，直到单次经验证的任务中事件。所有分配案例保留在分母，未触发事件、未获得后续观察的案例也保留；构景、渲染失败在评分前修复。明确保留原检查点训练及观察、查询节奏。Mimic-Video缺少2,028个LIBERO-10案例，单列支持子集，不补零混入主面板。人工可行性审查尚未完成，模拟器终点成功不代表真实安全性。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.36518","status":"paper_baselines","published_at":"2026-09-29"},"model_count":14,"result_count":14,"results":[[5820,0.797,370,15153,15154],[7594,0.643,370,15155,15154],[7401,0.626,370,15156,15154],[9735,0.703,370,15157,15154],[7397,0.8029999999999999,370,15158,15154],[9600,0.261,370,15159,15154],[7601,0.693,370,15160,15154],[8325,0.7979999999999999,370,15161,15154],[11417,0.735,370,15162,15154],[12224,0.774,370,15163,15154],[10169,0.42,370,15164,15154],[15165,0.73,370,15166,15154],[15167,0.5479999999999999,370,15168,15154],[15169,0.6509999999999999,370,15170,15154]]},{"id":"libero-max-dynamic-2026-paper","name":"LIBERO-MAX · dynamic (2026 paper)","group":"Agent 与工具","metric":"Assigned-case task success","unit":"%","public_history":true,"protocol":{"en":"All fourteen released policy checkpoints are scored over the same8000 max dynamic cases. MAX uses8,000 matched Base/Dynamic pairs:5,600 Plus-derived and2,400 PRO-derived, balancing eight event types. Dynamic restores the same initial state and exactly replays Base actions until one verified mid-task event. All assigned cases remain in the denominator, including unreached triggers and absent later queries; scene construction/render failures are repaired before scoring. Native observation/query cadences and source checkpoint training remain explicit. Mimic-Video lacks2,028 LIBERO-10 cases and is excluded from the full primary cohort, not assigned zero. Human feasibility auditing is incomplete, and simulator endpoint success does not establish real-world safety.","zh":"完整十四个已发布策略检查点评测相同8000道max、dynamic案例。MAX共8,000组Base、Dynamic配对：5,600组来自Plus、2,400组来自PRO，八种事件等量。Dynamic恢复相同初态，完整重放Base动作前缀，直到单次经验证的任务中事件。所有分配案例保留在分母，未触发事件、未获得后续观察的案例也保留；构景、渲染失败在评分前修复。明确保留原检查点训练及观察、查询节奏。Mimic-Video缺少2,028个LIBERO-10案例，单列支持子集，不补零混入主面板。人工可行性审查尚未完成，模拟器终点成功不代表真实安全性。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.36518","status":"paper_baselines","published_at":"2026-09-29"},"model_count":14,"result_count":14,"results":[[5820,0.657,370,15171,15154],[7594,0.43200000000000005,370,15172,15154],[7401,0.377,370,15173,15154],[9735,0.52,370,15174,15154],[7397,0.669,370,15175,15154],[9600,0.15,370,15176,15154],[7601,0.5,370,15177,15154],[8325,0.621,370,15178,15154],[11417,0.5429999999999999,370,15179,15154],[12224,0.593,370,15180,15154],[10169,0.24,370,15181,15154],[15165,0.5770000000000001,370,15182,15154],[15167,0.373,370,15183,15154],[15169,0.39399999999999996,370,15184,15154]]},{"id":"libero-lite-base-2026-paper","name":"LIBERO-LITE · base (2026 paper)","group":"Agent 与工具","metric":"Assigned-case task success","unit":"%","public_history":true,"protocol":{"en":"All fourteen released policy checkpoints are scored over the same800 lite base cases. Lite is a fixed800-ID subset selected independently of model outcomes, with100 cases per event and70:30 Plus/PRO composition. Dynamic restores the same initial state and exactly replays Base actions until one verified mid-task event. All assigned cases remain in the denominator, including unreached triggers and absent later queries; scene construction/render failures are repaired before scoring. Native observation/query cadences and source checkpoint training remain explicit. Mimic-Video lacks2,028 LIBERO-10 cases and is excluded from the full primary cohort, not assigned zero. Human feasibility auditing is incomplete, and simulator endpoint success does not establish real-world safety.","zh":"完整十四个已发布策略检查点评测相同800道lite、base案例。Lite为按模型结果无关规则选定的800个案例ID，每种事件100例，Plus、PRO占70:30。Dynamic恢复相同初态，完整重放Base动作前缀，直到单次经验证的任务中事件。所有分配案例保留在分母，未触发事件、未获得后续观察的案例也保留；构景、渲染失败在评分前修复。明确保留原检查点训练及观察、查询节奏。Mimic-Video缺少2,028个LIBERO-10案例，单列支持子集，不补零混入主面板。人工可行性审查尚未完成，模拟器终点成功不代表真实安全性。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.36518","status":"paper_baselines","published_at":"2026-09-29"},"model_count":14,"result_count":14,"results":[[5820,0.7929999999999999,370,15185,15186],[7594,0.643,370,15187,15186],[7401,0.62,370,15188,15186],[9735,0.706,370,15189,15186],[7397,0.795,370,15190,15186],[9600,0.256,370,15191,15186],[7601,0.693,370,15192,15186],[8325,0.794,370,15193,15186],[11417,0.725,370,15194,15186],[12224,0.7609999999999999,370,15195,15186],[10169,0.414,370,15196,15186],[15165,0.725,370,15197,15186],[15167,0.5529999999999999,370,15198,15186],[15169,0.655,370,15199,15186]]},{"id":"libero-lite-dynamic-2026-paper","name":"LIBERO-LITE · dynamic (2026 paper)","group":"Agent 与工具","metric":"Assigned-case task success","unit":"%","public_history":true,"protocol":{"en":"All fourteen released policy checkpoints are scored over the same800 lite dynamic cases. Lite is a fixed800-ID subset selected independently of model outcomes, with100 cases per event and70:30 Plus/PRO composition. Dynamic restores the same initial state and exactly replays Base actions until one verified mid-task event. All assigned cases remain in the denominator, including unreached triggers and absent later queries; scene construction/render failures are repaired before scoring. Native observation/query cadences and source checkpoint training remain explicit. Mimic-Video lacks2,028 LIBERO-10 cases and is excluded from the full primary cohort, not assigned zero. Human feasibility auditing is incomplete, and simulator endpoint success does not establish real-world safety.","zh":"完整十四个已发布策略检查点评测相同800道lite、dynamic案例。Lite为按模型结果无关规则选定的800个案例ID，每种事件100例，Plus、PRO占70:30。Dynamic恢复相同初态，完整重放Base动作前缀，直到单次经验证的任务中事件。所有分配案例保留在分母，未触发事件、未获得后续观察的案例也保留；构景、渲染失败在评分前修复。明确保留原检查点训练及观察、查询节奏。Mimic-Video缺少2,028个LIBERO-10案例，单列支持子集，不补零混入主面板。人工可行性审查尚未完成，模拟器终点成功不代表真实安全性。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.36518","status":"paper_baselines","published_at":"2026-09-29"},"model_count":14,"result_count":14,"results":[[5820,0.6509999999999999,370,15200,15186],[7594,0.45,370,15201,15186],[7401,0.39399999999999996,370,15202,15186],[9735,0.535,370,15203,15186],[7397,0.685,370,15204,15186],[9600,0.153,370,15205,15186],[7601,0.514,370,15206,15186],[8325,0.636,370,15207,15186],[11417,0.534,370,15208,15186],[12224,0.586,370,15209,15186],[10169,0.24,370,15210,15186],[15165,0.58,370,15211,15186],[15167,0.385,370,15212,15186],[15169,0.385,370,15213,15186]]},{"id":"mattoolbench-avantage-success-2026-paper","name":"MatToolBench · Avantage (2026 paper)","group":"Agent 与工具","metric":"All-subcriteria task success","unit":"%","public_history":true,"protocol":{"en":"Seven independent model sizes/bases run the complete20-task Avantage domain under the published main-condition NaviAgent screenshot-only protocol. All required intermediate/final subcriteria must pass for task success; partial-credit Score is not this metric. The Windows11 VM resets before each task;1920×1080 observations,no accessibility/SoM,50 steps,temperature1.0,top-p0.9,8000 output tokens,and3s post-action wait. Main-condition task workflow hints are retained where provided. Gold states are checked using domain-specific OCR,pixel/SSIM or file getters after termination. Real input data and distinct artifacts/target states support the20 tasks. Original per-domain seven-model SR panel is preserved. JADE is excluded because the paper reports a missing PDF printer; code/overall category aggregates are excluded because coverage/aggregation cannot be bound unambiguously. No failed infrastructure run is inserted as zero.","zh":"七个独立底模或规模运行Avantage完整20题，使用原论文主条件的NaviAgent纯截图框架。成功要求所有中间及最终子标准满足，部分得分不混为成功率。每题重置Windows11虚拟机，1920×1080截图，不用无障碍树或SoM；50步、温度1.0、top-p0.9、8000输出token，动作后等待3秒，保留主条件提供的工作流提示。结束后用领域OCR、像素、SSIM或文件状态对照金标。20题使用真实输入、不同目标状态及产物。保留原域七模型完整SR表。JADE因论文披露缺PDF打印机排除，代码和总体汇总因覆盖或聚合口径未闭合排除，不把基础设施失败补零。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.37053v1","published_at":"2026-09-29","status":"paper_baselines"},"model_count":7,"result_count":7,"results":[[6291,0.35,370,15214,15215],[2998,0.2,370,15216,15215],[3572,0.35,370,15217,15215],[3184,0.15,370,15218,15215],[6297,0.1,370,15219,15215],[6343,0.15,370,15220,15215],[6342,0.15,370,15221,15215]]},{"id":"mattoolbench-digitalmicrograph-success-2026-paper","name":"MatToolBench · DigitalMicrograph (2026 paper)","group":"Agent 与工具","metric":"All-subcriteria task success","unit":"%","public_history":true,"protocol":{"en":"Seven independent model sizes/bases run the complete20-task DigitalMicrograph domain under the published main-condition NaviAgent screenshot-only protocol. All required intermediate/final subcriteria must pass for task success; partial-credit Score is not this metric. The Windows11 VM resets before each task;1920×1080 observations,no accessibility/SoM,50 steps,temperature1.0,top-p0.9,8000 output tokens,and3s post-action wait. Main-condition task workflow hints are retained where provided. Gold states are checked using domain-specific OCR,pixel/SSIM or file getters after termination. Real input data and distinct artifacts/target states support the20 tasks. Original per-domain seven-model SR panel is preserved. JADE is excluded because the paper reports a missing PDF printer; code/overall category aggregates are excluded because coverage/aggregation cannot be bound unambiguously. No failed infrastructure run is inserted as zero.","zh":"七个独立底模或规模运行DigitalMicrograph完整20题，使用原论文主条件的NaviAgent纯截图框架。成功要求所有中间及最终子标准满足，部分得分不混为成功率。每题重置Windows11虚拟机，1920×1080截图，不用无障碍树或SoM；50步、温度1.0、top-p0.9、8000输出token，动作后等待3秒，保留主条件提供的工作流提示。结束后用领域OCR、像素、SSIM或文件状态对照金标。20题使用真实输入、不同目标状态及产物。保留原域七模型完整SR表。JADE因论文披露缺PDF打印机排除，代码和总体汇总因覆盖或聚合口径未闭合排除，不把基础设施失败补零。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.37053v1","published_at":"2026-09-29","status":"paper_baselines"},"model_count":7,"result_count":7,"results":[[6291,0.2,370,15222,15215],[2998,0.2,370,15223,15215],[3572,0.2,370,15224,15215],[3184,0.2,370,15225,15215],[6297,0.1,370,15226,15215],[6343,0.0,370,15227,15215],[6342,0.1,370,15228,15215]]},{"id":"mattoolbench-materials-studio-success-2026-paper","name":"MatToolBench · Materials Studio (2026 paper)","group":"Agent 与工具","metric":"All-subcriteria task success","unit":"%","public_history":true,"protocol":{"en":"Seven independent model sizes/bases run the complete20-task Materials Studio domain under the published main-condition NaviAgent screenshot-only protocol. All required intermediate/final subcriteria must pass for task success; partial-credit Score is not this metric. The Windows11 VM resets before each task;1920×1080 observations,no accessibility/SoM,50 steps,temperature1.0,top-p0.9,8000 output tokens,and3s post-action wait. Main-condition task workflow hints are retained where provided. Gold states are checked using domain-specific OCR,pixel/SSIM or file getters after termination. Real input data and distinct artifacts/target states support the20 tasks. Original per-domain seven-model SR panel is preserved. JADE is excluded because the paper reports a missing PDF printer; code/overall category aggregates are excluded because coverage/aggregation cannot be bound unambiguously. No failed infrastructure run is inserted as zero.","zh":"七个独立底模或规模运行Materials Studio完整20题，使用原论文主条件的NaviAgent纯截图框架。成功要求所有中间及最终子标准满足，部分得分不混为成功率。每题重置Windows11虚拟机，1920×1080截图，不用无障碍树或SoM；50步、温度1.0、top-p0.9、8000输出token，动作后等待3秒，保留主条件提供的工作流提示。结束后用领域OCR、像素、SSIM或文件状态对照金标。20题使用真实输入、不同目标状态及产物。保留原域七模型完整SR表。JADE因论文披露缺PDF打印机排除，代码和总体汇总因覆盖或聚合口径未闭合排除，不把基础设施失败补零。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.37053v1","published_at":"2026-09-29","status":"paper_baselines"},"model_count":7,"result_count":7,"results":[[6291,0.15,370,15229,15215],[2998,0.1,370,15230,15215],[3572,0.2,370,15231,15215],[3184,0.3,370,15232,15215],[6297,0.0,370,15233,15215],[6343,0.0,370,15234,15215],[6342,0.0,370,15235,15215]]},{"id":"mattoolbench-vesta-success-2026-paper","name":"MatToolBench · VESTA (2026 paper)","group":"Agent 与工具","metric":"All-subcriteria task success","unit":"%","public_history":true,"protocol":{"en":"Seven independent model sizes/bases run the complete20-task VESTA domain under the published main-condition NaviAgent screenshot-only protocol. All required intermediate/final subcriteria must pass for task success; partial-credit Score is not this metric. The Windows11 VM resets before each task;1920×1080 observations,no accessibility/SoM,50 steps,temperature1.0,top-p0.9,8000 output tokens,and3s post-action wait. Main-condition task workflow hints are retained where provided. Gold states are checked using domain-specific OCR,pixel/SSIM or file getters after termination. Real input data and distinct artifacts/target states support the20 tasks. Original per-domain seven-model SR panel is preserved. JADE is excluded because the paper reports a missing PDF printer; code/overall category aggregates are excluded because coverage/aggregation cannot be bound unambiguously. No failed infrastructure run is inserted as zero.","zh":"七个独立底模或规模运行VESTA完整20题，使用原论文主条件的NaviAgent纯截图框架。成功要求所有中间及最终子标准满足，部分得分不混为成功率。每题重置Windows11虚拟机，1920×1080截图，不用无障碍树或SoM；50步、温度1.0、top-p0.9、8000输出token，动作后等待3秒，保留主条件提供的工作流提示。结束后用领域OCR、像素、SSIM或文件状态对照金标。20题使用真实输入、不同目标状态及产物。保留原域七模型完整SR表。JADE因论文披露缺PDF打印机排除，代码和总体汇总因覆盖或聚合口径未闭合排除，不把基础设施失败补零。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.37053v1","published_at":"2026-09-29","status":"paper_baselines"},"model_count":7,"result_count":7,"results":[[6291,0.2,370,15236,15215],[2998,0.25,370,15237,15215],[3572,0.25,370,15238,15215],[3184,0.2,370,15239,15215],[6297,0.1,370,15240,15215],[6343,0.1,370,15241,15215],[6342,0.1,370,15242,15215]]},{"id":"omnivcbench-mcq-2026-paper","name":"OmniVCBench · MCQ (2026 paper)","group":"数学与科学","metric":"Six-option MCQ accuracy","unit":"%","public_history":true,"protocol":{"en":"Sixteen configurations on the same6077 six-option questions from1080 source papers:2371 L1 evidence-conditioned inference,1986 L2 mechanistic explanation,1720 L3 evidence-grounded hypothesis proposal. At least15 underlying released bases; Qwen3-VL8B SFT remains the same base with explicit adaptation. The primary item-macro accuracy comes from original0/1 predictions on all6077 items; Table34 preserves three decimals and source-paper clustered CIs. Candidate input is figure plus question/options,without source title/caption/context; reference evidence is used only for construction/scoring. Three biology PhDs unanimously approved6077 of7923 candidates. Questions use one composite figure/panel crop and may require several subfigures; old2011–2017 source papers and unknown pretraining exposure remain part of the cohort. Distractors are rewritten model errors, screened for science,identity,length and ambiguity. Open-response1–5 judge scores have separate validity handling and are not substituted for MCQ accuracy; four evidence-acquisition pilots and humans are not ranked models.","zh":"16配置回答相同6077道六选一问题，来自1080篇论文；L1证据推断2371、L2机制解释1986、L3证据约束假说1720。至少15独立发布底模，Qwen3VL8B的SFT仍计原8B底模并标明适配。使用全部6077条原始0/1预测的逐题平均准确率，表34保留三位小数和按源论文聚类区间。候选仅见图与问题选项，不见论文标题、图注和上下文；后者用于构建与评分。三位生物博士从7923候选中一致批准6077题。单张复合图或裁图可要求跨子图推理；2011至2017年源论文及未知预训练接触情况保留。干扰项来自模型错误，经科学性、身份、长度和歧义筛选。开放回答1至5裁判分另有有效性处理，不替代MCQ；四个证据获取试点和人工参考不当排名模型。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.37773v1","published_at":"2026-09-29","status":"paper_baselines"},"model_count":15,"result_count":16,"results":[[4048,0.55093,370,15243,15244],[5295,0.50304,370,15245,15244],[7838,0.49366,370,15246,15244],[5297,0.43738999999999995,370,15247,15244],[6342,0.34277,370,15248,15244],[6342,0.33816,370,15249,15244],[6298,0.30476,370,15250,15244],[2654,0.30278,370,15251,15244],[2687,0.28879,370,15252,15244],[2754,0.26625,370,15253,15244],[6389,0.25539,370,15254,15244],[2771,0.22495,370,15255,15244],[6341,0.21688,370,15256,15244],[6246,0.20865999999999998,370,15257,15244],[11887,0.17310999999999999,370,15258,15244],[15259,0.15863,370,15260,15244]]},{"id":"robochrono-current-action-2026-paper","name":"RoboChrono · current action (2026 paper)","group":"视觉与视频","metric":"Four-option accuracy","unit":"%","public_history":true,"protocol":{"en":"Eighteen model configurations share the full 5407 applicable current action questions in the original34713-question benchmark,39 scenarios,1816 execution videos and about25h. Six choice tracks total29306 questions; the5407 temporal-localisation questions remain separate. Zero-shot inference uses only the permitted observed frames or question-supplied images; future footage is hidden in action prediction. View matching is generated only for synchronised multiview recordings. Correct annotated choice receives1,otherwise0; individual task tracks avoid ambiguous cross-task aggregate weights. Runs with response parse failures≥50% are excluded by the source; all18 published complete rows are retained,including valid low scores. The Qwen3-VL8B Thinking row folds to the8B base while retaining the mode. At least six independent bases remain even after folding adaptation/mode variants. Published protocol uses official closed interfaces and released local weights; exact per-model decoding/checkpoint revisions are not all supplied. Chance/control rows and conditional-only localisation scores are excluded.","zh":"18模型配置使用原34713题基准中current action完整适用面板，共5407题；语料含39场景、1816执行视频、约25小时。六个选择题轨合计29306题，5407道时间定位题另列。零样本推理只用获准历史帧或题目图像，动作预测不见未来视频；跨视角题只从同步多视角记录生成。匹配标注选项计1，否则0；逐轨保留，避免跨任务总分权重歧义。来源排除解析失败率≥50%的运行，完整发表18行均保留，包括真实低分。Qwen3VL8B的Thinking行折叠到8B底模，模式保留；即使合并适配及模式仍超过六独立底模。闭源走官方接口、开源用发布权重，论文未逐模型绑定所有解码和版本修订。随机参考及只用可解析秒数子集的定位分不纳入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.36605v1","published_at":"2026-09-29","status":"paper_baselines"},"model_count":17,"result_count":18,"results":[[4977,0.894,370,15261,15262],[15263,0.752,370,15264,15262],[6721,0.754,370,15265,15262],[15266,0.73,370,15267,15262],[5843,0.697,370,15268,15262],[6343,0.709,370,15269,15262],[6238,0.684,370,15270,15262],[15271,0.618,370,15272,15262],[6342,0.653,370,15273,15262],[6342,0.608,370,15274,15262],[6254,0.57,370,15275,15262],[15276,0.574,370,15277,15262],[7555,0.578,370,15278,15262],[5136,0.284,370,15279,15262],[6341,0.509,370,15280,15262],[15281,0.409,370,15282,15262],[15283,0.503,370,15284,15262],[2766,0.433,370,15285,15262]]},{"id":"robochrono-next-action-2026-paper","name":"RoboChrono · next action (2026 paper)","group":"视觉与视频","metric":"Four-option accuracy","unit":"%","public_history":true,"protocol":{"en":"Eighteen model configurations share the full 4880 applicable next action questions in the original34713-question benchmark,39 scenarios,1816 execution videos and about25h. Six choice tracks total29306 questions; the5407 temporal-localisation questions remain separate. Zero-shot inference uses only the permitted observed frames or question-supplied images; future footage is hidden in action prediction. View matching is generated only for synchronised multiview recordings. Correct annotated choice receives1,otherwise0; individual task tracks avoid ambiguous cross-task aggregate weights. Runs with response parse failures≥50% are excluded by the source; all18 published complete rows are retained,including valid low scores. The Qwen3-VL8B Thinking row folds to the8B base while retaining the mode. At least six independent bases remain even after folding adaptation/mode variants. Published protocol uses official closed interfaces and released local weights; exact per-model decoding/checkpoint revisions are not all supplied. Chance/control rows and conditional-only localisation scores are excluded.","zh":"18模型配置使用原34713题基准中next action完整适用面板，共4880题；语料含39场景、1816执行视频、约25小时。六个选择题轨合计29306题，5407道时间定位题另列。零样本推理只用获准历史帧或题目图像，动作预测不见未来视频；跨视角题只从同步多视角记录生成。匹配标注选项计1，否则0；逐轨保留，避免跨任务总分权重歧义。来源排除解析失败率≥50%的运行，完整发表18行均保留，包括真实低分。Qwen3VL8B的Thinking行折叠到8B底模，模式保留；即使合并适配及模式仍超过六独立底模。闭源走官方接口、开源用发布权重，论文未逐模型绑定所有解码和版本修订。随机参考及只用可解析秒数子集的定位分不纳入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.36605v1","published_at":"2026-09-29","status":"paper_baselines"},"model_count":17,"result_count":18,"results":[[4977,0.764,370,15286,15262],[15263,0.511,370,15287,15262],[6721,0.538,370,15288,15262],[15266,0.55,370,15289,15262],[5843,0.471,370,15290,15262],[6343,0.509,370,15291,15262],[6238,0.426,370,15292,15262],[15271,0.39,370,15293,15262],[6342,0.459,370,15294,15262],[6342,0.453,370,15295,15262],[6254,0.428,370,15296,15262],[15276,0.377,370,15297,15262],[7555,0.435,370,15298,15262],[5136,0.308,370,15299,15262],[6341,0.323,370,15300,15262],[15281,0.349,370,15301,15262],[15283,0.401,370,15302,15262],[2766,0.34,370,15303,15262]]},{"id":"robochrono-goal-conditioned-next-action-2026-paper","name":"RoboChrono · goal-conditioned next action (2026 paper)","group":"视觉与视频","metric":"Four-option accuracy","unit":"%","public_history":true,"protocol":{"en":"Eighteen model configurations share the full 4880 applicable goal-conditioned next action questions in the original34713-question benchmark,39 scenarios,1816 execution videos and about25h. Six choice tracks total29306 questions; the5407 temporal-localisation questions remain separate. Zero-shot inference uses only the permitted observed frames or question-supplied images; future footage is hidden in action prediction. View matching is generated only for synchronised multiview recordings. Correct annotated choice receives1,otherwise0; individual task tracks avoid ambiguous cross-task aggregate weights. Runs with response parse failures≥50% are excluded by the source; all18 published complete rows are retained,including valid low scores. The Qwen3-VL8B Thinking row folds to the8B base while retaining the mode. At least six independent bases remain even after folding adaptation/mode variants. Published protocol uses official closed interfaces and released local weights; exact per-model decoding/checkpoint revisions are not all supplied. Chance/control rows and conditional-only localisation scores are excluded.","zh":"18模型配置使用原34713题基准中goal-conditioned next action完整适用面板，共4880题；语料含39场景、1816执行视频、约25小时。六个选择题轨合计29306题，5407道时间定位题另列。零样本推理只用获准历史帧或题目图像，动作预测不见未来视频；跨视角题只从同步多视角记录生成。匹配标注选项计1，否则0；逐轨保留，避免跨任务总分权重歧义。来源排除解析失败率≥50%的运行，完整发表18行均保留，包括真实低分。Qwen3VL8B的Thinking行折叠到8B底模，模式保留；即使合并适配及模式仍超过六独立底模。闭源走官方接口、开源用发布权重，论文未逐模型绑定所有解码和版本修订。随机参考及只用可解析秒数子集的定位分不纳入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.36605v1","published_at":"2026-09-29","status":"paper_baselines"},"model_count":17,"result_count":18,"results":[[4977,0.836,370,15304,15262],[15263,0.588,370,15305,15262],[6721,0.631,370,15306,15262],[15266,0.599,370,15307,15262],[5843,0.561,370,15308,15262],[6343,0.566,370,15309,15262],[6238,0.495,370,15310,15262],[15271,0.464,370,15311,15262],[6342,0.501,370,15312,15262],[6342,0.508,370,15313,15262],[6254,0.464,370,15314,15262],[15276,0.438,370,15315,15262],[7555,0.458,370,15316,15262],[5136,0.347,370,15317,15262],[6341,0.335,370,15318,15262],[15281,0.415,370,15319,15262],[15283,0.421,370,15320,15262],[2766,0.423,370,15321,15262]]},{"id":"robochrono-frame-matching-2026-paper","name":"RoboChrono · frame matching (2026 paper)","group":"视觉与视频","metric":"Four-option accuracy","unit":"%","public_history":true,"protocol":{"en":"Eighteen model configurations share the full 5325 applicable frame matching questions in the original34713-question benchmark,39 scenarios,1816 execution videos and about25h. Six choice tracks total29306 questions; the5407 temporal-localisation questions remain separate. Zero-shot inference uses only the permitted observed frames or question-supplied images; future footage is hidden in action prediction. View matching is generated only for synchronised multiview recordings. Correct annotated choice receives1,otherwise0; individual task tracks avoid ambiguous cross-task aggregate weights. Runs with response parse failures≥50% are excluded by the source; all18 published complete rows are retained,including valid low scores. The Qwen3-VL8B Thinking row folds to the8B base while retaining the mode. At least six independent bases remain even after folding adaptation/mode variants. Published protocol uses official closed interfaces and released local weights; exact per-model decoding/checkpoint revisions are not all supplied. Chance/control rows and conditional-only localisation scores are excluded.","zh":"18模型配置使用原34713题基准中frame matching完整适用面板，共5325题；语料含39场景、1816执行视频、约25小时。六个选择题轨合计29306题，5407道时间定位题另列。零样本推理只用获准历史帧或题目图像，动作预测不见未来视频；跨视角题只从同步多视角记录生成。匹配标注选项计1，否则0；逐轨保留，避免跨任务总分权重歧义。来源排除解析失败率≥50%的运行，完整发表18行均保留，包括真实低分。Qwen3VL8B的Thinking行折叠到8B底模，模式保留；即使合并适配及模式仍超过六独立底模。闭源走官方接口、开源用发布权重，论文未逐模型绑定所有解码和版本修订。随机参考及只用可解析秒数子集的定位分不纳入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.36605v1","published_at":"2026-09-29","status":"paper_baselines"},"model_count":17,"result_count":18,"results":[[4977,0.983,370,15322,15262],[15263,0.954,370,15323,15262],[6721,0.921,370,15324,15262],[15266,0.849,370,15325,15262],[5843,0.797,370,15326,15262],[6343,0.72,370,15327,15262],[6238,0.853,370,15328,15262],[15271,0.848,370,15329,15262],[6342,0.69,370,15330,15262],[6342,0.715,370,15331,15262],[6254,0.778,370,15332,15262],[15276,0.656,370,15333,15262],[7555,0.465,370,15334,15262],[5136,0.245,370,15335,15262],[6341,0.396,370,15336,15262],[15281,0.398,370,15337,15262],[15283,0.237,370,15338,15262],[2766,0.26,370,15339,15262]]},{"id":"robochrono-view-matching-2026-paper","name":"RoboChrono · view matching (2026 paper)","group":"视觉与视频","metric":"Four-option accuracy","unit":"%","public_history":true,"protocol":{"en":"Eighteen model configurations share the full 4835 applicable view matching questions in the original34713-question benchmark,39 scenarios,1816 execution videos and about25h. Six choice tracks total29306 questions; the5407 temporal-localisation questions remain separate. Zero-shot inference uses only the permitted observed frames or question-supplied images; future footage is hidden in action prediction. View matching is generated only for synchronised multiview recordings. Correct annotated choice receives1,otherwise0; individual task tracks avoid ambiguous cross-task aggregate weights. Runs with response parse failures≥50% are excluded by the source; all18 published complete rows are retained,including valid low scores. The Qwen3-VL8B Thinking row folds to the8B base while retaining the mode. At least six independent bases remain even after folding adaptation/mode variants. Published protocol uses official closed interfaces and released local weights; exact per-model decoding/checkpoint revisions are not all supplied. Chance/control rows and conditional-only localisation scores are excluded.","zh":"18模型配置使用原34713题基准中view matching完整适用面板，共4835题；语料含39场景、1816执行视频、约25小时。六个选择题轨合计29306题，5407道时间定位题另列。零样本推理只用获准历史帧或题目图像，动作预测不见未来视频；跨视角题只从同步多视角记录生成。匹配标注选项计1，否则0；逐轨保留，避免跨任务总分权重歧义。来源排除解析失败率≥50%的运行，完整发表18行均保留，包括真实低分。Qwen3VL8B的Thinking行折叠到8B底模，模式保留；即使合并适配及模式仍超过六独立底模。闭源走官方接口、开源用发布权重，论文未逐模型绑定所有解码和版本修订。随机参考及只用可解析秒数子集的定位分不纳入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.36605v1","published_at":"2026-09-29","status":"paper_baselines"},"model_count":17,"result_count":18,"results":[[4977,0.675,370,15340,15262],[15263,0.536,370,15341,15262],[6721,0.451,370,15342,15262],[15266,0.444,370,15343,15262],[5843,0.392,370,15344,15262],[6343,0.307,370,15345,15262],[6238,0.313,370,15346,15262],[15271,0.413,370,15347,15262],[6342,0.324,370,15348,15262],[6342,0.315,370,15349,15262],[6254,0.283,370,15350,15262],[15276,0.344,370,15351,15262],[7555,0.304,370,15352,15262],[5136,0.608,370,15353,15262],[6341,0.289,370,15354,15262],[15281,0.266,370,15355,15262],[15283,0.222,370,15356,15262],[2766,0.258,370,15357,15262]]},{"id":"robochrono-frame-ordering-2026-paper","name":"RoboChrono · frame ordering (2026 paper)","group":"视觉与视频","metric":"Four-option accuracy","unit":"%","public_history":true,"protocol":{"en":"Eighteen model configurations share the full 3979 applicable frame ordering questions in the original34713-question benchmark,39 scenarios,1816 execution videos and about25h. Six choice tracks total29306 questions; the5407 temporal-localisation questions remain separate. Zero-shot inference uses only the permitted observed frames or question-supplied images; future footage is hidden in action prediction. View matching is generated only for synchronised multiview recordings. Correct annotated choice receives1,otherwise0; individual task tracks avoid ambiguous cross-task aggregate weights. Runs with response parse failures≥50% are excluded by the source; all18 published complete rows are retained,including valid low scores. The Qwen3-VL8B Thinking row folds to the8B base while retaining the mode. At least six independent bases remain even after folding adaptation/mode variants. Published protocol uses official closed interfaces and released local weights; exact per-model decoding/checkpoint revisions are not all supplied. Chance/control rows and conditional-only localisation scores are excluded.","zh":"18模型配置使用原34713题基准中frame ordering完整适用面板，共3979题；语料含39场景、1816执行视频、约25小时。六个选择题轨合计29306题，5407道时间定位题另列。零样本推理只用获准历史帧或题目图像，动作预测不见未来视频；跨视角题只从同步多视角记录生成。匹配标注选项计1，否则0；逐轨保留，避免跨任务总分权重歧义。来源排除解析失败率≥50%的运行，完整发表18行均保留，包括真实低分。Qwen3VL8B的Thinking行折叠到8B底模，模式保留；即使合并适配及模式仍超过六独立底模。闭源走官方接口、开源用发布权重，论文未逐模型绑定所有解码和版本修订。随机参考及只用可解析秒数子集的定位分不纳入。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.36605v1","published_at":"2026-09-29","status":"paper_baselines"},"model_count":17,"result_count":18,"results":[[4977,0.683,370,15358,15262],[15263,0.329,370,15359,15262],[6721,0.359,370,15360,15262],[15266,0.328,370,15361,15262],[5843,0.304,370,15362,15262],[6343,0.329,370,15363,15262],[6238,0.29,370,15364,15262],[15271,0.264,370,15365,15262],[6342,0.284,370,15366,15262],[6342,0.277,370,15367,15262],[6254,0.294,370,15368,15262],[15276,0.254,370,15369,15262],[7555,0.271,370,15370,15262],[5136,0.496,370,15371,15262],[6341,0.27,370,15372,15262],[15281,0.286,370,15373,15262],[15283,0.204,370,15374,15262],[2766,0.262,370,15375,15262]]},{"id":"omni-storybench-complete-narration-bertscore-2026-paper","name":"Omni-StoryBench · narration fidelity (2026 paper)","group":"生成与创作","metric":"BERTScore F1","unit":"score","public_history":true,"protocol":{"en":"Narration fidelity track only. Eight underlying backbones with complete900/900 narration outputs yield21 original pipeline configurations. Each input is a current storybook page,narration and structured next-page conditions; one human-reviewed transition per source–book pair,900 in total. Generation must continue the next page; source annotations were model-assisted and revised by16 human reviewers. BERTScorer uses English,rescale_with_baseline=False,and reports F1 averaged over900 references on native0–1; this is text similarity,not accuracy or the multimodal total. FLUX.1-dev/FLUX.1-Kontext/Nitro-T1.2B image experts and Parler-large/VoxCPM2 TTS experts remain explicit configurations,not additional narration bases. AnyGPT,Omni-Diffusion and Dynin-Omni use native output paths; only Dynin generates all three modalities simultaneously. Full original32-row metric and coverage tables are retained. The5 incomplete narration backbones and Ground truth reference are excluded by coverage,not padded; judge/total scores with per-system/per-judge validity masks are not mixed into this fixed900 panel. The BERTScorer default checkpoint/package revision is not separately pinned in the paper.","zh":"此轨只评旁白忠实度。八底模均产生900条完整旁白，保留21个原管线配置。输入为当前绘本页、旁白与结构化下一页条件；每个来源和书籍组合保留一个经人工复核的转场，共900。模型生成下一页内容，原注释由模型辅助、16名人工复核修订。BERTScorer设英文、rescale_with_baseline=False，报告900参考上的F1均值，保留原生0至1文本相似度，不当准确率或多模态总分。图像专家FLUX.1dev、FLUX.1Kontext、NitroT1.2B和语音专家Parlerlarge、VoxCPM2逐行写入配置，不额外计旁白底模。AnyGPT、OmniDiffusion和Dynin使用原生路径，只有Dynin三模态单次同时生成。保留原始32行指标表和覆盖表；五个旁白不完整底模及Ground truth参考按覆盖排除，不补零；各系统各裁判有效样本不同的裁判总分不混入固定900题面板。论文未另行固定BERTScorer默认检查点和包修订。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2609.37317v1","published_at":"2026-09-29","status":"paper_baselines"},"score_max":1,"score_decimals":3,"saturation_excluded":true,"scale_label":{"en":"Native score / 1","zh":"原生分数 / 1"},"model_count":8,"result_count":21,"results":[[3184,0.882,370,15376,15377],[3184,0.882,370,15378,15377],[3184,0.882,370,15379,15377],[3184,0.882,370,15380,15377],[5722,0.882,370,15381,15377],[5722,0.882,370,15382,15377],[5722,0.882,370,15383,15377],[5722,0.882,370,15384,15377],[3181,0.882,370,15385,15377],[3181,0.882,370,15386,15377],[3181,0.882,370,15387,15377],[3181,0.882,370,15388,15377],[2687,0.872,370,15389,15377],[2687,0.872,370,15390,15377],[2687,0.872,370,15391,15377],[4969,0.879,370,15392,15377],[4969,0.879,370,15393,15377],[4969,0.879,370,15394,15377],[15395,0.802,370,15396,15377],[12064,0.872,370,15397,15377],[15398,0.908,370,15399,15377]]},{"id":"llm-generated-world-models-2026-paper","name":"LLM-Generated World Models (2026 paper)","group":"知识与推理","metric":"Mean F1","unit":"score","public_history":true,"protocol":{"en":"Primary-source results extracted from LLM-Generated World Models (arXiv:2611.14419, table S6.T2, published 2026-11-10). Retains 5 published evaluation configurations across 5 distinct models on 'Mean F1' (native score scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《LLM-Generated World Models》原始论文（arXiv:2611.14419，表 S6.T2，发布日期 2026-11-10）。完整保留“Mean F1”列下已发表的 5 组评测配置成绩（覆盖 5 个独立模型，采用原生分值口径（不计入准确率饱和度分档））；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.14419v1","published_at":"2026-11-10","status":"paper_baselines"},"model_count":5,"result_count":5,"score_min":0,"score_max":1,"score_decimals":3,"results":[[755,0.1709,370,15400,15401],[3261,0.2696,370,15402,15401],[762,0.4981,370,15403,15401],[796,0.4719,370,15404,15401],[3403,0.4087,370,15405,15401]]},{"id":"are-bench-2026-paper","name":"ARE-bench (2026 paper)","group":"Agent 与工具","metric":"Overall ISR","unit":"%","public_history":true,"protocol":{"en":"Primary-source results extracted from ARE-bench (arXiv:2611.14364, table S4.T5, published 2026-11-10). Retains 7 published evaluation configurations across 7 distinct models on 'Overall ISR' (comparable percentage scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《ARE-bench》原始论文（arXiv:2611.14364，表 S4.T5，发布日期 2026-11-10）。完整保留“Overall ISR”列下已发表的 7 组评测配置成绩（覆盖 7 个独立模型，采用可比百分比口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.14364v1","published_at":"2026-11-10","status":"paper_baselines"},"model_count":7,"result_count":7,"results":[[5618,0.6622,370,15406,15407],[2959,0.5689,370,15408,15407],[3153,0.3956,370,15409,15407],[3157,0.3778,370,15410,15407],[3396,0.3067,370,15411,15407],[3155,0.28,370,15412,15407],[5300,0.2578,370,15413,15407]]},{"id":"mmdm-2026-paper","name":"MMDM (2026 paper)","group":"知识与推理","metric":"Overall ALL","unit":"%","public_history":true,"protocol":{"en":"Primary-source results extracted from MMDM (arXiv:2611.14309, table S4.T3, published 2026-11-10). Retains 28 published evaluation configurations across 28 distinct models on 'Overall ALL' (comparable percentage scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《MMDM》原始论文（arXiv:2611.14309，表 S4.T3，发布日期 2026-11-10）。完整保留“Overall ALL”列下已发表的 28 组评测配置成绩（覆盖 28 个独立模型，采用可比百分比口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.14309v1","published_at":"2026-11-10","status":"paper_baselines"},"model_count":28,"result_count":28,"results":[[15414,0.672,370,15415,15416],[15417,0.614,370,15418,15416],[15419,0.733,370,15420,15416],[15421,0.623,370,15422,15416],[15423,0.566,370,15424,15416],[15425,0.76,370,15426,15416],[15427,0.751,370,15428,15416],[15429,0.744,370,15430,15416],[15431,0.671,370,15432,15416],[15433,0.646,370,15434,15416],[5546,0.642,370,15435,15416],[15436,0.624,370,15437,15416],[15438,0.621,370,15439,15416],[15440,0.583,370,15441,15416],[15442,0.577,370,15443,15416],[6178,0.574,370,15444,15416],[15445,0.572,370,15446,15416],[5549,0.569,370,15447,15416],[15448,0.565,370,15449,15416],[5548,0.557,370,15450,15416],[15451,0.533,370,15452,15416],[15453,0.49,370,15454,15416],[15455,0.487,370,15456,15416],[15457,0.446,370,15458,15416],[15459,0.424,370,15460,15416],[15461,0.388,370,15462,15416],[15463,0.322,370,15464,15416],[15465,0.267,370,15466,15416]]},{"id":"hx2dx-2026-paper","name":"Hx2Dx (2026 paper)","group":"医疗","metric":"Correct","unit":"raw","public_history":true,"protocol":{"en":"Primary-source results extracted from Hx2Dx (arXiv:2611.14122, table S9.T2, published 2026-11-10). Retains 7 published evaluation configurations across 7 distinct models on 'Correct' (raw reported metric scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《Hx2Dx》原始论文（arXiv:2611.14122，表 S9.T2，发布日期 2026-11-10）。完整保留“Correct”列下已发表的 7 组评测配置成绩（覆盖 7 个独立模型，采用原始指标口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.14122v1","published_at":"2026-11-10","status":"paper_baselines"},"model_count":7,"result_count":7,"score_decimals":2,"results":[[3529,366.0,370,15467,15468],[3512,327.0,370,15469,15468],[15470,276.0,370,15471,15468],[3414,192.0,370,15472,15468],[11955,62.0,370,15473,15468],[4937,63.0,370,15474,15468],[363,77.0,370,15475,15468]]},{"id":"spy-bench-2026-paper","name":"SPY-Bench (2026 paper)","group":"安全与可靠性","metric":"w/ scenario · HMS","unit":"score","public_history":true,"protocol":{"en":"Primary-source results extracted from SPY-Bench (arXiv:2611.13997, table S3.T4, published 2026-11-10). Retains 21 published evaluation configurations across 21 distinct models on 'w/ scenario · HMS' (native score scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《SPY-Bench》原始论文（arXiv:2611.13997，表 S3.T4，发布日期 2026-11-10）。完整保留“w/ scenario · HMS”列下已发表的 21 组评测配置成绩（覆盖 21 个独立模型，采用原生分值口径（不计入准确率饱和度分档））；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.13997v1","published_at":"2026-11-10","status":"paper_baselines"},"model_count":21,"result_count":21,"score_min":0,"score_max":100,"score_decimals":2,"results":[[15476,15.25,370,15477,15478],[15479,25.92,370,15480,15478],[15481,11.93,370,15482,15478],[15483,26.56,370,15484,15478],[2827,33.11,370,15485,15478],[2848,46.1,370,15486,15478],[15487,13.35,370,15488,15478],[15489,27.87,370,15490,15478],[15491,28.48,370,15492,15478],[15493,41.59,370,15494,15478],[15495,36.64,370,15496,15478],[15497,12.33,370,15498,15478],[15499,16.64,370,15500,15478],[15501,21.73,370,15502,15478],[15503,36.64,370,15504,15478],[15505,35.78,370,15506,15478],[15507,21.46,370,15508,15478],[15509,39.86,370,15510,15478],[15511,48.31,370,15512,15478],[67,22.96,370,15513,15478],[15514,42.25,370,15515,15478]]},{"id":"bioxarena-pro-2026-paper","name":"BioXArena-Pro (2026 paper)","group":"Agent 与工具","metric":"Mean","unit":"%","public_history":true,"protocol":{"en":"Primary-source results extracted from BioXArena-Pro (arXiv:2611.13994, table S3.T4, published 2026-11-10). Retains 22 published evaluation configurations across 7 distinct models on 'Mean' (comparable percentage scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《BioXArena-Pro》原始论文（arXiv:2611.13994，表 S3.T4，发布日期 2026-11-10）。完整保留“Mean”列下已发表的 22 组评测配置成绩（覆盖 7 个独立模型，采用可比百分比口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.13994v1","published_at":"2026-11-10","status":"paper_baselines"},"model_count":7,"result_count":22,"results":[[2958,0.551,370,15516,15517],[2959,0.532,370,15518,15517],[2958,0.532,370,15519,15517],[2959,0.532,370,15520,15517],[755,0.529,370,15521,15517],[2959,0.527,370,15522,15517],[2959,0.496,370,15523,15517],[2958,0.484,370,15524,15517],[5284,0.478,370,15525,15517],[2983,0.478,370,15526,15517],[2983,0.472,370,15527,15517],[2958,0.467,370,15528,15517],[2983,0.467,370,15529,15517],[2959,0.465,370,15530,15517],[2958,0.465,370,15531,15517],[762,0.452,370,15532,15517],[2983,0.444,370,15533,15517],[14452,0.414,370,15534,15517],[5284,0.406,370,15535,15517],[5284,0.389,370,15536,15517],[2983,0.374,370,15537,15517],[5284,0.336,370,15538,15517]]},{"id":"autonomous-rtl-and-synthesis-optimization-2026-paper","name":"Autonomous RTL and Synthesis Optimization (2026 paper)","group":"Agent 与工具","metric":"(a) Reference-based: RTL-OPT","unit":"raw","public_history":true,"protocol":{"en":"Primary-source results extracted from Autonomous RTL and Synthesis Optimization (arXiv:2611.13713, table S2.T1, published 2026-11-10). Retains 9 published evaluation configurations across 9 distinct models on '(a) Reference-based: RTL-OPT' (raw reported metric scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《Autonomous RTL and Synthesis Optimization》原始论文（arXiv:2611.13713，表 S2.T1，发布日期 2026-11-10）。完整保留“(a) Reference-based: RTL-OPT”列下已发表的 9 组评测配置成绩（覆盖 9 个独立模型，采用原始指标口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.13713v1","published_at":"2026-11-10","status":"paper_baselines"},"model_count":9,"result_count":9,"score_decimals":2,"results":[[354,5.6,370,15539,15540],[6957,5.6,370,15541,15540],[9073,13.9,370,15542,15540],[1487,13.9,370,15543,15540],[15544,-0.7,370,15545,15540],[12229,0.6,370,15546,15540],[15547,-3.1,370,15548,15540],[15549,-1.4,370,15550,15540],[15551,-5.8,370,15552,15540]]},{"id":"llm-next-step-recommendations-2026-paper","name":"LLM Next-Step Recommendations (2026 paper)","group":"知识与推理","metric":"P3","unit":"%","public_history":true,"protocol":{"en":"Primary-source results extracted from LLM Next-Step Recommendations (arXiv:2611.13667, table S4.T4, published 2026-11-10). Retains 6 published evaluation configurations across 6 distinct models on 'P3' (comparable percentage scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《LLM Next-Step Recommendations》原始论文（arXiv:2611.13667，表 S4.T4，发布日期 2026-11-10）。完整保留“P3”列下已发表的 6 组评测配置成绩（覆盖 6 个独立模型，采用可比百分比口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.13667v1","published_at":"2026-11-10","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[3453,1.0,370,15553,15554],[3568,0.96,370,15555,15554],[762,1.0,370,15556,15554],[2973,0.93,370,15557,15554],[3482,1.0,370,15558,15554],[3386,1.0,370,15559,15554]]},{"id":"phyrubric-bench-2026-paper","name":"PhyRubric-Bench (2026 paper)","group":"数学与科学","metric":"PC · ρ","unit":"score","public_history":true,"protocol":{"en":"Primary-source results extracted from PhyRubric-Bench (arXiv:2611.13576, table A3.T10, published 2026-11-10). Retains 15 published evaluation configurations across 9 distinct models on 'PC · ρ' (native score scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《PhyRubric-Bench》原始论文（arXiv:2611.13576，表 A3.T10，发布日期 2026-11-10）。完整保留“PC · ρ”列下已发表的 15 组评测配置成绩（覆盖 9 个独立模型，采用原生分值口径（不计入准确率饱和度分档））；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.13576v1","published_at":"2026-11-10","status":"paper_baselines"},"model_count":9,"result_count":15,"score_min":0,"score_max":1,"score_decimals":3,"results":[[6342,0.021,370,15560,15561],[6342,0.043,370,15562,15561],[6343,0.146,370,15563,15561],[6343,0.143,370,15564,15561],[6297,0.123,370,15565,15561],[6297,0.13,370,15566,15561],[15567,0.153,370,15568,15561],[15569,0.105,370,15570,15561],[15569,0.112,370,15571,15561],[15572,0.195,370,15573,15561],[15574,0.202,370,15575,15561],[15574,0.237,370,15576,15561],[2962,0.264,370,15577,15561],[2962,0.333,370,15578,15561],[15579,0.317,370,15580,15561]]},{"id":"med-evidence-bench-2026-paper","name":"Med Evidence Bench (2026 paper)","group":"医疗","metric":"Length-adjusted ETS","unit":"score","public_history":true,"protocol":{"en":"Primary-source results extracted from Med Evidence Bench (arXiv:2611.13562, table Sx8.p11.1, published 2026-11-10). Retains 30 published evaluation configurations across 9 distinct models on 'Length-adjusted ETS' (native score scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《Med Evidence Bench》原始论文（arXiv:2611.13562，表 Sx8.p11.1，发布日期 2026-11-10）。完整保留“Length-adjusted ETS”列下已发表的 30 组评测配置成绩（覆盖 9 个独立模型，采用原生分值口径（不计入准确率饱和度分档））；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.13562v1","published_at":"2026-11-10","status":"paper_baselines"},"model_count":9,"result_count":30,"score_min":0,"score_max":1,"score_decimals":3,"results":[[15581,0.5308,370,15582,15583],[3185,0.2948,370,15584,15583],[3404,0.3002,370,15585,15583],[3204,0.3177,370,15586,15583],[3154,0.3407,370,15587,15583],[4048,0.3713,370,15588,15583],[3155,0.4105,370,15589,15583],[15590,0.3853,370,15591,15583],[3157,0.3191,370,15592,15583],[15593,0.2421,370,15594,15583],[15581,0.6579,370,15595,15583],[3185,0.376,370,15596,15583],[3404,0.3461,370,15597,15583],[3204,0.4074,370,15598,15583],[3154,0.3382,370,15599,15583],[4048,0.3978,370,15600,15583],[3155,0.456,370,15601,15583],[15590,0.3515,370,15602,15583],[3157,0.4257,370,15603,15583],[15593,0.2881,370,15604,15583],[15581,0.3855,370,15605,15583],[3185,0.2019,370,15606,15583],[3404,0.2477,370,15607,15583],[3204,0.215,370,15608,15583],[3154,0.3435,370,15609,15583],[4048,0.341,370,15610,15583],[3155,0.3584,370,15611,15583],[15590,0.424,370,15612,15583],[3157,0.1972,370,15613,15583],[15593,0.1894,370,15614,15583]]},{"id":"deputybench-2026-paper","name":"DeputyBench (2026 paper)","group":"安全与可靠性","metric":"PSR","unit":"%","public_history":true,"protocol":{"en":"Primary-source results extracted from DeputyBench (arXiv:2611.13355, table S3.T1, published 2026-11-09). Retains 12 published evaluation configurations across 12 distinct models on 'PSR' (comparable percentage scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《DeputyBench》原始论文（arXiv:2611.13355，表 S3.T1，发布日期 2026-11-09）。完整保留“PSR”列下已发表的 12 组评测配置成绩（覆盖 12 个独立模型，采用可比百分比口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.13355v1","published_at":"2026-11-09","status":"paper_baselines"},"model_count":12,"result_count":12,"results":[[5508,0.00833,370,15615,15616],[5506,0.00875,370,15617,15616],[15618,0.00926,370,15619,15616],[2959,0.00893,370,15620,15616],[2961,0.00901,370,15621,15616],[2968,0.0076,370,15622,15616],[2983,0.00806,370,15623,15616],[6234,0.00646,370,15624,15616],[4881,0.00793,370,15625,15616],[6452,0.00788,370,15626,15616],[15627,0.0069,370,15628,15616],[5417,0.00563,370,15629,15616]]},{"id":"agentscript-eval-2026-paper","name":"AgentScript-Eval (2026 paper)","group":"Agent 与工具","metric":"Pass rate","unit":"%","public_history":true,"protocol":{"en":"Primary-source results extracted from AgentScript-Eval (arXiv:2611.13183, table S3.T1, published 2026-11-09). Retains 10 published evaluation configurations across 5 distinct models on 'Pass rate' (comparable percentage scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《AgentScript-Eval》原始论文（arXiv:2611.13183，表 S3.T1，发布日期 2026-11-09）。完整保留“Pass rate”列下已发表的 10 组评测配置成绩（覆盖 5 个独立模型，采用可比百分比口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.13183v1","published_at":"2026-11-09","status":"paper_baselines"},"model_count":5,"result_count":10,"results":[[3479,0.313,370,15630,15631],[3196,0.291,370,15632,15631],[15633,0.281,370,15634,15631],[3756,0.209,370,15635,15631],[3568,0.174,370,15636,15631],[3401,0.465,370,15637,15631],[3401,0.515,370,15638,15631],[3401,0.436,370,15639,15631],[3401,0.611,370,15640,15631],[3401,0.453,370,15641,15631]]},{"id":"vigor-2026-paper","name":"VIGOR (2026 paper)","group":"视觉与视频","metric":"Pointing","unit":"%","public_history":true,"protocol":{"en":"Primary-source results extracted from VIGOR (arXiv:2611.13150, table S4.T2, published 2026-11-09). Retains 26 published evaluation configurations across 26 distinct models on 'Pointing' (comparable percentage scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《VIGOR》原始论文（arXiv:2611.13150，表 S4.T2，发布日期 2026-11-09）。完整保留“Pointing”列下已发表的 26 组评测配置成绩（覆盖 26 个独立模型，采用可比百分比口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.13150v1","published_at":"2026-11-09","status":"paper_baselines"},"model_count":26,"result_count":26,"results":[[2780,0.277,370,15642,15643],[6344,0.137,370,15644,15643],[2766,0.277,370,15645,15643],[6246,0.365,370,15646,15643],[2800,0.238,370,15647,15643],[6341,0.319,370,15648,15643],[15649,0.315,370,15650,15643],[2771,0.346,370,15651,15643],[6866,0.342,370,15652,15643],[2687,0.372,370,15653,15643],[6298,0.345,370,15654,15643],[2485,0.271,370,15655,15643],[2754,0.356,370,15656,15643],[2728,0.352,370,15657,15643],[2654,0.382,370,15658,15643],[15659,0.32,370,15660,15643],[6342,0.322,370,15661,15643],[6865,0.352,370,15662,15643],[2697,0.367,370,15663,15643],[6345,0.38,370,15664,15643],[8277,0.318,370,15665,15643],[6864,0.323,370,15666,15643],[6391,0.358,370,15667,15643],[6343,0.356,370,15668,15643],[2706,0.357,370,15669,15643],[2642,0.387,370,15670,15643]]},{"id":"gtagym-2026-paper","name":"GTAGym (2026 paper)","group":"视觉与视频","metric":"Overall","unit":"%","public_history":true,"protocol":{"en":"Primary-source results extracted from GTAGym (arXiv:2611.13028, table S4.T2, published 2026-11-09). Retains 12 published evaluation configurations across 6 distinct models on 'Overall' (comparable percentage scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《GTAGym》原始论文（arXiv:2611.13028，表 S4.T2，发布日期 2026-11-09）。完整保留“Overall”列下已发表的 12 组评测配置成绩（覆盖 6 个独立模型，采用可比百分比口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.13028v1","published_at":"2026-11-09","status":"paper_baselines"},"model_count":6,"result_count":12,"results":[[4977,0.664,370,15671,15672],[5618,0.66,370,15673,15672],[3385,0.338,370,15674,15672],[3396,0.389,370,15675,15672],[3155,0.284,370,15676,15672],[5300,0.242,370,15677,15672],[4977,0.709,370,15678,15672],[5618,0.669,370,15679,15672],[3385,0.382,370,15680,15672],[3396,0.418,370,15681,15672],[3155,0.331,370,15682,15672],[5300,0.342,370,15683,15672]]},{"id":"buysideflow-2026-paper","name":"BuySideFlow (2026 paper)","group":"Agent 与工具","metric":"Overall","unit":"%","public_history":true,"protocol":{"en":"Primary-source results extracted from BuySideFlow (arXiv:2611.12966, table S4.T2, published 2026-11-07). Retains 9 published evaluation configurations across 5 distinct models on 'Overall' (comparable percentage scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《BuySideFlow》原始论文（arXiv:2611.12966，表 S4.T2，发布日期 2026-11-07）。完整保留“Overall”列下已发表的 9 组评测配置成绩（覆盖 5 个独立模型，采用可比百分比口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.12966v1","published_at":"2026-11-07","status":"paper_baselines"},"model_count":5,"result_count":9,"results":[[3196,0.35,370,15684,15685],[3201,0.309,370,15686,15685],[3181,0.299,370,15687,15685],[3186,0.196,370,15688,15685],[3189,0.162,370,15689,15685],[3181,0.24,370,15690,15685],[3196,0.368,370,15691,15685],[3196,0.287,370,15692,15685],[3201,0.223,370,15693,15685]]},{"id":"earthrecord-2026-paper","name":"EarthRecord (2026 paper)","group":"Agent 与工具","metric":"Overall","unit":"%","public_history":true,"protocol":{"en":"Primary-source results extracted from EarthRecord (arXiv:2611.12934, table S2.T2, published 2026-11-06). Retains 14 published evaluation configurations across 14 distinct models on 'Overall' (comparable percentage scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《EarthRecord》原始论文（arXiv:2611.12934，表 S2.T2，发布日期 2026-11-06）。完整保留“Overall”列下已发表的 14 组评测配置成绩（覆盖 14 个独立模型，采用可比百分比口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.12934v1","published_at":"2026-11-06","status":"paper_baselines"},"model_count":14,"result_count":14,"results":[[3384,0.854,370,15694,15695],[3394,0.841,370,15696,15695],[2960,0.814,370,15697,15695],[3382,0.81,370,15698,15695],[2958,0.77,370,15699,15695],[2964,0.735,370,15700,15695],[3424,0.726,370,15701,15695],[5394,0.681,370,15702,15695],[244,0.54,370,15703,15695],[3392,0.412,370,15704,15695],[15705,0.195,370,15706,15695],[2976,0.186,370,15707,15695],[15708,0.239,370,15709,15695],[15710,0.137,370,15711,15695]]},{"id":"workflowarena-2026-paper","name":"WorkflowArena (2026 paper)","group":"Agent 与工具","metric":"Overall","unit":"%","public_history":true,"protocol":{"en":"Primary-source results extracted from WorkflowArena (arXiv:2611.12856, table S4.T1, published 2026-11-09). Retains 25 published evaluation configurations across 9 distinct models on 'Overall' (comparable percentage scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《WorkflowArena》原始论文（arXiv:2611.12856，表 S4.T1，发布日期 2026-11-09）。完整保留“Overall”列下已发表的 25 组评测配置成绩（覆盖 9 个独立模型，采用可比百分比口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.12856v1","published_at":"2026-11-09","status":"paper_baselines"},"model_count":9,"result_count":25,"results":[[2959,0.3695,370,15712,15713],[2959,0.3411,370,15714,15713],[2959,0.3323,370,15715,15713],[2961,0.3321,370,15716,15713],[2958,0.308,370,15717,15713],[2958,0.2982,370,15718,15713],[2960,0.295,370,15719,15713],[2959,0.2914,370,15720,15713],[2958,0.2811,370,15721,15713],[2960,0.2804,370,15722,15713],[2958,0.278,370,15723,15713],[2960,0.2759,370,15724,15713],[2958,0.2734,370,15725,15713],[2960,0.273,370,15726,15713],[2960,0.2563,370,15727,15713],[2961,0.2526,370,15728,15713],[2961,0.2324,370,15729,15713],[2959,0.2202,370,15730,15713],[2961,0.1978,370,15731,15713],[2961,0.1688,370,15732,15713],[2967,0.1648,370,15733,15713],[2976,0.1332,370,15734,15713],[6452,0.1084,370,15735,15713],[3512,0.0608,370,15736,15713],[2980,0.0443,370,15737,15713]]},{"id":"autobuddy-bench-2026-paper","name":"AutoBuddy-Bench (2026 paper)","group":"Agent 与工具","metric":"End to end","unit":"%","public_history":true,"protocol":{"en":"Primary-source results extracted from AutoBuddy-Bench (arXiv:2611.12393, table S5.T1, published 2026-11-09). Retains 17 published evaluation configurations across 17 distinct models on 'End to end' (comparable percentage scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《AutoBuddy-Bench》原始论文（arXiv:2611.12393，表 S5.T1，发布日期 2026-11-09）。完整保留“End to end”列下已发表的 17 组评测配置成绩（覆盖 17 个独立模型，采用可比百分比口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.12393v1","published_at":"2026-11-09","status":"paper_baselines"},"model_count":17,"result_count":17,"results":[[5456,0.698,370,15738,15739],[5142,0.68,370,15740,15739],[4972,0.654,370,15741,15739],[15742,0.635,370,15743,15739],[3208,0.631,370,15744,15739],[3185,0.592,370,15745,15739],[4946,0.541,370,15746,15739],[5108,0.523,370,15747,15739],[5368,0.522,370,15748,15739],[3153,0.488,370,15749,15739],[3187,0.481,370,15750,15739],[5107,0.47,370,15751,15739],[5155,0.434,370,15752,15739],[5150,0.325,370,15753,15739],[5160,0.297,370,15754,15739],[3201,0.272,370,15755,15739],[5111,0.234,370,15756,15739]]},{"id":"triad-2026-paper","name":"TRIAD (2026 paper)","group":"视觉与视频","metric":"State · Cars","unit":"score","public_history":true,"protocol":{"en":"Primary-source results extracted from TRIAD (arXiv:2611.12364, table S4.T2, published 2026-11-09). Retains 21 published evaluation configurations across 21 distinct models on 'State · Cars' (native score scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《TRIAD》原始论文（arXiv:2611.12364，表 S4.T2，发布日期 2026-11-09）。完整保留“State · Cars”列下已发表的 21 组评测配置成绩（覆盖 21 个独立模型，采用原生分值口径（不计入准确率饱和度分档））；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.12364v1","published_at":"2026-11-09","status":"paper_baselines"},"model_count":21,"result_count":21,"score_min":0,"score_max":1,"score_decimals":3,"results":[[15757,0.062,370,15758,15759],[15760,0.069,370,15761,15759],[15762,0.05,370,15763,15759],[15764,0.291,370,15765,15759],[15766,0.101,370,15767,15759],[15768,0.272,370,15769,15759],[15770,0.177,370,15771,15759],[15772,0.085,370,15773,15759],[15774,0.076,370,15775,15759],[15776,0.142,370,15777,15759],[15778,0.092,370,15779,15759],[15780,0.107,370,15781,15759],[15782,0.17,370,15783,15759],[15784,0.194,370,15785,15759],[15786,0.146,370,15787,15759],[15788,0.195,370,15789,15759],[15790,0.13,370,15791,15759],[15792,0.092,370,15793,15759],[15794,0.195,370,15795,15759],[15796,0.259,370,15797,15759],[15798,0.258,370,15799,15759]]},{"id":"geomind-2026-paper","name":"GeoMind (2026 paper)","group":"视觉与视频","metric":"AVG","unit":"%","public_history":true,"protocol":{"en":"Primary-source results extracted from GeoMind (arXiv:2611.12175, table S4.T2, published 2026-11-09). Retains 12 published evaluation configurations across 9 distinct models on 'AVG' (comparable percentage scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《GeoMind》原始论文（arXiv:2611.12175，表 S4.T2，发布日期 2026-11-09）。完整保留“AVG”列下已发表的 12 组评测配置成绩（覆盖 9 个独立模型，采用可比百分比口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.12175v1","published_at":"2026-11-09","status":"paper_baselines"},"model_count":9,"result_count":12,"results":[[7592,0.207,370,15800,15801],[10681,0.113,370,15802,15801],[9732,0.125,370,15803,15801],[15804,0.183,370,15805,15801],[15806,0.264,370,15807,15801],[15808,0.24,370,15809,15801],[353,0.111,370,15810,15801],[766,0.046,370,15811,15801],[6978,0.159,370,15812,15801],[353,0.101,370,15813,15801],[766,0.04,370,15814,15801],[6978,0.169,370,15815,15801]]},{"id":"fraudbench-2026-paper","name":"FraudBench (2026 paper)","group":"专业工作","metric":"ℛ A F","unit":"%","public_history":true,"protocol":{"en":"Primary-source results extracted from FraudBench (arXiv:2611.12167, table S5.T3, published 2026-11-09). Retains 20 published evaluation configurations across 20 distinct models on 'ℛ A F' (comparable percentage scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《FraudBench》原始论文（arXiv:2611.12167，表 S5.T3，发布日期 2026-11-09）。完整保留“ℛ A F”列下已发表的 20 组评测配置成绩（覆盖 20 个独立模型，采用可比百分比口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.12167v1","published_at":"2026-11-09","status":"paper_baselines"},"model_count":20,"result_count":20,"results":[[15816,0.091,370,15817,15818],[15819,0.118,370,15820,15818],[15821,0.131,370,15822,15818],[15823,0.014,370,15824,15818],[15825,0.019,370,15826,15818],[15827,0.013,370,15828,15818],[3708,0.007,370,15829,15818],[2055,0.007,370,15830,15818],[15831,0.006,370,15832,15818],[15833,0.006,370,15834,15818],[15835,0.019,370,15836,15818],[15837,0.013,370,15838,15818],[15839,0.038,370,15840,15818],[15841,0.006,370,15842,15818],[15843,0.006,370,15844,15818],[15845,0.013,370,15846,15818],[15847,0.006,370,15848,15818],[15849,0.026,370,15850,15818],[15851,0.025,370,15852,15818],[6278,0.006,370,15853,15818]]},{"id":"swe-time-2026-paper","name":"SWE-Time (2026 paper)","group":"编程","metric":"SWE-Time Avg.","unit":"%","public_history":true,"protocol":{"en":"Primary-source results extracted from SWE-Time (arXiv:2611.12083, table S3.T2, published 2026-11-09). Retains 15 published evaluation configurations across 15 distinct models on 'SWE-Time Avg.' (comparable percentage scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《SWE-Time》原始论文（arXiv:2611.12083，表 S3.T2，发布日期 2026-11-09）。完整保留“SWE-Time Avg.”列下已发表的 15 组评测配置成绩（覆盖 15 个独立模型，采用可比百分比口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.12083v1","published_at":"2026-11-09","status":"paper_baselines"},"model_count":15,"result_count":15,"results":[[3479,0.619,370,15854,15855],[3572,0.485,370,15856,15855],[3412,0.279,370,15857,15855],[2983,0.545,370,15858,15855],[3405,0.44,370,15859,15855],[3261,0.459,370,15860,15855],[3515,0.597,370,15861,15855],[3654,0.56,370,15862,15855],[3560,0.609,370,15863,15855],[3603,0.309,370,15864,15855],[2977,0.6,370,15865,15855],[3490,0.555,370,15866,15855],[5290,0.243,370,15867,15855],[3581,0.62,370,15868,15855],[5101,0.587,370,15869,15855]]},{"id":"industrial-level-chip-floorplanning-2026-paper","name":"Industrial-Level Chip Floorplanning (2026 paper)","group":"Agent 与工具","metric":"Q mean","unit":"score","public_history":true,"protocol":{"en":"Primary-source results extracted from Industrial-Level Chip Floorplanning (arXiv:2611.12048, table S4.T1, published 2026-11-09). Retains 9 published evaluation configurations across 6 distinct models on 'Q mean' (native score scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《Industrial-Level Chip Floorplanning》原始论文（arXiv:2611.12048，表 S4.T1，发布日期 2026-11-09）。完整保留“Q mean”列下已发表的 9 组评测配置成绩（覆盖 6 个独立模型，采用原生分值口径（不计入准确率饱和度分档））；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.12048v1","published_at":"2026-11-09","status":"paper_baselines"},"model_count":6,"result_count":9,"score_min":0,"score_max":100,"score_decimals":2,"results":[[15870,57.97,370,15871,15872],[15873,31.17,370,15874,15872],[15875,22.73,370,15876,15872],[15877,16.18,370,15878,15872],[15879,18.92,370,15880,15872],[15881,15.52,370,15882,15872],[15877,0.0,370,15883,15872],[15879,15.25,370,15884,15872],[15881,0.0,370,15885,15872]]},{"id":"cheminsight-2026-paper","name":"ChemInsight (2026 paper)","group":"数学与科学","metric":"Depth Score","unit":"score","public_history":true,"protocol":{"en":"Primary-source results extracted from ChemInsight (arXiv:2611.11794, table S3.T1, published 2026-11-09). Retains 10 published evaluation configurations across 10 distinct models on 'Depth Score' (native score scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《ChemInsight》原始论文（arXiv:2611.11794，表 S3.T1，发布日期 2026-11-09）。完整保留“Depth Score”列下已发表的 10 组评测配置成绩（覆盖 10 个独立模型，采用原生分值口径（不计入准确率饱和度分档））；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.11794v1","published_at":"2026-11-09","status":"paper_baselines"},"model_count":10,"result_count":10,"score_min":0,"score_max":1,"score_decimals":3,"results":[[26,0.107,370,15886,15887],[2058,0.069,370,15888,15887],[2602,0.059,370,15889,15887],[3004,0.067,370,15890,15887],[3009,0.047,370,15891,15887],[767,0.034,370,15892,15887],[15893,0.028,370,15894,15887],[769,0.021,370,15895,15887],[15896,0.021,370,15897,15887],[6292,0.024,370,15898,15887]]},{"id":"fincutbench-2026-paper","name":"FinCutBench (2026 paper)","group":"专业工作","metric":"Localization Stability","unit":"%","public_history":true,"protocol":{"en":"Primary-source results extracted from FinCutBench (arXiv:2611.11588, table S6.T2, published 2026-11-08). Retains 9 published evaluation configurations across 9 distinct models on 'Localization Stability' (comparable percentage scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《FinCutBench》原始论文（arXiv:2611.11588，表 S6.T2，发布日期 2026-11-08）。完整保留“Localization Stability”列下已发表的 9 组评测配置成绩（覆盖 9 个独立模型，采用可比百分比口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.11588v2","published_at":"2026-11-08","status":"paper_baselines"},"model_count":9,"result_count":9,"results":[[3385,0.962,370,15899,15900],[5300,1.0,370,15901,15900],[2582,0.996,370,15902,15900],[2576,1.0,370,15903,15900],[4977,1.0,370,15904,15900],[4979,0.902,370,15905,15900],[5519,0.999,370,15906,15900],[6415,0.996,370,15907,15900],[6214,0.896,370,15908,15900]]},{"id":"realembedbench-2026-paper","name":"RealEmbedBench (2026 paper)","group":"编程","metric":"Full","unit":"score","public_history":true,"protocol":{"en":"Primary-source results extracted from RealEmbedBench (arXiv:2611.11469, table S6.T10, published 2026-11-08). Retains 6 published evaluation configurations across 6 distinct models on 'Full' (native score scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《RealEmbedBench》原始论文（arXiv:2611.11469，表 S6.T10，发布日期 2026-11-08）。完整保留“Full”列下已发表的 6 组评测配置成绩（覆盖 6 个独立模型，采用原生分值口径（不计入准确率饱和度分档））；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.11469v1","published_at":"2026-11-08","status":"paper_baselines"},"model_count":6,"result_count":6,"score_min":0,"score_max":1,"score_decimals":3,"results":[[15909,0.5388,370,15910,15911],[15912,0.4916,370,15913,15911],[2948,0.4901,370,15914,15911],[8463,0.7076,370,15915,15911],[15916,0.5095,370,15917,15911],[15918,0.3823,370,15919,15911]]},{"id":"surgknowbench-2026-paper","name":"SurgKnowBench (2026 paper)","group":"医疗","metric":"Phase Rec. Acc.","unit":"%","public_history":true,"protocol":{"en":"Primary-source results extracted from SurgKnowBench (arXiv:2611.11437, table S2.T2, published 2026-11-08). Retains 17 published evaluation configurations across 17 distinct models on 'Phase Rec. Acc.' (comparable percentage scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《SurgKnowBench》原始论文（arXiv:2611.11437，表 S2.T2，发布日期 2026-11-08）。完整保留“Phase Rec. Acc.”列下已发表的 17 组评测配置成绩（覆盖 17 个独立模型，采用可比百分比口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.11437v1","published_at":"2026-11-08","status":"paper_baselines"},"model_count":17,"result_count":17,"results":[[354,0.247,370,15920,15921],[6214,0.184,370,15922,15921],[2774,0.238,370,15923,15921],[2471,0.141,370,15924,15921],[6150,0.189,370,15925,15921],[2626,0.232,370,15926,15921],[4151,0.177,370,15927,15921],[2697,0.195,370,15928,15921],[2763,0.213,370,15929,15921],[3029,0.214,370,15930,15921],[2453,0.22,370,15931,15921],[6625,0.237,370,15932,15921],[2444,0.236,370,15933,15921],[2710,0.241,370,15934,15921],[15935,0.248,370,15936,15921],[8169,0.246,370,15937,15921],[15938,0.197,370,15939,15921]]},{"id":"warbias-2026-paper","name":"WarBias (2026 paper)","group":"安全与可靠性","metric":"All","unit":"%","public_history":true,"protocol":{"en":"Primary-source results extracted from WarBias (arXiv:2611.11284, table A6.T12, published 2026-11-08). Retains 12 published evaluation configurations across 6 distinct models on 'All' (comparable percentage scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《WarBias》原始论文（arXiv:2611.11284，表 A6.T12，发布日期 2026-11-08）。完整保留“All”列下已发表的 12 组评测配置成绩（覆盖 6 个独立模型，采用可比百分比口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.11284v2","published_at":"2026-11-08","status":"paper_baselines"},"model_count":6,"result_count":12,"results":[[5344,0.681,370,15940,15941],[587,0.361,370,15942,15941],[6668,0.739,370,15943,15941],[15944,0.584,370,15945,15941],[15946,0.723,370,15947,15941],[15948,0.521,370,15949,15941],[5344,0.442,370,15950,15941],[587,0.419,370,15951,15941],[6668,0.653,370,15952,15941],[15944,0.587,370,15953,15941],[15946,0.617,370,15954,15941],[15948,0.563,370,15955,15941]]},{"id":"deepfinance-2026-paper","name":"DeepFinance (2026 paper)","group":"专业工作","metric":"Score","unit":"%","public_history":true,"protocol":{"en":"Primary-source results extracted from DeepFinance (arXiv:2611.10926, table S5.T3, published 2026-11-07). Retains 22 published evaluation configurations across 22 distinct models on 'Score' (comparable percentage scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《DeepFinance》原始论文（arXiv:2611.10926，表 S5.T3，发布日期 2026-11-07）。完整保留“Score”列下已发表的 22 组评测配置成绩（覆盖 22 个独立模型，采用可比百分比口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.10926v1","published_at":"2026-11-07","status":"paper_baselines"},"model_count":22,"result_count":22,"results":[[15956,0.576,370,15957,15958],[15959,0.548,370,15960,15958],[15961,0.546,370,15962,15958],[15963,0.543,370,15964,15958],[15965,0.472,370,15966,15958],[15967,0.456,370,15968,15958],[15969,0.442,370,15970,15958],[15971,0.414,370,15972,15958],[15973,0.4,370,15974,15958],[15975,0.37,370,15976,15958],[15977,0.346,370,15978,15958],[15979,0.341,370,15980,15958],[15981,0.303,370,15982,15958],[15983,0.298,370,15984,15958],[15985,0.275,370,15986,15958],[15987,0.273,370,15988,15958],[15989,0.264,370,15990,15958],[15991,0.215,370,15992,15958],[15993,0.203,370,15994,15958],[15995,0.145,370,15996,15958],[15997,0.131,370,15998,15958],[15999,0.109,370,16000,15958]]},{"id":"endoscop-v-2026-paper","name":"EndoSCOP-V (2026 paper)","group":"视觉与视频","metric":"Strict Overall","unit":"%","public_history":true,"protocol":{"en":"Primary-source results extracted from EndoSCOP-V (arXiv:2611.10626, table S5.T3, published 2026-11-07). Retains 18 published evaluation configurations across 18 distinct models on 'Strict Overall' (comparable percentage scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《EndoSCOP-V》原始论文（arXiv:2611.10626，表 S5.T3，发布日期 2026-11-07）。完整保留“Strict Overall”列下已发表的 18 组评测配置成绩（覆盖 18 个独立模型，采用可比百分比口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.10626v1","published_at":"2026-11-07","status":"paper_baselines"},"model_count":18,"result_count":18,"results":[[3763,0.598,370,16001,16002],[4070,0.584,370,16003,16002],[2584,0.582,370,16004,16002],[6343,0.564,370,16005,16002],[8171,0.55,370,16006,16002],[16007,0.546,370,16008,16002],[6342,0.542,370,16009,16002],[8167,0.542,370,16010,16002],[8170,0.517,370,16011,16002],[9409,0.516,370,16012,16002],[4969,0.515,370,16013,16002],[8524,0.467,370,16014,16002],[4970,0.445,370,16015,16002],[16016,0.434,370,16017,16002],[16018,0.614,370,16019,16002],[16020,0.594,370,16021,16002],[3572,0.583,370,16022,16002],[16023,0.779,370,16024,16002]]},{"id":"infoorgbench-2026-paper","name":"InfoOrgBench (2026 paper)","group":"Agent 与工具","metric":"Avg Score","unit":"score","public_history":true,"protocol":{"en":"Primary-source results extracted from InfoOrgBench (arXiv:2611.10600, table S4.T3, published 2026-11-07). Retains 45 published evaluation configurations across 9 distinct models on 'Avg Score' (native score scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《InfoOrgBench》原始论文（arXiv:2611.10600，表 S4.T3，发布日期 2026-11-07）。完整保留“Avg Score”列下已发表的 45 组评测配置成绩（覆盖 9 个独立模型，采用原生分值口径（不计入准确率饱和度分档））；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.10600v1","published_at":"2026-11-07","status":"paper_baselines"},"model_count":9,"result_count":45,"score_min":0,"score_max":100,"score_decimals":2,"results":[[325,55.6,370,16025,16026],[325,57.9,370,16027,16026],[325,56.7,370,16028,16026],[325,52.9,370,16029,16026],[325,52.5,370,16030,16026],[2970,65.0,370,16031,16026],[2970,65.3,370,16032,16026],[2970,66.5,370,16033,16026],[2970,65.9,370,16034,16026],[2970,66.3,370,16035,16026],[2961,71.6,370,16036,16026],[2961,71.7,370,16037,16026],[2961,72.4,370,16038,16026],[2961,71.1,370,16039,16026],[2961,70.5,370,16040,16026],[16041,56.1,370,16042,16026],[16041,57.3,370,16043,16026],[16041,56.3,370,16044,16026],[16041,56.7,370,16045,16026],[16041,53.8,370,16046,16026],[2968,68.6,370,16047,16026],[2968,69.7,370,16048,16026],[2968,69.5,370,16049,16026],[2968,70.2,370,16050,16026],[2968,69.5,370,16051,16026],[3408,59.4,370,16052,16026],[3408,60.3,370,16053,16026],[3408,60.3,370,16054,16026],[3408,59.9,370,16055,16026],[3408,60.2,370,16056,16026],[2973,64.5,370,16057,16026],[2973,65.8,370,16058,16026],[2973,65.6,370,16059,16026],[2973,63.5,370,16060,16026],[2973,64.4,370,16061,16026],[2976,61.2,370,16062,16026],[2976,62.7,370,16063,16026],[2976,63.1,370,16064,16026],[2976,62.4,370,16065,16026],[2976,63.0,370,16066,16026],[2960,82.6,370,16067,16026],[2960,82.5,370,16068,16026],[2960,82.4,370,16069,16026],[2960,83.2,370,16070,16026],[2960,80.9,370,16071,16026]]},{"id":"motiongraphicsbench-2026-paper","name":"MotionGraphicsBench (2026 paper)","group":"视觉与视频","metric":"Program","unit":"%","public_history":true,"protocol":{"en":"Primary-source results extracted from MotionGraphicsBench (arXiv:2611.10592, table A3.T5, published 2026-11-07). Retains 16 published evaluation configurations across 16 distinct models on 'Program' (comparable percentage scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《MotionGraphicsBench》原始论文（arXiv:2611.10592，表 A3.T5，发布日期 2026-11-07）。完整保留“Program”列下已发表的 16 组评测配置成绩（覆盖 16 个独立模型，采用可比百分比口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.10592v1","published_at":"2026-11-07","status":"paper_baselines"},"model_count":16,"result_count":16,"results":[[16072,0.276,370,16073,16074],[16075,0.38,370,16076,16074],[16077,0.61,370,16078,16074],[3077,0.42,370,16079,16074],[16080,0.31,370,16081,16074],[12041,0.515,370,16082,16074],[16083,0.434,370,16084,16074],[16085,0.141,370,16086,16074],[16087,0.192,370,16088,16074],[16089,0.116,370,16090,16074],[3079,0.25,370,16091,16074],[12050,0.21,370,16092,16074],[16093,0.292,370,16094,16074],[16095,0.083,370,16096,16074],[16097,0.082,370,16098,16074],[16099,0.293,370,16100,16074]]},{"id":"yrc-bench-2026-paper","name":"YRC-Bench (2026 paper)","group":"指令与长文","metric":"COMET","unit":"score","public_history":true,"protocol":{"en":"Primary-source results extracted from YRC-Bench (arXiv:2611.10555, table A1.T6, published 2026-11-07). Retains 9 published evaluation configurations across 9 distinct models on 'COMET' (native score scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《YRC-Bench》原始论文（arXiv:2611.10555，表 A1.T6，发布日期 2026-11-07）。完整保留“COMET”列下已发表的 9 组评测配置成绩（覆盖 9 个独立模型，采用原生分值口径（不计入准确率饱和度分档））；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.10555v1","published_at":"2026-11-07","status":"paper_baselines"},"model_count":9,"result_count":9,"score_min":0,"score_max":1,"score_decimals":3,"results":[[5344,0.759374,370,16101,16102],[5300,0.788221,370,16103,16102],[16104,0.750249,370,16105,16102],[16106,0.720437,370,16107,16102],[16108,0.789696,370,16109,16102],[16110,0.811545,370,16111,16102],[5116,0.807057,370,16112,16102],[16113,0.801992,370,16114,16102],[6864,0.76862,370,16115,16102]]},{"id":"afve-bench-2026-paper","name":"AFVE-Bench (2026 paper)","group":"指令与长文","metric":"Clean (all h > 0 )","unit":"%","public_history":true,"protocol":{"en":"Primary-source results extracted from AFVE-Bench (arXiv:2611.10541, table Sx4.T3, published 2026-11-07). Retains 11 published evaluation configurations across 11 distinct models on 'Clean (all h > 0 )' (comparable percentage scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《AFVE-Bench》原始论文（arXiv:2611.10541，表 Sx4.T3，发布日期 2026-11-07）。完整保留“Clean (all h > 0 )”列下已发表的 11 组评测配置成绩（覆盖 11 个独立模型，采用可比百分比口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.10541v1","published_at":"2026-11-07","status":"paper_baselines"},"model_count":11,"result_count":11,"results":[[3568,0.2188,370,16116,16117],[3261,0.0586,370,16118,16117],[268,0.3086,370,16119,16117],[2055,0.0859,370,16120,16117],[3490,0.3961,370,16121,16117],[362,0.1953,370,16122,16117],[3601,0.1639,370,16123,16117],[345,0.0391,370,16124,16117],[2058,0.2063,370,16125,16117],[16126,0.1133,370,16127,16117],[16128,0.0312,370,16129,16117]]},{"id":"wordeval-2026-paper","name":"WordEval (2026 paper)","group":"专业工作","metric":"Overall · Case","unit":"%","public_history":true,"protocol":{"en":"Primary-source results extracted from WordEval (arXiv:2611.10475, table S4.T3, published 2026-11-07). Retains 9 published evaluation configurations across 9 distinct models on 'Overall · Case' (comparable percentage scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《WordEval》原始论文（arXiv:2611.10475，表 S4.T3，发布日期 2026-11-07）。完整保留“Overall · Case”列下已发表的 9 组评测配置成绩（覆盖 9 个独立模型，采用可比百分比口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.10475v1","published_at":"2026-11-07","status":"paper_baselines"},"model_count":9,"result_count":9,"results":[[16130,0.871,370,16131,16132],[16133,0.667,370,16134,16132],[16135,0.618,370,16136,16132],[16137,0.565,370,16138,16132],[16139,0.555,370,16140,16132],[16141,0.528,370,16142,16132],[16143,0.475,370,16144,16132],[16145,0.445,370,16146,16132],[16147,0.828,370,16148,16132]]},{"id":"turnflow-2026-paper","name":"TurnFlow (2026 paper)","group":"知识与推理","metric":"Overall","unit":"%","public_history":true,"protocol":{"en":"Primary-source results extracted from TurnFlow (arXiv:2611.10412, table S3.T2, published 2026-11-07). Retains 9 published evaluation configurations across 9 distinct models on 'Overall' (comparable percentage scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《TurnFlow》原始论文（arXiv:2611.10412，表 S3.T2，发布日期 2026-11-07）。完整保留“Overall”列下已发表的 9 组评测配置成绩（覆盖 9 个独立模型，采用可比百分比口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.10412v1","published_at":"2026-11-07","status":"paper_baselines"},"model_count":9,"result_count":9,"results":[[3155,0.5959,370,16149,16150],[16151,0.5878,370,16152,16150],[3153,0.5286,370,16153,16150],[16154,0.5265,370,16155,16150],[3198,0.5041,370,16156,16150],[3185,0.5,370,16157,16150],[3190,0.498,370,16158,16150],[268,0.4735,370,16159,16150],[3763,0.4286,370,16160,16150]]},{"id":"siturecbench-2026-paper","name":"SituRecBench (2026 paper)","group":"视觉与视频","metric":"NDCG@10 (%)","unit":"score","public_history":true,"protocol":{"en":"Primary-source results extracted from SituRecBench (arXiv:2611.10349, table S3.T3, published 2026-11-07). Retains 14 published evaluation configurations across 7 distinct models on 'NDCG@10 (%)' (native score scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《SituRecBench》原始论文（arXiv:2611.10349，表 S3.T3，发布日期 2026-11-07）。完整保留“NDCG@10 (%)”列下已发表的 14 组评测配置成绩（覆盖 7 个独立模型，采用原生分值口径（不计入准确率饱和度分档））；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.10349v1","published_at":"2026-11-07","status":"paper_baselines"},"model_count":7,"result_count":14,"score_min":0,"score_max":100,"score_decimals":2,"results":[[2627,15.0,370,16161,16162],[6342,17.7,370,16163,16162],[353,26.6,370,16164,16162],[3189,42.4,370,16165,16162],[755,43.5,370,16166,16162],[762,48.2,370,16167,16162],[3412,56.0,370,16168,16162],[6342,13.7,370,16169,16162],[2627,16.4,370,16170,16162],[353,20.2,370,16171,16162],[755,33.8,370,16172,16162],[3189,35.5,370,16173,16162],[762,43.0,370,16174,16162],[3412,43.7,370,16175,16162]]},{"id":"mario-2026-paper","name":"MaRiO (2026 paper)","group":"Agent 与工具","metric":"Avg.","unit":"score","public_history":true,"protocol":{"en":"Primary-source results extracted from MaRiO (arXiv:2611.10344, table S3.T2, published 2026-11-07). Retains 23 published evaluation configurations across 23 distinct models on 'Avg.' (native score scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《MaRiO》原始论文（arXiv:2611.10344，表 S3.T2，发布日期 2026-11-07）。完整保留“Avg.”列下已发表的 23 组评测配置成绩（覆盖 23 个独立模型，采用原生分值口径（不计入准确率饱和度分档））；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.10344v1","published_at":"2026-11-07","status":"paper_baselines"},"model_count":23,"result_count":23,"score_min":0,"score_max":100,"score_decimals":2,"results":[[3649,39.4,370,16176,16177],[3644,46.6,370,16178,16177],[766,43.7,370,16179,16177],[764,56.7,370,16180,16177],[16181,36.9,370,16182,16177],[9591,36.0,370,16183,16177],[16184,38.7,370,16185,16177],[16186,38.8,370,16187,16177],[7709,43.2,370,16188,16177],[2478,41.4,370,16189,16177],[2471,40.6,370,16190,16177],[16191,40.1,370,16192,16177],[7803,33.3,370,16193,16177],[16194,33.2,370,16195,16177],[2766,34.7,370,16196,16177],[2728,38.1,370,16197,16177],[2911,37.9,370,16198,16177],[16199,32.4,370,16200,16177],[13731,37.7,370,16201,16177],[16202,35.8,370,16203,16177],[2646,42.7,370,16204,16177],[16205,29.9,370,16206,16177],[16207,32.9,370,16208,16177]]},{"id":"longtail-swe-2026-paper","name":"Longtail-SWE (2026 paper)","group":"编程","metric":"All-3","unit":"%","public_history":true,"protocol":{"en":"Primary-source results extracted from Longtail-SWE (arXiv:2611.10304, table S3.T1, published 2026-11-06). Retains 9 published evaluation configurations across 9 distinct models on 'All-3' (comparable percentage scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《Longtail-SWE》原始论文（arXiv:2611.10304，表 S3.T1，发布日期 2026-11-06）。完整保留“All-3”列下已发表的 9 组评测配置成绩（覆盖 9 个独立模型，采用可比百分比口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.10304v1","published_at":"2026-11-06","status":"paper_baselines"},"model_count":9,"result_count":9,"results":[[16209,0.7376,370,16210,16211],[16212,0.7327,370,16213,16211],[16214,0.6337,370,16215,16211],[16216,0.5693,370,16217,16211],[16093,0.4851,370,16218,16211],[16219,0.3911,370,16220,16211],[16221,0.2376,370,16222,16211],[16223,0.2079,370,16224,16211],[16097,0.0941,370,16225,16211]]},{"id":"deepresearch-rewardbench-2026-paper","name":"DeepResearch RewardBench (2026 paper)","group":"Agent 与工具","metric":"Average","unit":"score","public_history":true,"protocol":{"en":"Primary-source results extracted from DeepResearch RewardBench (arXiv:2611.10156, table S3.T2, published 2026-11-06). Retains 19 published evaluation configurations across 19 distinct models on 'Average' (native score scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《DeepResearch RewardBench》原始论文（arXiv:2611.10156，表 S3.T2，发布日期 2026-11-06）。完整保留“Average”列下已发表的 19 组评测配置成绩（覆盖 19 个独立模型，采用原生分值口径（不计入准确率饱和度分档））；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.10156v1","published_at":"2026-11-06","status":"paper_baselines"},"model_count":19,"result_count":19,"score_min":0,"score_max":100,"score_decimals":2,"results":[[9318,78.87,370,16226,16227],[16228,77.79,370,16229,16227],[16230,77.79,370,16231,16227],[16232,77.04,370,16233,16227],[16234,71.29,370,16235,16227],[16236,66.62,370,16237,16227],[16238,64.35,370,16239,16227],[16240,63.75,370,16241,16227],[16242,61.63,370,16243,16227],[16244,59.82,370,16245,16227],[16246,55.29,370,16247,16227],[16248,54.53,370,16249,16227],[16250,54.23,370,16251,16227],[16252,51.61,370,16253,16227],[16254,49.7,370,16255,16227],[16256,36.71,370,16257,16227],[16258,36.41,370,16259,16227],[16260,31.09,370,16261,16227],[16262,25.68,370,16263,16227]]},{"id":"culturalcrosswords-2026-paper","name":"CulturalCrosswords (2026 paper)","group":"知识与推理","metric":"E","unit":"%","public_history":true,"protocol":{"en":"Primary-source results extracted from CulturalCrosswords (arXiv:2611.10044, table A7.T9, published 2026-11-05). Retains 11 published evaluation configurations across 11 distinct models on 'E' (comparable percentage scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《CulturalCrosswords》原始论文（arXiv:2611.10044，表 A7.T9，发布日期 2026-11-05）。完整保留“E”列下已发表的 11 组评测配置成绩（覆盖 11 个独立模型，采用可比百分比口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.10044v2","published_at":"2026-11-05","status":"paper_baselines"},"model_count":11,"result_count":11,"results":[[2960,0.881,370,16264,16265],[2958,0.899,370,16266,16265],[3196,0.832,370,16267,16265],[2962,0.911,370,16268,16265],[2983,0.676,370,16269,16265],[3386,0.756,370,16270,16265],[3412,0.565,370,16271,16265],[2956,0.841,370,16272,16265],[2959,0.826,370,16273,16265],[3578,0.588,370,16274,16265],[3479,0.249,370,16275,16265]]},{"id":"seeing-what-execution-needs-2026-paper","name":"Seeing What Execution Needs (2026 paper)","group":"视觉与视频","metric":"Success","unit":"%","public_history":true,"protocol":{"en":"Primary-source results extracted from Seeing What Execution Needs (arXiv:2611.09794, table A6.T8, published 2026-11-06). Retains 7 published evaluation configurations across 7 distinct models on 'Success' (comparable percentage scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《Seeing What Execution Needs》原始论文（arXiv:2611.09794，表 A6.T8，发布日期 2026-11-06）。完整保留“Success”列下已发表的 7 组评测配置成绩（覆盖 7 个独立模型，采用可比百分比口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.09794v2","published_at":"2026-11-06","status":"paper_baselines"},"model_count":7,"result_count":7,"results":[[4039,0.325,370,16276,16277],[3493,0.429,370,16278,16277],[2974,0.425,370,16279,16277],[362,0.283,370,16280,16277],[4337,0.403,370,16281,16277],[6342,0.394,370,16282,16277],[16283,0.26,370,16284,16277]]},{"id":"animweb-bench-2026-paper","name":"AnimWeb-Bench (2026 paper)","group":"编程","metric":"Animation Fidelity · CLIP","unit":"score","public_history":true,"protocol":{"en":"Primary-source results extracted from AnimWeb-Bench (arXiv:2611.09769, table S4.tab1, published 2026-11-06). Retains 14 published evaluation configurations across 14 distinct models on 'Animation Fidelity · CLIP' (native score scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《AnimWeb-Bench》原始论文（arXiv:2611.09769，表 S4.tab1，发布日期 2026-11-06）。完整保留“Animation Fidelity · CLIP”列下已发表的 14 组评测配置成绩（覆盖 14 个独立模型，采用原生分值口径（不计入准确率饱和度分档））；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.09769v1","published_at":"2026-11-06","status":"paper_baselines"},"model_count":14,"result_count":14,"score_min":0,"score_max":1,"score_decimals":3,"results":[[2958,0.4303,370,16285,16286],[3394,0.3511,370,16287,16286],[762,0.3355,370,16288,16286],[2956,0.366,370,16289,16286],[755,0.2922,370,16290,16286],[2968,0.296,370,16291,16286],[5664,0.2793,370,16292,16286],[3482,0.2148,370,16293,16286],[2983,0.2117,370,16294,16286],[12842,0.1908,370,16295,16286],[2977,0.3218,370,16296,16286],[3512,0.2836,370,16297,16286],[3529,0.2733,370,16298,16286],[16299,0.1584,370,16300,16286]]},{"id":"abhbench-2026-paper","name":"ABHBench (2026 paper)","group":"Agent 与工具","metric":"CE","unit":"score","public_history":true,"protocol":{"en":"Primary-source results extracted from ABHBench (arXiv:2611.09326, table S5.T3, published 2026-11-06). Retains 12 published evaluation configurations across 12 distinct models on 'CE' (native score scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《ABHBench》原始论文（arXiv:2611.09326，表 S5.T3，发布日期 2026-11-06）。完整保留“CE”列下已发表的 12 组评测配置成绩（覆盖 12 个独立模型，采用原生分值口径（不计入准确率饱和度分档））；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.09326v1","published_at":"2026-11-06","status":"paper_baselines"},"model_count":12,"result_count":12,"score_min":0,"score_max":1,"score_decimals":3,"results":[[4039,0.805,370,16301,16302],[16303,0.707,370,16304,16302],[6390,0.665,370,16305,16302],[6901,0.879,370,16306,16302],[801,0.955,370,16307,16302],[353,0.832,370,16308,16302],[6275,0.869,370,16309,16302],[345,0.915,370,16310,16302],[16311,0.931,370,16312,16302],[4075,0.636,370,16313,16302],[1464,0.787,370,16314,16302],[16315,0.885,370,16316,16302]]},{"id":"klear-2026-paper","name":"KLEAR (2026 paper)","group":"知识与推理","metric":"All · A3","unit":"%","public_history":true,"protocol":{"en":"Primary-source results extracted from KLEAR (arXiv:2611.09223, table S5.T5, published 2026-11-06). Retains 7 published evaluation configurations across 7 distinct models on 'All · A3' (comparable percentage scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《KLEAR》原始论文（arXiv:2611.09223，表 S5.T5，发布日期 2026-11-06）。完整保留“All · A3”列下已发表的 7 组评测配置成绩（覆盖 7 个独立模型，采用可比百分比口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.09223v1","published_at":"2026-11-06","status":"paper_baselines"},"model_count":7,"result_count":7,"results":[[16317,0.455,370,16318,16319],[16320,0.282,370,16321,16319],[8591,0.234,370,16322,16319],[2278,0.603,370,16323,16319],[3417,0.689,370,16324,16319],[2961,0.818,370,16325,16319],[2960,0.923,370,16326,16319]]},{"id":"khmer-pfgen-bench-2026-paper","name":"Khmer PFGen-Bench (2026 paper)","group":"知识与推理","metric":"Score","unit":"score","public_history":true,"protocol":{"en":"Primary-source results extracted from Khmer PFGen-Bench (arXiv:2611.09222, table A3.T1, published 2026-11-06). Retains 19 published evaluation configurations across 19 distinct models on 'Score' (native score scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《Khmer PFGen-Bench》原始论文（arXiv:2611.09222，表 A3.T1，发布日期 2026-11-06）。完整保留“Score”列下已发表的 19 组评测配置成绩（覆盖 19 个独立模型，采用原生分值口径（不计入准确率饱和度分档））；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.09222v1","published_at":"2026-11-06","status":"paper_baselines"},"model_count":19,"result_count":19,"score_min":0,"score_max":1,"score_decimals":3,"results":[[16327,0.875,370,16328,16329],[16330,0.853,370,16331,16329],[5408,0.846,370,16332,16329],[5365,0.837,370,16333,16329],[4048,0.832,370,16334,16329],[16335,0.815,370,16336,16329],[5874,0.783,370,16337,16329],[6338,0.747,370,16338,16329],[3185,0.725,370,16339,16329],[5480,0.712,370,16340,16329],[3191,0.671,370,16341,16329],[16342,0.651,370,16343,16329],[6886,0.646,370,16344,16329],[1583,0.609,370,16345,16329],[5411,0.547,370,16346,16329],[16347,0.349,370,16348,16329],[16349,0.348,370,16350,16329],[16351,0.303,370,16352,16329],[16353,0.28,370,16354,16329]]},{"id":"ataribench-2026-paper","name":"AtariBench (2026 paper)","group":"视觉与视频","metric":"Overall","unit":"%","public_history":true,"protocol":{"en":"Primary-source results extracted from AtariBench (arXiv:2611.08755, table S4.T2, published 2026-11-05). Retains 11 published evaluation configurations across 7 distinct models on 'Overall' (comparable percentage scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《AtariBench》原始论文（arXiv:2611.08755，表 S4.T2，发布日期 2026-11-05）。完整保留“Overall”列下已发表的 11 组评测配置成绩（覆盖 7 个独立模型，采用可比百分比口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.08755v1","published_at":"2026-11-05","status":"paper_baselines"},"model_count":7,"result_count":11,"results":[[16355,0.623,370,16356,16357],[16358,0.657,370,16359,16357],[16360,0.753,370,16361,16357],[16362,0.833,370,16363,16357],[3,0.813,370,16364,16357],[13363,0.837,370,16365,16357],[3079,0.8,370,16366,16357],[16367,0.863,370,16368,16357],[3078,0.823,370,16369,16357],[16370,0.637,370,16371,16357],[3075,0.68,370,16372,16357]]},{"id":"xstack-2026-paper","name":"XStack (2026 paper)","group":"Agent 与工具","metric":"+weights","unit":"%","public_history":true,"protocol":{"en":"Primary-source results extracted from XStack (arXiv:2611.08472, table S3.T4, published 2026-11-05). Retains 7 published evaluation configurations across 7 distinct models on '+weights' (comparable percentage scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《XStack》原始论文（arXiv:2611.08472，表 S3.T4，发布日期 2026-11-05）。完整保留“+weights”列下已发表的 7 组评测配置成绩（覆盖 7 个独立模型，采用可比百分比口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.08472v1","published_at":"2026-11-05","status":"paper_baselines"},"model_count":7,"result_count":7,"results":[[4983,0.75,370,16373,16374],[4048,0.45,370,16375,16374],[16376,0.4,370,16377,16374],[3396,0.25,370,16378,16374],[5134,0.2,370,16379,16374],[5298,0.1,370,16380,16374],[5297,0.05,370,16381,16374]]},{"id":"interactive-visual-gym-2026-paper","name":"Interactive Visual Gym (2026 paper)","group":"视觉与视频","metric":"Interaction quality Overall","unit":"%","public_history":true,"protocol":{"en":"Primary-source results extracted from Interactive Visual Gym (arXiv:2611.08286, table S4.T2, published 2026-11-05). Retains 15 published evaluation configurations across 15 distinct models on 'Interaction quality Overall' (comparable percentage scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《Interactive Visual Gym》原始论文（arXiv:2611.08286，表 S4.T2，发布日期 2026-11-05）。完整保留“Interaction quality Overall”列下已发表的 15 组评测配置成绩（覆盖 15 个独立模型，采用可比百分比口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.08286v1","published_at":"2026-11-05","status":"paper_baselines"},"model_count":15,"result_count":15,"results":[[2962,0.871,370,16382,16383],[2983,0.894,370,16384,16383],[2960,0.865,370,16385,16383],[2958,0.863,370,16386,16383],[16387,0.845,370,16388,16383],[6580,0.891,370,16389,16383],[16390,0.538,370,16391,16383],[5300,0.862,370,16392,16383],[16393,0.339,370,16394,16383],[16395,0.33,370,16396,16383],[5725,0.392,370,16397,16383],[9195,0.582,370,16398,16383],[6574,0.385,370,16399,16383],[8222,0.372,370,16400,16383],[16401,0.279,370,16402,16383]]},{"id":"tcsexam-2026-paper","name":"TCSExam (2026 paper)","group":"数学与科学","metric":"Open conjectures Resolved","unit":"%","public_history":true,"protocol":{"en":"Primary-source results extracted from TCSExam (arXiv:2611.07962, table S5.T3, published 2026-11-05). Retains 9 published evaluation configurations across 8 distinct models on 'Open conjectures Resolved' (comparable percentage scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《TCSExam》原始论文（arXiv:2611.07962，表 S5.T3，发布日期 2026-11-05）。完整保留“Open conjectures Resolved”列下已发表的 9 组评测配置成绩（覆盖 8 个独立模型，采用可比百分比口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.07962v1","published_at":"2026-11-05","status":"paper_baselines"},"model_count":8,"result_count":9,"results":[[3384,0.0,370,16403,16404],[2958,0.0,370,16405,16404],[2967,0.0,370,16406,16404],[2977,0.0,370,16407,16404],[3154,0.0,370,16408,16404],[5664,0.0,370,16409,16404],[2966,0.0,370,16410,16404],[2974,0.0,370,16411,16404],[16412,0.0,370,16413,16404]]},{"id":"beyond-instruction-override-2026-paper","name":"Beyond Instruction Override (2026 paper)","group":"Agent 与工具","metric":"ASR All","unit":"%","public_history":true,"protocol":{"en":"Primary-source results extracted from Beyond Instruction Override (arXiv:2611.07640, table A5.T12, published 2026-11-05). Retains 25 published evaluation configurations across 5 distinct models on 'ASR All' (comparable percentage scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《Beyond Instruction Override》原始论文（arXiv:2611.07640，表 A5.T12，发布日期 2026-11-05）。完整保留“ASR All”列下已发表的 25 组评测配置成绩（覆盖 5 个独立模型，采用可比百分比口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.07640v1","published_at":"2026-11-05","status":"paper_baselines"},"model_count":5,"result_count":25,"higher_is_better":false,"saturation_excluded":true,"results":[[7176,0.232,370,16414,16415],[244,0.174,370,16416,16415],[6420,0.206,370,16417,16415],[3442,0.111,370,16418,16415],[268,0.129,370,16419,16415],[7176,0.221,370,16420,16415],[244,0.175,370,16421,16415],[6420,0.186,370,16422,16415],[3442,0.081,370,16423,16415],[268,0.118,370,16424,16415],[7176,0.166,370,16425,16415],[244,0.103,370,16426,16415],[6420,0.106,370,16427,16415],[3442,0.047,370,16428,16415],[268,0.098,370,16429,16415],[7176,0.207,370,16430,16415],[244,0.129,370,16431,16415],[6420,0.141,370,16432,16415],[3442,0.053,370,16433,16415],[268,0.111,370,16434,16415],[7176,0.171,370,16435,16415],[244,0.078,370,16436,16415],[6420,0.094,370,16437,16415],[3442,0.028,370,16438,16415],[268,0.075,370,16439,16415]]},{"id":"jailbound-2026-paper","name":"JailBound (2026 paper)","group":"安全与可靠性","metric":"Avg.","unit":"%","public_history":true,"protocol":{"en":"Primary-source results extracted from JailBound (arXiv:2611.07530, table A6.T17, published 2026-11-05). Retains 45 published evaluation configurations across 45 distinct models on 'Avg.' (comparable percentage scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《JailBound》原始论文（arXiv:2611.07530，表 A6.T17，发布日期 2026-11-05）。完整保留“Avg.”列下已发表的 45 组评测配置成绩（覆盖 45 个独立模型，采用可比百分比口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.07530v1","published_at":"2026-11-05","status":"paper_baselines"},"model_count":45,"result_count":45,"results":[[1828,0.502,370,16440,16441],[1804,0.261,370,16442,16441],[562,0.412,370,16443,16441],[16444,0.24,370,16445,16441],[1830,0.355,370,16446,16441],[1826,0.498,370,16447,16441],[14412,0.377,370,16448,16441],[1748,0.419,370,16449,16441],[1645,0.565,370,16450,16441],[6340,0.379,370,16451,16441],[4969,0.49,370,16452,16441],[4070,0.606,370,16453,16441],[2584,0.523,370,16454,16441],[2590,0.547,370,16455,16441],[16456,0.331,370,16457,16441],[10899,0.324,370,16458,16441],[16459,0.439,370,16460,16441],[5535,0.431,370,16461,16441],[7607,0.386,370,16462,16441],[1810,0.444,370,16463,16441],[561,0.41,370,16464,16441],[1649,0.558,370,16465,16441],[565,0.334,370,16466,16441],[602,0.563,370,16467,16441],[16468,0.418,370,16469,16441],[16470,0.648,370,16471,16441],[16472,0.618,370,16473,16441],[16474,0.587,370,16475,16441],[595,0.605,370,16476,16441],[16477,0.509,370,16478,16441],[16479,0.317,370,16480,16441],[16481,0.589,370,16482,16441],[4479,0.407,370,16483,16441],[550,0.335,370,16484,16441],[1864,0.632,370,16485,16441],[16486,0.677,370,16487,16441],[6214,0.242,370,16488,16441],[6704,0.208,370,16489,16441],[6705,0.175,370,16490,16441],[345,0.295,370,16491,16441],[353,0.363,370,16492,16441],[3748,0.329,370,16493,16441],[6130,0.475,370,16494,16441],[764,0.362,370,16495,16441],[766,0.392,370,16496,16441]]},{"id":"reqmembench-2026-paper","name":"ReqMemBench (2026 paper)","group":"编程","metric":"Full History (C1)","unit":"%","public_history":true,"protocol":{"en":"Primary-source results extracted from ReqMemBench (arXiv:2611.07307, table S5.T2, published 2026-11-05). Retains 6 published evaluation configurations across 6 distinct models on 'Full History (C1)' (comparable percentage scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《ReqMemBench》原始论文（arXiv:2611.07307，表 S5.T2，发布日期 2026-11-05）。完整保留“Full History (C1)”列下已发表的 6 组评测配置成绩（覆盖 6 个独立模型，采用可比百分比口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.07307v1","published_at":"2026-11-05","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[2958,0.1081,370,16497,16498],[3394,0.0541,370,16499,16498],[762,0.027,370,16500,16498],[2973,0.0541,370,16501,16498],[3384,0.0541,370,16502,16498],[3393,0.0,370,16503,16498]]},{"id":"geoworld-2026-paper","name":"GeoWorld (2026 paper)","group":"Agent 与工具","metric":"Task-Only","unit":"%","public_history":true,"protocol":{"en":"Primary-source results extracted from GeoWorld (arXiv:2611.17267, table S5.T3, published 2026-11-12). Retains 9 published evaluation configurations across 9 distinct models on 'Task-Only' (comparable percentage scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《GeoWorld》原始论文（arXiv:2611.17267，表 S5.T3，发布日期 2026-11-12）。完整保留“Task-Only”列下已发表的 9 组评测配置成绩（覆盖 9 个独立模型，采用可比百分比口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.17267v1","published_at":"2026-11-12","status":"paper_baselines"},"model_count":9,"result_count":9,"results":[[762,0.027,370,16504,16505],[755,0.047,370,16506,16505],[3568,0.027,370,16507,16505],[3482,0.06,370,16508,16505],[3386,0.141,370,16509,16505],[3391,0.101,370,16510,16505],[3157,0.047,370,16511,16505],[2977,0.027,370,16512,16505],[5521,0.034,370,16513,16505]]},{"id":"auditing-ai-peer-reviewers-2026-paper","name":"Auditing AI peer reviewers (2026 paper)","group":"数学与科学","metric":"Total","unit":"raw","public_history":true,"protocol":{"en":"Primary-source results extracted from Auditing AI peer reviewers (arXiv:2611.17052, table A7.T9, published 2026-11-12). Retains 16 published evaluation configurations across 8 distinct models on 'Total' (raw reported metric scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《Auditing AI peer reviewers》原始论文（arXiv:2611.17052，表 A7.T9，发布日期 2026-11-12）。完整保留“Total”列下已发表的 16 组评测配置成绩（覆盖 8 个独立模型，采用原始指标口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.17052v1","published_at":"2026-11-12","status":"paper_baselines"},"model_count":8,"result_count":16,"score_decimals":2,"results":[[16514,16.6,370,16515,16516],[16517,0.8,370,16518,16516],[16519,25.4,370,16520,16516],[16521,32.2,370,16522,16516],[16523,-7.9,370,16524,16516],[16525,7.8,370,16526,16516],[16527,9.6,370,16528,16516],[16529,9.3,370,16530,16516],[16514,28.3,370,16531,16516],[16517,17.2,370,16532,16516],[16519,40.0,370,16533,16516],[16521,38.9,370,16534,16516],[16523,-9.4,370,16535,16516],[16525,1.1,370,16536,16516],[16527,3.3,370,16537,16516],[16529,-4.4,370,16538,16516]]},{"id":"analoglayoutworld-2026-paper","name":"AnalogLayoutWorld (2026 paper)","group":"Agent 与工具","metric":"Avg. Score","unit":"score","public_history":true,"protocol":{"en":"Primary-source results extracted from AnalogLayoutWorld (arXiv:2611.16991, table S4.T1, published 2026-11-12). Retains 14 published evaluation configurations across 11 distinct models on 'Avg. Score' (native score scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《AnalogLayoutWorld》原始论文（arXiv:2611.16991，表 S4.T1，发布日期 2026-11-12）。完整保留“Avg. Score”列下已发表的 14 组评测配置成绩（覆盖 11 个独立模型，采用原生分值口径（不计入准确率饱和度分档））；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.16991v1","published_at":"2026-11-12","status":"paper_baselines"},"model_count":11,"result_count":14,"score_min":0,"score_max":1,"score_decimals":3,"results":[[16539,0.8154,370,16540,16541],[16542,0.1504,370,16543,16541],[16544,0.1974,370,16545,16541],[16544,0.2246,370,16546,16541],[16547,0.1075,370,16548,16541],[15879,0.4779,370,16549,16541],[15879,0.4632,370,16550,16541],[16551,0.2555,370,16552,16541],[16551,0.2378,370,16553,16541],[16554,0.0958,370,16555,16541],[16556,0.29,370,16557,16541],[16558,0.2633,370,16559,16541],[16560,0.7483,370,16561,16541],[16562,0.7023,370,16563,16541]]},{"id":"visual-cipher-bench-2026-paper","name":"Visual Cipher Bench (2026 paper)","group":"视觉与视频","metric":"Acc.","unit":"%","public_history":true,"protocol":{"en":"Primary-source results extracted from Visual Cipher Bench (arXiv:2611.16861, table S3.T1, published 2026-11-12). Retains 29 published evaluation configurations across 8 distinct models on 'Acc.' (comparable percentage scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《Visual Cipher Bench》原始论文（arXiv:2611.16861，表 S3.T1，发布日期 2026-11-12）。完整保留“Acc.”列下已发表的 29 组评测配置成绩（覆盖 8 个独立模型，采用可比百分比口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.16861v1","published_at":"2026-11-12","status":"paper_baselines"},"model_count":8,"result_count":29,"results":[[2958,0.267,370,16564,16565],[2958,0.225,370,16566,16565],[2958,0.217,370,16567,16565],[2958,0.117,370,16568,16565],[2958,0.05,370,16569,16565],[2960,0.217,370,16570,16565],[2960,0.158,370,16571,16565],[2960,0.117,370,16572,16565],[2960,0.083,370,16573,16565],[2960,0.05,370,16574,16565],[2959,0.175,370,16575,16565],[2959,0.217,370,16576,16565],[2959,0.108,370,16577,16565],[2959,0.083,370,16578,16565],[2959,0.0,370,16579,16565],[2976,0.008,370,16580,16565],[2976,0.0,370,16581,16565],[2976,0.0,370,16582,16565],[2976,0.0,370,16583,16565],[2976,0.0,370,16584,16565],[2976,0.0,370,16585,16565],[2961,0.008,370,16586,16565],[2961,0.0,370,16587,16565],[2961,0.0,370,16588,16565],[2961,0.0,370,16589,16565],[2961,0.008,370,16590,16565],[3512,0.0,370,16591,16565],[3560,0.0,370,16592,16565],[2967,0.0,370,16593,16565]]},{"id":"dashbench-2026-paper","name":"DashBench (2026 paper)","group":"视觉与视频","metric":"Avg Agree","unit":"%","public_history":true,"protocol":{"en":"Primary-source results extracted from DashBench (arXiv:2611.16767, table S4.T1, published 2026-11-12). Retains 25 published evaluation configurations across 25 distinct models on 'Avg Agree' (comparable percentage scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《DashBench》原始论文（arXiv:2611.16767，表 S4.T1，发布日期 2026-11-12）。完整保留“Avg Agree”列下已发表的 25 组评测配置成绩（覆盖 25 个独立模型，采用可比百分比口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.16767v1","published_at":"2026-11-12","status":"paper_baselines"},"model_count":25,"result_count":25,"results":[[2058,0.3475,370,16594,16595],[6133,0.3441,370,16596,16595],[16597,0.2067,370,16598,16595],[16599,0.1848,370,16600,16595],[16601,0.1721,370,16602,16595],[16603,0.1643,370,16604,16595],[16605,0.1607,370,16606,16595],[16607,0.1278,370,16608,16595],[16609,0.1261,370,16610,16595],[16611,0.1234,370,16612,16595],[16613,0.1145,370,16614,16595],[13091,0.0864,370,16615,16595],[16616,0.0776,370,16617,16595],[16618,0.0774,370,16619,16595],[16620,0.0633,370,16621,16595],[16622,0.0544,370,16623,16595],[16624,0.0438,370,16625,16595],[16626,0.0157,370,16627,16595],[16628,0.1001,370,16629,16595],[16630,0.0591,370,16631,16595],[16632,0.0276,370,16633,16595],[16634,0.0142,370,16635,16595],[16636,0.011,370,16637,16595],[16638,0.0087,370,16639,16595],[16640,0.0032,370,16641,16595]]},{"id":"musetom-2026-paper","name":"MuSeToM (2026 paper)","group":"知识与推理","metric":"opentom Acc","unit":"%","public_history":true,"protocol":{"en":"Primary-source results extracted from MuSeToM (arXiv:2611.16662, table A7.T24, published 2026-11-12). Retains 18 published evaluation configurations across 18 distinct models on 'opentom Acc' (comparable percentage scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《MuSeToM》原始论文（arXiv:2611.16662，表 A7.T24，发布日期 2026-11-12）。完整保留“opentom Acc”列下已发表的 18 组评测配置成绩（覆盖 18 个独立模型，采用可比百分比口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.16662v1","published_at":"2026-11-12","status":"paper_baselines"},"model_count":18,"result_count":18,"results":[[353,0.7263,370,16642,16643],[7609,0.4775,370,16644,16643],[3419,0.882,370,16645,16643],[16646,0.5112,370,16647,16643],[16648,0.7576,370,16649,16643],[9052,0.7207,370,16650,16643],[8544,0.6252,370,16651,16643],[4828,0.6774,370,16652,16643],[784,0.7376,370,16653,16643],[513,0.8186,370,16654,16643],[769,0.7761,370,16655,16643],[16656,0.6316,370,16657,16643],[6292,0.813,370,16658,16643],[3009,0.6854,370,16659,16643],[2288,0.626,370,16660,16643],[16661,0.7632,370,16662,16643],[3005,0.8339,370,16663,16643],[2576,0.8339,370,16664,16643]]},{"id":"singularity-bench-2026-paper","name":"Singularity-Bench (2026 paper)","group":"Agent 与工具","metric":"Mid-training Acc","unit":"%","public_history":true,"protocol":{"en":"Primary-source results extracted from Singularity-Bench (arXiv:2611.16591, table S5.T5, published 2026-11-12). Retains 11 published evaluation configurations across 11 distinct models on 'Mid-training Acc' (comparable percentage scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《Singularity-Bench》原始论文（arXiv:2611.16591，表 S5.T5，发布日期 2026-11-12）。完整保留“Mid-training Acc”列下已发表的 11 组评测配置成绩（覆盖 11 个独立模型，采用可比百分比口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.16591v1","published_at":"2026-11-12","status":"paper_baselines"},"model_count":11,"result_count":11,"results":[[16665,0.319,370,16666,16667],[16668,0.178,370,16669,16667],[16670,0.201,370,16671,16667],[16672,0.155,370,16673,16667],[16674,0.099,370,16675,16667],[16676,0.091,370,16677,16667],[16678,0.051,370,16679,16667],[16680,0.134,370,16681,16667],[16682,0.118,370,16683,16667],[16684,0.045,370,16685,16667],[16686,0.039,370,16687,16667]]},{"id":"crossploit-2026-paper","name":"CrossPloit (2026 paper)","group":"安全与可靠性","metric":"MM","unit":"score","public_history":true,"protocol":{"en":"Primary-source results extracted from CrossPloit (arXiv:2611.16588, table S6.T7, published 2026-11-12). Retains 18 published evaluation configurations across 16 distinct models on 'MM' (native score scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《CrossPloit》原始论文（arXiv:2611.16588，表 S6.T7，发布日期 2026-11-12）。完整保留“MM”列下已发表的 18 组评测配置成绩（覆盖 16 个独立模型，采用原生分值口径（不计入准确率饱和度分档））；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.16588v1","published_at":"2026-11-12","status":"paper_baselines"},"model_count":16,"result_count":18,"score_min":0,"score_max":100,"score_decimals":2,"results":[[16688,28.2,370,16689,16690],[16691,27.1,370,16692,16690],[16693,30.9,370,16694,16690],[16695,23.9,370,16696,16690],[16697,20.0,370,16698,16690],[16699,20.9,370,16700,16690],[16701,16.7,370,16702,16690],[16703,15.9,370,16704,16690],[16705,15.1,370,16706,16690],[16707,24.6,370,16708,16690],[16709,25.8,370,16710,16690],[16693,27.7,370,16711,16690],[16712,24.7,370,16713,16690],[16714,22.2,370,16715,16690],[16716,22.1,370,16717,16690],[16701,15.2,370,16718,16690],[16719,14.1,370,16720,16690],[16721,14.3,370,16722,16690]]},{"id":"rea-bench-2026-paper","name":"ReA-Bench (2026 paper)","group":"Agent 与工具","metric":"Overall (177)","unit":"score","public_history":true,"protocol":{"en":"Primary-source results extracted from ReA-Bench (arXiv:2611.16493, table A2.T16a, published 2026-11-12). Retains 32 published evaluation configurations across 7 distinct models on 'Overall (177)' (native score scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《ReA-Bench》原始论文（arXiv:2611.16493，表 A2.T16a，发布日期 2026-11-12）。完整保留“Overall (177)”列下已发表的 32 组评测配置成绩（覆盖 7 个独立模型，采用原生分值口径（不计入准确率饱和度分档））；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.16493v1","published_at":"2026-11-12","status":"paper_baselines"},"model_count":7,"result_count":32,"score_min":0,"score_max":100,"score_decimals":2,"results":[[16723,1.7,370,16724,16725],[16723,6.8,370,16726,16725],[16723,4.0,370,16727,16725],[16723,2.8,370,16728,16725],[16729,0.0,370,16730,16725],[16729,5.1,370,16731,16725],[16729,4.0,370,16732,16725],[16729,6.2,370,16733,16725],[16734,2.8,370,16735,16725],[16734,9.0,370,16736,16725],[16734,11.3,370,16737,16725],[16734,11.3,370,16738,16725],[16739,5.1,370,16740,16725],[16739,12.4,370,16741,16725],[16739,14.1,370,16742,16725],[16739,19.8,370,16743,16725],[16744,2.8,370,16745,16725],[16744,2.8,370,16746,16725],[16744,7.3,370,16747,16725],[16744,8.5,370,16748,16725],[16749,3.4,370,16750,16725],[16749,4.5,370,16751,16725],[16749,6.2,370,16752,16725],[16749,7.9,370,16753,16725],[16754,49.2,370,16755,16725],[16754,47.5,370,16756,16725],[16754,57.1,370,16757,16725],[16754,54.2,370,16758,16725],[16759,49.7,370,16760,16725],[16759,61.0,370,16761,16725],[16759,56.5,370,16762,16725],[16759,48.6,370,16763,16725]]},{"id":"ricapbench-2026-paper","name":"RICapBench (2026 paper)","group":"视觉与视频","metric":"RelCap- F1 all","unit":"score","public_history":true,"protocol":{"en":"Primary-source results extracted from RICapBench (arXiv:2611.16413, table S5.T2, published 2026-11-12). Retains 15 published evaluation configurations across 13 distinct models on 'RelCap- F1 all' (native score scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《RICapBench》原始论文（arXiv:2611.16413，表 S5.T2，发布日期 2026-11-12）。完整保留“RelCap- F1 all”列下已发表的 15 组评测配置成绩（覆盖 13 个独立模型，采用原生分值口径（不计入准确率饱和度分档））；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.16413v1","published_at":"2026-11-12","status":"paper_baselines"},"model_count":13,"result_count":15,"score_min":0,"score_max":100,"score_decimals":2,"results":[[16764,8.38,370,16765,16766],[7967,6.4,370,16767,16766],[16768,5.38,370,16769,16766],[16770,2.61,370,16771,16766],[16772,2.39,370,16773,16766],[16764,10.0,370,16774,16766],[7967,7.91,370,16775,16766],[16776,4.44,370,16777,16766],[16778,14.03,370,16779,16766],[16780,1.18,370,16781,16766],[16782,2.43,370,16783,16766],[16784,1.85,370,16785,16766],[3157,1.81,370,16786,16766],[5617,0.72,370,16787,16766],[5519,1.41,370,16788,16766]]},{"id":"ai-risk-behaviors-2026-paper","name":"AI risk behaviors (2026 paper)","group":"综合","metric":"for an AI","unit":"%","public_history":true,"protocol":{"en":"Primary-source results extracted from AI risk behaviors (arXiv:2611.16318, table S4.T2, published 2026-11-12). Retains 9 published evaluation configurations across 9 distinct models on 'for an AI' (comparable percentage scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《AI risk behaviors》原始论文（arXiv:2611.16318，表 S4.T2，发布日期 2026-11-12）。完整保留“for an AI”列下已发表的 9 组评测配置成绩（覆盖 9 个独立模型，采用可比百分比口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.16318v1","published_at":"2026-11-12","status":"paper_baselines"},"model_count":9,"result_count":9,"results":[[5142,0.0948,370,16789,16790],[5107,0.095,370,16791,16790],[4972,0.095,370,16792,16790],[5148,0.0958,370,16793,16790],[16794,0.0986,370,16795,16790],[5144,0.095,370,16796,16790],[5475,0.095,370,16797,16790],[4952,0.0918,370,16798,16790],[5104,0.0958,370,16799,16790]]},{"id":"vibeprevisbench-2026-paper","name":"VibePrevisBench (2026 paper)","group":"视觉与视频","metric":"Overall","unit":"score","public_history":true,"protocol":{"en":"Primary-source results extracted from VibePrevisBench (arXiv:2611.15789, table S4.T2, published 2026-11-11). Retains 11 published evaluation configurations across 11 distinct models on 'Overall' (native score scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《VibePrevisBench》原始论文（arXiv:2611.15789，表 S4.T2，发布日期 2026-11-11）。完整保留“Overall”列下已发表的 11 组评测配置成绩（覆盖 11 个独立模型，采用原生分值口径（不计入准确率饱和度分档））；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.15789v1","published_at":"2026-11-11","status":"paper_baselines"},"model_count":11,"result_count":11,"score_min":0,"score_max":100,"score_decimals":2,"results":[[16800,77.7,370,16801,16802],[16803,76.1,370,16804,16802],[16805,75.0,370,16806,16802],[16807,74.1,370,16808,16802],[16809,70.2,370,16810,16802],[16811,69.6,370,16812,16802],[16813,67.5,370,16814,16802],[5329,64.7,370,16815,16802],[16816,63.7,370,16817,16802],[16818,61.3,370,16819,16802],[16820,61.1,370,16821,16802]]},{"id":"orbit-2026-paper","name":"Orbit (2026 paper)","group":"知识与推理","metric":"h","unit":"score","public_history":true,"protocol":{"en":"Primary-source results extracted from Orbit (arXiv:2611.15718, table A9.T9, published 2026-11-11). Retains 30 published evaluation configurations across 20 distinct models on 'h' (native score scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《Orbit》原始论文（arXiv:2611.15718，表 A9.T9，发布日期 2026-11-11）。完整保留“h”列下已发表的 30 组评测配置成绩（覆盖 20 个独立模型，采用原生分值口径（不计入准确率饱和度分档））；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.15718v1","published_at":"2026-11-11","status":"paper_baselines"},"model_count":20,"result_count":30,"score_min":0,"score_max":100,"score_decimals":2,"results":[[16822,1.16,370,16823,16824],[16825,1.59,370,16826,16824],[16827,0.43,370,16828,16824],[16829,0.92,370,16830,16824],[16831,1.69,370,16832,16824],[16833,0.77,370,16834,16824],[16835,0.89,370,16836,16824],[16837,1.28,370,16838,16824],[16839,0.39,370,16840,16824],[16841,0.59,370,16842,16824],[16843,1.53,370,16844,16824],[16845,0.94,370,16846,16824],[16847,0.88,370,16848,16824],[16849,1.62,370,16850,16824],[16851,0.74,370,16852,16824],[796,0.72,370,16853,16824],[4040,0.8,370,16854,16824],[362,0.68,370,16855,16824],[16856,0.7,370,16857,16824],[16858,0.74,370,16859,16824],[796,0.85,370,16860,16824],[4040,0.82,370,16861,16824],[362,0.76,370,16862,16824],[16856,0.72,370,16863,16824],[16858,0.78,370,16864,16824],[796,0.55,370,16865,16824],[4040,0.91,370,16866,16824],[362,0.54,370,16867,16824],[16856,1.11,370,16868,16824],[16858,0.88,370,16869,16824]]},{"id":"decarena-2026-paper","name":"DecArena (2026 paper)","group":"Agent 与工具","metric":"Out of context","unit":"score","public_history":true,"protocol":{"en":"Primary-source results extracted from DecArena (arXiv:2611.15521, table A4.T6, published 2026-11-11). Retains 54 published evaluation configurations across 8 distinct models on 'Out of context' (native score scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《DecArena》原始论文（arXiv:2611.15521，表 A4.T6，发布日期 2026-11-11）。完整保留“Out of context”列下已发表的 54 组评测配置成绩（覆盖 8 个独立模型，采用原生分值口径（不计入准确率饱和度分档））；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.15521v1","published_at":"2026-11-11","status":"paper_baselines"},"model_count":8,"result_count":54,"score_min":0,"score_max":100,"score_decimals":2,"results":[[16870,69.4,370,16871,16872],[2593,59.9,370,16873,16872],[5320,91.2,370,16874,16872],[6340,22.6,370,16875,16872],[16876,25.7,370,16877,16872],[16878,37.5,370,16879,16872],[16870,97.2,370,16880,16872],[2593,72.3,370,16881,16872],[5320,93.6,370,16882,16872],[6340,25.1,370,16883,16872],[16876,31.5,370,16884,16872],[16878,36.1,370,16885,16872],[16870,38.8,370,16886,16872],[2593,36.4,370,16887,16872],[5320,36.7,370,16888,16872],[6340,45.1,370,16889,16872],[16876,46.3,370,16890,16872],[16878,45.7,370,16891,16872],[16870,65.9,370,16892,16872],[2593,69.5,370,16893,16872],[5320,68.7,370,16894,16872],[16870,68.5,370,16895,16872],[2593,67.2,370,16896,16872],[5320,70.1,370,16897,16872],[16870,80.1,370,16898,16872],[2593,80.8,370,16899,16872],[5320,99.3,370,16900,16872],[16870,62.5,370,16901,16872],[2593,82.3,370,16902,16872],[5320,57.8,370,16903,16872],[16870,56.8,370,16904,16872],[2593,53.3,370,16905,16872],[5320,71.5,370,16906,16872],[16870,47.5,370,16907,16872],[2593,63.1,370,16908,16872],[5320,56.8,370,16909,16872],[16870,59.8,370,16910,16872],[2593,40.5,370,16911,16872],[5320,66.3,370,16912,16872],[6340,56.7,370,16913,16872],[16876,64.9,370,16914,16872],[16878,71.5,370,16915,16872],[6340,31.8,370,16916,16872],[5828,33.5,370,16917,16872],[4969,36.3,370,16918,16872],[6340,51.5,370,16919,16872],[5828,57.2,370,16920,16872],[4969,56.6,370,16921,16872],[6340,60.2,370,16922,16872],[5828,53.7,370,16923,16872],[4969,56.4,370,16924,16872],[6340,22.6,370,16925,16872],[16876,25.7,370,16926,16872],[16878,37.5,370,16927,16872]]},{"id":"verifier-choice-is-a-benchmark-2026-paper","name":"Verifier Choice is a Benchmark (2026 paper)","group":"视觉与视频","metric":"A all %","unit":"%","public_history":true,"protocol":{"en":"Primary-source results extracted from Verifier Choice is a Benchmark (arXiv:2611.15433, table A6.T7, published 2026-11-11). Retains 20 published evaluation configurations across 5 distinct models on 'A all %' (comparable percentage scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《Verifier Choice is a Benchmark》原始论文（arXiv:2611.15433，表 A6.T7，发布日期 2026-11-11）。完整保留“A all %”列下已发表的 20 组评测配置成绩（覆盖 5 个独立模型，采用可比百分比口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.15433v1","published_at":"2026-11-11","status":"paper_baselines"},"model_count":5,"result_count":20,"results":[[16928,0.8346,370,16929,16930],[16928,0.8496,370,16931,16930],[16928,0.8246,370,16932,16930],[16928,0.825,370,16933,16930],[8141,0.7053,370,16934,16930],[8141,0.6934,370,16935,16930],[8141,0.7645,370,16936,16930],[8141,0.7645,370,16937,16930],[2584,0.8566,370,16938,16930],[2584,0.8189,370,16939,16930],[2584,0.8526,370,16940,16930],[2584,0.7825,370,16941,16930],[4070,0.8474,370,16942,16930],[4070,0.8088,370,16943,16930],[4070,0.8482,370,16944,16930],[4070,0.8127,370,16945,16930],[4969,0.8303,370,16946,16930],[4969,0.8162,370,16947,16930],[4969,0.8439,370,16948,16930],[4969,0.8096,370,16949,16930]]},{"id":"mechbench-2026-paper","name":"MechBench (2026 paper)","group":"知识与推理","metric":"Open-ended","unit":"score","public_history":true,"protocol":{"en":"Primary-source results extracted from MechBench (arXiv:2611.15342, table S5.T3, published 2026-11-11). Retains 7 published evaluation configurations across 7 distinct models on 'Open-ended' (native score scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《MechBench》原始论文（arXiv:2611.15342，表 S5.T3，发布日期 2026-11-11）。完整保留“Open-ended”列下已发表的 7 组评测配置成绩（覆盖 7 个独立模型，采用原生分值口径（不计入准确率饱和度分档））；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.15342v1","published_at":"2026-11-11","status":"paper_baselines"},"model_count":7,"result_count":7,"score_min":0,"score_max":100,"score_decimals":2,"results":[[3196,96.7,370,16950,16951],[4979,97.8,370,16952,16951],[5727,95.2,370,16953,16951],[5343,73.1,370,16954,16951],[16955,65.7,370,16956,16951],[16957,40.8,370,16958,16951],[16959,60.7,370,16960,16951]]},{"id":"casescope-2026-paper","name":"CaseScope (2026 paper)","group":"医疗","metric":"CaseScope-Atlas All","unit":"%","public_history":true,"protocol":{"en":"Primary-source results extracted from CaseScope (arXiv:2611.15325, table S11.T8, published 2026-11-11). Retains 28 published evaluation configurations across 28 distinct models on 'CaseScope-Atlas All' (comparable percentage scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《CaseScope》原始论文（arXiv:2611.15325，表 S11.T8，发布日期 2026-11-11）。完整保留“CaseScope-Atlas All”列下已发表的 28 组评测配置成绩（覆盖 28 个独立模型，采用可比百分比口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.15325v1","published_at":"2026-11-11","status":"paper_baselines"},"model_count":28,"result_count":28,"results":[[2771,0.1,370,16961,16962],[2754,0.098,370,16963,16962],[6391,0.11,370,16964,16962],[2485,0.082,370,16965,16962],[2774,0.088,370,16966,16962],[2728,0.096,370,16967,16962],[2654,0.094,370,16968,16962],[6342,0.121,370,16969,16962],[4070,0.142,370,16970,16962],[6300,0.089,370,16971,16962],[2626,0.121,370,16972,16962],[5536,0.126,370,16973,16962],[6630,0.104,370,16974,16962],[6345,0.055,370,16975,16962],[8529,0.015,370,16976,16962],[16977,0.094,370,16978,16962],[8173,0.07,370,16979,16962],[16980,0.048,370,16981,16962],[16982,0.108,370,16983,16962],[4970,0.018,370,16984,16962],[8167,0.048,370,16985,16962],[8531,0.0,370,16986,16962],[16987,0.006,370,16988,16962],[16989,0.019,370,16990,16962],[6990,0.072,370,16991,16962],[10857,0.001,370,16992,16962],[10856,0.076,370,16993,16962],[16994,0.086,370,16995,16962]]},{"id":"acrossplace-2026-paper","name":"AcrossPlace (2026 paper)","group":"视觉与视频","metric":"I2T · R@10","unit":"%","public_history":true,"protocol":{"en":"Primary-source results extracted from AcrossPlace (arXiv:2611.15314, table S5.T3, published 2026-11-11). Retains 13 published evaluation configurations across 8 distinct models on 'I2T · R@10' (comparable percentage scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《AcrossPlace》原始论文（arXiv:2611.15314，表 S5.T3，发布日期 2026-11-11）。完整保留“I2T · R@10”列下已发表的 13 组评测配置成绩（覆盖 8 个独立模型，采用可比百分比口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.15314v1","published_at":"2026-11-11","status":"paper_baselines"},"model_count":8,"result_count":13,"results":[[16996,0.1274,370,16997,16998],[16999,0.1362,370,17000,16998],[17001,0.1604,370,17002,16998],[17003,0.1593,370,17004,16998],[17003,0.2401,370,17005,16998],[17006,0.3681,370,17007,16998],[17006,0.4079,370,17008,16998],[17009,0.528,370,17010,16998],[17009,0.4947,370,17011,16998],[17012,0.4569,370,17013,16998],[17012,0.4533,370,17014,16998],[17015,0.4145,370,17016,16998],[17015,0.4477,370,17017,16998]]},{"id":"castworld-2026-paper","name":"CastWorld (2026 paper)","group":"Agent 与工具","metric":"Versions","unit":"score","public_history":true,"protocol":{"en":"Primary-source results extracted from CastWorld (arXiv:2611.15262, table A1.T2, published 2026-11-11). Retains 15 published evaluation configurations across 15 distinct models on 'Versions' (native score scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《CastWorld》原始论文（arXiv:2611.15262，表 A1.T2，发布日期 2026-11-11）。完整保留“Versions”列下已发表的 15 组评测配置成绩（覆盖 15 个独立模型，采用原生分值口径（不计入准确率饱和度分档））；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.15262v1","published_at":"2026-11-11","status":"paper_baselines"},"model_count":15,"result_count":15,"score_min":0,"score_max":100,"score_decimals":2,"results":[[2958,1.22,370,17018,17019],[2960,1.408,370,17020,17019],[3201,1.064,370,17021,17019],[3490,1.1,370,17022,17019],[2983,1.004,370,17023,17019],[17024,1.124,370,17025,17019],[2959,1.076,370,17026,17019],[5791,1.428,370,17027,17019],[6886,1.056,370,17028,17019],[2961,1.04,370,17029,17019],[2584,1.036,370,17030,17019],[2976,1.072,370,17031,17019],[4969,1.004,370,17032,17019],[2582,1.02,370,17033,17019],[4070,1.044,370,17034,17019]]},{"id":"mvdrivebench-2026-paper","name":"MVDriveBench (2026 paper)","group":"知识与推理","metric":"Driving Reasoning Avg","unit":"score","public_history":true,"protocol":{"en":"Primary-source results extracted from MVDriveBench (arXiv:2611.15225, table S3.T2, published 2026-11-11). Retains 23 published evaluation configurations across 23 distinct models on 'Driving Reasoning Avg' (native score scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《MVDriveBench》原始论文（arXiv:2611.15225，表 S3.T2，发布日期 2026-11-11）。完整保留“Driving Reasoning Avg”列下已发表的 23 组评测配置成绩（覆盖 23 个独立模型，采用原生分值口径（不计入准确率饱和度分档））；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.15225v1","published_at":"2026-11-11","status":"paper_baselines"},"model_count":23,"result_count":23,"score_min":0,"score_max":100,"score_decimals":2,"results":[[3196,94.0,370,17035,17036],[2058,93.6,370,17037,17036],[353,91.5,370,17038,17036],[764,93.4,370,17039,17036],[6130,91.4,370,17040,17036],[2261,89.4,370,17041,17036],[2576,87.5,370,17042,17036],[17043,73.1,370,17044,17036],[5482,87.9,370,17045,17036],[2675,85.2,370,17046,17036],[2166,89.1,370,17047,17036],[2706,85.0,370,17048,17036],[6391,90.0,370,17049,17036],[2584,73.0,370,17050,17036],[5536,46.7,370,17051,17036],[7392,82.6,370,17052,17036],[17053,66.9,370,17054,17036],[17055,73.4,370,17056,17036],[17057,66.9,370,17058,17036],[8517,67.8,370,17059,17036],[7066,70.4,370,17060,17036],[17061,73.1,370,17062,17036],[2632,72.1,370,17063,17036]]},{"id":"rslivebench-uav-2026-paper","name":"RSLiveBench-UAV (2026 paper)","group":"视觉与视频","metric":"Reasoning · Mean","unit":"score","public_history":true,"protocol":{"en":"Primary-source results extracted from RSLiveBench-UAV (arXiv:2611.14953, table A3.T2, published 2026-11-11). Retains 30 published evaluation configurations across 30 distinct models on 'Reasoning · Mean' (native score scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《RSLiveBench-UAV》原始论文（arXiv:2611.14953，表 A3.T2，发布日期 2026-11-11）。完整保留“Reasoning · Mean”列下已发表的 30 组评测配置成绩（覆盖 30 个独立模型，采用原生分值口径（不计入准确率饱和度分档））；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.14953v1","published_at":"2026-11-11","status":"paper_baselines"},"model_count":30,"result_count":30,"score_min":0,"score_max":100,"score_decimals":2,"results":[[17064,0.0,370,17065,17066],[17067,0.89,370,17068,17066],[17069,9.82,370,17070,17066],[17071,6.04,370,17072,17066],[17073,14.37,370,17074,17066],[15276,9.18,370,17075,17066],[7555,8.88,370,17076,17066],[15271,10.46,370,17077,17066],[17078,12.55,370,17079,17066],[17080,13.39,370,17081,17066],[17082,16.4,370,17083,17066],[17084,15.07,370,17085,17066],[17086,18.3,370,17087,17066],[7066,25.05,370,17088,17066],[8177,20.09,370,17089,17066],[7556,18.3,370,17090,17066],[17091,17.93,370,17092,17066],[8179,25.52,370,17093,17066],[8868,22.24,370,17094,17066],[15266,14.75,370,17095,17066],[17096,13.7,370,17097,17066],[5828,7.48,370,17098,17066],[2654,10.82,370,17099,17066],[2687,15.98,370,17100,17066],[6150,18.9,370,17101,17066],[4070,19.1,370,17102,17066],[17103,19.33,370,17104,17066],[17105,24.29,370,17106,17066],[17107,38.61,370,17108,17066],[17109,38.81,370,17110,17066]]},{"id":"joyfrontbench-2026-paper","name":"JoyFrontBench (2026 paper)","group":"编程","metric":"All pass","unit":"%","public_history":true,"protocol":{"en":"Primary-source results extracted from JoyFrontBench (arXiv:2611.14952, table S4.T1, published 2026-11-11). Retains 5 published evaluation configurations across 5 distinct models on 'All pass' (comparable percentage scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《JoyFrontBench》原始论文（arXiv:2611.14952，表 S4.T1，发布日期 2026-11-11）。完整保留“All pass”列下已发表的 5 组评测配置成绩（覆盖 5 个独立模型，采用可比百分比口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.14952v1","published_at":"2026-11-11","status":"paper_baselines"},"model_count":5,"result_count":5,"results":[[2982,0.386,370,17111,17112],[2977,0.2,370,17113,17112],[762,0.4,370,17114,17112],[3404,0.214,370,17115,17112],[3154,0.143,370,17116,17112]]},{"id":"toolturns-2026-paper","name":"ToolTurns (2026 paper)","group":"Agent 与工具","metric":"FU 2","unit":"%","public_history":true,"protocol":{"en":"Primary-source results extracted from ToolTurns (arXiv:2611.14925, table S4.T4, published 2026-11-11). Retains 9 published evaluation configurations across 9 distinct models on 'FU 2' (comparable percentage scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《ToolTurns》原始论文（arXiv:2611.14925，表 S4.T4，发布日期 2026-11-11）。完整保留“FU 2”列下已发表的 9 组评测配置成绩（覆盖 9 个独立模型，采用可比百分比口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.14925v1","published_at":"2026-11-11","status":"paper_baselines"},"model_count":9,"result_count":9,"results":[[354,0.503,370,17117,17118],[5406,0.62,370,17119,17118],[4958,0.555,370,17120,17118],[3401,0.686,370,17121,17118],[3191,0.575,370,17122,17118],[3763,0.601,370,17123,17118],[7148,0.635,370,17124,17118],[3419,0.49,370,17125,17118],[17126,0.648,370,17127,17118]]},{"id":"mlirbench-2026-paper","name":"MLIRBench (2026 paper)","group":"编程","metric":"L5","unit":"score","public_history":true,"protocol":{"en":"Primary-source results extracted from MLIRBench (arXiv:2611.14881, table S5.T4, published 2026-11-11). Retains 9 published evaluation configurations across 9 distinct models on 'L5' (native score scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《MLIRBench》原始论文（arXiv:2611.14881，表 S5.T4，发布日期 2026-11-11）。完整保留“L5”列下已发表的 9 组评测配置成绩（覆盖 9 个独立模型，采用原生分值口径（不计入准确率饱和度分档））；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.14881v1","published_at":"2026-11-11","status":"paper_baselines"},"model_count":9,"result_count":9,"score_min":0,"score_max":100,"score_decimals":2,"results":[[3412,94.94,370,17128,17129],[283,59.11,370,17130,17129],[6278,52.59,370,17131,17129],[3490,51.67,370,17132,17129],[325,53.57,370,17133,17129],[3489,32.78,370,17134,17129],[12364,26.88,370,17135,17129],[11961,21.18,370,17136,17129],[17137,18.15,370,17138,17129]]},{"id":"multilingual-document-understanding-2026-paper","name":"Multilingual Document Understanding (2026 paper)","group":"指令与长文","metric":"AVG","unit":"score","public_history":true,"protocol":{"en":"Primary-source results extracted from Multilingual Document Understanding (arXiv:2611.14758, table S4.T2, published 2026-11-10). Retains 23 published evaluation configurations across 23 distinct models on 'AVG' (native score scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《Multilingual Document Understanding》原始论文（arXiv:2611.14758，表 S4.T2，发布日期 2026-11-10）。完整保留“AVG”列下已发表的 23 组评测配置成绩（覆盖 23 个独立模型，采用原生分值口径（不计入准确率饱和度分档））；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.14758v1","published_at":"2026-11-10","status":"paper_baselines"},"model_count":23,"result_count":23,"score_min":0,"score_max":100,"score_decimals":2,"results":[[17139,40.6,370,17140,17141],[2639,46.1,370,17142,17141],[17143,35.4,370,17144,17141],[17145,35.5,370,17146,17141],[17147,50.9,370,17148,17141],[17149,52.7,370,17150,17141],[17151,49.9,370,17152,17141],[16603,47.4,370,17153,17141],[17154,35.3,370,17155,17141],[17156,35.0,370,17157,17141],[17158,56.1,370,17159,17141],[17160,57.6,370,17161,17141],[17162,65.4,370,17163,17141],[16607,62.1,370,17164,17141],[17165,46.8,370,17166,17141],[17167,51.4,370,17168,17141],[17169,68.1,370,17170,17141],[17171,69.0,370,17172,17141],[2614,59.2,370,17173,17141],[17174,55.8,370,17175,17141],[17176,66.8,370,17177,17141],[17178,66.3,370,17179,17141],[17180,65.8,370,17181,17141]]},{"id":"tau2-bench-2026-paper","name":"tau2-bench (2026 paper)","group":"Agent 与工具","metric":"Overall","unit":"score","public_history":true,"protocol":{"en":"Primary-source results extracted from tau2-bench (arXiv:2611.14581, table S10.T3, published 2026-11-10). Retains 8 published evaluation configurations across 8 distinct models on 'Overall' (native score scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《tau2-bench》原始论文（arXiv:2611.14581，表 S10.T3，发布日期 2026-11-10）。完整保留“Overall”列下已发表的 8 组评测配置成绩（覆盖 8 个独立模型，采用原生分值口径（不计入准确率饱和度分档））；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.14581v1","published_at":"2026-11-10","status":"paper_baselines"},"model_count":8,"result_count":8,"score_min":0,"score_max":1,"score_decimals":3,"results":[[17182,0.502,370,17183,17184],[17185,0.513,370,17186,17184],[17187,0.543,370,17188,17184],[17189,0.542,370,17190,17184],[17191,0.685,370,17192,17184],[17193,0.647,370,17194,17184],[17195,0.664,370,17196,17184],[17197,0.766,370,17198,17184]]},{"id":"roborisk-2026-paper","name":"RoboRisk (2026 paper)","group":"视觉与视频","metric":"RoboRisk-R · SSR","unit":"%","public_history":true,"protocol":{"en":"Primary-source results extracted from RoboRisk (arXiv:2611.14115, table S5.T2, published 2026-11-10). Retains 10 published evaluation configurations across 5 distinct models on 'RoboRisk-R · SSR' (comparable percentage scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《RoboRisk》原始论文（arXiv:2611.14115，表 S5.T2，发布日期 2026-11-10）。完整保留“RoboRisk-R · SSR”列下已发表的 10 组评测配置成绩（覆盖 5 个独立模型，采用可比百分比口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.14115v1","published_at":"2026-11-10","status":"paper_baselines"},"model_count":5,"result_count":10,"results":[[17199,0.6923,370,17200,17201],[17199,0.8462,370,17202,17201],[17203,0.4615,370,17204,17201],[17203,0.8462,370,17205,17201],[17206,0.5385,370,17207,17201],[17206,0.8462,370,17208,17201],[17209,0.6154,370,17210,17201],[17209,0.8462,370,17211,17201],[17212,0.2308,370,17213,17201],[17212,0.5385,370,17214,17201]]},{"id":"the-narrative-gap-2026-paper","name":"The Narrative Gap (2026 paper)","group":"知识与推理","metric":"Output cap","unit":"raw","public_history":true,"protocol":{"en":"Primary-source results extracted from The Narrative Gap (arXiv:2611.12750, table A2.T6, published 2026-11-09). Retains 15 published evaluation configurations across 14 distinct models on 'Output cap' (raw reported metric scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《The Narrative Gap》原始论文（arXiv:2611.12750，表 A2.T6，发布日期 2026-11-09）。完整保留“Output cap”列下已发表的 15 组评测配置成绩（覆盖 14 个独立模型，采用原始指标口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.12750v1","published_at":"2026-11-09","status":"paper_baselines"},"model_count":14,"result_count":15,"score_decimals":2,"results":[[4281,3000.0,370,17215,17216],[5150,3000.0,370,17217,17216],[17218,3000.0,370,17219,17216],[17220,3000.0,370,17221,17216],[17222,16384.0,370,17223,17216],[457,3000.0,370,17224,17216],[4164,3000.0,370,17225,17216],[5144,3000.0,370,17226,17216],[17227,3000.0,370,17228,17216],[17229,3000.0,370,17230,17216],[11303,3000.0,370,17231,17216],[11288,3000.0,370,17232,17216],[17233,3000.0,370,17234,17216],[420,3000.0,370,17235,17216],[17236,3000.0,370,17237,17216]]},{"id":"do-underwater-image-quality-metrics-2026-paper","name":"Do Underwater Image Quality Metrics (2026 paper)","group":"视觉与视频","metric":"Noise","unit":"%","public_history":true,"protocol":{"en":"Primary-source results extracted from Do Underwater Image Quality Metrics (arXiv:2611.18429, table A3.T8, published 2026-11-13). Retains 71 published evaluation configurations across 71 distinct models on 'Noise' (comparable percentage scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《Do Underwater Image Quality Metrics》原始论文（arXiv:2611.18429，表 A3.T8，发布日期 2026-11-13）。完整保留“Noise”列下已发表的 71 组评测配置成绩（覆盖 71 个独立模型，采用可比百分比口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.18429v1","published_at":"2026-11-13","status":"paper_baselines"},"model_count":71,"result_count":71,"results":[[17238,0.12,370,17239,17240],[17241,0.05,370,17242,17240],[17243,0.12,370,17244,17240],[17245,0.07,370,17246,17240],[17247,0.11,370,17248,17240],[17249,0.12,370,17250,17240],[17251,0.05,370,17252,17240],[17253,0.25,370,17254,17240],[17255,0.63,370,17256,17240],[17257,0.09,370,17258,17240],[17259,0.2,370,17260,17240],[17261,0.59,370,17262,17240],[9956,0.47,370,17263,17240],[9957,0.1,370,17264,17240],[17265,0.28,370,17266,17240],[17267,0.12,370,17268,17240],[17269,0.24,370,17270,17240],[9953,0.28,370,17271,17240],[17272,0.18,370,17273,17240],[17274,0.86,370,17275,17240],[17276,0.02,370,17277,17240],[17278,0.04,370,17279,17240],[17280,0.22,370,17281,17240],[17282,0.04,370,17283,17240],[17284,0.28,370,17285,17240],[17286,0.06,370,17287,17240],[17288,0.65,370,17289,17240],[17290,0.01,370,17291,17240],[17292,0.2,370,17293,17240],[9955,0.05,370,17294,17240],[17295,0.16,370,17296,17240],[17297,0.12,370,17298,17240],[17299,0.12,370,17300,17240],[17301,0.23,370,17302,17240],[17303,0.02,370,17304,17240],[17305,0.13,370,17306,17240],[17307,0.3,370,17308,17240],[17309,0.0,370,17310,17240],[17311,0.0,370,17312,17240],[17313,0.16,370,17314,17240],[17315,0.23,370,17316,17240],[17317,0.52,370,17318,17240],[17319,0.25,370,17320,17240],[17321,0.02,370,17322,17240],[17323,0.11,370,17324,17240],[9144,0.04,370,17325,17240],[17326,0.06,370,17327,17240],[17328,0.02,370,17329,17240],[17330,0.24,370,17331,17240],[17332,0.07,370,17333,17240],[17334,0.38,370,17335,17240],[17336,0.74,370,17337,17240],[17338,0.01,370,17339,17240],[17340,0.23,370,17341,17240],[17342,0.13,370,17343,17240],[17344,0.05,370,17345,17240],[17346,0.14,370,17347,17240],[17348,0.13,370,17349,17240],[17350,0.0,370,17351,17240],[17352,0.18,370,17353,17240],[17354,0.34,370,17355,17240],[17356,0.72,370,17357,17240],[17358,0.68,370,17359,17240],[17360,0.37,370,17361,17240],[17362,0.51,370,17363,17240],[17364,0.41,370,17365,17240],[17366,0.68,370,17367,17240],[17368,0.15,370,17369,17240],[17370,0.02,370,17371,17240],[11796,0.05,370,17372,17240],[17373,0.38,370,17374,17240]]},{"id":"jfinbench-2026-paper","name":"JFinBench (2026 paper)","group":"专业工作","metric":"Score","unit":"%","public_history":true,"protocol":{"en":"Primary-source results extracted from JFinBench (arXiv:2611.18130, table S5.T3, published 2026-11-13). Retains 12 published evaluation configurations across 12 distinct models on 'Score' (comparable percentage scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《JFinBench》原始论文（arXiv:2611.18130，表 S5.T3，发布日期 2026-11-13）。完整保留“Score”列下已发表的 12 组评测配置成绩（覆盖 12 个独立模型，采用可比百分比口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.18130v1","published_at":"2026-11-13","status":"paper_baselines"},"model_count":12,"result_count":12,"results":[[17375,0.958,370,17376,17377],[15956,0.936,370,15957,17377],[17378,0.888,370,17379,17377],[15977,0.834,370,15978,17377],[16093,0.799,370,17380,17377],[17381,0.731,370,17382,17377],[17383,0.625,370,17384,17377],[17385,0.621,370,17386,17377],[17387,0.591,370,17388,17377],[17389,0.582,370,17390,17377],[17391,0.577,370,17392,17377],[17393,0.509,370,17394,17377]]},{"id":"g-cgec-eval-2026-paper","name":"G-CGEC-Eval (2026 paper)","group":"Agent 与工具","metric":"𝐅 𝟏","unit":"%","public_history":true,"protocol":{"en":"Primary-source results extracted from G-CGEC-Eval (arXiv:2611.17966, table A2.T5, published 2026-11-13). Retains 16 published evaluation configurations across 6 distinct models on '𝐅 𝟏' (comparable percentage scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《G-CGEC-Eval》原始论文（arXiv:2611.17966，表 A2.T5，发布日期 2026-11-13）。完整保留“𝐅 𝟏”列下已发表的 16 组评测配置成绩（覆盖 6 个独立模型，采用可比百分比口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.17966v1","published_at":"2026-11-13","status":"paper_baselines"},"model_count":6,"result_count":16,"results":[[3412,0.0495,370,17395,17396],[2576,0.0265,370,17397,17396],[2584,0.0085,370,17398,17396],[3196,0.0022,370,17399,17396],[7223,0.0018,370,17400,17396],[3412,0.1823,370,17401,17396],[7223,0.1401,370,17402,17396],[3763,0.1368,370,17403,17396],[2584,0.1348,370,17404,17396],[3196,0.1237,370,17405,17396],[3196,0.2283,370,17406,17396],[7223,0.1817,370,17407,17396],[3763,0.1031,370,17408,17396],[2584,0.052,370,17409,17396],[3196,0.1461,370,17410,17396],[3196,0.2097,370,17411,17396]]},{"id":"webarena-pro-2026-paper","name":"WebArena-Pro (2026 paper)","group":"Agent 与工具","metric":"Overall SR","unit":"%","public_history":true,"protocol":{"en":"Primary-source results extracted from WebArena-Pro (arXiv:2611.17955, table S4.T5, published 2026-11-13). Retains 13 published evaluation configurations across 13 distinct models on 'Overall SR' (comparable percentage scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《WebArena-Pro》原始论文（arXiv:2611.17955，表 S4.T5，发布日期 2026-11-13）。完整保留“Overall SR”列下已发表的 13 组评测配置成绩（覆盖 13 个独立模型，采用可比百分比口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.17955v1","published_at":"2026-11-13","status":"paper_baselines"},"model_count":13,"result_count":13,"results":[[5672,0.05,370,17412,17413],[4995,0.13,370,17414,17413],[3403,0.157,370,17415,17413],[17416,0.223,370,17417,17413],[7950,0.303,370,17418,17413],[3497,0.237,370,17419,17413],[17420,0.303,370,17421,17413],[3578,0.253,370,17422,17413],[3412,0.48,370,17423,17413],[17424,0.557,370,17425,17413],[17426,0.607,370,17427,17413],[5506,0.623,370,17428,17413],[2959,0.623,370,17429,17413]]},{"id":"outfitterbench-2026-paper","name":"OutfitterBench (2026 paper)","group":"Agent 与工具","metric":"g = 7 Mean gain","unit":"score","public_history":true,"protocol":{"en":"Primary-source results extracted from OutfitterBench (arXiv:2611.17933, table A4.T10, published 2026-11-13). Retains 14 published evaluation configurations across 7 distinct models on 'g = 7 Mean gain' (native score scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《OutfitterBench》原始论文（arXiv:2611.17933，表 A4.T10，发布日期 2026-11-13）。完整保留“g = 7 Mean gain”列下已发表的 14 组评测配置成绩（覆盖 7 个独立模型，采用原生分值口径（不计入准确率饱和度分档））；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.17933v1","published_at":"2026-11-13","status":"paper_baselines"},"model_count":7,"result_count":14,"score_min":0,"score_max":1,"score_decimals":3,"results":[[17430,0.064,370,17431,17432],[17433,0.667,370,17434,17432],[17435,0.453,370,17436,17432],[17437,0.227,370,17438,17432],[3296,0.623,370,17439,17432],[12316,0.302,370,17440,17432],[16412,0.184,370,17441,17432],[17430,0.052,370,17442,17432],[17433,0.775,370,17443,17432],[17435,0.0,370,17444,17432],[17437,0.186,370,17445,17432],[3296,0.19,370,17446,17432],[12316,0.316,370,17447,17432],[16412,0.0,370,17448,17432]]},{"id":"mineralexp-2026-paper","name":"MineralExp (2026 paper)","group":"Agent 与工具","metric":"Overall · F1","unit":"score","public_history":true,"protocol":{"en":"Primary-source results extracted from MineralExp (arXiv:2611.17931, table S5.T3, published 2026-11-13). Retains 28 published evaluation configurations across 7 distinct models on 'Overall · F1' (native score scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《MineralExp》原始论文（arXiv:2611.17931，表 S5.T3，发布日期 2026-11-13）。完整保留“Overall · F1”列下已发表的 28 组评测配置成绩（覆盖 7 个独立模型，采用原生分值口径（不计入准确率饱和度分档））；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.17931v1","published_at":"2026-11-13","status":"paper_baselines"},"model_count":7,"result_count":28,"score_min":0,"score_max":100,"score_decimals":2,"results":[[2960,63.84,370,17449,17450],[2976,50.53,370,17451,17450],[2961,60.42,370,17452,17450],[2970,53.87,370,17453,17450],[3153,46.77,370,17454,17450],[5300,55.78,370,17455,17450],[17456,50.22,370,17457,17450],[2960,60.73,370,17458,17450],[2976,49.36,370,17459,17450],[2961,60.51,370,17460,17450],[2970,56.81,370,17461,17450],[3153,46.77,370,17462,17450],[5300,53.43,370,17463,17450],[17456,48.87,370,17464,17450],[2960,59.9,370,17465,17450],[2976,48.13,370,17466,17450],[2961,60.75,370,17467,17450],[2970,51.98,370,17468,17450],[3153,46.2,370,17469,17450],[5300,52.23,370,17470,17450],[17456,47.65,370,17471,17450],[2960,62.06,370,17472,17450],[2976,54.1,370,17473,17450],[2961,59.96,370,17474,17450],[2970,53.63,370,17475,17450],[3153,47.5,370,17476,17450],[5300,54.73,370,17477,17450],[17456,50.11,370,17478,17450]]},{"id":"omniscore-bench-2026-paper","name":"OmniScore-Bench (2026 paper)","group":"视觉与视频","metric":"NED","unit":"score","public_history":true,"protocol":{"en":"Primary-source results extracted from OmniScore-Bench (arXiv:2611.17870, table S4.T9, published 2026-11-13). Retains 48 published evaluation configurations across 12 distinct models on 'NED' (native score scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《OmniScore-Bench》原始论文（arXiv:2611.17870，表 S4.T9，发布日期 2026-11-13）。完整保留“NED”列下已发表的 48 组评测配置成绩（覆盖 12 个独立模型，采用原生分值口径（不计入准确率饱和度分档））；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.17870v1","published_at":"2026-11-13","status":"paper_baselines"},"model_count":12,"result_count":48,"score_min":0,"score_max":100,"score_decimals":2,"higher_is_better":false,"results":[[3512,89.6,370,17479,17480],[3595,90.8,370,17481,17480],[3560,81.1,370,17482,17480],[2977,71.7,370,17483,17480],[2967,64.4,370,17484,17480],[3529,89.6,370,17485,17480],[2956,64.4,370,17486,17480],[2959,46.3,370,17487,17480],[2983,84.4,370,17488,17480],[2962,46.2,370,17489,17480],[2960,28.0,370,17490,17480],[2958,23.5,370,17491,17480],[3512,86.1,370,17492,17480],[3595,89.9,370,17493,17480],[3560,81.1,370,17494,17480],[2977,69.3,370,17495,17480],[2967,64.1,370,17496,17480],[3529,86.9,370,17497,17480],[2956,69.2,370,17498,17480],[2959,55.1,370,17499,17480],[2983,79.8,370,17500,17480],[2962,29.0,370,17501,17480],[2960,23.5,370,17502,17480],[2958,21.4,370,17503,17480],[3512,86.4,370,17504,17480],[3595,91.8,370,17505,17480],[3560,76.1,370,17506,17480],[2977,70.9,370,17507,17480],[2967,62.8,370,17508,17480],[3529,83.7,370,17509,17480],[2956,59.3,370,17510,17480],[2959,43.5,370,17511,17480],[2983,85.7,370,17512,17480],[2962,52.0,370,17513,17480],[2960,30.5,370,17514,17480],[2958,26.8,370,17515,17480],[3512,89.0,370,17516,17480],[3595,96.1,370,17517,17480],[3560,76.2,370,17518,17480],[2977,74.5,370,17519,17480],[2967,67.0,370,17520,17480],[3529,93.1,370,17521,17480],[2956,77.1,370,17522,17480],[2959,39.9,370,17523,17480],[2983,98.0,370,17524,17480],[2962,88.3,370,17525,17480],[2960,55.2,370,17526,17480],[2958,50.6,370,17527,17480]]},{"id":"octave-2026-paper","name":"OCTAVE (2026 paper)","group":"编程","metric":"Richness","unit":"score","public_history":true,"protocol":{"en":"Primary-source results extracted from OCTAVE (arXiv:2611.17681, table S4.T6, published 2026-11-13). Retains 11 published evaluation configurations across 11 distinct models on 'Richness' (native score scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《OCTAVE》原始论文（arXiv:2611.17681，表 S4.T6，发布日期 2026-11-13）。完整保留“Richness”列下已发表的 11 组评测配置成绩（覆盖 11 个独立模型，采用原生分值口径（不计入准确率饱和度分档））；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.17681v1","published_at":"2026-11-13","status":"paper_baselines"},"model_count":11,"result_count":11,"score_min":0,"score_max":100,"score_decimals":2,"results":[[3153,27.6,370,17528,17529],[8423,26.9,370,17530,17529],[3151,36.4,370,17531,17529],[3149,38.2,370,17532,17529],[227,25.8,370,17533,17529],[3154,31.8,370,17534,17529],[3156,38.5,370,17535,17529],[3185,25.3,370,17536,17529],[3196,27.9,370,17537,17529],[4048,31.5,370,17538,17529],[5404,47.6,370,17539,17529]]},{"id":"aligned-conflicting-or-unspecified-2026-paper","name":"Aligned, Conflicting, or Unspecified? (2026 paper)","group":"医疗","metric":"w/ text","unit":"%","public_history":true,"protocol":{"en":"Primary-source results extracted from Aligned, Conflicting, or Unspecified? (arXiv:2611.17680, table S1.T1, published 2026-11-13). Retains 10 published evaluation configurations across 10 distinct models on 'w/ text' (comparable percentage scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《Aligned, Conflicting, or Unspecified?》原始论文（arXiv:2611.17680，表 S1.T1，发布日期 2026-11-13）。完整保留“w/ text”列下已发表的 10 组评测配置成绩（覆盖 10 个独立模型，采用可比百分比口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.17680v1","published_at":"2026-11-13","status":"paper_baselines"},"model_count":10,"result_count":10,"results":[[2960,0.6556,370,17540,17541],[2976,0.5576,370,17542,17541],[2961,0.569,370,17543,17541],[2970,0.5097,370,17544,17541],[5344,0.4527,370,17545,17541],[17546,0.4493,370,17547,17541],[9458,0.4846,370,17548,17541],[17549,0.4128,370,17550,17541],[17551,0.561,370,17552,17541],[17553,0.5485,370,17554,17541]]},{"id":"benchmarks-don-t-talk-back-2026-paper","name":"Benchmarks Don't Talk Back (2026 paper)","group":"知识与推理","metric":"SCoRe","unit":"%","public_history":true,"protocol":{"en":"Primary-source results extracted from Benchmarks Don't Talk Back (arXiv:2611.17563, table S4.T1, published 2026-11-12). Retains 14 published evaluation configurations across 12 distinct models on 'SCoRe' (comparable percentage scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《Benchmarks Don't Talk Back》原始论文（arXiv:2611.17563，表 S4.T1，发布日期 2026-11-12）。完整保留“SCoRe”列下已发表的 14 组评测配置成绩（覆盖 12 个独立模型，采用可比百分比口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.17563v1","published_at":"2026-11-12","status":"paper_baselines"},"model_count":12,"result_count":14,"results":[[17555,0.985,370,17556,17557],[17558,0.229,370,17559,17557],[17560,0.415,370,17561,17557],[17562,0.722,370,17563,17557],[17564,0.781,370,17565,17557],[17566,0.537,370,17567,17557],[17568,0.37,370,17569,17557],[17570,0.376,370,17571,17557],[17572,0.375,370,17573,17557],[17574,0.315,370,17575,17557],[5408,0.516,370,17576,17557],[6214,0.493,370,17577,17557],[5408,0.521,370,17578,17557],[6214,0.446,370,17579,17557]]},{"id":"talk-to-act-2026-paper","name":"Talk to Act (2026 paper)","group":"视觉与视频","metric":"Acc.","unit":"%","public_history":true,"protocol":{"en":"Primary-source results extracted from Talk to Act (arXiv:2611.17549, table S4.T1, published 2026-11-12). Retains 5 published evaluation configurations across 5 distinct models on 'Acc.' (comparable percentage scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《Talk to Act》原始论文（arXiv:2611.17549，表 S4.T1，发布日期 2026-11-12）。完整保留“Acc.”列下已发表的 5 组评测配置成绩（覆盖 5 个独立模型，采用可比百分比口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.17549v1","published_at":"2026-11-12","status":"paper_baselines"},"model_count":5,"result_count":5,"results":[[8777,0.362,370,17580,17581],[17582,0.488,370,17583,17581],[17584,0.59,370,17585,17581],[17586,0.357,370,17587,17581],[17588,0.487,370,17589,17581]]},{"id":"unfinished-bench-2026-paper","name":"Unfinished-Bench (2026 paper)","group":"Agent 与工具","metric":"Om.","unit":"score","public_history":true,"protocol":{"en":"Primary-source results extracted from Unfinished-Bench (arXiv:2611.17460, table S4.T4, published 2026-11-12). Retains 12 published evaluation configurations across 12 distinct models on 'Om.' (native score scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《Unfinished-Bench》原始论文（arXiv:2611.17460，表 S4.T4，发布日期 2026-11-12）。完整保留“Om.”列下已发表的 12 组评测配置成绩（覆盖 12 个独立模型，采用原生分值口径（不计入准确率饱和度分档））；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.17460v1","published_at":"2026-11-12","status":"paper_baselines"},"model_count":12,"result_count":12,"score_min":0,"score_max":1,"score_decimals":3,"results":[[62,1.0,370,17590,17591],[4906,0.0,370,17592,17591],[2585,0.0,370,17593,17591],[8494,0.0,370,17594,17591],[5320,1.0,370,17595,17591],[6585,0.0,370,17596,17591],[3492,0.0,370,17597,17591],[17598,0.0,370,17599,17591],[3403,0.0,370,17600,17591],[3529,0.0,370,17601,17591],[3512,0.0,370,17602,17591],[3515,0.0,370,17603,17591]]},{"id":"parsebench-2-2026-paper","name":"ParseBench-2 (2026 paper)","group":"视觉与视频","metric":"Pages limited by the cap (%) All","unit":"%","public_history":true,"protocol":{"en":"Primary-source results extracted from ParseBench-2 (arXiv:2611.17400, table A3.T18, published 2026-11-12). Retains 35 published evaluation configurations across 35 distinct models on 'Pages limited by the cap (%) All' (comparable percentage scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《ParseBench-2》原始论文（arXiv:2611.17400，表 A3.T18，发布日期 2026-11-12）。完整保留“Pages limited by the cap (%) All”列下已发表的 35 组评测配置成绩（覆盖 35 个独立模型，采用可比百分比口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.17400v1","published_at":"2026-11-12","status":"paper_baselines"},"model_count":35,"result_count":35,"results":[[17604,0.42,370,17605,17606],[17607,0.51,370,17608,17606],[17609,0.52,370,17610,17606],[17611,0.51,370,17612,17606],[17613,0.51,370,17614,17606],[17615,0.5,370,17616,17606],[17617,0.5,370,17618,17606],[17619,0.49,370,17620,17606],[17621,0.5,370,17622,17606],[17623,0.53,370,17624,17606],[17625,0.54,370,17626,17606],[17627,0.41,370,17628,17606],[17629,0.53,370,17630,17606],[17631,0.4,370,17632,17606],[8230,0.44,370,17633,17606],[17634,0.39,370,17635,17606],[17636,0.39,370,17637,17606],[17638,0.42,370,17639,17606],[17640,0.41,370,17641,17606],[2959,0.36,370,17642,17606],[17643,0.43,370,17644,17606],[17645,0.57,370,17646,17606],[17647,0.46,370,17648,17606],[17649,0.39,370,17650,17606],[17651,0.39,370,17652,17606],[3415,0.37,370,17653,17606],[3405,0.37,370,17654,17606],[17655,0.44,370,17656,17606],[17657,0.37,370,17658,17606],[2961,0.37,370,17659,17606],[17660,0.45,370,17661,17606],[2970,0.39,370,17662,17606],[17663,0.47,370,17664,17606],[4219,0.38,370,17665,17606],[4035,0.41,370,17666,17606]]},{"id":"omni-workspace-2026-paper","name":"Omni-Workspace (2026 paper)","group":"Agent 与工具","metric":"Avg.","unit":"score","public_history":true,"protocol":{"en":"Primary-source results extracted from Omni-Workspace (arXiv:2611.19202, table S3.T1, published 2026-11-14). Retains 18 published evaluation configurations across 18 distinct models on 'Avg.' (native score scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《Omni-Workspace》原始论文（arXiv:2611.19202，表 S3.T1，发布日期 2026-11-14）。完整保留“Avg.”列下已发表的 18 组评测配置成绩（覆盖 18 个独立模型，采用原生分值口径（不计入准确率饱和度分档））；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.19202v1","published_at":"2026-11-14","status":"paper_baselines"},"model_count":18,"result_count":18,"score_min":0,"score_max":100,"score_decimals":2,"results":[[3415,65.93,370,17667,17668],[3482,59.89,370,17669,17668],[4033,55.62,370,17670,17668],[2602,48.35,370,17671,17668],[7223,46.44,370,17672,17668],[17673,39.38,370,17674,17668],[17675,25.97,370,17676,17668],[8381,13.52,370,17677,17668],[8551,8.01,370,17678,17668],[7737,5.63,370,17679,17668],[4961,64.78,370,17680,17668],[3189,58.34,370,17681,17668],[3490,55.16,370,17682,17668],[3198,52.13,370,17683,17668],[2582,51.27,370,17684,17668],[2590,42.41,370,17685,17668],[5485,20.25,370,17686,17668],[6150,15.46,370,17687,17668]]},{"id":"do-vlms-really-look-at-2026-paper","name":"Do VLMs Really Look at (2026 paper)","group":"视觉与视频","metric":"Counterfactual SCS (%) Overall","unit":"%","public_history":true,"protocol":{"en":"Primary-source results extracted from Do VLMs Really Look at (arXiv:2611.19184, table S3.T3, published 2026-11-14). Retains 6 published evaluation configurations across 6 distinct models on 'Counterfactual SCS (%) Overall' (comparable percentage scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《Do VLMs Really Look at》原始论文（arXiv:2611.19184，表 S3.T3，发布日期 2026-11-14）。完整保留“Counterfactual SCS (%) Overall”列下已发表的 6 组评测配置成绩（覆盖 6 个独立模型，采用可比百分比口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.19184v1","published_at":"2026-11-14","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[17688,0.134,370,17689,17690],[17691,0.393,370,17692,17690],[17693,0.281,370,17694,17690],[17695,0.326,370,17696,17690],[2832,0.279,370,17697,17690],[2824,0.342,370,17698,17690]]},{"id":"raven-bench-2026-paper","name":"RAVEN-Bench (2026 paper)","group":"视觉与视频","metric":"S coh","unit":"score","public_history":true,"protocol":{"en":"Primary-source results extracted from RAVEN-Bench (arXiv:2611.18880, table S5.T2, published 2026-11-13). Retains 26 published evaluation configurations across 26 distinct models on 'S coh' (native score scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《RAVEN-Bench》原始论文（arXiv:2611.18880，表 S5.T2，发布日期 2026-11-13）。完整保留“S coh”列下已发表的 26 组评测配置成绩（覆盖 26 个独立模型，采用原生分值口径（不计入准确率饱和度分档））；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.18880v1","published_at":"2026-11-13","status":"paper_baselines"},"model_count":26,"result_count":26,"score_min":0,"score_max":100,"score_decimals":2,"results":[[3184,22.8,370,17699,17700],[3412,30.2,370,17701,17700],[3424,20.3,370,17702,17700],[17703,8.3,370,17704,17700],[6245,13.8,370,17705,17700],[17706,14.5,370,17707,17700],[17708,17.1,370,17709,17700],[6152,18.2,370,17710,17700],[6150,19.7,370,17711,17700],[6144,16.2,370,17712,17700],[5995,19.5,370,17713,17700],[5725,15.1,370,17714,17700],[8136,21.5,370,17715,17700],[17716,17.8,370,17717,17700],[6349,21.9,370,17718,17700],[17719,23.8,370,17720,17700],[13731,12.5,370,17721,17700],[17722,6.9,370,17723,17700],[6632,15.5,370,17724,17700],[5485,23.4,370,17725,17700],[5534,22.1,370,17726,17700],[2613,27.9,370,17727,17700],[7153,27.1,370,17728,17700],[17729,33.4,370,17730,17700],[17731,19.8,370,17732,17700],[16784,23.6,370,17733,17700]]},{"id":"infihall-bench-2026-paper","name":"InfiHall-Bench (2026 paper)","group":"视觉与视频","metric":"Overall","unit":"score","public_history":true,"protocol":{"en":"Primary-source results extracted from InfiHall-Bench (arXiv:2611.18053, table Sx6.T3, published 2026-11-13). Retains 7 published evaluation configurations across 7 distinct models on 'Overall' (native score scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《InfiHall-Bench》原始论文（arXiv:2611.18053，表 Sx6.T3，发布日期 2026-11-13）。完整保留“Overall”列下已发表的 7 组评测配置成绩（覆盖 7 个独立模型，采用原生分值口径（不计入准确率饱和度分档））；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.18053v1","published_at":"2026-11-13","status":"paper_baselines"},"model_count":7,"result_count":7,"score_min":0,"score_max":100,"score_decimals":2,"results":[[12059,47.49,370,17734,17735],[8013,67.08,370,17736,17735],[8528,46.69,370,17737,17735],[17738,34.26,370,17739,17735],[17740,50.83,370,17741,17735],[17742,56.42,370,17743,17735],[17744,65.77,370,17745,17735]]},{"id":"fugibench-2026-paper","name":"FuGIBench (2026 paper)","group":"视觉与视频","metric":"[ 0 , 1 ]","unit":"score","public_history":true,"protocol":{"en":"Primary-source results extracted from FuGIBench (arXiv:2611.16400, table S6.T2, published 2026-11-12). Retains 6 published evaluation configurations across 6 distinct models on '[ 0 , 1 ]' (native score scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《FuGIBench》原始论文（arXiv:2611.16400，表 S6.T2，发布日期 2026-11-12）。完整保留“[ 0 , 1 ]”列下已发表的 6 组评测配置成绩（覆盖 6 个独立模型，采用原生分值口径（不计入准确率饱和度分档））；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.16400v1","published_at":"2026-11-12","status":"paper_baselines"},"model_count":6,"result_count":6,"score_min":0,"score_max":1,"score_decimals":3,"results":[[6150,0.298,370,17746,17747],[6138,0.301,370,17748,17747],[6629,0.3,370,17749,17747],[17750,0.217,370,17751,17747],[17752,0.2,370,17753,17747],[17754,0.092,370,17755,17747]]},{"id":"wordinferbench-2026-paper","name":"WordInferBench (2026 paper)","group":"Agent 与工具","metric":"76.9%","unit":"%","public_history":true,"protocol":{"en":"Primary-source results extracted from WordInferBench (arXiv:2611.15189, table S4.T2, published 2026-11-11). Retains 6 published evaluation configurations across 6 distinct models on '76.9%' (comparable percentage scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《WordInferBench》原始论文（arXiv:2611.15189，表 S4.T2，发布日期 2026-11-11）。完整保留“76.9%”列下已发表的 6 组评测配置成绩（覆盖 6 个独立模型，采用可比百分比口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.15189v1","published_at":"2026-11-11","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[13298,0.764,370,17756,17757],[17758,0.661,370,17759,17757],[17760,0.575,370,17761,17757],[17762,0.518,370,17763,17757],[13304,0.478,370,17764,17757],[17765,0.535,370,17766,17757]]},{"id":"cinemme-2026-paper","name":"CineMME (2026 paper)","group":"视觉与视频","metric":"MCQs Accuracy (%) Avg.","unit":"%","public_history":true,"protocol":{"en":"Primary-source results extracted from CineMME (arXiv:2611.15161, table S4.T2, published 2026-11-11). Retains 17 published evaluation configurations across 15 distinct models on 'MCQs Accuracy (%) Avg.' (comparable percentage scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《CineMME》原始论文（arXiv:2611.15161，表 S4.T2，发布日期 2026-11-11）。完整保留“MCQs Accuracy (%) Avg.”列下已发表的 17 组评测配置成绩（覆盖 15 个独立模型，采用可比百分比口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.15161v1","published_at":"2026-11-11","status":"paper_baselines"},"model_count":15,"result_count":17,"results":[[9747,0.2422,370,17767,17768],[7967,0.3088,370,17769,17768],[17770,0.2269,370,17771,17768],[7502,0.2325,370,17772,17768],[17773,0.3111,370,17774,17768],[5722,0.325,370,17775,17768],[16599,0.2975,370,17776,17768],[16601,0.3319,370,17777,17768],[17778,0.3688,370,17779,17768],[2904,0.2244,370,17780,17768],[17778,0.2412,370,17781,17768],[17782,0.3834,370,17783,17768],[8381,0.313,370,17784,17768],[9494,0.5844,370,17785,17768],[2058,0.635,370,17786,17768],[17787,0.612,370,17788,17768],[17789,0.6766,370,17790,17768]]},{"id":"sciresearch-eval-2026-paper","name":"SciResearch-Eval (2026 paper)","group":"Agent 与工具","metric":"Score at default","unit":"score","public_history":true,"protocol":{"en":"Primary-source results extracted from SciResearch-Eval (arXiv:2611.21541, table S4.T3, published 2026-11-16). Retains 9 published evaluation configurations across 9 distinct models on 'Score at default' (native score scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《SciResearch-Eval》原始论文（arXiv:2611.21541，表 S4.T3，发布日期 2026-11-16）。完整保留“Score at default”列下已发表的 9 组评测配置成绩（覆盖 9 个独立模型，采用原生分值口径（不计入准确率饱和度分档））；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.21541v1","published_at":"2026-11-16","status":"paper_baselines"},"model_count":9,"result_count":9,"score_min":0,"score_max":100,"score_decimals":2,"results":[[17791,0.02,370,17792,17793],[17794,0.037,370,17795,17793],[17796,0.75,370,17797,17793],[17798,0.77,370,17799,17793],[17800,1.35,370,17801,17793],[17802,1.39,370,17803,17793],[17804,1.43,370,17805,17793],[17806,1.76,370,17807,17793],[17808,2.07,370,17809,17793]]},{"id":"cae-bench-2026-paper","name":"CAE-Bench (2026 paper)","group":"Agent 与工具","metric":"Overall","unit":"score","public_history":true,"protocol":{"en":"Primary-source results extracted from CAE-Bench (arXiv:2611.21273, table S4.T4, published 2026-11-16). Retains 12 published evaluation configurations across 12 distinct models on 'Overall' (native score scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《CAE-Bench》原始论文（arXiv:2611.21273，表 S4.T4，发布日期 2026-11-16）。完整保留“Overall”列下已发表的 12 组评测配置成绩（覆盖 12 个独立模型，采用原生分值口径（不计入准确率饱和度分档））；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.21273v1","published_at":"2026-11-16","status":"paper_baselines"},"model_count":12,"result_count":12,"score_min":0,"score_max":100,"score_decimals":2,"results":[[3408,40.95,370,17810,17811],[2976,47.82,370,17812,17811],[762,60.23,370,17813,17811],[3394,57.93,370,17814,17811],[2960,65.24,370,17815,17811],[2958,64.85,370,17816,17811],[2983,58.3,370,17817,17811],[2962,62.49,370,17818,17811],[3512,50.41,370,17819,17811],[5899,41.59,370,17820,17811],[3411,40.28,370,17821,17811],[3396,27.1,370,17822,17811]]},{"id":"inclusionmed-2026-paper","name":"InclusionMed (2026 paper)","group":"医疗","metric":"Agency","unit":"score","public_history":true,"protocol":{"en":"Primary-source results extracted from InclusionMed (arXiv:2611.21218, table S5.T2, published 2026-11-16). Retains 22 published evaluation configurations across 22 distinct models on 'Agency' (native score scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《InclusionMed》原始论文（arXiv:2611.21218，表 S5.T2，发布日期 2026-11-16）。完整保留“Agency”列下已发表的 22 组评测配置成绩（覆盖 22 个独立模型，采用原生分值口径（不计入准确率饱和度分档））；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.21218v1","published_at":"2026-11-16","status":"paper_baselines"},"model_count":22,"result_count":22,"score_min":0,"score_max":100,"score_decimals":2,"results":[[2962,45.7,370,17823,17824],[2959,44.7,370,17825,17824],[2956,46.1,370,17826,17824],[2958,36.1,370,17827,17824],[2960,35.4,370,17828,17824],[3512,42.6,370,17829,17824],[2968,37.5,370,17830,17824],[2967,38.2,370,17831,17824],[2982,38.7,370,17832,17824],[2977,33.9,370,17833,17824],[2983,30.6,370,17834,17824],[3408,29.0,370,17835,17824],[3154,32.3,370,17836,17824],[3492,31.0,370,17837,17824],[2976,24.7,370,17838,17824],[2975,33.0,370,17839,17824],[3534,30.8,370,17840,17824],[2971,30.9,370,17841,17824],[2980,25.7,370,17842,17824],[2986,25.9,370,17843,17824],[3493,8.9,370,17844,17824],[17845,13.4,370,17846,17824]]},{"id":"civilbench-2026-paper","name":"CivilBench (2026 paper)","group":"知识与推理","metric":"Total","unit":"%","public_history":true,"protocol":{"en":"Primary-source results extracted from CivilBench (arXiv:2611.21105, table S4.T2, published 2026-11-16). Retains 17 published evaluation configurations across 17 distinct models on 'Total' (comparable percentage scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《CivilBench》原始论文（arXiv:2611.21105，表 S4.T2，发布日期 2026-11-16）。完整保留“Total”列下已发表的 17 组评测配置成绩（覆盖 17 个独立模型，采用可比百分比口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.21105v1","published_at":"2026-11-16","status":"paper_baselines"},"model_count":17,"result_count":17,"results":[[17847,0.4313,370,17848,17849],[2754,0.6307,370,17850,17849],[17851,0.6513,370,17852,17849],[6389,0.6173,370,17853,17849],[8518,0.5885,370,17854,17849],[2478,0.5195,370,17855,17849],[2780,0.5533,370,17856,17849],[6150,0.6637,370,17857,17849],[17858,0.4927,370,17859,17849],[17860,0.6388,370,17861,17849],[17862,0.6595,370,17863,17849],[2531,0.6667,370,17864,17849],[5406,0.6194,370,17865,17849],[345,0.6276,370,17866,17849],[4064,0.6512,370,17867,17849],[2058,0.6965,370,17868,17849],[7801,0.7109,370,17869,17849]]},{"id":"alive-bench-2026-paper","name":"ALIVE-Bench (2026 paper)","group":"Agent 与工具","metric":"Depth","unit":"score","public_history":true,"protocol":{"en":"Primary-source results extracted from ALIVE-Bench (arXiv:2611.20956, table S5.T4, published 2026-11-16). Retains 12 published evaluation configurations across 12 distinct models on 'Depth' (native score scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《ALIVE-Bench》原始论文（arXiv:2611.20956，表 S5.T4，发布日期 2026-11-16）。完整保留“Depth”列下已发表的 12 组评测配置成绩（覆盖 12 个独立模型，采用原生分值口径（不计入准确率饱和度分档））；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.20956v1","published_at":"2026-11-16","status":"paper_baselines"},"model_count":12,"result_count":12,"score_min":0,"score_max":100,"score_decimals":2,"results":[[6130,3.43,370,17870,17871],[6133,3.13,370,17872,17871],[3183,2.45,370,17873,17871],[6978,3.44,370,17874,17871],[2058,3.06,370,17875,17871],[17876,2.82,370,17877,17871],[1464,3.28,370,17878,17871],[5722,1.82,370,17879,17871],[2600,2.2,370,17880,17871],[2613,1.81,370,17881,17871],[7938,2.57,370,17882,17871],[6625,1.57,370,17883,17871]]},{"id":"accelerate-bench-2026-paper","name":"Accelerate Bench (2026 paper)","group":"Agent 与工具","metric":"Score@1","unit":"%","public_history":true,"protocol":{"en":"Primary-source results extracted from Accelerate Bench (arXiv:2611.20882, table S4.T1, published 2026-11-16). Retains 16 published evaluation configurations across 16 distinct models on 'Score@1' (comparable percentage scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《Accelerate Bench》原始论文（arXiv:2611.20882，表 S4.T1，发布日期 2026-11-16）。完整保留“Score@1”列下已发表的 16 组评测配置成绩（覆盖 16 个独立模型，采用可比百分比口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.20882v1","published_at":"2026-11-16","status":"paper_baselines"},"model_count":16,"result_count":16,"results":[[2961,0.674,370,17884,17885],[2959,0.594,370,17886,17885],[2960,0.492,370,17887,17885],[2958,0.472,370,17888,17885],[5728,0.452,370,17889,17885],[14569,0.45,370,17890,17885],[2977,0.425,370,17891,17885],[3385,0.401,370,17892,17885],[3154,0.397,370,17893,17885],[3396,0.393,370,17894,17885],[3185,0.379,370,17895,17885],[2964,0.379,370,17896,17885],[3201,0.341,370,17897,17885],[227,0.34,370,17898,17885],[2974,0.339,370,17899,17885],[5114,0.255,370,17900,17885]]},{"id":"detourbench-2026-paper","name":"DetourBench (2026 paper)","group":"Agent 与工具","metric":"Success rate","unit":"%","public_history":true,"protocol":{"en":"Primary-source results extracted from DetourBench (arXiv:2611.20844, table S5.T2, published 2026-11-16). Retains 5 published evaluation configurations across 5 distinct models on 'Success rate' (comparable percentage scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《DetourBench》原始论文（arXiv:2611.20844，表 S5.T2，发布日期 2026-11-16）。完整保留“Success rate”列下已发表的 5 组评测配置成绩（覆盖 5 个独立模型，采用可比百分比口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.20844v1","published_at":"2026-11-16","status":"paper_baselines"},"model_count":5,"result_count":5,"results":[[3485,0.96,370,17901,17902],[3296,0.92,370,17903,17902],[25,0.613,370,17904,17902],[50,0.6,370,17905,17902],[12316,0.587,370,17906,17902]]},{"id":"curse-bench-2026-paper","name":"CuRSE-Bench (2026 paper)","group":"视觉与视频","metric":"Cohen’s d","unit":"score","public_history":true,"protocol":{"en":"Primary-source results extracted from CuRSE-Bench (arXiv:2611.20821, table S6.T7, published 2026-11-16). Retains 8 published evaluation configurations across 8 distinct models on 'Cohen’s d' (native score scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《CuRSE-Bench》原始论文（arXiv:2611.20821，表 S6.T7，发布日期 2026-11-16）。完整保留“Cohen’s d”列下已发表的 8 组评测配置成绩（覆盖 8 个独立模型，采用原生分值口径（不计入准确率饱和度分档））；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.20821v1","published_at":"2026-11-16","status":"paper_baselines"},"model_count":8,"result_count":8,"score_min":0,"score_max":1,"score_decimals":3,"results":[[797,0.05,370,17907,17908],[4463,0.39,370,17909,17908],[587,0.27,370,17910,17908],[10811,0.44,370,17911,17908],[3184,0.44,370,17912,17908],[3196,0.44,370,17913,17908],[7480,0.4,370,17914,17908],[5115,0.36,370,17915,17908]]},{"id":"eegprobe-2026-paper","name":"EEGProbe (2026 paper)","group":"视觉与视频","metric":"EEG Cross-modal Alignment · EVM","unit":"%","public_history":true,"protocol":{"en":"Primary-source results extracted from EEGProbe (arXiv:2611.20779, table S4.T3, published 2026-11-16). Retains 11 published evaluation configurations across 11 distinct models on 'EEG Cross-modal Alignment · EVM' (comparable percentage scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《EEGProbe》原始论文（arXiv:2611.20779，表 S4.T3，发布日期 2026-11-16）。完整保留“EEG Cross-modal Alignment · EVM”列下已发表的 11 组评测配置成绩（覆盖 11 个独立模型，采用可比百分比口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.20779v1","published_at":"2026-11-16","status":"paper_baselines"},"model_count":11,"result_count":11,"results":[[762,0.24,370,17916,17917],[5519,0.22,370,17918,17917],[3153,0.2,370,17919,17917],[17920,0.3,370,17921,17917],[3408,0.16,370,17922,17917],[6275,0.3,370,17923,17917],[4961,0.26,370,17924,17917],[12527,0.24,370,17925,17917],[4968,0.26,370,17926,17917],[4969,0.18,370,17927,17917],[6149,0.24,370,17928,17917]]},{"id":"industrialknoweval-2026-paper","name":"IndustrialKnowEval (2026 paper)","group":"知识与推理","metric":"Mean","unit":"score","public_history":true,"protocol":{"en":"Primary-source results extracted from IndustrialKnowEval (arXiv:2611.20695, table S5.T4, published 2026-11-16). Retains 18 published evaluation configurations across 18 distinct models on 'Mean' (native score scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《IndustrialKnowEval》原始论文（arXiv:2611.20695，表 S5.T4，发布日期 2026-11-16）。完整保留“Mean”列下已发表的 18 组评测配置成绩（覆盖 18 个独立模型，采用原生分值口径（不计入准确率饱和度分档））；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.20695v1","published_at":"2026-11-16","status":"paper_baselines"},"model_count":18,"result_count":18,"score_min":0,"score_max":1,"score_decimals":3,"results":[[17929,0.7265,370,17930,17931],[17932,0.7285,370,17933,17931],[764,0.7487,370,17934,17931],[782,0.7181,370,17935,17931],[787,0.7113,370,17936,17931],[17937,0.6415,370,17938,17931],[552,0.6443,370,17939,17931],[345,0.7351,370,17940,17931],[2058,0.7694,370,17941,17931],[5365,0.8293,370,17942,17931],[587,0.4633,370,17943,17931],[542,0.6232,370,17944,17931],[5660,0.395,370,17945,17931],[7590,0.5578,370,17946,17931],[5126,0.6571,370,17947,17931],[5344,0.6741,370,17948,17931],[6301,0.6775,370,17949,17931],[9111,0.7373,370,17950,17931]]},{"id":"pixelbench-2026-paper","name":"PixelBench (2026 paper)","group":"视觉与视频","metric":"All (100) Score","unit":"score","public_history":true,"protocol":{"en":"Primary-source results extracted from PixelBench (arXiv:2611.20628, table S6.T2, published 2026-11-16). Retains 12 published evaluation configurations across 12 distinct models on 'All (100) Score' (native score scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《PixelBench》原始论文（arXiv:2611.20628，表 S6.T2，发布日期 2026-11-16）。完整保留“All (100) Score”列下已发表的 12 组评测配置成绩（覆盖 12 个独立模型，采用原生分值口径（不计入准确率饱和度分档））；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.20628v1","published_at":"2026-11-16","status":"paper_baselines"},"model_count":12,"result_count":12,"score_min":0,"score_max":100,"score_decimals":2,"results":[[17951,83.76,370,17952,17953],[17954,78.49,370,17955,17953],[2982,78.04,370,17956,17953],[17378,77.35,370,17957,17953],[6328,76.92,370,17958,17953],[17959,75.7,370,17960,17953],[17961,74.98,370,17962,17953],[16093,74.45,370,17963,17953],[17964,69.51,370,17965,17953],[17966,67.41,370,17967,17953],[16219,65.52,370,17968,17953],[2576,55.6,370,17969,17953]]},{"id":"vahbench-2026-paper","name":"VAHBench (2026 paper)","group":"Agent 与工具","metric":"S ¯","unit":"score","public_history":true,"protocol":{"en":"Primary-source results extracted from VAHBench (arXiv:2611.20535, table S3.T2, published 2026-11-16). Retains 31 published evaluation configurations across 6 distinct models on 'S ¯' (native score scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《VAHBench》原始论文（arXiv:2611.20535，表 S3.T2，发布日期 2026-11-16）。完整保留“S ¯”列下已发表的 31 组评测配置成绩（覆盖 6 个独立模型，采用原生分值口径（不计入准确率饱和度分档））；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.20535v1","published_at":"2026-11-16","status":"paper_baselines"},"model_count":6,"result_count":31,"score_min":0,"score_max":100,"score_decimals":2,"results":[[17970,7.44,370,17971,17972],[12316,14.32,370,17973,17972],[2028,9.22,370,17974,17972],[17437,25.82,370,17975,17972],[17976,14.2,370,17977,17972],[17970,31.34,370,17978,17972],[12316,39.92,370,17979,17972],[2028,30.18,370,17980,17972],[17437,42.02,370,17981,17972],[17976,35.87,370,17982,17972],[17970,29.26,370,17983,17972],[12316,21.64,370,17984,17972],[2028,16.06,370,17985,17972],[17437,30.02,370,17986,17972],[17976,24.25,370,17987,17972],[17970,46.2,370,17988,17972],[12316,49.34,370,17989,17972],[2028,44.06,370,17990,17972],[17437,51.84,370,17991,17972],[17976,47.86,370,17992,17972],[17970,6.48,370,17993,17972],[12316,8.2,370,17994,17972],[2028,1.88,370,17995,17972],[17437,17.3,370,17996,17972],[17976,8.47,370,17997,17972],[17970,26.7,370,17998,17972],[12316,31.1,370,17999,17972],[2028,23.46,370,18000,17972],[17437,41.68,370,18001,17972],[17976,30.74,370,18002,17972],[18003,26.9,370,18004,17972]]},{"id":"reconpotbench-2026-paper","name":"ReconPotBench (2026 paper)","group":"安全与可靠性","metric":"Authenticity Score Avg.","unit":"score","public_history":true,"protocol":{"en":"Primary-source results extracted from ReconPotBench (arXiv:2611.20528, table S4.T2, published 2026-11-16). Retains 24 published evaluation configurations across 21 distinct models on 'Authenticity Score Avg.' (native score scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《ReconPotBench》原始论文（arXiv:2611.20528，表 S4.T2，发布日期 2026-11-16）。完整保留“Authenticity Score Avg.”列下已发表的 24 组评测配置成绩（覆盖 21 个独立模型，采用原生分值口径（不计入准确率饱和度分档））；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.20528v1","published_at":"2026-11-16","status":"paper_baselines"},"model_count":21,"result_count":24,"score_min":0,"score_max":1,"score_decimals":3,"results":[[18005,0.17,370,18006,18007],[5016,0.29,370,18008,18007],[18009,0.19,370,18010,18007],[6957,0.21,370,18011,18007],[4080,0.24,370,18012,18007],[18013,0.2,370,18014,18007],[3748,0.3,370,18015,18007],[6133,0.3,370,18016,18007],[3225,0.27,370,18017,18007],[3009,0.25,370,18018,18007],[769,0.26,370,18019,18007],[7019,0.03,370,18020,18007],[7719,0.13,370,18021,18007],[1921,0.12,370,18022,18007],[797,0.21,370,18023,18007],[18024,0.04,370,18025,18007],[18026,0.03,370,18027,18007],[18028,0.18,370,18029,18007],[18030,0.15,370,18031,18007],[18032,0.2,370,18033,18007],[18034,0.21,370,18035,18007],[18036,0.12,370,18037,18007],[18038,0.14,370,18039,18007],[18040,0.19,370,18041,18007]]},{"id":"evident-2026-paper","name":"EVIDENT (2026 paper)","group":"视觉与视频","metric":"Korean · GSR","unit":"score","public_history":true,"protocol":{"en":"Primary-source results extracted from EVIDENT (arXiv:2611.20488, table S4.T2, published 2026-11-16). Retains 24 published evaluation configurations across 6 distinct models on 'Korean · GSR' (native score scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《EVIDENT》原始论文（arXiv:2611.20488，表 S4.T2，发布日期 2026-11-16）。完整保留“Korean · GSR”列下已发表的 24 组评测配置成绩（覆盖 6 个独立模型，采用原生分值口径（不计入准确率饱和度分档））；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.20488v1","published_at":"2026-11-16","status":"paper_baselines"},"model_count":6,"result_count":24,"score_min":0,"score_max":100,"score_decimals":2,"results":[[18042,45.33,370,18043,18044],[18045,30.0,370,18046,18044],[18047,27.33,370,18048,18044],[18049,4.67,370,18050,18044],[18051,0.0,370,18052,18044],[5487,10.0,370,18053,18044],[18042,42.0,370,18054,18044],[18045,12.0,370,18055,18044],[18047,16.0,370,18056,18044],[18049,6.0,370,18057,18044],[18051,0.0,370,18058,18044],[5487,10.0,370,18059,18044],[18042,54.0,370,18060,18044],[18045,50.0,370,18061,18044],[18047,46.0,370,18062,18044],[18049,4.0,370,18063,18044],[18051,0.0,370,18064,18044],[5487,6.0,370,18065,18044],[18042,40.0,370,18066,18044],[18045,28.0,370,18067,18044],[18047,20.0,370,18068,18044],[18049,4.0,370,18069,18044],[18051,0.0,370,18070,18044],[5487,14.0,370,18071,18044]]},{"id":"logic-chain-breaking-point-2026-paper","name":"Logic Chain Breaking Point (2026 paper)","group":"知识与推理","metric":"Max out","unit":"raw","public_history":true,"protocol":{"en":"Primary-source results extracted from Logic Chain Breaking Point (arXiv:2611.20447, table S3.T1, published 2026-11-16). Retains 16 published evaluation configurations across 16 distinct models on 'Max out' (raw reported metric scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《Logic Chain Breaking Point》原始论文（arXiv:2611.20447，表 S3.T1，发布日期 2026-11-16）。完整保留“Max out”列下已发表的 16 组评测配置成绩（覆盖 16 个独立模型，采用原始指标口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.20447v1","published_at":"2026-11-16","status":"paper_baselines"},"model_count":16,"result_count":16,"score_decimals":2,"results":[[7429,128.0,370,18072,18073],[18074,128.0,370,18075,18073],[7419,8192.0,370,18076,18073],[7407,8192.0,370,18077,18073],[18078,8192.0,370,18079,18073],[18080,128.0,370,18081,18073],[18082,128.0,370,18083,18073],[18084,128.0,370,18085,18073],[18086,128.0,370,18087,18073],[18088,128.0,370,18089,18073],[18090,8192.0,370,18091,18073],[18092,6144.0,370,18093,18073],[18094,6144.0,370,18095,18073],[18096,8192.0,370,18097,18073],[18098,8192.0,370,18099,18073],[18100,8192.0,370,18101,18073]]},{"id":"dominobench-2026-paper","name":"DominoBench (2026 paper)","group":"数学与科学","metric":"H3 all","unit":"score","public_history":true,"protocol":{"en":"Primary-source results extracted from DominoBench (arXiv:2611.20385, table A6.T8, published 2026-11-16). Retains 27 published evaluation configurations across 27 distinct models on 'H3 all' (native score scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《DominoBench》原始论文（arXiv:2611.20385，表 A6.T8，发布日期 2026-11-16）。完整保留“H3 all”列下已发表的 27 组评测配置成绩（覆盖 27 个独立模型，采用原生分值口径（不计入准确率饱和度分档））；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.20385v1","published_at":"2026-11-16","status":"paper_baselines"},"model_count":27,"result_count":27,"score_min":0,"score_max":100,"score_decimals":2,"results":[[2983,62.5,370,18102,18103],[18104,61.1,370,18105,18103],[2962,60.4,370,18106,18103],[2959,59.9,370,18107,18103],[3380,57.3,370,18108,18103],[18109,52.5,370,18110,18103],[4037,48.8,370,18111,18103],[2961,45.4,370,18112,18103],[5934,44.6,370,18113,18103],[2970,43.4,370,18114,18103],[283,33.9,370,18115,18103],[18116,28.0,370,18117,18103],[5300,27.6,370,18118,18103],[2990,25.3,370,18119,18103],[5320,24.5,370,18120,18103],[2698,21.1,370,18121,18103],[18122,17.4,370,18123,18103],[325,17.1,370,18124,18103],[8990,13.7,370,18125,18103],[2585,13.4,370,18126,18103],[3764,12.8,370,18127,18103],[6343,10.5,370,18128,18103],[2626,8.7,370,18129,18103],[2642,8.2,370,18130,18103],[6150,2.4,370,18131,18103],[2754,1.0,370,18132,18103],[6238,0.0,370,18133,18103]]},{"id":"trait-collapse-in-text-to-image-generation-2026-paper","name":"Trait Collapse in Text-to-Image Generation (2026 paper)","group":"视觉与视频","metric":"SCS","unit":"%","public_history":true,"protocol":{"en":"Primary-source results extracted from Trait Collapse in Text-to-Image Generation (arXiv:2611.20381, table S6.T2, published 2026-11-16). Retains 14 published evaluation configurations across 14 distinct models on 'SCS' (comparable percentage scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《Trait Collapse in Text-to-Image Generation》原始论文（arXiv:2611.20381，表 S6.T2，发布日期 2026-11-16）。完整保留“SCS”列下已发表的 14 组评测配置成绩（覆盖 14 个独立模型，采用可比百分比口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.20381v1","published_at":"2026-11-16","status":"paper_baselines"},"model_count":14,"result_count":14,"results":[[12059,0.00585,370,18134,18135],[18136,0.00587,370,18137,18135],[8013,0.00614,370,18138,18135],[8090,0.00605,370,18139,18135],[14842,0.00665,370,18140,18135],[18141,0.0056,370,18142,18135],[18143,0.00564,370,18144,18135],[18145,0.00642,370,18146,18135],[18147,0.00603,370,18148,18135],[18149,0.00701,370,18150,18135],[7290,0.00702,370,18151,18135],[6365,0.00707,370,18152,18135],[18153,0.0074,370,18154,18135],[18155,0.0073,370,18156,18135]]},{"id":"mr-bench-2026-paper","name":"MR-Bench (2026 paper)","group":"视觉与视频","metric":"NS","unit":"%","public_history":true,"protocol":{"en":"Primary-source results extracted from MR-Bench (arXiv:2611.20349, table S3.T5, published 2026-11-16). Retains 23 published evaluation configurations across 23 distinct models on 'NS' (comparable percentage scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《MR-Bench》原始论文（arXiv:2611.20349，表 S3.T5，发布日期 2026-11-16）。完整保留“NS”列下已发表的 23 组评测配置成绩（覆盖 23 个独立模型，采用可比百分比口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.20349v1","published_at":"2026-11-16","status":"paper_baselines"},"model_count":23,"result_count":23,"results":[[6138,0.946,370,18157,18158],[6136,0.925,370,18159,18158],[864,0.896,370,18160,18158],[9197,0.904,370,18161,18158],[2728,0.607,370,18162,18158],[2913,0.407,370,18163,18158],[2754,0.425,370,18164,18158],[797,0.993,370,18165,18158],[18166,0.007,370,18167,18158],[18168,0.836,370,18169,18158],[18170,0.848,370,18171,18158],[18172,0.936,370,18173,18158],[18174,0.919,370,18175,18158],[18176,0.913,370,18177,18158],[18178,0.902,370,18179,18158],[18180,0.931,370,18181,18158],[18182,0.912,370,18183,18158],[18184,0.906,370,18185,18158],[18186,0.907,370,18187,18158],[18188,0.912,370,18189,18158],[18190,0.911,370,18191,18158],[18192,0.914,370,18193,18158],[18194,0.938,370,18195,18158]]},{"id":"llm-inference-on-desktop-scale-nvidia-2026-paper","name":"LLM Inference on Desktop-Scale NVIDIA (2026 paper)","group":"综合","metric":"KV capacity (req.)","unit":"raw","public_history":true,"protocol":{"en":"Primary-source results extracted from LLM Inference on Desktop-Scale NVIDIA (arXiv:2611.20289, table S4.T2, published 2026-11-15). Retains 17 published evaluation configurations across 17 distinct models on 'KV capacity (req.)' (raw reported metric scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《LLM Inference on Desktop-Scale NVIDIA》原始论文（arXiv:2611.20289，表 S4.T2，发布日期 2026-11-15）。完整保留“KV capacity (req.)”列下已发表的 17 组评测配置成绩（覆盖 17 个独立模型，采用原始指标口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.20289v1","published_at":"2026-11-15","status":"paper_baselines"},"model_count":17,"result_count":17,"score_decimals":2,"results":[[18196,161.0,370,18197,18198],[18199,337.0,370,18200,18198],[18201,344.0,370,18202,18198],[18203,181.0,370,18204,18198],[18205,212.0,370,18206,18198],[18207,368.0,370,18208,18198],[18209,48.0,370,18210,18198],[18211,63.0,370,18212,18198],[18213,494.0,370,18214,18198],[18215,627.0,370,18216,18198],[18217,685.0,370,18218,18198],[18219,179.0,370,18220,18198],[18221,2002.0,370,18222,18198],[18223,951.0,370,18224,18198],[18225,169.0,370,18226,18198],[18227,225.0,370,18228,18198],[18229,68.0,370,18230,18198]]},{"id":"swe-cowork-2026-paper","name":"SWE-CoWork (2026 paper)","group":"编程","metric":"Requirements obtained (%) All","unit":"%","public_history":true,"protocol":{"en":"Primary-source results extracted from SWE-CoWork (arXiv:2611.20218, table A7.T9, published 2026-11-15). Retains 11 published evaluation configurations across 11 distinct models on 'Requirements obtained (%) All' (comparable percentage scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《SWE-CoWork》原始论文（arXiv:2611.20218，表 A7.T9，发布日期 2026-11-15）。完整保留“Requirements obtained (%) All”列下已发表的 11 组评测配置成绩（覆盖 11 个独立模型，采用可比百分比口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.20218v1","published_at":"2026-11-15","status":"paper_baselines"},"model_count":11,"result_count":11,"results":[[2958,0.753,370,18231,18232],[5275,0.707,370,18233,18232],[3515,0.663,370,18234,18232],[3157,0.566,370,18235,18232],[18236,0.506,370,18237,18232],[5899,0.51,370,18238,18232],[2977,0.452,370,18239,18232],[3512,0.395,370,18240,18232],[4881,0.367,370,18241,18232],[3654,0.35,370,18242,18232],[3492,0.239,370,18243,18232]]},{"id":"sg-legalclarify-2026-paper","name":"SG-LegalClarify (2026 paper)","group":"专业工作","metric":"Temp.","unit":"score","public_history":true,"protocol":{"en":"Primary-source results extracted from SG-LegalClarify (arXiv:2611.19643, table A2.T4, published 2026-11-14). Retains 17 published evaluation configurations across 17 distinct models on 'Temp.' (native score scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《SG-LegalClarify》原始论文（arXiv:2611.19643，表 A2.T4，发布日期 2026-11-14）。完整保留“Temp.”列下已发表的 17 组评测配置成绩（覆盖 17 个独立模型，采用原生分值口径（不计入准确率饱和度分档））；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.19643v1","published_at":"2026-11-14","status":"paper_baselines"},"model_count":17,"result_count":17,"score_min":0,"score_max":1,"score_decimals":3,"results":[[3453,1.0,370,18244,18245],[3479,1.0,370,18246,18245],[3568,1.0,370,18247,18245],[762,1.0,370,18248,18245],[2973,1.0,370,18249,18245],[3408,1.0,370,18250,18245],[5734,0.7,370,18251,18245],[3490,0.6,370,18252,18245],[2977,0.6,370,18253,18245],[6664,0.6,370,18254,18245],[7229,0.6,370,18255,18245],[18256,0.0,370,18257,18245],[4973,0.7,370,18258,18245],[363,0.0,370,18259,18245],[18260,1.0,370,18261,18245],[18262,1.0,370,18263,18245],[11955,0.0,370,18264,18245]]},{"id":"rta-bench-2026-paper","name":"rta-bench (2026 paper)","group":"Agent 与工具","metric":"Decisions","unit":"raw","public_history":true,"protocol":{"en":"Primary-source results extracted from rta-bench (arXiv:2611.19530, table A2.T13, published 2026-11-14). Retains 48 published evaluation configurations across 6 distinct models on 'Decisions' (raw reported metric scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《rta-bench》原始论文（arXiv:2611.19530，表 A2.T13，发布日期 2026-11-14）。完整保留“Decisions”列下已发表的 48 组评测配置成绩（覆盖 6 个独立模型，采用原始指标口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.19530v1","published_at":"2026-11-14","status":"paper_baselines"},"model_count":6,"result_count":48,"score_decimals":2,"results":[[3496,204.0,370,18265,18266],[3496,240.0,370,18267,18266],[3496,218.0,370,18268,18266],[3392,44.0,370,18269,18266],[3392,4.0,370,18270,18266],[3392,66.0,370,18271,18266],[3409,152.0,370,18272,18266],[3409,156.0,370,18273,18266],[3409,170.0,370,18274,18266],[3415,94.0,370,18275,18266],[3415,119.0,370,18276,18266],[3415,113.0,370,18277,18266],[3415,124.0,370,18278,18266],[3415,78.0,370,18279,18266],[3415,143.0,370,18280,18266],[3415,135.0,370,18281,18266],[3415,125.0,370,18282,18266],[3415,126.0,370,18283,18266],[3415,153.0,370,18284,18266],[3415,140.0,370,18285,18266],[3415,124.0,370,18286,18266],[3408,188.0,370,18287,18266],[3408,154.0,370,18288,18266],[3408,138.0,370,18289,18266],[3408,130.0,370,18290,18266],[3408,112.0,370,18291,18266],[3408,123.0,370,18292,18266],[3408,159.0,370,18293,18266],[3408,191.0,370,18294,18266],[3408,159.0,370,18295,18266],[3408,173.0,370,18296,18266],[3408,162.0,370,18297,18266],[3408,157.0,370,18298,18266],[3408,212.0,370,18299,18266],[3408,142.0,370,18300,18266],[3408,167.0,370,18301,18266],[3408,167.0,370,18302,18266],[3408,155.0,370,18303,18266],[3408,211.0,370,18304,18266],[8573,1.0,370,18305,18266],[8573,5.0,370,18306,18266],[8573,20.0,370,18307,18266],[8573,1.0,370,18308,18266],[8573,5.0,370,18309,18266],[8573,8.0,370,18310,18266],[8573,1.0,370,18311,18266],[8573,5.0,370,18312,18266],[8573,1.0,370,18313,18266]]},{"id":"xlsar-bench-2026-paper","name":"XLSAR-Bench (2026 paper)","group":"视觉与视频","metric":"Avg.","unit":"%","public_history":true,"protocol":{"en":"Primary-source results extracted from XLSAR-Bench (arXiv:2611.19364, table S3.T3, published 2026-11-14). Retains 16 published evaluation configurations across 8 distinct models on 'Avg.' (comparable percentage scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《XLSAR-Bench》原始论文（arXiv:2611.19364，表 S3.T3，发布日期 2026-11-14）。完整保留“Avg.”列下已发表的 16 组评测配置成绩（覆盖 8 个独立模型，采用可比百分比口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.19364v1","published_at":"2026-11-14","status":"paper_baselines"},"model_count":8,"result_count":16,"results":[[6886,0.6745,370,18314,18315],[5876,0.6326,370,18316,18315],[5816,0.4731,370,18317,18315],[18318,0.469,370,18319,18315],[18320,0.2503,370,18321,18315],[9997,0.505,370,18322,18315],[16780,0.2605,370,18323,18315],[18324,0.6077,370,18325,18315],[6886,0.7017,370,18326,18315],[5876,0.641,370,18327,18315],[5816,0.5483,370,18328,18315],[18318,0.5456,370,18329,18315],[18320,0.2461,370,18330,18315],[9997,0.5476,370,18331,18315],[16780,0.2457,370,18332,18315],[18324,0.6012,370,18333,18315]]},{"id":"forecastgym-2026-paper","name":"ForecastGym (2026 paper)","group":"Agent 与工具","metric":"Accuracy","unit":"%","public_history":true,"protocol":{"en":"Primary-source results extracted from ForecastGym (arXiv:2611.23045, table S6.T4, published 2026-11-17). Retains 6 published evaluation configurations across 6 distinct models on 'Accuracy' (comparable percentage scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《ForecastGym》原始论文（arXiv:2611.23045，表 S6.T4，发布日期 2026-11-17）。完整保留“Accuracy”列下已发表的 6 组评测配置成绩（覆盖 6 个独立模型，采用可比百分比口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.23045v1","published_at":"2026-11-17","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[2973,0.642,370,18334,18335],[3568,0.597,370,18336,18335],[2983,0.628,370,18337,18335],[3153,0.66,370,18338,18335],[2960,0.653,370,18339,18335],[18340,0.719,370,18341,18335]]},{"id":"the-shared-word-2026-paper","name":"The Shared Word (2026 paper)","group":"知识与推理","metric":"Acc.","unit":"%","public_history":true,"protocol":{"en":"Primary-source results extracted from The Shared Word (arXiv:2611.22935, table S4.T3, published 2026-11-17). Retains 27 published evaluation configurations across 25 distinct models on 'Acc.' (comparable percentage scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《The Shared Word》原始论文（arXiv:2611.22935，表 S4.T3，发布日期 2026-11-17）。完整保留“Acc.”列下已发表的 27 组评测配置成绩（覆盖 25 个独立模型，采用可比百分比口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.22935v1","published_at":"2026-11-17","status":"paper_baselines"},"model_count":25,"result_count":27,"results":[[18342,0.07,370,18343,18344],[18345,0.093,370,18346,18344],[18347,0.148,370,18348,18344],[18349,0.15,370,18350,18344],[18351,0.051,370,18352,18344],[18353,0.142,370,18354,18344],[18355,0.131,370,18356,18344],[8579,0.14,370,18357,18344],[8591,0.329,370,18358,18344],[3688,0.158,370,18359,18344],[8975,0.156,370,18360,18344],[5596,0.298,370,18361,18344],[11955,0.393,370,18362,18344],[18363,0.258,370,18364,18344],[18365,0.273,370,18366,18344],[18367,0.308,370,18368,18344],[5344,0.285,370,18369,18344],[4070,0.199,370,18370,18344],[18371,0.103,370,18372,18344],[18373,0.09,370,18374,18344],[5343,0.121,370,18375,18344],[780,0.563,370,18376,18344],[18377,0.592,370,18378,18344],[18379,0.541,370,18380,18344],[16080,0.79,370,18381,18344],[18382,0.699,370,18383,18344],[12041,0.888,370,18384,18344]]},{"id":"think-before-you-code-2026-paper","name":"Think Before You Code (2026 paper)","group":"编程","metric":"Overall","unit":"score","public_history":true,"protocol":{"en":"Primary-source results extracted from Think Before You Code (arXiv:2611.22905, table S4.T3, published 2026-11-17). Retains 9 published evaluation configurations across 9 distinct models on 'Overall' (native score scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《Think Before You Code》原始论文（arXiv:2611.22905，表 S4.T3，发布日期 2026-11-17）。完整保留“Overall”列下已发表的 9 组评测配置成绩（覆盖 9 个独立模型，采用原生分值口径（不计入准确率饱和度分档））；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.22905v1","published_at":"2026-11-17","status":"paper_baselines"},"model_count":9,"result_count":9,"score_min":0,"score_max":1,"score_decimals":3,"results":[[3184,0.436,370,18385,18386],[2058,0.411,370,18387,18386],[12364,0.353,370,18388,18386],[6399,0.391,370,18389,18386],[2055,0.386,370,18390,18386],[2261,0.383,370,18391,18386],[3572,0.312,370,18392,18386],[18393,0.422,370,18394,18386],[7265,0.381,370,18395,18386]]},{"id":"social-signal-bench-2026-paper","name":"Social Signal Bench (2026 paper)","group":"视觉与视频","metric":"Eng. MF1","unit":"score","public_history":true,"protocol":{"en":"Primary-source results extracted from Social Signal Bench (arXiv:2611.22704, table A7.T10, published 2026-11-17). Retains 17 published evaluation configurations across 15 distinct models on 'Eng. MF1' (native score scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《Social Signal Bench》原始论文（arXiv:2611.22704，表 A7.T10，发布日期 2026-11-17）。完整保留“Eng. MF1”列下已发表的 17 组评测配置成绩（覆盖 15 个独立模型，采用原生分值口径（不计入准确率饱和度分档））；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.22704v1","published_at":"2026-11-17","status":"paper_baselines"},"model_count":15,"result_count":17,"score_min":0,"score_max":100,"score_decimals":2,"results":[[2979,43.0,370,18396,18397],[2983,35.1,370,18398,18397],[6724,40.0,370,18399,18397],[2962,39.4,370,18400,18397],[2960,33.5,370,18401,18397],[6143,38.2,370,18402,18397],[18403,20.7,370,18404,18397],[18405,58.4,370,18406,18397],[3412,46.5,370,18407,18397],[2983,35.8,370,18408,18397],[6724,47.2,370,18409,18397],[18410,37.7,370,18411,18397],[18412,26.2,370,18413,18397],[18414,12.4,370,18415,18397],[13706,32.9,370,18416,18397],[18417,31.1,370,18418,18397],[18419,45.8,370,18420,18397]]},{"id":"spatialcast-2026-paper","name":"SpatialCast (2026 paper)","group":"视觉与视频","metric":"Real (200) Overall","unit":"%","public_history":true,"protocol":{"en":"Primary-source results extracted from SpatialCast (arXiv:2611.22657, table S4.T2, published 2026-11-17). Retains 15 published evaluation configurations across 15 distinct models on 'Real (200) Overall' (comparable percentage scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《SpatialCast》原始论文（arXiv:2611.22657，表 S4.T2，发布日期 2026-11-17）。完整保留“Real (200) Overall”列下已发表的 15 组评测配置成绩（覆盖 15 个独立模型，采用可比百分比口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.22657v1","published_at":"2026-11-17","status":"paper_baselines"},"model_count":15,"result_count":15,"results":[[2960,0.795,370,18421,18422],[2962,0.63,370,18423,18422],[18424,0.345,370,18425,18422],[18426,0.365,370,18427,18422],[18428,0.335,370,18429,18422],[18430,0.365,370,18431,18422],[18432,0.34,370,18433,18422],[18434,0.385,370,18435,18422],[18436,0.305,370,18437,18422],[18438,0.355,370,18439,18422],[18440,0.33,370,18441,18422],[18442,0.345,370,18443,18422],[18444,0.36,370,18445,18422],[18446,0.335,370,18447,18422],[18448,0.345,370,18449,18422]]},{"id":"mm-reportbench-2026-paper","name":"MM-ReportBench (2026 paper)","group":"医疗","metric":"Overall","unit":"score","public_history":true,"protocol":{"en":"Primary-source results extracted from MM-ReportBench (arXiv:2611.22514, table S4.T2, published 2026-11-17). Retains 20 published evaluation configurations across 20 distinct models on 'Overall' (native score scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《MM-ReportBench》原始论文（arXiv:2611.22514，表 S4.T2，发布日期 2026-11-17）。完整保留“Overall”列下已发表的 20 组评测配置成绩（覆盖 20 个独立模型，采用原生分值口径（不计入准确率饱和度分档））；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.22514v1","published_at":"2026-11-17","status":"paper_baselines"},"model_count":20,"result_count":20,"score_min":0,"score_max":100,"score_decimals":2,"results":[[18450,22.3,370,18451,18452],[4066,21.2,370,18453,18452],[1573,21.3,370,18454,18452],[786,20.3,370,18455,18452],[5648,19.5,370,18456,18452],[3196,23.5,370,18457,18452],[353,64.3,370,18458,18452],[345,63.0,370,18459,18452],[764,66.8,370,18460,18452],[6133,63.8,370,18461,18452],[2129,65.0,370,18462,18452],[7685,64.8,370,18463,18452],[3578,63.6,370,18464,18452],[6297,61.4,370,18465,18452],[769,68.9,370,18466,18452],[18467,63.6,370,18468,18452],[17546,53.4,370,18469,18452],[18470,67.9,370,18471,18452],[18472,65.7,370,18473,18452],[18474,63.9,370,18475,18452]]},{"id":"sciworkbench-2026-paper","name":"SciWorkBench (2026 paper)","group":"Agent 与工具","metric":"Physics","unit":"score","public_history":true,"protocol":{"en":"Primary-source results extracted from SciWorkBench (arXiv:2611.22490, table S5.T4, published 2026-11-17). Retains 13 published evaluation configurations across 13 distinct models on 'Physics' (native score scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《SciWorkBench》原始论文（arXiv:2611.22490，表 S5.T4，发布日期 2026-11-17）。完整保留“Physics”列下已发表的 13 组评测配置成绩（覆盖 13 个独立模型，采用原生分值口径（不计入准确率饱和度分档））；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.22490v1","published_at":"2026-11-17","status":"paper_baselines"},"model_count":13,"result_count":13,"score_min":0,"score_max":100,"score_decimals":2,"results":[[18476,56.17,370,18477,18478],[18479,51.73,370,18480,18478],[18481,57.23,370,18482,18478],[18483,42.76,370,18484,18478],[18485,42.01,370,18486,18478],[18487,43.39,370,18488,18478],[18489,38.21,370,18490,18478],[18491,36.61,370,18492,18478],[18493,25.5,370,18494,18478],[18495,35.57,370,18496,18478],[18497,28.51,370,18498,18478],[18499,30.49,370,18500,18478],[18501,17.5,370,18502,18478]]},{"id":"contrasense-2026-paper","name":"ContraSense (2026 paper)","group":"视觉与视频","metric":"Contra-Acc( % )","unit":"raw","public_history":true,"protocol":{"en":"Primary-source results extracted from ContraSense (arXiv:2611.22360, table S4.T2, published 2026-11-17). Retains 20 published evaluation configurations across 8 distinct models on 'Contra-Acc( % )' (raw reported metric scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《ContraSense》原始论文（arXiv:2611.22360，表 S4.T2，发布日期 2026-11-17）。完整保留“Contra-Acc( % )”列下已发表的 20 组评测配置成绩（覆盖 8 个独立模型，采用原始指标口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.22360v1","published_at":"2026-11-17","status":"paper_baselines"},"model_count":8,"result_count":20,"score_decimals":2,"results":[[18503,-42.27,370,18504,18505],[18503,35.89,370,18506,18505],[18503,80.97,370,18507,18505],[13726,-45.08,370,18508,18505],[13726,37.55,370,18509,18505],[13726,81.74,370,18510,18505],[7711,-44.19,370,18511,18505],[7711,54.02,370,18512,18505],[7711,91.95,370,18513,18505],[2471,-37.93,370,18514,18505],[4457,-33.46,370,18515,18505],[4457,47.13,370,18516,18505],[4457,80.72,370,18517,18505],[18518,-33.59,370,18519,18505],[18518,52.11,370,18520,18505],[18518,85.44,370,18521,18505],[353,-33.33,370,18522,18505],[353,60.03,370,18523,18505],[353,86.08,370,18524,18505],[345,-26.05,370,18525,18505]]},{"id":"officeprism-2026-paper","name":"OfficePrism (2026 paper)","group":"Agent 与工具","metric":"Full med.","unit":"score","public_history":true,"protocol":{"en":"Primary-source results extracted from OfficePrism (arXiv:2611.22262, table S6.T4, published 2026-11-17). Retains 10 published evaluation configurations across 10 distinct models on 'Full med.' (native score scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《OfficePrism》原始论文（arXiv:2611.22262，表 S6.T4，发布日期 2026-11-17）。完整保留“Full med.”列下已发表的 10 组评测配置成绩（覆盖 10 个独立模型，采用原生分值口径（不计入准确率饱和度分档））；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.22262v1","published_at":"2026-11-17","status":"paper_baselines"},"model_count":10,"result_count":10,"score_min":0,"score_max":100,"score_decimals":2,"results":[[4952,70.2,370,18526,18527],[5104,70.2,370,18528,18527],[5149,67.9,370,18529,18527],[3209,67.6,370,18530,18527],[440,67.1,370,18531,18527],[411,65.5,370,18532,18527],[1321,66.2,370,18533,18527],[18534,63.6,370,18535,18527],[438,67.6,370,18536,18527],[420,61.2,370,18537,18527]]},{"id":"where-does-hindsight-enter-clinical-2026-paper","name":"Where Does Hindsight Enter Clinical (2026 paper)","group":"医疗","metric":"V2 − V3","unit":"%","public_history":true,"protocol":{"en":"Primary-source results extracted from Where Does Hindsight Enter Clinical (arXiv:2611.22241, table S5.T2, published 2026-11-17). Retains 10 published evaluation configurations across 10 distinct models on 'V2 − V3' (comparable percentage scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《Where Does Hindsight Enter Clinical》原始论文（arXiv:2611.22241，表 S5.T2，发布日期 2026-11-17）。完整保留“V2 − V3”列下已发表的 10 组评测配置成绩（覆盖 10 个独立模型，采用可比百分比口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.22241v1","published_at":"2026-11-17","status":"paper_baselines"},"model_count":10,"result_count":10,"results":[[797,0.198,370,18538,18539],[4466,0.207,370,18540,18539],[4463,0.184,370,18541,18539],[1621,0.205,370,18542,18539],[587,0.221,370,18543,18539],[18544,0.198,370,18545,18539],[18546,0.203,370,18547,18539],[18548,0.071,370,18549,18539],[18550,0.181,370,18551,18539],[18552,0.219,370,18553,18539]]},{"id":"roboteam-2026-paper","name":"RoboTeam (2026 paper)","group":"视觉与视频","metric":"per task","unit":"raw","public_history":true,"protocol":{"en":"Primary-source results extracted from RoboTeam (arXiv:2611.22138, table S2.T3, published 2026-11-17). Retains 13 published evaluation configurations across 12 distinct models on 'per task' (raw reported metric scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《RoboTeam》原始论文（arXiv:2611.22138，表 S2.T3，发布日期 2026-11-17）。完整保留“per task”列下已发表的 13 组评测配置成绩（覆盖 12 个独立模型，采用原始指标口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.22138v1","published_at":"2026-11-17","status":"paper_baselines"},"model_count":12,"result_count":13,"score_decimals":2,"results":[[6649,184.0,370,18554,18555],[18556,471.0,370,18557,18555],[18558,421.0,370,18559,18555],[18560,0.0,370,18561,18555],[18562,298.0,370,18563,18555],[18564,212.0,370,18565,18555],[4946,115.0,370,18566,18555],[18567,0.0,370,18568,18555],[18567,0.0,370,18569,18555],[18570,14.0,370,18571,18555],[18572,3.0,370,18573,18555],[4955,12.0,370,18574,18555],[6037,3.0,370,18575,18555]]},{"id":"atlas-finance-2026-paper","name":"ATLAS-Finance (2026 paper)","group":"专业工作","metric":"Mean score","unit":"%","public_history":true,"protocol":{"en":"Primary-source results extracted from ATLAS-Finance (arXiv:2611.22085, table S5.T3, published 2026-11-17). Retains 11 published evaluation configurations across 11 distinct models on 'Mean score' (comparable percentage scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《ATLAS-Finance》原始论文（arXiv:2611.22085，表 S5.T3，发布日期 2026-11-17）。完整保留“Mean score”列下已发表的 11 组评测配置成绩（覆盖 11 个独立模型，采用可比百分比口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.22085v1","published_at":"2026-11-17","status":"paper_baselines"},"model_count":11,"result_count":11,"results":[[755,0.006,370,18576,18577],[2956,0.004,370,18578,18577],[2958,0.004,370,18579,18577],[2977,0.006,370,18580,18577],[2974,0.007,370,18581,18577],[3515,0.007,370,18582,18577],[2983,0.01,370,18583,18577],[762,0.005,370,18584,18577],[3386,0.007,370,18585,18577],[3492,0.009,370,18586,18577],[3482,0.009,370,18587,18577]]},{"id":"diagtrace-2026-paper","name":"DiagTrace (2026 paper)","group":"医疗","metric":"Event R","unit":"%","public_history":true,"protocol":{"en":"Primary-source results extracted from DiagTrace (arXiv:2611.22009, table A1.T5, published 2026-11-16). Retains 7 published evaluation configurations across 7 distinct models on 'Event R' (comparable percentage scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《DiagTrace》原始论文（arXiv:2611.22009，表 A1.T5，发布日期 2026-11-16）。完整保留“Event R”列下已发表的 7 组评测配置成绩（覆盖 7 个独立模型，采用可比百分比口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.22009v1","published_at":"2026-11-16","status":"paper_baselines"},"model_count":7,"result_count":7,"results":[[18588,0.577,370,18589,18590],[18591,0.575,370,18592,18590],[18593,0.665,370,18594,18590],[18595,0.666,370,18596,18590],[18597,0.655,370,18598,18590],[18599,0.654,370,18600,18590],[18601,0.661,370,18602,18590]]},{"id":"embedgym-2026-paper","name":"EmbedGym (2026 paper)","group":"Agent 与工具","metric":"zephyr ( N = 107 ) · L3-I","unit":"score","public_history":true,"protocol":{"en":"Primary-source results extracted from EmbedGym (arXiv:2611.21845, table S6.T3, published 2026-11-16). Retains 8 published evaluation configurations across 8 distinct models on 'zephyr ( N = 107 ) · L3-I' (native score scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《EmbedGym》原始论文（arXiv:2611.21845，表 S6.T3，发布日期 2026-11-16）。完整保留“zephyr ( N = 107 ) · L3-I”列下已发表的 8 组评测配置成绩（覆盖 8 个独立模型，采用原生分值口径（不计入准确率饱和度分档））；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.21845v1","published_at":"2026-11-16","status":"paper_baselines"},"model_count":8,"result_count":8,"score_min":0,"score_max":100,"score_decimals":2,"results":[[3408,71.0,370,18603,18604],[762,93.0,370,18605,18604],[2976,82.0,370,18606,18604],[2960,95.0,370,18607,18604],[2983,92.0,370,18608,18604],[2974,93.0,370,18609,18604],[2990,58.0,370,18610,18604],[5320,51.0,370,18611,18604]]},{"id":"ejbench-2026-paper","name":"EJBench (2026 paper)","group":"知识与推理","metric":"Envelope","unit":"%","public_history":true,"protocol":{"en":"Primary-source results extracted from EJBench (arXiv:2611.21707, table S5.T3, published 2026-11-16). Retains 12 published evaluation configurations across 12 distinct models on 'Envelope' (comparable percentage scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《EJBench》原始论文（arXiv:2611.21707，表 S5.T3，发布日期 2026-11-16）。完整保留“Envelope”列下已发表的 12 组评测配置成绩（覆盖 12 个独立模型，采用可比百分比口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.21707v1","published_at":"2026-11-16","status":"paper_baselines"},"model_count":12,"result_count":12,"results":[[2958,0.00771,370,18612,18613],[2960,0.0075,370,18614,18613],[2959,0.00771,370,18615,18613],[2964,0.00688,370,18616,18613],[2979,0.00771,370,18617,18613],[2961,0.0075,370,18618,18613],[2992,0.00708,370,18619,18613],[3512,0.00583,370,18620,18613],[2970,0.00542,370,18621,18613],[2976,0.00521,370,18622,18613],[18623,0.00203,370,18624,18613],[4070,0.00115,370,18625,18613]]},{"id":"a-3-bench-2026-paper","name":"A$^3$Bench (2026 paper)","group":"知识与推理","metric":"Average","unit":"%","public_history":true,"protocol":{"en":"Primary-source results extracted from A$^3$Bench (arXiv:2611.24762, table S4.T2, published 2026-11-18). Retains 15 published evaluation configurations across 13 distinct models on 'Average' (comparable percentage scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《A$^3$Bench》原始论文（arXiv:2611.24762，表 S4.T2，发布日期 2026-11-18）。完整保留“Average”列下已发表的 15 组评测配置成绩（覆盖 13 个独立模型，采用可比百分比口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.24762v1","published_at":"2026-11-18","status":"paper_baselines"},"model_count":13,"result_count":15,"results":[[9076,0.3986,370,18626,18627],[12282,0.3631,370,18628,18627],[2774,0.4146,370,18629,18627],[2731,0.3911,370,18630,18627],[2654,0.3804,370,18631,18627],[17722,0.1734,370,18632,18627],[2646,0.4057,370,18633,18627],[18634,0.3425,370,18635,18627],[18636,0.443,370,18637,18627],[6349,0.3397,370,18638,18627],[18639,0.3737,370,18640,18627],[3644,0.3937,370,18641,18627],[18636,0.3726,370,18642,18627],[18643,0.4119,370,18644,18627],[2654,0.4362,370,18645,18627]]},{"id":"absbench-2026-paper","name":"ABSBench (2026 paper)","group":"视觉与视频","metric":"V + T","unit":"raw","public_history":true,"protocol":{"en":"Primary-source results extracted from ABSBench (arXiv:2611.24517, table Sx6.T2, published 2026-11-18). Retains 5 published evaluation configurations across 5 distinct models on 'V + T' (raw reported metric scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《ABSBench》原始论文（arXiv:2611.24517，表 Sx6.T2，发布日期 2026-11-18）。完整保留“V + T”列下已发表的 5 组评测配置成绩（覆盖 5 个独立模型，采用原始指标口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.24517v1","published_at":"2026-11-18","status":"paper_baselines"},"model_count":5,"result_count":5,"score_decimals":2,"results":[[3401,135.1,370,18646,18647],[18648,103.4,370,18649,18647],[18650,86.2,370,18651,18647],[3196,137.3,370,18652,18647],[3496,69.9,370,18653,18647]]},{"id":"videoace-2026-paper","name":"VideoACE (2026 paper)","group":"视觉与视频","metric":"Gain","unit":"%","public_history":true,"protocol":{"en":"Primary-source results extracted from VideoACE (arXiv:2611.24184, table S5.T2, published 2026-11-18). Retains 5 published evaluation configurations across 5 distinct models on 'Gain' (comparable percentage scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《VideoACE》原始论文（arXiv:2611.24184，表 S5.T2，发布日期 2026-11-18）。完整保留“Gain”列下已发表的 5 组评测配置成绩（覆盖 5 个独立模型，采用可比百分比口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.24184v1","published_at":"2026-11-18","status":"paper_baselines"},"model_count":5,"result_count":5,"results":[[2962,0.073,370,18654,18655],[2983,0.11,370,18656,18655],[3405,0.142,370,18657,18655],[5899,0.127,370,18658,18655],[6923,0.03,370,18659,18655]]},{"id":"roughcut-bench-2026-paper","name":"RoughCut-Bench (2026 paper)","group":"视觉与视频","metric":"Avg. IoU","unit":"score","public_history":true,"protocol":{"en":"Primary-source results extracted from RoughCut-Bench (arXiv:2611.24098, table S3.T1, published 2026-11-18). Retains 8 published evaluation configurations across 8 distinct models on 'Avg. IoU' (native score scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《RoughCut-Bench》原始论文（arXiv:2611.24098，表 S3.T1，发布日期 2026-11-18）。完整保留“Avg. IoU”列下已发表的 8 组评测配置成绩（覆盖 8 个独立模型，采用原生分值口径（不计入准确率饱和度分档））；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.24098v1","published_at":"2026-11-18","status":"paper_baselines"},"model_count":8,"result_count":8,"score_min":0,"score_max":1,"score_decimals":3,"results":[[2602,0.354,370,18660,18661],[18662,0.358,370,18663,18661],[3184,0.315,370,18664,18661],[3196,0.336,370,18665,18661],[6412,0.351,370,18666,18661],[4961,0.336,370,18667,18661],[7937,0.308,370,18668,18661],[9492,0.326,370,18669,18661]]},{"id":"hygeobench-2026-paper","name":"HyGeoBench (2026 paper)","group":"视觉与视频","metric":"OA","unit":"score","public_history":true,"protocol":{"en":"Primary-source results extracted from HyGeoBench (arXiv:2611.24068, table S4.T2, published 2026-11-18). Retains 24 published evaluation configurations across 23 distinct models on 'OA' (native score scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《HyGeoBench》原始论文（arXiv:2611.24068，表 S4.T2，发布日期 2026-11-18）。完整保留“OA”列下已发表的 24 组评测配置成绩（覆盖 23 个独立模型，采用原生分值口径（不计入准确率饱和度分档））；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.24068v1","published_at":"2026-11-18","status":"paper_baselines"},"model_count":23,"result_count":24,"score_min":0,"score_max":100,"score_decimals":2,"results":[[3512,31.31,370,18670,18671],[18672,15.67,370,18673,18671],[17731,12.13,370,18674,18671],[18675,17.83,370,18676,18671],[17069,18.16,370,18677,18671],[18678,24.51,370,18679,18671],[5725,32.03,370,18680,18671],[2654,28.22,370,18681,18671],[18682,29.44,370,18683,18671],[4070,32.97,370,18684,18671],[2959,48.2,370,18685,18671],[18686,48.05,370,18687,18671],[2968,42.0,370,18688,18671],[2982,47.64,370,18689,18671],[3512,38.29,370,18690,18671],[2967,45.79,370,18691,18671],[3529,41.69,370,18692,18671],[5361,45.12,370,18693,18671],[18694,41.4,370,18695,18671],[2960,42.28,370,18696,18671],[18697,39.64,370,18698,18671],[18699,46.23,370,18700,18671],[18701,44.77,370,18702,18671],[18703,50.62,370,18704,18671]]},{"id":"futurebench-2026-paper","name":"FutureBench (2026 paper)","group":"知识与推理","metric":"Verify","unit":"%","public_history":true,"protocol":{"en":"Primary-source results extracted from FutureBench (arXiv:2611.23601, table S5.T2, published 2026-11-17). Retains 11 published evaluation configurations across 6 distinct models on 'Verify' (comparable percentage scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《FutureBench》原始论文（arXiv:2611.23601，表 S5.T2，发布日期 2026-11-17）。完整保留“Verify”列下已发表的 11 组评测配置成绩（覆盖 6 个独立模型，采用可比百分比口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.23601v1","published_at":"2026-11-17","status":"paper_baselines"},"model_count":6,"result_count":11,"results":[[1573,0.559,370,18705,18706],[18707,0.65,370,18708,18706],[351,0.658,370,18709,18706],[3748,0.625,370,18710,18706],[5929,0.642,370,18711,18706],[1573,0.634,370,18712,18706],[18707,0.625,370,18713,18706],[351,0.824,370,18714,18706],[3748,0.78,370,18715,18706],[5929,0.837,370,18716,18706],[18717,0.56,370,18718,18706]]},{"id":"can-llms-understand-what-they-2026-paper","name":"Can LLMs Understand What They (2026 paper)","group":"专业工作","metric":"Propagation","unit":"%","public_history":true,"protocol":{"en":"Primary-source results extracted from Can LLMs Understand What They (arXiv:2611.23533, table S4.T6, published 2026-11-17). Retains 6 published evaluation configurations across 6 distinct models on 'Propagation' (comparable percentage scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《Can LLMs Understand What They》原始论文（arXiv:2611.23533，表 S4.T6，发布日期 2026-11-17）。完整保留“Propagation”列下已发表的 6 组评测配置成绩（覆盖 6 个独立模型，采用可比百分比口径）；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.23533v1","published_at":"2026-11-17","status":"paper_baselines"},"model_count":6,"result_count":6,"results":[[2058,0.1336,370,18719,18720],[5136,0.0504,370,18721,18720],[587,0.6464,370,18722,18720],[5648,0.3442,370,18723,18720],[5344,0.2805,370,18724,18720],[786,0.0732,370,18725,18720]]},{"id":"rfbench-2026-paper","name":"RFBench (2026 paper)","group":"视觉与视频","metric":"Strict all ≤ 3 dB","unit":"score","public_history":true,"protocol":{"en":"Primary-source results extracted from RFBench (arXiv:2611.23410, table A9.T8, published 2026-11-17). Retains 7 published evaluation configurations across 7 distinct models on 'Strict all ≤ 3 dB' (native score scale); non-model baselines and incomplete rows are excluded without zero-imputation.","zh":"数据直接提取自《RFBench》原始论文（arXiv:2611.23410，表 A9.T8，发布日期 2026-11-17）。完整保留“Strict all ≤ 3 dB”列下已发表的 7 组评测配置成绩（覆盖 7 个独立模型，采用原生分值口径（不计入准确率饱和度分档））；非模型基线（如随机或人工参考）及缺失单元格均严格剔除，不做补零处理。"},"source":{"name":"Paper authors","url":"https://arxiv.org/abs/2611.23410v1","published_at":"2026-11-17","status":"paper_baselines"},"model_count":7,"result_count":7,"score_min":0,"score_max":100,"score_decimals":2,"results":[[18726,0.0,370,18727,18728],[18729,0.4,370,18730,18728],[4958,0.8,370,18731,18728],[18732,0.4,370,18733,18728],[6342,0.0,370,18734,18728],[18735,1.6,370,18736,18728],[18737,2.8,370,18738,18728]]}]}
