Mizan bias benchmark: testing AI models
Mizan is a bias benchmark that tests whether AI models apply double standards on sensitive topics.
Why this: Covers model and benchmark · Published within 6 hours
35 items · All topics →
Mizan is a bias benchmark that tests whether AI models apply double standards on sensitive topics.
Why this: Covers model and benchmark · Published within 6 hours
Moonshot AI stopped chasing coding benchmark scores and is steering its Kimi models toward biology, chemistry and pharma research, where science tasks demand more capable models.
Why this: Covers model and benchmark · Published within 24 hours
In its MLPerf debut the Vera Rubin platform led across training, post-training and agentic inference, with NVIDIA sweeping every category.
Why this: Official source · Covers inference and Agent · Published within 24 hours
This study pits heuristics written by a frontier LLM coding agent against commercial solvers on unsolved job-shop scheduling instances.
Why this: Covers model and Agent · Relevant to this topic
This dataset uses real-world privacy fixes to test whether coding agents catch the privacy defects that regulations like the GDPR require software to avoid.
Why this: Covers Agent and benchmark · Relevant to this topic
This benchmark places LLM agents in heterogeneous multi-agent economies to test long-horizon task handling and even business management.
Why this: Covers LLM and Agent · Relevant to this topic
OCTAVE breaks end-to-end HTML5 game generation into capability-resolved evaluation dimensions instead of a single aggregate score, testing whether an agent can coordinate rules, interaction, state, and presentation.
Why this: Covers Agent and benchmark · Relevant to this topic
This benchmark continuously tests LLMs on financial capability using the latest Japanese certification exams, keeping the questions fresh.
Why this: Covers model and LLM · Relevant to this topic
This benchmark tests whether LLMs can spot experimental protocols that read fluently but are physically infeasible.
Why this: Covers model and LLM · Relevant to this topic
This work looks beyond binary task success for coding agents, analyzing behavioral trajectories to reveal how success and failure arise.
Why this: Covers Agent and benchmark · Relevant to this topic
This study notes that deployed models face contested answers while benchmarks test them alone, and proposes learning through self-disagreement.
Why this: Covers model and benchmark · Relevant to this topic
Using the reference agent bundled with the Vals Finance Agent Benchmark, the researchers test whether it drifts, how to diagnose that, and how to intervene in an auditable way.
Why this: Covers Agent and benchmark · Relevant to this topic
This framework provides realistic, controllable and reproducible simulation plus scalable benchmarking for e-commerce web agents.
Why this: Covers Agent and benchmark · Relevant to this topic
This study evaluates LLMs' risk propensity across contexts and task demands, systematically measuring whether models lean risk-averse or risk-tolerant.
Why this: Covers model and LLM · Relevant to this topic
BigCodeBench evaluates LLM code generation with tasks involving diverse function calls and complex instructions, closer to real development than benchmarks that only test simple snippets.
Why this: Covers model and LLM · Relevant to this topic
This work proposes a portable screen that first checks whether an LLM's confidence signal actually varies with correctness before using it for abstention, routing, and escalation decisions.
Why this: Covers LLM and benchmark · Relevant to this topic
REFORGE benchmarks LLM reverse-engineering capability through decompiled binary function naming, responding to real threat-intelligence reports of LLMs being used for reverse engineering.
Why this: Covers model and LLM · Relevant to this topic
Coding agents often stumble at the finish line — most of a feature built, yet a requirement dropped or tests left incomplete — and RL trained on coding tasks carries across benchmarks, the study finds.
Why this: Covers Agent and benchmark · Relevant to this topic
CastWorld moves agentic time-series forecasting beyond single-shot number prediction toward self-directed, context-aware interaction, and judges it on more than accuracy alone.
Why this: Covers Agent and benchmark · Relevant to this topic
When IoT time series meet external text like expert reports, this benchmark diagnoses how forecasting models behave when those signals agree, conflict, or go unspecified.
Why this: Covers multimodal and benchmark · Relevant to this topic
When GPU rental marketplace quotes go missing, daily price benchmarks can distort, and this study stress-tests the common median rule under four controlled missingness patterns.
Why this: Covers benchmark · Relevant to this topic
This synthetic benchmark tests whether causal discovery methods can find which recorded variables truly drive product quality, yield or energy use.
Why this: Covers benchmark · Relevant to this topic
This benchmark tests frontier models on 70 realistic workplace requests across 13 domains, each graded against an expert rubric.
Why this: Covers model and benchmark · Relevant to this topic
This benchmark brings Chinese grammatical error correction to handwritten essay images, requiring edits grounded in the image rather than text-to-text conversion.
Why this: Covers benchmark · Relevant to this topic
LEGAL-UQA provides a parallel English-Urdu benchmark for testing whether legal information assistants reliably help Urdu speakers understand their constitutional rights.
Why this: Covers benchmark · Relevant to this topic
ViroGym gives protein language models a realistic large-scale benchmark to systematically test their zero-shot prediction of viral protein mutation effects.
Why this: Covers model and benchmark · Relevant to this topic
TransitLM provides a large-scale dataset and benchmark so models can plan public transit routes without structured map infrastructure.
Why this: Covers benchmark · Relevant to this topic
URSA provides a chemistry-aware benchmark for evaluating synthesis planning that finds reaction pathways for target molecules, serving retrosynthesis in drug discovery.
Why this: Covers benchmark · Relevant to this topic
NaijaFacts addresses the flaw that multilingual benchmarks for African languages mostly test translation rather than local knowledge, specifically testing models' grasp of facts about Nigeria.
Why this: Covers benchmark · Relevant to this topic
Abacus.AI ran one fine-tuning recipe across three very different open-weight models and saw the improvements land in totally different spots — a more telling result than any single benchmark score.
Why this: Covers model and benchmark · Relevant to this topic
The new model doubles success rates on some tasks at best, while DeepMind also released an open safety benchmark to probe jailbreak risk.
Why this: Covers model and Gemini · Relevant to this topic
OpenAI's new BioBasicsBench runs agents through 100 tasks to gauge how they interpret raw lab data and tackle routine wet-lab work.
Why this: Official source · Covers Agent and benchmark
Phonon TTS brings high-quality speech synthesis on-device with leading benchmark results, suited to latency- and privacy-sensitive scenarios.
Why this: Official source · Covers benchmark
LifeVal is OpenAI's new benchmark for gauging how well AI helps with everyday tasks — finding childcare, managing finances, handling government paperwork, and the like.
Why this: Official source · Covers benchmark
The piece runs both databases on an everyday laptop: response times land between a few milliseconds and a few dozen, with the precise numbers varying by CPU, memory, and workload.
Why this: Covers benchmark · Relevant to this topic