# SentinelBench

> SentinelBench is an open-source benchmark from Microsoft Research AI Frontiers that evaluates AI agents on 100 long-running monitoring tasks across 10 synthetic web environments. Agents must wait, watch, and act when scripted events change the environment underfoot; success is scored against the environment's database state, and the benchmark also measures reaction time and token cost. Paper: arXiv:2606.05342 (June 2026).

Key facts:
- 100 tasks across 10 environments: MicroMail (email), MicroChat (team chat), MicroDin (professional network), MicroHub (code hosting), MicroHood (stock trading), MicroGram (photo sharing), MicroTube (video), MicroFy (music), MicroLendar (calendar), MicroScholar (academic search)
- Task axes: passive vs. active action requirement, absolute vs. relative success criteria; 20 no-op tasks whose condition never occurs punish agents that declare success blindly
- Each task replays a scripted event timeline over a 10-minute window by default; a speed_factor stretches tasks arbitrarily (e.g., 40 minutes)
- Best baseline: GPT-5.4 (low reasoning) with a wait_for(condition, timeout) tool: 75% success, $0.23 median API cost per task, 51.7 s median reaction time
- Tool design matters as much as model choice: on 40-minute tasks, wait_for solves 69/100 at $0.48 median while a plain sleep(time) tool solves 56/100 at $4.65 — 9.7x more expensive for 13 fewer tasks
- Authors: Matheus Kunzler Maldaner (University of Florida), Adam Fourney, Amanda Swearngin, Hussein Mozannar, Gagan Bansal, Maya Murad, Rafah Hosn, Saleema Amershi (Microsoft Research)

## Resources

- [Paper, arXiv:2606.05342](https://arxiv.org/abs/2606.05342): full benchmark description, task generation, and baseline results
- [Code repository](https://github.com/microsoft/sentinel_environments): environments, task scenarios, synthetic catalogs, and evaluation harness (MIT license)
- [Microsoft Research blog: "Tell me when"](https://www.microsoft.com/en-us/research/blog/tell-me-when-building-agents-that-can-wait-monitor-and-act/): motivation and agent design for monitoring tasks
- [Playable demo](https://sentinelbench.com/): try real benchmark tasks yourself in the browser
- [llms-full.txt](https://sentinelbench.com/llms-full.txt): extended summary with full baseline tables and example task prompts

## Citation

```bibtex
@misc{maldaner2026sentinelbench,
  title={SentinelBench: A Benchmark for Long-Running Monitoring Agents},
  author={Matheus Kunzler Maldaner and Adam Fourney and Amanda Swearngin and Hussein Mozannar and Gagan Bansal and Maya Murad and Rafah Hosn and Saleema Amershi},
  year={2026},
  eprint={2606.05342},
  archivePrefix={arXiv},
  primaryClass={cs.AI}
}
```