SentinelBench evaluates AI agents on long-running monitoring tasks: 100 tasks across 10 synthetic web environments including email, team chat, professional networking, code hosting, stock trading, photo sharing, video, music, calendar, and academic search. Each task replays a scripted sequence of events, so pages change underfoot while the agent watches, waits, and acts at the right moment. Success is scored against the environment's database state; the benchmark also measures reaction time and token cost. By Matheus Kunzler Maldaner, Adam Fourney, Amanda Swearngin, Hussein Mozannar, Gagan Bansal, Maya Murad, Rafah Hosn, and Saleema Amershi (Microsoft Research AI Frontiers).
Paper (arXiv:2606.05342) · Code on GitHub · Microsoft Research blog: "Tell me when"
Three models, each paired with a plain sleep(time) tool or a purpose-built
wait_for(condition, timeout) tool. Success rate over all 100 tasks; median API
cost per task (prices as of May 15, 2026); median reaction time on successful tasks.
| Model | Waiting tool | Success rate | Median cost / task | Median reaction time |
|---|---|---|---|---|
| GPT-5.4 (low reasoning) | wait_for | 75% | $0.23 | 51.7 s |
| GPT-5.4 (low reasoning) | sleep | 68% | $1.17 | 42.3 s |
| GPT-4o | wait_for | 48% | $0.13 | 22.8 s |
| GPT-4o | sleep | 46% | $0.29 | 48.6 s |
| Qwen 3.5:9B | wait_for | 48% | $0.01 | 60.1 s |
| Qwen 3.5:9B | sleep | 49% | $0.02 | 123.8 s |
When tasks stretch to 40 minutes (speed_factor 0.25), GPT-5.4 with wait_for
solves 69/100 tasks at a median $0.48 per task, while sleep solves 56/100 at
$4.65 — 9.7× more expensive for 13 fewer tasks.
This site is a playable demo: open an environment and try a real benchmark task yourself. MicroMail (email), MicroChat (team chat), MicroDin (professional network), MicroHub (code hosting), MicroHood (stock trading), MicroGram (photo sharing), MicroTube (video), MicroFy (music), MicroLendar (calendar), MicroScholar (academic search).