Commit Graph

12 Commits

Author SHA1 Message Date
cthomas
1d360cc239 feat: dump thread state on event loop hang (#8388) 2026-01-12 10:57:48 -08:00
cthomas
8da7a08861 feat: add more granular logging for event loop (#8318) 2026-01-12 10:57:48 -08:00
cthomas
c66b852978 feat: add more data to event loop logging (#8299) 2026-01-12 10:57:48 -08:00
cthomas
8f4074d2fc feat: dump traces on event loop saturation (#8185) 2026-01-12 10:57:48 -08:00
cthomas
a6dd98d843 feat: add event loop saturation detection (#8177)
Co-authored-by: jnjpng <jin@letta.com>
2026-01-12 10:57:48 -08:00
Sarah Wooders
c9ad2fd7c4 chore: move things to debug logging (#6610) 2025-12-15 12:03:09 -08:00
Kian Jones
94c2921711 chore: walk back some temporary debugging stuff (#6332)
* first pass

* uv lock
2025-11-24 19:10:27 -08:00
Kian Jones
ea845d379d feat: improved watchdog thread (#6252)
log even when event loop is not blocked to clarify whether event loop is blocked during 'freeze' and attempt to estimate main event loop load
2025-11-24 19:09:33 -08:00
Kian Jones
19210c01b0 feat: add stacktrace dumping on event loop block (#6211)
Add stack trace dumping to watchdog

When a hang is detected, now logs:
- Full stack trace of all threads
- Exact file:line of blocking code
- Function names in the call chain

This shows you WHY the event loop is blocked, not just that it's blocked.
2025-11-24 19:09:33 -08:00
Kian Jones
71bce718f7 Add lightweight event loop watchdog monitoring (#6209)
* Add lightweight event loop watchdog monitoring

- Thread-based watchdog detects event loop hangs >15s
- Runs independently, won't interfere with normal operation
- Disabled in test environments
- Minimal overhead, just heartbeat checks every 5s

* actually test it

* Add test script to validate watchdog detects hangs

Run with: uv run python test_watchdog_hang.py

Tests:
- Normal operation (no false positives)
- Short blocks under threshold (no alerts)
- Long blocks over threshold (correctly alerts)
2025-11-24 19:09:33 -08:00
Kian Jones
a5e435c56f fix: register memory monitor at startup (#6195)
register monitor at startup
2025-11-24 19:09:32 -08:00
Kian Jones
848aa962b6 feat: add memory tracking to core (#6179)
* add memory tracking to core

* move to asyncio from threading.Thread

* remove threading.thread all the way

* delay decorator monitoring initialization until after event loop is registered

* context manager to decorator

* add psutil
2025-11-24 19:09:32 -08:00