Overview
Pipeline heartbeats provide a way to monitor the health of your pipeline by sending periodic heartbeat frames through the system. When enabled, the pipeline will send heartbeat frames every second and monitor their progress through the pipeline.Enabling Heartbeats
Heartbeats can be enabled by settingenable_heartbeats to True in the PipelineParams:
How It Works
When heartbeats are enabled:- The pipeline sends a
HeartbeatFrameevery second - The frame traverses through all processors in the pipeline, from source to sink
- The pipeline monitors how long it takes for heartbeat frames to complete their journey
- If a heartbeat frame isn’t received within 10 seconds, a warning is logged
Monitoring Output
The system will log:- Trace-level logs showing heartbeat processing time
- Warning messages if heartbeats aren’t received within the monitoring window
Use Cases
Heartbeat monitoring is useful for:- Detecting pipeline stalls or blockages
- Monitoring processing latency through the pipeline
- Identifying performance issues in specific processors
- Ensuring the pipeline remains responsive
Configuration
The heartbeat system uses two timing values:- Interval (default 1.0s) — how often heartbeat frames are sent. Configurable via
heartbeats_period_secsinPipelineParams. - Monitor window (10x the interval) — how long to wait before logging a warning if no heartbeat is received.
The heartbeat interval is configurable via the
heartbeats_period_secs parameter in PipelineParams. The monitor window is always 10x the interval.