Monitoring
Create a custom monitor
Get alerted when a pipeline or table processes too few or too many rows, falls behind on ingestion, or grows a PostgreSQL replication slot.
Custom monitors sit next to the checks Artie runs on every pipeline for source, replication and destination health. Users with the View only role see monitors but can’t create, edit or delete them.
Create a monitor
Open the monitor form
In the Artie dashboard, open Monitors and select New monitor.
To start from one pipeline, open the pipeline and select New monitor in its Monitors section. The form then shows that pipeline in place of the Pipeline field.
Choose a pipeline and monitor type
Select a Pipeline, then a Monitor type. Draft pipelines aren’t listed.
Large PostgreSQL replication slot appears only when the pipeline’s source is PostgreSQL.
Optionally limit the monitor to a table
To watch one table, select it under Table (optional). Leave the field empty to watch the whole pipeline.
The field appears for low data volume, high data volume, and high ingestion lag monitors. Pipelines with a Kafka destination support pipeline-wide monitors only.
Name the monitor
Artie fills in Name from the monitor type, pipeline and table. Edit it if you want, up to 255 characters.
Alert emails for data volume and ingestion lag monitors include the text in Description.
Preview the condition and configure alerts
Use the chart preview to check the condition against the last 24 hours, or pick another time range.
Set the Detection method and its fields, then choose Who to notify. Each field is described in Detection and notifications.
Save the monitor
Select Create monitor. Artie saves the monitor and runs it.
The monitor appears in the Custom monitors table with the status Not run yet, Healthy or Needs attention.
Monitor types
Each type evaluates one metric, at the scope shown in the last column.
| Monitor | What it evaluates | Threshold unit | Scope |
|---|---|---|---|
| Low data volume | Fewer processed rows than the threshold or expected range | Rows | Pipeline or table |
| High data volume | More processed rows than the threshold or expected range | Rows | Pipeline or table |
| High ingestion lag | Ingestion lag at or above the threshold, or above its expected range | Minutes | Pipeline or table |
| Large PostgreSQL replication slot | A replication slot’s size at or above the threshold | GB | PostgreSQL pipeline |
A pipeline with a Large PostgreSQL replication slot monitor uses that monitor’s fixed threshold in place of the built-in slot-size check. The built-in check alerts at 5% of your database size, kept between 10 GB and 100 GB. If 65% of max_slot_wal_keep_size is lower, it alerts there instead. When Artie can’t read the database’s size, it alerts at 10 GB. See Replication slot keeps growing.
Default low data volume monitor
Twelve hours after a pipeline’s first deploy, Artie adds a low data volume monitor if the pipeline doesn’t have one. The monitor is named Low data volume - PIPELINE_NAME, where PIPELINE_NAME is the pipeline’s name.
It uses the Threshold method with a threshold of 1 row and a rollup window of 720 minutes. It alerts everyone when the pipeline has no data for 12 hours. Edit its threshold or delete it if that alert doesn’t fit the pipeline.
Detection and notifications
Low data volume, high data volume, and high ingestion lag monitors use one of two detection methods. New monitors start on Anomaly detection.
- Anomaly detection alerts when the metric departs from its expected range.
- Threshold alerts when the configured value is crossed.
Large PostgreSQL replication slot monitors always use a threshold.
Anomaly detection settings
These three fields appear when Detection method is Anomaly detection.
| Field | Default | What it sets |
|---|---|---|
| Tolerance | 2 | Standard deviations a data point must be from the expected value to count as an anomaly |
| Trigger window (minutes) | 15 | Minutes the data must stay anomalous before the monitor alerts |
| Recovery window (minutes) | 15 | Minutes the data must stay in its normal range before the monitor recovers |
Threshold settings
The threshold fields depend on the monitor type.
| Monitor | Field | Default | What it sets |
|---|---|---|---|
| Low and high data volume | Threshold (rows) | 1 | Row count that triggers the alert |
| Low and high data volume | Rollup window (minutes) | 60 | Minutes of processed rows that Artie sums and compares to the threshold |
| High ingestion lag | Threshold (minutes) | 1 | Lag that triggers the alert, compared with the lag over the last minute |
| Large PostgreSQL replication slot | Threshold (GB) | 10 | Slot size that triggers the alert |
| Large PostgreSQL replication slot | Alert | After 10 minutes | How long the slot must stay at or above the threshold |
Alert accepts Immediately, After 10 minutes, After 20 minutes or After 30 minutes.
A low data volume threshold monitor doesn’t alert until the pipeline is older than its rollup window. Saving a change to the rollup window re-evaluates the alert with the new window.
Notifications
Who to notify sets which workspace members get the alert email: Everyone, No one or Specific users. With Specific users, pick the members under Users. Members who opted out of alert emails don’t receive them.
Two checkboxes appear only when the workspace has the matching setup:
- Also notify this workspace’s additional alert emails appears when the workspace has additional alert emails. It’s on by default.
- Also notify PagerDuty appears when PagerDuty is connected. It’s off by default.
A pipeline’s setting to disable alert emails doesn’t apply to custom monitors.
Monitor history
In Monitors, select a monitor’s name in the Custom monitors table to open its detail page. The page shows the pipeline, the table if one is set, the alert threshold, and a chart for the selected time range.
Alert history lists each alert with Triggered at, Status (Active or Resolved) and Resolved at. Use it to check whether a threshold or anomaly setting reflects normal pipeline behavior before changing the monitor.
Edit, clone or delete a monitor
On the monitor’s detail page:
- Select Edit monitor to change its settings. The pipeline and monitor type stay fixed.
- Select More options > Clone monitor to copy it to the same or another pipeline. Artie adds
(clone)to the name. - Select More options > Delete monitor, then confirm. The monitor stops alerting, and the deletion can’t be undone.
Artie deletes a table monitor on its next run if the monitor’s table no longer exists.
Was this page helpful?
- Need help? Message us on Slack or email [email protected].
- Want a walkthrough? Book a demo.
- LLM? Read llms.txt.