Scenarios
You can configure alarm policies based on logs or metrics to send alarms when specific criteria are met, for example, excessive number of error logs or abnormally long API response times.
Prerequisites
Data has been uploaded to log topics.
Operation Steps
Log in to the CLS console and go to the Alarm Policy management page. Click New in the upper-left corner and configure the following main information. 1. Basic Information
Alarm Name: specifies a string of 1 to 255 characters in length.
Enabling Status: specifies whether to enable the current alarm policy. When the alarm policy is disabled, no alarm or notification will be generated.
Tag: Tags are assigned to the current alarm policy for multi-dimensional resource classification and management.
Configuration Method:
Custom Configuration: This method requires you to manually specify the log topics to monitor, the query statements, and the trigger conditions. It offers high flexibility but is slightly more complex to configure. This configuration method is described in detail later in this document.
Template-based Configuration: This method uses preset templates to create alarm policies, eliminating the need to manually write query statements and trigger conditions, making configuration simple. Currently, it only supports TKE audit logs and event logs. Support for logs from more cloud products will be added in the future. For details, see Alarm Policy Templates. 2. Monitoring Object
Select the log or metric topics you want to monitor. Two selection methods are supported:
All Statements Use the Same Monitoring Object (Default): This option supports multiple selections, allowing you to select up to 20 topics. Cross-region selection is not supported. The system evaluates each topic individually to determine whether it meets the trigger conditions. If multiple topics meet the trigger conditions simultaneously, multiple alarms are generated concurrently.
Specify a monitored object for each statement: This option allows you to set a topic individually for each execution statement. Only single selection is supported, and cross-region selection is not allowed. The results of each execution statement are aggregated to determine whether the trigger conditions are met. For example, an alarm is triggered when the number of error logs in Topic A plus the number of error logs in Topic B is greater than 10.
3. Monitoring Task
Execution statement: refers to a query statement that operates on log or metric topics.
Query statement: refers to a log search and analysis statement. For details, see Syntax Rules.
Example 1: Obtains logs with error status through status:error.
Example 2: Calculate the average response time for the domain `domain:aaa.com` using domain:"aaa.com" | select avg(request_time) as Latency.
Query Time Range: The maximum time range is the last 24 hours.
Query statement: PromQL. For details, see Syntax Rules.
Example 1: An alarm is triggered when the filesystem usage percentage is greater than 50%: filesystem_usage_pct > 0.5.
Example 2: An alarm is triggered when the maximum CPU load over the last 5 minutes is greater than 1: max_over_time(node_load1[5m]) > 1.
Trigger condition: An alarm is triggered when the trigger condition are met. Two configuration modes are supported: Interaction and Statement
Interaction Mode:
Select the alarm severity level, execution statement, variable name, and comparison method in sequence. Then, set the alarm threshold.
Multiple alarm severity levels are supported. When a condition is met, an alarm of the corresponding level is triggered. If multiple conditions are met simultaneously, multiple alarms will be triggered. Different notification rules can be configured in a notification channel group based on the alarm level. For example, "Critical" alarms are notified via phone calls, "Warning" alarms are notified via SMS, and "Info" alarms are not notified.
Statement Mode:
Alarm severity levels: Three levels are supported: Info, Warning, and Critical. When are met, an alarm of the corresponding level is triggered. If multiple condition are met simultaneously, multiple alarms will be triggered.
Trigger by group: specifies whether to enable the group-based trigger feature.
When the feature is enabled, if an execution statement returns multiple results, the system groups these results by the specified grouping field. Each group is evaluated independently against the trigger condition. If a group meets the condition, a separate alarm is generated for that group.
Grouping Field(s): Select at least one field from the result of the execution statement to serve as the basis for grouping.
Example:
Execution statement: * | select avg(request_time) as request_time, domain group by domain order by request_time desc limit 5.
Trigger condition: $1.request_time > 5.
Preview:
A total of one alarm was triggered:
|
aaa.com | 12.56 | Criterion met; one consolidated alarm triggered |
bbb.com | 9.45 |
|
ccc.com | 7.23 |
|
ddd.com | 5.21 |
|
eee.com | 4.78 | Criterion not met; no alarm triggered |
$1.domain is used as a group field. Five groups are generated, of which groups 1 to 4 meet the trigger conditions. Each group triggers one alert, resulting in a total of 4 alerts.
|
aaa.com | Group 1 | 12.56 | Criterion met; one alarm separately triggered |
bbb.com | Group 2 | 9.45 | Criterion met; one alarm separately triggered |
ccc.com | Group 3 | 7.23 | Criterion met; one alarm separately triggered |
ddd.com | Group 4 | 5.21 | Criterion met; one alarm separately triggered |
eee.com | Group 5 | 4.78 | Criterion not met; no alarm triggered |
Note:
When group-based trigger is enabled, a single alarm execution may have a large number of results meeting the trigger condition, potentially generating numerous simultaneous alarms and causing an alarm storm. Therefore, please configure the grouping field(s) and trigger condition appropriately.
When specifying grouping field(s), you can divide execution results into up to 1,000 groups. No alarms will be triggered for excessive groups.
Execution Cycle: refers to the execution frequency of the monitoring task. Two configuration modes are supported:
|
Fixed interval | Executes the monitoring task at fixed time intervals. Supported range: 1 to 1440 minutes. | Executes the monitoring task every 5 minutes. |
Fixed time | Executes the monitoring task at a fixed time point. Supported range: 00:00 to 23:59. | Executes the monitoring task daily at 02:00. |
Cron expression | Executes the monitoring task according to a Cron expression. Minimum granularity: minute. | */5 8-20 * * *
Executes the monitoring task every 5 minutes from 08:00 to 20:00 daily. |
4. Multi-dimensional Analysis
When an alarm is triggered, you can use multi-dimensional analysis to perform further analysis on raw logs, and add the analysis result to the alarm notification to help locate the cause of the alarm. Multi-dimensional analysis does not affect the alarm trigger condition.
|
| Obtains the raw logs that meet the search condition of the execution statement. The log fields to display, the number of entries, and the display format can be configured. For example, when an alarm is triggered due to excessive error logs, you can view the detailed logs in the alarm. |
TOP 5 log fields and proportion statistics | For all the logs within the time range when the alarm is triggered, groups logs by specified fields and counts entries to obtain top 5 field values and their percentages. For example, when an alarm is triggered due to excessive error logs, you can view the top 5 URLs and top 5 response status codes. |
Custom Search and Analysis | For all logs within the time range when the alarm is triggered, execute a custom search and analysis statement.
Example 1: Obtain the time consumed by each API request via * | select avg(timeCost) as time,URL group by URL.
Example 2: Obtain error logs via status:>499. |
Note:
"Raw Logs" and "TOP 5 log fields and proportion statistics" automatically associate with the search conditions specified in the query statement (excluding SQL filtering conditions such as where). This indicates that multi-dimensional analysis needs to be performed on the raw logs that meet the conditions.
Multi-dimensional analysis is not currently supported for metric topics.
5. Alarm Notification
Alarm Frequency:
Duration: A notification will be sent only after the trigger condition is met constantly for a specified number of times (default value: 1, valid range: 1–10).
Interval: After a notification is sent, no additional notifications will be sent within the specified interval. For example, alarming once every 15 minutes means that at most one alarm notification will be received within a 15-minute window.
Notification Method:
It supports notification methods such as SMS, email, phone call, WeChat, WeCom, DingTalk, Lark, and custom API callback (webhook). Two configuration types are supported, and you only need to select one.
|
| Used only in CLS. | It focuses only on log monitoring alarms and does not cover other types of monitoring alarms in TCOP, such as cloud product monitoring alarms. |
| Created and shared in TCOP. | You need to monitor both logs and other types of monitoring alarms in TCOP, such as cloud product monitoring alarms. Use the On-call Schedule and Alarm Message Delivery to SCF features. |
Extra Category Field:
Classify alarms using key-value pairs in the format key:value. Keys support only letters, digits, and underscores, and must start with a letter. Maximum key length: 50 characters. Maximum value length: 200 characters. You can configure different notification rules based on the fields in the notification groups, or use {{.Label}} to reference the fields in additional notification content and custom API callbacks.
For example, you can set a classification field as service:payService (indicating that the service name for the current alarm is payService).
Configure notification rules in the notification channel group to send alarms to the corresponding team when the service field contains payService.
Use {{.Label}} or {{.Label.service}} in custom API callback configuration to obtain the field value ({"service":"payService"} or payService, respectively). The third-party platform can identify the service name for this alarm when receiving the callback.
Additional notification content:
Add extra text to alarm notifications to enrich their content. You can dynamically generate this text using alarm notification variables. Detailed Explanation of Execution Cycle and Query Time Range
CLS automatically schedules the actual execution time of alarm policies to avoid performance degradation caused due to simultaneous execution of multiple alarm policies. Therefore, the actual execution time of alarm policies may vary by seconds from the specified alarm execution cycle.
For example, two alarm policies created at 12:00:00, both set to be executed every minute, may have slightly different actual execution times:
Actual execution times for the first alarm policy: 12:01:12, 12:02:12, 12:03:12, 12:04:11, 12:05:11, and so on
Actual execution times for the second alarm policy: 12:01:27, 12:02:27, 12:03:27, 12:04:27, 12:05:27, and so on.
To ensure no alarms are missed during automatic scheduling, the alarm query time range will be automatically rounded and an additional 30-second query delay will be added. (Since data reporting typically has a delay of approximately 10 seconds, strictly executing alarms based on actual times may cause missed alarms due to unreported latest data).
For example, if an alarm policy executes once every minute and queries data from the most recent minute each time, the actual execution times and corresponding query time ranges are as follows:
|
12:01:12 | 11:59:30 – 12:00:30 |
12:02:12 | 12:00:30 – 12:01:30 |
12:03:12 | 12:01:30 – 12:02:30 |
12:04:11 | 12:02:30 – 12:03:30 |
12:05:11 | 12:03:30 – 12:04:30 |
Cron Expression
Alarm policy execution cycles support Cron expressions with a minimum granularity of minutes. A Cron expression consists of five space-separated time fields, each representing a time unit. The structure is as follows:
The description of each field is as follows:
Field 1: Minute (0–59)
Field 2: Hour (0–23)
Field 3: Day (1–31)
Field 4: Month (1–12)
Field 5: Day of Week (0–6) (0 represents Sunday)
Note:
Cron expressions are executed in the UTC+8 time zone across all regions.
Special characters can be used in expressions to represent more complex time rules:
*: matches any value. For example, using * in the Minute field means that the task executes once every minute.
?: indicates no specific value and can only be used in the Day and Day of Week fields.
–: indicates a range. For example, using 5–20 in the Minute field means that the task executes once every minute between from minute 5 to minute 20.
/: indicates execution starting from a specified time at fixed intervals. For example, using 5/20 in the Minute field means that the task executes once every 20 minutes starting from minute 5, that is, at minutes 5, 25, and 45.
,: lists enumeration values. If 5,20 is used for the Minutes field, a task is executed at 5 and 20 minutes, respectively.
Example:
|
0 12 * * * | Executes once daily at 12:00 (noon). |
15 10 * * * | A task is executed once at 10:15 AM every day. |
* 14 * * * | Executes once every minute daily from 14:00 to 14:59. |
0/5 14 * * * | Executes once every 5 minutes daily from 14:00 to 14:55. |
0/5 14,18 * * * | A task is executed every 5 minutes between 14:00 and 14:55 and between 18:00 and 18:55 every day. |
0-5 14 * * * | Executes once every minute daily from 14:00 to 14:05. |
10,44 14 * 3 3 | Executes at 14:10 and 14:44 every Wednesday in March. |
15 10 * * 1-5 | Executes once daily at 10:15 AM from Monday to Friday. |
Best Practices