Monitoring policies

The rate and number of incoming events serve as an important indicator of the state of the system. For example, you can detect when there are too many events, too few, or none at all. Monitoring policies are designed to detect such situations. In a policy, you can specify a lower threshold, an optional upper threshold, and the way the events are counted: by frequency or by total number.

The policy must be applied to the event source. After applying the policy, you can monitor the status of the source: green means everything is OK, red means the stream is outside the configured threshold. If the status is red, an event of the Monitoring type generated. The event is generated in the Main tenant. Users with the General administrator role and other roles, if they have access to the Main tenant, can search for events of the Monitoring type. You can also configure notifications to be sent to an arbitrary email address. Policies for monitoring the sources of events are displayed in the table under Source status → Monitoring policies. You can sort the table by clicking the column header of the relevant setting. Clicking a policy opens the data area with policy settings. The settings can be edited.

Algorithm for applying a monitoring policy

Monitoring policies are applied to an event source in accordance with the following algorithm:

1. The event stream is counted at the collector.
2. The Core server gets information about the stream from the collectors every 15 seconds.
3. The obtained data is stored on the Core server in the Victoria Metrics time series database, and the data storage depth on the Core server is 15 days.
4. An inventory of event sources is taken once per minute.
5. The stream is counted separately for each event source in accordance with the following rules:
   • If a monitoring policy is applied to the event source, the number displayed for the event stream is counted for the time period specified in the policy.

     Depending on the policy type, the number of the event stream is counted as the number of events (for the byCount policy type) or as the number events per second (EPS, for the byEPS policy type). You can look up how the stream is counted for the applied policy in the Stream column on the List of event sources page.

   • If no monitoring policy is applied to the event source, the number for the event stream corresponds to the last value.
6. The event stream is checked against the constraints of the policy once a minute.

If the event stream from the source crosses the thresholds specified in the monitoring policy, information about this is recorded in the following way:

• A notification about a monitoring policy getting triggered is sent to the email addresses specified in the policy.
• A stream monitoring informational event of type 5(Type=5) is generated. The fields of the event are described in the table below.

  Fields of the monitoring event

  Event field name	Field value
  ID	Unique ID of the event.
  Timestamp	Event time.
  Type	Type of the audit event. For the audit event, the value is 5 (monitoring).
  Name	Name of the monitoring policy.
  DeviceProduct	KUMA
  DeviceCustomString1	The value from the value field in the notification. Displays the value of the metric for which the notification was sent.

The generated monitoring event is sent to the following resources:

• All storages of the Main tenant
• All correlators of the Main tenant
• All correlators of the tenant in which the event source is located

Managing monitoring policies

To add a monitoring policy:

1. In the KUMA web interface, under Source status → Monitoring policies, click Add policy and define the settings in the opened window:
   1. In the Policy name field, enter a unique name for the policy you are creating. The name must contain 1 to 128 Unicode characters.
   2. In the Tenant drop-down list, select the tenant that will own the policy. Your tenant selection determines the specific sources of events that can covered by the monitoring policy.
   3. In the Policy type drop-down list, select one of the following options:
      • byCount—by the number of events over a certain period of time.
      • byEPS—by the number of events per second over a certain period of time. The average value over the entire period is calculated. You can additionally track spikes during specific periods.
   4. In the Lower limit and Upper limit fields, define the boundaries representing normal behavior. Deviations outside of these boundaries will trigger the monitoring policy, create an alert, and forward notifications.
   5. In the Count interval field, specify the period during which the monitoring policy must take into account the data from the monitoring source. The maximum value is 14 days.
   6. If you selected the byEPS policy type, in the Control interval, minutes field, specify the control time interval (in minutes) within which the number of events must cross the threshold for the monitoring policy to trigger:
      • If, during this time period, all checks (performed once per minute) find that the stream is crossing the threshold, the monitoring policy is triggered.
      • If, during this time period, one of the checks (performed once per minute) finds that that the stream is within the thresholds, the monitoring policy is not triggered, and the count of check results is reset.

      If you do not specify the frequency of measurement, the monitoring policy is triggered immediately after the stream is found to cross the threshold.

   7. If necessary, specify the email addresses to which notifications about the activation of the KUMA monitoring policy should be sent. To add each address, click the Email button.

      To forward notifications, you must configure a connection to the SMTP server.

2. Click Add.

The monitoring policy will be added.

To apply a monitoring policy:

1. In the KUMA web console, in the Source status → Event sources section, select one or more event sources from the list by selecting check boxes next to the names of the event sources. You can also select all event sources in the list by selecting the Select all check box.

   After you select the event sources to which you want to apply a monitoring policy, the Apply policy button becomes available on the toolbar if any policies are available.

2. Click Apply policy.
3. This opens the Enable policy window; in that window, select a policy from the drop-down list. You can also use the context search to select a policy in the drop-down list. The selected monitoring policy must belong to the Shared tenant or to the tenant of the event source. After applying the policy, the status of the event source becomes green, and the Monitoring policy, Stream, Lower limit, and Upper limit columns are filled with information from the applied policy.
4. Click OK.

The monitoring policy is applied to the selected event sources.

To delete a monitoring policy:

1. In the KUMA web interface, in the Source status → Monitoring policies section, select one or more monitoring policies that you want to delete.
2. Click Delete policy and confirm the action.

The selected monitoring policies are deleted.

You cannot remove preinstalled monitoring policies or policies that have been assigned to data sources.

Page top