About event sources

The regular expressions and event normalizing rules specified in the configuration file are grouped by event sources that are represented by Source elements. Usually these event sources are devices that issue events which afterward are checked by Feed Service. Every Source element contains a set of rules. There can be one or more Source elements in the InputSettings > RegExps element.

A Source element has the following attributes:

• id

  A unique identifier of the event source.

  There must be an event source with the default identifier (<Source id="default">). Rules of the default event source have lower priority than rules specific to the event source. Rules specific to the event source are applied first. Rules of the default event source are applied next. If a rule specific to the event source and a rule of the default event source have the same name, the rule of the default event source is applied only if the rule specific to the event source had no matches.

  There are two special event sources that you can use: http_single_lookup (<Source id="http_single_lookup">) and http_file_lookup (<Source id="http_file_lookup">). The rules of the http_single_lookup event source are used when single values are searched for by means of CyberTrace Web. The rules of the http_file_lookup event source are used when hashes of specified files or indicators in log files are searched for by means of CyberTrace Web. Therefore, if you have to search for values contained in a log file of some special format, you are advised to specify rules for the http_file_lookup event source. If the configuration file contains the http_single_lookup event source or the http_file_lookup event source, we strongly recommend that you not remove the regular expressions specified in these special event sources by default and, instead, edit them as you like.

  In detection events, the identifier of the event source can be referred to by the %SourceId% pattern.

• ip

  The IP address of the event source. The ip attribute cannot be set for the default, http_single_lookup, and http_file_lookup event sources.

  If an event has arrived from an event source that has the specified IP address, the event is processed by using the rules contained in this Source element. If the IP address of the event source is not among those specified in the ip attribute of the Source elements, the host name of the event source is determined and a Source element is sought for that has this host name in the hostname attribute; the rules from that Source element are used for processing the event.

• hostname

  The host name of the event source. The value of the host name is extracted from the event. In syslog events, the host name follows the timestamp (https://tools.ietf.org/html/rfc5424). For example, in an event Feb 2 11:57:59 sample-hostname alert: sample event text the host name is sample-hostname.

  The hostname attribute cannot be set for the default, http_single_lookup, and http_file_lookup event sources.

  If an event has arrived from an event source that has the specified host name and the IP address of the event source is not among those specified in the ip attribute of the Source elements, the event is processed by using the rules contained in this Source element.

The way of how Feed Service chooses rules from different Source elements is described in the following flow chart.

Choosing a rule

Note that event normalizing rules are applied first, and regular expressions are applied afterwards.

The regular expressions of the default event source for finding URLs, IP addresses, and hashes are universal, that is, they can be used for parsing events issued by most devices. They can be used for parsing events that contain multiple URLs, but cannot be used, for example, for parsing events that contain URLs with no protocol specified. The use of universal regular expressions lowers the performance of Feed Service compared to use of device-specific regular expressions. Also, the universal regular expressions do not handle the dispersal, in an event, of different parts of a URL (for example, the host and the path). The universal regular expressions for finding hashes can extract symbol sequences that actually are not hashes.

Page top