Collector

A collector is an application component that receives messages from event sources, processes them, and transmits them to a storage, correlator, and/or third-party services to identify alerts.

For each collector, you need to configure one connector and one normalizer. You can also configure an unlimited number of additional normalizers, filters, enrichment rules, and aggregation rules. To enable the collector to send normalized events to other services, specific destinations must be added. Normally, two destinations are used: the storage and the correlator.

The collector operation algorithm includes the following steps:

1. Receiving messages from event sources

   To receive messages, you must configure an active or passive connector. The passive connector can only receive messages from the event source, while the active connector can initiate a connection to the event source, such as a database management system.

   Connectors can also vary by type. The choice of connector type depends on the transport protocol for transmitting messages. For example, for an event source that transmits messages over TCP, you must install a TCP type connector.

   The program has the following connector types available:

   • internal
   • tcp
   • udp
   • netflow
   • sflow
   • nats-jetstream
   • kafka
   • http
   • sql
   • file
   • diode
   • ftp
   • nfs
   • wmi
   • wec
   • snmp
2. Event parsing and normalization

   Events received by the connector are processed using the normalizer and normalization rules set by the user. The choice of normalizer depends on the format of the messages received from the event source. For example, you must select a CEF-type root normalizer for a source that sends events in CEF format.

   The following normalizers are available in the program:

   • JSON
   • CEF
   • Regexp
   • Syslog (as per RFC3164 and RFC5424)
   • CSV
   • Key-value
   • XML
   • NetFlow v5
   • NetFlow v9
   • IPFIX (v10)
3. Filtering of normalized events

   You can configure filters that allow you to filter out events that meet specified conditions. Events that do not meet the filtering conditions will be sent for processing.

4. Enrichment and conversion of normalized events

   Enrichment rules let you to supplement event contents with information from internal and external sources. The program has the following enrichment sources:

   • constants
   • cybertrace
   • dictionaries
   • dns
   • events
   • ldap
   • templates
   • timezone data
   • geographic data

   Mutation rules let you convert event field contents in accordance with the defined criteria. The program has the following conversion methods:

   • lower—converts all characters to lower case.
   • upper—converts all characters to upper case.
   • regexp—extracts a substring using RE2 regular expressions.
   • substring—gets a substring based on the specified numbers of the start and end positions.
   • replace—replaces text with the entered string.
   • trim—deletes the specified characters.
   • append—adds characters to the end of the field value.
   • prepend—adds characters to the beginning of the field value.
5. Aggregation of normalized events

   You can configure aggregation rules to reduce the number of similar events that are transmitted to the storage and/or the correlator. Configuring aggregation rules lets you combine several events into one event. This helps you reduce the load on the services responsible for further event processing, conserves storage space and the license quota for events per second (EPS). For example, you can aggregate into one event all events involving network connections made using the same protocol (transport and application layers) between two IP addresses and received during a specified time interval.

6. Transmission of normalized events

   After all the processing stages are completed, the event is sent to configured destinations.