Collector

A collector is an application component that receives messages from event sources, processes them, and transmits them to a storage, correlator, and/or third-party services to identify alerts.

For each collector, you need to configure one connector and one normalizer. You can also configure an unlimited number of additional Normalizers, Filters, Enrichment rules, and Aggregation rules. To enable the collector to send normalized events to other services, specific destinations must be added. Normally, two destinations are used: the storage and the correlator.

The collector operation algorithm includes the following steps:

  1. Receiving messages from event sources

    To receive messages, you must configure an active or passive connector. The passive connector can only receive messages from the event source, while the active connector can initiate a connection to the event source, such as a database management system.

    Connectors can also vary by type. The choice of connector type depends on the transport protocol for transmitting messages. For example, for an event source that transmits messages over TCP, you must install a TCP type connector.

    The program has the following connector types available:

    • tcp
    • udp
    • netflow
    • sflow
    • nats-jetstream
    • kafka
    • kata/edr
    • http
    • sql
    • file
    • 1c-xml
    • 1c-log
    • diode
    • ftp
    • nfs
    • vmware
    • wmi
    • wec
    • snmp-trap
    • elastic
    • etw
  2. Event parsing and normalization

    Events received by the connector are processed using the normalizer and normalization rules set by the user. The choice of normalizer depends on the format of the messages received from the event source. For example, you must select a CEF-type root normalizer for a source that sends events in CEF format.

    The following normalizers are available in the program:

    • JSON
    • CEF
    • Regexp
    • Syslog (as per RFC3164 and RFC5424)
    • CSV
    • Key-value
    • XML
    • NetFlow v5
    • NetFlow v9
    • SQL
    • IPFIX (v10)
  3. Filtering of normalized events

    You can configure filters to select events satisfying specified conditions to send such events for processing.

  4. Enrichment and conversion of normalized events

    Enrichment rules let you to supplement event contents with information from internal and external sources. The program has the following enrichment sources:

    • constants
    • cybertrace
    • dictionaries
    • dns
    • events
    • ldap
    • templates
    • timezone data
    • geographic data

    Mutation rules let you convert event field contents in accordance with the defined criteria. The program has the following conversion methods:

    • lower—converts all characters to lower case.
    • upper—converts all characters to upper case.
    • regexp—extracts a substring using RE2 regular expressions.
    • substring—gets a substring based on the specified numbers of the start and end positions.
    • replace—replaces text with the entered string.
    • trim—deletes the specified characters.
    • append—adds characters to the end of the field value.
    • prepend—adds characters to the beginning of the field value.
  5. Aggregation of normalized events

    You can configure aggregation rules to reduce the number of similar events that are transmitted to the storage and/or the correlator. Configuring aggregation rules lets you combine several events into one event. This helps you reduce the load on the services responsible for further event processing, conserves storage space and the license quota for events per second (EPS). For example, you can aggregate into one event all events involving network connections made using the same protocol (transport and application layers) between two IP addresses and received during a specified time interval.

  6. Transmission of normalized events

    After all the processing stages are completed, the event is sent to configured destinations.

Page top