Collector

A collector is an application component that receives messages from event sources, processes them, and transmits them to a storage, correlator, and/or third-party services to identify alerts.

For each collector, you need to configure one connector and one normalizer. You can also configure an unlimited number of additional Normalizers, Filters, Enrichment rules, and Aggregation rules. To enable the collector to send normalized events to other services, specific destinations must be added. Normally, two destinations are used: the storage and the correlator.

The collector operation algorithm includes the following steps:

  1. Receiving messages from event sources

    To receive messages, you must configure an active or passive connector. The passive connector can only receive messages from the event source, while the active connector can initiate a connection to the event source, such as a database management system.

    Connectors can also vary by type. The choice of connector type depends on the transport protocol for transmitting messages. For example, for an event source that transmits messages over TCP, you must install a TCP type connector.

    The program has the following connector types available:

    • internal
    • tcp
    • udp
    • netflow
    • nats
    • kafka
    • http
    • sql
    • file
    • ftp
    • nfs
    • wmi
    • wec
    • snmp
  2. Event parsing and normalization

    Events received by the connector are processed using the parser and normalization rules set by the user. The choice of normalizer depends on the format of the messages received from the event source. For example, you must select a CEF-type root normalizer for a source that sends events in CEF format.

    The following normalizers are available in the program:

    • JSON
    • CEF
    • Regexp
    • Syslog (as per RFC3164 and RFC5424)
    • CSV
    • Key-value
    • XML
    • NetFlow v5
    • NetFlow v9
    • IPFIX (v10)
  3. Filtering of normalized events

    You can configure filters that allow you to select only the events that satisfy the specified conditions for further processing. Events that do not meet the filtering conditions are eliminated at this stage and are not processed further.

  4. Enrichment and conversion of normalized events

    Enrichment rules let you to supplement event contents with information from internal and external sources. The program has the following enrichment sources:

    • constant
    • cybertrace
    • dictionary
    • dns
    • event
    • ldap
    • template

    Mutation rules let you convert event contents in accordance with the defined criteria. The program has the following conversion methods:

    • lower—converts all characters to lower case.
    • upper—converts all characters to upper case.
    • regexp—extracts a substring using RE2 regular expressions.
    • substring—selects text strings by specified item numbers.
    • replace—replaces text with the entered string.
    • trim—deletes the specified characters.
    • append—adds characters to the end of the field value.
    • prepend—adds characters to the beginning of the field value.
  5. Aggregation of normalized events

    You can configure aggregation rules to reduce the number of similar events that are transmitted to the storage and/or the correlator. For example, you can aggregate into one event all messages about network connections transmitted over the same protocol (transport and application layers) between two IP addresses and received during a specified time interval. If aggregation rules are configured, multiple events will be processed and saved as a single event. This helps you reduce the load on the services responsible for further event processing, saves you storage space, and reduces events processed per second (EPS) count.

  6. Transmission of normalized events

    After all the processing stages are completed, the event is sent to configured destinations.

Page top