Collector

A collector is an application component that receives messages from event sources, processes these messages, and sends them to a storage, correlator, and/or third-party services to identify alerts.

For each collector, one connector and one normalizer must be configured. You can also configure any number of additional normalizers, filters, enrichment rules, and aggregation rules. For the collector to send normalized events to other services, you must add destinations. Normally, two destinations are used: a storage and a correlator.

The collector iterates over the following steps:

  1. Receive messages from event sources

    To receive messages, you must configure an active or passive connector. A passive connector only listens for messages from an event source, while an active connector actively polls an event source, such as a database management system.

    Connectors can have different types. The choice of connector type depends on the transport protocol used for messaging. For example, if your event source sends messages over TCP, you must install a connector of the TCP type.

    The application has the following connector types available:

    • tcp
    • udp
    • netflow
    • sflow
    • nats-jetstream
    • kafka
    • kata/edr
    • http
    • sql
    • file
    • 1c-xml
    • 1c-log
    • diode
    • ftp
    • nfs
    • vmware
    • wmi
    • wec
    • snmp-trap
    • elastic
    • etw
  2. Parse and normalize events

    Events received by the connector are processed using the normalizer and normalization rules set by the user. The choice of normalizer depends on the format of messages coming from the event source. For example, if your event source sends messages in the CEF format, you must select a normalizer of the CEF type.

    The following normalizers are available in the application:

    • JSON
    • CEF
    • Regexp
    • Syslog (as per RFC3164 and RFC5424)
    • CSV
    • Key-value
    • XML
    • NetFlow (the same normalizer for NetFlow v5, NetFlow v9 and IPFIX)
    • NetFlow v5
    • NetFlow v9
    • SQL
    • IPFIX (v10)
  3. Filter normalized events

    You can configure filters to identify events that satisfy certain conditions and only send such events for processing.

  4. Enrich and convert normalized events

    Enrichment rules let you add supplementary information from internal and external sources to the events. The application can use the following enrichment sources:

    • constants
    • cybertrace
    • dictionaries
    • dns
    • events
    • ldap
    • templates
    • timezone data
    • geographic data

    Mutation rules let you convert event field contents in accordance with the defined criteria. The application offers the following conversion methods:

    • lower: convert all characters to lower case.
    • upper: convert all characters to upper case.
    • regexp: extract a substring using RE2 regular expressions.
    • substring—gets a substring based on the specified numbers of the start and end positions.
    • replace: replace some text with a string.
    • trim: delete the specified characters.
    • append: add characters to the end of the field value.
    • prepend: adds characters to the beginning of the field value.
  5. Aggregate normalized events

    You can configure aggregation rules to reduce the number of similar events that are transmitted to the storage and/or the correlator. Configuring aggregation rules lets you combine several events into one event. This helps you reduce the load on the services responsible for further event processing, conserves storage space and the license quota for events per second (EPS). For example, you can aggregate into one event all events involving network connections made using the same protocol (transport and application layers) between two IP addresses and received during a specified time interval.

  6. Send out normalized events

    Having passed through all processing steps, the event is sent to the configured destinations.

Page top