Step 3. Event parsing

This is a required step of the Installation Wizard. On the Event parsing tab of the Installation Wizard, select or create a normalizer resource whose settings will define the rules for converting raw events into normalized events. You can add more than one normalizer to implement complex processing logic.

When creating a new normalizer in the Installation Wizard, by default it is saved in the set of resources for the collector and cannot be used in other collectors. You can use the Save normalizer check box to create a separate resource.

If, when changing the settings of a collector resource set, you change or delete conversions in a normalizer connected to it, the edits will not be saved, and the normalizer resource itself may be corrupted. If you need to modify conversions in a normalizer that is already part of a service, the changes must be made directly to the resource under Resources → Normalizers in the web interface.

Adding a normalizer

To add an existing normalizer to a resource set:

1. Click the Add event parsing button.

   The Event parsing window will open with the normalizer settings and an active Normalization scheme tab.

2. In the Normalizer drop-down list, select the required normalizer.

   The Event parsing window will display the parameters of the selected normalizer. You can open the selected resource for editing in a new browser tab using the button.

3. Click OK.

The normalizer is displayed as a dark circle on the Event parsing tab of the Installation Wizard. Clicking on the circle will open the normalizer options for editing. When you hover over the circle, a plus sign is displayed: click on it to add more normalizers (see below).

To create a new normalizer:

1. Select Create new from the Normalizer drop-down list.

   The Event parsing window will open with the normalizer settings and an active Normalization scheme tab.

2. If you want to keep the normalizer as a separate resource, select the Save normalizer check box. This check box is cleared by default.
3. In the Name field, enter a unique name for the normalizer. The name must contain from 1 to 128 Unicode characters.
4. In the Parsing method drop-down list, select the type of events to receive. Depending on your choice, you can use the preconfigured rules for matching event fields or set your own rules. When you select some parsing methods, additional parameter fields required for filling in may become available.

   Available parsing methods:

   • json

     This parsing method is used to process JSON data.

     When processing files with hierarchically arranged data, you can access the fields of nested objects by specifying the names of the parameters dividing them by a period. For example, the username parameter from the string "user": {"username": "system: node: example-01"} can be accessed by using the user.username query.

   • cef

     This parsing method is used to process CEF data.

     When choosing this method, you can use the preconfigured rules for converting events to the KUMA format by clicking the Apply default mapping button.

   • regexp

     This parsing method is used to create custom rules for processing JSON data.

     In the Normalization parameter block field, add a regular expression (RE2 syntax) with named capture groups. The name of a group and its value will be interpreted as the field and the value of the raw event, which can be converted into an event field in KUMA format.

     To add event handling rules:

     1. Copy an example of the data you want to process to the Event examples field. This is an optional but recommended step.
     2. In the Normalization parameter block field add a regular expression with named capture groups in RE2 syntax, for example "(?P<name>regexp)".

        You can add multiple regular expressions by using the Add regular expression button. If you need to remove the regular expression, use the button.

     3. Click the Copy field names to the mapping table button.

        Capture group names are displayed in the KUMA field column of the Mapping table. Now you can select the corresponding KUMA field in the column next to each capture group. Otherwise, if you named the capture groups in accordance with the CEF format, you can use the automatic CEF mapping by selecting the Use CEF syntax for normalization check box.

     Event handling rules were added.

   • syslog

     This parsing method is used to process data in syslog format.

     When choosing this method, you can use the preconfigured rules for converting events to the KUMA format by clicking the Apply default mapping button.

   • csv

     This parsing method is used to create custom rules for processing CSV data.

     When choosing this method, you must specify the separator of values in the string in the Delimiter field. Any single-byte ASCII character can be used as a delimiter.

   • kv

     This parsing method is used to process data in key-value pair format.

     If you select this method, you must provide values in the following required fields:

     • Pair delimiter—specify a character that will serve as a delimiter for key-value pairs. You can specify any one-character (1 byte) value, provided that the character does not match the value delimiter.
     • Value delimiter—specify a character that will serve as a delimiter between the key and the value. You can specify any one-character (1 byte) value, provided that the character does not match the delimiter of key-value pairs.
   • xml

     This parsing method is used to process XML data.

     When this method is selected in the parameter block XML Attributes you can specify the key attributes to be extracted from tags. If an XML structure has several attributes with different values in the same tag, you can indicate the necessary value by specifying its key in the Source column of the Mapping table.

     To add key XML attributes,

     Click the Add field button, and in the window that appears, specify the path to the required attribute.

     You can add more than one attribute. Attributes can be removed one at a time using the cross icon or all at once using the Reset button.

     If XML key attributes are not specified, then in the course of field mapping the unique path to the XML value will be represented by a sequence of tags.

   • netflow5

     This parsing method is used to process data in the NetFlow v5 format.

     When choosing this method, you can use the preconfigured rules for converting events to the KUMA format by clicking the Apply default mapping button.

     In mapping rules, the protocol type for netflow5 is not indicated in the fields of KUMA events by default. When parsing data in NetFlow format on the Enrichment normalizer tab, you should create a constant data enrichment rule that adds the netflow value to the DeviceProduct target field.

   • netflow9

     This parsing method is used to process data in the NetFlow v9 format.

     When choosing this method, you can use the preconfigured rules for converting events to the KUMA format by clicking the Apply default mapping button.

     In mapping rules, the protocol type for netflow9 is not indicated in the fields of KUMA events by default. When parsing data in NetFlow format on the Enrichment normalizer tab, you should create a constant data enrichment rule that adds the netflow value to the DeviceProduct target field.

   • ipfix

     This parsing method is used to process IPFIX data.

     When choosing this method, you can use the preconfigured rules for converting events to the KUMA format by clicking the Apply default mapping button.

     In mapping rules, the protocol type for ipfix is not indicated in the fields of KUMA events by default. When parsing data in NetFlow format on the Enrichment normalizer tab, you should create a constant data enrichment rule that adds the netflow value to the DeviceProduct target field.

   • sql—this method becomes available only when using a sql type connector.

     This parsing method is used to process SQL data.

5. In the Keep raw log drop-down list, specify whether the original raw event should be stored in the newly created normalized event. Available values:
   • Never—do not save the raw event This is the default setting.
   • Only errors—save the raw event in the Raw field of the normalized event if errors occurred when parsing it. This value is convenient to use when debugging a service. In this case, every time an event has a non-empty Raw field, you know there was a problem.
   • Always—always save the raw event in the Raw field of the normalized event.
6. In the Keep extra fields drop-down list, choose whether you want to store the raw event fields in the normalized event if no mapping rules have been configured for them (see below). The data is stored in the Extra event field. By default, fields are not saved.
7. Copy an example of the data you want to process to the Event examples field. This is an optional but recommended step.

   Event examples can also be loaded from a TSV, CSV, or TXT file by using the Load from file button.

8. Configure the mapping of the raw event fields to event fields in KUMA format In the Mapping table:
   1. In the Source column, provide the name of the raw event field that you want to convert into the KUMA event field.

      Clicking the button next to the field names in the Source column opens the Conversion window, in which you can use the Add conversion button to create rules for modifying the original data before they are written to the KUMA event fields.

      Available conversions

      Conversions are changes that can be applied to a value before it gets written to the event field. The conversion type is selected from a drop-down list.

      Available conversions:

      • lower—is used to make all characters of the value lowercase
      • upper—is used to make all characters of the value uppercase
      • regexp – used to convert a value using the regular expression RE2. When this conversion type is selected, the field appears where regular expression should be added.
      • substring—is used to extract characters in the position range specified in the Start and End fields. These fields appear when this conversion type is selected.
      • replace—is used to replace specified character sequence with the other character sequence. When this type of conversion is selected, new fields appear:
        • Replace chars—in this field you can specify the character sequence that should be replaced.
        • With chars—in this field you can specify the characters sequence should be used instead of replaced characters.
      • trim—used to simultaneously remove the characters specified in the Chars field from the leading and end positions of the value. The field appears when this type of conversion is selected. For example, a trim conversion with the Micromon value applied to Microsoft-Windows-Sysmon results in soft-Windows-Sys.
      • append is used to add the characters specified in the Constant field to the end of the event field value. The field appears when this type of conversion is selected.
      • prepend—used to prepend the characters specified in the Constant field to the start of the event field value. The field appears when this type of conversion is selected.
      • replace with regexp—is used to replace RE2 regular expression results with the character sequence.
        • Expression—in this field you can specify the regular expression which results that should be replaced.
        • With chars—in this field you can specify the characters sequence should be used instead of replaced characters.
   2. In the KUMA field column, select the required KUMA event field from the drop-down list. You can search for fields by entering their names in the field.
   3. If the name of the KUMA event field selected at the previous step begins with DeviceCustom*, you can add a unique custom label in the Label field if necessary.

   New table rows can be added by using the Add row button. Rows can be deleted individually using the button or all at once using the Clear all button.

   If you have loaded data into the Event examples field, the table will have an Examples column containing examples of values carried over from the raw event field to the KUMA event field.

9. Click OK.

The normalizer is displayed as a dark circle on the Event parsing tab of the Installation Wizard. Clicking on the circle will open the normalizer options for editing. When you hover over the circle, a plus sign is displayed: click on it to add more normalizers (see below).

Enriching normalized events with additional data

You can add additional data to the newly created normalized events by creating enrichment rules in the normalizer similar to those in enrichment rule resources. These enrichment rules are stored in the normalizer resource where they were created. There can be more than one enrichment rule.

To add enrichment rules to the normalizer:

1. Select the normalizer and go to the Enrichment tab in the Event parsing window.
2. Click the Add enrichment button.

   The enrichment rule parameter block appears. Close the parameter block using the button.

3. Select the enrichment type from the Source kind drop-down list. Depending on the selected type, you may see advanced settings that will also need to be completed.

   Available Enrichment rule source types:

   • constant

     This type of enrichment is used when a constant needs to be added to an event field. Settings of this type of enrichment:

     • In the Constant field, specify the value that should be added to the event field. The value should not be longer than 255 Unicode characters. If you leave this field blank, the existing event field value will be cleared.
     • In the Target field drop-down list, select the KUMA event field to which you want to write the data.

   • dictionary

     This type of enrichment is used if you need to add a value from the dictionary to the event field.

     When this type is selected in the Dictionary name drop-down list, you must select the dictionary that will provide the values. In the Key fields settings block, you must use the Add field button to select the event fields whose values will be used for dictionary entry selection.

   • event

     This type of enrichment is used when you need to write a value from another event field to the current event field. Settings of this type of enrichment:

     • In the Target field drop-down list, select the KUMA event field to which you want to write the data.
     • In the Source field drop-down list, select the event field whose value will be written to the target field.
     • Clicking the button opens the Conversion window in which you can, using the Add conversion button, create rules for modifying the original data before writing them to the KUMA event fields.

       Available conversions

       Conversions are changes that can be applied to a value before it gets written to the event field. The conversion type is selected from a drop-down list.

       Available conversions:

       • lower—is used to make all characters of the value lowercase
       • upper—is used to make all characters of the value uppercase
       • regexp – used to convert a value using the regular expression RE2. When this conversion type is selected, the field appears where regular expression should be added.
       • substring—is used to extract characters in the position range specified in the Start and End fields. These fields appear when this conversion type is selected.
       • replace—is used to replace specified character sequence with the other character sequence. When this type of conversion is selected, new fields appear:
         • Replace chars—in this field you can specify the character sequence that should be replaced.
         • With chars—in this field you can specify the characters sequence should be used instead of replaced characters.
       • trim—used to simultaneously remove the characters specified in the Chars field from the leading and end positions of the value. The field appears when this type of conversion is selected. For example, a trim conversion with the Micromon value applied to Microsoft-Windows-Sysmon results in soft-Windows-Sys.
       • append is used to add the characters specified in the Constant field to the end of the event field value. The field appears when this type of conversion is selected.
       • prepend—used to prepend the characters specified in the Constant field to the start of the event field value. The field appears when this type of conversion is selected.
       • replace with regexp—is used to replace RE2 regular expression results with the character sequence.
         • Expression—in this field you can specify the regular expression which results that should be replaced.
         • With chars—in this field you can specify the characters sequence should be used instead of replaced characters.
   • template

     This type of enrichment is used when you need to write a value obtained by processing Go templates into the event field. Settings of this type of enrichment:

     • Put the Go template into the Template field.

       Event field names are passed in the {{.EventField}} format, where EventField is the name of the event field from which the value must be passed to the script.

       Example: Attack on {{.DestinationAddress}} from {{.SourceAddress}}.

     • In the Target field drop-down list, select the KUMA event field to which you want to write the data.
4. Click OK.

Enrichment rules are added to the normalizer, and the Event parsing window is closed.

Creating a structure of normalizers

You can create several extra normalizers within a normalizer. This allows you to customize complex event handling logic.

The sequence in which normalizers are created matters: events are processed sequentially, and their path is shown using arrows.

To create an extra normalizer:

• Create the initial normalizer (see above).

  The created normalizer will be displayed in the window as a dark circle.

• Hover over the initial normalizer and click the plus sign button that appears.
• In the Add normalizer to normalization scheme window, specify the conditions under which the data will be sent to the extra normalizer:
  • If you want to send only events with specific fields to the extra normalizer, list them in the Fields to pass into normalizer field.
  • If you want to send only events in which certain fields have been assigned specific values to the extra normalizer, specify the name of the event field in the Use normalizer for events with specific event field values field and the value that should match it in the Condition value field.

  The data processed by these conditions can be preconverted by clicking the button. This opens the Conversion window, in which you can use the Add conversion button to create rules for modifying the original data before it is written to the KUMA event fields.

  Available conversions

  Conversions are changes that can be applied to a value before it gets written to the event field. The conversion type is selected from a drop-down list.

  Available conversions:

  • lower—is used to make all characters of the value lowercase
  • upper—is used to make all characters of the value uppercase
  • regexp – used to convert a value using the regular expression RE2. When this conversion type is selected, the field appears where regular expression should be added.
  • substring—is used to extract characters in the position range specified in the Start and End fields. These fields appear when this conversion type is selected.
  • replace—is used to replace specified character sequence with the other character sequence. When this type of conversion is selected, new fields appear:
    • Replace chars—in this field you can specify the character sequence that should be replaced.
    • With chars—in this field you can specify the characters sequence should be used instead of replaced characters.
  • trim—used to simultaneously remove the characters specified in the Chars field from the leading and end positions of the value. The field appears when this type of conversion is selected. For example, a trim conversion with the Micromon value applied to Microsoft-Windows-Sysmon results in soft-Windows-Sys.
  • append is used to add the characters specified in the Constant field to the end of the event field value. The field appears when this type of conversion is selected.
  • prepend—used to prepend the characters specified in the Constant field to the start of the event field value. The field appears when this type of conversion is selected.
  • replace with regexp—is used to replace RE2 regular expression results with the character sequence.
    • Expression—in this field you can specify the regular expression which results that should be replaced.
    • With chars—in this field you can specify the characters sequence should be used instead of replaced characters.
• Click OK.

  This will open the Event parsing window, in which you can configure the rules for processing events as you did in the initial normalizer (see above). The Keep raw log parameter is not available. The Event examples field displays the values specified when the initial normalizer was created.

• Specify the extra normalizer settings similar to the initial normalizer
• Click OK.

The extra normalizer is displayed as a dark block that indicates the conditions under which this normalizer will be used. The conditions can be changed by moving your mouse cursor over the extra normalizer and clicking the button showing the pencil image. If you hover the mouse pointer over the extra normalizer, a plus button appears, which you can use to create a new extra normalizer. To delete a normalizer, use the button with the trash icon.

Proceed to the next step of the Installation Wizard.

Page top