Event parsing settings

You can configure the rules for converting incoming events to the KUMA format when creating event parsing rules in the normalizer settings window, on the Normalization scheme tab.

Available settings:

• Name (required)—name of the parsing rules. Must contain 1 to 128 Unicode characters. The name of the main parsing rule is used as the name of the normalizer.
• Tenant (required)—name of the tenant that owns the resource.

  This setting is not available for extra parsing rules.

• Parsing method (required)—drop-down list for selecting the type of incoming events. Depending on your choice, you can use the preconfigured rules for matching event fields or set your own rules. When you select some parsing methods, additional parameter fields required for filling in may become available.

  Available parsing methods:

  • json

    This parsing method is used to process JSON data where each object, including its nested objects, occupies a single line in a file.

    When processing files with hierarchically arranged data, you can access the fields of nested objects by specifying the names of the parameters dividing them by a period. For example, the username parameter from the string "user": {"username": "system: node: example-01"} can be accessed by using the user.username query.

    Files are processed line by line. Multi-line objects with nested structures may be normalized incorrectly.

    In complex normalization schemes where additional normalizers are used, all nested objects are processed at the first normalization level, except for cases when the extra normalization conditions are not specified and, therefore, the event being processed is passed to the additional normalizer in its entirety.

    Newline characters can be \n and \r\n. Strings must be UTF-8 encoded.

    If you want to send the raw event for advanced normalization, at each nesting level in the Advanced event parsing window, select Yes in the Keep raw event drop-down list.

  • cef

    This parsing method is used to process CEF data.

    When choosing this method, you can use the preconfigured rules for converting events to the KUMA format by clicking the Apply default mapping button.

  • regexp

    This parsing method is used to create custom rules for processing data in a format using regular expressions.

    In the Normalization parameter block field, add a regular expression (RE2 syntax) with named capture groups. The name of a group and its value will be interpreted as the field and the value of the raw event, which can be converted into an event field in KUMA format.

    To add event handling rules:

    1. Copy an example of the data you want to process to the Event examples field. This is an optional but recommended step.
    2. In the Normalization parameter block field add a regular expression with named capture groups in RE2 syntax, for example "(?P<name>regexp)". The regular expression added to the Normalization parameter must exactly match the event. Also, when developing the regular expression, it is recommended to use special characters that match the starting and ending positions of the text: ^, $.

       You can add multiple regular expressions by using the Add regular expression button. If you need to remove the regular expression, use the button.

    3. Click the Copy field names to the mapping table button.

       Capture group names are displayed in the KUMA field column of the Mapping table. Now you can select the corresponding KUMA field in the column next to each capture group. Otherwise, if you named the capture groups in accordance with the CEF format, you can use the automatic CEF mapping by selecting the Use CEF syntax for normalization check box.

    Event handling rules were added.

  • syslog

    This parsing method is used to process data in syslog format.

    When choosing this method, you can use the preconfigured rules for converting events to the KUMA format by clicking the Apply default mapping button.

  • csv

    This parsing method is used to create custom rules for processing CSV data.

    When choosing this method, you must specify the separator of values in the string in the Delimiter field. Any single-byte ASCII character can be used as a delimiter.

  • kv

    This parsing method is used to process data in key-value pair format.

    If you select this method, you must provide values in the following required fields:

    • Pair delimiter—specify a character that will serve as a delimiter for key-value pairs. You can specify any one-character (1 byte) value, provided that the character does not match the value delimiter.
    • Value delimiter—specify a character that will serve as a delimiter between the key and the value. You can specify any one-character (1 byte) value, provided that the character does not match the delimiter of key-value pairs.
  • xml

    This parsing method is used to process XML data in which each object, including its nested objects, occupies a single line in a file. Files are processed line by line.

    If you want to send the raw event for advanced normalization, at each nesting level in the Advanced event parsing window, select Yes in the Keep raw event drop-down list.

    When this method is selected in the parameter block XML attributes you can specify the key attributes to be extracted from tags. If an XML structure has several attributes with different values in the same tag, you can indicate the necessary value by specifying its key in the Source column of the Mapping table.

    To add key XML attributes,

    Click the Add field button, and in the window that appears, specify the path to the required attribute.

    You can add more than one attribute. Attributes can be removed one at a time using the cross icon or all at once using the Reset button.

    If XML key attributes are not specified, then in the course of field mapping the unique path to the XML value will be represented by a sequence of tags.

    Tag numbering

    Tag numbering is available as of KUMA 2.1.3. This functionality allows automatically numbering tags in XML events, which lets you parse an event with identical tags or unnamed tags, such as <Data>.

    As an example, we will use the Tag numbering functionality to number the tags of the EventData attribute of Microsoft Windows PowerShell event ID 800.

    

    To parse such events, you must:

    • Configure tag numbering.
    • Configure data mapping for numbered tags with KUMA event fields.

    KUMA 3.0.x supports using XML attributes and Tag numbering functionality at the same time in the same extra normalizer. If an attribute contains unnamed tags or identical tags, we recommend using the Tag numbering functionality. If the attribute contains only named tags, use XML attributes. To use this functionality in extra normalizers, you must sequentially enable the "Keep raw event" setting in each extra normalizer along the path that the event follows to the target extra normalizer, and in the target extra normalizer itself.

    For an example of this functionality in action, you can refer to the MicrosoftProducts normalizer — the "Keep raw event" setting is enabled sequentially in the "AD FS" and "424" extra normalizers.

    To configure parsing of events with identically named or unnamed tags:

    1. Create a new normalizer or open an existing normalizer for editing.
    2. In the Basic event parsing window of the normalizer, in the Parsing method drop-down list, select 'xml' and in the Tag numbering field, click Add field.

       In the displayed field, enter the full path to the tag to whose elements you want to assign a number. For example, Event.EventData.Data. The first number to be assigned to a tag is 0. If the tag is empty, for example, <Data />, it is also assigned a number.

    3. To configure data mapping, under Mapping, click Add row and do the following:
       1. In the new row, in the Source field, enter the full path to the tag and its index. For the Microsoft Windows event from the example above, the full path with indices look like this:
          • Event.EventData.Data.0
          • Event.EventData.Data.1
          • Event.EventData.Data.2 and so on
       2. In the KUMA field drop-down list, select the field in the KUMA event that will receive the value from the numbered tag after parsing.
    4. To save changes:
       • If you created a new normalizer, click Save.
       • If you edited an existing normalizer, click Update configuration in the collector to which the normalizer is linked.

    Parsing is configured.

  • netflow5

    This parsing method is used to process data in the NetFlow v5 format.

    When choosing this method, you can use the preconfigured rules for converting events to the KUMA format by clicking the Apply default mapping button. If the netflow5 type is selected for the main parsing, extra normalization is not available.

    In mapping rules, the protocol type for netflow5 is not indicated in the fields of KUMA events by default. When parsing data in NetFlow format, on the Enrichment normalizer tab, you must create a constant data enrichment rule that adds the netflow value to the DeviceProduct target field.

  • netflow9

    This parsing method is used to process data in the NetFlow v9 format.

    When choosing this method, you can use the preconfigured rules for converting events to the KUMA format by clicking the Apply default mapping button. If the netflow9 type is selected for the main parsing, extra normalization is not available.

    In mapping rules, the protocol type for netflow9 is not indicated in the fields of KUMA events by default. When parsing data in NetFlow format, on the Enrichment normalizer tab, you must create a constant data enrichment rule that adds the netflow value to the DeviceProduct target field.

  • sflow5

    This parsing method is used to process data in sflow5 format.

    When choosing this method, you can use the preconfigured rules for converting events to the KUMA format by clicking the Apply default mapping button. If the sflow5 type is selected for the main parsing, extra normalization is not available.

  • ipfix

    This parsing method is used to process IPFIX data.

    When choosing this method, you can use the preconfigured rules for converting events to the KUMA format by clicking the Apply default mapping button. If the ipfix type is selected for the main parsing, extra normalization is not available.

    In mapping rules, the protocol type for ipfix is not indicated in the fields of KUMA events by default. When parsing data in NetFlow format, on the Enrichment normalizer tab, you must create a constant data enrichment rule that adds the netflow value to the DeviceProduct target field.

  • sql

    The normalizer uses this method to process data obtained by making a selection from the database.

• Keep raw event (required)—in this drop-down list, indicate whether you need to store the raw event in the newly created normalized event. Available values:
  • Don't save—do not save the raw event. This is the default setting.
  • Only errors—save the raw event in the Raw field of the normalized event if errors occurred when parsing it. This value is convenient to use when debugging a service. In this case, every time an event has a non-empty Raw field, you know there was a problem.

    If fields containing the names *Address or *Date* do not comply with normalization rules, these fields are ignored. No normalization error occurs in this case, and the values of the fields are not displayed in the Raw field of the normalized event even if the Keep raw event → Only errors option was selected.

  • Always—always save the raw event in the Raw field of the normalized event.

  This setting is not available for extra parsing rules.

• Keep extra fields (required)—in this drop-down list, you can choose whether you want to save fields and their values if no mapping rules have been configured for them (see below). This data is saved as an array in the Extra event field. Normalized events can be searched and filtered based on the data stored in the Extra field.

  Filtering based on data from the Extra event field

  Conditions for filters based on data from the Extra event field:

  • Condition—If.
  • Left operand—event field.
  • In this event field, you can specify one of the following values:
    • Extra field.
    • Value from the Extra field in the following format:

      Extra.<field name>

      For example, Extra.app.

      A value of this type is specified manually.

    • Value from the array written to the Extra field in the following format:

      Extra.<field name>.<array element>

      For example, Extra.array.0.

      The values in the array are numbered starting from 0.

      A value of this type is specified manually.

      To work with a value from the Extra field at depth 3 and below, use backquotes ``. For example, `Extra.lev1.lev2.lev3`.

  • Operator – =.
  • Right operand—constant.
  • Value—the value by which you need to filter events.

  By default, no extra fields are saved.

• Description—resource description: up to 4,000 Unicode characters.

  This setting is not available for extra parsing rules.

• Event examples—in this field, you can provide an example of data that you want to process.

  This setting is not available for the following parsing methods: netflow5, netflow9, sflow5, ipfix, sql.

  The Event examples field is populated with data obtained from the raw event if the event was successfully parsed and the type of data obtained from the raw event matches the type of the KUMA field.

  For example, the value "192.168.0.1" enclosed in quotation marks is not displayed in the SourceAddress field, in this case the value 192.168.0.1 is displayed in the Event examples field.

• Mapping settings block—here you can configure mapping of raw event fields to fields of the event in KUMA format:
  • Source—column for the names of the raw event fields that you want to convert into KUMA event fields.

    Clicking the button next to the field names in the Source column opens the Conversion window, in which you can use the Add conversion button to create rules for modifying the original data before they are written to the KUMA event fields. In the Conversion window, you can swap the added rules by dragging them by the icon; you can also delete them using the icon.

    Available conversions

    Conversions are changes that can be applied to a value before it gets written to the event field. The conversion type is selected from a drop-down list.

    Available conversions:

    • lower—is used to make all characters of the value lowercase
    • upper—is used to make all characters of the value uppercase
    • regexp – used to convert a value using the regular expression RE2. When this conversion type is selected, the field appears where regular expression should be added.
    • substring—is used to extract characters in the position range specified in the Start and End fields. These fields appear when this conversion type is selected.
    • replace—is used to replace specified character sequence with the other character sequence. When this type of conversion is selected, new fields appear:
      • Replace chars—in this field you can specify the character sequence that should be replaced.
      • With chars—in this field you can specify the characters sequence should be used instead of replaced characters.
    • trim—used to simultaneously remove the characters specified in the Chars field from the leading and end positions of the value. The field appears when this type of conversion is selected. For example, a trim conversion with the Micromon value applied to Microsoft-Windows-Sysmon results in soft-Windows-Sys.
    • append is used to add the characters specified in the Constant field to the end of the event field value. The field appears when this type of conversion is selected.
    • prepend—used to prepend the characters specified in the Constant field to the start of the event field value. The field appears when this type of conversion is selected.
    • replace with regexp—is used to replace RE2 regular expression results with the character sequence.
      • Expression—in this field you can specify the regular expression which results that should be replaced.
      • With chars—in this field you can specify the characters sequence should be used instead of replaced characters.
    • Converting encoded strings to text:
      • decodeHexString—used to convert a HEX string to text.
      • decodeBase64String—used to convert a Base64 string to text.
      • decodeBase64URLString—used to convert a Base64url string to text.

      When converting a corrupted string or if conversion error occur, corrupted data may be written to the event field.

      During event enrichment, if the length of the encoded string exceeds the size of the field of the normalized event, the string is truncated and is not decoded.

      If the length of the decoded string exceeds the size of the event field into which the decoded value is to be written, such a string is truncated to fit the size of the event field.

    Conversions when using the extended event schema

    Whether or not a conversion can be used depends on the type of extended event schema field being used:

    • For an additional field of the "String" type, all types of conversions are available.
    • For fields of the "Number" and "Float" types, the following types of conversions are available: regexp, substring, replace, trim, append, prepend, replaceWithRegexp, decodeHexString, decodeBase64String, decodeBase64URLString.
    • For fields of "Array of strings", "Array of numbers", and "Array of floats" types, the following types of conversions are available: append, prepend.

  • KUMA field—drop-down list for selecting the required fields of KUMA events. You can search for fields by entering their names in the field.
  • Label—in this column, you can add a unique custom label to event fields that begin with DeviceCustom* and Flex*.

  New table rows can be added by using the Add row button. Rows can be deleted individually using the button or all at once using the Clear all button.

  If you have loaded data into the Event examples field, the table will have an Examples column containing examples of values carried over from the raw event field to the KUMA event field.

  If the size of the KUMA event field is less than the length of the value placed in it, the value is truncated to the size of the event field.

Extended event schema

When normalizing events, extended event schema fields can be used in addition to standard KUMA event schema fields. Information about the types of extended event schema fields is shown in the table below.

Using many unique fields of the extended event schema can reduce the performance of the system, increase the amount of disk space required for storing events, and make the information difficult to understand.

We recommend consciously choosing a minimal set of additional fields of the extended event schema that you want to use in normalizers and correlation.

To use extended event schema fields:

• Open an existing event normalizer or create a new event normalizer.
• Specify the basic settings of the normalizer.
• Click "Add row".
• For the "Source" setting, enter the name of the source field in the raw event.
• For the "KUMA field" setting, enter the name of the extended event schema field that you are creating (see the table below). You can also use an existing field of the extended event schema.

  Fields of the extended data model of normalized events:

  Field name Specified in the KUMA field setting	Data type	Availability in the normalizer	Description
  S.<field name>	String	All types	Field of the "String" type
  N.<field name>	Number	All types	Field of the "Number" type
  F.<field name>	Float	All types	Field of the "Float" type
  SA.<field name>	Array of strings	KV, JSON	Field of the "Array of strings" type The order of the array elements is the same as the order of the elements of the raw event.
  NA.<field name>	Array of integers	KV, JSON	A field of the "Array of integers" type. The order of the array elements is the same as the order of the elements of the raw event.
  FA.<field name>	Array of floats	KV, JSON	Field of the "Array of floats" type The order of the array elements is the same as the order of the elements of the raw event.

The prefixes "S.", "N.", "F.", "SA.", "NA.", "FA." are required when creating fields of the extended event schema; the prefixes must be strictly uppercase.

Replace <field name> with the field name. You may use letters of the English alphabet and numerals in the field name. The space character is not allowed.

• Click OK.
• Click Save to finish editing the event normalizer.

The normalizer is saved, and the additional field is created. After saving the normalizer, the additional field can be used in normalizers and other resources. If you do not save the new normalizer with an extended event schema field, then to use the extended event schema field in enriching the normalizer itself, you must add this field. To do so, for the selected normalizer, in the Basic event parsing window on the Enrichment tab, in the Target field drop-down list, select Add <field type>.

Note: If the data in the fields of the raw event does not match the type of the KUMA field, the value is not saved during the normalization of events if type conversion cannot be performed. For example, the string "test" cannot be written to the DeviceCustomNumber1 KUMA field of the Number type.

If you want to minimize the load on the storage server when searching events, preparing reports, and performing other operations on events in storage, use KUMA event schema fields as your first preference, extended event schema fields as your second preference, and the Extra fields as your last resort.