Step 3. Event parsing

Expand all | Collapse all

This is a required step of the Installation Wizard. On the Event parsing tab of the Installation Wizard, click the + Add event parsing button to open the Basic event parsing window, and in that window, in the Normalizer drop-down list, select or create a normalizer. In normalizer settings, define the rules for converting raw events into normalized events. You can add multiple event parsing rules to the normalizer to implement complex event processing logic. You can test the normalizer using test events.

When you create a new normalizer in the Installation Wizard, the default normalizer is saved in the resource set for the collector. You cannot use the created normalizer in other collectors. You can select the Save normalizer check box to create the normalizer as a separate resource. This will make the normalizer available for selection in other collectors of the tenant.

Adding a normalizer to a resource set

To add a normalizer to a resource set:

1. Click the + Add event parsing button.

   This opens the Basic event parsing window with the normalizer settings. By default, the Normalization scheme tab is selected.

2. In the Normalizer drop-down list, select a normalizer. You can select a normalizer that belongs to the tenant of the collector or to the Shared tenant.

   The normalizer settings are displayed.

   If you want to edit the settings of the normalizer, in the Normalizer drop-down list, click the pencil icon next to the name of the normalizer to open the Edit normalizer window, and in that window, click the dark circle. This opens the Basic event parsing window with the normalizer settings. If you want to edit additional parsing settings, in the Edit normalizer window, hover over the dark circle and click the plus symbol that appears. This opens the Additional event parsing with settings of the additional parsing. For more details on configuring the additional parsing of events, see the Creating the structure of event normalization rules section below.

3. Click OK.

The normalizer is added and displayed as a dark circle on the Event parsing tab of the Installation Wizard. You can click the dark circle to view the settings of the normalizer.

To create a new normalizer in the collector:

1. Click the + Add event parsing button.

   This opens the Basic event parsing window with the normalizer settings. By default, the Normalization scheme tab is selected.

2. If you want to create the normalizer as a separate resource, select the Save normalizer check box. If the check box is selected, the normalizer becomes available for selection in other collectors of the tenant. This check box is cleared by default.
3. In the Name field, enter a unique name for the normalizer. Maximum length of the name: 128 Unicode characters.
4. In the Parsing method drop-down list, select the type of events to receive. Depending on the selected value, you can use the predefined event field matching rules or define rules manually. When you select some parsing methods, additional settings may become available that you must specify.

   Available parsing methods:

   • json

     This parsing method is used to process JSON data where each object, including its nested objects, occupies a single line in a file.

     When processing files with hierarchically structured data, you can reference the fields of nested objects using the dot notation. For example, the username parameter from the string "user": {"username": "system: node: example-01"} can be accessed by using the user.username query.

     Files are processed line by line. Multi-line objects with nested structures may be normalized incorrectly.

     In complex normalization schemes where additional normalizers are used, all nested objects are processed at the first normalization level, except for cases when the extra normalization conditions are not specified and, therefore, the event being processed is passed to the extra normalizer in its entirety.

     You can use \n and \r\n as newline characters. Strings must be UTF-8 encoded.

     If you want to send the raw event for advanced normalization, at each nesting level in the Advanced event parsing window, select Yes in the Keep raw event drop-down list.

   • cef

     This parsing method is used to process CEF data.

     If you select this parsing method, you can use the predefined rules for converting events to the KUMA format by clicking Apply default mapping.

   • regexp

     This parsing method is used to create custom rules for processing data in a format using regular expressions.

     You must add a regular expression (RE2 syntax) with named capturing groups to the field under Normalization. The name of the capturing group and its value are considered the field and value of the raw event that can be converted to an event field in KUMA format.

     To add event handling rules:

     1. If necessary, copy an example of the data you want to process to the Event examples field. We recommend completing this step.
     2. In the field under Normalization, add a RE2 regular expression with named capturing groups, for example, "(?P<name>regexp)". The regular expression added to the field under Normalization must exactly match the event. When designing the regular expression, we recommend using special characters that match the starting and ending positions of the text: ^, $.

        You can add multiple regular expressions or remove regular expressions. To add a regular expression, click Add regular expression. To remove a regular expression, click the delete icon next to it.

     3. Click the Copy field names to the mapping table button.

        Capture group names are displayed in the KUMA field column of the Mapping table. You can select the corresponding KUMA field in the column opposite each capturing group. If you followed the CEF format when naming the capturing groups, you can use automatic CEF mapping by selecting the Use CEF syntax for normalization check box.

     Event handling rules are added.

   • syslog

     This parsing method is used to process data in syslog format.

     If you select this parsing method, you can use the predefined rules for converting events to the KUMA format by clicking Apply default mapping.

     To parse events in rfc5424 format with a structured-data section, in the Keep extra fields drop-down list, select Yes. This makes the values from the structured-data section available in the Extra fields.

   • csv

     This parsing method is used to create custom rules for processing CSV data.

     When choosing this parsing method, you must specify the separator of values in the string in the Delimiter field. Any single-byte ASCII character can be used as a delimiter for values in a string.

   • kv

     This parsing method is used to process data in key-value pair format. Available parsing method settings are listed in the table below.

     Available parsing method settings

     Setting	Description
     Pair delimiter	The character used to separate key-value pairs. You can specify any single-character (1 byte) value. The specified value must not match the value specified in the Value delimiter field.
     Value delimiter	The character used to separate a key from its value. You can specify any single-character (1 byte) value. The specified value must not match the value specified in the Pair delimiter field.

   • xml

     This parsing method is used to process XML data in which each object, including nested objects, occupies a single line in a file. Files are processed line by line.

     If you want to send the raw event for advanced normalization, at each nesting level in the Advanced event parsing window, select Yes in the Keep raw event drop-down list.

     If you select this parsing method, under XML attributes, you can specify the key XML attributes to be extracted from tags. If an XML structure has multiple XML attributes with different values in the same tag, you can identify the necessary value by specifying the key of the value in the Source column of the Mapping table.

     To add key XML attributes:

     1. Click + Add field.
     2. This opens a window; in that window, specify the path to the XML attribute.

     You can add multiple XML attributes or remove XML attributes. To remove an individual XML attribute, click the delete icon next to it. To remove all XML attributes, click Reset.

     If XML key attributes are not specified, then in the course of field mapping the unique path to the XML value will be represented by a sequence of tags.

     Tag numbering

     Starting with KUMA 2.1.3, you can use automatic tag numbering in XML events. This lets you parse an event with the identical tags or unnamed tags, such as <Data>.

     As an example, we will number the tags of the EventData attribute of the Microsoft Windows PowerShell event ID 800.

     <Event xmlns="http://schemas .microsoft.com/win/2004/08/events/event">

     <System>

     <Provider Name="Microsoft-Windows-ActiveDirectory_DomainService" Guid="{0e8478c5-3605-4e8c-8497-1e730c959516}" EventSourceName="NTDS" />

     <EventID Qualifiers="0000">0000</EventID>

     <Version>@</Version>

     <Level>4</Level>

     <Task>15</Task>

     <Opcode>0</Opcode>

     <Keywords >0x8080000000000000</Keywords>

     <TimeCreated SystemTime="2000-01-01T00:00:00.659495900Z" />

     <EventRecordID>55647</EventRecordID>

     <Correlation />

     <Execution ProcessID="1" ThreadID="1" />

     <Channel>service</Channel>

     <Computer>computer</Computer>

     <Security UserID="0000" />

     </System>

     <EventData>

     <Data>583</Data>

     <Data>36</Data>

     <Data>192.168.0.1:5084</Data>

     <Data>level</Data>

     <Data>name, lDAPDisplayName</Data>

     <Data />

     <Data>5545</Data>

     <Data>3</Data>

     <Data>0</Data>

     <Data>0</Data>

     <Data>0</Data>

     <Data>15</Data>

     <Data>none</Data>

     </EventData>

     </Event>

     To parse events with identical tags or unnamed tags, you need to configure tag numbering and data mapping for numbered tags with KUMA event fields.

     KUMA 3.0.x supports using XML attributes and tag numbering at the same time in the same extra normalizer. If an XML attribute contains unnamed tags or identical tags, we recommend using tag numbering. If the XML attribute contains only named tags, we recommend using XML attributes.

     To use XML attributes and tag numbering in extra normalizers, you must sequentially enable the Keep raw event setting in each extra normalizer along the path that the event follows to the target extra normalizer, and in the target extra normalizer itself.

     For an example of how tag numbering works, you can refer to the MicrosoftProducts normalizer. The Keep raw event setting is enabled sequentially in both AD FS and 424 extra normalizers.

     To set up the parsing of events with unnamed or identical tags:

     1. Open an existing normalizer or create a new normalizer.
     2. In the Basic event parsing window of the normalizer, in the Parsing method drop-down list, select xml.
     3. In the Tag numbering field, click + Add field.
     4. In the displayed field, enter the full path to the tag to whose elements you want to assign a number, for example, Event.EventData.Data. The first tag gets number 0. If the tag is empty, for example, <Data />, it is also assigned a number.
     5. To configure data mapping, under Mapping, click + Add row and do the following:
        1. In the displayed row, in the Source field, enter the full path to the tag and the index of the tag. For example, for the Microsoft Windows PowerShell event ID 800 from the example above, the full paths to tags and tag indices are as follows:
           • Event.EventData.Data.0
           • Event.EventData.Data.1
           • Event.EventData.Data.2 and so on.
        2. In the KUMA field drop-down list, select the field in the KUMA event that will receive the value from the numbered tag after parsing.
     6. Save changes in one of the following ways:
        • If you created a new normalizer, click Save.
        • If you edited an existing normalizer, in the collector to which the normalizer is linked, click Update configuration.

     Parsing is configured.

   • netflow

     This parsing method is used to process data in all supported NetFlow protocol formats: NetFlow v5, NetFlow v9, and IPFIX.

     If you select this parsing method, you can use the predefined rules for converting events to the KUMA format by clicking Apply default mapping. This takes into account the source fields of all NetFlow versions (NetFlow v5, NetFlow v9, and IPFIX).

     If the netflow parsing method is selected for the main parsing, extra normalization is not available.

     The default mapping rules for the netflow parsing method do not specify the protocol type in KUMA event fields. When parsing data in NetFlow format, on the Enrichment normalizer tab, you must create a constant data enrichment rule that adds the netflow value to the DeviceProduct target field.

   • netflow5

     This parsing method is used to process data in the NetFlow v5 format.

     If you select this parsing method, you can use the predefined rules for converting events to the KUMA format by clicking Apply default mapping. If the netflow5 parsing method is selected for the main parsing, extra normalization is not available.

     The default mapping rules for the netflow5 parsing method do not specify the protocol type in KUMA event fields. When parsing data in NetFlow format, on the Enrichment normalizer tab, you must create a constant data enrichment rule that adds the netflow value to the DeviceProduct target field.

   • netflow9

     This parsing method is used to process data in the NetFlow v9 format.

     If you select this parsing method, you can use the predefined rules for converting events to the KUMA format by clicking Apply default mapping. If the netflow9 parsing method is selected for the main parsing, extra normalization is not available.

     The default mapping rules for the netflow9 parsing method do not specify the protocol type in KUMA event fields. When parsing data in NetFlow format, on the Enrichment normalizer tab, you must create a constant data enrichment rule that adds the netflow value to the DeviceProduct target field.

   • sflow5

     This parsing method is used to process data in sflow5 format.

     If you select this parsing method, you can use the predefined rules for converting events to the KUMA format by clicking Apply default mapping. If the sflow5 parsing method is selected for the main parsing, extra normalization is not available.

   • ipfix

     This parsing method is used to process IPFIX data.

     If you select this parsing method, you can use the predefined rules for converting events to the KUMA format by clicking Apply default mapping. If the ipfix parsing method is selected for the main parsing, extra normalization is not available.

     The default mapping rules for the ipfix parsing method do not specify the protocol type in KUMA event fields. When parsing data in NetFlow format, on the Enrichment normalizer tab, you must create a constant data enrichment rule that adds the netflow value to the DeviceProduct target field.

   • sql

     The normalizer uses this parsing method to process data obtained by making a selection from the database.

     This parsing method becomes available only if at the Transport step of the Installation Wizard you specified a connector of the sql type.

   • normalizer

     This parsing method allows you to reuse a previously created normalizer as the primary or extra normalizer for processing events.

     If you select this parsing method, in the Normalizer drop-down list, you need to select one of the previously created normalizers of the json, cef, regexp, syslog, csv, kv, xml, or sql type. Normalizers with the normalizer parsing method cannot be selected.

5. In the Keep raw event drop-down list, specify whether you want to store the original raw event in the re-created normalized event.
   • Don't save—do not save the raw event. This value is selected by default.
   • Only errors—save the raw event in the Raw field of the normalized event if errors occurred when parsing it. You can use this value to debug the service. In this case, an event having a non-empty Raw field will indicate problems.
   • Always—always save the raw event in the Raw field of the normalized event.
6. In the Keep extra fields drop-down list, choose whether you want to store the raw event fields in the normalized event if no mapping is configured for them (see step 8 of these instructions):
   • No. This value is selected by default.
   • Yes. The original event fields are saved in the Extra field of the normalized event.

   This field is not available if the Parsing method is set to normalizer.

7. If necessary, provide an example of the data you want to process to the Event examples field. We recommend completing this step.

   You can add multiple event examples. Every new example must be specified on a new line. At the same time, in the Mapping table, its parsing result will be displayed in the Examples column. This lets you check whether the fields of different types of incoming events are mapped correctly.

   Limitations involved in parsing event examples

   When parsing the examples of data specified in the Event examples field, the following parameters are not taken into account in the extra normalizer:

   • Enrichment configured in the root normalizer
   • Filters specified in the extra normalizer
   • The Use raw event setting
   • Conversions in the root normalizer
   • Conversions for the value of the Field to pass into normalizer field.

   This field is not available if the Parsing method is set to normalizer.

8. In the Mapping table, configure the mapping of raw event fields to KUMA event fields:
   1. Click + Add row.

      You can add multiple table rows or delete table rows. To delete a table row, select the check box next to it and click the Delete button.

   2. In the Source column, specify the name of the raw event field that you want to map to the KUMA event field.

      For details about the field name format, refer to the Normalized event data model article. For a description of the mapping, refer to the Mapping fields of predefined normalizers article.

      If you want to create rules for modifying the fields of the original event before writing to the KUMA event fields, click the settings icon next to the field name to open the Conversion window, and in that window, click + Add conversion. You can reorder the rules or delete the rules. To reorder rules, use the reorder icons. To delete a rule, click the delete icon next to it.

      Available conversions

      Conversions are modifications that are applied to a value before it is written to the event field. You can select one of the following conversion types from the drop-down list:

      • entropy is used for converting the value of the source field using the information entropy calculation function and placing the conversion result in the target field of the float type. The result of the conversion is a number. Calculating the information entropy allows detecting DNS tunnels or compromised passwords, for example, when a user enters the password instead of the login and the password gets logged in plain text.
      • lower—is used to make all characters of the value lowercase
      • upper—is used to make all characters of the value uppercase
      • regexp – used to convert a value using a specified RE2 regular expression. When you select this type of conversion, a field is displayed in which you must specify the RE2 regular expression.
      • substring is used to extract characters in a specified range of positions. When you select this type of conversion, the Start and End fields are displayed, in which you must specify the range of positions.
      • replace—is used to replace specified character sequence with the other character sequence. When you select this type of conversion, the following fields are displayed:
        • Replace chars specifies the sequence of characters to be replaced.
        • With chars is the character sequence to be used instead of the character sequence being replaced.
      • trim removes the specified characters from the beginning and from the end of the event field value. When you select this type of conversion, the Chars field is displayed in which you must specify the characters. For example, if a trim conversion with the Micromon value is applied to Microsoft-Windows-Sysmon, the new value is soft-Windows-Sys.
      • append appends the specified characters to the end of the event field value. When you select this type of conversion, the Constant field is displayed in which you must specify the characters.
      • prepend prepends the specified characters to the beginning of the event field value. When you select this type of conversion, the Constant field is displayed in which you must specify the characters.
      • replace with regexp is used to replace RE2 regular expression results with the specified character sequence. When you select this type of conversion, the following fields are displayed:
        • Expression is the RE2 regular expression whose results you want to replace.
        • With chars is the character sequence to be used instead of the character sequence being replaced.
      • Converting encoded strings to text:
        • decodeHexString—used to convert a HEX string to text.
        • decodeBase64String—used to convert a Base64 string to text.
        • decodeBase64URLString—used to convert a Base64url string to text.

        When converting a corrupted string or if conversion error occur, corrupted data may be written to the event field.

        During event enrichment, if the length of the encoded string exceeds the size of the field of the normalized event, the string is truncated and is not decoded.

        If the length of the decoded string exceeds the size of the event field into which the decoded value is to be written, the string is truncated to fit the size of the event field.

      • Converting the IP address to the IPv4 format:
        • ipDecimalToDotted converts an IP address in decimal format to an IP address in IPv4 format, in which octets are separated by dots.
        • ipHexToDotted converts a hexadecimal IP address to a decimal IP address in IPv4 format, in which octets are separated by dots.

      Conversions when using the extended event schema

      Whether or not a conversion can be used depends on the type of extended event schema field being used:

      • For an additional field of the String type, all types of conversions are available.
      • For fields of the Number and Float types, the following types of conversions are available: regexp, substring, replace, trim, append, prepend, replaceWithRegexp, decodeHexString, decodeBase64String, and decodeBase64URLString.
      • For fields of Array of strings, Array of numbers, and Array of floats types, the following types of conversions are available: append and prepend.

      If at the Transport step of the Installation Wizard, you specified a connector of the file type, you can pass the name or path of the file being processed by the collector to the KUMA event field. To do this, in the Source column, specify one of the following values:

      • $kuma_fileSourceName to pass the name of the file being processed by the collector in the KUMA event field.
      • $kuma_fileSourcePath to pass the path to the file being processed by the collector in the KUMA event field.

      When you use a file connector, the new variables in the normalizer will only work with destinations of the internal type.

   3. In the KUMA field column, select a KUMA event field from the drop-down list. You can find the KUMA event field by typing its name. If the name of the KUMA event field begins with DeviceCustom* or Flex*, enter a unique custom label in the Label field, if necessary.

   If you want KUMA to enrich events with asset information, and the asset information to be available in the alert card when a correlation rule is triggered, in the Mapping table, configure a mapping of host address and host name fields depending on the purpose of the asset. For example, you can configure a mapping for SourceAddress and SourceHostName, or DestinationAddress and DestinationHostName KUMA event fields. As a result of enrichment, the event card includes a SourceAssetID or DestinationAssetID KUMA event field, and a link to the asset card. As a result of enrichment, asset information is also available in the alert card.

   If you have loaded data into the Event examples field, the table will have an Examples column containing examples of values carried over from the raw event field to the KUMA event field.

   The Mapping table is not available if the Parsing method is set to normalizer.

9. Click OK.

The normalizer is created and displayed as a dark circle on the Event parsing tab of the Installation Wizard. You can click the dark circle to view the settings of the normalizer. If you want to edit additional parsing settings, hover over the dark circle and click the plus symbol that appears. This opens the Additional event parsing with settings of the additional parsing. For more details on configuring the additional parsing of events, see the Creating the structure of event normalization rules section below.

When displaying a normalization scheme, the structure of nested normalizers is not displayed. To view this structure, you need to go to the normalizer card by clicking its name link. If you are using a nested normalizer with extra normalization, keep in mind that the structure of the output data may change. In that case, it is important to understand which field is passed to the other extra normalizer.

Enriching normalized events with additional data

You can create enrichment rules in the normalizer to add extra data to created normalized events. Enrichment rules are stored in the normalizer in which they were created.

The Enrichment tab is not available if on the Normalization scheme tab, the Parsing method is set to normalizer.

To add enrichment rules to the normalizer:

1. Select the main or additional normalization rule to open a window, and in that window, select the Enrichment tab.
2. Click the + Add enrichment button.

   The enrichment rule settings section is displayed. You can add multiple enrichment rules or delete enrichment rules. To delete an enrichment rule, click the delete icon next to it.

3. In the Source kind drop-down list, select the type of the enrichment source. When you select some enrichment source types, additional settings may become available that you must specify.

   Available Enrichment rule source types:

   • constant

     This type of enrichment is used when a constant needs to be added to an event field. Available enrichment type settings are listed in the table below.

     Available enrichment type settings

     Setting	Description
     Constant	The value to be added to the event field. Maximum length of the value: 255 Unicode characters. If you leave this field blank, the existing event field value is removed.
     Target field	The KUMA event field that you want to populate with the data.

     If you are using the event enrichment functions for extended schema fields of String, Number, or Float type with a constant, the constant is added to the field.

     If you are using the event enrichment functions for extended schema fields of Array of strings, Array of numbers, or Array of floats type with a constant, the constant is added to the elements of the array.

   • dictionary

     This type of enrichment is used if you need to add a value from the dictionary of the Dictionary type. Available enrichment type settings are listed in the table below.

     Available enrichment type settings

     Setting	Description
     Dictionary name	The dictionary from which the values are to be taken.
     Key fields	Event fields whose values are to be used for selecting a dictionary entry. To add an event field, click Add field. You can add multiple event fields.

     If you are using event enrichment with the dictionary type selected as the Source kind setting, and an array field is specified in the Key enrichment fields setting, when an array is passed as the dictionary key, the array is serialized into a string in accordance with the rules of serializing a single value in the TSV format.

     Example: The Key fields setting of the enrichment uses the SA.StringArrayOne extended schema field. The SA.StringArrayOne extended schema field contains the values "a", "b", "c". The following values are passed to the dictionary as the key: ['a','b','c'].

     If the Key enrichment fields setting uses an extended schema array field and a regular event schema field, the field values are separated by the | character when the dictionary is queried.

     Example: The Key enrichment fields setting uses the SA.StringArrayOne extended schema field and the Code string field. The SA.StringArrayOne extended schema field contains the values "a", "b", "c", and the Code string field contains the myCode sequence of characters. The following values are passed to the dictionary as the key: ['a','b','c']|myCode.

   • table

     This type of enrichment is used if you need to add a value from the dictionary of the Table type. Available enrichment type settings are listed in the table below.

     Available enrichment type settings

     Setting	Description
     Dictionary name	The dictionary from which the values are to be taken.
     Key fields	Event fields whose values are to be used for selecting a dictionary entry. To add an event field, click Add field. You can add multiple event fields.
     Mapping	Event fields for data transfer: Dictionary field specifies dictionary fields from which data is to be transmitted. The available fields depend on the selected dictionary resource. KUMA field specifies event fields to which data is to be transmitted. For some of the selected fields (*custom* and *flex*), in the Label column, you can specify a name for the data written there.

     The first field in the table (Dictionary field) is taken as the key with which the fields selected from the event as key fields are matched (KUMA field). As the key in the Dictionary field, you must select an indicator of compromise by which the enrichment is to be performed, for example, IP address, URL, or hash. In the rule, you must select the event field that corresponds to the selected indicator of compromise in the dictionary field.

     If you want to select multiple key fields, you can specify them using | as a separator (when specifying in the web interface or importing as a CSV file), for example, <IP address>|<user name>.

     You can add new table rows or delete table rows. To add a new table row, click Add new element. To delete a row in the table, click the button.

   • event

     This type of enrichment is used when you need to write a value from another event field to the current event field. Available enrichment type settings are listed in the table below.

     Available enrichment type settings

     Setting	Description
     Target field	The KUMA event field that you want to populate with the data.
     Source field	The event field whose value is written to the target field.

     Clicking opens the Conversion window, in which you can click Add conversion to create rules for modifying the source data before writing them to the KUMA event fields. You can reorder and delete created rules. To change the position of a rule, click next to it. To delete a rule, click next to it.

     Available conversions

     Conversions are modifications that are applied to a value before it is written to the event field. You can select one of the following conversion types from the drop-down list:

     • entropy is used for converting the value of the source field using the information entropy calculation function and placing the conversion result in the target field of the float type. The result of the conversion is a number. Calculating the information entropy allows detecting DNS tunnels or compromised passwords, for example, when a user enters the password instead of the login and the password gets logged in plain text.
     • lower—is used to make all characters of the value lowercase
     • upper—is used to make all characters of the value uppercase
     • regexp – used to convert a value using a specified RE2 regular expression. When you select this type of conversion, a field is displayed in which you must specify the RE2 regular expression.
     • substring is used to extract characters in a specified range of positions. When you select this type of conversion, the Start and End fields are displayed, in which you must specify the range of positions.
     • replace—is used to replace specified character sequence with the other character sequence. When you select this type of conversion, the following fields are displayed:
       • Replace chars specifies the sequence of characters to be replaced.
       • With chars is the character sequence to be used instead of the character sequence being replaced.
     • trim removes the specified characters from the beginning and from the end of the event field value. When you select this type of conversion, the Chars field is displayed in which you must specify the characters. For example, if a trim conversion with the Micromon value is applied to Microsoft-Windows-Sysmon, the new value is soft-Windows-Sys.
     • append appends the specified characters to the end of the event field value. When you select this type of conversion, the Constant field is displayed in which you must specify the characters.
     • prepend prepends the specified characters to the beginning of the event field value. When you select this type of conversion, the Constant field is displayed in which you must specify the characters.
     • replace with regexp is used to replace RE2 regular expression results with the specified character sequence. When you select this type of conversion, the following fields are displayed:
       • Expression is the RE2 regular expression whose results you want to replace.
       • With chars is the character sequence to be used instead of the character sequence being replaced.
     • Converting encoded strings to text:
       • decodeHexString—used to convert a HEX string to text.
       • decodeBase64String—used to convert a Base64 string to text.
       • decodeBase64URLString—used to convert a Base64url string to text.

       When converting a corrupted string or if conversion error occur, corrupted data may be written to the event field.

       During event enrichment, if the length of the encoded string exceeds the size of the field of the normalized event, the string is truncated and is not decoded.

       If the length of the decoded string exceeds the size of the event field into which the decoded value is to be written, the string is truncated to fit the size of the event field.

     • Converting the IP address to the IPv4 format:
       • ipDecimalToDotted converts an IP address in decimal format to an IP address in IPv4 format, in which octets are separated by dots.
       • ipHexToDotted converts a hexadecimal IP address to a decimal IP address in IPv4 format, in which octets are separated by dots.

     Conversions when using the extended event schema

     Whether or not a conversion can be used depends on the type of extended event schema field being used:

     • For an additional field of the String type, all types of conversions are available.
     • For fields of the Number and Float types, the following types of conversions are available: regexp, substring, replace, trim, append, prepend, replaceWithRegexp, decodeHexString, decodeBase64String, and decodeBase64URLString.
     • For fields of Array of strings, Array of numbers, and Array of floats types, the following types of conversions are available: append and prepend.

     When using enrichment of events that have event selected as the Source kind and the extended event schema fields are used as arguments, the following special considerations apply:

     • If the source extended event schema field has the Array of strings type, and the target extended event schema field has the String type, the values are written to the target extended event schema field in TSV format.

       Example: the SA.StringArray extended event schema field contains values: "string1", "string2", "string3". An event enrichment operation is performed. The result of the event enrichment operation is written to the DeviceCustomString1 extended event schema field. The DeviceCustomString1 extended event schema field contains values: ["string1", "string2", "string3"].

     • If the source and target extended event schema fields have the Array of strings type, values of the source extended event schema field are added to the values of the target extended event schema field, and the "," character is used as the delimiter.

       Example: The SA.StringArrayOne field of the extended event scheme contains the ["string1", "string2", "string3"] values, and the SA.StringArrayTwo field of the extended event scheme contains the ["string4", "string5", "string6"] values. An event enrichment operation is performed. The result of the event enrichment operation is written to the SA.StringArrayTwo field of the extended event scheme. The SA.StringArrayTwo extended event schema field contains values: ["string4", "string5", "string6", "string1", "string2", "string3"].

   • template

     This type of enrichment is used when you need to write the result of processing Go templates into the event field. We recommend matching the value and the size of the field. Available enrichment type settings are listed in the table below.

     Available enrichment type settings

     Setting	Description
     Template	The Go template. Event field names are passed in the {{.EventField}} format, where EventField is the name of the event field from which the value must be passed to the script, for example, {{.DestinationAddress}} attacked from {{.SourceAddress}}.
     Target field	The KUMA event field that you want to populate with the data.

     If you are using an enrichment of events in which the Source kind is template, and the target field has the String type, and the source field is an extended event schema field containing an array of strings, you can use one of the following examples for the template:

     • {{.SA.StringArrayOne}}
     • {{- range $index, $element := . SA.StringArrayOne -}}

       {{- if $index}}, {{end}}"{{$element}}"{{- end -}}

     To convert the data in an array field in a template into the TSV format, use the toString function, for example:

     template {{toString .SA.StringArray}}

4. In the Target field drop-down list, select the KUMA event field to which you want to write the data. You cannot select a value if in the Source kind drop-down list, you selected table.
5. If you want to enable details in the normalizer log, enable the Debug toggle switch. The toggle switch is turned off by default.
6. Click OK.

The event enrichment rules with additional data are added to the normalizer into the selected parsing rule.

Configuring parsing linked to IPv4 addresses

If at the Transport step of the Installation Wizard, you specified a connector of the udp, tcp, or http type, you can forward events from multiple IPv4 addresses from sources of different types to the same collector, and have the collector apply the specified normalizers. To do this, you need to specify several IPv4 addresses and select the normalizer that you want to apply to events coming from the specified IPv4 addresses. The following types of normalizers are available: json, cef, regexp, syslog, csv, kv, and xml.

If you select a connector type other than udp, tcp, or http in a collector with configured normalizers and linking to IPv4 addresses, the Parsing settings tab is no longer displayed, and only the first of the previously specified normalizers is specified on the Event parsing tab of the Installation Wizard. The Parsing settings tab is hidden immediately, and the changes are applied after the resource is saved. If you want to restore the previous settings, exit the Installation Wizard without saving.

For normalizers of the Syslog and regexp types, you can use of a chain of normalizers. In this case, you can specify additional normalization conditions depending on the value of the DeviceProcessName field. The difference from extra normalization is that you can specify shared normalizers.

To configure parsing with linking to IPv4 addresses:

1. Select the Parsing settings tab and click the + Event source button.

   A group of parsing settings is displayed. You can add multiple parsings or delete parsings. To remove a parsing, click the delete icon next to it.

2. In the IP address(es) field, enter the IPv4 address from which events will be sent. You can specify multiple IP addresses separated by commas. The length of the IPv4 address list is unlimited; however, we recommend limiting the number of IPv4 addresses to keep the load on the collector balanced. You must specify a value in this field if you want to apply multiple normalizers in one collector.

   The IPv4 address must be unique for each normalizer. KUMA checks the uniqueness of IPv4 addresses, and if you specify the same IPv4 address for different normalizers, the Field must be unique message is displayed.

   If you want to send all events to the same normalizer without specifying IPv4 addresses, we recommend creating a separate collector. To improve performance, we recommend creating a separate collector with one normalizer if you want to apply the same normalizer to events from a large number of IPv4 addresses.

3. In the Normalizer drop-down list, create or select a normalizer. You can click the arrow icon next to the drop-down list to select the Parsing schemas tab.

   Normalization will be triggered if at the Transport step of the Installation Wizard, you specified a connector of the udp, tcp, or http type. For a http connector, you must specify the header of the event source. Taking into account the available connectors, the following normalizer types are available for automatic source recognition: json, cef, regexp, syslog, csv, kv, and xml.

4. If you selected a normalizer of the Syslog or regexp type, and you want to add a conditional normalization, click the + Add conditional normalization button. Conditional normalization is available if in the Mapping table of the main normalizer, you configured the mapping of the source event field to the DeviceProcessName KUMA event field. Under Condition, in the DeviceProcessName field, specify a process name, and in the drop-down list, create or select a normalizer. You can specify multiple combinations of the DeviceProcessName KUMA event field and a normalizer. Normalization is performed until the first match.

Parsing with linking to IPv4 addresses is configured.

Creating a structure of event normalization rules

To implement complex event processing logic, you can add multiple event parsing rules to the normalizer. Events are switched between the parsing rules depending on the specified conditions. Events are processed sequentially in accordance with the creation order of the parsing rules. The event processing path is displayed as arrows.

To create an additional parsing rule:

1. Create a normalizer. For more information on creating normalizers, see the Adding a normalizer to a resource set section earlier in this article.

   The created normalizer is displayed as a dark circle on the Event parsing tab of the Installation Wizard.

2. Hover over the dark circle and click the plus sign that appears.
3. In the Additional event parsing window that opens, configure the additional event parsing rule:
   • Extra normalization conditions tab:

     If you want to send a raw event for extra normalization, select Yes in the Keep raw event drop-down list. The default value is No. We recommend passing a raw event to normalizers of json and xml types. If you want to send a raw event for extra normalization to the second, third, etc nesting levels, at each nesting level, select Yes in the Keep raw event drop-down list.

     To send only the events with a specific field to the additional normalizer, specify the field in the Field to pass into normalizer field.

     On this tab, you can define other conditions that must be satisfied for the event to be sent for additional parsing.

   • Normalization scheme tab:

     On this tab, you can configure event processing rules, similar to the main normalizer settings. The Keep raw event setting is not available. The Event examples field displays the values specified when the normalizer was created.

   • Enrichment tab:

     On this tab, you can configure enrichment rules for events. For more details on configuring enrichment rules, see the Enriching normalized events with additional data section earlier in this article. This tab is not available if on the Normalization scheme tab, the Parsing method is set to normalizer.

4. Click OK.

An additional parsing rule is added to the normalizer and displayed as a dark box. The dark box specifies the conditions that trigger the additional parsing rule.

You can do the following:

• You can chick the additional parsing rule to edit its settings.
• You can find an additional parsing rule by entering its name in the field in the upper part of the window.
• Create a new additional parsing rule. To do this, hover over the additional parsing rule and click on the plus icon that appears.
• Duplicate an additional parsing rule. To do this, hover over the additional parsing rule and click on the duplicate icon.
• Delete an additional parsing rule. To do this, hover over the additional parsing rule and click on the trash can icon.
• Change the position and order of additional normalizers. To do this, select the check box next to the extra normalizer and in the toolbar, click Move. This opens the Move normalizer window that opens, select the required values in the New parent normalizer and Insert after normalizer drop-down lists and click Save. If nothing is selected in the Insert after normalizer drop-down list, the additional normalizer is moved to the beginning of the sequence of additional normalizers in the selected parent normalizer. In the lower part of the additional normalizer section, the number of additional normalizers associated with it is indicated.

  You can move multiple additional normalizers of the same level together. If you move an additional normalizer that has child normalizers, the whole chain is moved.

• Adjust the display of the structure of normalization rules in the toolbar using Collapse by default, Expand all, and Compact view buttons.

  By default, all additional normalizers at second level and below are collapsed. The main normalizer is the zero level, followed by the first level of extra normalizers, and the rest of the extra normalizers are collapsed. When you go to the normalization scheme, the scheme is displayed collapsed by default. For each item in the normalizer tree, you can collapse the child items using the button or expand them using the button. The buttons are displayed when you hover over the parent normalizer.

Proceed to the next step of the Installation Wizard.

Page top