Adding a custom or third-party feed of email type

This section explains how to add a custom or third-party feed of the email type and change its settings.

To add a feed:

In the Filtering rules for feeds section, click Custom feeds.
The Custom feed window opens.
In the Feed type field, specify the email type of feed.
The Custom feed window changes to the following:

Adding a feed of email type
For the feed of email type, specify the following information:
- Feed name
  In the feed name, you can use Latin letters, numbers, underscores, and hyphens. The name must differ from other feed names that are already used.
  
  Do not use FalsePositive or InternalTI as the feed name, since they are reserved for the built-in supplier names of Kaspersky CyberTrace.
  
  Do not use the @ character in the feed name if basic authentication is used, and the user name or password contains @.
- Feed type
  Specify the email type of feed.
  
  Select the Use TLS/SSL check box to enable interaction with the mail server by using TLS/SSL certificates.
  
  The Use TLS/SSL check box is selected by default. Uncheck it if the mail server does not use TLS/SSL.
- Host
  Host name or IP address of the mail server to connect to. This field cannot be empty.
- Port
  Mail server port to connect to. This field cannot be empty.
- Mail server type
  Protocol for interaction with the mail server. Select either POP3 or IMAP. By default, IMAP is selected.
- Vendor name
  From the drop-down list, select the name of the feed vendor or create a new vendor by selecting [Add new vendor].
- Confidence
  The level of confidence of the feed. This field cannot be empty. The range of possible values is from 1 to 100.
  
  The preset values are 100 for feeds from Kaspersky, 50 for OSINT feeds, and 50 for third-party feeds. You can change these values.
- Days to process emails
  Number of days needed for processing messages from the mail server. This field cannot be empty.
  
  Specify an integer positive number. The maximum value is 365. The preset value is 7.
- Login
  User account for connecting to the mail server. This field cannot be empty.
- Password
  Account password for connecting to the mail server. This field cannot be empty.
- Check connection
  Click the Check connection button to connect to the mail server. The result will be displayed in the pop-up window.

After you fill in the fields above, click Next to continue to another settings window, and then specify filtering rules for email messages and parsing rules for the message body and message attachment.

After adding an email feed, update the Retention period value (preset value is 365 days).

Configuring filtering and parsing rules for feeds of email type

The window below contains the fields for configuring filtering and parsing rules:

Custom feed window. Message attachments parsing.

Filtering and parsing rules

Adding message filters

Message filters set the rules for selecting emails for parsing. You can select emails by subject and sender. All rules use the AND condition.

To add message filters:

In the Email filtering rules section of the Custom feed window, click the Add rule button.
You can add one or more rules. If no rules are added, parsing will be applied to all emails from the server.
For every field, specify the following information:
- Field for filter
  Filtering rules can be applied to one of the following fields from an email message:
  - From
    The sender of the email message.
  - Subject
    The subject of the email message.
- Filtration condition
  Select one of the following values:
  - Contains (by default)
    The value from the email message must contain the value from this field.
    
    This value is default.
  - Not contains
    The value from the email message must not contain the value from this field.
  - Match
    The value from the email message must be equal to the value from this field.
  - Not match
    The value from the email message must not be equal to the value from this field.
- Value
  Criterion that is used to filter email messages from the mail server.
  
  This field cannot be empty.

The email server forms the From field in two variants: sender@mail.ru or sender<sender@mail.ru>.

If Field for filter has From, and Filtration condition has Match, the Value will be compared with the sender@mail.ru value (if the From field has sender@mail.ru), or will be compared with the value in parentheses (if the From field has sender<sender@mail.ru>).

Configuring parsing rules for the message attachment

These rules contain settings for attachment types, as well as regular expressions for parsing indicators and attachment context.

To specify parsing rules for the message attachment:

Go to Message attachments parsing.
The message attachment parsing is active by default. You can apply one or several types of attachment. The attachment of one type can be applied only once.
In the Attachment section, specify the fields that depend on the type of the attachment you select.
- Attachment type
  From the drop-down list, select the type of the attachment. You can use one of the following values as the attachment type:
  - csv
    For a CSV attachment, specify a delimiter to split the columns. Semicolon (;) is used as a preset delimiter.
    
    The fields for specifying in the Attachment section are: Field type, Field name, Column number.
  - json
    The fields for specifying in the Attachment section are: Field type, Field name, Root element.
  - stix1
    If parsing rules for stix1 and xml are simultaneously specified in the email feed settings, the stix1 file will be parsed first.
    
    If the stix1 file has been parsed without errors, it is no longer processed as a regular xml file.
  - stix2
    If parsing rules for stix2 and json are simultaneously specified in the email feed settings, the stix2 file will be parsed first.
    
    If the stix2 file has been parsed without errors, it is no longer processed as a regular json file.
  - pdf
    The fields for specifying in the Attachment section are: Field type, Field name, Regular expression.
    
    In the Regular expression field you can choose the preset regular expressions, or correct them, if necessary.
    
    For URL and IP indicators, the preset regular expression is configured for receiving indicators, the value of which has a dot enclosed in brackets (for example: badurl[.].com).
    
    In Kaspersky CyberTrace starting from version 4.2, the regular expressions for MD5, SHA1, and SHA256 can extract the value fragments from longer indicators (such as longer hashes or URLs, if they consist of the a-f characters and numbers). It is recommended to replace default regular expressions for MD5, SHA1, and SHA256 with more specific ones. For example, for MD5 you can use [^\da-fA-F]([\da-fA-F]{32})[^\da-fA-F] instead of ([\da-fA-F]{32}).
  - xml
    For an XML attachment, specify the root element. This allows you to use the names of feed elements relative to the root element. Which element to specify as the root depends on the level of nesting in a given feed. See the example of the root element for an XML feed in Step 4 of section "Adding a custom or third-party feed".
    
    The fields for specifying in the Attachment section are: Field type, Field name, Element.
- Field type
  Select an indicator type.
  
  This field is not available for stix1 and stix2.
- Field name
  This name will be referred to in the matching process.
  
  In the field name, you can use Latin letters, numbers, underscores, and hyphens. The name must contain at least one Latin letter.
  
  This field is not available for stix1 and stix2.
- Root element
  Specify the root element for the JSON attachment type. You can specify a root element value with any nesting level. Define the limits of the nesting level with a "/" character.
  
  The root element parameter can be empty. If it is not empty, the value of the root element should not contain empty nesting levels (substring "//"), and should not start or end with a "/" character.
  
  You cannot use wildcards in the root element for JSON feeds.
Use regular expressions if you want to apply exclusion rules.
Click Check parsing to check the parsing settings for attachments.
The first 50 strings of the result feed will be displayed including indicators of the processed messages bodies and their attachments.
Click Save.

Configuring parsing rules for the message body

These rules contain regular expressions for parsing indicators and message body context.

To specify parsing rules for the message body:

Select Message body parsing in the Custom feed window.
In the Rules section, specify the following fields:
- Field type
  Select an indicator type.
- Field name
  This name will be referred to in the matching process.
  
  In the field name, you can use Latin letters, numbers, underscores, and hyphens. The name must contain at least one Latin letter.
- Regular expression
  In the Regular expression field, you can choose the preset regular expressions or correct them, if necessary.
  
  For URL and IP indicators, the preset regular expression is configured for receiving indicators, the value of which has a dot enclosed in brackets (for example: badurl[.].com).
  
  In Kaspersky CyberTrace starting from version 4.2, the regular expressions for MD5, SHA1, and SHA256 can extract the value fragments from longer indicators (such as longer hashes or URLs, if they consist of the a-f characters and numbers). It is recommended to replace default regular expressions for MD5, SHA1, and SHA256 with more specific ones. For example, for MD5 you can use [^\da-fA-F]([\da-fA-F]{32})[^\da-fA-F] instead of ([\da-fA-F]{32}).
  
  Click Add new expression to apply one more regular expression.
  
  Use regular expressions, if you want to apply exclusion rules.
Click Check parsing to check the parsing settings for the message body.
The first 50 strings of the result feed will be displayed, including indicators of the bodies of the processed messages and their attachments.
Click Save.

Page top