Adding a custom or third-party feed

This section explains how to add a custom or third-party feed, and change its settings. Make sure that the General tenant is selected from the drop-down list that has all available tenants, which is located in the upper-left area of the window.

You can add feeds with only one field of the URL or DOMAIN type. That is, if you mark one field in a feed as URL or DOMAIN, do not mark another field in this feed as URL or DOMAIN. The URL and DOMAIN types are considered the same field type.

When you add a feed, it is automatically enabled in all settings tenants.

Adding a custom feed

To add a feed:

1. In the Filtering rules for feeds section, click Add custom feed.

   The Custom feed window opens:

   

   Adding a custom or third-party feed

2. For any custom or third-party feed, specify the following information:
   • Feed name

     In the feed name, you can use Latin letters, numbers, underscores, and hyphens. The name must differ from other feed names that are already used.

     Do not use FalsePositive or InternalTI as the feed name, since they are reserved for the built-in supplier names of Kaspersky CyberTrace.

     Do not use the @ character in the feed name.

   • Feed type

     This type can be one of the following:

     • json

       If a feed in the JSON format contains a field with a subnet mask value, Kaspersky CyberTrace discloses data only if it is a first-level field. If this field is nested, Kaspersky CyberTrace cannot disclose data.

       If you download Kaspersky Threat Data Feeds from https://wlinfo.kaspersky.com, the JSON format is used. You cannot change this value.

     • stix

       If a feed is in the STIX format, you also need to choose the STIX version:

       • For STIX 1.0 or 1.1, select 1
       • For STIX 2.0, select 2
       • For STIX 2.1, select 2.1
     • csv
     • xml
     • misp
     • email

       For more information on email feeds, see section "Adding a custom or third-party feed of email type".

     • PDF
   • Vendor name

     From the drop-down list, select the name of the feed vendor or add a new one.

   • Confidence

     The level of confidence of the feed. This field cannot be empty. The range of possible values is from 1 to 100.

     The preset values are 100 for feeds from Kaspersky, 50 for OSINT feeds, and 50 for third-party feeds. You can change these values.

     Level of confidence is provided in the Feeds > Feed > confidence attribute of the Kaspersky CyberTrace Service configuration file.

   • Path to the feed

     You can specify the path in one of the following forms:

     • Full path on the computer where Kaspersky CyberTrace is installed
     • Network path

       The specified network path is available for the active user account, while Kaspersky CyberTrace Service and Feed Utility run under the LocalService account. Therefore, if you need to download custom and third-party feeds from a network directory, give the LocalService user account access to this network directory.

       The network directory must be mapped.

       You can only specify the network path in Windows.

     • HTTP(S) address

       You can download Kaspersky Threat Data Feeds and differential feeds, which were not added at the moment of the product release, from https://wlinfo.kaspersky.com. For information about differential feeds, see subsection "Downloading differential feeds" from section "Working with feeds".

       You can use an IPv4 or an IPv6 address. An IPv6 address must be enclosed in square brackets. For more information, see RFC 2732.

     • FTP address

     For pdf feeds, it is allowed to specify only a local directory or network folder as a path in Windows.

   • Certificate

     Path to the certificate that gives access to the feed. The full path must be specified.

     You can only specify the certificate path if the feed will be downloaded over an HTTPS connection.

     If you download Kaspersky Threat Data Feeds from https://wlinfo.kaspersky.com, the field contains the preset value Kaspersky Lab certificate. You cannot change this value.

   • Authentication type

     The authentication type can be Basic or None.

     The basic authentication scheme is available if the path to the feed is an HTTP(S) or FTP address. For this type of authentication, enter the following settings:

     • User name

       This field cannot be empty.

     • Password

     Authentication type is provided in the Settings > Feeds > Feed parameter of the Feed Utility configuration file.

3. For a STIX feed, also specify the following information:
   • Get from a TAXII server

     If this check box is selected, the STIX feed must be downloaded from the TAXII™ server.

     For a STIX 2.0 feed, specify a TAXII 2.0 server. For a STIX 2.1 feed, specify a TAXII 2.1 server.

     When a STIX feed is downloaded from the TAXII server, Kaspersky CyberTrace parses this feed and counts the number of indicators.

   • Collection name

     The name of the collection that must be downloaded from the TAXII server. Note that you can specify only one collection name at a time.

     Kaspersky CyberTrace does not support TAXII feeds that have information about the reputation of one object. IBM feeds such as xfe.ipr and xfe.url are not supported.

4. For CSV, XML, and JSON feeds, specify the following information:
   • For a CSV feed, specify a delimiter. After that, the rule will be applied immediately and the columns will be split. By default, a semicolon (;) is used as a delimiter.
   • For an XML feed, you may specify the root element. This allows you to use the names of feed elements relative to the root element. Which element to specify as the root depends on the level of nesting in a given feed.

     You cannot use wildcard characters (the asterisk (*) or question mark (?)) to specify the path, only the root element.

   • For a JSON feed, you may specify the root element.

     You can specify root element value with any nesting level. Define the limits of the nesting level by using a "/" character.

     The root element parameter can be empty. If it is not empty, the value of the root element should not contain empty nesting levels (substring "//"), and should not start or end with a "/" character.

     You cannot use wildcards in the root element for JSON feeds.

   The following example demonstrates the root element for an XML feed, where the root element is root:

   <root> <url>http</url> <ip>1</ip> <url>https</url> <ip>2</ip> </root>

   The following example demonstrates the root element for an XML feed, where the root element is root/element*:

   <root> <element1> <url>http</url> <ip>1</ip> </element1> <element2> <url>https</url> <ip>2</ip> </element2> </root>

   The following example demonstrates the root element for a JSON feed for obtaining IP indicators from an ip field, where the root element is items/ipv4:

   { "id": 4321, "items": [ { "dateDetected": "2022-01-01T01:23:45+00:00", "dateLastSeen": "2022-05-28T12:34:56+00:00", "ipv4": [ { "asn": "AS123456, LLC", "city": "Moscow", "countryName": "Russian", "ip": "192.0.0.144" }, { "asn": "AS654321, LLC", "city": "Moscow", "countryCode": "RU", "countryName": "Russian", "ip": "191.0.0.145" } ], "ipv6": [] }, ... ] }

After you specify the initial settings for a custom or third-party feed, the feed is fully loaded and its 50 strings are displayed so that you can choose the fields to be used in the matching process (see subsection "Configuring feed fields to be used for matching (CSV, JSON, XML feeds)" below).

Selecting feed fields for matching

This is relevant for feeds in the following formats: CSV, JSON, or XML. After a STIX or Kaspersky feed is added, Kaspersky CyberTrace fully loads it for use without need for manual configuration of fields.

For PDF or email feeds, you can click "Check parsing" to check the parsing settings for PDF files or email messages (including attachments).

The first 50 strings of the result feed will be displayed, including indicators of the processed messages bodies and their attachments.

In some cases, such as when a STIX feed is too large and/or the TAXII server used for downloading the feed is too slow, it may take Kaspersky CyberTrace up to an hour to load a STIX feed.

After adding a pdf feed, update the Retention period value (preset value is 365 days).

Configuring feed fields to be used for matching (CSV, JSON, XML, and PDF feeds)

When choosing fields for matching from differential feeds, ignore all fields inside the metadata element.

To choose feed fields to be used for matching, specify the following information for each field:

• Field type

  One of the following values can be used as the field type:

  • URL
  • MD5
  • SHA1
  • SHA256
  • IP
  • DOMAIN
  • CONTEXT

    Note that there must be at least one field with a type other than CONTEXT. Such fields are used for matching. When such a field is involved in the detection process, a detection event is generated with the %FEED_NAME%_%FIELD_TYPE% category, where %FEED_NAME% is the feed name and %FIELD_TYPE% is the field type.

    Note that you can add only one field of the URL or DOMAIN type with regard to one feed.

Specify all CONTEXT fields that you want to add to a CyberTrace indicator. To properly indicate the detecting and context fields for the feed downloaded from https://wlinfo.kaspersky.com, contact your Technical Account Manager (TAM).

• Field name

  This name will be referred to in the matching process.

  In the field name, you can use Latin letters, numbers, underscores, and hyphens. The name must contain at least one Latin letter.

• Regular expression

  In this field, specify a regular expression for a PDF feed as a parsing rule.

  For URL and IP indicators (PDF feeds), the preset regular expression is configured for receiving indicators, the value of which has a dot enclosed in brackets (for example: badurl[.].com).

  In CyberTrace version 4.2, the regular expressions for MD5, SHA1 and SHA256 can extract the value fragments from longer indicators (such as longer hashes or URLs, if they consisting of the a-f characters and numbers). It is recommended to replace default regular expressions for MD5, SHA1 and SHA256 with more specific ones. For example, for MD5 you can use [^\da-fA-F]([\da-fA-F]{32})[^\da-fA-F] instead of ([\da-fA-F]{32}).

• Reference to the data in the feed:
  • For a CSV feed, specify the column number.
  • For a JSON feed, specify the field name.

    Specify the full path to the element relative to the root element. You cannot use wildcard characters (the asterisk (*) or question mark (?)) to specify the path or the root element.

    For JSON feeds, the name of the field is case-sensitive. Specify field names in the same case as they are in a JSON feed.

    To specify a nested field, use a slash (/): for example, mainField/subField.

  • For an XML feed, specify the element.

    Specify the full path to the element relative to the root element. You cannot use wildcard characters (the asterisk (*) or question mark (?)) to specify the path, only the root element. The path is case sensitive.

    In the following example, if you specified root/element* as the root element, then the full path to the elements relative to the root element is url and ip, not root/element1/url or root/element2/ip:

  <root> <element1> <url>http</url> <ip>1</ip> </element1> <element2> <url>https</url> <ip>2</ip> </element2> </root>

When adding a custom or third-party feed, feeds updating can be performed. In this case, you will be notified about it and a new feed will not be added. We recommend that you wait a while, and then try to add a feed again.