Working with feeds

Feed Utility is a tool that can download, filter, and compile Kaspersky Threat Data Feeds according to a specified set of rules defined in its configuration file. These rules can also be set by using Kaspersky CyberTrace Web.

Downloading

Feed Utility downloads archives containing feeds from the update servers. Each downloaded archive contains one feed. Before downloading Kaspersky Threat Data Feeds, Feed Utility checks whether they are newer than those being used. Before downloading OSINT and third-party feeds, Feed Utility does not perform such checking.

Feed Utility uses a certificate for authentication. The certificate also defines which Kaspersky Threat Data Feeds can be downloaded by Feed Utility. For example, if you have a demo certificate, Feed Utility can download only demo feeds.

If you have trouble downloading feeds from third-party sources, check section "Feed Utility troubleshooting", subsection "An SSL error occurred while downloading a third-party feed".

Downloading diff feeds

Feed Utility supports the downloading of delta updates for specific Kaspersky Threat Data Feeds. Such feeds are called diff feeds in this document. Diff feeds are similar to Kaspersky Threat Data Feeds updated under the regular scenario, but have different feed IDs. For the list of available diff feeds, refer to section "About Kaspersky Threat Data Feeds".

To download a diff feed:

Specify the ID of the diff feed in the Feed Utility configuration file.

For diff feeds, there are snapshots and diff parts available on the update servers. A snapshot is a full version of the feed generated daily. A diff part of the feed contains changes that must be applied to the feed to make it up-to-date.

Feed Utility updates diff feeds as follows:

  1. The snapshot is downloaded in the following cases:
    • The feed is being downloaded for the first time.
    • The feed was updated a long time ago and it is not feasible to download diff parts.

      The feasibility of downloading diff parts is determined by the update server.

  2. One or more archives with diff parts are downloaded if they are newer than the snapshot downloaded in step 1 or the feed file is currently in use.

    When an archive with a diff part is downloaded, it is renamed to %FILE_NAME_SRC%_%TIMESTAMP%.zip, where %FILE_NAME_SRC% is the initial file name (without an extension) and %TIMESTAMP% is the timestamp of the diff part publication in the yyyy-mm-dd HH:MM:SS format.

  3. If Feed Utility is running in processing mode (with the -p option), the previously downloaded and unpacked diff parts are applied to the snapshot located in the WorkDir directory (for more information about the WorkDir parameter, see section "Configuration file parameters (Feed Utility)").

    When an archive with a diff part is unpacked, it is renamed to %FILE_NAME_SRC%_%TIMESTAMP%.json, where %FILE_NAME_SRC% is the initial file name (without an extension) and %TIMESTAMP% is the timestamp of the diff part publication in the yyyy-mm-dd HH:MM:SS format.

By default, Feed Utility downloads diff parts of a feed in parallel. To enable sequential downloading, set the SequentialDownload parameter of the Feed Utility configuration file to True.

Diff versions of Kaspersky Threat Data Feeds (if they exist) can be downloaded by using the same certificate as for the regular versions.

Processing and filtering

After the archives containing feeds are downloaded, Feed Utility unpacks the archives and processes the original feed files. The feed files are modified according to a combination of feed rules, filtering rules, and other parameters specified in the Feed Utility configuration file. These parameters define the data that will be included in the resulting feeds, the output format of the resulting feeds, and the maximum number of records in the resulting feed.

Filtering is the process of modifying the original feed files according to specified filtering criteria. Filtering criteria are defined in the filtering rules for each feed. Depending on the intended Feed Utility usage scenario, you may want to create a feed that uses only a subset of information contained in the original feed. This can be achieved by using a combination of feed rules and filtering rules.

Default filtering rules

The default Feed Utility configuration file that is shipped in the Kaspersky CyberTrace distribution kit contains the following filtering rule for IP Reputation Data Feed and Demo IP Reputation Data Feed:

Only feed records whose threat_score parameter is not less than 75 are detected.

Kaspersky considers malicious those IP addresses whose threat_score is not less than 75. IP addresses whose threat_score is less than 75 are considered related to spam and posing no threat.

If you reduce the boundary value or remove the filter, you will have many detections by Demo IP Reputation Data Feed and IP Reputation Data Feed that will be mere notifications about detecting spam IP addresses.

To remove the filter:

  1. Open the Settings > Feeds page of Kaspersky CyberTrace Web.
  2. For Demo IP Reputation Data Feed (or IP Reputation Data Feed) remove the filtering rule for the threat_score field.

Compiling

If you use Feed Utility with Feed Service, feeds that contain URL masks must be converted to binary format. Feed Utility compiles the URL masks extracted from these feeds and creates binary files which are then used by Feed Service to quickly match URLs from received events to URL masks. Compiling is performed automatically by Feed Utility, if the UrlMatcherField option is specified in the feed rules.

Reloading

When notified, Feed Service reloads the feeds for use, that is, it unloads the old feeds from memory and loads the new ones.

Updating feeds

Page top