Working with feeds

Feed Utility is a tool that can download, filter, and compile Kaspersky Threat Data Feeds according to a specified set of rules defined in its configuration file. These rules can also be set by using Kaspersky CyberTrace Web.

Downloading

Feed Utility downloads archives containing feeds from the Kaspersky update servers. Each downloaded archive contains one feed. Before downloading Kaspersky Threat Data Feeds, Feed Utility checks whether they are newer than those being used. Before downloading OSINT and third-party feeds, Feed Utility does not perform such checking.

Feed Utility uses a certificate for authentication. The certificate also defines which Kaspersky Threat Data Feeds can be downloaded by Feed Utility. For example, if you have a demo certificate, Feed Utility can download only demo feeds.

If you have trouble downloading feeds from third-party sources, check section "Feed Utility troubleshooting", subsection "An SSL error occurred while downloading a third-party feed".

If Kaspersky releases a new data feed and this feed is available with your certificate, add a new Feed element in the Feed Utility configuration file:

After you add a new Feed element, Feed Utility can process the new feed.

To get the list of data feeds available with your certificate, specify the -l / --list option when running Feed Utility on the command line.

Downloading differential feeds

Feed Utility supports the downloading of differential updates for specific Kaspersky Threat Data Feeds. Such feeds are called differential feeds in this document. Differential feeds are similar to Kaspersky Threat Data Feeds updated under the regular scenario, but have different feed IDs. See also the list of available differential feeds.

To download a differential feed:

Specify the ID of the differential feed in the Feed Utility configuration file.

For differential feeds, there are snapshots and differential parts available on the update servers. A snapshot is a full version of the feed generated daily. A differential part of the feed contains changes that must be applied to the feed to make it up-to-date.

Feed Utility updates differential feeds as follows:

1. The snapshot is downloaded in the following cases:
   • The feed is being downloaded for the first time.
   • The feed was updated a long time ago and it is not feasible to download differential parts.

     The feasibility of downloading differential parts is determined by the update server.

2. One or more archives with differential parts are downloaded if they are newer than the snapshot downloaded in step 1 or the feed file is currently in use.

   When an archive with a differential part is downloaded, it is renamed to %FILE_NAME_SRC%_%TIMESTAMP%.zip, where %FILE_NAME_SRC% is the initial file name (without an extension) and %TIMESTAMP% is the timestamp of the differential part publication in the yyyy-mm-dd HH:MM:SS format.

3. If Feed Utility is running in processing mode (with the -p option), the previously downloaded and unpacked differential parts are applied to the snapshot located in the WorkDir directory (learn more about the WorkDir parameter in the Feed Utility configuration file).

   When an archive with a differential part is unpacked, it is renamed to %FILE_NAME_SRC%_%TIMESTAMP%.json, where %FILE_NAME_SRC% is the initial file name (without an extension) and %TIMESTAMP% is the timestamp of the differential part publication in the yyyy-mm-dd HH:MM:SS format.

By default, Feed Utility downloads differential parts of a feed in parallel. To enable sequential downloading, set the SequentialDownload parameter of the Feed Utility configuration file to True.

Differential versions of Kaspersky Threat Data Feeds (if they exist) can be downloaded by using the same certificate as for the regular versions.

Processing and filtering

After the archives containing feeds are downloaded, Feed Utility unpacks the archives and processes the original feed files. The feed files are modified according to a combination of feed rules, filtering rules, and other parameters specified in the Feed Utility configuration file. These parameters define the data that will be included in the resulting feeds, the output format of the resulting feeds, and the maximum number of records in the resulting feed.

Filtering is the process of modifying the original feed files according to specified filtering criteria. Filtering criteria are defined in the filtering rules for each feed. Depending on the intended Feed Utility usage scenario, you may want to create a feed that uses only a subset of information contained in the original feed. This can be achieved by using a combination of feed rules and filtering rules.

Default filtering rules

The default Feed Utility configuration file that is shipped in the Kaspersky CyberTrace distribution kit contains the following filtering rule for IP Reputation Data Feed and Demo IP Reputation Data Feed:

Only feed records whose threat_score parameter is not less than 75 are downloaded.

Kaspersky considers malicious those IP addresses whose threat_score is not less than 75. IP addresses whose threat_score is less than 75 are considered related to spam and posing no significant threat.

If you reduce the boundary value or remove the filter, you will have many detections alerts by Demo IP Reputation Data Feed and IP Reputation Data Feed. These alerts will be notifications about detecting spam IP addresses.

To remove the filter:

1. Open the Settings > Feeds page of Kaspersky CyberTrace Web.
2. For Demo IP Reputation Data Feed (or IP Reputation Data Feed) remove the filtering rule for the threat_score field.

Compiling

If you use Feed Utility with Kaspersky CyberTrace Service, feeds that contain URL masks must be converted to binary format. Feed Utility compiles the URL masks extracted from these feeds and creates binary files which are then used by Kaspersky CyberTrace Service to quickly match URLs from received events to URL masks. Compiling is performed automatically by Feed Utility, if the UrlMatcherField option is specified in the feed rules.

Reloading

When notified, Kaspersky CyberTrace Service reloads the feeds for use, that is, it unloads the old feeds from memory and loads the new ones.

Updating feeds