Feed Utility configuration parameters

Feed Utility reads the configuration parameters, feed rules, filtering rules, and parsing rules for feeds from the configuration file. This file is in XML format and has several groups of parameters.

The paths in the configuration file must contain only the characters used in the operating system locale, otherwise Feed Utility will not work.

Feed (feed rules, filtering rules, and parsing rules)

The Feed parameter contains rules for a particular feed. This element has several types of nested parameters:

• Feed rules specify how this particular feed must be processed by Feed Utility.
• Filtering rules are criteria that Feed Utility uses to filter the original feed files. Filtering rules are a part of feed rules for each feed.
• Parsing rules are rules for custom feeds (OSINT feeds, and other feeds that are not Kaspersky Threat Data Feeds). These parameters specify how each feed must be parsed by Feed Utility.

This parameter has the following attributes:

• enabled

  Specifies whether Feed Utility must download and process this feed.

  If enabled is true, Feed Utility downloads and processes the feed. If enabled is false, Feed Utility skips this feed.

The following example demonstrates how feed rules, filtering rules, and parsing rules are nested in the configuration file.

FeedsDir

The FeedsDir parameter specifies the directory where Feed Utility puts processed feed files.

WorkDir

The WorkDir parameter specifies the directory where Feed Utility puts the downloaded and unpacked feed files.

If this parameter is not specified, Feed Utility uses the default temporary directory of the operating system.

WorkDir cannot be equal to FeedsDir.

CertFile

The CertFile parameter specifies the path to the certificate file. This certificate is used by Feed Utility to download feeds.

The certificate file must be in the PEM format.

SourceIPs

The SourceIPs parameter specifies the IP addresses that are used by Feed Utility to download feeds.

This parameter is optional. If it is omitted or has an empty value, Feed Utility resolves Kaspersky server addresses by their domain names.

You can specify one or more IPv4 addresses in this parameter. To specify several IP addresses, use the semicolon (";") as a delimiter.

The following example demonstrates specifying IP addresses in the SourceIPs parameter.

SourceDomains

The SourceDomains parameter specifies the domain names that are used by Feed Utility to download feeds.

You can specify one or more domain names in this parameter. To specify several domain names, use the semicolon (";") as a delimiter. Feed Utility will attempt to download feeds from the specified domain names in the order they appear in the configuration file.

When SourceDomains and SourceIPs parameters are used together, domains specified in the SourceDomains parameter are used before IP addresses specified in the SourceIPs parameter. If all attempts to download feeds fail, Feed Utility will generate an error message.

You can use Unicode symbols in this parameter.

The following example demonstrates specifying IP addresses in the SourceDomains parameter.

CreateExternalFeedInfoList path="PATH"

This parameter is obsolete. It is ignored in the current version of Kaspersky CyberTrace.

The CreateExternalFeedInfoList parameter specifies whether a list of supported OSINT feeds must be generated. This parameter is mandatory.

If this parameter is 1, Feed Utility creates a list of supported OSINT feeds, osint_feed_list.conf, in a directory specified in the path attribute. If you added any custom or third-party feeds to Kaspersky CyberTrace, Feed Utility also creates a list of these feeds, custom_feed_list.conf, in the same directory as osint_feed_list.conf.

If this parameter is 0, Feed Utility does not create a list of supported OSINT feeds.

The following example demonstrates specifying a path where the list must be created. In this example, the list will be created in a directory where Feed Utility binary is located.

NotifyKTFS path="PATH"

The NotifyKTFS parameter specifies whether Kaspersky CyberTrace Service must be notified about the feed updates.

This parameter can be used only with json output format.

If this parameter is 1, Feed Utility notifies Kaspersky CyberTrace Service that the feeds must be reloaded. A path to the Kaspersky CyberTrace Service binary file must be specified in the path attribute of this parameter.

If this parameter is 0, Feed Utility does not notify Kaspersky CyberTrace Service.

EULA

The EULA parameter specifies whether the terms of the End User License Agreement (EULA) were accepted by a user.

If this value is accepted, the terms of the EULA were accepted.

If this value is rejected, the terms of the EULA were not accepted. In this case, Feed Utility cannot be used.

RetryCount

The RetryCount parameter specifies the number of attempts to download a Kaspersky Threat Data Feed. Feed Utility tries to re-download a feed when a connection timeout, partial downloading, and other errors occur.

If the specified number of attempts were unsuccessful, Feed Utility displays an error message and continues its operation.

This parameter is used only for Kaspersky Threat Data Feeds. OSINT feeds and other custom feeds will not be re-downloaded by Feed Utility.

This parameter is optional. If this parameter is not specified, Feed Utility uses the default value of 10.

If this parameter is 0, the number of attempts is not limited.

SequentialDownload

The SequentialDownload parameter specifies whether Feed Utility must download feeds in sequential or parallel mode.

If this value is 1 or true, Feed Utility downloads feeds in sequential mode, one by one.

If this value is 0 or false, Feed Utility downloads feeds in parallel mode, all feeds at the same time.

By default, this parameter has the value of 0.

OutputFormat

The OutputFormat parameter defines the output format for all feeds. This parameter can have the following values:

• json

  The feeds are in JSON format. The feed files have a .json extension.

  This is the default value. If the OutputFormat parameter is omitted, this value is used to define the output format.

• txt

  The feeds are in plain text format (UTF-8 with BOM). The feed files have a .txt extension.

  • delimiter attribute

    In this format, record fields are separated with a delimiter. The default delimiter is ";". To specify a custom delimiter, use the delimiter attribute as follows:

    <OutputFormat delimiter="%delimiter%">txt</OutputFormat>

    Here, substitute %delimiter% with a symbol that must be used as a delimiter.

  • indicatorPerLine attribute

    To output one record field per line, set the indicatorPerLine attribute to 1 as follows:

    <OutputFormat indicatorPerLine="1">txt</OutputFormat>

    If you use this attribute, subfields specified in the RequiredFields feed rule must have the same parent field. For example, "files/MD5;files/SHA1" is valid, while "files/MD5;whois/domain" is invalid and will result in an error.

  If this output format is specified, all feed rules in the configuration file must include a RequiredFields parameter. The RequiredFields parameter specifies the order in which the fields are written to the output feed.

• csv

  Same as txt. The feed files have a .csv extension.

  You can use delimiter and indicatorPerLine attributes.

• openioc

  The feeds are in OpenIOC format. The feed files have an .ioc extension.

  You can specify the version of the OpenIOC format in the version attribute: it can be either 1.0, or 1.1. If the attribute is omitted, version 1.1 is used.

  Converting feeds to OpenIOC 1.0 format has some restrictions. Phishing URL Data Feed and Malicious URL Data Feed cannot be converted to OpenIOC 1.0 format; an error message is printed instead. For other feeds, only hash and IP address fields are converted. Converting feeds to OpenIOC 1.1 format has no such restrictions.

  It is not recommended to use the RequiredFields element. In this case, the feed will be missing the fields that are mandatory for converting to OpenIOC format.

  It is not recommended to use the RecordsCount parameter, since it is not intended for this format. The results of using this parameter may be unpredictable.

  Feeds in OpenIOC format take significantly more hard drive space than the original feed files.

• stix

  The feeds will be in STIX™ format. The files will have an .xml extension.

  For STIX format, feeds with URL masks must have the type field.

  You can specify the version of the STIX format in the version attribute: it can be 1, 2.0 or 2.1. If value 1 is specified, the feed will be in STIX 1.1 format. If the attribute is omitted, value 1 is used.

  It is not recommended to use the RequiredFields element. In this case, the feed will be missing the fields that are mandatory for converting to STIX format.

  It is not recommended to use the RecordsCount parameter, since it is not intended for this format. The results of using this parameter may be unpredictable.

  Feeds in STIX format take significantly more hard drive space than the original feed files.

The following example demonstrates how the OutputFormat parameter is nested in the configuration file.

CreateDiff

The CreateDiff parameter specifies whether Feed Utility must create feed diffs. Feed diffs are files that contain differences between the old and new version of a processed feed file. This parameter affects all feeds created by Feed Utility as follows:

• If this parameter is 0, Feed Utility does not create feed diffs. This is the default value.
• If this parameter is 1, Feed Utility creates feed diffs.

If CreateDiff is 1, and new versions of feeds are downloaded, two additional files are created for each feed (%feed_name% is the name of the feed file):

• The %feed_name%_new.json file contains records that were added to the new version of the feed file.
• The %feed_name%_del.json file contains records that were deleted in the new version of the feed file.

Feed diffs can be created only for feeds in JSON format that are contained in a single file:

• The OutputFormat parameter must have the json value.
• For each feed, the UrlMatcherField parameter must be omitted or have an empty value.
• For each feed, the RecordsCount parameter must not have the perFile attribute, or this attribute must have a value of 0.

To create feed diffs, Feed Utility uses a key field in the old and new version of the feed:

• If this feed contains non-nested id, MD5, ip, url, or domain field, this field is used as a key field.
• If none of the fields above are present, Feed Utility attempts to search for a field with unique values across the feed. If this attempt is not successful, a warning is generated.

The following example demonstrates how the OutputFormat parameter is nested in the configuration file.

ProxySettings

The ProxySettings parameter specifies proxy settings for Feed Utility. If you specify a proxy server, Feed Utility will download feeds using the specified parameters.

The user name and password for the proxy are stored in the Feed Utility configuration file. This information is not provided to Kaspersky.

Proxy settings are specified in the following parameters:

• Host

  Host of the proxy server.

  You can specify a domain name or an IP address in this parameter. Both IPv4 and IPv6 addresses are supported.

• Port

  Port of the proxy server.

• User

  Encrypted user name for proxy server authentication.

  If a proxy server does not require authentication, leave this parameter empty.

  This parameter is stored encrypted. Use the --set-proxy command-line option to set this parameter. If you do not use this option and enter your user name as plain text, connection to the proxy server will not be established.

• Password

  Encrypted password for proxy server authentication.

  If a proxy server does not require authentication, leave this parameter empty.

  This parameter is stored encrypted. Use the --set-proxy command-line option to set this parameter. If you do not use this option and enter your password as plain text, connection to the proxy server will not be established.

The following example demonstrates how proxy settings are nested in the configuration file.

LogSettings

The LogSettings parameter defines how Feed Utility logs its activity.

If you enable logging, Feed Utility can write to the log files any of the following information that can be considered private, security-related, or sensitive: Feed Utility configuration parameters, proxy host and port, and operations performed while downloading and processing feeds.

If logging is enabled, Feed Utility writes to log files the information about free hard drive space that available for the work and feed directories. Also, starting from this version, an average speed that the feeds have while loading will be written to logs.

Log files are regular text files. All information written to the log files is not encrypted. The log files have standard inherited access rights. We recommend that you assign the directory for storing log files the appropriate rights so that only the administrator can read the log files.

Log files are stored until they are explicitly deleted by a user.

Feed Utility does not send log files or any data contained in them to Kaspersky. For technical support purposes, your Technical Account Manager (TAM) can ask you to provide log files.

Logging settings are specified in the following parameters:

• EnableLog

  Enables logging.

  If this value is 1 or true, Feed Utility logs its activity.

  If this value is 0 or false, Feed Utility does not log its activity.

• LogsDir

  Directory where Feed Utility stores its log files.

• CleanOldLog

  Enables removal of old log files.

  If this value is 0, upon initialization, Feed Utility keeps old log files.

  If this value is 1, upon initialization, Feed Utility deletes old log files.

The following example demonstrates how logging settings are nested in the configuration file.