Feed rules

Individual feed rules in the Feed elements specify how each feed must be processed by Feed Utility.

By default, the configuration file has entries containing feed rules for all feeds. Entries for commercial feeds are disabled. If you use a commercial certificate to download feeds, uncomment the entries for feeds that are available with your certificate.

Example of feed rules

The following is an example of feed rules. Feed rules are specified individually for each feed.

<Settings> ... <Feeds> <Feed enabled="true"> <Name>Botnet_CnC_URL_Data_Feed</Name> <FeedID>65</FeedID> <Filters> <Field name="geo" value="RU"/> </Filters> <UrlMatcherField toRegex="false">mask</UrlMatcherField> <RequiredFields>mask;geo;first_seen;last_seen</RequiredFields> <RecordsCount perFile="100" total="1000" /> <FeedFields/> </Feed> <Feed enabled="false"> <Name>CustomFeed</Name> <Path>./custom_example/example_feed.json</Path> <Parsing type="json"> <MD5 type="MD5">files/MD5</MD5> </Parsing> <FeedFields/> </Feed> ... </Feeds> ... </Settings>

Feed

This parent element contains feed rules for a feed.

Name

This element specifies the name of the downloaded feed file.

After Feed Utility unpacks downloaded feeds, it searches for a file with a name that begins with a specified string. If several file names begin with the specified string, Feed Utility prints an error message and stops processing feeds. In this case, you must manually resolve this conflict, for example, by deleting the extra files from the directory where Feed Utility unpacks them. This directory is specified by the WorkDir parameter.

This parameter is case-insensitive.

This parameter must have a unique value. No two Feed elements can have the same value in this parameter.

FeedID

This element applies only to Kaspersky feeds.

The element specifies the identifier of a feed. Feed Utility uses this parameter to download feeds from the update servers.

The enabled attribute specifies whether the feed must be processed by Feed Utility:

• If this attribute is true, Feed Utility processes this feed.
• If this attribute is false, Feed Utility skips this feed.

Path

The Path parameter specifies the path and the file name for a custom or third-party feed.

The following value types are supported:

• Path and filename on the computer where the Feed Utility binary is located.
• FTP or FTPS path and file name.
• HTTP or HTTPS URL.
• Network path (supported only in the Windows operating system).

This parameter has the following attributes:

• cert

  Specifies the path to a certificate that will be used to download this feed via HTTPS or FTPS. This attribute is optional. If it is not specified, Feed Utility will not use a certificate.

Authentication type is provided in the Settings > Feeds > Feed parameter of the Feed Utility configuration file.

For a feed of the email type, select a connection protocol by specifying POP3 or IMAP as a value for type attribute.

If a connection protocol is selected, specify the IPv4/Host name and the port of email server in the Path element.

Authorization

Basic authentication settings for custom or third-party feeds.

This parameter has two nested elements:

• User

  Encrypted user name for basic authentication on a server from which the feeds are downloaded.

  Use the --set-basic-auth command-line option to set this parameter. For STIX feeds that are downloaded from the TAXII™ server, use the --set-taxii option.

  This element cannot be empty.

• Password

  Encrypted password for basic authentication on a server from which the feeds are downloaded.

  Use the --set-basic-auth command-line option to set this parameter. For STIX feeds that are downloaded from the TAXII server, use the --set-taxii option.

If this type of authentication is not required, do not specify this parameter.

TAXII

The TAXII parameter specifies the location of a STIX feed. This element must contain the address of a Poll service of a TAXII server.

This parameter has the following attributes:

• collection_name

  The name of the data collection for this STIX feed.

• version

  The version of the TAXII server. The possible values of this attribute are 1, 2.0, and 2.1. For a STIX 2.0 feed, specify a TAXII 2.0 server. For a STIX 2.1 feed, specify a TAXII 2.1 server.

  By default, the value of this attribute is 1.

The following example demonstrates specifying the location of a STIX feed:

<TAXII collection_name="example-collection" version="2.1">http://192.0.2.10:9000</TAXII>

TAXII server authorization settings

Credentials for TAXII server authorization.

This parameter has two nested elements:

• User

  User name for authorization on a TAXII server.

  This parameter is stored encrypted. Use the --set-taxii command-line option to set this parameter. If you do not use this option and enter your user name as plain text, connection to the TAXII server will not be established.

• Password

  Password for authorization on a TAXII server.

  This parameter is stored encrypted. Use the --set-taxii command-line option to set this parameter. If you do not use this option and enter your password as plain text, connection to the TAXII server will not be established.

The following example demonstrates usage of this parameter:

<Feed> <Name>TAXII</Name> <TAXII collection_name="example-collection" version="2.1">http://192.0.2.10:9000</TAXII> <Authorization> <User>zQYq33rAY7dgImLtk8W0jQ==</User> <Password>OUYWpkPDoH+vv/IFfCrshw==</Password> </Authorization> </Feed>

Filters

This element specifies filtering rules for the feed. Each filtering rule is defined in a Field element.

The Filters element is optional. If a Filters element contains no nested Field elements, Feed Utility treats this situation as if the Filters element is omitted. If there is no Filters element, no filtering is performed.

UrlMatcherField

This element defines how feeds with URL masks are processed by Feed Utility.

If you use Feed Utility as a part of Kaspersky CyberTrace together with Kaspersky CyberTrace Service, feeds that contain URL masks must be converted to binary format.

If you use Feed Utility without Kaspersky CyberTrace Service, you do not need to compile masks, so the UrlMatcherField element is not required in the Feed Utility configuration file.

This element has a value and an optional toRegex attribute:

• The value of this element specifies the name of the field in the feed that contains the URL mask.
• The optional toRegex attribute of this element defines how the utility must process the feed.
  • If the toRegex attribute is 0, false, or is omitted, Feed Utility compiles the feed and creates binary files that can be used by Kaspersky CyberTrace Service.
  • If the toRegex attribute is 1, or true, Feed Utility processes URL masks in the feeds so that they are converted to regular expressions.

    For each URL mask, Feed Utility creates a regular expression that matches all URLs covered by the URL mask. For example, if a URL mask is botnet.example.com, the regular expression for it is ((\/{2}|^)botnet\.example\.com(\/.*|$)). Like the URL mask, this regular expression matches all URLs in the botnet.example.com domain.

    The toRegex attribute can be set to 1 only for those feeds that have the type field. Also, if the RequiredFields element is present among the feed rules, it must contain the type field.

For json output format, you can do the following:

• Compile feeds with URL masks to a binary file that can be used by Kaspersky CyberTrace Service.

  If you use Kaspersky CyberTrace Service, the UrlMatcherField element is mandatory for feeds that contain URL masks: Malicious URL Feed, Phishing URL Feed, Botnet C&C URL Feed, and IoT URL Feed.

  <UrlMatcherField>%field_name%</UrlMatcherField>

• Convert URL masks to regular expressions.

  <UrlMatcherField toRegex=1>%field_name%</UrlMatcherField>

• Do not convert URL masks.

  Omit the UrlMatcherField element.

For csv and txt output formats, you can do the following:

• Convert URL masks to regular expressions.

  <UrlMatcherField toRegex=1>%field_name%</UrlMatcherField>

• Do not convert URL masks.

  Omit the UrlMatcherField element.

For openioc and stix output formats, omit this element. You cannot use this element with these output formats.

RequiredFields

This element specifies fields that are included in the processed feed. This element is mandatory for txt and csv output formats. If the RequiredFields element is omitted, all fields of a record are written to the processed feed.

Field names are separated by a semi-colon (";"). The slash ("/") in a field name indicates a nested field (in terms of JSON format).

This element defines fields in the resulting feed; it does not work like a filtering rule. For example, if a <RequiredFields>id;mask</RequiredFields> feed rule is defined for a feed, the records in the processed feed will have only id and mask fields. Records that have at least one of the specified fields (id or mask) will also be included. Records that do not have at least one of the specified fields will be excluded because the absence of specified fields results in an empty record written to the processed feed. If you want to filter a feed so that only records with all the specified fields are included in resulting feed, you must use filtering rules. For information about using the RequiredFields element together with filtering criteria, see subsection "Excluding records with missing fields" in the "Filtering rules" section.

Record fields are written to csv and txt formats in the order they are listed in the RequiredFields element. Record fields are written in json format in the order in which they appear in the source feed. For openioc and stix formats, the order of records is not defined; records are written in the order of processing.

FeedFields

This element lists all fields present in a feed.

Do not change this parameter. Feed Utility automatically writes field values to it.

RecordsCount

This element specifies the maximum number of records that will be included in the processed feed.

This element has two attributes:

• The total attribute of this element defines the total number of first-level records that must be included in the processed files. Feed Utility includes only those records that match the filtering rules. If this value is 0, this number is not limited, and all records are included.

  Note that the total attribute refers to first-level records, so the processed feed can contain more nested records than the value of total. For example, the original feed contains the following data:

  [ { "time": "23.02.1992 15:43", "urls":[ { "url" : "http://url.biz/sample1" }, { "url" : "http://url.biz/sample2" } ] }, { "time": "24.02.1992 15:43", "urls":[ { "url" : "http://url.biz/sample3" }, { "url" : "http://url.biz/sample4" } ] } ]

  You can specify the following settings for this feed in the Feed Utility configuration file:

  <OutputFormat>csv</OutputFormat> <RequiredFields>urls/url</RequiredFields> <RecordsCount total="1" />

  The processed feed will contain two records because they are nested in one first-level record.

  http://url.biz/sample1 http://url.biz/sample2

• The perFile attribute of this element defines the maximum number of records that must be included in a single file. If the total number of records in one processed feed file is greater than this value, Feed Utility creates an additional feed file.

  For example, if total is 1000, perFile is 300, and Feed Utility finds 650 records that match filtering rules, then three files are created (two files with 300 records and one with 50 records).

  This attribute cannot be used in the following cases:

  • The perFile attribute cannot be used for feeds with URL masks that are compiled into binary format.
  • The perFile attribute cannot be used with custom feeds that have parsing rules with the type attribute set to url or domain types. Such feeds are also compiled into binary format by Feed Utility. To use the perFile attribute for these types of fields, you must change the field type to context.

Parsing

This element contains parsing rules for custom feeds.

Rules for feeds of email type

Feed Utility can download third-party feeds of email type from a mail server.

Configure mail server connection settings in the following elements of the kl_feed_util configuration file:

• MailboxConnection

  This element is obligatory, if POP3/IMAP are enabled in the Path element.

  This element has the following attributes:

  • use_tls

    Specify true or false values for use_tls attribute to enable or disable use of TLS/SSL for interacting with the mail server. The preset value is true.

  • receive_period

    The maximum value for this attribute is 360, the preset value is 7, the minimum possible value is 1.

    Feed Utility processes only emails received after the last email feed update.

    If an email feed had not been updated before, Feed Utility processes all the emails over the last number of days set in the receive_period attribute.

• MailboxConnection/User

  The value for this element is encoded. For connecting to mail server, Feed Utility encodes user name using the --set-mailbox parameter.

• MailboxConnection/Password

  The value for this element is encoded. For connecting to mail server, Feed Utility encodes password using the --set-mailbox parameter.

  For checking connection with a mail server, use --check-mailbox-connection attribute.

• MailboxConnection/Filters

  This section allows to specify one or more filtering rules for mail server messages. This section is optional. For more information, see section "Filtering rules".

• Parsing

  To set the parsing rules for a feed of email type, see section "Parsing rules".

The following example demonstrates mail server connection settings:

<Feed enabled="true"> <Name>Email_OSINT</Name> <Path type="pop3">myemailserv.com:3125</Path> <MailboxConnection use_tls="false" receive_period="7"> <User>==asd</User> <Password>==ewq</Password> <Filters> <Filter field="from" condition="not_match">mywife@ya.ru</Filter> <Filter field="subject" condition="contains">Best IoCs ever</Filter> </Filters> </MailboxConnection> <Parsing type="messageBody"> <ip type="ip">(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})</ip> </Parsing> <Parsing type="messageAttach"> <Attach type="csv"> <ip type="ip">1</ip> </Attach> <Attach type="pdf"> <url type="url">1</url> </Attach> </Parsing> <Excluded> <Item>VALUE1</Item> <Item>VALUE2</Item> </Excluded> </Feed>

Page top