Feed rules

Individual feed rules in the Feed elements specify how each feed must be processed by Feed Utility.

By default, the configuration file has entries containing feed rules for all feeds. Entries for commercial feeds are commented. If you use a commercial certificate to download feeds, uncomment the entries for feeds that are available with your certificate.

Example of feed rules

The following is an example of feed rules. Feed rules are specified individually for each feed.

<Settings> ... <Feeds> <Feed enabled="true"> <Name>Botnet_CnC_URL_Data_Feed</Name> <FeedID>65</FeedID> <Filters> <Field name="geo" value="RU"/> </Filters> <UrlMatcherField toRegex="false">mask</UrlMatcherField> <RequiredFields>mask;geo;first_seen;last_seen</RequiredFields> <RecordsCount perFile="100" total="1000" /> <FeedFields/> </Feed> <Feed enabled="false"> <Name>CustomFeed</Name> <Path>./custom_example/example_feed.json</Path> <Parsing type="json"> <MD5 type="MD5">files/MD5</MD5> </Parsing> <FeedFields/> </Feed> ... </Feeds> ... </Settings>

Feed

This parent element contains feed rules for a feed. For more information about this parameter, see section "Configuration file parameters".

Name

This element specifies the name of the downloaded feed file.

After Feed Utility unpacks downloaded feeds, it searches for a file with a name that begins with a specified string. If several file names begin with the specified string, Feed Utility prints an error message and stops processing feeds. In this case, you must manually resolve this conflict, for example, by deleting the extra files from the directory where Feed Utility unpacks them. This directory is specified by the WorkDir parameter.

This parameter is case-insensitive.

This parameter must have a unique value. No two Feed elements can have the same value in this parameter.

FeedID

This element specifies the identifier of a feed. Feed Utility uses this parameter to download feeds from the update servers.

The enabled attribute specifies whether the feed must be processed by Feed Utility:

• If this attribute is true, Feed Utility processes this feed.
• If this attribute is false, Feed Utility skips this feed.

Path

The Path parameter specifies the path and the file name for a custom or third-party feed.

Following value types are supported:

• Path and filename on the computer where Feed Utility binary is located.
• FTP or FTPS path and file name.
• HTTP or HTTPS URL.
• Network path (supported only in the Windows operating system).

This parameter has the following attributes:

• cert

  Specifies the path to a certificate that will be used to download this feed via HTTPS or FTPS. This attribute is optional. If it is not specified, Feed Utility will not use a certificate.

Authentication type is provided in the Settings > Feeds > Feed parameter of the Feed Utility configuration file.

Authorization

Basic authentication settings for custom or third-party feeds.

This parameter has two nested elements:

• User

  Encrypted user name for basic authentication on a server from which the feeds are downloaded.

  Use the --set-basic-auth command-line option to set this parameter. For STIX feeds that are downloaded from the TAXII server, use the --set-taxii option.

  This element cannot be empty.

• Password

  Encrypted password for basic authentication on a server from which the feeds are downloaded.

  Use the --set-basic-auth command-line option to set this parameter. For STIX feeds that are downloaded from the TAXII server, use the --set-taxii option.

If this type of authentication is not required, do not specify this parameter.

TAXII

The TAXII parameter specifies the location of a STIX feed. This element must contain the address of a Poll service of a TAXII server.

This parameter has the following attributes:

• collection_name

  The name of the data collection for this STIX feed.

• cert

  Path to a certificate that must be used to download STIX feeds. This attribute must be used when downloading STIX feeds using the HTTPS or FTPS protocol.

The following example demonstrates specifying the location of a STIX feed:

<TAXII collection_name="example-collection">http://192.0.2.10:9000</TAXII>

The following example demonstrates usage of this parameter:

<Feed> <Name>TAXII</Name> <TAXII collection_name="example-collection">http://192.0.2.10:9000</TAXII> <Authorization> <User>zQYq33rAY7dgImLtk8W0jQ==</User> <Password>OUYWpkPDoH+vv/IFfCrshw==</Password> </Authorization> </Feed>

Filters

This element specifies filtering rules for the feed. Each filtering rule is defined in a Field element. For more information, see section "Filtering rules".

The Filters element is optional. If a Filters element contains no nested Field elements, Feed Utility treats this situation as if the Filters element is omitted. If there is no Filters element, no filtering is performed.

UrlMatcherField

This element defines how feeds with URL masks are processed by Feed Utility.

If you use Feed Utility as a part of CyberTrace together with Feed Service, feeds that contain URL masks must be converted to binary format.

If you use Feed Utility without Feed Service, you do not need to compile masks, so the UrlMatcherField element is not required in the FeedUtility configuration file.

This element has a value and an optional toRegex attribute:

• The value of this element specifies the name of the field in the feed that contains the URL mask.
• The optional toRegex attribute of this element defines how the utility must process the feed.
  • If the toRegex attribute is 0, false, or is omitted, Feed Utility compiles the feed and creates binary files that can be used by Feed Service.
  • If the toRegex attribute is 1, or true, Feed Utility processes URL masks in the feeds so that they are converted to regular expressions.

    For each URL mask, Feed Utility creates a regular expression that matches all URLs covered by the URL mask. For example, if a URL mask is botnet.example.com, the regular expression for it is ((\/{2}|^)botnet\.example\.com(\/.*|$)). Like the URL mask, this regular expression matches all URLs in the botnet.example.com domain.

    The toRegex attribute can be set to 1 only for those feeds that have the type field. Also, if the RequiredFields element is present among the feed rules, it must contain the type field.

For json output format, you can do the following:

• Compile feeds with URL masks to a binary file that can be used by Feed Service.

  If you use Feed Service, the UrlMatcherField element is mandatory for feeds that contain URL masks: Malicious URL Feed, Phishing URL Feed, Botnet CnC URL Feed, and IoT URL Feed.

  <UrlMatcherField>%field_name%</UrlMatcherField>

• Convert URL masks to regular expressions.

  <UrlMatcherField toRegex=1>%field_name%</UrlMatcherField>

• Do not convert URL masks.

  Omit the UrlMatcherField element.

For csv and txt output formats, you can do the following:

• Convert URL masks to regular expressions.

  <UrlMatcherField toRegex=1>%field_name%</UrlMatcherField>

• Do not convert URL masks.

  Omit the UrlMatcherField element.

For openioc and stix output formats, omit this element. You cannot use this element with these output formats.

RequiredFields

This element specifies fields that are included in the processed feed. This element is mandatory for txt and csv output formats. If the RequiredFields element is omitted, all fields of a record are written to the processed feed.

Field names are separated by a semi-colon (";"). The slash ("/") in a field name indicates a nested field (in terms of JSON format).

This element defines fields in the resulting feed; it does not work like a filtering rule. For example, if a <RequiredFields>id;mask</RequiredFields> feed rule is defined for a feed, the records in the processed feed will have only id and mask fields. Records that have at least one of the specified fields (id or mask) will also be included. Records that do not have at least one of the specified fields will be excluded because the absence of specified fields results in an empty record written to the processed feed. If you want to filter a feed so that only records with all the specified fields are included in resulting feed, you must use filtering rules. For information about using the RequiredFields element together with filtering criteria, see subsection "Excluding records with missing fields" in the "Filtering rules" section.

Record fields are written to csv and txt formats in the order they are listed in the RequiredFields element. Record fields are written in json format in the order in which they appear in the source feed. For openioc and stix formats, the order of records is not defined; records are written in the order of processing.

FeedFields

This element lists all fields present in a feed.

Do not change this parameter. Feed Utility automatically writes field values to it.

RecordsCount

This element specifies the maximum number of records that will be included in the processed feed.

This element has two attributes:

• The total attribute of this element defines the total number of first-level records that must be included in the processed files. Feed Utility includes only those records that match the filtering rules. If this value is 0, this number is not limited, and all records are included.

  Note that the total attribute refers to first-level records, so the processed feed can contain more nested records than the value of total. For example, the original feed contains the following data:

  [ { "time": "23.02.1992 15:43", "urls":[ { "url" : "http://url.biz/sample1" }, { "url" : "http://url.biz/sample2" } ] }, { "time": "24.02.1992 15:43", "urls":[ { "url" : "http://url.biz/sample3" }, { "url" : "http://url.biz/sample4" } ] } ]

  You can specify the following settings for this feed in the Feed Utility configuration file:

  <OutputFormat>csv</OutputFormat> <RequiredFields>urls/url</RequiredFields> <RecordsCount total="1" />

  The processed feed will contain two records because they are nested in one first-level record.

  http://url.biz/sample1 http://url.biz/sample2

• The perFile attribute of this element defines the maximum number of records that must be included in a single file. If the total number of records in one processed feed file is greater than this value, Feed Utility creates an additional feed file.

  For example, if total is 1000, perFile is 300, and Feed Utility finds 650 records that match filtering rules, then three files are created (two files with 300 records and one with 50 records).

  This attribute cannot be used in the following cases:

  • The perFile attribute cannot be used for feeds with URL masks that are compiled into binary format.
  • The perFile attribute cannot be used with custom feeds that have parsing rules with the type attribute set to url or domain types. Such feeds are also compiled into binary format by Feed Utility. To use the perFile attribute for these types of fields, you must change the field type to context.

Parsing

This element contains parsing rules for custom feeds. For more information, see section "Parsing rules".

Page top