Feed rules

Individual feed rules in the Feed elements specify how each feed must be processed by Feed Utility.

By default, the configuration file has entries containing feed rules for all feeds. Entries for commercial feeds are commented. If you use a commercial certificate to download feeds, uncomment the entries for feeds that are available with your certificate.

Example of feed rules

The following is an example of feed rules. Feed rules are specified individually for each feed.

<Settings>

...

<Feeds>

<Feed enabled="true">

<Name>Botnet_CnC_URL_Data_Feed</Name>

<FeedID>65</FeedID>

<Filters>

<Field name="geo" value="RU"/>

</Filters>

<UrlMatcherField toRegex="false">mask</UrlMatcherField>

<RequiredFields>mask;geo;first_seen;last_seen</RequiredFields>

<RecordsCount perFile="100" total="1000" />

<FeedFields/>

</Feed>

<Feed enabled="false">

<Name>CustomFeed</Name>

<Path>./custom_example/example_feed.json</Path>

<Parsing type="json">

<MD5 type="MD5">files/MD5</MD5>

</Parsing>

<FeedFields/>

</Feed>

...

</Feeds>

...

</Settings>

Feed

This parent element contains feed rules for a feed. For more information about this parameter, see section "Configuration file parameters".

Name

This element specifies the name of the downloaded feed file.

After Feed Utility unpacks downloaded feeds, it searches for a file with a name that begins with a specified string. If several file names begin with the specified string, Feed Utility prints an error message and stops processing feeds. In this case, you must manually resolve this conflict, for example, by deleting the extra files from the directory where Feed Utility unpacks them. This directory is specified by the WorkDir parameter.

This parameter is case-insensitive.

This parameter must have a unique value. No two Feed elements can have the same value in this parameter.

FeedID

This element specifies the identifier of a feed. Feed Utility uses this parameter to download feeds from the update servers.

The enabled attribute specifies whether the feed must be processed by Feed Utility:

Path

The Path parameter specifies the path and the file name for a custom or third-party feed.

Following value types are supported:

This parameter has the following attributes:

Authentication type is provided in the Settings > Feeds > Feed parameter of the Feed Utility configuration file.

Authorization

Basic authentication settings for custom or third-party feeds.

This parameter has two nested elements:

If this type of authentication is not required, do not specify this parameter.

TAXII

The TAXII parameter specifies the location of a STIX feed. This element must contain the address of a Poll service of a TAXII server.

This parameter has the following attributes:

The following example demonstrates specifying the location of a STIX feed:

<TAXII collection_name="example-collection">http://192.0.2.10:9000</TAXII>

Authorization

Credentials for TAXII server authorization.

This parameter has two nested elements:

The following example demonstrates usage of this parameter:

<Feed>

<Name>TAXII</Name>

<TAXII collection_name="example-collection">http://192.0.2.10:9000</TAXII>

<Authorization>

<User>zQYq33rAY7dgImLtk8W0jQ==</User>

<Password>OUYWpkPDoH+vv/IFfCrshw==</Password>

</Authorization>

</Feed>

Filters

This element specifies filtering rules for the feed. Each filtering rule is defined in a Field element. For more information, see section "Filtering rules".

The Filters element is optional. If a Filters element contains no nested Field elements, Feed Utility treats this situation as if the Filters element is omitted. If there is no Filters element, no filtering is performed.

UrlMatcherField

This element defines how feeds with URL masks are processed by Feed Utility.

If you use Feed Utility as a part of CyberTrace together with Feed Service, feeds that contain URL masks must be converted to binary format.

If you use Feed Utility without Feed Service, you do not need to compile masks, so the UrlMatcherField element is not required in the FeedUtility configuration file.

This element has a value and an optional toRegex attribute:

For json output format, you can do the following:

For csv and txt output formats, you can do the following:

For openioc and stix output formats, omit this element. You cannot use this element with these output formats.

RequiredFields

This element specifies fields that are included in the processed feed. This element is mandatory for txt and csv output formats. If the RequiredFields element is omitted, all fields of a record are written to the processed feed.

Field names are separated by a semi-colon (";"). The slash ("/") in a field name indicates a nested field (in terms of JSON format).

This element defines fields in the resulting feed; it does not work like a filtering rule. For example, if a <RequiredFields>id;mask</RequiredFields> feed rule is defined for a feed, the records in the processed feed will have only id and mask fields. Records that have at least one of the specified fields (id or mask) will also be included. Records that do not have at least one of the specified fields will be excluded because the absence of specified fields results in an empty record written to the processed feed. If you want to filter a feed so that only records with all the specified fields are included in resulting feed, you must use filtering rules. For information about using the RequiredFields element together with filtering criteria, see subsection "Excluding records with missing fields" in the "Filtering rules" section.

Record fields are written to csv and txt formats in the order they are listed in the RequiredFields element. Record fields are written in json format in the order in which they appear in the source feed. For openioc and stix formats, the order of records is not defined; records are written in the order of processing.

FeedFields

This element lists all fields present in a feed.

Do not change this parameter. Feed Utility automatically writes field values to it.

RecordsCount

This element specifies the maximum number of records that will be included in the processed feed.

This element has two attributes:

Parsing

This element contains parsing rules for custom feeds. For more information, see section "Parsing rules".

Page top