Feed Service Guide

This section explains how Feed Service works, how to configure it, and how to manage it from the command line or by using a script.

About Feed Service

Feed Service is one of the components of Kaspersky CyberTrace. It runs as a service and works as follows:

1. Feed Service listens on a specified port for incoming events.
2. Feed Service matches indicators in received events against those in the feeds that are used.
3. Feed Service generates events in the specified format that contains detected indicators on the basis of the incoming events and the feed records involved in the detection process.
4. Feed Service sends the generated events to the specified address and port.

Managing Feed Service

This section describes how to manage Feed Service.

Managing Feed Service by using systemd (Linux)

You can manage Feed Service by using systemd.

You can use all standard systemd commands to manage Feed Service.

Checking the service status

To check the status of Feed Service, use the following command:

systemctl status cybertrace.service

Starting, stopping, restarting the service

To start Feed Service, use the following command:

systemctl start cybertrace.service

Feed Service begins to receive events only after it has loaded the feeds into memory and has sent the KL_ALERT_ServiceStarted event.

To stop Feed Service, use the following command:

systemctl stop cybertrace.service

To restart Feed Service, use the following command:

systemctl restart cybertrace.service

Adding and removing the service from the list of services loaded on boot

By default, Feed Service is loaded on boot. You can change this behavior with the following commands.

To add Feed Service to the list of services loaded on boot, use the following command:

systemctl enable cybertrace.service

To remove Feed Service from the list of services loaded on boot, use the following command:

systemctl disable cybertrace.service

Managing Feed Service using the script (Windows)

The %service_dir%\bin\kl_control.bat file is a script for managing Feed Service.

Command-line options

The script supports the following command-line options:

• start—Starts Feed Service and the watchdog Windows service.

  The script prints the information about whether this operation succeeded or failed.

  Feed Service begins to receive events only after it has loaded the feeds into memory and has sent the KL_ALERT_ServiceStarted event.

• stop—Stops Feed Service and the watchdog Windows service.

  The script prints the information about whether this operation succeeded or failed.

• restart—Restarts the services of Kaspersky CyberTrace: stops them and starts them again.

  The script prints the information about whether this operation succeeded or failed.

• status—Prints the information about the status of the services of Kaspersky CyberTrace.
• reloaddb—Instructs Feed Service to reload the feeds it is using.
• help—Prints the Help information: how to use the kl_control.bat script.

You can use only one option at a time.

The script can also be launched without parameters. In this case, it prints the list of the available options and their descriptions.

Example

The following command starts Feed Service:

Managing Feed Service from the command line (Linux)

You can run the Feed Service binary file from the command line.

We recommend that you use systemd to manage Feed Service. Running Feed Service from the command line is intended for troubleshooting purposes.

The binary file uses the flags described in the following table.

Flags for kl_feed_service

Example

The following command runs Feed Service in watchdog mode and specifies serv.conf as the configuration file.

Managing Feed Service from the command line (Windows)

You can run the Feed Service binary file from the command line.

We recommend managing Feed Service by using the script. Running Feed Service from the command line is intended for troubleshooting purposes.

The binary file uses the flags described in the following table.

Flags for kl_feed_service.exe

If no flag is specified, the kl_feed_service.exe program prints the list of available flags to the screen.

Feed Service uses the configuration file that resides in the same folder as the kl_feed_service.exe binary file.

Feed Service configuration reference

This section describes the configuration file of Feed Service.

We recommend that you configure Feed Service by using the Settings > Service tab. Use this section only for configuration that cannot be done from the web interface.

About the configuration file (Feed Service)

Feed Service reads settings from a configuration file.

Editing the configuration file

If the configuration file is absent or does not follow the rules specified in this section, Feed Service does not start and prints an error message.

To avoid potential issues, make a backup copy of the Feed Service configuration file before you make any changes in the file. If Kaspersky CyberTrace does not work properly after you have reconfigured Feed Service, replace the configuration file with its backup copy.

Applying changes made to the configuration file

You have to reload the configuration file or restart Feed Service after you edit the configuration file for the changes to take effect.

You can apply changes made to certain elements of the configuration file by reloading the configuration file but without restarting Feed Service.

These elements include:

• InputSettings > RegExps
• Feeds
• OutputSettings > RecordFieldContextFormat
• OutputSettings > AlertFormat
• OutputSettings > EventFormat

To reload the configuration file without restarting Feed Service,

Run the script for managing Feed Service with the reloadconfig option.

To apply changes made to other elements of the configuration file, you have to restart Feed Service.

To restart Feed Service,

Run the script for managing Feed Service with the restart option.

Configuration file location (Linux)

In Linux, the configuration file used by Feed Service can be specified in the following ways:

• When Feed Service is started from the command line, the path to a configuration file can be specified as an argument. By default, this path is ../etc/kl_feed_service.conf.
• When Feed Service is managed using the script, the path to the configuration file is specified in the script parameters. By default, this path is %service_dir%/etc/kl_feed_service.conf.

Configuration file location (Windows)

The configuration file used by Feed Service is named kl_feed_service.conf and resides in the same directory as the Feed Service binary file.

Case sensitivity

All the names used in the configuration file (for example, pattern names or feed fields) are case-sensitive.

Using special characters

To use special characters (for example, an ampersand or angle brackets) in regular expressions and other parameters, enclose the text of the elements in a CDATA section.

The following example uses braces around parameters:

EULA

Specifies whether the terms of the End User License Agreement (EULA) were accepted by a user.

Attributes

This element has no attributes.

Possible values

This element has the following possible values.

EULA element possible values

Example

The following is an example of a EULA element.

IndicatorDatabaseSettings

Defines the indicator database settings.

Path

IndicatorDatabaseSettings

Attributes

This element has no attributes.

Nested elements

This element is a container for the following nested elements:

• ConnectionString

  Specifies the IP address and port that will be used for accessing the indicator database.

  For more information about this element, see the "IndicatorDatabaseSettings > ConnectionString" section below.

IndicatorDatabaseSettings > ConnectionString

This element must contain a string with an IP address and a port. See the example below.

This element is mandatory.

Example

The following is an example of this element.

Domains

Contains configuration for tenants.

Path

Domains

Attributes

This element has no attributes.

Nested elements

This element is a container for the following nested element:

• Domain

Example

The following is an example of this element.

Domain

Defines configuration for a single feed in a tenant.

Path

Domains > Domain

Attributes

This element has the following attributes.

Domain element attributes

Nested elements

This element is a container for the following nested elements:

• Description

  Tenant description

  This element is optional.

• InputSettings

  Input connection settings specific for this tenant. See InputSettings.

• OutputSettings

  Output connection settings specific for this tenant. See OutputSettings.

  This section is for a specific tenant. It cannot have AlertConnectionString and AlertFormat as nested elements.

• Feeds

  Feed settings for the feeds used in this tenant.

  For more information about this element, see the Domain > Feeds section below.

Domain > Feeds

Feed settings for the feeds used in the tenant.

This element is a container for the following nested element:

• Feed

  Settings for a specific feed in a tenant.

Example

The following is an example of this element.

Domain > Feeds > Feed

Defines configuration for a single tenant.

Path

Domains > Domain > Feeds > Feed

Attributes

This element has the following attributes.

Feed element attributes

Nested elements

This element is a container for the following nested element:

• ActionableFields

  Actionable fields for this feed in this tenant.

  The specified actionable fields replace actionable fields that are defined for this feed in the Feeds > Feed > ActionableFields element (if such fields are defined).

  This element is optional.

  For more information, see Feeds > Feed > ActionableFields.

Example

The following is an example of this element.

InputSettings

Contains input settings for the General tenant.

This element defines the IP address and port that Feed Service will listen on for incoming events, normalizing rules for processing the events, and regular expressions for parsing the events.

Path

InputSettings

Attributes

This element has no attributes.

Nested elements

This element is a container for the following nested elements:

• RegExps

  Contains Boost regular expressions that are used to parse incoming events originating from different sources.

• ConnectionString

  Specifies the IP address and port (or the Windows-named pipe) that the service will listen on for incoming events.

• EventDelimiter

  Specifies the rule for the splitting of incoming events,

Example

The following is an example of this element.

RegExps

Contains Boost regular expressions that are used to parse incoming events originating from different sources.

Path

InputSettings > RegExps

Attributes

This element has no attributes.

Nested elements

This element is a container for the following nested element:

• Source

  Contains parameters for a specific event source.

Example

The following is an example of this element.

Source

Contains parameters for a specific event source.

The regular expressions and event normalizing rules specified in the configuration file are grouped by event sources that are represented by Source elements. Usually these event sources are devices that issue events, which afterward are checked by Feed Service. Every Source element contains a set of rules. There can be one or more Source elements in the InputSettings > RegExps element.

Rules workflow

The way Feed Service chooses rules from different Source elements is described in the following flow chart.

Choosing a rule

Note that event normalizing rules are applied first and regular expressions are applied afterwards.

The regular expressions of the default event source for finding URLs, IP addresses, and hashes are universal; that is, they can be used for parsing events issued by most devices. They can be used for parsing events that contain multiple URLs, but cannot be used, for example, for parsing events that contain URLs with no protocol specified. The use of universal regular expressions lowers the performance of Feed Service, compared to using device-specific regular expressions. Also, the universal regular expressions do not handle the dispersal, in an event, of different parts of a URL (for example, the host and the path). The universal regular expressions for finding hashes can extract symbol sequences that actually are not hashes.

Special event sources

There must be an event source with the default identifier (<Source id="default">). Rules of the default event source have lower priority than rules specific to the event source. Rules specific to the event source are applied first. Rules of the default event source are applied next. If a rule specific to the event source and a rule of the default event source have the same name, the rule of the default event source is applied only if the rule specific to the event source had no matches.

There are two special event sources that you can use: http_single_lookup (<Source id="http_single_lookup">) and http_file_lookup (<Source id="http_file_lookup">).

The rules of the http_single_lookup event source are used when single values are searched for by using CyberTrace Web.

The rules of the http_file_lookup event source are used when hashes of specified files or indicators in log files are searched for by using CyberTrace Web. Therefore, if you have to search for values contained in a log file that has a special format, we recommend specifying rules for the http_file_lookup event source.

If the configuration file contains the http_single_lookup event source or the http_file_lookup event source, we strongly recommend that you do not remove the regular expressions specified in these special event sources by default and, instead, edit them as needed.

Path

InputSettings > RegExps > Source

Attributes

This element has the following attributes.

Source element attributes

Nested elements

This element is a container for the following nested elements:

• Regular expressions

  Regular expressions that are used to parse incoming events originating from this source.

  Each regular expression is a separate element with the name of the regular expression.

• NormalizingRules

  Rules for modifying incoming events.

Example

The following is an example of this element.

Source > Regular expression

Defines a regular expression for an event source.

Path

InputSettings > RegExps > Source > %RegexpName%

This element has the name of the regular expression.

Attributes

This element has the following attributes.

%RegexpName% element attributes

Value

This element contains a Boost regular expression.

For more information about specifying values for this parameter, see section "About regular expressions".

Example

The following is an example of this element.

Source > NormalizingRules

Defines rules for modifying incoming events.

For more information about normalizing rules, see Adding normalizing rules.

Path

InputSettings > RegExps > Source > NormalizingRules

Attributes

This element has no attributes.

Nested elements

This element is a container for the following nested elements:

• Replace

  These elements specify replacing rules.

• Ignore

  These elements specify ignoring rules.

NormalizingRules > Replace

Defines a replacing rule.

This element has the following attributes.

Replace element attributes

NormalizingRules > Ignore

Defines an ignoring rule

This element has the following attributes.

Ignore element attributes

Example

The following is an example of this element.

ConnectionString

Specifies the IP address and port (or the Windows-named pipe) that the service will listen on for incoming events.

This element is mandatory.

Path

InputSettings > ConnectionString

Attributes

This element has no attributes.

Value

The value is formatted as <ip_address>:<port> (if an IP address and port are used) or as \\.\pipe\<pipe_name> (if a Windows-named pipe is used).

The IP address must consist of four decimal octets, each separated by a dot. The value in each octet must be less than 256.

Example

The following is an example of this element.

EventDelimiter

Specifies the rule for the splitting of incoming events.

Path

InputSettings > EventDelimiter

Attributes

This element has no attributes.

Value

The value rule must have the following format:

<EventDelimiter>%START_EVENT_SYMBOLS%</EventDelimiter>

The %START_EVENT_SYMBOLS% value contains the regular expression that corresponds to the beginning of the substring of the incoming event.

The rule processes all occurrences of the %START_EVENT_SYMBOLS% value. If the %START_EVENT_SYMBOLS% value is found in the string of the incoming event, this string will be split into multiple events by the addition of the newline character (\n) before every matched substring.

If the %START_EVENT_SYMBOLS% value does not match any substring of the incoming string, the incoming string is considered a single event.

If an event contains newline characters (\n), such an event will be split by using both the %START_EVENT_SYMBOLS% value and newline characters as event delimiters.

Example

The following is an example of this element.

Feeds

Defines how events must be checked against feeds.

Path

Feeds

Attributes

This element has the following attributes:

Feeds element attributes

Nested elements

This element is a container for the following nested element:

• Feed

  Every Feed element describes a feed.

  A configuration file must contain at least one Feed element.

Example

The following is an example of this element.

Feed

Describes a feed or supplier.

Path

Feeds > Feed

Attributes

This element has the following attributes:

Feed element attributes

Nested elements

This element is a container for the following nested elements:

• Field

  This element is obsolete starting from Kaspersky CyberTrace version 4.0.

  A Field element specifies the rules for checking an event against the records of the feed.

  For more information about this element, see section "About feed matching rules".

• ActionableFields

  Defines actionable fields.

Example

The following is an example of this element.

Feed > ActionableFields

Defines the fields that must be inserted into the outgoing events apart from the context. An outgoing event contains context and actionable fields.

Path

Feeds > Feed > ActionableFields

Attributes

This element has no attributes.

Nested elements

This element is a container for the following nested element:

• ActionableField

  Defines a single actionable field.

ActionableFields > ActionableField

Defines a single actionable field.

This element has the following attributes:

ActionableField element attributes

Example

The following is an example of this element.

About feed matching rules

Starting from Kaspersky CyberTrace version 4.0, the Feeds > Feed > Field element is obsolete. Information in this section applies only to Kaspersky CyberTrace 3.1.0 and below.

Feed Service checks incoming events against the feeds specified in the Feeds > Feed elements. For each specified feed, matching rules are set with one or more Field elements. Each Field element describes how a particular field in the feed must match the data from incoming events.

The following is an example of a Field element:

The Field element must have the following attributes:

• name

  The name of the field in the feed must be specified in the name attribute. Note that field names are case-sensitive. For example, fields "md5" and "MD5" are different fields.

  To specify nested fields, use the '/' delimiter. For example, name="detail/info" specifies the info field in a feed that has the following content:[ { "hash":"234D123...", "detail": [ { "info" : "some value" } ] } ].

• matching_type

  The type of matching must be specified in the matching_type attribute.

  The following values are possible:

  • "Url"

    The event field will be checked for conformance to the URL masks stored in the feed. For more information about the masks stored in feeds, contact your technical account manager (TAM).

  • "Exact"

    Comparison of two strings will be performed: the event field and the field stored in the feed.

  • "Hash"

    The event field will be checked to determine whether it is equal to the one stored in the feed. This matching type is used only for hashes.

• input_regexp_to_match

  Name of a regular expression that is used for matching. The value of this attribute must be one of the regular expression names from the InputSettings > RegExps element.

• category

  The category that will be used in the outgoing event.

All the attributes of the Field element are required.

The following is an example of the feed matching rules for a specific feed:

The following is an example of the feed matching rules for a specific feed for ArcSight:

If you have events that contain the event source IP address, we recommend that you check them against IP Reputation Data Feed. This must be done because the event source may be a bot or a malicious device of some other kind that takes part in a DoS attack on the user resources. To check such events against IP Reputation Data Feed, add an SRC_IP regular expression to find the event source IP addresses. Also, add a rule for IP Reputation Data Feed to use the SRC_IP regular expression, so that the configuration file will contain the following records:

Also, add the reference to the value found by the SRC_IP regular expression (%SRC_IP%) to the OutputSettings > EventFormat element.

IoCExports

Contains indicators export settings.

Path

IoCExports

Attributes

This element has the following attributes.

IoCExports element attributes

Nested elements

The IoCExports element is a container for the following nested element:

• IoCExport

Example

The following is an example of this element.

IoCExport

Defines settings for a single report with exported data.

Path

IoCExports > IoCExport

Attributes

This element has the following attributes.

IoCExport element attributes

Nested elements

The IoCExports > IoCExport element is a container for the following nested element:

• Filter

  A configuration file must contain at least one Filter element.

Example

The following is an example of this element.

IoCExport > Filter

Defines the filtering rules for data to be exported.

Path

IoCExports > IoCExport > Filter

Attributes

This element has the following attributes.

Filter element attributes

Example

The following is an example of this element.

RetroScan

Contains retrospective scan settings.

Path

RetroScan

Attributes

This element has the following attributes.

RetroScan element attributes

SendEventFilters

Contains filtering rules for detection events from Kaspersky CyberTrace. You can specify several filtering rules at once.

Path

SendEventFilters

Attributes

This element has no attributes.

Nested elements

This element is a container for the following nested element:

• Filter

  A filtering rule.

SendEventFilters > Filter

This element defines a filtering rule.

For more information about this element and possible values of its attributes, see section "Working with indicators".

This element has the following attributes:

ActionableField element attributes

Example

The following is an example of this element.

OutputSettings

Contains output settings for the General tenant.

Defines the address and port of the event target software to send the outgoing events to, and the format of the outgoing events.

Path

OutputSettings

Attributes

This element has no attributes.

Nested elements

This element is a container for the following nested elements:

• EventFormat

  Specifies the format of outgoing events.

  For more information about the values of this element, see About event formats and patterns.

  The EventFormat element is mandatory.

• RecordFieldContextFormat

  Specifies how context fields must be added to an event.

  For more information about the values of this element, see About event formats and patterns.

  The RecordFieldContextFormat element is mandatory.

• ActionableFieldContextFormat

  Specifies how actionable fields must be added to an event.

  For more information about the values of this element, see About event formats and patterns.

  The ActionableFieldContextFormat element is mandatory.

• AlertFormat

  Specifies the format for outgoing events that inform the event target software of the Feed Service state.

  For more information about the values of this element, see About event formats and patterns.

  The AlertFormat element is optional. If it is absent from the configuration file, no notification is made.

• ConnectionString

  Specifies the IP address and port (or the Windows-named pipe) to which the service will send outgoing events.

  The ConnectionString element is mandatory.

  For more information about this element, see the "OutputSettings > ConnectionString" section below.

• AlertConnectionString

  Specifies the IP address (or host) and port to which the service will send service alerts.

  The AlertConnectionString element is optional.

  For more information about this element, see the "OutputSettings > AlertConnectionString" section below.

• FinishedEventFormat

  Specifies the format of the informational event that is generated for each processed event.

  The FinishedEventFormat element is mandatory.

  For more information about this element, see the "OutputSettings > FinishedEventFormat" section below.

OutputSettings > ConnectionString

Specifies the IP address (or host) and port to which the service will send service alerts.

The string is formatted as <ip_address>:<port> (if an IP address and port are used) or as \\.\pipe\<pipe_name> (if a Windows-named pipe is used). The IP address must consist of four decimal octets, each separated by a dot. The value in each octet must be less than 256.

OutputSettings > AlertConnectionString

Specifies the IP address (or host) and port to which the service will send service alerts.

The value of this element is formatted as <ip_address>:<port> (if an IP address and port are used) or as \\.\pipe\<pipe_name> (if a Windows-named pipe is used). The IP address must consist of four decimal octets, each separated by a dot. The value in each octet must be less than 256.

The AlertConnectionString element is optional. If the element is omitted, the enabled attribute with the false value is used for this element.

This element has the following attributes:

AlertConnectionString element attributes

OutputSettings > FinishedEventFormat

Specifies the format of the informational event that is generated after an event is processed.

If this parameter is enabled, Kaspersky CyberTrace will generate an informational event for each event that it processes. An informational event is generated even if there were no detections.

The FinishedEventFormat element is mandatory.

The value of this element specifies the event format. You can use the %RecordContext% pattern and regular expression names in the format. For more information about patterns, see About event formats and patterns.

The %RecordContext% pattern will provide the following fields, if used:

• category

  It is "LookupFinished" for events of this type.

• sent_events

  The number of events sent to a SIEM solution.

• total

  Concatenation of the following substrings formed for every category assigned to detection events:

  <category>:<number_of_detections>;

  If there were no detections, the sent_events parameter is set to 0, and the total string is empty.

This element has the following attributes:

FinishedEventFormat element attributes

Example

The following is an example of this element.

ServiceSettings

Defines settings for the Feed Service process.

Path

ServiceSettings

Attributes

This element has no attributes.

Nested elements

This element is a container for the following nested elements:

• Bases

  Specifies the path to the directory that contains feeds from Kaspersky. If a relative path is set, it is calculated relative to the directory that contains the service binary file.

  The Bases element is mandatory.

• BasesBackup

  Specifies the path to the directory that contains backup version of feeds from Kaspersky. If a relative path is set, it is calculated relative to the directory that contains the service binary file.

  The BasesBackup element is mandatory.

• BasesDownload

  Specifies the path to the directory that contains downloaded feeds from Kaspersky. If a relative path is set, it is calculated relative to the directory that contains the service binary file.

  The BasesDownload element is mandatory.

• TemporaryDir

  The directory for temporary files.

  The TemporaryDir element is optional. If it is omitted, the default value is used.

  In Linux, the default value is /tmp.

  In Windows, the default value is %TEMP% (the current Windows user's temporary folder).

• OutdatedBasesAlertPeriod

  The time interval in hours following the last feed update, after which a notification about an outdated feed is sent to the event target. To turn off notifications, set this parameter to 0. This setting is taken into account for every feed that has no outdated_alert_period attribute.

  The OutdatedBasesAlertPeriod element is optional. If it is omitted, the default value 0 is used.

• ScannersCount

  The number of scanners. Every scanner handles a single TCP connection.

  If you want to run Feed Service in watchdog mode, specify one scanner in addition to the number of scanners needed for Feed Service itself. This must be done because the watchdog module uses an additional scanner.

  The ScannersCount element is optional. If it is omitted, the default value 9 is used.

• ScanningThreadsPerScanner

  The number of threads per scanner.

  The ScanningThreadsPerScanner element is optional. If it is omitted, the default value 8 is used.

• EventSendingRetriesCount

  Number of times Feed Service tries to resend a detection event to a SIEM solution if the first attempt at sending fails. If the value of EventSendingRetriesCount is 0, Feed Service sends each detection event one time and does not attempt to resend it.

  Maximum possible value is 10. The preset value is 3.

  The EventSendingRetriesCount element is mandatory.

• EventSendingRetriesTimеout

  Time interval between attempts made by Feed Service to resend a detection event to a SIEM solution, in seconds. Maximum possible value is 60.

  The EventSendingRetriesTimеout element is mandatory.

  The preset value is 10.

• FeedsRollbackEnabled

  Specifies if feeds rollback is enabled or disabled.

  If feeds rollback is enabled, feeds are rolled back when Kaspersky CyberTrace fails to upload new indicators into the Matching engine after feeds are updated. Kaspersky CyberTrace removes new indicators from the database and uses the previous feeds.

  Possible values:

  • true — feeds rollback is enabled.
  • false — feeds rollback is disabled.

  Kaspersky CyberTrace reads FeedsRollbackEnabled only during initialization and does not reread it after.

  By default, there is no FeedsRollbackEnabled element in the configuration file. If this element is missing, feeds rollback is enabled.

Example

The following is an example of this element.

GUISettings

Defines settings for the CyberTrace Web.

Path

GUISettings

Attributes

This element has no attributes.

Nested elements

This element is a container for the following nested elements:

• HTTPServer

  Contains the CyberTrace HTTP service parameters.

• FeedUtil

  Contains the Feed Utility parameters.

• AuthenticationServers

  Defines settings for connection to the LDAP servers.

Example

The following is an example of this element.

HTTPServer

Contains the CyberTrace HTTP service parameters.

Path

GUISettings > HTTPServer

Attributes

This element has no attributes.

Optional

The HTTPServer element is optional. If it is omitted, the default values are used.

Nested elements

This element is a container for the following nested elements:

• ConnectionString

  Specifies the IP address and port where the CyberTrace HTTP service is available.

  The ConnectionString element is optional. If it is omitted, the 127.0.0.1:443 IP address and port are used. After the installation process is complete, the default value of the HTTPServer > ConnectionString element is 0.0.0.0:443.

• SSLCertificatePath

  Path to the PEM-formatted certificate on a local computer for HTTPS connections. If a relative path is specified, it is calculated relative to the executable file.

  For security reasons, do not store your certificate in a shared folder accessible over a network and do not specify the path to a network shared folder containing your certificate.

  The SSLCertificatePath element is optional. If it is omitted, the ../httpsrv/kl_feed_service_cert.pem file is used.

• SSLPrivateKeyPath

  Path to the PEM-formatted private key on a local computer for HTTPS connections. If a relative path is specified, it is calculated relative to the executable file.

  For security reasons, do not store your private key in a shared folder accessible over a network and do not specify the path to a network shared folder containing your private key.

  The SSLPrivateKeyPath element is optional. If it is omitted, the ../httpsrv/kl_feed_service_private.pem file is used.

• TemplatesPath

  Path to the directory that contains layout pages for CyberTrace HTTP Service. If a relative path is specified, it is calculated relative to the executable file.

  The TemplatesPath element is optional. If it is omitted, the ../httpsrv/templates/ directory is used.

• ResourcesIP

  Contains the IP address or hostname that is used in the %IndicatorInfo% field of detection events, in the service events to notify that a retrospective scan completes, and in the result of exporting an indicator.

  The ResourcesIP element is mandatory. The default value is 127.0.0.1.

Example

The following is an example of this element.

FeedUtil

Contains the Feed Utility parameters.

Path

GUISettings > FeedUtil

Attributes

This element has no attributes.

Nested elements

This element is a container for the following nested element:

• ConfigurationPath

  Path to the Feed Utility configuration file.

  The ConfigurationPath element is mandatory.

Example

The following is an example of this element.

AuthenticationServers

Defines settings for connection to the LDAP servers.

Path

GUISettings > AuthenticationServers

Attributes

This element has no attributes.

Mandatory

This element is mandatory.

Nested elements

This element is a container for the following nested element:

• AuthenticationServer

  Contains the LDAP connection settings parameters.

Example

The following is an example of this element.

AuthenticationServers > AuthenticationServer

Contains the LDAP connection settings parameters.

Path

GUISettings > AuthenticationServers > AuthenticationServer

Optional

The AuthenticationServer element is optional.

Attributes

This element has the following attributes.

AuthenticationServer element attributes

Nested elements

This element is a container for the following nested elements:

• ConnectionString

  Contains the connection parameters for the LDAP server.

  For more information about this element, see subsection "AuthenticationServer > ConnectionString" below.

• DomainName

  The path to the database that contains the user accounts that can access Kaspersky CyberTrace.

  For more information about this element, see subsection "AuthenticationServer > DomainName" below.

• AdministratorAccountsFilter

  Contains filtering rules for user accounts that must be assigned the Administrator role.

  The AdministratorAccountsFilter element must not contain the value that is specified in the AnalystAccountsFilter element.

  The AdministratorAccountsFilter element can be empty.

• AnalystAccountsFilter

  Contains filtering rules for user accounts that must be assigned the Analyst role.

  The AnalystAccountsFilter element must not contain the value that is specified in the AdministratorAccountsFilter element.

  The AnalystAccountsFilter element can be empty. If the value is not specified, all user accounts that access Kaspersky CyberTrace will have the Analyst role.

  This element is mandatory.

AuthenticationServer > ConnectionString

IP address or FQDN (fully qualified domain name), and port of the LDAP server.

This element is mandatory and cannot be empty.

This element has the following attributes.

ConnectionString element attributes

AuthenticationServer > DomainName

The path to the database that contains the user accounts that can access Kaspersky CyberTrace.

This element is mandatory and cannot be empty.

This element has the following attributes.

DomainName element attributes

Example

The following is an example of this element.

Feed Service logging

This section describes how Feed Service logs its activity.

Enabling logging

By default, logging is disabled. To enable logging, use the kl_feed_service_log.conf file in the bin directory where the binary file of the service is located. Fill in the kl_feed_service_log.conf file as described in this section. This file is used by Feed Service and the watchdog module. A change in the contents of the kl_feed_service_log.conf file results in the new settings being applied; this process takes several seconds.

Enabling logging decreases the performance of Feed Service. Use logging only if you encounter problems and errors.

Logging and data security

If you enable logging, Feed Service can write to the log files any of the following information that can be considered private, security-related, or sensitive:

• Unmodified data (URLs, IP addresses, hashes, and other data) as it is received by Feed Service.
• The results of checking events against a feed.

Log files are regular text files. No information written to the log files is encrypted. The log files have standard inherited access rights. We recommend that you assign the directory for storing log files the appropriate rights so that only the administrator can read the log files.

Kaspersky CyberTrace does not send log files or any data contained in them to Kaspersky. For technical support purposes, your Technical Account Manager can ask you to provide log files.

Log files are stored until they are explicitly deleted by a user. If the Append parameter in the logging configuration file is 0, the previous log files are deleted when Feed Service is started. If the Append parameter in the logging configuration file is 1, the information is retained during the full cycle of Feed Service usage

If you uninstall Kaspersky CyberTrace, these log files will not be deleted if the directory with log files is located outside of the Feed Service installation directory (as specified by the LogsFolder parameter).

For more information about data written to the log files, see subsection "Log file contents" below.

Logging configuration file

The kl_feed_service_log.conf file is an XML file. Its fields are described in the table below.

Configuration file example

The following kl_feed_service_log.conf file example enables logging at the dbg logging level. Logs will be stored in the logs subdirectory of the directory where the Feed Service binary file resides.

Log files name format

Feed Service writes messages to files named "kl_feed_service-<pid>-<date_time>.log" or "kl_feed_service-<pid>-<date_time>_<index>.log".

The watchdog module writes messages to files named "kl_feed_servicewd-<pid>-<date_time>.log" or "kl_feed_servicewd-<pid>-<date_time>_<index>.log".

Log file contents

If the err logging level is used, Feed Service writes the following information to the log:

• Feed Service version and PID of the service process.
• The Feed Service configuration file parameters.
• Path to the directory with feeds.
• Errors occurred when normalizing a URL.
• Errors occurred while establishing a TCP connection.
• Watchdog messages:
  • Feed Service freezing or crashing.
  • Restarting Feed Service.
• Errors that occurred while changing the state of CyberTrace Web interface elements. Identifiers of user sessions where these errors occurred.
• Information about errors that occurred while handling CyberTrace Web HTTP requests:
  • Request method, GET or POST.
  • Request path for GET and POST requests.
  • GET request parameters. For POST requests, the request body is not logged.
  • Total elapsed time for handling the request.

If the inf logging level is used, Feed Service writes the following information to the log:

• Information to be written at the err logging level.
• Establishing or closing a TCP connection.
• Receiving and sending TCP requests and responses.
• Switching Feed Service to same-socket mode.
• Putting new databases into effect.
• Detecting an event that matches some record in a feed.
• Incoming event.

  Note that incoming events can contain private data, and so we recommend that you protect the log files from unauthorized access.

• Messages about navigating CyberTrace Web pages. Identifiers of user sessions for these messages.

If the dbg logging level is used, Feed Service writes the following information to the log:

• Information to be written at the inf logging level.
• Substrings in events information that regular expressions match.
• The result of checking an event against a feed.
• Sending test messages to and from the watchdog module.
• Messages about actions performed on CyberTrace Web pages. Identifiers of user sessions for these messages.
• Information about successful CyberTrace Web HTTP requests:
  • Request method, GET or POST.
  • Request path for GET and POST requests.
  • GET request parameters. For POST requests, the request body is not logged.
  • Total elapsed time for handling the request.

About resending detection events

By default, if an error occurs when Feed Service is sending a detection event to a SIEM solution, this event is sent again after a period of time specified in the configuration file.

You can configure how many times Feed Service attempts to send each detection event and the interval between attempts by using the EventSendingRetriesCount and EventSendingRetriesTimеout elements of the configuration file.

Feed Service in ReplyBack mode

In some cases, it is necessary for Feed Service to send responsive events back to the event source to the same socket from which the original events are received (ReplyBack mode).

To shift Feed Service to ReplyBack mode:

Send X-KF-ReplyBack as the first message after a TCP connection with Feed Service is established.

During the current TCP session every responsive event will be sent back to the same socket from which the original event was received. The setting specified in the OutputSettings > ConnectionString element of the configuration file will be ignored.

You can configure Feed Service to send an informational event indicating that event checking has finished.

To enable sending finishing events, perform one of the following actions:

• Set the enable attribute of the OutputSettings > FinishedEventFormat element of the Feed Service configuration file to true.
• Send X-KF-SendFinishedEventX-KF-ReplyBack as the first message after a TCP connection with Feed Service is established.

  In this case, the enable attribute of the OutputSettings > FinishedEventFormat element in the Feed Service configuration file will be ignored.

To instruct Feed Service that it must shift to ReplyBack mode and send finishing events:

Send X-KF-SendFinishedEventX-KF-ReplyBack as the first message after a TCP connection with Feed Service is established.

By default, in ReplyBack mode, Kaspersky CyberTrace does not save the statistics of the detection events received during the current connection.

To save the statistics of the detection events in ReplyBack mode:

• If you want Feed Service to send finishing events, send X-KF-SendFinishedEventX-KF-ReplyBackX-KF-SaveStatistic as the first message after a TCP connection with Feed Service is established.
• If you do not want Feed Service to send finishing events, send X-KF-ReplyBackX-KF-SaveStatistic as the first message after a TCP connection with Feed Service is established.

Features of event processing by Feed Service

This section outlines the way in which Feed Service processes incoming events.

Feed Service receives and sends events using TCP. Events are encoded in UTF-8.

Feed Service processes special characters in incoming events as follows:

• Feed Service treats the line feed character (0x0A) as the delimiter between events.

  Thus, text before a line feed character and text after it belong to different events.

• Feed Service ignores the carriage return character (0x0D).

Limitations on Feed Service incoming events

Feed Service processes events that are no larger than 64 kilobytes (KB). If an incoming event exceeds 64 KB, Feed Service splits it into several events and processes them separately. Thus, an indicator can be split in two and may not be extracted and checked.

We recommend that you configure the event source (the SIEM software), or the normalization rules, or both, so that the incoming events will not exceed 64 KB in size.