Kaspersky CyberTrace

Contents

Log Scanner Guide

The Log Scanner utility is a command-line application that allows you to send data to Feed Service for checking against feeds. You can check hashes, IP addresses, or URLs one by one or in batch mode, that is, by sending Feed Service files that contain data to check.

This chapter explains how to use the Log Scanner utility.

In this section

Command-line options

Examples of usage scenarios

Configuration file (Log Scanner)

Recommendations on using Log Scanner
Page top

Command-line options

In Linux, the Log Scanner utility is launched from the command line as follows:

./log_scanner [-h|--help] [-r|--report] [-c|--config] [[-p|--path]|[-s|--hash]|[-u|--url]|[-i|--ip]] [value]

In Windows, the Log Scanner utility is launched from the command line as follows:

log_scanner.exe [-h|--help] [-r|--report] [-c|--config] [[-p|--path]|[-s|--hash]|[-u|--url]|[-i|--ip]] [value]

The following table explains the command-line options.

Command-line options of Log Scanner

Option

Description

-h

‑‑help

Prints the usage message to the screen.

If this option is specified, all other options are ignored.

-r

‑‑report

If this option is specified, Feed Service will return the response to Log Scanner in the same socket in which the request was sent, and Log Scanner will save the result in a text file. The output file is named log_scanner_report%current_time%.txt, where %current_time% is the date and time (including seconds) of creation of the output file. The location of the output file is set in the OutputDir element of the Log Scanner configuration file.

If a URL, IP address, or hash is found in Kaspersky Threat Data Feeds, its category and context information is written to the output. After the entire input is processed, the following information is written to the output:

  • The number of requests sent to Feed Service
  • The number of detections received from Feed Service
  • The time taken to perform all of the checking

    If this option is not specified, Feed Service will generate output according to the settings specified in its configuration file.

If this option is specified, make sure that the value of the enable attribute of the OutputSetting > FinishedEventFormat element in the Feed Service configuration file is not false.

-c

‑‑config

Path to the configuration file. It can be either an absolute or a relative path. A relative path is calculated relative to the directory from which you run Log Scanner.

By default, Log Scanner uses the log_scanner.conf configuration file that is placed in the directory from which you run Log Scanner.

-p

‑‑path

Path to a directory or text file that contains URLs, IP addresses, and hashes to check against Kaspersky Threat Data Feeds. It can be an absolute or a relative path. A relative path is calculated relative to the directory that contains the Log Scanner binary file. If the path to a directory is specified, all files contained in it and all its all-level subdirectories are processed.

Each line of each processed file is sent to Feed Service as the data to be checked. No further formatting is applied. Feed Service will parse the lines by using the regular expressions set in its configuration file.

You can specify several paths; in this case, use the -p option before every path. For example:

./log_scanner -p log1.txt -p log2.txt

log_scanner.exe -p log1.txt -p log2.txt

-s

‑‑hash

Hashes to be checked against Kaspersky Threat Data Feeds. They can be MD5 hashes, SHA1 hashes, or SHA256 hashes; Log Scanner determines the type of a hash on the basis of its length. If several hashes are specified, they must be separated by space symbols. For example:

./log_scanner -s A8315A5D4C8ACB982372C16B83BAEAAA -s A72C5B99F2706B00718279C9533A3648

log_scanner.exe -s A8315A5D4C8ACB982372C16B83BAEAAA -s A72C5B99F2706B00718279C9533A3648

-i

‑‑ip

IP addresses to be checked against Kaspersky Threat Data Feeds. If several IP addresses are specified, they must be separated by space symbols. For example:

./log_scanner -i 15.54.33.54 -i 45.62.66.69

log_scanner.exe -i 15.54.33.54 -i 45.62.66.69

-u

‑‑url

URLs to be checked against Kaspersky Threat Data Feeds. If several URLs are specified, they must be separated by space symbols. For example:

./log_scanner -u http://example.com/malware_test -u http://example.com/phishing_test

log_scanner.exe -u http://example.com/malware_test -u http://example.com/phishing_test

Do not use the -u option to check URLs that contain an ampersand (&). To check a URL that contains an ampersand, copy the URL to a text file and check the file by using the -p option, as described above.

If you specify none of the -p, -s, -u, or -i options, and specify only the value to check, this value will be treated as the path to the file or directory to be scanned.

The Log Scanner utility uses the current locale of the operating system.

Page top

Examples of usage scenarios

This section contains examples of using Log Scanner in some situations.

Checking several log files

All log files that you pass for scanning must be in UTF-8 encoding. If your log files have a different encoding, make sure to convert them to UTF-8.

If you have feeds that are not compiled and a directory containing log files, you can check the log files by performing the following procedure.

To check several log files:

  1. In the Feed Service configuration file kl_feed_service.conf specify the feeds to be used, normalizing rules to process events in the log files, and regular expressions to parse events.
  2. Start Feed Service:

    systemctl start cybertrace.service (in LInux)

    %service_dir%\bin\kl_control.bat start (in Windows)

  3. Run the Log Scanner utility and specify the directory that contains log files. For example:

    ./log_scanner -r –p ../logs (in Linux)

    log_scanner.exe -r –p ..\logs (in Windows)

  4. Stop Feed Service by running the following command:

    systemctl stop cybertrace.service

    %service_dir%\bin\kl_control.bat stop (in Windows)

After Log Scanner finishes its work, the directory specified by the OutputDir element of the log_scanner.conf configuration file will contain a report about the URLs and hashes detected by Feed Service.

Checking several URLs and hashes

If you have to check several URLs and hashes, perform the following procedure.

To check several URLs and hashes:

  1. Start Feed Service by running the following command:

    systemctl start cybertrace.service (in LInux)

    %service_dir%\bin\kl_control.bat start (in Windows)

  2. Run Log Scanner and specify the hashes to be checked. For example:

    ./log_scanner -r -s A72C5B99F2706B00718279C9533A3648 -s 6AA0321FA9D82D652AB53882D7CF9E592B4439B8 (in LInux)

    log_scanner.exe -r -s A72C5B99F2706B00718279C9533A3648 -s 6AA0321FA9D82D652AB53882D7CF9E592B4439B8 (in Windows)

  3. Run Log Scanner and specify the URLs to be checked. For example:

    ./log_scanner -r –u test.mav.example.com?bad_url=1 -u test.phishing.example.com/psh/test?p=1&p=2 (in LInux)

    log_scanner.exe -r –u test.mav.example.com?bad_url=1 -u test.phishing.example.com/psh/test?p=1&p=2 (in Windows)

  4. Stop Feed Service by running the following command:

    systemctl stop cybertrace.service (in LInux)

    %service_dir%\bin\kl_control.bat stop (in Windows)

After Log Scanner finishes its work, the directory specified by the OutputDir element of the log_scanner.conf configuration file will contain a report about the URLs detected by Feed Service and a report about the detected hashes.

Page top

Configuration file (Log Scanner)

The Log Scanner configuration file is an XML file that contains parameters described in the table below. If this file is not present in the directory or some parameters are not present in the file, the default values are used for the missing parameters.

Configuration file parameters

Parameter

Description

Verbose

Affects the Log Scanner output to the console. If the value contained in the Verbose element is "False" or "0", or the element is omitted, little information is printed to the console. Otherwise, detailed information is printed.

ThreadsCount

Maximum number of threads that Log Scanner can use when processing input data.

By default, up to 8 threads are used.

OutputDir

Directory that will contain the output file. It can be either an absolute or a relative path. A relative path is calculated relative to the directory that contains the Log Scanner binary file.

If the OutputDir parameter is not set, the output file is stored in the directory where the Log Scanner binary file resides.

Pattern

The utility sends requests to Feed Service in the format specified in the Pattern element. The following parameters can be used:

  • %IP%—The value to be checked if the utility is called with the -i (--ip) parameter.
  • %MD5%—The value to be checked if the utility is called with the -s (--hash) parameter and the value is an MD5 hash.
  • %SHA1%—The value to be checked if the utility is called with the -s (--hash) parameter and the value is an SHA1 hash.
  • %SHA256%—The value to be checked if the utility is called with the -s (--hash) parameter and the value is an SHA256 hash.
  • %URL%—The value to be checked if the utility is called with the -u (--url) parameter.

    By default, the following value is used:

    ip=%IP% md5=%MD5% sha1=%SHA1% sha256=%SHA256% url=%URL%

Connection

Specifies the IP address and port (or the Windows-named pipe, or UNIX socket) to which Log Scanner will send the received data.

  • If you use one of non-supported SIEM solutions, the Connection parameter should specify how to connect to that solution.
  • If you do not use a SIEM solution, the Connection parameter should specify how to connect to Feed Service.

    The value depends on the way in which Log Scanner interacts with a SIEM solution or Feed Service.

  • If they interact using TCP/IP, specify in the Connection element the IP address and port on which Feed Service receives events.
  • If they interact through a Windows-named pipe, specify in the Connection element the named pipe on which Feed Service receives events. The pipe name must be specified in format \\.\pipe\<pipe_name>.
  • If they interact through a UNIX socket, specify in the Connection element the socket on which Feed Service receives events.

    By default, the data is sent to 127.0.0.1:9999.

SocketTimeout

Number of seconds that Log Scanner waits for the socket or pipe specified in the Connection parameter to resume sending data.

If the value of this parameter is 0, Log Scanner waits indefinitely.

The maximum value of this parameter that you can set is 1000.

By default, the timeout is 15 seconds.

Configuration file example

<Settings>

<Verbose>0</Verbose>

<ThreadsCount>8</ThreadsCount>

<OutputDir>../log_scanner_reports</OutputDir>

<Pattern>ip=%IP% md5=%MD5% sha1=%SHA1% sha256=%SHA256% url=%URL%</Pattern>

<Connection>127.0.0.1:9999</Connection>

<SocketTimeout>15</SocketTimeout>

</Settings>

Page top

Recommendations on using Log Scanner

We recommend that you use Feed Service together with Log Scanner in the following cases:

  • You have to check some log files and save the check result to a file.

    It can be useful while investigating information security incidents, when the SIEM solution you use is unavailable, or if you do not use any SIEM solution.

  • You have to check some log files and send the check results to the SIEM solution used.

Configuring Feed Service and Log Scanner

Feed Service and Log Scanner must interact correctly, so their corresponding parameters must be set according to each other as follows:

  • The port set in the Settings > Connection element of the Log Scanner configuration file must accord with the port specified in the InputSettings > ConnectionString element of the Feed Service configuration file.
  • The number of threads specified in the Settings > ThreadsCount element of the Log Scanner configuration file must not be greater than that specified in the ServiceSettings > ScannersCount element of the Feed Service configuration file. If Feed Service runs in watchdog mode, the number of threads specified in the Settings > ThreadsCount element of the Log Scanner configuration file must be less than that specified in the ServiceSettings > ScannersCount element of the Feed Service configuration file.
  • The data sent by Log Scanner to Feed Service—either lines of log files or strings created on the basis of the Settings > Pattern element of the Log Scanner configuration file—must be parseable by regular expressions specified in the Configuration > InputSettings > RegExps element of the Feed Service configuration file.

Configuration files examples

The following is an excerpt from a sample Feed Service configuration file.

<Configuration>

<InputSettings>

<RegExps>

<Source id="default">

<!--You can use them in the OutputSettings->EventFormat string with the pattern %REGEXPNAME%-->

...

<RE_MD5>md5=(.*?)(?:$|\s)</RE_MD5>

<RE_SHA1>sha1=(.*?)(?:$|\s)</RE_SHA1>

<RE_SHA256>sha256=(.*?)(?:$|\s)</RE_SHA256>

<RE_URL>url=(.*?)(?:$|\s)</RE_URL>

<RE_IP>ip=(.*?)(?:$|\s)</RE_IP>

</Source>

</RegExps>

<ConnectionString>127.0.0.1:9999</ConnectionString> <!-- <ip>:<port>. Threat Feed Service listens for <ip>:<port>. <port> must be available -->

</InputSettings>

 

<Feeds per_scan_detect_limit="10000">...</Feeds>

 

<OutputSettings>

...

<FinishedEventFormat>LookupFinished</FinishedEventFormat>

</OutputSettings>

 

<ServiceSettings>

...

<ScannersCount>9</ScannersCount> <!-- 1 tcp connection = 1 scanner -->

</ServiceSettings>

</Configuration>

The following is an excerpt from a Log Scanner configuration file that corresponds to the Feed Service configuration file provided above.

<Settings>

...

<ThreadsCount>8</ThreadsCount>

<Pattern>ip=%IP% md5=%MD5% sha1=%SHA1% sha256=%SHA256% url=%URL%</Pattern>

<Connection>127.0.0.1:9999</Connection>

</Settings>

When using these configuration files, Log Scanner sends the requests to the IP address 127.0.0.1 and port 9999, and Feed Service listens on port 9999 for data to check. Both Log Scanner and Feed Service use up to 8 threads for transferring and processing data, and one thread is used by the watchdog module (in LInux) or watchdog service (in Windows). If correct URLs, IP addresses, and hashes are sent to Feed Service for checking, they will be successfully parsed by using the regular expressions specified in the Feed Service configuration file.

Managing check results

After data is checked by Feed Service, you can either send the check results to event target software or save them to a file:

  • For sending check results to event target software, set the correct value of the "OutputSettings > ConnectionString" element of the Feed Service configuration file.
  • For saving check results to a file, pass the -r option when running Log Scanner from the command line as follows:

    ./log_scanner -r -p file_to_check (in Linux)

    log_scanner.exe -r -p file_to_check (in Windows)

    The value of the enable attribute of the OutputSettings > FinishedEventFormat element in the Feed Service configuration file must not be false.

Report example

The report content depends on the value of the OutputSettings > EventFormat element of the Feed Service configuration file.

The following is an example of a report sent by Feed Service to Log Scanner.

- KL_Data_Feed_Service_v1 LEEF:1.0|Kaspersky Lab|SIEM Service|1.0|KL_Malicious_URL|url=malicious_domain_21.com/folder/load.php?| IP=91.202.63.117, 196.254.10.200, 194.190.253.19, 185.56.137.11, 178.62.5.157, 173.194.222.211, 159.253.145.183, 87.250.250.135, 82.145.209.252, 74.125.205.211 first_seen=11.01.2016 07:17 geo=ru, ua, kz, by, de, ro, az, cz, uz, md id=9491494 last_seen=14.01.2016 13:36 mask=malicious_domain_21.com/folder/load.php?* popularity=5 type=21

Total number of objects sent to KTFS: 1

Total number of detects received from KTFS: 1

Total scan time: 00:00:01.032

Page top