Recommendations on using Log Scanner

We recommend that you use Feed Service together with Log Scanner in the following cases:

• You have to check some log files and save the check result to a file.

  It can be useful while investigating information security incidents, when the SIEM solution you use is unavailable, or if you do not use any SIEM solution.

• You have to check some log files and send the check results to the SIEM solution used.

Configuring Feed Service and Log Scanner

Feed Service and Log Scanner must interact correctly, so their corresponding parameters must be set according to each other as follows:

• The port set in the Settings > Connection element of the Log Scanner configuration file must accord with the port specified in the InputSettings > ConnectionString element of the Feed Service configuration file.
• The number of threads specified in the Settings > ThreadsCount element of the Log Scanner configuration file must not be greater than that specified in the ServiceSettings > ScannersCount element of the Feed Service configuration file. If Feed Service runs in watchdog mode, the number of threads specified in the Settings > ThreadsCount element of the Log Scanner configuration file must be less than that specified in the ServiceSettings > ScannersCount element of the Feed Service configuration file.
• The data sent by Log Scanner to Feed Service—either lines of log files or strings created on the basis of the Settings > Pattern element of the Log Scanner configuration file—must be parsable by regular expressions specified in the Configuration > InputSettings > RegExps element of the Feed Service configuration file.

Configuration files examples

The following is an excerpt from a sample Feed Service configuration file.

<Configuration> <InputSettings> <RegExps> <Source id="default"> <!--You can use them in the OutputSettings->EventFormat string with the pattern %REGEXPNAME%--> ... <RE_MD5>md5=(.*?)(?:$|\s)</RE_MD5> <RE_SHA1>sha1=(.*?)(?:$|\s)</RE_SHA1> <RE_SHA256>sha256=(.*?)(?:$|\s)</RE_SHA256> <RE_URL>url=(.*?)(?:$|\s)</RE_URL> <RE_IP>ip=(.*?)(?:$|\s)</RE_IP> </Source> </RegExps> <ConnectionString>127.0.0.1:9999</ConnectionString> <!-- <ip>:<port>. Threat Feed Service listens for <ip>:<port>. <port> must be available --> </InputSettings>   <Feeds per_scan_detect_limit="10000">...</Feeds>   <OutputSettings> ... <FinishedEventFormat>LookupFinished</FinishedEventFormat> </OutputSettings>   <ServiceSettings> ... <ScannersCount>9</ScannersCount> <!-- 1 tcp connection = 1 scanner --> </ServiceSettings> </Configuration>

The following is an excerpt from a Log Scanner configuration file that corresponds to the Feed Service configuration file provided above.

<Settings> ... <ThreadsCount>8</ThreadsCount> <Pattern>ip=%IP% md5=%MD5% sha1=%SHA1% sha256=%SHA256% url=%URL%</Pattern> <Connection>127.0.0.1:9999</Connection> </Settings>

When using these configuration files, Log Scanner sends the requests to the IP address 127.0.0.1 and port 9999, and Feed Service listens on port 9999 for data to check. Both Log Scanner and Feed Service use up to 8 threads for transferring and processing data, and one thread is used by the watchdog module (in Linux) or watchdog service (in Windows). If correct URLs, IP addresses, and hashes are sent to Feed Service for checking, they will be successfully parsed by using the regular expressions specified in the Feed Service configuration file.

Managing check results

After data is checked by Feed Service, you can either send the check results to event target software or save them to a file:

• For sending check results to event target software, set the correct value of the "OutputSettings > ConnectionString" element of the Feed Service configuration file.
• For saving check results to a file, pass the -r option when running Log Scanner from the command line as follows:

  ./log_scanner -r -p file_to_check (in Linux)

  log_scanner.exe -r -p file_to_check (in Windows)

  The value of the enable attribute of the OutputSettings > FinishedEventFormat element in the Feed Service configuration file must not be false.

Report example

The report content depends on the value of the OutputSettings > EventFormat element of the Feed Service configuration file.

The following is an example of a report sent by Feed Service to Log Scanner.

- KL_Data_Feed_Service_v1 LEEF:1.0|Kaspersky Lab|SIEM Service|1.0|KL_Malicious_URL|url=malicious_domain_21.com/folder/load.php?| IP=91.202.63.117, 196.254.10.200, 194.190.253.19, 185.56.137.11, 178.62.5.157, 173.194.222.211, 159.253.145.183, 87.250.250.135, 82.145.209.252, 74.125.205.211 first_seen=11.01.2016 07:17 geo=ru, ua, kz, by, de, ro, az, cz, uz, md id=9491494 last_seen=14.01.2016 13:36 mask=malicious_domain_21.com/folder/load.php?* popularity=5 type=21 Total number of objects sent to KTFS: 1 Total number of detects received from KTFS: 1 Total scan time: 00:00:01.032

Page top