After feeds are initially converted and imported into the MISP instance, you can make Kaspersky Threat Feed App for MISP run periodically at a specific interval using the cron utility.
Update interval
The specific update interval of a feed depends on the MISP hardware, feed type, and its record limit. We recommend to run an update every 30 minutes.
To schedule a conversion every 30 minutes:
crontab -e
*/30 * * * * python %service_dir%/main.py --full_update_interval_h %update_interval%
In the above command, replace %update_interval% with an interval (in hours) for running a full update.
The cron utility will now execute the main.py script every 30 minutes.
Instead of python, specify the full path for the python 3 interpreter. For instance: /bin/python3.
If the converter script does not finish the feed conversion process by the time the cron utility runs the converter again, then an attempt to run it again will produce an error. After the converter script finishes the feed conversion process, it can be run again.
If Kaspersky Threat Feed App for MISP runs on the same machine as MISP, we recommend to modify the cron job by adding a command that runs the fetching process after the conversion is properly completed. To do this, modify the previous cron job as follows:
*/30 * * * * python %service_dir%/main.py --full_update_interval_h %update_interval% && %MISP_dir%/app/Console/cake Server fetchFeed %MISP_user_id% %feed_id%
In the above command, replace:
%update_interval% with an interval (in hours) for running a full update.%MISP_dir% with the full path to the folder where MISP is installed.%MISP_user_id% with the ID of the MISP user whose account will be used to run the fetching.%feed_id% with the feed identifier assigned to it by MISP. Feed identifiers are available in the MISP UI.If more than one Kaspersky Threat Data Feed is to be added, then the above command should be transformed as follows:
*/30 * * * * python %service_dir%/main.py --full_update_interval_h %update_interval% && (%MISP_dir%/app/Console/cake Server fetchFeed %MISP_user_id% %feed_id_1%; %MISP_dir%/app/Console/cake Server fetchFeed %MISP_user_id% %feed_id_2%; ... ; %MISP_dir%/app/Console/cake Server fetchFeed %MISP_user_id% %feed_id_n%)
Please refer to MISP documentation to see which parameters can be used for the Server fetchFeed command.
Make sure that the user that will start the cron task has permission to run the %MISP_dir%/app/Console/cake file. To do this, add this user to the group www-data: usermod -a -G www-data %username%, where username is the user under which the cron task will be started. You may need to log out and log in again to make changes applied.
Otherwise, if Kaspersky Threat Feed App for MISP does not run on the same machine as MISP, set a cron task for regular fetching directly on the MISP instance or use the following API command (curl example):
curl --insecure -i -X POST -H "Authorization: %auth_key%" -H "Accept:application/json" -H "content-type: application/json" %MISP_url%/feeds/fetchFromFeed/%feed_id%
In the above command, replace:
%auth_key% with the AUTH key of the MISP instance.%MISP_url% with the URL or IP address of the MISP instance.%feed_id% with the feed identifier that was assigned to it by MISP. Feed identifiers are available in the MISP UI.The --insecure parameter causes curl to establish insecure SSL connections. This may create security issues. Use it only for evaluation purposes.
Conversion performance
The initial conversion process converts all records (up tp RECORDS_COUNT) from all the enabled feeds to MISP format. Subsequent converter calls calculate the diff with existing feeds but make no changes in records that were changed. The full update will launch only if the interval between the time of the last full update and the current time exceeds the value defined in the full_update_interval_h parameter (by default, 12 hours).
Loading all the feeds into MISP can take up to several days, but you can create a pool of MISP instances with one instance for each feed. In this case, set up the converter on each node to convert only one feed.
If done regularly, subsequent feed updates should not take more than an hour. The less frequently the updates are done, the bigger the diff and, consequently, the longer the update process.
Page top