Detaching, archiving, and attaching partitions

If you want to optimize disk space and speed up queries in KUMA, you can detach data partitions in ClickHouse, archive partitions, or move partitions to a drive. If necessary, you can later reattach the partitions you need and perform data processing.

Detaching partitions

To detach partitions:

  1. Determine the shard on all replicas of which you want to detach the partition.
  2. Get the partition ID using the following command:

    sudo /opt/kaspersky/kuma/clickhouse/bin/client.sh -d kuma --multiline --query "SELECT partition, name FROM system.parts;" |grep 20231130

    In this example, the command returns the partition ID for November 30, 2023.

  3. One each replica of the shard, detach the partition using the following command and specifying the partition ID:

    sudo /opt/kaspersky/kuma/clickhouse/bin/client.sh -d kuma --multiline --query "ALTER TABLE events_local_v2 DETACH PARTITION ID '<partition ID>'"

As a result, the partition is detached on all replicas of the shard. Now you can move the data directory to a drive or archive the partition.

Archiving partitions

To archive detached partitions:

  1. Find the detached partition in disk subsystem of the server:

    sudo find /opt/kaspersky/kuma/clickhouse/data/ -name <ID of the detached partition>\*

  2. Change to the 'detached' directory that contains the detached partition, and while in the 'detached' directory, perform the archival:

    sudo cd <path to the 'detached' directory containing the detached partition>

    sudo zip -9 -r detached.zip *

    For example:

    sudo cd /opt/kaspersky/kuma/clickhouse/data/store/d5b/d5bdd8d8-e1eb-4968-95bd-d8d8e1eb3968/detached/

    sudo zip -9 -r detached.zip *

The partition is archived.

Attaching partitions

To attach archived partitions to KUMA:

  1. Increase the Retention period value.

    KUMA deletes data based on the date specified in the Timestamp field, which records the time when the event is received, and based on the Retention period value that you set for the storage.

    Before restoring archived data, make sure that the Retention period value overlaps the date in the Timestamp field. If this is not the case, the archived data will be deleted within 1 hour.

  2. Place the archive partition in the 'detached' section of your storage and unpack the archive:

    sudo unzip detached.zip -d <path to the 'detached' directory>

    For example:

    sudo unzip detached.zip -d /opt/kaspersky/kuma/clickhouse/data/store/d5b/d5bdd8d8-e1eb-4968-95bd-d8d8e1eb3968/detached/

  3. Run the command to attach the partition:

    sudo /opt/kaspersky/kuma/clickhouse/bin/client.sh -d kuma --multiline --query "ALTER TABLE events_local_v2 ATTACH PARTITION ID '<partition ID>'"

    Repeat the steps of unpacking the archive and attaching the partition on each replica of the shard.

As a result, the archived partition is attached and its events are again available for search.

Page top