Event route tracing

To find out which connection the events were received from, you can use the Trace event route setting. You may need to trace the route of an event when multiple agents send events to the same collector, for example, in a large infrastructure where agents are installed on dedicated WEC servers and multiple agents send events to the same collector. The event route information can be useful for troubleshooting event routing.

For the Trace event route toggle switch to become available, you must specify at least one internal destination in the agent. Also, to trace event routes, a connector of the internal type must be specified in the collector that is receiving events from the agent. After configuring and saving the agent, information about the agent's route is added to the S.KL_EventRoute field of the extended event schema.

The S.KL_EventRoute field is displayed only for new events that are received by the collector after enabling the Trace event route setting. Other services through which the event passes, including the collector, correlator (only correlation rules of the 'simple' type), and router, parse the S.KL_EventRoute field, and, if the field is not empty, add their own information to the field when they process the event.

If there is a proxy between the sender server and the agent, the proxy address is specified in the S.KL_EventRoute field on the agent, and the collector adds the proxy address to the initial part of the route in the S.KL.EventRoute field.

Connectors of the tcp/udp/http type use the internal protocol to pass the address of the host that sent the event in the S.KL_EventRoute field. If there is a proxy between the sender server and the agent, the proxy is specified indicated in the S.KL_EventRoute field. The WEC, WMI, and ETW agents use the internal protocol to pass the host name of the Windows server on which the agent is installed in the S.KL_EventRoute field.

You can enable event route tracing in one of the following ways:

Create a new agent, specify the internal destination in it and enable the Trace event route option.
In an existing agent, add a tab with a new connection and specify the internal destination on this tab. After you specify the internal destination, the Trace event route option becomes available. Enable the Trace event route toggle switch. After you save the agent settings, restart the agent to apply the changes.
Unlink the automatically created agent from the collector to make the agent available for editing its settings, or duplicate the automatically created agent and proceed to configure the settings in the duplicate agent. After you specify the internal destination in the editable agent, the Trace event route option becomes available. Enable the Trace event route toggle switch. After you save the agent settings, restart the agent to apply the changes.

The Event tracing log section appears in the event card, alert card and correlation event card. This section displays information from the S.KL_EventRoute field in a processed form. Service IDs are converted to service names and displayed as clickable links. Clicking a service name opens a new browser tab with the card of the service. If you rename the service, the name of the service also changes when the event is displayed in the card, both for new events and for events that have already been received and processed. If you delete a service in the Active services section, the Event tracing log section displays Deleted instead of the hyperlink. The rest of the route information is not deleted and continues to be displayed: connector type, FQDN, and the value of the SourceAddress field. To view raw route information, you can add the S.KL_EventRoute column to the event table.

Using route information

You can use the 'Extract from JSON' function or other ClickHouse functions to get the parts of the route that you need for debugging. For more information about functions, please refer to the ClickHouse documentation: https://clickhouse.com/docs/en/sql-reference/functions/json-functions#jsonextractstringjson-indices-or-keys

The following table gives examples of queries.

Query example	Description
SELECT *, simpleJSONExtractRaw(S.KL_EventRoute, 'version') as version FROM `events` Where Type !=4 ORDER BY Timestamp DESC LIMIT 25	This query displays events for which the version of the collector is displayed in the version column.
SELECT *, simpleJSONExtractRaw(S.KL_EventRoute, 'connectorKind') as connectorKind FROM `events` Where Type !=4 ORDER BY Timestamp DESC LIMIT 250	This query displays events for which the type of the collector is displayed in the connectorKind column.
SELECT *, simpleJSONExtractRaw(S.KL_EventRoute, 'connectorKind') as connectorKind FROM `events` Where Type !=4 and simpleJSONExtractRaw(S.KL_EventRoute, 'connectorKind') != '"wec"' ORDER BY Timestamp DESC LIMIT 250	This query displays events received from connectors of a type other than 'wec'.

If want to select by a different value from the S.KL_EventRoute field, you can look up the name of your parameter in the events table in the Raw column and use it in your query.

Page top