Event route tracing

To find out which connection the events were received from, you can use the Trace event route setting. You may need to trace the route of an event when multiple agents send events to the same collector, for example, in a large infrastructure where agents are installed on dedicated WEC servers and multiple agents send events to the same collector. The event route information can be useful for troubleshooting event routing.

For the Trace event route toggle switch to become available, you must specify at least one internal destination in the agent. Also, to trace event routes, a connector of the internal type must be specified in the collector that is receiving events from the agent. After configuring and saving the agent, information about the agent's route is added to the S.KL_EventRoute field of the extended event schema.

The S.KL_EventRoute field is displayed only for new events that are received by the collector after enabling the Trace event route setting. Other services through which the event passes, including the collector, correlator (only correlation rules of the 'simple' type), and router, parse the S.KL_EventRoute field, and, if the field is not empty, add their own information to the field when they process the event.

If the event passes through multiple hosts on its way from the sender to the agent, the address of the last host that established a connection with the agent is saved in the S.KL_EventRoute field. If there is a proxy server between the sender and the agent, the address of this proxy is written to the field, and no separate proxy configuration is required on the agent. The tcp/udp/http connectors pass the address of the host from which the event was received via the internal protocol to S.KL_EventRoute, and the WEC, WMI, and ETW agents specify the host name of the Windows server on which they are installed.

You can enable event route tracing in one of the following ways:

Create a new agent, specify the internal destination in it and enable the Trace event route option.
In an existing agent, add a tab with a new connection and specify the internal destination on this tab. After you specify the internal destination, the Trace event route option becomes available. Enable the Trace event route toggle switch. After you save the agent settings, restart the agent to apply the changes. In this case, route tracing only affects the tab on which the internal destination is selected, not the entire agent.
Unlink the automatically created agent from the collector to make the agent available for editing its settings, or duplicate the automatically created agent and proceed to configure the settings in the duplicate agent. After you specify the internal destination in the editable agent, the Trace event route option becomes available. Enable the Trace event route toggle switch. After you save the agent settings, restart the agent to apply the changes. In this case, route tracing only affects the tabs on which the internal destination is selected, not the entire agent.

The Event tracing log section appears in the event card, alert card and correlation event card. This section displays information from the S.KL_EventRoute field in a processed form. Service IDs are converted to service names and displayed as clickable links. Clicking a service name opens a new browser tab with the card of the service. If you rename the service, the name of the service also changes when the event is displayed in the card, both for new events and for events that have already been received and processed. If you delete a service in the Active services section, the Event tracing log section displays Deleted instead of the hyperlink. The rest of the route information is not deleted and is still displayed, including the connector type, FQDN, the value of the SourceAddress field, and other route parameters from the S.KL_EventRoute field. To view raw route information, you can add the S.KL_EventRoute column to the event table.

Using route information

You can use the 'Extract from JSON' function or other ClickHouse functions to get the parts of the route that you need for debugging. For more information about functions, please refer to the ClickHouse documentation: https://clickhouse.com/docs/en/sql-reference/functions/json-functions#jsonextractstringjson-indices-or-keys

The following table gives examples of queries.

Query example	Description
SELECT *, simpleJSONExtractRaw(S.KL_EventRoute, 'version') as version FROM `events` Where Type !=4 ORDER BY Timestamp DESC LIMIT 25	This query displays events for which the version of the collector is displayed in the version column.
SELECT *, simpleJSONExtractRaw(S.KL_EventRoute, 'connectorKind') as connectorKind FROM `events` Where Type !=4 ORDER BY Timestamp DESC LIMIT 250	This query displays events for which the type of the collector is displayed in the connectorKind column.
SELECT *, simpleJSONExtractRaw(S.KL_EventRoute, 'connectorKind') as connectorKind FROM `events` Where Type !=4 and simpleJSONExtractRaw(S.KL_EventRoute, 'connectorKind') != '"wec"' ORDER BY Timestamp DESC LIMIT 250	This query displays events received from connectors of a type other than 'wec'.

If want to select by a different value from the S.KL_EventRoute field, you can look up the name of your parameter in the events table in the Raw column and use it in your query.

Page top