Documentation

Enable trace logging

Learn how to enable trace logging to help you identify performance bottlenecks and troubleshoot problems in queries.

When you enable trace logging for a query, InfluxDB propagates your trace ID through system processes and collects additional log information. InfluxData Support can then use the trace ID that you provide to filter, collate, and analyze log information for the query run. The tracing system follows the OpenTelemetry traces model for providing observability into a request.

Avoid unnecessary tracing

Only enable tracing for a query when you need to request troubleshooting help from InfluxDB Support. To manage resources, InfluxDB has an upper limit for the number of trace requests. Too many traces can cause InfluxDB to evict log information.

To enable tracing for a query, include the influx-trace-id header in your query request.

Syntax

Use the following syntax for the influx-trace-id header:

influx-trace-id: TRACE_ID:1112223334445:0:1

In the header value, replace the following:

  • TRACE_ID: a unique string, 8-16 bytes long, encoded as hexadecimal (32 maximum hex characters). The trace ID should uniquely identify the query run.
  • :1112223334445:0:1: InfluxDB constant values (required, but ignored)

Example

The following examples show how to create and pass a trace ID to enable query tracing in InfluxDB:

Use the InfluxDBClient3 InfluxDB Python client and pass the headers argument in the query() method.

from influxdb_client_3 import InfluxDBClient3
import secrets

def use_flightcalloptions_trace_header():
  print('# Use FlightCallOptions to enable tracing.')
  client = InfluxDBClient3(token=f"
DATABASE_TOKEN
"
,
host=f"cluster-id.a.influxdb.io", database=f"
DATABASE_NAME
"
)
# Generate a trace ID for the query: # 1. Generate a random 8-byte value as bytes. # 2. Encode the value as hexadecimal. random_bytes = secrets.token_bytes(8) trace_id = random_bytes.hex() # Append required constants to the trace ID. trace_value = f"{trace_id}:1112223334445:0:1" # Encode the header key and value as bytes. # Create a list of header tuples. headers = [((b"influx-trace-id", trace_value.encode('utf-8')))] sql = "SELECT * FROM home WHERE time >= now() - INTERVAL '30 days'" influxql = "SELECT * FROM home WHERE time >= -90d" # Use the query() headers argument to pass the list as FlightCallOptions. client.query(sql, headers=headers) client.close() use_flightcalloptions_trace_header()

Use the InfluxDBClient3 InfluxDB Python client and flight.ClientMiddleware to pass and inspect headers.

Tracing response header

With tracing enabled and a valid trace ID in the request, InfluxDB’s DoGet action response contains a header with the trace ID that you sent.

Trace response header syntax
trace-id: TRACE_ID

Inspect Flight response headers

To inspect Flight response headers when using a client library, pass a FlightClientMiddleware instance that defines a middleware callback function for the onHeadersReceived event (the particular function name you use depends on the client library language).

The following example uses Python client middleware that adds request headers and extracts the trace ID from the DoGet response headers:

import pyarrow.flight as flight

class TracingClientMiddleWareFactory(flight.ClientMiddleware):
  # Defines a custom middleware factory that returns a middleware instance.
    def __init__(self):
        self.request_headers = []
        self.response_headers = []
        self.traces  = []

    def addRequestHeader(self, header):
        self.request_headers.append(header)

    def addResponseHeader(self, header):
        self.response_headers.append(header)

    def addTrace(self, traceid):
        self.traces.append(traceid)

    def createTrace(self, traceid):
      # Append InfluxDB constants to the trace ID.
      trace = f"{traceid}:1112223334445:0:1"

      # To the list of request headers,
      # add a tuple with the header key and value as bytes.
      self.addRequestHeader((b"influx-trace-id", trace.encode('utf-8')))

    def start_call(self, info):
        return TracingClientMiddleware(info.method, self)

class TracingClientMiddleware(flight.ClientMiddleware):
  # Defines middleware with client event callback methods.
    def __init__(self, method, callback_obj):
        self._method = method
        self.callback = callback_obj

    def call_completed(self, exception):
      print('callback: call_completed')
      if(exception):
        print(f"  ...with exception: {exception}")

    def sending_headers(self):
      print('callback: sending_headers: ', self.callback.request_headers)
      if len(self.callback.request_headers) > 0:
        return dict(self.callback.request_headers)

    def received_headers(self, headers):
      self.callback.addResponseHeader(headers)
      # For the DO_GET action, extract the trace ID from the response headers.
      if str(self._method) == "FlightMethod.DO_GET" and "trace-id" in headers:
          trace_id = headers["trace-id"][0]
          self.callback.addTrace(trace_id)

from influxdb_client_3 import InfluxDBClient3
import secrets

def use_middleware_trace_header():
  print('# Use Flight client middleware to enable tracing.')

  # Instantiate the middleware.
  res = TracingClientMiddleWareFactory()

  # Instantiate the client, passing in the middleware instance that provides
  # event callbacks for the request.
  client = InfluxDBClient3(token=f"
DATABASE_TOKEN
"
,
host=f"cluster-id.a.influxdb.io", database=f"
DATABASE_NAME
"
,
flight_client_options={"middleware": (res,)}) # Generate a trace ID for the query: # 1. Generate a random 8-byte value as bytes. # 2. Encode the value as hexadecimal. random_bytes = secrets.token_bytes(8) trace_id = random_bytes.hex() res.createTrace(trace_id) sql = "SELECT * FROM home WHERE time >= now() - INTERVAL '30 days'" client.query(sql) client.close() assert trace_id in res.traces[0], "Expect trace ID in DoGet response." use_middleware_trace_header()

Replace the following:

  • DATABASE_NAME: your InfluxDB Cloud Dedicated database
  • DATABASE_TOKEN: a database token with sufficient permissions to the specified database

Store or log your query trace ID to ensure you can provide it to InfluxData Support for troubleshooting.

After you run your query with tracing enabled, do the following:

  • Remove the tracing header from subsequent runs of the query (to avoid unnecessary tracing).
  • Provide the trace ID in a request to InfluxData Support.

Retrieve system information for a query

If you enable trace logging for a query, the trace-id appears in the system.queries table.


Was this page helpful?

Thank you for your feedback!


The future of Flux

Flux is going into maintenance mode. You can continue using it as you currently are without any changes to your code.

Read more

InfluxDB v3 enhancements and InfluxDB Clustered is now generally available

New capabilities, including faster query performance and management tooling advance the InfluxDB v3 product line. InfluxDB Clustered is now generally available.

InfluxDB v3 performance and features

The InfluxDB v3 product line has seen significant enhancements in query performance and has made new management tooling available. These enhancements include an operational dashboard to monitor the health of your InfluxDB cluster, single sign-on (SSO) support in InfluxDB Cloud Dedicated, and new management APIs for tokens and databases.

Learn about the new v3 enhancements


InfluxDB Clustered general availability

InfluxDB Clustered is now generally available and gives you the power of InfluxDB v3 in your self-managed stack.

Talk to us about InfluxDB Clustered