Is it possible to add a custom metadata identifier when using Exemplars?

[Rebwon]

Hello. We are currently upgrading from Prometheus old client 0.x to 1.x.

In the previous version, when using ExemplarSampler and Exemplar, the structure allowed us to include separate custom metadata, so in our system we used one additional identifier for management purposes.

However, while reviewing the 1.x version during the upgrade, it appears that the SpanContext and Exemplar classes only provide trace_id and span_id as identifiers, and do not seem to allow adding separate custom metadata. It is possible that I may have missed an extension point.

Is there any way to include a separate custom metadata identifier when using Exemplars?

[Rebwon]

Is there anyone who could respond to this issue?

[ADITYA-CODE-SOURCE]

Hi @Rebwon,
Yes — this is possible in 1.x.
Custom metadata can be attached to exemplars as additional exemplar labels. So while SpanContext only provides trace_id and span_id automatically, you can still include your own extra identifier explicitly.
For example:

Exemplar exemplar = Exemplar.builder()
    .value(2.0)
    .traceId("abc")
    .spanId("123")
    .labels(Labels.of("custom_metadata", "your_value"))
    .build();

Or when recording directly on a metric:
counter.incWithExemplar(1.0, Labels.of("custom_metadata", "your_value"));
So if your 0.x setup used an additional identifier for management purposes, the 1.x equivalent is to pass it as an extra exemplar label.

[Rebwon]

@ADITYA-CODE-SOURCE

Thank you for your response.

The approach you suggested seems to involve programmatically adding custom labels directly to the exemplar.

Is there a way to have these added automatically, similar to how they are included via SpanContext?

[jaydeluca]

this is not currently possible, but I am exploring what it would take to add it

[jaydeluca]

@Rebwon would this type of API suit your needs?

Counter counter =
    Counter.builder()
        .name("requests_total")
        .exemplarLabelsSupplier(() -> Labels.of("management_id", "mgmt-42"))
        .build();

[Rebwon]

@jaydeluca

My requirement is to configure this globally, rather than at the point where individual metrics are created.

While inserting custom labels into exemplars is possible for metrics that we define ourselves, this approach does not work for metrics that are registered by external libraries that we do not control.

[jaydeluca]

@Rebwon thanks for that clarification. I think we can support this, but we have a few other priorities at the moment. If you are interested in contributing it feel free, otherwise I can work on it when time allows