Issue when using @Timed annotation with prometheus-metrics-instrumentation-dropwizard #1321

Activity

1. zeitlinger commented on Apr 8, 2025

   

   zeitlinger

   on Apr 8, 2025

   Member

   More actions

   This will be fixed in prometheus/OpenMetrics#286 - which basically says that metric names are not changed or checked in client sdks
2. kpur-sbab commented on Apr 10, 2025

   

   kpur-sbab

   on Apr 10, 2025

   Author

   More actions

   Great thanks. We are doing this as a temporary fix for DW 4.x

   prometheusRegistry.register(new DropwizardExports(environment.metrics(),
                   (name, metric) -> !(metric instanceof Timer) || !name.matches(".*[._]total.*")));
           environment.admin()
                   .addServlet("prometheusMetrics", new PrometheusMetricsServlet(prometheusRegistry))
                   .addMapping("/prometheusMetrics");
   
3. YellowFlora commented on Feb 18, 2026

   

   YellowFlora

   on Feb 18, 2026

   More actions

   Hello @zeitlinger,

   In the current client_java version the .total suffix is still dropped by AbstractDropwizardExports because it calls sanitizeMetricName, which removes the .total suffix.

   My understanding was that this behavior was supposed to change so that Dropwizard metrics ending in .total are handled in a way that avoids name collisions. Is that correct, or is the .total suffix still intentionally dropped?

   The issue is that when a Dropwizard metric like <servicepath>.uploadDocumentFile.total and a metric without the suffix both get sanitized to the same name (<servicepath>_uploadDocumentFile), the merge of snapshots later fails due to duplicate names with identical (empty) label sets.

   This causes a failure in TextFormatUtil.mergeDuplicates, because it tries to combine metrics with the same name and identical (empty) labels, resulting in a DuplicateLabelsException.

   Example test illustrating the failure:

   @Test
   void testMergeDuplicates_sameName_mergesLabels() {
     SummarySnapshot counter1 =
       SummarySnapshot.builder()
         .name("<servicepath>_uploadDocumentFile")
         .dataPoint(
           SummarySnapshot.SummaryDataPointSnapshot.builder()
             .count(100)
             .build())
         .build();
   
     SummarySnapshot counter2 =
       SummarySnapshot.builder()
         .name("<servicepath>_uploadDocumentFile")
         .dataPoint(
           SummarySnapshot.SummaryDataPointSnapshot.builder()
             .count(10)
             .build())
         .build();
   
     MetricSnapshots snapshots = new MetricSnapshots(counter1, counter2);
   
     assertThrows(DuplicateLabelsException.class, () -> TextFormatUtil.mergeDuplicates(snapshots));
   }

   So my question is: should .total still be stripped at this stage, or should the exporter avoid dropping it to prevent duplicate metric name collisions?
4. zeitlinger commented on Mar 6, 2026

   

   zeitlinger

   on Mar 6, 2026

   Member

   More actions

   @YellowFlora You're right that .total is currently stripped, and that's working as designed per the OpenMetrics 1.0 spec. This will change as part of #1912 (OpenMetrics 2.0 support), which will stop treating _total/.total as reserved suffixes that get stripped during sanitization.

   👍React with 👍1Reacted by YellowFlora
5. josephlbarnett commented on Mar 27, 2026

   

   josephlbarnett

   on Mar 27, 2026

   More actions

   rather than filtering them out (we use async requests so the .total is more useful than the raw one), we map ".total" -> ".async" like so:

   .customLabelMapper(
                       object : CustomLabelMapper(
                           listOf(MapperConfig(
                               // This config won't actually match what we want because the * glob stops at '.'s
                               // but we do need a non-empty list of MapperConfigs to use a CustomLabelMapper
                               "*.total", $$"${0}.total", mapOf())),
                       ) {
                           override fun getName(dropwizardName: String?): String {
                               // this one actually remaps **.total -> **.async
                               return super.getName(dropwizardName?.replace("\\.total$".toRegex(), ".async"))
                           }
                       },
                   )

   not sure if there's a less awkward way to set it up with the current code?
6. josephlbarnett

   added a commit that references this issue on Mar 27, 2026

   WIP: prometheus metrics scraping

   ...

   1e118bd
7. josephlbarnett

   added a commit that references this issue on May 6, 2026

   WIP: prometheus metrics scraping

   ...

   5c3b100
8. josephlbarnett commented on Jun 11, 2026

   

   josephlbarnett

   on Jun 11, 2026

   Last edited by josephlbarnett

   More actions

   also fwiw, it seemed like a workaround was not needed in 1.6.1, but is needed again in 1.7.0, i think per #2100 and/or #2124