Repository navigation
manifests: count telemetry by service on kind - #1070
Conversation
4b9dab5 to
3224a52
Compare
| # max_streams bounds the cardinality. There is one stream for each pair | ||
| # of a count and a service, thus a limit far above the count of | ||
| # services is sufficient. | ||
| deltatocumulative: | ||
| max_streams: 1000 |
There was a problem hiding this comment.
I just tested this locally and in max_streams streams are actually per source resource, and since serverboot gives every process its own service.instance.id 1000 is not a lot. Maybe we can use locally, but in meter.yaml we can configure a lot higher value?
There was a problem hiding this comment.
Consider this will be used by benchmarking. How about 10000 in kind and 100k in meter.yaml?
At a few hundred bytes per tracked datapoint that's roughly 40~50 MB of data in memory, negligible against the memory limit of collector.
There was a problem hiding this comment.
10000 on kind sounds right to me. I measured ~5KiB per stream end to end, so 10k is ~ 50MiB.
For 100k on the meter the memory is OK with 263MiB live heap, 675MiB RSS, but it becomes 38.5 MiB per scrape at 100k series, and we scrape every 10s from a local Prometheus.
I'd take 20000 there and add sum(otelcol_deltatocumulative_datapoints{error="limit"}) must be 0 + streams_tracked / streams_limit to the pass criteria, since a limit hit deletes series from the scrape with no log line at all.
Wdyt?
There was a problem hiding this comment.
sg, updated
715ea0c to
c671804
Compare
Krisztian F (krisztianfekete)
left a comment
Krisztian F (krisztianfekete)
left a comment
There was a problem hiding this comment.
There are some flakes (?), but LGTM!
The volume measurement worked on GKE only. There the managed collector
cannot hold a connector, thus benchmarking/telemetry/meter.yaml puts a
second collector in front of it and counts the data as it goes through.
On kind the collector is ours, thus it can do the count itself and the
cluster needs no meter and no extra hop.
Add the count connector to the kind collector. It now gives the same two
counts as the meter, substrate_spans_total{service_name} and
substrate_datapoints_total{service_name}, on the Prometheus exporter port
that it already opens. The connector resolves each key from the attributes
of the item, then the scope, then the resource, thus service.name resolves
with no transform in front of it. Drop the transform from the meter also.
Give the collector its otelcol_* counters also. The default reader binds
to localhost, thus nothing outside the pod could read it. Bind it to each
interface and open port 8888. The counters are the positive control of a
measurement: without them a service that reports zero volume and data that
never arrived look the same.
One scrape job reads both ports. The prometheus.io/scrape annotation can
name one port only, and port 8889 re-exports each metric that the pipelines
push. The pods of ateapi, atelet, and atenet-router already carry that
annotation on 9090, thus an annotation on the collector would bring each of
their series in a second time and sum() would give double. The job keeps
the substrate_* and otelcol_* names only.
The actors need no change. benchmarking/workloads/deploy.sh reads the
endpoint from the ate-otel-config ConfigMap, and the kind overlay of that
ConfigMap already names this collector.
Follow-up to agent-substrate#749.
c671804 to
6c96cd3
Compare
The volume measurement in current benchmarking repo worked on GKE only. There the managed collector cannot hold a connector, thus benchmarking/telemetry/meter.yaml puts a second collector in front of it and counts the data as it goes through.
This PR adds the support to kind cluster to count the telemetry data.
Follow-up to #749.