Skip to content

Measuring energy on a single machine running multiple concurrent GPU processes (Federated Learning simulation) #936

Open

Description

@AustinTapp

Dear authors,

Thanks for your nice work.

I have been using codecarbon for some time, and I was wondering what is the recommended way to use CodeCarbon to measure energy/emissions on a single machine where multiple GPU processes run in parallel?

Our use case is a federated learning simulation that launches many federated “clients” as separate processes on one GPU.

Right now, we run one EmissionsTracker per process (each simulated client), and this seems to be working reasonably. However, if we track per process, how does CodeCarbon derive the GPU power consumption? Is per-PID attribution supported via NVML, and could that double-count energy cost when multiple processes share a GPU? How should we think about CPU/RAM attribution when multiple processes are active? Are there docs/examples for multi-process, multi-GPU scenarios we should follow?

Thanks in advance! Any pointers to best practices or examples would be greatly appreciated.

Best regards,
Austin Tapp

Activity

  1. benoit-cty commented on Sep 24, 2025

    @benoit-cty
    Contributor

    Hello,

    Sorry, CodeCarbon can't do per-process with NVML yet. So you have to run CodeCarbon on the whole machine.

    If you have multiple GPU, CodeCarbon will monitor all of them.

    You could specify which GPU you want to monitor by giving the GPU id, so you could do per-GPU monitoring.

    For CPU/RAM attribution, the process mode is supposed to work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions