Repository navigation
Adding memory profiling ? #216
Description
As memory_profiler is no longer maintained, I see this project as a perfect place to continue on it's work and provide a full powerful line_profiler that combine both information on time and memory consumption in one place, providing for instance some profiling results like this:
Line # Mem usage Increment Hits Time Line Contents
======================================================================
3 @profile
4 def my_func():
5 7.676 MiB 7.676 MiB 1 0.20 ms a = [1] * (10 ** 6)
6 160.301 MiB 152.625 MiB 1 0.25 ms b = [2] * (2 * 10 ** 7)
7 7.813 MiB -152.488 MiB 1 0.02 ms del b
8 7.813 MiB 0.000 MiB 1 1.200 s sleep(1.2)
9 7.813 MiB 0.000 MiB 1 0.01 ms return a
I'd really like to know what do you think about this possibility ?
I've also an idea of letting the user add more columns to this results based on his use case, for example:
def energy_tracker(time_inc, memory_inc): #maybe other args
return TIME_RATE * time_inc + MEM_RATE * memory_inc
profiler.register_trackers({"Energy": energy_tracker})
Which would add a new column called Energy with the given formula applied
Activity
Erotemic commentedon Jun 13, 2023
I think this is beyond the scope of this project. Tracking memory will impact the performance of tracking time.
For energy tracking, check out codecarbon.
Abdelgha-4 commentedon Jun 13, 2023
I guess if that's the main concern, we could give the user an option to choose what he wants to track.
This would allow both, having a powerful all-in-one profiler that can provide insights and diagnosis for the code from multiple angles, while also keeping the option of having a lightweight and fast time-only profiler.
Thank you for the reference, it's indeed a great package, but it lacks the line-by-line details and also the flexibility that "user specific columns" might add (for example a cost function, or a function to highlight only specific consumption patterns or some other custom logic ..)
Erotemic commentedon Jun 14, 2023
If you feel like you can write the logic to make it work, by all means go for it. I think it would be an interesting tool, and I may even use it. However, I don't have the bandwidth to entertain adding a new scope to this project. As such I'm going to close this issue as it pertains to this repository.
But don't let that stop you from implementing the feature. If you can get a working proof of concept, I'd love to hear about it and I'd link to it in the README here in related work. You're more than welcome to take this source code as a starting point and redistribute it under a new name.
Also I think scalene does have some sort of memory profiling (but probably not line-by-line).
Abdelgha-4 commentedon Jun 14, 2023
I perfectly understand, thank you for the encouragement!
Since I'm not that familiar with Cython (which is heavily used by line_profiler), I guess I'll start the other way around, forking memory_profiler (which is in pure python) and I'll see what I can get from there.
Anyway, it was nice hearing your opinion. Thank you for your time.