Scalarizer Tracker

[ValerianRey]

This issue tracks the candidate methods that could maybe be implemented as Scalarizer in torchjd.scalarization.

Status	Name	Ref	Stateful	Existing implementations	Special Remarks
🟡 In progress (#731)	DWA (Dynamic Weight Averaging)	End-to-End Multi-Task Learning with Attention (CVPR 2019, 1836 citations)	Yes, previous epochs' average losses	official, LibMTL	Need an extra step method to tell the scalarizer we switched epochs.
🔵 To investigate	FAMO	FAMO: Fast Adaptive Multitask Optimization (NeurIPS 2023, 124 citations)	?	official, LibMTL	Not sure this is a Scalarizer, need more investigation.
🔵 To investigate	GradNorm	GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks (ICML 2018, 2334 citations)	Yes, trainable state	unofficial, LibMTL	Not sure this is a Scalarizer, need more investigation.
🔵 To investigate	COSMOS	Scalable Pareto Front Approximation for Deep Multi-Objective Learning (ICDM 2021, 95 citations)	No	official, LibMoon
🔵 To investigate	PBI (Penalty-based Boundary Intersection)	MOEA/D: A Multiobjective Evolutionary Algorithm Based on Decomposition (IEEE Transactions on evolutionary computation, 2007, 11728 citations)	?	LibMoon	This comes from the evolutionary algorithms field, so let's only implement it if it's relevant to gradient-based methods.

[PierreQuinton]

Let's keep this tracking issue clean. We will discuss this in a PR, when created.