[FEATURE] More granular config options for control of liger and optimizer

[vadimkantorov]

1. Would be nice to add optimizer_fused, optimizer_foreach config options. It allows to better control the memory footprint of the optimizer

   Upstream issue:

   • Add optimizer fused=True and foreach=False options in the config to allow sidestepping current Adam's foreach=True intermediate param-sized extra allocation verl-project/verl#2556

2. Would be nice to better control _apply_liger_kernel_to_instance(model=actor_module) ->
   _apply_liger_kernel_to_instance(model=actor_module, fused_linear_cross_entropy=liger_fused_linear_cross_entropy, cross_entropy=liger_cross_entropy). This is needed because otherwise Verl's fused FusedLinearCrossEntropy kernel conflicts with Liger kernels - at least used to conflict

   Upstream issue:

   • use_liger's _apply_liger_kernel_to_instance(model=actor_module) should be called as _apply_liger_kernel_to_instance(model=actor_module, fused_linear_cross_entropy=False) when use_fused_kernels=True verl-project/verl#2609

[vadimkantorov]

Should I send a PR for these?

[pan-x-c]

Thanks for reporting these issues, we will verify these options next Monday. If you would like, you can send a PR to help us speed up the verification process.

[vadimkantorov]

Probably just controlling for fused_linear_cross_entropy=False is enough for enabling both fused_kernels and liger. Ideally, these patches should be sent upstream to verl too...

[chenyushuo]

We have fixed the setting of fused_linear_cross_entropy in verl/fsdp_workers.py.

[vadimkantorov]

Thanks :) The remaining item in this issue would be introducing granular options fused=True/False (and maybe foreach=True/False) for the optimizer.

By default fused=False and foreach=True is used for now:

• Transition Adam and maybe other optimizers to fused code path by default to avoid foreach=True-specific VRAM peak due to temp TensorList for bias-corrected moments pytorch/pytorch#158371 (comment)

fused=True is less memory-hungry, so would be nice to have an option to enable fused or disable foreach