add gc.freeze to minimize memory usage in celery worker

[wjddn279]

discuss: https://lists.apache.org/thread/33hdp3hm705mzgrltv7o3468wvwbjsr3
related: #60505, #60505

Body

In a Celery worker, the Celery Main Process runs by default, and 16 ForkPoolWorkers are created as sub-processes of that process (based on default settings).

It operates based on COW (Copy-On-Write), which is the default memory behavior of fork, and through this, heavy modules (numpy, k8s) are pre-loaded. However, it was observed that as soon as ForkPoolWorkers are created, they have a higher PSS than the Main Process, confirming that memory increased due to COW.

As with other PRs, gc.freeze was applied, and it was confirmed that memory usage per process was reduced to less than half.

Benchmark

This test is a memory snapshot of celery worker container taken after 10 hours have elapsed when 100 tasks per minute were executed every minute, deploying with Docker

AS-IS (docker image 3.1.7)

TO-BE (apply with gc.freeze)

---

Was generative AI tooling used to co-author this PR?

• Yes (please specify the tool below)

---

• Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
• For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
• When adding dependency, check compliance with the ASF 3rd Party License Policy.
• For significant user-facing changes create newsfragment: {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

[wjddn279]

@hussein-awala @dheerajturaga

hello, code owners of celery workers...
Can I get some reviews about this?

[potiuk]

Sorry missed it!