Workaround · Performance · Blocks work
Hundreds of Airflow workers exhaust Postgres connections, solved with PgBouncer
1 source thread · first seen 2022-02
Summary
At busy times Airflow launches hundreds of worker processes each needing a database connection, so client-side pooling does not help and a server-side pooler is used.
- Affects
- Teams running Airflow clusters on Postgres
- Workaround
- Place PgBouncer in front of the Postgres metadata database
Evidence
Excerpts are copied word for word from the source; follow the link to read it in full.
“each of them need at least 1 connection to Postgresql to save its result”