Pain point · Performance · Blocks work
Passing large data between tasks eats memory that is never released
2 source threads · first seen 2023-10 · last seen 2024-04
Summary
A user who otherwise likes Prefect's design finds that large task inputs and outputs stay in memory even after deleting them, unless result persistence is disabled on the task.
- Affects
- data engineers passing large data frames
- Workaround
- Disable persist_result and cache_result_in_memory on tasks
Evidence
Excerpts are copied word for word from the source; follow the link to read it in full.
“Memory consumption continues to increase with the number of task runs”
“This eaten up memory cannot be released, or at least, I could not find any option to do so”