datafusion icon indicating copy to clipboard operation
datafusion copied to clipboard

UpdateD pool.rs

Open TouchstoneTheDev opened this issue 2 years ago • 5 comments

Which issue does this PR close?

Closes #6934

Rationale for this change

The rationale for this change is to add a debug dump of the MemoryManager to help diagnose memory exhaustion issues in certain queries. The PR proposes implementing the Display trait for the memory pools to produce a report of the reservations sorted in descending order, and adding a log message prior to returning a resources exhausted error.

What changes are included in this PR?

Implementing the Display trait for the memory pools to produce a report of the reservations sorted in descending order.

Are these changes tested?

Yes, the changes are tested. The PR includes updates to the MemoryManager tests to include the new functionality.

Are there any user-facing changes?

No, there are no user-facing changes. The changes are internal to the MemoryManager and are intended to help diagnose memory exhaustion issues in certain queries.

TouchstoneTheDev avatar Jul 13 '23 07:07 TouchstoneTheDev

Hi @tanmay-veer , this pr only includes the test, do you push the latest version?

Weijun-H avatar Jul 13 '23 08:07 Weijun-H

Yes i have only included tests

TouchstoneTheDev avatar Jul 13 '23 09:07 TouchstoneTheDev

I guess I was hoping to see code that would generate a report like this

GreedyPool 25 allocations, 25630532 used, 332 free, 43942344 capacity
  321433: GroupedHashAggregateStream[0]
  1233: GroupedHashAggregateStream[2]
  24: Deduplicate

In the debug log

debug!("Pool Exhausted while trying to allocate {additional} bytes for {reservation} :\n{self}", self)

alamb avatar Jul 13 '23 13:07 alamb

I think this is not waiting on review currently -- it is waiting for feedback to be addressed, so marking as "Draft"

alamb avatar Jul 14 '23 19:07 alamb

Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days.

github-actions[bot] avatar May 05 '24 01:05 github-actions[bot]