learning-spark icon indicating copy to clipboard operation
learning-spark copied to clipboard
verified publisher icon databricks

Basic ideas to solve Spark OOM: Count all the high frequence words in a big table

Open guotong1988 opened this issue 4 years ago • 0 comments

The detail question is:

I want to count all the high frequence words in a big table.

I split each sentence of each row, then flatmap to one word per row, then groupby, then count the word number in each group.

It will OOM.

guotong1988 avatar Mar 19 '21 01:03 guotong1988