jsvine
Pickle implementation for PDF and Page objects
Would love it if there was a way a pickling implementation could be implemented for Page or PDF objects. Currently none exists and makes working with things like multiprocessing a bit harder if one is interested in speeding up pdf processing over pages. Any advice or workaround from the community would be much appreciated :)
Hi @rajathsalegame, and thanks for the suggestion. I agree that this could be a useful feature. Unfortunately for your multiprocessing use-case, a lot of the heavy processing load currently is handled by the pdfminer.six dependency, which I don't believe supports pickling.
To better understand the request, could you provide a bit more detail about your goals with multiprocessing and at what specific stage in your pipeline you'd be pickling/unpickling?
Hi, in my case, I need to apply page.dedupe_chars().extract_words(keep_blank_chars=True) on every pages in a PDF file. For one file with 20 pages, it takes about 16 sec, taking up most of the time consumtion in my pipline. It will be great for pdfplumber to support pickle and multiprocessing.
BTW, if I do something like:
import pandas as pf
from joblib import Parallel, delayed
def parse_page(page):
df_page = pd.DataFrame(page.dedupe_chars().extract_words(keep_blank_chars=True))
with pdfplumber.open(pdf_file_or_path) as pdf:
result_parse = Parallel(-1, 'threading')(delayed(parse_page)(page) for page in pdf.pages)
It seems the error info is not about page being unpickable, but:
File "<ipython-input-50-192c80c0ccbc>", line 207, in parse_page
df_page = pd.DataFrame(page.dedupe_chars().extract_words(keep_blank_chars=True))
File "/home/myname/miniconda3/envs/diamondforce/lib/python3.10/site-packages/pdfplumber/page.py", line 403, in dedupe_chars
p._objects = {kind: objs for kind, objs in self.objects.items()}
File "/home/myname/miniconda3/envs/diamondforce/lib/python3.10/site-packages/pdfplumber/page.py", line 215, in objects
self._objects: Dict[str, T_obj_list] = self.parse_objects()
File "/home/myname/miniconda3/envs/diamondforce/lib/python3.10/site-packages/pdfplumber/page.py", line 275, in parse_objects
for obj in self.iter_layout_objects(self.layout._objs):
File "/home/myname/miniconda3/envs/diamondforce/lib/python3.10/site-packages/pdfplumber/page.py", line 161, in layout
interpreter.process_page(self.page_obj)
File "/home/myname/miniconda3/envs/diamondforce/lib/python3.10/site-packages/pdfminer/pdfinterp.py", line 997, in process_page
self.render_contents(page.resources, page.contents, ctm=ctm)
File "/home/myname/miniconda3/envs/diamondforce/lib/python3.10/site-packages/pdfminer/pdfinterp.py", line 1016, in render_contents
self.execute(list_value(streams))
File "/home/myname/miniconda3/envs/diamondforce/lib/python3.10/site-packages/pdfminer/pdfinterp.py", line 1021, in execute
parser = PDFContentParser(streams)
File "/home/myname/miniconda3/envs/diamondforce/lib/python3.10/site-packages/pdfminer/pdfinterp.py", line 251, in __init__
PSStackParser.__init__(self, None) # type: ignore[arg-type]
File "/home/myname/miniconda3/envs/diamondforce/lib/python3.10/site-packages/pdfminer/psparser.py", line 545, in __init__
PSBaseParser.__init__(self, fp)
File "/home/myname/miniconda3/envs/diamondforce/lib/python3.10/site-packages/pdfminer/psparser.py", line 193, in __init__
self.seek(0)
File "/home/myname/miniconda3/envs/diamondforce/lib/python3.10/site-packages/pdfminer/pdfinterp.py", line 263, in seek
self.fillfp()
File "/home/myname/miniconda3/envs/diamondforce/lib/python3.10/site-packages/pdfminer/pdfinterp.py", line 256, in fillfp
strm = stream_value(self.streams[self.istream])
File "/home/myname/miniconda3/envs/diamondforce/lib/python3.10/site-packages/pdfminer/pdftypes.py", line 217, in stream_value
x = resolve1(x)
File "/home/myname/miniconda3/envs/diamondforce/lib/python3.10/site-packages/pdfminer/pdftypes.py", line 118, in resolve1
x = x.resolve(default=default)
File "/home/myname/miniconda3/envs/diamondforce/lib/python3.10/site-packages/pdfminer/pdftypes.py", line 106, in resolve
return self.doc.getobj(self.objid)
File "/home/myname/miniconda3/envs/diamondforce/lib/python3.10/site-packages/pdfminer/pdfdocument.py", line 866, in getobj
obj = self._getobj_parse(index, objid)
File "/home/myname/miniconda3/envs/diamondforce/lib/python3.10/site-packages/pdfminer/pdfdocument.py", line 840, in _getobj_parse
(_, obj) = self._parser.nextobject()
File "/home/myname/miniconda3/envs/diamondforce/lib/python3.10/site-packages/pdfminer/psparser.py", line 656, in nextobject
self.do_keyword(pos, token)
File "/home/myname/miniconda3/envs/diamondforce/lib/python3.10/site-packages/pdfminer/pdfparser.py", line 79, in do_keyword
(objid, genno) = (int(objid), int(genno)) # type: ignore[arg-type]
TypeError: int() argument must be a string, a bytes-like object or a real number, not 'PSKeyword'
"""
Does this make multiprocess with pdfplumber a bit easier?
For anyone who's interested, a work-around for multiprocessing for pdfplumber is to start multiprocessing before opening the pdf file with pdfplumber:
from joblib import Parallel, delayed
import pdfplumber
import pandas as pd
def plumber_parsepage(pdf_path: str,
idx_page: int = 0,
) -> pd.DataFrame:
with pdfplumber.open(pdf_path) as pdf:
page = pdf.pages[idx_page]
df_page = pd.DataFrame(page.dedupe_chars().extract_words(keep_blank_chars=True))
return df_page
with pdfplumber.open(pdf_file_or_path) as pdf:
num_page = len(pdf.pages)
result = Parallel(-1)(delayed(plumber_parsepage)(pdf_file_or_path, i) for i in range(num_page))
For anyone who's interested, a work-around for
multiprocessingforpdfplumberis to start multiprocessing before opening the pdf file withpdfplumber:from joblib import Parallel, delayed import pdfplumber import pandas as pd def plumber_parsepage(pdf_path: str, idx_page: int = 0, ) -> pd.DataFrame: with pdfplumber.open(pdf_path) as pdf: page = pdf.pages[idx_page] df_page = pd.DataFrame(page.dedupe_chars().extract_words(keep_blank_chars=True)) return df_page with pdfplumber.open(pdf_file_or_path) as pdf: num_page = len(pdf.pages) result = Parallel(-1)(delayed(plumber_parsepage)(pdf_file_or_path, i) for i in range(num_page))
However, pdfplumber.open() itself costs too much time if there are 100 or more pages in the pdf file.
So the pickle implementation is really useful!
i know... my solution is split large files first and multicore and after put together and small files one on each core
you can try if you have windows ;)
https://github.com/kalle07/parsing
