Jeremy Wiebe comments

Results 6 comments of


                                            Jeremy Wiebe

Issues with serialization on persistance

Thanks for this. I don't believe any of us active contributors have been very focused on performance tuning, and thus we haven't really spent time ensuring capabilities like persistence are...

Loading ARC files produces record size errror

When the next version of webarchive-commons (1.1.7) is released it will fix this issue.

use WET files from CommonCrawl

RecordLoader.loadArchives() only supports ARC and WARC files. We use the webarchive-commons library for processing these, which does not have WET support. On Tue, Sep 27, 2016 at 12:06 PM, David...

use WET files from CommonCrawl

I'll leave it to @lintool and @ianmilligan1 to comment on this feature's desirability. I would suggest that if you do decide to implement a WET reader, which seems pretty straightforward,...

Heap space!! java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:3332)

@ruebot, you say: > Upon investigation, this does not seem to be a problem with a large ARC/WARC or large collection. Can you clarify this? Are you saying that jobs...

Heap space!! java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:3332)

There's a cruel irony in the fact that mere hours before @ruebot's comment I deleted all my old memory profiles from tests I ran trying to understand this issue a...