aut

aut copied to clipboard

Output filtered data to WARC Format

10

Users may desire outputs in WARC format after filtering their RDD[ArchiveRecord].

greebie

Heap space!! java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:3332)

21

**Describe the bug** On a number FDLP and Stanford collections, we run into this space heap space error, and it kills the Spark job. Upon investigation, this does not seem...

ruebot

Bump jsoup from 1.14.2 to 1.15.3

1

Bumps [jsoup](https://github.com/jhy/jsoup) from 1.14.2 to 1.15.3. Release notes Sourced from jsoup's releases. jsoup 1.15.3 jsoup 1.15.3 is out now, and includes a security fix for potential XSS attacks, along with...

dependabot[bot]

`s3a` URLs don't work as in documentation

1

EDIT: this helped, the doc may need to be updated: ``` sc.hadoopConfiguration.set("fs.defaultFS", "s3a://commoncrawl/") ``` **Describe the bug** According to the docs, `aut` should be able to read data from `s3a`...

acruise

Bumps [org.apache.tika:tika-core](https://github.com/apache/tika) from 1.23 to 3.2.2. Changelog Sourced from org.apache.tika:tika-core's changelog. Release 4.0.0-BETA1 - ??? BREAKING CHANGES Moved towards default json based configuration (TIKA-4544 and many others). tika-pipes implementation modules...

dependabot[bot]

Replace Boilerpipe functionality with more modern Readability clone

1

**Problem** Class ExtractBoilerpipeText doesn't fully do what it purports to; ie. it sometimes leaves (often large) portions of eg. header and comment thread text in the output. [Boilerpipe](https://github.com/kohlschutter/boilerpipe) was last...

mjsuhonos