vinaygoel
java io error, likely incorrect versions
There is a lot of setup for the environment for these exercises. When I'm trying to run through things, I'm getting a Java IO error that is pretty vague. I'm thinking I'm running a version of something that doesn't mix well with a version of something else.
Could you provide a list of requirements along with their versions?
- Docker
- Hadoop
- Java
- Pig
- Python?
- And anything else I might be missing
This is an off request, but I wanted to know if different methods to generate warc files could produce any inconsistencies. I'm trying your examples with producing warc files with wget as opposed to the provided downloads.
Regardless, thanks for these tools. If I can get my bug resolved, I'll post the results for anyone else who might run into this too.
For future reference, I managed to get it working with the following versions:
- Java 8 (JDK 8u222)
- Webarchive commons 1.1.9
- Hadoop 2.9.2
- Apache pig 0.16.0
- Jython 2.5.4-rc1
- JYSON 1.0.2
- Datafu 0.0.10
- Elephant bird 4.17
- Piggybank 0.16.0
The most important thing is to run Java 8 and a version of apache pig <= 0.16.0
