pylinkvalidator
pylinkvalidator copied to clipboard
Fix issue with Base64 encoding for basic auth
Simple fix to resolve the newline encoded at the end of the basic auth string.
Symptom:
ERROR Crawled 1 urls with 1 error(s) in 0.01 seconds
Start URL(s): http://foobar.foo
error (<type 'exceptions.ValueError'>): Invalid header value 'Basic YWlyc291cmNlOmJsYWhibGFo\n': http://foobar.foo
Cause:
As others have reported (e.g. here there's a problem with using the encode methods for Base64 encoding as they append a newline character, which results in an invalid header. This allegedly started to occur after a security fix made to Python.
Fix:
Replace the Base64 encode method with b64encode instead - this doesn't append a newline.