return &mut str from encode_hex

[elichai]

Add utf8 checks in debug mode, and return &mut str from hex_encode so the caller can use the hex as str without needing to do the utf8 conversion themselves.