PBKDF2 formats: Support output dkLen other than multiples of native hash size (even shorter than it)

[magnumripper]

https://www.openwall.com/lists/john-users/2020/11/27/9

Since our current formats seem to be buggy (they accept such hashes but fails to crack them), this is fairly high prio.

I picture we could accept any length of input hash, down to say 48 bits. Should be trivial to implement.

[magnumripper]

PBKDF2-HMAC-SHA1 has a test vector of $pbkdf2-hmac-sha1$1000.fd11cde0.27de197171e6d49fc5f55c9ef06c0d8751cd7250 with password "3956". I tried simply dropping the last few bytes from it - it was still accepted but failed to crack. Prefered behavior is accept, crack, write full hash to pot.

[magnumripper]

In https://github.com/openwall/john/issues/4991#issuecomment-1010842963, @rbsec wrote:

389-ds uses PBKDF2-HMAC-SHA256 with a 64 byte salt and a 256 byte key length (their source code and a python implementation) - so it would be definitely be useful to support longer key lengths.

That's off topic for #4991 but on topic here. Apparently Jim's unifying efforts ended up with some regression. In pbkdf2_hmac_sha256_fmt_plug.c there's this:

#define MAX_CIPHERTEXT_LENGTH   1024 /* Bump this and code will adopt */

...but that macro is no longer used (so should be removed). In pbkdf2_hmac_common.h there's this now:

#define PBKDF2_SHA256_BINARY_SIZE           32
(...)
#define PBKDF2_SHA256_MAX_BINARY_SIZE       (4 * PBKDF2_SHA256_BINARY_SIZE)

I take it our current format supports max. 128 bytes.

OTOH any longer hash could simply be truncated to 128 bytes (or even 32 bytes) and would (due to the nature of PBKDF2) crack reliably.

[rbsec]

@magnumripper apologies - I've deleted my comment from the other thread - thanks for moving it here.

Truncating the hashes to 32 bytes seems to work fine, and I'm having to mangle their format anyway (390-ds uses a strange format starting with {PBKDF2_SHA256}, and then with the iterations, salt and output all base64 encoded), so cutting them off at 32 bytes is a nice simple fix.

I couldn't get John to accept anything longer than 32 bytes - but that doesn't really matter as I can crack the shorter versions.

Thanks