openwall
We lack developer documentation
What we need to document:
- How to implement a new format.
- Which format to pick for a template
- How to advance from a simple implementation to one with SIMD support
- When and how to implement thin formats
(And I am sure lots of other stuff)
- How the plugin "system" works (and when to use / not use it)
- How to update the autoconf stuff
-
split()vsprepare()
Openwall's wiki is almost dead and that might be partly because it's horrible to use: It's the worst wiki [the software] I've encountered as a writer.
On the other hand, this Github wiki is, in my opinion, wonderful. It's simple and intuitive. I have some pages here mostly for my own notes.
https://github.com/magnumripper/JohnTheRipper/wiki/Developer-Help-Documentation
I have added some place holder pages, listing all items from this post. I really would like to be able to control what shows up in that 'Pages' sidebar. It will get HEAVILY cluttered and unusable.
The place holders are annoying. Please either drop them for now, or fill them in.
Another question: How to decide whether or not an OpenCL format should be OMP or not. E.g., rar-opencl is OMP, some other formats are OMP as well, but (at least for my Haswell with beignet) I don't really know why.
It's pretty obvious: For example, the CUDA version of WPAPSK does significant post-processing on CPU, so should (and does AFAIK) use OpenMP for that. The OpenCL version does it all on GPU, so there's simply nowhere you could put an OpenMP pragma.
OK, then I might need a real GPU instead of Intel's Haswell to see the benefit of using OpenMP for rar-opencl. For me. there's no difference running a non-OMP version, and the CPU is still mostly idle.
You got it backwards. Again, the OpenCL version of WPA does not and can not utilize OpenMP, because all work is done on GPU.
NO, I didn't get it backwards. I didn't mention WPA at all. I just discussed rar-opencl. May be here the CPU has work to do, but at least with my Haswell, there's so little work to do for the CPU that the CPU is idle most of the time, even in a --disable-openmp build.
Oh, my bad. The thing with RAR is we have very good early rejection nowadays. Whenever that code fails to reject early, we need to do, worst case, an AES decryption of a whole file in the archive. That file is potentially huge.
Can that AES decryption of a whole file benefit from OpenCL? I just ask because I intend to run rar-opencl in parallel ro rar (where I would use default OMP_NUM_THREADS on CPU), and I am afraid that enabling OpenCL for GPU might hurt perfrmance rather than help.
The --test benchmarks just indicate that I can combine --format=rar and --format=rar-opencl, and get more or less the same speeds as when I run --format=rar and --format=rar-opencl individually.
AES can be done on GPU but I don't think it's worth it. But I'm pretty sure it will be worthwhile to run CPU and GPU in parallel, because of the mentioned early rejection stuff.
We should also document the use of error() and pexit(). And why to use them (there are very good reasons) instead of exit() when possible.
We also have a doc/HACKING.md file now. I am looking for suggestions about topics which could be added to this file.
