Honor brew prefix when hacking linker path

[oggy]

This package was failing to install for me because I'm using a custom brew prefix. We already honor the prefix further down, so we should do it here too.