applications fail with connection refused.
Run zathura from the terminal. Errors out with this:
No protocol specified
Unable to init server: Could not connect: Connection refused
(zathura:11): Gtk-WARNING **: 11:00:51.082: cannot open display: :1
Removed Xauthority and Xinitrc variables, now zathura works, but firefox does not. Firefox fails with the same exact error message without Gtk warnings
No protocol specified is the X11 protocol version of Permission denied and so would be related to the xauthority environment (and xauth).
This would be something specific to your setup and perhaps how you have manage xauth. You will have to look into these aspects as there isn't anything that sx specifically does, besides adding and removing its own authorisation entry, to manage these permissions.
Yes, firefox is run through xephyr, is it not supported by sx?
By design sx uses the tty number as the value for DISPLAY and so makes starting nested servers difficult because they aren't executed from a /dev/tty context.
Instead, to give your user access under xephyr, you can use xhost +si:localuser:"$USER". See Xsecurity(7).
Other alternatives are sticking with startx or xinit as they provide access to the xserver command-line or modifying sx to do what you want. (It's intended to be illustrative and small enough to edit.)