[Work half done, requesting assistance]: Identifying controller as a joycon

[AlexIII]

Feature Description

There's an old branch experiment/identify-as-joycon in the repo which was last updated 1.5 years ago. We had a game night coming and no spare joycons, so I've hustly rebased some essential parts of that branch onto the current master which enabled us to use ps4 controller as a joycon. It works and we had an absolute blast playing, but there's remain some problems which are to be addressed before it could become a pull request.

Critical problems:

• Motion controls need to be translated in order to compensate for joycon orientation.

Not-so-critical problems:

• The console offers to update the emulated 'joycon' firmware on controller connect. I understand it comes from this function not reporting the desired fw version.

https://github.com/ndeadly/MissionControl/blob/540d1fdb23d02cd6d3adae1adfb76bceac671b3e/mc_mitm/source/controllers/emulated_switch_controller.cpp#L278

I've tried to rebase commits f400a206e4b and fbad1e4e236 but not only this doesn't help, after the reported version change, suddenly the motion controls stop working altogether! No idea what's going on here.

Apart from these two everything else seem to be working fine. I really want to finish this, but I need advice on what to do with these problems. I've never touched switch homebrew scene before, my experience is like literally 4 hours, please excuse me if I missed something that is considered plain obvious.

Here's my branch and diff

Rationale

Mario Party refuses to work with emulated pro-controller. It exclusively accepts joycons only. Maybe there's other games like that, I don't know.

Additional Details

No response

[ndeadly]

Hey, sorry I've been away on holidays and haven't had a chance to respond until now.

I haven't touched this branch in a while because there are some outstanding issues that I hadn't found a solution to and I had more important things to work on. Seems you already stumbled across the main one (joycon motion controls). If this interests you and you'd be willing to put in the time to resolve the remaining problems, I'd be grateful, as would many others who have asked about it.

Motion controls need to be translated in order to compensate for joycon orientation.

I revisited this branch at the beginning of last year when I added support for motion controls. I found that my Joycon motion data was somehow different to that of a Pro Controller/what has been documented, even though the documentation suggests they should be the same. I'm wondering whether a new format has been introduced at some point. @Banz99 and myself spent a bit of time trying to figure it out but ultimately benched it to work on other things. I have some python scripts I wrote to parse and explore some recorded packets if you'd like to pick up where I left off.

The console offers to update the emulated 'joycon' firmware on controller connect. I understand it comes from this function not reporting the desired fw version.

This issue is actually caused by incorrect joycon MCU data being reported by this function

https://github.com/ndeadly/MissionControl/blob/540d1fdb23d02cd6d3adae1adfb76bceac671b3e/mc_mitm/source/controllers/emulated_switch_controller.cpp#L438-L454

The current code is reporting the data from a Pro Controller. I thought I made a commit to add the Joycon data. Maybe it got stashed away somewhere and lost. In addition to the firmware version in the function you mentioned, the MCU data from a Joycon on the current firmware should be added.

The other issue I encountered was that I couldn't find a way to force the orientation of the controller to horizontal. Various actions could switch the controller between vertical and horizontal, ~~but from memory the functions that can query the orientation don't work from a sysmodule context, so there was no way to detect the switch. This would also need to be resolved somehow.~~ @Banz99 reminded me that we did figure that part out in the end, it just required constant checking of the status every packet, which may or may not cause performance problems.

If you'd like to work on this and want to discuss implementation details further, it would probably be best to contact me directly on discord @ ndeadly#5317

[ndeadly]

Some updated Joycon data logged from my controller

fw_major = 0x04 fw_minor = 0x19

mcu data:

0x01, 0x00, 0xff, 0x00, 0x08, 0x00, 0x1b, 0x06,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0xf6

[AlexIII]

Thank you for the answer. Before reading your explanation I was under impression that everything was just about to work, like only a minor fix was needed. But now it seems like it would require further reverse engineering and I don't feel confident I can take up on a task like that. Though, I still wish to try and poke around a bit more. If I may, I have a few more questions.

1. Can you recommend some testing util for Switch that shows data received from the motion sensor(s)? Like, xyz values and graphs. Right now every time I want to test motion controls I have to start Mario Party which takes a lot of time.
2. In your experiments, were motion controls registered by a game at all (in emulated joycon mode)? In my case motion controls work in Mario Party mini-games, but the axes are horribly messed up: rotated and inverted. Still, with trial and error, it is possible to find such controller orientation in which you can play the game.
3. I've tried to modify HandleHidCommandMcuWrite() and fw_ver in HandleHidCommandGetDeviceInfo as you've suggested, it helped to get rid off the update notification. However, as I mentioned earlier, if I change reported fw_ver to anything other than {0x03, 0x48} (Pro Controller) the motion controls stop to being registered by the game at all. Did you experience anything similar?

[Banz99]

Thank you for the answer. Before reading your explanation I was under impression that everything was just about to work, like only a minor fix was needed. But now it seems like it would require further reverse engineering and I don't feel confident I can take up on a task like that. Though, I still wish to try and poke around a bit more. If I may, I have a few more questions.

1. Can you recommend some testing util for Switch that shows data received from the motion sensor(s)? Like, xyz values and graphs. Right now every time I want to test motion controls I have to start Mario Party which takes a lot of time.
2. In your experiments, were motion controls registered by a game at all (in emulated joycon mode)? In my case motion controls work in Mario Party mini-games, but the axes are horribly messed up: rotated and inverted. Still, with trial and error, it is possible to find such controller orientation in which you can play the game.
3. I've tried to modify HandleHidCommandMcuWrite() and fw_ver in HandleHidCommandGetDeviceInfo as you've suggested, it helped to get rid off the update notification. However, as I mentioned earlier, if I change reported fw_ver to anything other than {0x03, 0x48} (Pro Controller) the motion controls stop to being registered by the game at all. Did you experience anything similar?

I'm very interested in what happens in your last two points. Based on our observation the axes were completely nonsensical as if the bit packing wasn't coherent to a pro controller at all, not just rotated and inverted (suggesting some non byte data subdivision or some quaternions shenanigans). I personally tested joycons motion in skyward sword hd, (can't remember if ndeadly did as well and with what), but maybe super mario party being an older title and your fw reporting lower versions doesn't activate this undocumented mode? Might be worth investigating. We just put our research on hold given there were other priorities and we were left completely in the dark. Feel free to join the discord server, your contribution might prove very useful.

[ndeadly]

But now it seems like it would require further reverse engineering and I don't feel confident I can take up on a task like that.

I don't think it necessarily needs to involve static analysis of the binary, if that's what you're thinking. I was just prodding around in some packets I recorded and plotting the data as a timeseries. I could see that there are definitely some signals in there, but things are a bit scrambled. My thinking was either that the data was packed differently into the available space, or even a different representation (eg, quaternions or rotation matrices) was being used. If you're comfortable with manipulating binary data that's probably the only skill you would need.

Can you recommend some testing util for Switch that shows data received from the motion sensor(s)? Like, xyz values and graphs. Right now every time I want to test motion controls I have to start Mario Party which takes a lot of time

Unfortunately I don't have anything great for this either. I thought about writing something to do it, but it was going to be too much trouble given the lack of higher level graphics libraries available for the console. When @Banz99 and I were testing, we resorted to either logging the controller input packets and plotting on PC as mentioned above, or using a modified version of the libnx opengl "lenny" example that rotates the geometry according to the P1 motion controls as a kind of sanity check to see whether the motion data was anywhere near what we would expect to see. I agree that having to boot up a game is pretty tedious, which is why I have shyed away from features involving motion controls and rumble when there are hardware support issues to tackle.

In your experiments, were motion controls registered by a game at all (in emulated joycon mode)? In my case motion controls work in Mario Party mini-games, but the axes are horribly messed up: rotated and inverted. Still, with trial and error, it is possible to find such controller orientation in which you can play the game.

When I initially messed with this, I didn't support motion controls. I can't remember exactly, but I think it was only @Banz99 who looked at the behaviour involving motion controls. I was mostly dealing with the "scientific" side of acquiring data recordings and producing some plots. From memory, the accelerometer data is fine and it's just the gyro that is in this messed up format. The axes would still need to be rotated, but it's possible some games can appear to work more or less fine if they're more concerned with the accelerometer data. It's also possible that some games are initialising the motion controls in different modes, which could explain this undocumented format we're seeing.

I've tried to modify HandleHidCommandMcuWrite() and fw_ver in HandleHidCommandGetDeviceInfo as you've suggested, it helped to get rid off the update notification. However, as I mentioned earlier, if I change reported fw_ver to anything other than {0x03, 0x48} (Pro Controller) the motion controls stop to being registered by the game at all. Did you experience anything similar?

Can't say I have. I'm pretty sure I was swapping in the latest Joycon firmware version without issue at the time I made that branch. But then again, I didn't support motion controls at the time. I guess this is uncharted territory.

[AlexIII]

I now moved to Mario Kart for testing motion controls and found that with the following mapping of IMU data everything seems to work (at least in this game)

right joycon
x := -y
y := -x
z := -z

left joycon
x := y
y := -x
z := z

I've updated my branch accordingly.

Still, I cannot make sense why motion controls stop responding when I send correct joycon fw version. For now I'm leaving it as it is (i.e. reporting pro controller fw version), since it seems to function fine and the pesky update message is only a minor inconvenience really.

[brendongl]

I know this is pretty niche, but can we get way to register pro controllers as joy-cons WITHOUT motion data for now?

There is only 1 game I know of which it would be super useful for. And that is Cluckpuck 99. It only allows party mode with joy-cons (don't ask me why) and doesn't use motion controls. There may be others without motion controls, but this would be a godsend if the everything else is already implemented but you didn't want to push this further since most people want it for motion control data.

[AlexIII]

@brendongl You can try out the branch I published. Seem like this issue is not popular enough to warrant a merge into the main build =)

[brendongl]

I am a noob and have no idea how to download a fork/branch. Any guidance would be appreciated!

[AlexIII]

I've uploaded the binaries and updated the readme file. Check it out, should be relatively easy to install now. Let me know if there are problems.