TolikPylypchuk
Press/drag fails for some macOS apps
Some macOS apps seem to monitor mouse events in some way that SharpHook doesn't cover. Often, it is click-drag operations that don't work.
Bitwig is such an example. If I use ShatrpHook to position the mouse cursor over a Mute button in the app and press the left mouse button, it works as expected.
But if I move the mouse cursor to a volume fader, click and hold the left mouse button and move the mouse cursor, it does not work. It should "grab" the volume fader and move it, which it does when I do it manually but not when using SharpHook. It registers the left mouse button press (evident since the object is being selected), but nothing else happens when the mouse cursor moves; the volume fader handle isn't brought along with it. The app behaves as if I press the mouse button and then keep the mouse stationary.
What could cause such a behavior?
Hello! Thanks for posting this issue! I will look into it. macOS has some weird specifics about mouse events, this is probably happening because some event flag hasn't been set by libuiohook.
Could you please also provide a minimal reproducible example so that it's easier to debug?
I realize that having example code for debugging would simplify things a lot, and I'd love to supply that if I had something like that.
My application is a Stream Deck plugin, and as such, it's not a self-sustained executable and cannot be run outside of the Stream Deck environment.
To give you a minimal reproducible example, I must start from scratch and invent something new, and the question is if it is quicker if I do it than if you do it.
I'm very interested in solving this problem and will help however I can. I'm just not sure that creating a new solution from scratch is the best way to help.
I can look at it, but I can't say when I can have a new solution to share.
Från: Tolik Pylypchuk @.> Skickat: den 6 augusti 2024 20:48 Till: TolikPylypchuk/SharpHook @.> Kopia: Gunnar Carlson @.>; Author @.> Ämne: Re: [TolikPylypchuk/SharpHook] Press/drag fails for some macOS apps (Issue #116)
Could you please also provide a minimal reproducible example so that it's easier to debug?
— Reply to this email directly, view it on GitHubhttps://github.com/TolikPylypchuk/SharpHook/issues/116#issuecomment-2271929133, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AKMO447WZAD2IZXR67SFBFLZQEK6BAVCNFSM6AAAAABL6FSPUCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENZRHEZDSMJTGM. You are receiving this because you authored the thread.Message ID: @.***>
Yeah, no problem, I will try to reproduce it on my own then. If I can't, then we'll have to think of something.
It was easier than I thought to extract the things from my code. With the following code, I can replicate the issue on my mac.
using SharpHook;
using SharpHook.Native;
namespace MouseMoveTest
{
internal class Program
{
//TODO: Set startX and startY to a suitable screen location
static short startX = 466;
static short startY = 280;
static short stepSize = -2;
static int stepDelay = 50;
static short currentX;
static short currentY;
static EventSimulator? simulator;
static void Main(string[] args)
{
simulator = new EventSimulator();
currentX = startX;
currentY = startY;
// Move to start position
MoveMouse(0, 0);
// Press left mouse button
PressMouseButton();
// Move mouse left for 1 second
for (int i = 0; i < 20; i++)
{
MoveMouse(stepSize, 0);
Thread.Sleep(stepDelay);
}
// release mouse button
ReleaseMouseButton();
}
static void MoveMouse(short offsetX, short offsetY)
{
simulator?.SimulateMouseMovement((short)(currentX + offsetX), (short)(currentY + offsetY));
currentX += offsetX;
currentY += offsetY;
}
static void PressMouseButton()
{
simulator?.SimulateMousePress(currentX, currentY, SharpHook.Native.MouseButton.Button1);
}
static void ReleaseMouseButton()
{
simulator?.SimulateMouseRelease(currentX, currentY, SharpHook.Native.MouseButton.Button1);
}
}
}
I've tried running this code on macOS Sonoma, and everything works fine. I've tried this out with Visual Studio Code, Safari, and JetBrains Rider, and text was selected in these applications, meaning the mouse cursor really was dragged.
I've also checked the code of libuiohook - it does simulate kCGEvent{Left,Right,Other}MouseDragged if a button was perviously clicked.
Could it be a problem with the target application? Maybe, it doesn't handle simulated drag events correctly? Sorry, I don't have the ability to install Bigwig as it appears to be a huge application. Do you have other examples of applications that misbehave when mouse dragging is simulated? If not, then unfortunately, it appears that there's nothing I can really do here.
I don't know how you can see Bitwig as a huge app. It uses 1.75 GB RAM on my Mac.
I know there are other apps with similar problems but I haven't the details. I have asked to get more info about that, but I know they are advanced graphical apps so they will probably be larger than Bitwig.
Could it be a problem with the target application? Maybe, it doesn't handle simulated drag events correctly?
Those are interesting questions. I didn't think an app could detect that it was a simulated event? If an app reacts to physical mouse movement but not to simulated mouse movement, does that mean that it uses some technology that is not possible to simulate?
Ok, my choice of words was poor - I meant that Bitwig looks complex.
Regarding the problem being in the target app - here's an example. SharpHook can simulate text entry, but on macOS applications are not required to handle such events, meaning that text entry simulation may simply not work in some apps and there's nothing SharpHook can do about it. Maybe, it's a similar situation here where Bitwig doesn't handle drag events, instead it handles mouse click + mouse move? This is just an assumption though.
Could you please try to reproduce this in other applications like browsers or text editors? If you see that the mouse really is dragged (e.g. text is selected) then I think SharpHook simulates this corectly.
As I mentioned in my initial report, this problem only affects some apps. I thought it affected more apps, but it turns out that Bitwig may be the only one. For the majority of apps, it works as expected.
So, the verdict is that if it doesn't work, it doesn't work, and there's nothing we can do about it?
Yes, I believe that there's nothing we can currently do. Maybe, you can contact Bitwig and submit a bug report for it?
I'm going to close this issue, but we can continue the discussion, and I will reopen it if some new info comes up about this bug.
I installed Karabiner-EventViewer on my Mac to see if I could get some more information on what was going on. It doesn't track mouse movements, but it does log mouse buttons.
When using the physical mouse, the mouse button presses (and releases) were logged as expected. When using SharpHook, nothing was logged.
I don't know what I want to say with this, other than there are clearly different ways to detect mouse events on a Mac.
Hm, ok, I will take a more detailed look at this. Thanks for checking this!
I've tried investigating this issue using Karabiner-EventViewer, and yes, it doesn't show any simulated mouse events. I don't know what can be changed about simulating mouse events on macOS, and I didn't find anything that can be useful. Of course, I pretty much have surface-level knowledge about the way macOS simulates events so I may have missed something.
Currently I don't think I can do much about this issue, but I will keep this open for a while - maybe something will come up.
I'm going to close this issue due to inactivity and not being able to do anything about it. We can continue the discussion anyway, and I will reopen the issue if new information comes up.
Some of my users have been reporting this issue too, and I experience it in my VM as well. One thing we all have in common is that we can't drag files around in Finder only by simulating a MouseButton press with SharpHook., while pressing down a physical mouse button does let us drag files around.
I've tried running this code on macOS Sonoma, and everything works fine. I've tried this out with Visual Studio Code, Safari, and JetBrains Rider, and text was selected in these applications, meaning the mouse cursor really was dragged.
@TolikPylypchuk I thought that might be worth sharing, in case you never found a way to replicate the issue without having to install professional music production software.
Additionally, resizing a window does not animate the resizing when the mouse press was simulated, while a real press does animate the resizing in real-time. Interestingly though, the window does get resized all the same once a mouse release is simulated.
Hi @BlackCoyote! Thanks for the note! I've reopened the issue and will look into it.
Is it possible that what the behaviour I expected is fundamentally not possible?
The behaviour I expected is that if i simulate a mouse press with SharpHook, and then move my physical mouse, it would be considered dragging, but it sort of looks like every mouse move event needs to include a flag for whether the mouse is being dragged, and since the physical mouse button was never pressed the event sent by the physical mouse never have the dragging flag.
Does that mean it is not possible, or would there be a way to intercept mouse moves and add flags?
I understand this is fundamentally a bit different from the original issue where the mouse movement was also being simulated by SharpHook, apologies for that.
@BlackCoyote yeah, your issue is different from the original issue and I don't think it can be fixed.
The problem is that macOS has different event types for mouse moves and mouse drags (for some reason). When simulating events, libuiohook manages these events itself. Basically, when simulating a mouse button press, it saves the currently pressed button, and when simulating mouse move, it checks the saved currently pressed button. If there is one, it simulates dragging. Otherwise, it simulates moving. I guess, when moving a physical cursor, macOS ignores the simulated mouse button press event which is weird, but I don't know if we can do anything about it.
Conversely, if you press a physical mouse button, and then simulate moving the mouse, libuiohook won't know that it actually needs to drag the mouse, and will move it instead. There are 2 different mouse event types in libuiohook and SharpHook - MouseMoved and MouseDragged, but they are only different in global hooks. They work exactly the same when simulating events, libuiohook just doesn't care about the difference, it manages these event types on its own.
So I'm going to close this issue again, sorry.