SlicerT UI update

[DanielKauss]

Most of the credit goes to @RoxasKH who designed the new UI.

Fixes #7060. This PR updates the existing SlicerT UI to make it bigger and resizable. Also has some small bug fixes and changes to the plugin window related files to allow them to be resizable. Here is the new UI:

The new minimum size is 516x400, and it can be resized without limits.

Changes outside SlicerT

I started this PR a few months ago, so these changes may have been implemented already somewhere else.

The pluginView class now has the functions setResizable and isResizable to allow plugins to make their window resizable. This gets checked in the InstrumentTrackWindow to set a few window flags.

Also changed a few things about how effects, LFO and arpeggio act when resized. Again, this might not be needed anymore.

SlicerT bugfixes

• Fixed active note highlight not turning off when note stops
• Double clicking on the seeker to create a slice
• Note snapping integer division rounding error

Sorry for the huge amount of random numbers in the code, but it's mostly just correcting the UI elements offsets. Anyway, if there is anything I should fix/change please let me know, thank you.

[michaelgregorius]

@DanielKauss, I only did a very quick review but overall I like that this PR lays the base to free the instruments from their confined and tiny windows. :smiley:

Another good candidate for a resizable and large instrument window would be AudioFileProcessor.

Did you also test the rendering performance of the wave view with very large samples? It looks like you are caching the waveform in a QPixmap anyway though.

[DanielKauss]

Did you also test the rendering performance of the wave view with very large samples? It looks like you are caching the waveform in a QPixmap anyway though.

Did this already on the original PR. The waveform only gets drawn if the seeker changes, and a while ago I also optimized the waveform drawing code, although I don't know if that is still in the codebase. Also tested it again just now and seems fine.

[michaelgregorius]

I have just tested the PR locally and noticed that resizing is very slow and sluggish after having loaded a sample that's 15 seconds long. I guess it will be even worse with longer samples. Based on the performance I get the impression that the code simply renders every sample into the caching pixmap. This will then lead to problems as soon as the pixmap needs to be updated.

A 15 seconds long sample has around 720000 samples at a standard sample rate of 48 kHz. If the view is around 1000 pixels wide then this means that 720 samples are rendered on one pixel column.

@DanielKauss, if I remember correctly, the view for the samples in the sample track has some optimizations. Can you please check?

[DanielKauss]

I have just tested the PR locally and noticed that resizing is very slow and sluggish after having loaded a sample that's 15 seconds long. I guess it will be even worse with longer samples. Based on the performance I get the impression that the code simply renders every sample into the caching pixmap. This will then lead to problems as soon as the pixmap needs to be updated.

This is somewhat strange. I just tested with a 2 minute sample, and everything worked smoothly. It didn't update at the full 60 fps when resizing, only something like 25, which I think is acceptable for something the user won't do much. And this was with a 2 minute sample, which is exaggerated anyway. And yes, while resizing, the sample gets redrawn on every resizeEvent. But it also gets redrawn every time you move the seeker, and that runs smoothly.

A 15 seconds long sample has around 720000 samples at a standard sample rate of 48 kHz. If the view is around 1000 pixels wide then this means that 720 samples are rendered on one pixel column.

No, the waveform drawing function is optimized so that it only draws a maximum of 512 samples per pixel, and I think that is the worst case scenario.

@DanielKauss, if I remember correctly, the view for the samples in the sample track has some optimizations. Can you please check?

Doesn't look like it, seems like we both just use the SampleWaveform::visualize function. It seems like the sample track doesn't even cache the pixmap, and redraws it on every paintEvent, so it should be even slower.

Is the lag on resizing really that noticeable? I didn't change any of the drawing code from the original SlicerT, and there everything ran smoothly. If the lag is only minimal, and only present while resizing, I don't think it's worth optimizing for.

[michaelgregorius]

Is the lag on resizing really that noticeable? I didn't change any of the drawing code from the original SlicerT, and there everything ran smoothly. If the lag is only minimal, and only present while resizing, I don't think it's worth optimizing for.

Unfortunately for me it can go well into "several seconds" territory. I have recorded a resizing action here:

Bildschirmaufnahme_20240818_103949.webm

What's interesting is that at the beginning it's ok but that it gets slower and slower as the window gets larger. This might indicate that it is not so much the sample processing but rather the rendering that creates the problems.

[DanielKauss]

Unfortunately for me it can go well into "several seconds" territory. I have recorded a resizing action here:

Pretty strange. I have created a small patch to measure the timings. Can you please apply it to this branch and send me the resulting console output when you resize the window? If you don't have a build system set up or something I can send you an executable. Patch is here:

timing_patch_slicerT.patch.txt

For me, I get the following output, using a 2 minute sample and after resizing for around 5 minutes to check for memory leaks:

-------------- resize event start ----------------
Resize Pixmaps: 11
Draw seeker waveform: 6433
Draw editor waveform: 26822
Draw seeker UI: 592
Draw editor UI: 6577
Full resize Event: 40467

Time is in microseconds, and as you can see drawing the big waveform is indeed what takes up the most time. However, for me it still only takes 26 milliseconds, so no idea how you would get a multiple second long delay

What's interesting is that at the beginning it's ok but that it gets slower and slower as the window gets larger. This might indicate that it is not so much the sample processing but rather the rendering that creates the problems.

This makes sense, since we have to process ~512 samples per pixel-column, and with a wider resolution this just increases. So both the processing time and rendering have to increase.

I would be grateful if you could test this out for me, and maybe also provide me with some of your system details. Sorry if this is asking for too much, but thank you anyway.

[michaelgregorius]

@DanielKauss, I have applied the patch and have also added the current time at the start of the resize event to the output. It surprised me that the output indicated an update rate of around 20 Hz/frames per second, i.e. 0.05 seconds per resize event. The visual appearance was laggy like in the video above though.

Then it dawned on me that the differences we both observe in performance might be a "Wayland vs. X11" effect as I am running under Wayland. Testing everything under X11 gave a rather smooth output.

Attached you can find the output of a resizing action under Wayland and under X11: resizeTiming-wayland.zip resizeTiming-x11.zip

So the problem seems to be that the Wayland compositor runs at a certain frame rate, in my case 60 Hz, and that if an application is not able to render at that rate then some of its output is dropped. This is a consequence of Waylands "every frame is perfect" approach. This is likely also the reason that I see some updates at the beginning of the resize action but then only very few intermediate ones in between. All other frames are simply dropped and I see the previous rendering of the application.

So the problem statement must be corrected to: Resizing the SlicerT window with large samples is laggy under Wayland. Or put differently: LMMS must ensure that it can render at 60 Hz. Otherwise these effects might also show up in other places.

[DanielKauss]

So the problem statement must be corrected to: Resizing the SlicerT window with large samples is laggy under Wayland. Or put differently: LMMS must ensure that it can render at 60 Hz. Otherwise these effects might also show up in other places.

Wayland could definitely be the problem here. I tried switching to Wayland, but opening LMMS immediately crashes the whole WM (cinnamon). Wayland in general barely works on my system at all, and I have no experience with it either, so I can't really work on this.

Optimizing the UI even further to always render more than 60 fps seems unfeasible, especially when for the resize we have to redraw everything. The only band aid fix I can think of is to only redraw the window once the user has stopped resizing, but this would involve some kind of timer, similar to this SO question, and would look worse on all the other systems.

Anyway, I can't really work on this, so you should decide what we should do with this. However, this seems like an underlying issue in either Qt, Wayland or the compositor, so this might be difficult to work out.

[michaelgregorius]

@DanielKauss, I have created https://github.com/LMMS/lmms/issues/7462 to deal with the inefficient rendering. If Wayland becomes more and more commonplace it will make LMMS look bad if it drops so many frames in certain situations.

Other applications also manage to render large waveform views so it should be doable.

[DanielKauss]

I'll leave this here in case someone else does a review. Using Qt layouts for the resizing is a waste of time, and would only complicate the code even further. With the current method, I can place each object on the exact pixel that I want, where as layouts do not have that level of precision. Even if I did use layouts, all the hard coded numbers would have to be implemented as well, just in the form of offsets.

And it would only really affect around 12 lines: the knob and button move functions in the resize event, and the text drawing in the paint event.

As for performance, I would hope that doing a few additions is insubstantial in comparison to all the drawing stuff, which would not be affected at all if I switched to layouts.

And before this PR, SlicerT also had these values hard coded, just without taking into account the window size. The same is true for all the other plugins, they all move the knobs to hard coded positions without layouts (even something like Monstro) because it is way simpler to get perfect positions that way.

[michaelgregorius]

And before this PR, SlicerT also had these values hard coded, just without taking into account the window size. The same is true for all the other plugins, they all move the knobs to hard coded positions without layouts (even something like Monstro) because it is way simpler to get perfect positions that way.

The other plugins all have a set size and cannot be resized though. Layouts usually make life easier when working with resizable widgets.

However, I agree that adding layouts is not part of this PR because it already worked with hard-coded layouts before.

[Rossmaxx]

I'll slack on the layout for now, but would like to see #6992 (or atleast the bugfix) merged before this.

[DanielKauss]

I'll slack on the layout for now, but would like to see https://github.com/LMMS/lmms/pull/6992 (or atleast the bugfix) merged before this.

The bugfix was already merged separately in #7174.

[zonkmachine]

@DanielKauss This seem to have messed a bit with the LV2 layout. It mostly works though but in the case below a part of the gui has been cut off.

[zonkmachine]

I opened a separate issue for the layout regressions: https://github.com/LMMS/lmms/issues/7510

[sakertooth]

I'll leave this here in case someone else does a review. Using Qt layouts for the resizing is a waste of time, and would only complicate the code even further. With the current method, I can place each object on the exact pixel that I want, where as layouts do not have that level of precision. Even if I did use layouts, all the hard coded numbers would have to be implemented as well, just in the form of offsets.

And it would only really affect around 12 lines: the knob and button move functions in the resize event, and the text drawing in the paint event.

As for performance, I would hope that doing a few additions is insubstantial in comparison to all the drawing stuff, which would not be affected at all if I switched to layouts.

I wholeheartedly disagree. The whole point of using Qt layouts is that you do not have to worry about the precision yourself, because it will get it right for you anyways. There's a good chance that you wont get it right however, and something will feel misplaced in the arrangement of widgets. There were advancements to move away from using hard-coded positions, like in #6591, so I really don't like how we are taking two steps back here. I find this to be a lot more complicated than just using layouts.

I mean, just look at all the manual offset calculations done here, and when something changes (maybe we add a knob somewhere or something else that affects the widget arrangement), all of those numbers will be imprecise (assuming they aren't imprecise already) and will have to be changed. This is a major headache and we should not be making the problem bigger for ourselves.

You must remember that software is soft and malleable, which makes it very susceptible to change. Please take into consideration any changes that can be made in the future before making software decisions, even if you can't see the changes yourself in the moment.

And before this PR, SlicerT also had these values hard coded, just without taking into account the window size. The same is true for all the other plugins, they all move the knobs to hard coded positions without layouts (even something like Monstro) because it is way simpler to get perfect positions that way.

Just because other instruments still use hard coded positions does not mean that is how the codebase should be arranging its widgets. There are many things in the codebase that fall into the criteria of "this is the way it is done in the codebase, but its definitely not how we want to do things generally speaking".

It is far simpler to get perfect positioning using layouts, because all that needs to be done is to just add the widget to the layout, which lays out the child widgets precisely in the parent widget. If there's any more customization necessary to the layout afterwards, that is always an option.

[Rossmaxx]

@DanielKauss I am thinking of a compromise regarding the current glitches relating to effect view. How about reverting this PR partially, ie the resizeability but change the minimum size to something like 600 x 400 for slicer t alone?

@sakertooth is this fine with you too?

[sakertooth]

@DanielKauss I am thinking of a compromise regarding the current glitches relating to effect view. How about reverting this PR partially, ie the resizeability but change the minimum size to something like 600 x 400 for slicer t alone?

@sakertooth is this fine with you too?

I was already in the process of reverting this partially, so it's fine with me.

[DanielKauss]

I don't mind if you revert this, but you could also only revert the changes to the effect classes, it's only a few lines.

As for using layouts, if you have a good idea of how to implement them to actually make the code simpler, I would be happy to use them. However, I will once again say that layouts are unnecessary here and would only complicate the code further. There isn't really any need for dynamic positioning, since the only coordinates that change are the y coords of the lower components. Implementing layouts would only really "remove" the code that is in the resize event, which is just a few lines long. And then, the hardcoded offsets would have to be somewhere anyway, we would just be moving them around.

[sakertooth]

However, I will once again say that layouts are unnecessary here and would only complicate the code further. There isn't really any need for dynamic positioning, since the only coordinates that change are the y coords of the lower components. Implementing layouts would only really "remove" the code that is in the resize event, which is just a few lines long. And then, the hardcoded offsets would have to be somewhere anyway, we would just be moving them around.

How familiar are you with Qt layouts? It would effectively remove both the paintEvent and resizeEvent functions. Most of the UI would be specified in the constructor without any need for hardcoded offsets. And what I said about changes made to the UI still stands: It becomes increasingly difficult with since if you want to add a knob, the surrounding geometry may also have to be changed, which can easily be avoided when using layouts.

[michaelgregorius]

We should not throw away all the code of this PR as it prepares SlicerT for being resizable. And making the instrument windows resizable should definitively be a goal IMO.

As @DanielKauss has already noted it would suffice to revert the changes to the effect classes to fix the problems with the LV2 plugin.

To fix the problem with the "distorted" tabs it would suffice to remove the following line from the constructor in plugins/SlicerT/SlicerTView.cpp:

setResizable(true);

Once the problem with the instrument tabs is fixed the line can then simply be put back and SlicerT would be immediately resizable without having to recode everything or to do hunting in Git commits and messages.

Keeping much of the code would also be good for testing the resize functionality of the instrument tabs when it is being fixed.

[sakertooth]

We should not throw away all the code of this PR as it prepares SlicerT for being resizable. And making the instrument windows resizable should definitively be a goal IMO.

I disagree (but not completely). The thing I'm worried about is if this is the implementation for resizability we want. If for whatever reason it isn't, one way or another we would have to start over. There might be also prerequisites before we can safely make instrument windows bigger/resizable (e.g. things like the Envelope and LFO graphs may need to be painted using QPainter instead of using QPixmap's to avoid pixelation when resized).

I will do the partial revert that @michaelgregorius and @DanielKauss mentioned and see that's all that is needed to fix the regressions, anyhow.

[michaelgregorius]

We should not throw away all the code of this PR as it prepares SlicerT for being resizable. And making the instrument windows resizable should definitively be a goal IMO.

I disagree (but not completely). The thing I'm worried about is if this is the implementation for resizability we want. If for whatever reason it isn't, one way or another we would have to start over.

The simplest implementation would be to enable the resizing of the individual instrument view, e.g. the view that shows the waveform in SlicerT, and to have the other tabs at "fixed" sizes, i.e. such that they are in well spaced layouts but do not grow if the window becomes bigger for now.

From there on one could then think about other implementations. Some months ago I made a proposal for some changes which would already separate the individual instrument view from the other elements that are always the same anyway. Unfortunately, it was not met with great enthusiasm. You can find it here: https://github.com/LMMS/lmms/issues/2510#issuecomment-2032220170.

There might be also prerequisites before we can safely make instrument windows bigger/resizable (e.g. things like the Envelope and LFO graphs may need to be painted using QPainter instead of using QPixmap's to avoid pixelation when resized).

I agree. Having the text "Envelope" and "LFO" in the window that shows the envelope and the LFO waveform respectively is not that practical anyway.

I will do the partial revert that @michaelgregorius and @DanielKauss mentioned and see that's all that is needed to fix the regressions, anyhow.

:+1:

[sakertooth]

Did the partial revert as described, but the widget layout in SlicerT is incorrect now, so I have to fix this as well.

[JohannesLorenz]

I'm currently trying to do a fixup of #7512 which is a fixup of this PR.

Some things I would like to change about PluginView::isResizable:

1. make this a virtual boolean function and remove setResizable and m_isResizable. That way, it also matches EffectControlDialog::isResizable, which is also a virtual.
2. LMMS' PluginView design is weird: Both EffectView and InstrumentView inherit from PluginView, but while InstrumentView is the "whole" Instrument GUI (i.e. all custom knobs that the plugin designer wanted), EffectView is only the 3 common knobs in the Mixer window (Gate, Pan, etc) - and EffectControlDialog is the actual thing where plugin designers add their knobs etc. So, currently, we have EffectView and InstrumentView inherit PluginView (which does nothing else than inherit ModelView) and EffectControlDialog inherit ModelView. I would like to switch the inheritance of the two Effect classes: Let EffectView only inherit ModelView and let EffectControlDialog inherit PluginView.
3. The names of the effect classes could be switched/changed, too, since EffectView sounds like it does what InstrumentView does, but the above text says that this is totally wrong. This might be a bigger change though without functional improvement.

@michaelgregorius What do you think about my proposed changes?

[michaelgregorius]

@JohannesLorenz, I definitively agree with 1. Whether or not an effect can be resized depends on the implementation and therefore it should be a virtual function. Another argument is that such a function would indicate whether an effect would be resizable at all, i.e. it describes the underlying ability. It would then be up to the clients of that method to decide how to treat the result of isResizable, i.e. if they want to treat the effect view to be resizable or not. It does not make sense to have a method like setResizable. If the underlying implementation does not support resizing then it would not make sense to call setResizable(true).

I don't have a good overview of the instrument and effect inheritance hierarchy. But if I understand correctly, you find it confusing that InstrumentView describes the window that's shown in the MDI area whereas EffectView is the view which renders an effect in the effect rack. EffectControlDialog actually is the window that's shown in the MDI area. In that case I would agree and a renaming might make sense.

[sqrvrt]

@DanielKauss Could you please explain the reasoning behind putting size restrictions on subWin instead of ITW itself? It seems to me it's just hardcoding more values into it without any benefit.

[DanielKauss]

@DanielKauss Could you please explain the reasoning behind putting size restrictions on subWin instead of ITW itself? It seems to me it's just hardcoding more values into it without any benefit.

I honestly can't tell you why I chose the sub-window instead of the instrument track window, but if I remember correctly I copied the code from the AdjustTabSize function, which used to set the size of all tabs with:

w->setMinimumSize(INSTRUMENT_WIDTH - 4, INSTRUMENT_HEIGHT - 4 - 1);.

But instead of changing the size of all tabs individually, I changed the size of the sub-window with all tabs in it. And as you can see here, the hard-coded values that you say I have added without benefit were already in the code, just in the form above.

It's been a while, so I may not remember everything correctly, sorry.