Video mode change sometimes gives screen corruption or occasionally crash

[rakslice]

When you change the video mode (colour depth or resolution) in the mac Monitors control panel, sometimes the video mode change works, sometimes you get a corrupt video situation where it's drawing the same stuff multiple times at different sizes with colour channel mixups and such.

I modified the basiliskii at the final stage update_display_static* that blits into the surface pixels to do some memset 0 bars in specific locations instead right into the surface pixels, to try to take basiliskii's blitting out of the equation, and sure enough when the video mode change works those are as expected and when it goes badly those get scaled and positioned incorrectly too.

I get essentially the same output from libSDL for the video mode changes that work as I get for the mode changes that don't work:

01-16 02:33:20.973 15444 15487 V libSDL  : calling SDL_SetVideoMode(1366, 768, 32, 1)
01-16 02:33:20.973 15444 15484 I libSDL  : SDL_SetVideoMode(): application requested mode 1366x768 OpenGL 0 HW 1 BPP 24
01-16 02:33:20.973 15444 15484 I libSDL  : SDL_SetVideoMode(): application requested hardware video mode - forcing software video mode
01-16 02:33:20.973 15444 15484 E libSDL  : ERROR: Invalid window
01-16 02:33:20.973 15444 15484 E libSDL  : ERROR: Setting the swap interval is not supported
01-16 02:33:20.973 15444 15484 E libSDL  : ERROR: Getting the swap interval is not supported
01-16 02:33:20.973 15444 15484 E libSDL  : ERROR: GL_GetAttribute not supported
01-16 02:33:20.976 15444 15487 V libSDL  : SDL_SetVideoMode(): Requested mode: 1366x768x32, obtained mode 1366x768x24
01-16 02:33:20.978 15444 15487 V libSDL  : SDL_SetVideoMode(): returning surface 0xe5918e40

[rakslice]

tl;dr: it's memory corruption.

For its addressing modes where the mac memory is at a fixed offset in host system memory (see TECH), basiliskii needs the framebuffer to stay at a known host memory address, so they reallocate it on mode changes with mmap() with MAP_FIXED. This is extremely fraught because it defeats normal safeguards against whatever allocator you've got around not having its memory allocations happily overlapped.

Even though there's no reason to have that for the "virtual" the-mac-ram-is-just-byte-arrays addressing mode we're operating in here, disabling that was not implemented for SDL.

In one case of weird artifacts, switching to "Millions" at 1366x768, I happened to notice that the addresses of the framebuffer and the_buffer_copy (the one used for dirty rectangles figuring) were only 5 hex digits different, which isn't enough space between them for ~0x300000 bytes of framebuffer -- they were actually overlapping, which explains basically all the visual nonsense.

[rakslice]

To give an idea what it looks like for future reference:

[rakslice]

Since it might be relevant at some point, I'm running on a Fire HD 10 (2019) which has an MT8183, which is a 64-bit capable SoC but the Android OS that Amazon provides on it is 32-bit (so this is an armv8l system basically)