rr-debugger
Support x86 Android
It would be great if rr could be used to debug Firefox for Android (or other Android applications).
Since Android can run on an x86 device or emulator, hopefully this is easier to do than supporting ARM.
cc @jrmuizel
The approach that seems most practical to me getting this to work is using Anbox. Anbox lets you run android apps in a container on regular Linux. Given this you would a single rr recording of all of the processes running in the container.
The big missing piece keeping this from working right now would be rr support for ashm and binder. Ashm seems like it wouldn't be too hard to support. Binder might be harder.
Interesting. So Firefox works in Anbox?
I just tried it on Debian 10, and yes, I am able to launch and load a website in both Firefox for Android and Firefox Preview in Anbox.
A native attempt to run Anbox under rr results in this:
$ rr record anbox.appmgr
rr: Saving execution to trace directory `/home/botond/.local/share/rr/anbox.appmgr-1'.
[FATAL /home/botond/builds/rr/src/AutoRemoteSyscalls.cc:517:check_syscall_result()]
(task 19499 (rec:19499) at time 4104)
-> Assertion `false' failed to hold. Syscall mmap failed with errno EACCES
Tail of trace dump:
{
real_time:45819.752171 global_time:4084, event:`SYSCALL: epoll_pwait' (state:ENTERING_SYSCALL) tid:19509, ticks:5163
rax:0xffffffffffffffda rbx:0xffffffff rcx:0xffffffffffffffff rdx:0x80 rsi:0x7eff67ffe740 rdi:0x4 rbp:0x7eff67ffed40 rsp:0x7eff67ffe700 r8:0x0 r9:0xc420150a58 r10:0xffffffff r11:0x246 r12:0x0 r13:0x5 r14:0x5 r15:0x5 rip:0x55964d54c9f0 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x119 fs_base:0x7eff67fff700 gs_base:0x0
}
{
real_time:45819.752222 global_time:4085, event:`SYSCALL: pselect6' (state:EXITING_SYSCALL) tid:19506, ticks:2456
rax:0x0 rbx:0x0 rcx:0xffffffffffffffff rdx:0x0 rsi:0x0 rdi:0x0 rbp:0x7eff76532dc0 rsp:0x7eff76532db0 r8:0x7eff76532db0 r9:0x0 r10:0x0 r11:0x246 r12:0x7ffd00ae730e r13:0x7ffd00ae730f r14:0x7eff76533700 r15:0x7ffd00ae7390 rip:0x55964d54c313 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x10e fs_base:0x7eff76533700 gs_base:0x0
{ tid:19506, addr:0x7eff76532db0, length:0x10 }
}
{
real_time:45819.752347 global_time:4086, event:`SYSCALLBUF_FLUSH' tid:19506, ticks:2518
{ syscall:'clock_gettime', ret:0x0, size:0x20 }
}
{
real_time:45819.752368 global_time:4087, event:`SYSCALL: pselect6' (state:ENTERING_SYSCALL) tid:19506, ticks:2518
rax:0xffffffffffffffda rbx:0x0 rcx:0xffffffffffffffff rdx:0x0 rsi:0x0 rdi:0x0 rbp:0x7eff76532dc0 rsp:0x7eff76532db0 r8:0x7eff76532db0 r9:0x0 r10:0x0 r11:0x246 r12:0x7ffd00ae730e r13:0x7ffd00ae730f r14:0x7eff76533700 r15:0x7ffd00ae7390 rip:0x55964d54c313 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x10e fs_base:0x7eff76533700 gs_base:0x0
}
{
real_time:45819.752375 global_time:4088, event:`SYSCALLBUF_RESET' tid:19506, ticks:2518
}
{
real_time:45819.752578 global_time:4089, event:`SYSCALLBUF_FLUSH' tid:19499, ticks:4770498
{ syscall:'read', ret:0x0, size:0x10 }
}
{
real_time:45819.752588 global_time:4090, event:`SYSCALL: epoll_ctl' (state:ENTERING_SYSCALL) tid:19499, ticks:4770498
rax:0xffffffffffffffda rbx:0x1 rcx:0xffffffffffffffff rdx:0x5 rsi:0x2 rdi:0x4 rbp:0xc420215408 rsp:0xc4202153d0 r8:0xc4200322d0 r9:0x2 r10:0xc4202153fc r11:0x246 r12:0x0 r13:0xf2 r14:0x32 r15:0x2 rip:0x55964d54c9c8 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xe9 fs_base:0x7eff76674b80 gs_base:0x0
}
{
real_time:45819.752594 global_time:4091, event:`SYSCALLBUF_RESET' tid:19499, ticks:4770498
}
{
real_time:45819.752628 global_time:4092, event:`SYSCALL: epoll_ctl' (state:EXITING_SYSCALL) tid:19499, ticks:4770498
rax:0x0 rbx:0x1 rcx:0xffffffffffffffff rdx:0x5 rsi:0x2 rdi:0x4 rbp:0xc420215408 rsp:0xc4202153d0 r8:0xc4200322d0 r9:0x2 r10:0xc4202153fc r11:0x246 r12:0x0 r13:0xf2 r14:0x32 r15:0x2 rip:0x55964d54c9c8 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xe9 fs_base:0x7eff76674b80 gs_base:0x0
}
{
real_time:45819.752746 global_time:4093, event:`SYSCALLBUF_FLUSH' tid:19499, ticks:4771422
{ syscall:'close', ret:0x0, size:0x10 }
}
{
real_time:45819.752756 global_time:4094, event:`SYSCALL: fstatat64' (state:ENTERING_SYSCALL) tid:19499, ticks:4771422
rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0xc4202aa6b8 rsi:0xc42026eab0 rdi:0xffffffffffffff9c rbp:0x106 rsp:0x681ffe10 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0xffffffffffffffff r13:0x3a r14:0x39 r15:0xaa rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x106 fs_base:0x7eff76674b80 gs_base:0x0
}
{
real_time:45819.752763 global_time:4095, event:`SYSCALLBUF_RESET' tid:19499, ticks:4771422
}
{
real_time:45819.752804 global_time:4096, event:`SYSCALL: fstatat64' (state:EXITING_SYSCALL) tid:19499, ticks:4771422
rax:0x0 rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0xc4202aa6b8 rsi:0xc42026eab0 rdi:0xffffffffffffff9c rbp:0x106 rsp:0x681ffe10 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0xffffffffffffffff r13:0x3a r14:0x39 r15:0xaa rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x106 fs_base:0x7eff76674b80 gs_base:0x0
{ tid:19499, addr:0xc4202aa6b8, length:0x90 }
}
{
real_time:45819.753014 global_time:4097, event:`SYSCALLBUF_FLUSH' tid:19499, ticks:4836347
{ syscall:'geteuid', ret:0x3e8, size:0x10 }
}
{
real_time:45819.753052 global_time:4098, event:`SYSCALL: execve' (state:ENTERING_SYSCALL) tid:19499, ticks:4836347
rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0xc42028e600 rsi:0xc4202d8270 rdi:0xc42026ed20 rbp:0x3b rsp:0x681ffe10 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0xffffffffffffffff r13:0xd0 r14:0xcf r15:0x100 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x3b fs_base:0x7eff76674b80 gs_base:0x0
}
{
real_time:45819.753059 global_time:4099, event:`SYSCALLBUF_RESET' tid:19499, ticks:4836347
}
{
real_time:45819.753204 global_time:4100, event:`EXIT' tid:19507, ticks:2425
}
{
real_time:45819.753279 global_time:4101, event:`EXIT' tid:19509, ticks:5163
}
{
real_time:45819.753347 global_time:4102, event:`EXIT' tid:19508, ticks:716
}
{
real_time:45819.754360 global_time:4103, event:`EXIT' tid:19506, ticks:2518
}
=== Start rr backtrace:
rr(_ZN2rr13dump_rr_stackEv+0x35)[0x562b2cbadbc3]
rr(_ZN2rr9GdbServer15emergency_debugEPNS_4TaskE+0x174)[0x562b2ca4af3a]
rr(+0x30768d)[0x562b2ca6e68d]
rr(_ZN2rr21EmergencyDebugOstreamD1Ev+0x62)[0x562b2ca6e894]
rr(_ZN2rr18AutoRemoteSyscalls20check_syscall_resultEli+0x2e5)[0x562b2c9edef9]
rr(_ZN2rr18AutoRemoteSyscalls22infallible_syscall_ptrIJNS_10remote_ptrIvEEmiiimEEES3_iDpT_+0x1dd)[0x562b2c9f023b]
rr(_ZN2rr18AutoRemoteSyscalls23infallible_mmap_syscallENS_10remote_ptrIvEEmiiim+0xd0)[0x562b2c9ed95a]
rr(_ZN2rr12AddressSpace11map_rr_pageERNS_18AutoRemoteSyscallsE+0x13f)[0x562b2c9c5df5]
rr(_ZN2rr12AddressSpace17post_exec_syscallEPNS_4TaskE+0xaa)[0x562b2c9c6978]
rr(_ZN2rr4Task17post_exec_syscallEv+0x5f)[0x562b2cb78ee1]
rr(+0x357b33)[0x562b2cabeb33]
rr(+0x36e278)[0x562b2cad5278]
rr(+0x35bcdc)[0x562b2cac2cdc]
rr(_ZN2rr19rec_process_syscallEPNS_10RecordTaskE+0xd9)[0x562b2cac2db8]
rr(_ZN2rr13RecordSession21syscall_state_changedEPNS_10RecordTaskEPNS0_9StepStateE+0xfb9)[0x562b2caa8e51]
rr(_ZN2rr13RecordSession11record_stepEv+0x589)[0x562b2caadccf]
rr(+0x33a2c5)[0x562b2caa12c5]
rr(_ZN2rr13RecordCommand3runERSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS7_EE+0x2ae)[0x562b2caa1ce4]
rr(main+0x20c)[0x562b2cbc4efb]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xeb)[0x7f3ae665d09b]
rr(_start+0x2a)[0x562b2c9c455a]
=== End rr backtrace
Launch gdb with
gdb '-l' '10000' '-ex' 'set sysroot /' '-ex' 'target extended-remote 127.0.0.1:19499' /snap/core/8039/usr/lib/snapd/snap-confine
Interesting. Not sure what's going on there, there might be some security seccomp policy that needs to be disabled.
I don't think Kyle or I are really incentivized to work on this right now, but maybe later.
A good next step here would be to try getting anbox running without snap. Snap is probably contributing some extra selinux stuff that we can avoid.
A good next step here would be to try getting anbox running without snap.
I've spent some time trying to do this, and while I was able to build anbox from source, I haven't been able to get its various components to start up and interact successfully outside of snap.
@jrmuizel suggested a potential alternative strategy of modifying the snap package script to build and include rr as something that runs inside the snap package.
I'm investigating how much work this would be. What is the simplest anbox thing that we could run?
That's not an android thing though. I have a good idea of what's required for snap at this point, I want to exercise anbox.
Whoops, total thinko there :)
A simple command-line thing like ls? Or does it need to have a GUI (or perhaps to exercise the binder and ashmem kernel modules) to be interesting for this purpose?
The approach that seems most practical to me getting this to work is using Anbox. Anbox lets you run android apps in a container on regular Linux. Given this you would a single rr recording of all of the processes running in the container.
@jrmuizel @khuey Anbox struggles a bit more with Android apps than I had hoped. For example, it isn't able to run modern Chromium. https://github.com/anbox/anbox/issues/1637
There's also Waydroid, if someone can force the system image to be aarch64 or x86_64 only. https://twitter.com/zhuowei/status/1533159959575379971