pg_backtrace

pg_backtrace copied to clipboard

This PostgreSQL extension can be used to get more information about PostgreSQL backend execution when no debugger is available. It allows to dump stack trace when error is reported or exception is caught. So there three used cases of using this extension:

1. Find out the source of error. pg_backtrace extension provides "pg_backtrace.level" GUC which selects error level for which backtrace information is attached. Default value is ERROR. So once this extension is initialized, all errors will include backtrace information which is dumped both in log file and delivered to the client:

postgres=# select count(*)/0.0 from pg_class; ERROR: division by zero CONTEXT: postgres: knizhnik postgres [local] SELECT(numeric_div+0xbc) [0x7c5ebc] postgres: knizhnik postgres [local] SELECT() [0x5fe4e2] postgres: knizhnik postgres [local] SELECT() [0x610730] postgres: knizhnik postgres [local] SELECT() [0x6115ca] postgres: knizhnik postgres [local] SELECT(standard_ExecutorRun+0x15a) [0x60193a] postgres: knizhnik postgres [local] SELECT() [0x74168c] postgres: knizhnik postgres [local] SELECT(PortalRun+0x29e) [0x742a7e] postgres: knizhnik postgres [local] SELECT() [0x73e922] postgres: knizhnik postgres [local] SELECT(PostgresMain+0x1189) [0x73fde9] postgres: knizhnik postgres [local] SELECT() [0x47d5e0] postgres: knizhnik postgres [local] SELECT(PostmasterMain+0xd28) [0x6d0448] postgres: knizhnik postgres [local] SELECT(main+0x421) [0x47e511] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0) [0x7f6361a13830] postgres: knizhnik postgres [local] SELECT(_start+0x29) [0x47e589]

2. Determine current state of backend (assume that there is some long running query and you do not know where it spends most of time). It is possible to send SIGINT signal to backend and it print current stack in logfile:

2018-11-12 18:24:12.222 MSK [24457] LOG: Caught signal 2 2018-11-12 18:24:12.222 MSK [24457] CONTEXT: /lib/x86_64-linux-gnu/libpthread.so.0(+0x11390) [0x7f63624e3390] /lib/x86_64-linux-gnu/libc.so.6(epoll_wait+0x13) [0x7f6361afa9f3] postgres: knizhnik postgres [local] SELECT(WaitEventSetWait+0xbe) [0x71e4de] postgres: knizhnik postgres [local] SELECT(WaitLatchOrSocket+0x8b) [0x71e93b] postgres: knizhnik postgres [local] SELECT(pg_sleep+0x98) [0x7babd8] postgres: knizhnik postgres [local] SELECT() [0x5fe4e2] postgres: knizhnik postgres [local] SELECT() [0x6266a8] postgres: knizhnik postgres [local] SELECT(standard_ExecutorRun+0x15a) [0x60193a] postgres: knizhnik postgres [local] SELECT() [0x74168c] postgres: knizhnik postgres [local] SELECT(PortalRun+0x29e) [0x742a7e] postgres: knizhnik postgres [local] SELECT() [0x73e922] postgres: knizhnik postgres [local] SELECT(PostgresMain+0x1189) [0x73fde9] postgres: knizhnik postgres [local] SELECT() [0x47d5e0] postgres: knizhnik postgres [local] SELECT(PostmasterMain+0xd28) [0x6d0448] postgres: knizhnik postgres [local] SELECT(main+0x421) [0x47e511] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0) [0x7f6361a13830] postgres: knizhnik postgres [local] SELECT(_start+0x29) [0x47e589]

3. Get stack trace for SIGSEGV or SIGBUS signals (if dumping cores is disabled for some reasons):

2018-11-12 18:25:52.636 MSK [24518] LOG: Caught signal 11 2018-11-12 18:25:52.636 MSK [24518] CONTEXT: /home/knizhnik/postgresql/dist/lib/pg_backtrace.so(+0xe37) [0x7f6358838e37] /lib/x86_64-linux-gnu/libpthread.so.0(+0x11390) [0x7f63624e3390] /home/knizhnik/postgresql/dist/lib/pg_backtrace.so(pg_backtrace_sigsegv+0) [0x7f6358838fb0] postgres: knizhnik postgres [local] SELECT() [0x5fe474] postgres: knizhnik postgres [local] SELECT() [0x6266a8] postgres: knizhnik postgres [local] SELECT(standard_ExecutorRun+0x15a) [0x60193a] postgres: knizhnik postgres [local] SELECT() [0x74168c] postgres: knizhnik postgres [local] SELECT(PortalRun+0x29e) [0x742a7e] postgres: knizhnik postgres [local] SELECT() [0x73e922] postgres: knizhnik postgres [local] SELECT(PostgresMain+0x1189) [0x73fde9] postgres: knizhnik postgres [local] SELECT() [0x47d5e0] postgres: knizhnik postgres [local] SELECT(PostmasterMain+0xd28) [0x6d0448] postgres: knizhnik postgres [local] SELECT(main+0x421) [0x47e511] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0) [0x7f6361a13830] postgres: knizhnik postgres [local] SELECT(_start+0x29) [0x47e589]

As far as Postgres extension is loaded and initialized on first access to its functions, it is necessary to call pg_backtrace_init() function to be able to use this extension. This function function actually does nothing and _PG_init() registers signal handlers for SIGSEGV, SIGBUS and SIGINT and executor run hook which setups exception context.

This extension is using backtrace function which is available at most Unixes. As it was mentioned in backtrace documentation:

The symbol names may be unavailable without the use of special linker options.
For systems using the GNU linker, it is necessary to use the -rdynamic
linker option.  Note that names of "static" functions are not exposed,
and won't be available in the backtrace.

Postgres is built without -rdynamic option. This is why not all function addresses in the stack trace above are resolved. It is possible to use GDB (at development host with correspondent postgres binaries) or Linux addr2line utility to get resolve function addresses:

$ addr2line -e ~/postgresql/dist/bin/postgres -a 0x5fe4e2
0x00000000005fe4e2
execExprInterp.c:?

Usage:

create extension pg_backtrace; select pg_backtrace_init();