mysql-operator icon indicating copy to clipboard operation
mysql-operator copied to clipboard
verified publisher icon oracle

buffer overflow detected

Open welyss opened this issue 7 years ago • 1 comments

Versions

MySQL Operator Version: release: 0.3.0 Environment:

  • Kubernetes version (use kubectl version): Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.1", GitCommit:"eec55b9ba98609a46fee712359c7b5b365bdd920", GitTreeState:"clean", BuildDate:"2018-12-13T10:39:04Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.5", GitCommit:"2166946f41b36dea2c4626f90a77706f426cdea2", GitTreeState:"clean", BuildDate:"2019-03-25T15:19:22Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}

  • Cloud provider or hardware configuration:

  • OS (e.g. from /etc/os-release):

  • Kernel (e.g. uname -a):

  • Others:

What happened?

mysql server crash, fatal error, kubernetes health check faild, restart pod, pod[mysql server] start, fatal error crash.(loop) And i use cephrbd as persistence volume, so my /var/lib/mysql directory will exists even then pod crash. after mysql server crash, log is:

  • base=1000 ++ cat /etc/hostname ++ grep -o '[^-]*$'
  • index=1 ++ expr 1000 + 1
  • /entrypoint.sh --server_id=1001 --datadir=/var/lib/mysql --user=mysql --gtid_mode=ON --log-bin --binlog_checksum=NONE --enforce_gtid_consistency=ON --log-slave-updates=ON --binlog-format=ROW --master-info-repository=TABLE --relay-log-info-repository=TABLE --transaction-write-set-extraction=XXHASH64 --relay-log=mysqlcluster-grafana-1-relay-bin --report-host=mysqlcluster-grafana-1.mysqlcluster-grafana --log-error-verbosity=3 [Entrypoint] MySQL Docker Image 8.0.15-1.1.10 /dev/mapper/centos-root /etc/my.cnf xfs ro,relatime,attr2,inode64,noquota 0 0 [Entrypoint] Starting MySQL 8.0.15-1.1.10 *** buffer overflow detected ***: /usr/sbin/mysqld terminated ======= Backtrace: ========= /lib64/libc.so.6(__fortify_fail+0x37)[0x7efc74ec8aa7] /lib64/libc.so.6(+0x115c22)[0x7efc74ec6c22] /usr/sbin/mysqld(_ZN25Persisted_variables_cache19set_persist_optionsEb+0x11ca)[0xc7d1da] /usr/sbin/mysqld(_Z11mysqld_mainiPPc+0x22ce)[0xc40ece] /lib64/libc.so.6(__libc_start_main+0xf5)[0x7efc74dd3495] /usr/sbin/mysqld[0xc2f7e9] ======= Memory map: ======== 00400000-032b8000 r-xp 00000000 fc:10 3417641 /usr/sbin/mysqld 032b9000-03407000 r--p 02eb8000 fc:10 3417641 /usr/sbin/mysqld 03407000-0363d000 rw-p 03006000 fc:10 3417641 /usr/sbin/mysqld 0363d000-03750000 rw-p 00000000 00:00 0 039f8000-05106000 rw-p 00000000 00:00 0 [heap] 7efbdc000000-7efbdc022000 rw-p 00000000 00:00 0 7efbdc022000-7efbe0000000 ---p 00000000 00:00 0 7efbe4000000-7efbe4024000 rw-p 00000000 00:00 0 7efbe4024000-7efbe8000000 ---p 00000000 00:00 0 7efbe8000000-7efbe8024000 rw-p 00000000 00:00 0 7efbe8024000-7efbec000000 ---p 00000000 00:00 0 7efbec000000-7efbec20a000 rw-p 00000000 00:00 0 7efbec20a000-7efbf0000000 ---p 00000000 00:00 0 7efbf0000000-7efbf0024000 rw-p 00000000 00:00 0 7efbf0024000-7efbf4000000 ---p 00000000 00:00 0 7efbf4000000-7efbf4021000 rw-p 00000000 00:00 0 7efbf4021000-7efbf8000000 ---p 00000000 00:00 0 7efbf8000000-7efbf8024000 rw-p 00000000 00:00 0 7efbf8024000-7efbfc000000 ---p 00000000 00:00 0 7efbfc000000-7efbfc024000 rw-p 00000000 00:00 0 7efbfc024000-7efc00000000 ---p 00000000 00:00 0 7efc00000000-7efc00021000 rw-p 00000000 00:00 0 7efc00021000-7efc04000000 ---p 00000000 00:00 0 7efc04000000-7efc04022000 rw-p 00000000 00:00 0 7efc04022000-7efc08000000 ---p 00000000 00:00 0 7efc08000000-7efc08024000 rw-p 00000000 00:00 0 7efc08024000-7efc0c000000 ---p 00000000 00:00 0 7efc0c000000-7efc0c024000 rw-p 00000000 00:00 0 7efc0c024000-7efc10000000 ---p 00000000 00:00 0 7efc11118000-7efc11119000 ---p 00000000 00:00 0 7efc11119000-7efc11919000 rw-p 00000000 00:00 0 7efc11919000-7efc1191a000 ---p 00000000 00:00 0 7efc1191a000-7efc1211a000 rw-p 00000000 00:00 0 7efc1211a000-7efc1211b000 ---p 00000000 00:00 0 7efc1211b000-7efc1291b000 rw-p 00000000 00:00 0 7efc1291b000-7efc12ad9000 r-xp 00000000 fc:10 3417562 /usr/lib64/mysql/plugin/group_replication.so 7efc12ad9000-7efc12cd9000 ---p 001be000 fc:10 3417562 /usr/lib64/mysql/plugin/group_replication.so 7efc12cd9000-7efc12cde000 r--p 001be000 fc:10 3417562 /usr/lib64/mysql/plugin/group_replication.so 7efc12cde000-7efc12ce5000 rw-p 001c3000 fc:10 3417562 /usr/lib64/mysql/plugin/group_replication.so 7efc12ce5000-7efc137ff000 rw-p 00000000 00:00 0 7efc137ff000-7efc13800000 ---p 00000000 00:00 0 7efc13800000-7efc14000000 rw-p 00000000 00:00 0 7efc14000000-7efc14021000 rw-p 00000000 00:00 0 7efc14021000-7efc18000000 ---p 00000000 00:00 0 7efc18000000-7efc18021000 rw-p 00000000 00:00 0 7efc18021000-7efc1c000000 ---p 00000000 00:00 0 7efc1c000000-7efc1c021000 rw-p 00000000 00:00 0 7efc1c021000-7efc20000000 ---p 00000000 00:00 0 7efc20000000-7efc20021000 rw-p 00000000 00:00 0 7efc20021000-7efc24000000 ---p 00000000 00:00 0 7efc24000000-7efc24021000 rw-p 00000000 00:00 0 7efc24021000-7efc28000000 ---p 00000000 00:00 0 7efc282cd000-7efc282ce000 ---p 00000000 00:00 0 7efc282ce000-7efc28ace000 rw-p 00000000 00:00 0 7efc28ace000-7efc28acf000 ---p 00000000 00:00 0 7efc28acf000-7efc292cf000 rw-p 00000000 00:00 0 7efc292cf000-7efc292d0000 ---p 00000000 00:00 0 7efc292d0000-7efc29ad0000 rw-p 00000000 00:00 0 7efc29ad0000-7efc29ad1000 ---p 00000000 00:00 0 7efc29ad1000-7efc2a993000 rw-p 00000000 00:00 0 7efc2a993000-7efc2a994000 ---p 00000000 00:00 0 7efc2a994000-7efc2b194000 rw-p 00000000 00:00 0 7efc2b194000-7efc2b195000 ---p 00000000 00:00 0 7efc2b195000-7efc2b995000 rw-p 00000000 00:00 0 7efc2b995000-7efc2b996000 ---p 00000000 00:00 0 7efc2b996000-7efc2c196000 rw-p 00000000 00:00 0 7efc2c196000-7efc2c197000 ---p 00000000 00:00 0 7efc2c197000-7efc2c997000 rw-p 00000000 00:00 0 7efc2c997000-7efc2c998000 ---p 00000000 00:00 0 7efc2c998000-7efc2d198000 rw-p 00000000 00:00 0 7efc2d198000-7efc2d199000 ---p 00000000 00:00 0 7efc2d199000-7efc2d999000 rw-p 00000000 00:00 0 7efc2d999000-7efc2d99a000 ---p 00000000 00:00 0 7efc2d99a000-7efc2e19a000 rw-p 00000000 00:00 0 7efc2e19a000-7efc2e19b000 ---p 00000000 00:00 0 7efc2e19b000-7efc2ebb9000 rw-p 00000000 00:00 0 7efc2ebb9000-7efc2ebba000 ---p 00000000 00:00 0 7efc2ebba000-7efc31bbd000 rw-p 00000000 00:00 0 7efc31bbd000-7efc31bbe000 ---p 00000000 00:00 0 7efc31bbe000-7efc323be000 rw-p 00000000 00:00 0 7efc323be000-7efc323bf000 ---p 00000000 00:00 0 7efc323bf000-7efc32bbf000 rw-p 00000000 00:00 0 7efc32bbf000-7efc32bc0000 ---p 00000000 00:00 0 7efc32bc0000-7efc54000000 rw-p 00000000 00:00 0 7efc54000000-7efc54d98000 rw-p 00000000 00:00 0 7efc54d98000-7efc58000000 ---p 00000000 00:00 0 7efc58024000-7efc58229000 rw-p 00000000 00:00 0 7efc58229000-7efc5822a000 ---p 00000000 00:00 0 7efc5822a000-7efc58a2a000 rw-p 00000000 00:00 0 7efc58a2a000-7efc58a2b000 ---p 00000000 00:00 0 7efc58a2b000-7efc5922b000 rw-p 00000000 00:00 0 7efc5922b000-7efc5922c000 ---p 00000000 00:00 0 7efc5922c000-7efc59a2c000 rw-p 00000000 00:00 0 7efc59a2c000-7efc59a2d000 ---p 00000000 00:00 0 7efc59a2d000-7efc5a22d000 rw-p 00000000 00:00 0 7efc5a22d000-7efc5a22e000 ---p 00000000 00:00 0 7efc5a22e000-7efc5aa2e000 rw-p 00000000 00:00 0 7efc5aa2e000-7efc5aa2f000 ---p 00000000 00:00 0 7efc5aa2f000-7efc5b22f000 rw-p 00000000 00:00 0 7efc5b22f000-7efc5b230000 ---p 00000000 00:00 0 7efc5b230000-7efc5bdff000 rw-p 00000000 00:00 0 7efc5be0c000-7efc5be0d000 ---p 00000000 00:00 0 7efc5be0d000-7efc5be54000 rw-p 00000000 00:00 0 7efc5be54000-7efc5be55000 ---p 00000000 00:00 0 7efc5be55000-7efc5be9c000 rw-p 00000000 00:00 0 7efc5be9c000-7efc5be9d000 ---p 00000000 00:00 0 7efc5be9d000-7efc5bee4000 rw-p 00000000 00:00 0 7efc5bee4000-7efc5bee5000 ---p 00000000 00:00 0 7efc5bee5000-7efc5bf2b000 rw-p 00000000 00:00 0 7efc5bf2b000-7efc5bf2c000 ---p 00000000 00:00 0 7efc5bf2c000-7efc5bf72000 rw-p 00000000 00:00 0 7efc5bf72000-7efc5bf73000 ---p 00000000 00:00 0 7efc5bf73000-7efc5bfb9000 rw-p 00000000 00:00 0 7efc5bfb9000-7efc5bfba000 ---p 00000000 00:00 0 7efc5bfba000-7efc5c000000 rw-p 00000000 00:00 0 7efc5c000000-7efc5cc79000 rw-p 00000000 00:00 0 7efc5cc79000-7efc60000000 ---p 00000000 00:00 0 7efc60037000-7efc60a99000 rw-p 00000000 00:00 0 7efc60a99000-7efc60a9a000 ---p 00000000 00:00 0 7efc60a9a000-7efc6129a000 rw-p 00000000 00:00 0 7efc6129a000-7efc6129f000 rw-s 00000000 00:10 133170779 /[aio] (deleted) 7efc6129f000-7efc612a4000 rw-s 00000000 00:10 133170778 /[aio] (deleted) 7efc612a4000-7efc612a9000 rw-s 00000000 00:10 133170777 /[aio] (deleted) 7efc612a9000-7efc612ae000 rw-s 00000000 00:10 133170776 /[aio] (deleted) 7efc612ae000-7efc61808000 rw-p 00000000 00:00 0 7efc61808000-7efc61809000 ---p 00000000 00:00 0 7efc61809000-7efc6282b000 rw-p 00000000 00:00 0 7efc6282b000-7efc62837000 r-xp 00000000 fc:10 3153431 /usr/lib64/libnss_files-2.17.so 7efc62837000-7efc62a36000 ---p 0000c000 fc:10 3153431 /usr/lib64/libnss_files-2.17.so 7efc62a36000-7efc62a37000 r--p 0000b000 fc:10 3153431 /usr/lib64/libnss_files-2.17.so 7efc62a37000-7efc62a38000 rw-p 0000c000 fc:10 3153431 /usr/lib64/libnss_files-2.17.so 7efc62a38000-7efc7358e000 rw-p 00000000 00:00 0 7efc7358e000-7efc735ee000 r-xp 00000000 fc:10 3153453 /usr/lib64/libpcre.so.1.2.0 7efc735ee000-7efc737ee000 ---p 00060000 fc:10 3153453 /usr/lib64/libpcre.so.1.2.0 7efc737ee000-7efc737ef000 r--p 00060000 fc:10 3153453 /usr/lib64/libpcre.so.1.2.0 7efc737ef000-7efc737f0000 rw-p 00061000 fc:10 3153453 /usr/lib64/libpcre.so.1.2.0 7efc737f0000-7efc73814000 r-xp 00000000 fc:10 3153487 /usr/lib64/libselinux.so.1 7efc73814000-7efc73a13000 ---p 00024000 fc:10 3153487 /usr/lib64/libselinux.so.1 7efc73a13000-7efc73a14000 r--p 00023000 fc:10 3153487 /usr/lib64/libselinux.so.1 7efc73a14000-7efc73a15000 rw-p 00024000 fc:10 3153487 /usr/lib64/libselinux.so.1 7efc73a15000-7efc73a17000 rw-p 00000000 00:00 0 7efc73a17000-7efc73a2d000 r-xp 00000000 fc:10 3153473 /usr/lib64/libresolv-2.17.so 7efc73a2d000-7efc73c2c000 ---p 00016000 fc:10 3153473 /usr/lib64/libresolv-2.17.so 7efc73c2c000-7efc73c2d000 r--p 00015000 fc:10 3153473 /usr/lib64/libresolv-2.17.so 7efc73c2d000-7efc73c2e000 rw-p 00016000 fc:10 3153473 /usr/lib64/libresolv-2.17.so 7efc73c2e000-7efc73c30000 rw-p 00000000 00:00 0 7efc73c30000-7efc73c33000 r-xp 00000000 fc:10 3153386 /usr/lib64/libkeyutils.so.1.5 7efc73c33000-7efc73e32000 ---p 00003000 fc:10 3153386 /usr/lib64/libkeyutils.so.1.5 7efc73e32000-7efc73e33000 r--p 00002000 fc:10 3153386 /usr/lib64/libkeyutils.so.1.5 7efc73e33000-7efc73e34000 rw-p 00003000 fc:10 3153386 /usr/lib64/libkeyutils.so.1.5 7efc73e34000-7efc73e41000 r-xp 00000000 fc:10 3153392 /usr/lib64/libkrb5support.so.0.1 7efc73e41000-7efc74041000 ---p 0000d000 fc:10 3153392 /usr/lib64/libkrb5support.so.0.1 7efc74041000-7efc74042000 r--p 0000d000 fc:10 3153392 /usr/lib64/libkrb5support.so.0.1 7efc74042000-7efc74043000 rw-p 0000e000 fc:10 3153392 /usr/lib64/libkrb5support.so.0.1 7efc74043000-7efc74058000 r-xp 00000000 fc:10 3153523 /usr/lib64/libz.so.1.2.7 7efc74058000-7efc74257000 ---p 00015000 fc:10 3153523 /usr/lib64/libz.so.1.2.7 7efc74257000-7efc74258000 r--p 00014000 fc:10 3153523 /usr/lib64/libz.so.1.2.7 7efc74258000-7efc74259000 rw-p 00015000 fc:10 3153523 /usr/lib64/libz.so.1.2.7 7efc74259000-7efc74272000 r-xp 00000000 fc:10 3153382 /usr/lib64/libk5crypto.so.3.1 7efc74272000-7efc74471000 ---p 00019000 fc:10 3153382 /usr/lib64/libk5crypto.so.3.1 7efc74471000-7efc74473000 r--p 00018000 fc:10 3153382 /usr/lib64/libk5crypto.so.3.1 7efc74473000-7efc74474000 rw-p 0001a000 fc:10 3153382 /usr/lib64/libk5crypto.so.3.1 7efc74474000-7efc74477000 r-xp 00000000 fc:10 3153320 /usr/lib64/libcom_err.so.2.1 7efc74477000-7efc74676000 ---p 00003000 fc:10 3153320 /usr/lib64/libcom_err.so.2.1 7efc74676000-7efc74677000 r--p 00002000 fc:10 3153320 /usr/lib64/libcom_err.so.2.1 7efc74677000-7efc74678000 rw-p 00003000 fc:10 3153320 /usr/lib64/libcom_err.so.2.1 7efc74678000-7efc74751000 r-xp 00000000 fc:10 3153390 /usr/lib64/libkrb5.so.3.3 7efc74751000-7efc74950000 ---p 000d9000 fc:10 3153390 /usr/lib64/libkrb5.so.3.3 7efc74950000-7efc7495e000 r--p 000d8000 fc:10 3153390 /usr/lib64/libkrb5.so.3.3 7efc7495e000-7efc74961000 rw-p 000e6000 fc:10 3153390 /usr/lib64/libkrb5.so.3.3 7efc74961000-7efc749ab000 r-xp 00000000 fc:10 3153372 /usr/lib64/libgssapi_krb5.so.2.2 7efc749ab000-7efc74bab000 ---p 0004a000 fc:10 3153372 /usr/lib64/libgssapi_krb5.so.2.2 7efc74bab000-7efc74bac000 r--p 0004a000 fc:10 3153372 /usr/lib64/libgssapi_krb5.so.2.2 7efc74bac000-7efc74bae000 rw-p 0004b000 fc:10 3153372 /usr/lib64/libgssapi_krb5.so.2.2 7efc74bae000-7efc74bb0000 r-xp 00000000 fc:10 3153342 /usr/lib64/libfreebl3.so 7efc74bb0000-7efc74daf000 ---p 00002000 fc:10 3153342 /usr/lib64/libfreebl3.so 7efc74daf000-7efc74db0000 r--p 00001000 fc:10 3153342 /usr/lib64/libfreebl3.so 7efc74db0000-7efc74db1000 rw-p 00002000 fc:10 3153342 /usr/lib64/libfreebl3.so 7efc74db1000-7efc74f73000 r-xp 00000000 fc:10 3153311 /usr/lib64/libc-2.17.so 7efc74f73000-7efc75173000 ---p 001c2000 fc:10 3153311 /usr/lib64/libc-2.17.so 7efc75173000-7efc75177000 r--p 001c2000 fc:10 3153311 /usr/lib64/libc-2.17.so 7efc75177000-7efc75179000 rw-p 001c6000 fc:10 3153311 /usr/lib64/libc-2.17.so 7efc75179000-7efc7517e000 rw-p 00000000 00:00 0 7efc7517e000-7efc75193000 r-xp 00000000 fc:10 3153345 /usr/lib64/libgcc_s-4.8.5-20150702.so.1 7efc75193000-7efc75392000 ---p 00015000 fc:10 3153345 /usr/lib64/libgcc_s-4.8.5-20150702.so.1 7efc75392000-7efc75393000 r--p 00014000 fc:10 3153345 /usr/lib64/libgcc_s-4.8.5-20150702.so.1 7efc75393000-7efc75394000 rw-p 00015000 fc:10 3153345 /usr/lib64/libgcc_s-4.8.5-20150702.so.1 7efc75394000-7efc75495000 r-xp 00000000 fc:10 3153402 /usr/lib64/libm-2.17.so 7efc75495000-7efc75694000 ---p 00101000 fc:10 3153402 /usr/lib64/libm-2.17.so 7efc75694000-7efc75695000 r--p 00100000 fc:10 3153402 /usr/lib64/libm-2.17.so 7efc75695000-7efc75696000 rw-p 00101000 fc:10 3153402 /usr/lib64/libm-2.17.so 7efc75696000-7efc7577f000 r-xp 00000000 fc:10 3153503 /usr/lib64/libstdc++.so.6.0.19 7efc7577f000-7efc7597e000 ---p 000e9000 fc:10 3153503 /usr/lib64/libstdc++.so.6.0.19 7efc7597e000-7efc75986000 r--p 000e8000 fc:10 3153503 /usr/lib64/libstdc++.so.6.0.19 7efc75986000-7efc75988000 rw-p 000f0000 fc:10 3153503 /usr/lib64/libstdc++.so.6.0.19 7efc75988000-7efc7599d000 rw-p 00000000 00:00 0 7efc7599d000-7efc759a4000 r-xp 00000000 fc:10 3153483 /usr/lib64/librt-2.17.so 7efc759a4000-7efc75ba3000 ---p 00007000 fc:10 3153483 /usr/lib64/librt-2.17.so 7efc75ba3000-7efc75ba4000 r--p 00006000 fc:10 3153483 /usr/lib64/librt-2.17.so 7efc75ba4000-7efc75ba5000 rw-p 00007000 fc:10 3153483 /usr/lib64/librt-2.17.so 7efc75ba5000-7efc75ba7000 r-xp 00000000 fc:10 3153329 /usr/lib64/libdl-2.17.so 7efc75ba7000-7efc75da7000 ---p 00002000 fc:10 3153329 /usr/lib64/libdl-2.17.so 7efc75da7000-7efc75da8000 r--p 00002000 fc:10 3153329 /usr/lib64/libdl-2.17.so 7efc75da8000-7efc75da9000 rw-p 00003000 fc:10 3153329 /usr/lib64/libdl-2.17.so 7efc75da9000-7efc75fdd000 r-xp 00000000 fc:10 3153323 /usr/lib64/libcrypto.so.1.0.2k 7efc75fdd000-7efc761dd000 ---p 00234000 fc:10 3153323 /usr/lib64/libcrypto.so.1.0.2k 7efc761dd000-7efc761f9000 r--p 00234000 fc:10 3153323 /usr/lib64/libcrypto.so.1.0.2k 7efc761f9000-7efc76206000 rw-p 00250000 fc:10 3153323 /usr/lib64/libcrypto.so.1.0.2k 7efc76206000-7efc7620a000 rw-p 00000000 00:00 0 7efc7620a000-7efc76271000 r-xp 00000000 fc:10 3153499 /usr/lib64/libssl.so.1.0.2k 7efc76271000-7efc76471000 ---p 00067000 fc:10 3153499 /usr/lib64/libssl.so.1.0.2k 7efc76471000-7efc76475000 r--p 00067000 fc:10 3153499 /usr/lib64/libssl.so.1.0.2k 7efc76475000-7efc7647c000 rw-p 0006b000 fc:10 3153499 /usr/lib64/libssl.so.1.0.2k 7efc7647c000-7efc76484000 r-xp 00000000 fc:10 3153321 /usr/lib64/libcrypt-2.17.so 7efc76484000-7efc76683000 ---p 00008000 fc:10 3153321 /usr/lib64/libcrypt-2.17.so 7efc76683000-7efc76684000 r--p 00007000 fc:10 3153321 /usr/lib64/libcrypt-2.17.so 7efc76684000-7efc76685000 rw-p 00008000 fc:10 3153321 /usr/lib64/libcrypt-2.17.so 7efc76685000-7efc766b3000 rw-p 00000000 00:00 0 7efc766b3000-7efc766b4000 r-xp 00000000 fc:10 3417540 /usr/lib64/libaio.so.1.0.1 7efc766b4000-7efc768b3000 ---p 00001000 fc:10 3417540 /usr/lib64/libaio.so.1.0.1 7efc768b3000-7efc768b4000 r--p 00000000 fc:10 3417540 /usr/lib64/libaio.so.1.0.1 7efc768b4000-7efc768b5000 rw-p 00001000 fc:10 3417540 /usr/lib64/libaio.so.1.0.1 7efc768b5000-7efc768cc000 r-xp 00000000 fc:10 3153468 /usr/lib64/libpthread-2.17.so 7efc768cc000-7efc76acb000 ---p 00017000 fc:10 3153468 /usr/lib64/libpthread-2.17.so 7efc76acb000-7efc76acc000 r--p 00016000 fc:10 3153468 /usr/lib64/libpthread-2.17.so 7efc76acc000-7efc76acd000 rw-p 00017000 fc:10 3153468 /usr/lib64/libpthread-2.17.so 7efc76acd000-7efc76ad1000 rw-p 00000000 00:00 0 7efc76ad1000-7efc76af3000 r-xp 00000000 fc:10 3153290 /usr/lib64/ld-2.17.so 7efc76af3000-7efc76af4000 rw-p 00000000 00:00 0 7efc76af4000-7efc76af9000 rw-s 00000000 00:10 133170775 /[aio] (deleted) 7efc76af9000-7efc76afe000 rw-s 00000000 00:10 133170774 /[aio] (deleted) 7efc76afe000-7efc76b03000 rw-s 00000000 00:10 133170773 /[aio] (deleted) 7efc76b03000-7efc76b08000 rw-s 00000000 00:10 133170772 /[aio] (deleted) 7efc76b08000-7efc76b0d000 rw-s 00000000 00:10 133170771 /[aio] (deleted) 7efc76b0d000-7efc76b12000 rw-s 00000000 00:10 133170770 /[aio] (deleted) 7efc76b12000-7efc76cee000 rw-p 00000000 00:00 0 7efc76cee000-7efc76cf0000 rw-s 00000000 00:10 133170780 /[aio] (deleted) 7efc76cf0000-7efc76cf1000 rw-s 00000000 00:10 133170767 /[aio] (deleted) 7efc76cf1000-7efc76cf2000 rw-p 00000000 00:00 0 7efc76cf2000-7efc76cf3000 r--p 00021000 fc:10 3153290 /usr/lib64/ld-2.17.so 7efc76cf3000-7efc76cf4000 rw-p 00022000 fc:10 3153290 /usr/lib64/ld-2.17.so 7efc76cf4000-7efc76cf5000 rw-p 00000000 00:00 0 7ffd9313a000-7ffd9315b000 rw-p 00000000 00:00 0 [stack] 7ffd9315d000-7ffd93160000 r--p 00000000 00:00 0 [vvar] 7ffd93160000-7ffd93162000 r-xp 00000000 00:00 0 [vdso] ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall] 00:28:55 UTC - mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. Attempting to collect some information that could help diagnose the problem. As this is a crash and something is definitely wrong, the information collection process might fail.

key_buffer_size=8388608 read_buffer_size=131072 max_used_connections=0 max_threads=1024 thread_count=1 connection_count=0 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 412784 K bytes of memory Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x3f5d000 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 7ffd93158588 thread_stack 0x46000 /usr/sbin/mysqld(my_print_stacktrace(unsigned char*, unsigned long)+0x3d) [0x1cfae8d] /usr/sbin/mysqld(handle_fatal_signal+0x423) [0xe32883] /lib64/libpthread.so.0(+0xf5d0) [0x7efc768c45d0] /lib64/libc.so.6(gsignal+0x37) [0x7efc74de72c7] /lib64/libc.so.6(abort+0x148) [0x7efc74de89b8] /lib64/libc.so.6(+0x78de7) [0x7efc74e29de7] /lib64/libc.so.6(__fortify_fail+0x37) [0x7efc74ec8aa7] /lib64/libc.so.6(+0x115c22) [0x7efc74ec6c22] /usr/sbin/mysqld(Persisted_variables_cache::set_persist_options(bool)+0x11ca) [0xc7d1da] /usr/sbin/mysqld(mysqld_main(int, char**)+0x22ce) [0xc40ece] /lib64/libc.so.6(__libc_start_main+0xf5) [0x7efc74dd3495] /usr/sbin/mysqld() [0xc2f7e9]

Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0): is an invalid pointer Connection ID (thread ID): 6 Status: NOT_KILLED

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash.

see "Persisted_variables_cache::set_persist_options(bool)+0x11ca", i guess this is a mysql bug when server load /var/lib/mysql/mysqld-auto.cnf, maybe it's a wrong memery use. but there is a strange thing on this case. the content in /var/lib/mysql/mysqld-auto.cnf is:

bash-4.2$ cat /var/lib/mysql/mysqld-auto.cnf { "Version" : 1 , "mysql_server" : { "auto_increment_offset" : { "Value" : "7" , "Metadata" : { "Timestamp" : 1555055061959056 , "User" : "root" , "Host" : "mysqlcluster-grafana-0.mysqlcluster-grafana.dba.svc.cluster." } } , "auto_increment_increment" : { "Value" : "7" , "Metadata" : { "Timestamp" : 1555055061980954 , "User" : "root" , "Host" : "mysqlcluster-grafana-0.mysqlcluster-grafana.dba.svc.cluster." } } , "mysql_server_static_options" : { "group_replication_bootstrap_group" : { "Value" : "OFF" , "Metadata" : { "Timestamp" : 1555055066697921 , "User" : "root" , "Host" : "mysqlcluster-grafana-0.mysqlcluster-grafana.dba.svc.cluster." } } , "group_replication_group_name" : { "Value" : "c93d439c-5cf6-11e9-b8f8-9a5030f83f2b" , "Metadata" : { "Timestamp" : 1555055061950774 , "User" : "root" , "Host" : "mysqlcluster-grafana-0.mysqlcluster-grafana.dba.svc.cluster." } } , "group_replication_group_seeds" : { "Value" : "mysqlcluster-grafana-1.mysqlcluster-grafana:33061,mysqlcluster-grafana-2.mysqlcluster-grafana:33061" , "Metadata" : { "Timestamp" : 1555055533798898 , "User" : "root" , "Host" : "mysqlcluster-grafana-2.mysqlcluster-grafana.dba.svc.cluster." } } , "group_replication_ip_whitelist" : { "Value" : "10.0.0.0/8" , "Metadata" : { "Timestamp" : 1555055061948765 , "User" : "root" , "Host" : "mysqlcluster-grafana-0.mysqlcluster-grafana.dba.svc.cluster." } } , "group_replication_local_address" : { "Value" : "mysqlcluster-grafana-0.mysqlcluster-grafana:33061" , "Metadata" : { "Timestamp" : 1555055061974887 , "User" : "root" , "Host" : "mysqlcluster-grafana-0.mysqlcluster-grafana.dba.svc.cluster." } } , "group_replication_recovery_use_ssl" : { "Value" : "ON" , "Metadata" : { "Timestamp" : 1555055061956596 , "User" : "root" , "Host" : "mysqlcluster-grafana-0.mysqlcluster-grafana.dba.svc.cluster." } } , "group_replication_single_primary_mode" : { "Value" : "OFF" , "Metadata" : { "Timestamp" : 1555055061946783 , "User" : "root" , "Host" : "mysqlcluster-grafana-0.mysqlcluster-grafana.dba.svc.cluster." } } , "group_replication_ssl_mode" : { "Value" : "REQUIRED" , "Metadata" : { "Timestamp" : 1555055061961098 , "User" : "root" , "Host" : "mysqlcluster-grafana-0.mysqlcluster-grafana.dba.svc.cluster." } } , "group_replication_start_on_boot" : { "Value" : "ON" , "Metadata" : { "Timestamp" : 1555055061963265 , "User" : "root" , "Host" : "mysqlcluster-grafana-0.mysqlcluster-grafana.dba.svc.cluster." } } } } }

why auto_increment_offset and auto_increment_increment is "7" and i don't really know c lang very well, so i can't confirm whether the real reason of crash is a mysql bug with wrong memery use. somebody can help thanks!!!!!!

What you expected to happen?

mysqld on after restart pod

How to reproduce it (as minimally and precisely as possible)?

sorry, i don't really know how to reproduce

Anything else we need to know?

after crash, if i took the group_replication_ip_whitelist variable to custom my.cnf with configmap(because if i remove the file "/var/lib/mysql/mysqld-auto.cnf" with rm, mysqld boot cmd doesn't carry this variable, it will cause error about whitelist), then remove /var/lib/mysql/mysqld-auto.cnf, delete pod, so it will restart a new pod in statefulset under the cluster which created by mysql-operator, then it recover, and it works!!i don't want to do that always.......= =

welyss avatar Apr 19 '19 01:04 welyss

were you able to fix it? I'm currently facing the same issue. Edit: for reference, I'm also using a custom my.cnf with the following:

[mysqld]
default_authentication_plugin=mysql_native_password
group_replication_ip_whitelist=10.0.0.0/8

Your workaround works btw 👍

azaiter avatar Jul 15 '19 17:07 azaiter