client_golang icon indicating copy to clipboard operation
client_golang copied to clipboard
verified publisher icon prometheus

runtime error: index out of range [43] with length 30 in HandlerForTransactional

Open huanghy6268 opened this issue 1 year ago • 2 comments

Background

It panics suddenly and does not happen again.

Envs

Go version: go1.18

github.com/prometheus/client_golang: v1.14.0

github.com/prometheus/common: v0.39.0

Stack:

2024-10-25 18:59:34.483444|ERROR|-|-:-:-|-|-:-|-:-|-|adminhttp: panic serving <ip:port>: runtime error: index out of range [43] with length 30|-:-|
2024-10-25 18:59:34.483468|ERROR|-|-:-:-|-|-:-|-:-|-|goroutine 2062665 [running]:|-:-|
2024-10-25 18:59:34.483473|ERROR|-|-:-:-|-|-:-|-:-|-|net/http.(*conn).serve.func1()|-:-|
2024-10-25 18:59:34.483478|ERROR|-|-:-:-|-|-:-|-:-|-|	/go/go1.18/src/net/http/server.go:1825 +0xbf|-:-|
2024-10-25 18:59:34.483482|ERROR|-|-:-:-|-|-:-|-:-|-|panic({0x4510960, 0xc0094bd530})|-:-|
2024-10-25 18:59:34.483487|ERROR|-|-:-:-|-|-:-|-:-|-|	/go/go1.18/src/runtime/panic.go:844 +0x258|-:-|
2024-10-25 18:59:34.483491|ERROR|-|-:-:-|-|-:-|-:-|-|compress/flate.(*huffmanBitWriter).indexTokens(0xc01976a5a0, {0xc01563e000, 0x1641, 0xc012ac6d48?})|-:-|
2024-10-25 18:59:34.483496|ERROR|-|-:-:-|-|-:-|-:-|-|	/go/go1.18/src/compress/flate/huffman_bit_writer.go:551 +0x2a5|-:-|
2024-10-25 18:59:34.483501|ERROR|-|-:-:-|-|-:-|-:-|-|compress/flate.(*huffmanBitWriter).writeBlock(0xc01976a5a0, {0xc01563e000?, 0x39?, 0x1ef8?}, 0x0, {0x0, 0x0, 0x0})|-:-|
2024-10-25 18:59:34.483505|ERROR|-|-:-:-|-|-:-|-:-|-|	/go/go1.18/src/compress/flate/huffman_bit_writer.go:440 +0xcf|-:-|
2024-10-25 18:59:34.483509|ERROR|-|-:-:-|-|-:-|-:-|-|compress/flate.(*compressor).writeBlock(0xc01f7a6000, {0xc01563e000?, 0xc012ac6e68?, 0x59146a?}, 0xc01f7a6000?)|-:-|
2024-10-25 18:59:34.483513|ERROR|-|-:-:-|-|-:-|-:-|-|	/go/go1.18/src/compress/flate/deflate.go:170 +0x9c|-:-|
2024-10-25 18:59:34.483517|ERROR|-|-:-:-|-|-:-|-:-|-|compress/flate.(*compressor).deflate(0xc01f7a6000)|-:-|
2024-10-25 18:59:34.483522|ERROR|-|-:-:-|-|-:-|-:-|-|	/go/go1.18/src/compress/flate/deflate.go:415 +0x6d9|-:-|
2024-10-25 18:59:34.483526|ERROR|-|-:-:-|-|-:-|-:-|-|compress/flate.(*compressor).close(0xc01f7a6000)|-:-|
2024-10-25 18:59:34.483530|ERROR|-|-:-:-|-|-:-|-:-|-|	/go/go1.18/src/compress/flate/deflate.go:646 +0x43|-:-|
2024-10-25 18:59:34.483534|ERROR|-|-:-:-|-|-:-|-:-|-|compress/flate.(*Writer).Close(...)|-:-|
2024-10-25 18:59:34.483538|ERROR|-|-:-:-|-|-:-|-:-|-|	/go/go1.18/src/compress/flate/deflate.go:732|-:-|
2024-10-25 18:59:34.483542|ERROR|-|-:-:-|-|-:-|-:-|-|compress/gzip.(*Writer).Close(0xc0114b5970)|-:-|
2024-10-25 18:59:34.483547|ERROR|-|-:-:-|-|-:-|-:-|-|	/go/go1.18/src/compress/gzip/gzip.go:242 +0x87|-:-|
2024-10-25 18:59:34.483551|ERROR|-|-:-:-|-|-:-|-:-|-|panic({0x4510960, 0xc0094bd518})|-:-|
2024-10-25 18:59:34.483555|ERROR|-|-:-:-|-|-:-|-:-|-|	/go/go1.18/src/runtime/panic.go:844 +0x258|-:-|
2024-10-25 18:59:34.483560|ERROR|-|-:-:-|-|-:-|-:-|-|compress/flate.(*compressor).findMatch(0xc01ecbaa50?, 0x4857785?, 0xc012ac70a8?, 0x4a72d4?, 0x1?)|-:-|
2024-10-25 18:59:34.483564|ERROR|-|-:-:-|-|-:-|-:-|-|	/go/go1.18/src/compress/flate/deflate.go:254 +0x2bf|-:-|
2024-10-25 18:59:34.483568|ERROR|-|-:-:-|-|-:-|-:-|-|compress/flate.(*compressor).deflate(0xc01f7a6000)|-:-|
2024-10-25 18:59:34.483572|ERROR|-|-:-:-|-|-:-|-:-|-|	/go/go1.18/src/compress/flate/deflate.go:443 +0x215|-:-|
2024-10-25 18:59:34.483576|ERROR|-|-:-:-|-|-:-|-:-|-|compress/flate.(*compressor).write(0xc01f7a6000, {0xc0154b6000?, 0x1000, 0xc00a612a50?})|-:-|
2024-10-25 18:59:34.483580|ERROR|-|-:-:-|-|-:-|-:-|-|	/go/go1.18/src/compress/flate/deflate.go:554 +0x82|-:-|
2024-10-25 18:59:34.483584|ERROR|-|-:-:-|-|-:-|-:-|-|compress/flate.(*Writer).Write(...)|-:-|
2024-10-25 18:59:34.483590|ERROR|-|-:-:-|-|-:-|-:-|-|	/go/go1.18/src/compress/flate/deflate.go:712|-:-|
2024-10-25 18:59:34.483595|ERROR|-|-:-:-|-|-:-|-:-|-|compress/gzip.(*Writer).Write(0xc0114b5970, {0xc0154b6000, 0x1000, 0x1000})|-:-|
2024-10-25 18:59:34.483599|ERROR|-|-:-:-|-|-:-|-:-|-|	/go/go1.18/src/compress/gzip/gzip.go:196 +0x34a|-:-|
2024-10-25 18:59:34.483603|ERROR|-|-:-:-|-|-:-|-:-|-|bufio.(*Writer).Flush(0xc0158a51c0)|-:-|
2024-10-25 18:59:34.483613|ERROR|-|-:-:-|-|-:-|-:-|-|	/go/go1.18/src/bufio/bufio.go:628 +0x62|-:-|
2024-10-25 18:59:34.483617|ERROR|-|-:-:-|-|-:-|-:-|-|bufio.(*Writer).WriteString(0xc0158a51c0, {0x4830c59?, 0xc0158a51c0?})|-:-|
2024-10-25 18:59:34.483621|ERROR|-|-:-:-|-|-:-|-:-|-|	/go/go1.18/src/bufio/bufio.go:740 +0x30|-:-|
2024-10-25 18:59:34.483626|ERROR|-|-:-:-|-|-:-|-:-|-|strings.(*byteStringReplacer).WriteString(0xc0000ec600, {0x4f3f280, 0xc0158a51c0}, {0x4830c59, 0x1})|-:-|
2024-10-25 18:59:34.483631|ERROR|-|-:-:-|-|-:-|-:-|-|	/go/go1.18/src/strings/replace.go:565 +0x1ce|-:-|
2024-10-25 18:59:34.483635|ERROR|-|-:-:-|-|-:-|-:-|-|strings.(*Replacer).WriteString(0xc012ac7287?, {0x4f3f280?, 0xc0158a51c0?}, {0x4830c59?, 0x0?})|-:-|
2024-10-25 18:59:34.483639|ERROR|-|-:-:-|-|-:-|-:-|-|	/go/go1.18/src/strings/replace.go:103 +0x90|-:-|
2024-10-25 18:59:34.483643|ERROR|-|-:-:-|-|-:-|-:-|-|github.com/prometheus/common/expfmt.writeEscapedString({0x4f92218?, 0xc0158a51c0}, {0x4830c59, 0x1}, 0x58?)|-:-|
2024-10-25 18:59:34.483648|ERROR|-|-:-:-|-|-:-|-:-|-|	/root/go/pkg/mod/github.com/prometheus/[email protected]/expfmt/text_create.go:424 +0x66|-:-|
2024-10-25 18:59:34.483653|ERROR|-|-:-:-|-|-:-|-:-|-|github.com/prometheus/common/expfmt.writeLabelPairs({0x4f92218, 0xc0158a51c0}, {0xc01ea2fe20, 0x4, 0x0?}, {0x0, 0x0}, 0xc012ac73d0?)|-:-|
2024-10-25 18:59:34.483657|ERROR|-|-:-:-|-|-:-|-:-|-|	/root/go/pkg/mod/github.com/prometheus/[email protected]/expfmt/text_create.go:368 +0x1cd|-:-|
2024-10-25 18:59:34.483662|ERROR|-|-:-:-|-|-:-|-:-|-|github.com/prometheus/common/expfmt.writeSample({0x4f92218, 0xc0158a51c0}, {0x486b5e2?, 0x0?}, {0x0, 0x0}, 0xc01ed29260, {0x0, 0x0}, 0x24?, ...)|-:-|
2024-10-25 18:59:34.483668|ERROR|-|-:-:-|-|-:-|-:-|-|	/root/go/pkg/mod/github.com/prometheus/[email protected]/expfmt/text_create.go:296 +0x105|-:-|
2024-10-25 18:59:34.483673|ERROR|-|-:-:-|-|-:-|-:-|-|github.com/prometheus/common/expfmt.MetricFamilyToText({0x4f3f3a0?, 0xc0114b5970}, 0xc01ee061e0)|-:-|
2024-10-25 18:59:34.483678|ERROR|-|-:-:-|-|-:-|-:-|-|	/root/go/pkg/mod/github.com/prometheus/[email protected]/expfmt/text_create.go:161 +0x985|-:-|
2024-10-25 18:59:34.483682|ERROR|-|-:-:-|-|-:-|-:-|-|github.com/prometheus/common/expfmt.NewEncoder.func7(0x4a454c0?)|-:-|
2024-10-25 18:59:34.483686|ERROR|-|-:-:-|-|-:-|-:-|-|	/root/go/pkg/mod/github.com/prometheus/[email protected]/expfmt/encode.go:144 +0x27|-:-|
2024-10-25 18:59:34.483690|ERROR|-|-:-:-|-|-:-|-:-|-|github.com/prometheus/common/expfmt.encoderCloser.Encode(...)|-:-|
2024-10-25 18:59:34.483695|ERROR|-|-:-:-|-|-:-|-:-|-|	/root/go/pkg/mod/github.com/prometheus/[email protected]/expfmt/encode.go:50|-:-|
2024-10-25 18:59:34.483700|ERROR|-|-:-:-|-|-:-|-:-|-|github.com/prometheus/client_golang/prometheus/promhttp.HandlerForTransactional.func1({0x7f73371cf7e0, 0xc01e189cc0}, 0xc01c905000)|-:-|
2024-10-25 18:59:34.483704|ERROR|-|-:-:-|-|-:-|-:-|-|	/root/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/http.go:205 +0x7dc|-:-|
2024-10-25 18:59:34.483708|ERROR|-|-:-:-|-|-:-|-:-|-|net/http.HandlerFunc.ServeHTTP(0x4f478e0?, {0x7f73371cf7e0?, 0xc01e189cc0?}, 0xc01e292ae0?)|-:-|
2024-10-25 18:59:34.483712|ERROR|-|-:-:-|-|-:-|-:-|-|	/go/go1.18/src/net/http/server.go:2084 +0x2f|-:-|
2024-10-25 18:59:34.483717|ERROR|-|-:-:-|-|-:-|-:-|-|github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerInFlight.func1({0x7f73371cf7e0, 0xc01e189cc0}, 0x4f90900?)|-:-|
2024-10-25 18:59:34.483721|ERROR|-|-:-:-|-|-:-|-:-|-|	/root/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:60 +0xd4|-:-|
2024-10-25 18:59:34.483727|ERROR|-|-:-:-|-|-:-|-:-|-|net/http.HandlerFunc.ServeHTTP(0x4f90900?, {0x7f73371cf7e0?, 0xc01e189cc0?}, 0x0?)|-:-|
2024-10-25 18:59:34.483731|ERROR|-|-:-:-|-|-:-|-:-|-|	/go/go1.18/src/net/http/server.go:2084 +0x2f|-:-|
2024-10-25 18:59:34.483735|ERROR|-|-:-:-|-|-:-|-:-|-|github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1({0x4f90900?, 0xc00056f180?}, 0xc01c905000)|-:-|
2024-10-25 18:59:34.483742|ERROR|-|-:-:-|-|-:-|-:-|-|	/root/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:146 +0xb8|-:-|
2024-10-25 18:59:34.483747|ERROR|-|-:-:-|-|-:-|-:-|-|net/http.HandlerFunc.ServeHTTP(0xc002a94c30?, {0x4f90900?, 0xc00056f180?}, 0x0?)|-:-|
2024-10-25 18:59:34.483751|ERROR|-|-:-:-|-|-:-|-:-|-|	/go/go1.18/src/net/http/server.go:2084 +0x2f|-:-|
2024-10-25 18:59:34.483756|ERROR|-|-:-:-|-|-:-|-:-|-|<mask>/internal.(*adminService).metricsToggleable.func1({0x4f90900, 0xc00056f180}, 0xeb5ac1f9490c?)|-:-|
2024-10-25 18:59:34.483760|ERROR|-|-:-:-|-|-:-|-:-|-|	/root/go/pkg/mod/<mask>/internal/admin_mod.go:305 +0x89|-:-|
2024-10-25 18:59:34.483764|ERROR|-|-:-:-|-|-:-|-:-|-|net/http.HandlerFunc.ServeHTTP(0x7f7370415908?, {0x4f90900?, 0xc00056f180?}, 0x40f3e5?)|-:-|
2024-10-25 18:59:34.483768|ERROR|-|-:-:-|-|-:-|-:-|-|	/go/go1.18/src/net/http/server.go:2084 +0x2f|-:-|
2024-10-25 18:59:34.483773|ERROR|-|-:-:-|-|-:-|-:-|-|net/http.(*ServeMux).ServeHTTP(0x0?, {0x4f90900, 0xc00056f180}, 0xc01c905000)|-:-|
2024-10-25 18:59:34.483777|ERROR|-|-:-:-|-|-:-|-:-|-|	/go/go1.18/src/net/http/server.go:2462 +0x149|-:-|
2024-10-25 18:59:34.483781|ERROR|-|-:-:-|-|-:-|-:-|-|net/http.serverHandler.ServeHTTP({0xc01e2925a0?}, {0x4f90900, 0xc00056f180}, 0xc01c905000)|-:-|
2024-10-25 18:59:34.483785|ERROR|-|-:-:-|-|-:-|-:-|-|	/go/go1.18/src/net/http/server.go:2916 +0x43b|-:-|
2024-10-25 18:59:34.483789|ERROR|-|-:-:-|-|-:-|-:-|-|net/http.(*conn).serve(0xc008076640, {0x4f922f8, 0xc00dcd6690})|-:-|
2024-10-25 18:59:34.483794|ERROR|-|-:-:-|-|-:-|-:-|-|	/go/go1.18/src/net/http/server.go:1966 +0x5d7|-:-|
2024-10-25 18:59:34.483798|ERROR|-|-:-:-|-|-:-|-:-|-|created by net/http.(*Server).Serve|-:-|
2024-10-25 18:59:34.483803|ERROR|-|-:-:-|-|-:-|-:-|-|	/go/go1.18/src/net/http/server.go:3071 +0x4db|-:-|

huanghy6268 avatar Oct 28 '24 10:10 huanghy6268

👋🏽 Thanks!

First of all, it would be nice if you could upgrade to the latest client_golang. Then we might need to ask for some reproduction tests.

@vesari actually found a very similar (if not same) issue related to gzip https://github.com/golang/go/issues/40008#issuecomment-653198373 where the root cause was that gzip writer was used concurrently. Sincer our code is not using response writer concurrently maybe your middleware or server setup causes concurrent response writer use? 🤔 Do you mind giving us more context on how promhttp is used (ideally in the small isolate go test).

bwplotka avatar Dec 16 '24 14:12 bwplotka

Hello 👋 Looks like there was no activity on this issue for the last 3 months. Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗 If there will be no activity in the next 4 weeks, this issue will be closed (we can always reopen an issue if we need!).

stale[bot] avatar Jul 19 '25 06:07 stale[bot]