buger
json parser not closing goroutine?
Hi,
Thank you for the amazing library. I have a simple webserver which when gets a list of n(n <=35) IP addresses, spawns n goroutines, makes a GET requests and parses the JSON responses using json.parser and again waits for new request. Binary should therefore use memory/CPU only when it gets request and use almost 0 CPU/goroutine memory usage when it receives no request. I see it's behaviour as expected initially when the binary starts but after a couple of hours or a day, when I do htop, I see some goroutines still running with memory usage and CPU usage when it is not addressing any request. I started digging in to see what goroutines are running by adding a server endpoint which does this e:=pprof.Lookup("goroutine").WriteTo(w, 2) where w is http.ResponseWriter and with debug=2. When I look at the runtime log, I see bunch of following goroutines running with following log--
goroutine 20441 [runnable]:
github.com/buger/jsonparser.stringEnd(0x186f3e6, 0x1ce, 0xa1a, 0xb, 0x39384)
~/goprojects/src/github.com/buger/jsonparser/parser.go:152 +0x1c
github.com/buger/jsonparser.blockEnd(0x186ee6b, 0x749, 0xf95, 0x7b7d0000, 0xffffffff)
~/goprojects/src/github.com/buger/jsonparser/parser.go:190 +0x114
github.com/buger/jsonparser.searchKeys(0x186c000, 0x35b4, 0x3e00, 0x17b161c, 0x1, 0x1, 0x142b4a0)
~/goprojects/src/github.com/buger/jsonparser/parser.go:286 +0x82c
github.com/buger/jsonparser.internalGet(0x186c000, 0x35b4, 0x3e00, 0x17b161c, 0x1, 0x1, 0x0, 0xb8000005, 0x58f6a0, 0x3cd524, ...)
~/goprojects/src/github.com/buger/jsonparser/parser.go:876 +0x1f8
github.com/buger/jsonparser.Get(0x186c000, 0x35b4, 0x3e00, 0x17b161c, 0x1, 0x1, 0x186d345, 0x5, 0x2abb, 0x1, ...)
~/goprojects/src/github.com/buger/jsonparser/parser.go:870 +0x64
github.com/anugur/myproject/API.(*DeviceAPI).ParseResponse(0xaccf5c, 0x186c000, 0x35b4, 0x3e00, 0x35b4, 0x162e30a, 0x5, 0x1758a60, 0x11, 0x1872130, ...)
~/goprojects/src/github.com/anugur/myproject/API/BaseAPI.go:370 +0x1f88
I see goroutine with runnable event, even when it does not get any request. At line 370 of my code, all I do is data, _, _, err:=jsonparser.Get(read_values, "jsondata") where readvalues is byte array of following json-
{
"jsondata":{very big json }
}
Once I get data, I don't do any parsing on it, I just store it as it is. Also, I added waitgroups to each goroutine and have a print statement after wg.Wait(). As expected, after every request, wg.Wait() hits and print statement prints as expected, which means goroutines have been done. but maybe some of the requests didnt go through wg.Wait(), since I also see this-
goroutine 56590 [semacquire, 842 minutes]:
sync.runtime_Semacquire(0x162f258)
/usr/local/go/src/runtime/sema.go:56 +0x48
sync.(*WaitGroup).Wait(0x162f250)
/usr/local/go/src/sync/waitgroup.go:130 +0xbc
main.(*LOGAgent).Log(0x14d8f30, 0x17c6a40)
~/goprojects/src/github.com/anugur/myproject/Agents/Logs.go:328 +0x6e0
reflect.Value.call(0x5f2860, 0x14d8f30, 0x1213, 0x5fea25, 0x4, 0x1606cf0, 0x1, 0x1, 0x1678768, 0x13, ...)
By Looking at timestamp( 842 minutes), I feel maybe some goroutine didn't finish and hit wg.Done(), 842 minutes ago and is stuck in a infinite loop at jsonparser.stringEnd-buger/jsonparser/parser.go:152
Other running goroutines, which the debug showed were server related. Do you feel it is anything to do with the way json parser copies a very big Json? Is there a simple way to just copy a json element as it is without more resource computation?
PS: I don't see any memory growth or CPU usage over time as of now but I want to optimise my server functionality by running only when it gets a request because it runs on a embedded platform. I also trimmed most of the details to focus more about the issue. Tested on golang 1.12.1 and version of jsonparser is the one I fetched 2 week ago from go get.
I figured out that I had 2 devices with the same addresses due to which I was receiving a malformed JSON. Json parser library fails and gets stuck indefinitely when it encounters such malformed JSON. I could replicate it by writing sample code and running it with go 1.12.1. For now, I am checking the validity of JSON before sending it through JSON-parser. If anyone would like to look at it here's the sample code-
package main
import (
"encoding/json"
"fmt"
"github.com/buger/jsonparser"
)
var re=[]byte(`{"a":{"b":{"c":{`)
func main() {
unmarshallhere:=make(map[string]interface{})
d, _, _, err := jsonparser.Get(re, "d")
fmt.Println("It never comes here")
if err == nil {
json.Unmarshal(d, &unmarshallhere)
} else {
fmt.Println("got parsing error")
}
}
One of my colleagues helped me trace out that this commit on Oct 23,2018 broke the functioning. master versions before it could handle my test case without blocking.
Thanks, Aditya