Race condition when running example

[avcarrillo]

While trying to track down a memory leak with my extension, I found there is a race condition.

When running the example and using the -race flag with go run, there is a race condition

C:\Users..\go\src\test\osquery>go run -race main.go \.\pipe\osquery.em

WARNING: DATA RACE Read at 0x00c0000c6960 by goroutine 10: git.apache.org/thrift.git/lib/go/thrift.(*TStandardClient).Call() C:/Users/../go/src/git.apache.org/thrift.git/lib/go/thrift/client.go:82 +0x59 github.com/kolide/osquery-go/gen/osquery.(*ExtensionClient).Ping() C:/Users/../go/src/github.com/kolide/osquery-go/gen/osquery/osquery.go:1022 +0xfd github.com/kolide/osquery-go/gen/osquery.(*ExtensionManagerClient).Ping() :1 +0x7c github.com/kolide/osquery-go.(*ExtensionManagerClient).Ping() C:/Users/../osquery-go.(*ExtensionManagerServer).Run.func2() C:/Users/../go/src/github.com/kolide/osquery-go/server.go:194 +0x84

Previous write at 0x00c0000c6960 by goroutine 9: git.apache.org/thrift.git/lib/go/thrift.(*TStandardClient).Call() C:/Users/../go/src/git.apache.org/thrift.git/lib/go/thrift/client.go:82 +0x70 github.com/kolide/osquery-go/gen/osquery.(*ExtensionManagerClient).RegisterExtension() C:/Users/../go/src/github.com/kolide/osquery-go/gen/osquery/osquery.go:1850 +0x1d5 github.com/kolide/osquery-go.(*ExtensionManagerClient).RegisterExtension() C:/Users/../go/src/github.com/kolide/osquery-go/client.go:73 +0xa4 github.com/kolide/osquery-go.(*ExtensionManagerServer).Start.func1() C:/Users/../go/src/github.com/kolide/osquery-go/server.go:143 +0x1d1 github.com/kolide/osquery-go.(*ExtensionManagerServer).Start() C:/Users/../go/src/github.com/kolide/osquery-go/server.go:172 +0x55 github.com/kolide/osquery-go.(*ExtensionManagerServer).Run.func1() C:/Users/../go/src/github.com/kolide/osquery-go/server.go:186 +0x3f

Goroutine 10 (running) created at: github.com/kolide/osquery-go.(*ExtensionManagerServer).Run() C:/Users/../go/src/github.com/kolide/osquery-go/server.go:190 +0xa6 main.main() C:/Users/../go/src/test/osquery/main.go:23 +0x30c

Goroutine 9 (running) created at: github.com/kolide/osquery-go.(*ExtensionManagerServer).Run() C:/Users/../go/src/github.com/kolide/osquery-go/server.go:185 +0x7a main.main() C:/Users/../go/src/test/osquery/main.go:23 +0x30c

================== WARNING: DATA RACE Read at 0x00c0000d0095 by goroutine 10: git.apache.org/thrift.git/lib/go/thrift.(*TBinaryProtocol).WriteI32() c:/go/src/encoding/binary/binary.go:116 +0x4f git.apache.org/thrift.git/lib/go/thrift.(*TBinaryProtocol).WriteMessageBegin() C:/Users/../go/src/git.apache.org/thrift.git/lib/go/thrift/binary_protocol.go:78 +0x6f git.apache.org/thrift.git/lib/go/thrift.(*TStandardClient).Send() C:/Users/../go/src/git.apache.org/thrift.git/lib/go/thrift/client.go:37 +0x258 git.apache.org/thrift.git/lib/go/thrift.(*TStandardClient).Call() C:/Users/../go/src/git.apache.org/thrift.git/lib/go/thrift/client.go:85 +0x11d github.com/kolide/osquery-go/gen/osquery.(*ExtensionClient).Ping() C:/Users/../go/src/github.com/kolide/osquery-go/gen/osquery/osquery.go:1022 +0xfd github.com/kolide/osquery-go/gen/osquery.(*ExtensionManagerClient).Ping() :1 +0x7c github.com/kolide/osquery-go.(*ExtensionManagerClient).Ping() C:/Users/../go/src/github.com/kolide/osquery-go/client.go:58 +0x8d github.com/kolide/osquery-go.(*ExtensionManagerServer).Run.func2() C:/Users/../go/src/github.com/kolide/osquery-go/server.go:194 +0x84

Previous write at 0x00c0000d0095 by goroutine 9: git.apache.org/thrift.git/lib/go/thrift.(*TBinaryProtocol).WriteI32() c:/go/src/encoding/binary/binary.go:120 +0xbb git.apache.org/thrift.git/lib/go/thrift.(*TBinaryProtocol).WriteMapBegin() C:/Users/../go/src/git.apache.org/thrift.git/lib/go/thrift/binary_protocol.go:142 +0xc9 github.com/kolide/osquery-go/gen/osquery.(*ExtensionManagerRegisterExtensionArgs).writeField2() C:/Users/../go/src/github.com/kolide/osquery-go/gen/osquery/osquery.go:2746 +0xa84 github.com/kolide/osquery-go/gen/osquery.(*ExtensionManagerRegisterExtensionArgs).Write() C:/Users/../go/src/github.com/kolide/osquery-go/gen/osquery/osquery.go:2717 +0x241 git.apache.org/thrift.git/lib/go/thrift.(*TStandardClient).Send() C:/Users/../go/src/git.apache.org/thrift.git/lib/go/thrift/client.go:40 +0x29f git.apache.org/thrift.git/lib/go/thrift.(*TStandardClient).Call() C:/Users/../go/src/git.apache.org/thrift.git/lib/go/thrift/client.go:85 +0x11d github.com/kolide/osquery-go/gen/osquery.(*ExtensionManagerClient).RegisterExtension() C:/Users/../go/src/github.com/kolide/osquery-go/gen/osquery/osquery.go:1850 +0x1d5 github.com/kolide/osquery-go.(*ExtensionManagerClient).RegisterExtension() C:/Users/../go/src/github.com/kolide/osquery-go/client.go:73 +0xa4 github.com/kolide/osquery-go.(*ExtensionManagerServer).Start.func1() C:/Users/../go/src/github.com/kolide/osquery-go/server.go:143 +0x1d1 github.com/kolide/osquery-go.(*ExtensionManagerServer).Start() C:/Users/../go/src/github.com/kolide/osquery-go/server.go:172 +0x55 github.com/kolide/osquery-go.(*ExtensionManagerServer).Run.func1() C:/Users/../go/src/github.com/kolide/osquery-go/server.go:186 +0x3f

Goroutine 10 (running) created at: github.com/kolide/osquery-go.(*ExtensionManagerServer).Run() C:/Users/../go/src/github.com/kolide/osquery-go/server.go:190 +0xa6 main.main() C:/Users/../go/src/test/osquery/main.go:23 +0x30c

Goroutine 9 (running) created at: github.com/kolide/osquery-go.(*ExtensionManagerServer).Run() C:/Users/../go/src/github.com/kolide/osquery-go/server.go:185 +0x7a main.main() C:/Users/../go/src/test/osquery/main.go:23 +0x30c

================== WARNING: DATA RACE Write at 0x00c0000d0092 by goroutine 10: git.apache.org/thrift.git/lib/go/thrift.(*TBinaryProtocol).WriteI32() c:/go/src/encoding/binary/binary.go:117 +0x66 git.apache.org/thrift.git/lib/go/thrift.(*TBinaryProtocol).WriteMessageBegin() C:/Users/../go/src/git.apache.org/thrift.git/lib/go/thrift/binary_protocol.go:78 +0x6f git.apache.org/thrift.git/lib/go/thrift.(*TStandardClient).Send() C:/Users/../go/src/git.apache.org/thrift.git/lib/go/thrift/client.go:37 +0x258 git.apache.org/thrift.git/lib/go/thrift.(*TStandardClient).Call() C:/Users/../go/src/git.apache.org/thrift.git/lib/go/thrift/client.go:85 +0x11d github.com/kolide/osquery-go/gen/osquery.(*ExtensionClient).Ping() C:/Users/../go/src/github.com/kolide/osquery-go/gen/osquery/osquery.go:1022 +0xfd github.com/kolide/osquery-go/gen/osquery.(*ExtensionManagerClient).Ping() :1 +0x7c github.com/kolide/osquery-go.(*ExtensionManagerClient).Ping() C:/Users/../go/src/github.com/kolide/osquery-go/client.go:58 +0x8d github.com/kolide/osquery-go.(*ExtensionManagerServer).Run.func2() C:/Users/../go/src/github.com/kolide/osquery-go/server.go:194 +0x84

Previous write at 0x00c0000d0092 by goroutine 9: git.apache.org/thrift.git/lib/go/thrift.(*TBinaryProtocol).WriteI32() c:/go/src/encoding/binary/binary.go:117 +0x66 git.apache.org/thrift.git/lib/go/thrift.(*TBinaryProtocol).WriteMapBegin() C:/Users/../go/src/git.apache.org/thrift.git/lib/go/thrift/binary_protocol.go:142 +0xc9 github.com/kolide/osquery-go/gen/osquery.(*ExtensionManagerRegisterExtensionArgs).writeField2() C:/Users/../go/src/github.com/kolide/osquery-go/gen/osquery/osquery.go:2746 +0xa84 github.com/kolide/osquery-go/gen/osquery.(*ExtensionManagerRegisterExtensionArgs).Write() C:/Users/../go/src/github.com/kolide/osquery-go/gen/osquery/osquery.go:2717 +0x241 git.apache.org/thrift.git/lib/go/thrift.(*TStandardClient).Send() C:/Users/../go/src/git.apache.org/thrift.git/lib/go/thrift/client.go:40 +0x29f git.apache.org/thrift.git/lib/go/thrift.(*TStandardClient).Call() C:/Users/../go/src/git.apache.org/thrift.git/lib/go/thrift/client.go:85 +0x11d github.com/kolide/osquery-go/gen/osquery.(*ExtensionManagerClient).RegisterExtension() C:/Users/../go/src/github.com/kolide/osquery-go/gen/osquery/osquery.go:1850 +0x1d5 github.com/kolide/osquery-go.(*ExtensionManagerClient).RegisterExtension() C:/Users/../go/src/github.com/kolide/osquery-go/client.go:73 +0xa4 github.com/kolide/osquery-go.(*ExtensionManagerServer).Start.func1() C:/Users/../go/src/github.com/kolide/osquery-go/server.go:143 +0x1d1 github.com/kolide/osquery-go.(*ExtensionManagerServer).Start() C:/Users/../go/src/github.com/kolide/osquery-go/server.go:172 +0x55 github.com/kolide/osquery-go.(*ExtensionManagerServer).Run.func1() C:/Users/../go/src/github.com/kolide/osquery-go/server.go:186 +0x3f

Goroutine 10 (running) created at: github.com/kolide/osquery-go.(*ExtensionManagerServer).Run() C:/Users/../go/src/github.com/kolide/osquery-go/server.go:190 +0xa6 main.main() C:/Users/../go/src/test/osquery/main.go:23 +0x30c

Goroutine 9 (running) created at: github.com/kolide/osquery-go.(*ExtensionManagerServer).Run() C:/Users/../go/src/github.com/kolide/osquery-go/server.go:185 +0x7a main.main() C:/Users/../go/src/test/osquery/main.go:23 +0x30c

================== WARNING: DATA RACE Write at 0x00c0000d0094 by goroutine 10: git.apache.org/thrift.git/lib/go/thrift.(*TBinaryProtocol).WriteI32() c:/go/src/encoding/binary/binary.go:119 +0x9e git.apache.org/thrift.git/lib/go/thrift.(*TBinaryProtocol).WriteMessageBegin() C:/Users/../go/src/git.apache.org/thrift.git/lib/go/thrift/binary_protocol.go:78 +0x6f git.apache.org/thrift.git/lib/go/thrift.(*TStandardClient).Send() C:/Users/../go/src/git.apache.org/thrift.git/lib/go/thrift/client.go:37 +0x258 git.apache.org/thrift.git/lib/go/thrift.(*TStandardClient).Call() C:/Users/../go/src/git.apache.org/thrift.git/lib/go/thrift/client.go:85 +0x11d github.com/kolide/osquery-go/gen/osquery.(*ExtensionClient).Ping() C:/Users/../go/src/github.com/kolide/osquery-go/gen/osquery/osquery.go:1022 +0xfd github.com/kolide/osquery-go/gen/osquery.(*ExtensionManagerClient).Ping() :1 +0x7c github.com/kolide/osquery-go.(*ExtensionManagerClient).Ping() C:/Users/../go/src/github.com/kolide/osquery-go/client.go:58 +0x8d github.com/kolide/osquery-go.(*ExtensionManagerServer).Run.func2() C:/Users/../go/src/github.com/kolide/osquery-go/server.go:194 +0x84

Previous write at 0x00c0000d0094 by goroutine 9: git.apache.org/thrift.git/lib/go/thrift.(*TBinaryProtocol).WriteI32() c:/go/src/encoding/binary/binary.go:119 +0x9e git.apache.org/thrift.git/lib/go/thrift.(*TBinaryProtocol).WriteMapBegin() C:/Users/../go/src/git.apache.org/thrift.git/lib/go/thrift/binary_protocol.go:142 +0xc9 github.com/kolide/osquery-go/gen/osquery.(*ExtensionManagerRegisterExtensionArgs).writeField2() C:/Users/../go/src/github.com/kolide/osquery-go/gen/osquery/osquery.go:2746 +0xa84 github.com/kolide/osquery-go/gen/osquery.(*ExtensionManagerRegisterExtensionArgs).Write() C:/Users/../go/src/github.com/kolide/osquery-go/gen/osquery/osquery.go:2717 +0x241 git.apache.org/thrift.git/lib/go/thrift.(*TStandardClient).Send() C:/Users/../go/src/git.apache.org/thrift.git/lib/go/thrift/client.go:40 +0x29f git.apache.org/thrift.git/lib/go/thrift.(*TStandardClient).Call() C:/Users/../go/src/git.apache.org/thrift.git/lib/go/thrift/client.go:85 +0x11d github.com/kolide/osquery-go/gen/osquery.(*ExtensionManagerClient).RegisterExtension() C:/Users/../go/src/github.com/kolide/osquery-go/gen/osquery/osquery.go:1850 +0x1d5 github.com/kolide/osquery-go.(*ExtensionManagerClient).RegisterExtension() C:/Users/../go/src/github.com/kolide/osquery-go/client.go:73 +0xa4 github.com/kolide/osquery-go.(*ExtensionManagerServer).Start.func1() C:/Users/../go/src/github.com/kolide/osquery-go/server.go:143 +0x1d1 github.com/kolide/osquery-go.(*ExtensionManagerServer).Start() C:/Users/../go/src/github.com/kolide/osquery-go/server.go:172 +0x55 github.com/kolide/osquery-go.(*ExtensionManagerServer).Run.func1() C:/Users/../go/src/github.com/kolide/osquery-go/server.go:186 +0x3f

Goroutine 10 (running) created at: github.com/kolide/osquery-go.(*ExtensionManagerServer).Run() C:/Users/../go/src/github.com/kolide/osquery-go/server.go:190 +0xa6 main.main() C:/Users/../go/src/test/osquery/main.go:23 +0x30c

Goroutine 9 (running) created at: github.com/kolide/osquery-go.(*ExtensionManagerServer).Run() C:/Users/../go/src/github.com/kolide/osquery-go/server.go:185 +0x7a main.main() C:/Users/../go/src/test/osquery/main.go:23 +0x30c

exit status 2

[mbhatt1]

Any movement on this?

[zwass]

Are you running into an issue due to this? The race looks to be deep into the Thrift code.

[mbhatt1]

Hit it once before.

[0xpaulbenoit]

I have experienced osquery extension issues due to this as well

[avcarrillo]

@zwass , Yes, it causes a memory leak which grows over time. I have to continually restart the service to keep it from growing out of control

[avcarrillo]

Same as @zwass, It causes a memory leak, and I have to continually restart the service. The more queries I execute against my osquery-go extension, the quicker the leak grows

[lucasmrod]

AFAICS this was fixed by https://github.com/osquery/osquery-go/pull/108

git checkout d2e851b3991b0728fdacce7e0f3a2d51a7f524cb # this is the commit before #108

$ cd examples/table

$ go build -race -o main main.go

$ sudo ./main --socket /var/osquery/osquery.em
➜  table git:(d2e851b) ✗ sudo ./main --socket /var/osquery/osquery.em
==================
WARNING: DATA RACE
Read at 0x00c000017050 by goroutine 9:
  github.com/apache/thrift/lib/go/thrift.(*TStandardClient).Call()
      /Users/luk/gopath/pkg/mod/github.com/apache/[email protected]/lib/go/thrift/client.go:89 +0x93
  github.com/osquery/osquery-go/gen/osquery.(*ExtensionClient).Ping()
      /Users/luk/fleetdm/git/osquery-go/gen/osquery/osquery.go:1239 +0xe1
  github.com/osquery/osquery-go/gen/osquery.(*ExtensionManagerClient).Ping()
      <autogenerated>:1 +0x67
  github.com/osquery/osquery-go.(*ExtensionManagerClient).Ping()
      /Users/luk/fleetdm/git/osquery-go/client.go:47 +0x6b
  github.com/osquery/osquery-go.(*ExtensionManagerServer).Run.func2()
      /Users/luk/fleetdm/git/osquery-go/server.go:256 +0x7e

Now when trying in master as of today there's no race condition:

git log -n 1
dbeefc07d76e3edaf614647a20cc588c1b4f47e4

go build -race -o main main.go
sudo ./main --socket /var/osquery/osquery.em
<no output and table plugin example_table works as expected>

[lucasmrod]

@directionless @RebeccaMahany

[RebeccaMahany]

@lucasmrod thanks for looking into this -- I agree that this looks like it was fixed by locking client access. Probably good to close this out?