Data race on InMemoryChannel.Send during shutdown

[davecheney]

inMemoryChannelState.stop closes collectChan and controlChan then nil's them out

// Part of channel accept loop: Clean up and close telemetry channel
func (state *inMemoryChannelState) stop() {
	close(state.channel.collectChan)
	close(state.channel.controlChan)

	state.channel.collectChan = nil
	state.channel.controlChan = nil

	// Throttle can't close until transmitters are done using it.
	state.channel.waitgroup.Wait()
	state.channel.throttle.Stop()

	state.channel.throttle = nil
}

This is used as a signal to Send to elide the item.

// Queues a single telemetry item
func (channel *InMemoryChannel) Send(item *contracts.Envelope) {
	if item != nil && channel.collectChan != nil {
		channel.collectChan <- item
	}
}

However this is not safe to do in the presence of multiple goroutines. Assigning nil to channel.collectChan does not involve a memory barrier and it is not guaranteed by the memory model that other goroutines will see that collectChan is nil. This means the send may occur on a closed channel, or the send may occur on a nil channel, which blocks forever.