Panic caused by concurrent map iteration and map write

[dlmiddlecote]

What's going wrong?

Creating or deleting fields is causing one of our pilosa nodes to panic and crash.

fatal error: concurrent map iteration and map write

goroutine 1727583 [running]:
runtime.throw(0xe6062c, 0x26)
	/usr/local/go/src/runtime/panic.go:1116 +0x72 fp=0xc0adf09080 sp=0xc0adf09050 pc=0x437732
runtime.mapiternext(0xc0adf09168)
	/usr/local/go/src/runtime/map.go:853 +0x554 fp=0xc0adf09100 sp=0xc0adf09080 pc=0x410cf4
github.com/pilosa/pilosa.(*Index).AvailableShards(0xc054bcac80, 0x0)
	/go/pilosa/index.go:276 +0x1bf fp=0xc0adf091e8 sp=0xc0adf09100 pc=0x95d73f

This happens because there is no mutex locked when this code path is executed: https://github.com/pilosa/pilosa/blob/v1.4.1/index.go#L276

This occurs in one place only: https://github.com/pilosa/pilosa/blob/v1.4.1/server.go#L604 This doesn’t use idx.CreateField which manages the mutex, but instead calls idx.createField, which doesn't.

What was expected?

The panic to not happen; the map to be guarded by a mutex in all occurences.

Steps to reproduce the behavior

Create a pilosa cluster with 3 nodes and replication. Query one node, and create a field in the same index on another. When that field is replicated, the node with accepting that createfield will panic and crash when a query is happening at the same time.

Information about your environment (OS/architecture, CPU, RAM, cluster/solo, configuration, etc.)

Pilosa v1.4.1