Elasendra Error - shard or keyspace unavailable
We have three node elasendra cluster.We upgraded it recently, but after that I can see indices on each node are red. cluster state is showing error like: shard or keyspace unavailable . We tried to delete indices but it got hung and executing from last 30 min, so canceled it.
Elasendra version: 6.8.4.16
Errors in log:
[f_historical_stories_qa_2020_03_30][0], node[0fObMI6wRVqgLlsNtKHeCA], [P], recovery_source[local shards recovery], s[UNASSIGNED], unassigned_info[[reason=ALLOCATION_FAILED], at[1970-01-01T00:00:00.000Z], failed_attempts[1], delayed=false, details[shard or keyspace unavailable], allocation_status[deciders_no]]
curl -XGET 'http://localhost:9200/_cluster/state/?pretty=true'
{
"state" : "UNASSIGNED",
"primary" : true,
"node" : "2c19J9KBSlWuOA_iSwufGw",
"relocating_node" : null,
"shard" : 0,
"index" : "h_historical_stories_qa_2022_04_22",
"token_ranges" : [
"(-9223372036854775808,9142068006140963677]"
],
"recovery_source" : {
"type" : "LOCAL_SHARDS"
},
"unassigned_info" : {
"reason" : "ALLOCATION_FAILED",
"at" : "1970-01-01T00:00:00.000Z",
"failed_attempts" : 1,
"delayed" : false,
"details" : "shard or keyspace unavailable",
"allocation_status" : "deciders_no"
}
Appreciate for your help.
Thanks