Right now a node is removed from shard-to-node mappings in the Client if an proxy-invocations fails.
The node-proxy itself, which gets removed as well, might be re-established if a new shard is added, but the removed shard-to-node mappings are never re-established for that client.
Now a failing proxy-invocation does not necessarily mean that the proxy is corrupt, see KATTA-180 as a example.
So following approach would look a bit safer:
- remove a node-proxy only if x successive invocations are failed
- re-establish the proxy immediatly
- if the re-established proxy fails - remove the shard-to-node mapping
a) if 3 successive proxy operations are failing
b) if 1 proxy operation is failing with an exception which indicates a connection problem