facebook
Help. Replicated pool and reliable delete stream
Hi! I need some help to configure mcrouter. Here is the thing, I have about 20 webservers running memcached locally and I need to replicate data between them. My get policy is to always look localhost first and then another server, so MissFailoverRoute works for me. But I'm facing some difficulties to get reliable delete stream running. I've tried this: (my test environment with just few servers)
{
"pools": {
"commerce-webs": {
"servers": [
"192.168.122.165:11212",
"192.168.122.202:11212",
"192.168.122.224:11212",
"192.168.122.32:11212"
]
},
"local": {
"servers": [
"127.0.0.1:11212"
]
}
},
"route": {
"type": "OperationSelectorRoute",
"operation_policies": {
"set": "AllSyncRoute|Pool|commerce-webs",
"add": "AllSyncRoute|Pool|commerce-webs",
"delete": "AllSyncRoute|Pool|commerce-webs",
"get": {
"type": "MissFailoverRoute",
"children": [
"PoolRoute|local",
"PoolRoute|commerce-webs"
]
},
}
}
}
That config works fine to set,get and delete keys. But if I disconnect any server to test the delete stream, the asynclog expected to be in /var/spool/mcrouter is never created. So I've tried this:
{
"pools": {
"commerce-webs": {
"servers": [
"192.168.122.165:11212",
"192.168.122.202:11212",
"192.168.122.224:11212",
"192.168.122.32:11212"
]
},
"local": {
"servers": [
"127.0.0.1:11212"
]
}
},
"route": {
"type": "OperationSelectorRoute",
"operation_policies": {
"set": "AllSyncRoute|Pool|commerce-webs",
"add": "AllSyncRoute|Pool|commerce-webs",
"get": {
"type": "MissFailoverRoute",
"children": [
"PoolRoute|local",
"PoolRoute|commerce-webs"
]
},
"delete": {
"type": "AllSyncRoute",
"children": [
"PoolRoute|commerce-webs"
]
}
}
}
}
When I set the delete operation with PoolRoute, some times the asynclog is created, but the deletes is not replicated to all servers in the pool.
I don't know what I'm doing wrong here. Can anyone point me what I am missing?
Thanks, and sorry any language error... not english speaker
Just to clear things, the behavior I've got is:
- Delete command using "AllSyncRoute|Pool|poolname":
- When all servers are accessible, everything works. The key is deleted from all servers in pool.
- When some server is unreachable, the key is deleted from all servers but the sick server. Mcrouter returns 'server timeout'.The expected asynclog with the failure information is not created.
- Delete command using "AllSyncRoute|PoolRoute|poolname":
- Deletion occurs only in one or two servers in the pool. Mcrouter returns 'DELETED' sometimes the asynclog is created, sometimes not.
I'm confused because the documentation says:
"By default, mcrouter will save a record of every delete command it fails to deliver
downstream (either due network issues or an error response from a destination). "
https://github.com/facebook/mcrouter/wiki/Features#reliable-delete-stream
But, I can't make it work like this.
Thanks
@andreazevedo Desculpe marcar você, mas como vi que é brasileiro e meu inglês é bem fraquinho, talvez você possa me dar uma ajuda. Valeu!
Hi @gustavoantao, if I understood correctly, you want the cache data to be identical to all the machines in commerce-webs, e.g. 192.168.122.165:11212, 192.168.122.202:11212, etc. and that you want the deletes to go to all of them.
I think the problem is because you have PoolRoute setup for all the other servers as a single pool:
"commerce-webs": {
"servers": [
"192.168.122.165:11212",
"192.168.122.202:11212",
"192.168.122.224:11212",
"192.168.122.32:11212"
]
},
This will make mcrouter hash the key and only send it to a machine in the pool based on the hash of key. I think you would need to create a separate PoolRoute for each of the individual machines for the delete to be replicates to all those machines. The async-log should be created in that case for the machine that is down.
Yes, @djvaporize is right!
Your first solution broadcast deletes to all servers, but reliable delete stream doesn't work because it needs PoolRoute.
The second solution, on the other hand, enables the reliable delete stream (because it uses PoolRoute), but it doesn't broadcast the deletes to all servers (because PoolRoute will hash the key to choose a server).
For it to work the way you want, you will need to have one PoolRoute wrapping each server, as @djvaporize mentioned. It would look something like this:
{
"pools": {
"commerce-webs": {
"servers": [
"192.168.122.165:11212",
"192.168.122.202:11212",
"192.168.122.224:11212",
"192.168.122.32:11212"
]
},
"local": {
"servers": [
"127.0.0.1:11212"
]
},
"server-1": {
"servers": [
"192.168.122.165:11212",
]
},
"server-2": {
"servers": [
"192.168.122.202:11212",
]
},
...
},
"route": {
"type": "OperationSelectorRoute",
"operation_policies": {
"set": "AllSyncRoute|Pool|commerce-webs",
"add": "AllSyncRoute|Pool|commerce-webs",
"delete": {
"type": "AllSyncRoute",
"children": [
"PoolRoute|local",
"PoolRoute|server-1",
"PoolRoute|server-2",
...
}
"get": {
"type": "MissFailoverRoute",
"children": [
"PoolRoute|local",
"PoolRoute|commerce-webs"
]
},
}
}
}
Hi guys! Thank you very much. Now I understood the logic here. I'll make some tests with this config. Now I just want to figure out a way to setup theese single server poolls automatically, maybe playing with macros... I'll try and come back with my results.
Thanks again!!
Update
Thank you again guys! mcrouter is working perfectly here in my testing environment. I just want to share what I've done, maybe could help someone else in the future.
1 - File servers.json.
Here I put all my servers in single routes, even local route
{
"local": { "servers": ["127.0.0.1:11212"] },
"commerce1": { "servers": ["192.168.122.165:11212"] },
"commerce2": { "servers": ["192.168.122.202:11212"] },
"commerce3": { "servers": ["192.168.122.224:11212"] },
"commerce4": { "servers": ["192.168.122.32:11212"] }
}
2 - mcrouter.json.
And then with some macros I import these servers using PoolRoutes instead of Pool
{
"macros": {
"all-pools": {
"type": "constDef",
"result": "@import(file:/etc/mcrouter/servers.json)"
},
"pool-commerce": {
"type": "constDef",
"result" : {
"type" : "foreach",
"from" : "@sort(@keys(%all-pools%))",
"where": "@contains(%item%,commerce)",
"use" : [ "PoolRoute|%item%" ]
}
},
"pool-local": {
"type": "constDef",
"result" : {
"type" : "foreach",
"from" : "@sort(@keys(%all-pools%))",
"where": "@equals(%item%,local)",
"use" : [ "PoolRoute|%item%" ]
}
}
},
"pools": "%all-pools%",
"route": {
"type": "OperationSelectorRoute",
"operation_policies": {
"set": {
"type": "AllSyncRoute",
"children": "%pool-commerce%"
},
"add": {
"type": "AllSyncRoute",
"children": "%pool-commerce%"
},
"delete": {
"type": "AllSyncRoute",
"children": [
"%pool-local%",
"%pool-commerce%"
]
},
"get": {
"type": "MissFailoverRoute",
"children": [
"%pool-local%",
"%pool-commerce%"
]
}
}
}
}
:+1: