redis plugin: redis server downtime not handled gracefully

[infomofo]

I'd like to use a redis cache for content caching in my play server.

I have a play action that i'm wrapping with the Cached action, i.e.

 def action(input: String) = Cached.status(_ => "cachekey", 200, 43200) {
    Action {implicit request =>

The play plugin for redis works great. However i realized I'm introducing a new point of failure into my system.

If the redis server ever becomes unavailable, then my Cached server action call throws a hard exception

Caused by: redis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketException: Broken pipe
    at redis.clients.jedis.Connection.flush(Connection.java:70) ~[jedis-2.4.2.jar:na]
    at redis.clients.jedis.Connection.getStatusCodeReply(Connection.java:173) ~[jedis-2.4.2.jar:na]
    at redis.clients.jedis.Jedis.set(Jedis.java:54) ~[jedis-2.4.2.jar:na]
    at com.typesafe.plugin.RedisPlugin$$anon$1$$anonfun$set_$2.apply(RedisPlugin.scala:145) ~[play-plugins-redis_2.11-2.3.1.jar:2.3.1]
    at com.typesafe.plugin.RedisPlugin$$anon$1$$anonfun$set_$2.apply(RedisPlugin.scala:144) ~[play-plugins-redis_2.11-2.3.1.jar:2.3.1]
    at org.sedis.Pool.withJedisClient(sedis.scala:103) ~[sedis_2.11-1.2.2.jar:na]
    at com.typesafe.plugin.RedisPlugin$$anon$1.set_(RedisPlugin.scala:144) ~[play-plugins-redis_2.11-2.3.1.jar:2.3.1]
    at com.typesafe.plugin.RedisPlugin$$anon$1.set(RedisPlugin.scala:103) ~[play-plugins-redis_2.11-2.3.1.jar:2.3.1]
    at play.api.cache.Cache$.set(Cache.scala:61) ~[play-cache_2.11-2.3.9.jar:2.3.9]

Is there a recommended pattern to deal with redis server downtime gracefully using this plugin and pattern? Ideally I would like the play caching api itself to log a hard exception but return the result as if it were a cache miss.

[infomofo]

I've even tried to create a helper partial function to wrap the status of the cached call:

  def safeCache(key: String, ttl: Int)(f: => Result)(implicit app: Application) = {
    try {
      Cached.status(_ => key, 200, ttl) {
        Action { implicit request =>
          f
        }
      }
    } catch {
//      case e: redis.clients.jedis.exceptions.JedisConnectionException =>
      case e: Throwable =>
        Logger.error(s"Could not retrieve cache key for $key due to ${e.getMessage}", e)
        NewRelic.noticeError(e)
        Action { implicit request =>
          f
        }
    }
  }

But the actual error seems to be thrown from an iteratee outside my control so I can't even catch this exception:

Caused by: redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
        at redis.clients.util.Pool.getResource(Pool.java:42) ~[jedis-2.4.2.jar:na]
        at org.sedis.Pool.withJedisClient(sedis.scala:101) ~[sedis_2.11-1.2.2.jar:na]
        at com.typesafe.plugin.RedisPlugin$$anon$1.set_(RedisPlugin.scala:144) ~[play-plugins-redis_2.11-2.3.1.jar:2.3.1]
        at com.typesafe.plugin.RedisPlugin$$anon$1.set(RedisPlugin.scala:103) ~[play-plugins-redis_2.11-2.3.1.jar:2.3.1]
        at play.api.cache.Cache$.set(Cache.scala:61) ~[play-cache_2.11-2.3.9.jar:2.3.9]
        at play.api.cache.Cache$.set(Cache.scala:72) ~[play-cache_2.11-2.3.9.jar:2.3.9]
        at play.api.cache.Cached$$anonfun$play$api$cache$Cached$$handleResult$1.apply(Cached.scala:83) ~[play-cache_2.11-2.3.9.jar:2.3.9]
        at play.api.cache.Cached$$anonfun$play$api$cache$Cached$$handleResult$1.apply(Cached.scala:73) ~[play-cache_2.11-2.3.9.jar:2.3.9]
        at scala.PartialFunction$AndThen.applyOrElse(PartialFunction.scala:190) ~[scala-library-2.11.6.jar:na]
        at play.api.cache.Cached.play$api$cache$Cached$$handleResult(Cached.scala:88) ~[play-cache_2.11-2.3.9.jar:2.3.9]
        at play.api.cache.Cached$$anonfun$build$1$$anonfun$apply$6$$anonfun$apply$7.apply(Cached.scala:57) ~[play-cache_2.11-2.3.9.jar:2.3.9]
        at play.api.cache.Cached$$anonfun$build$1$$anonfun$apply$6$$anonfun$apply$7.apply(Cached.scala:57) ~[play-cache_2.11-2.3.9.jar:2.3.9]
        at play.api.libs.iteratee.Iteratee$$anonfun$map$1.apply(Iteratee.scala:471) ~[play-iteratees_2.11-2.3.9.jar:2.3.9]
        at play.api.libs.iteratee.Iteratee$$anonfun$map$1.apply(Iteratee.scala:471) ~[play-iteratees_2.11-2.3.9.jar:2.3.9]
        at play.api.libs.iteratee.Iteratee$$anonfun$flatMap$1$$anonfun$apply$13.apply(Iteratee.scala:495) ~[play-iteratees_2.11-2.3.9.jar:2.3.9]
        at play.api.libs.iteratee.Iteratee$$anonfun$flatMap$1$$anonfun$apply$13.apply(Iteratee.scala:495) ~[play-iteratees_2.11-2.3.9.jar:2.3.9]
        at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24) [scala-library-2.11.6.jar:na]
        at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24) [scala-library-2.11.6.jar:na]

[jroper]

You could submit a pull request with your suggested improvement. As for why your work around doesn't work, it's because iteratees are asynchronous. If you want to recover, you need to invoke recover on the iteratee, you can't just catch a synchronous exception from asynchronous code.

[kliewkliew]

https://github.com/playframework/play-plugins/pull/172 I have submitted a pull request that handles client redirection upon failover using Redis Sentinel

[rmmeans]

@infomofo this was a simple bug that I resolved in this PR: https://github.com/playframework/play-plugins/pull/171

However, since this repo is not actively maintained anymore I was encouraged on my last PR before that one to consider taking over this repo. I have forked the repo and moved it here (still in process of updating documentation on this original repo) and have published the fix for this issue you referenced here.

https://github.com/lifeway/play-redis