I think you need use dagger's dependency injection to get rid of the factory pattern.

[ltn614]

Since you use dagger's dependency injection, it's all about to provide object instance when you need, so why bother use factory pattern