Data importing too slow without threading

Open xros opened this issue 10 years ago • 2 comments

I found it would be way slow if importing massive data into ES.

By default jsonpyes does not use Epoll, so it runs on one thread. The I/O of it is very slow.

It would take more than 5 hours to import 3GB data to ES.

There should be an improvement.

Dec 28 '15 19:12 xros

This was solved by me. https://github.com/xros/json-py-es/pull/3

But the efficiency was not improved too much. Maybe it works greater in good servers.

Let's see. :smiley:

Dec 29 '15 05:12 xros

almost 5 times faster than before.

Here's the pull request. https://github.com/xros/jsonpyes/pull/17

Thanks to all people's contribution to this project :smiley:

Mar 30 '16 17:03 xros