Add Elasticsearch Bulk API, HTTP Server Source

[rwaweber]

Current Vector Version

Latest I suppose?

Use-cases

Being able to receive events from clients that already know how to write to elasticsearch's Bulk API, like elastic beats, logstash, or other tools/libraries.

An example use case, could be to run a vector ES Bulk API Server to receive events from a winlogbeat client which would use an elasticsearch output[2], as a means to potentially build a workaround for [1] (though admittedly a windows eventlog source for vector would likely be a better solution in the long run, though I'd wager is likely a lot more difficult to implement).

This would be extremely helpful for us, as we are doing something similar to this, but with a handful of logstash instances behind a loadbalancer with a corresponding beats inputs[3]. This doesn't perform particularly well because the custom beats protocol doesn't loadbalance particularly well[4] as part of it being a stateful TCP connection.

In theory, with an ES bulk API source, we'd be able to place a vector instance behind a loadbalancer and turn those same beats clients to this loadbalancer, and theoretically get better performance over HTTP(assumption being that HTTP is much easier to loadbalance than a custom TCP protocol).

I see this source looking very similar to the HEC source, though instead of being for splunk-clients this is more aimed at elasticsearch-client systems.

Attempted Solutions

Unfortunately I don't really have any attempted solutions at this time.

Proposal

I think I covered this in my use case, though I'm happy to revise/edit this to make it more readable!

References

[1] https://github.com/timberio/vector/issues/2719 [2] https://www.elastic.co/guide/en/beats/winlogbeat/current/elasticsearch-output.html [3] https://www.elastic.co/guide/en/logstash/current/plugins-inputs-beats.html [4] https://github.com/elastic/beats/issues/7824#issuecomment-409553329