fluent
'thrift_flume_event_server' not found
It seems the current code removed 'thrift_flume_event_server' yet in in_flume.rb, it is still requiring the missing file. I think the thrift definition has changed?
Yes, flume's thrift protocol definition is changed.
Currently, flume's thrift protocol does not support sinking into flume.
in_flume plugin is obsolete.
I thought the thrift protocol was updated couple of month ago here? so, that's outdated too?
Removing require is not enough?
If in_flume works with older flume, we can provide it.
I figured out you are using this thrift_source_protocol.rb, how about requiring this, would it work?
, how about requiring this, would it work?
No, it wouldn't. Currently, it does not work.
Current thrift protocol sources cannot use new and older thrift protocol together in the same directory because their sources has same name which will cause conflict.
To solve this issue, my idea is:
- create old-thrift directory and revive older thrift protocol
- in_flume uses old-thrift protocol
Will just using 0.1.1 solve this issue, @puyuan ?
You mean that you want to use older flume with fluentd?
I thought the thrift protocol was updated couple of month ago here? so, that's outdated too?
Partially yes, but Flume thrift protocol had been updated over two years ago: https://github.com/apache/flume/commit/60da3d8606415202f966017a18084cb59d3e64d1#diff-cbeeb727e5da413a8c0abe58b0264d22
And in_flume uses outdated thrift protocol but it does not work for now.
Basically, I am using the latest flume 1.6 and I cannot get this plugin to work. It is not clear from the readme, which version of flume this plugin is supporting.
On the other hand, if I use this flume sink ( flume-ng-fluentd-sink ) and forward to 24224, I can completely skip using fluent-plugin-flume.
So its a matter of using 1) flume thrift -> fluent-plugin-flume or 2) flume-ng-fluentd-sink -> fluent source forward
Apparently 1 is broken right now, and the plugin has issues requiring the rb files. I was trying to see if 1) could work, so I don't have to install additional jars in my flume installation in the case of 2)
Apparently 1 is broken right now, and the plugin has issues requiring the rb files.
Your understanding is partially right, but latest Flume thrift protocol does not have sending flume event to outside flume functionality.
Because FlumeEventServer in flume protocol had been removed.
1 is broken against latest Flume even if reviving old thrift protocol.
2 is recommended way for now.