[FLINK-11893] [Project Website] Update ecosystem to encourage sharing of the connectors

[becketqin]

Based on the discussion in the mailing list, we would like to encourage the connector authors to share their connectors with the community, even the connectors are not in the Flink repo.

This ticket updates the ecosystem page on the website to reflect the change. It also adds ecosystem page back to the navigation bar.

[fhueske]

Hi @becketqin, thanks for the PR!

Can you remove all changes generated by the build script? That will make the PR easier to review.

Thank you, Fabian

[rmetzger]

Thank you @becketqin ! I left some comments. Once these are resolved, I'm +1 to merge.

[becketqin]

@fhueske @rmetzger Thanks a lot for the review. I updated the PR based on your comments. I removed the changes to the html files from build script. I will do a rebuild and add them back before the PR is merged.

[rmetzger]

+1 to merge from my point of view

[becketqin]

@fhueske Ping. Do you have time to take another look?

[rmetzger]

I'd like to get this in asap, otherwise, the real community packages website will be done before this is merged :)

[becketqin]

@rmetzger Yeah... I agree. @fhueske could you take look? Maybe we can check in the current update and make further modifications in separate patches if needed?

[fhueske]

Hi, sorry @becketqin, I forgot about this PR. I'll have a look later today.

Thanks, Fabian

[becketqin]

@rmetzger @fhueske One thing not quite clear to me is the compatible Flink version section. As far as I understand, anything compatible with Flink X.Y should also be compatible with Flink X.Z as long as Z >= Y. Is that correct? If that is the case, I'll check at which Flink version was each connector added, and update the Compatible Flink Versions accordingly.

[fhueske]

Thanks for the update @becketqin. I agree that this page should be a quick reference to more detailed information. However, I also think that it should be complete (at least with respect to build-in connectors). One of the main problems is the large number of possible combinations (APIs, source/sink, encodings, etc.).

Regarding your question: This is usually true. However, we sometimes also drop support for outdated connectors or older versions of other systems.

[rmetzger]

@rmetzger @fhueske One thing not quite clear to me is the compatible Flink version section. As far as I understand, anything compatible with Flink X.Y should also be compatible with Flink X.Z as long as Z >= Y. Is that correct? If that is the case, I'll check at which Flink version was each connector added, and update the Compatible Flink Versions accordingly.

How about removing that information from the page? I would prefer to have something about our ecosystem on the Flink website, instead of discussing forever here.

[rmetzger]

Once this is merged, let's add https://issues.apache.org/jira/browse/FLINK-12783

[fhueske]

I'm fine leaving the compatibility version out.

As I said before, I think the information about available build-in connectors should be complete as this page is likely to be considered as source-of-truth wrt. connector availability and it would be a shame if users would be discouraged because they cannot find info about an available connector.

[becketqin]

@rmetzger Thanks for pushing this. I'll remove the compatible versions column.

I'm fine leaving the compatibility version out.

As I said before, I think the information about available build-in connectors should be complete as this page is likely to be considered as source-of-truth wrt. connector availability and it would be a shame if users would be discouraged because they cannot find info about an available connector.

Hi @fhueske , in general I agree that this page should serve as a good connector availability reference for the users. However, it is unclear to me what is considered complete. For example, some security mechanism may not be supported, or we only have append sink while user needs an upsert sink, or some source connector may support timestamp while some others do not. Is the availability information of these connectors incomplete? To the users who needs that, such connector is not available. But including them here will simply explode the page and make it difficult to maintain.

So I think we need to draw a line between "available" and "fully meet user requirements". Personally speaking, I would take connector type, supported API and compatible versions as necessary for users to consider a connector to be available. And other information, such as supported encoding, supported security mechanisms, whether it is an upsert or append sink, belong to whether the connector "fully meets user requirements". We can keep them in the individual pages of each connectors.

Thoughts?

[MartijnVisser]

Completed via https://issues.apache.org/jira/browse/FLINK-22922