linkedin
Streaming to kafka
I know it's a long shot, but asking anyway incase someone has already solved this. Can I pump the delta to kafka instead of keeping it in memory and waiting for an agent to poll the endpoint? is it possible with some tweaks?
My usecase - An Open source GoldenGate equivalent which will stream all the DML operation in the source Oracle to a compacted kafka topic.
It would require pretty substantial "tweaks" to Databus to do this, but the basic idea of using Kafka as the transport layer is sound and is, in fact, exactly what LinkedIn's follow-on project Brooklin does: https://github.com/linkedin/Brooklin/
Big caveat: currently no Oracle connector is available. A trigger-based one (as in Databus) is under development and may be open-sourced eventually, but there's no timetable for this. You could consider adapting Databus's trigger-based source code to create a Brooklin connector, however; it wouldn't be a small job, but it's also not nearly as large a job as converting Databus to use Kafka. This approach would still require the same kind of "databusification" of your Oracle DB (sy$txlog table, coalesce job, etc.) as Databus itself does.
From: jaisonpjohn [email protected] Sent: Thursday, August 29, 2019 11:52 AM To: linkedin/databus [email protected] Cc: Subscribed [email protected] Subject: [linkedin/databus] Streaming to kafka (#123)
I know it's a longcshot, but asking anyway incase someone has alreay solved this. Can I pump the delta to kafka instead of keeping it in memory and waiting for an agent to poll the endpoint? is it possible with some tweaks?
My usecase - Opensource GoldenGate equivalent which will stream all the DML operation in the source Oracle to a compacted kafka topic.
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Flinkedin%2Fdatabus%2Fissues%2F123%3Femail_source%3Dnotifications%26email_token%3DAAXBMP3ZT4YSES7EJSJEVALQHALHJA5CNFSM4ISFV2NKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HII4ENQ&data=02%7C01%7Cgroelofs%40linkedin.com%7Cc44841b6ad4145b66f5508d72cb210f7%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637027015611260481&sdata=vqmIjPa9XMbKpCRo1uavo%2FGSPgr4GwK58U2JgkYMu3I%3D&reserved=0, or mute the threadhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAAXBMP4OP3SX7FTMBQADV53QHALHJANCNFSM4ISFV2NA&data=02%7C01%7Cgroelofs%40linkedin.com%7Cc44841b6ad4145b66f5508d72cb210f7%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637027015611260481&sdata=CMi0BiaHkWeia0C5IDdK96HZSKqGU3DYFqCbsPoVoFw%3D&reserved=0.
