flink-cdc icon indicating copy to clipboard operation
flink-cdc copied to clipboard

update mysql-postgres-totorial.md

Open dk333 opened this issue 3 years ago • 4 comments

add postgres-cdc debezium.plugin.name to source table definition. without debezium.plugin.name, it will raise an exception.

dk333 avatar Aug 12 '22 03:08 dk333

@dk333 Thanks @dk333 for the contribution, but the plugin is not required, it's an optional config IIUC.

leonardBang avatar Aug 12 '22 06:08 leonardBang

@dk333 What exception will be thrown? I tried with my envrionment, it works well without any exception.

image

luoyuxia avatar Aug 13 '22 02:08 luoyuxia

the exception like this. and i do not have a plugin named decoderbufs. i have check debezium offical doc, it say that we should use 'debezium.plugin.name' = 'pgoutput' or 'debezium.plugin.name' = 'decoderbufs' . if i have installed pg 13, the plugin pgoutput is ready. so i put this option in it.

2022-08-15 11:13:38 io.debezium.DebeziumException: Creation of replication slot failed at io.debezium.connector.postgresql.PostgresConnectorTask.start(PostgresConnectorTask.java:134) at io.debezium.connector.common.BaseSourceTask.start(BaseSourceTask.java:130) at io.debezium.embedded.EmbeddedEngine.run(EmbeddedEngine.java:759) at io.debezium.embedded.ConvertingEngineBuilder$2.run(ConvertingEngineBuilder.java:188) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.postgresql.util.PSQLException: ERROR: could not access file "decoderbufs": No such file or directory at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2532) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2267) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:312) at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:448) at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:369) at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:310) at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:296) at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:273) at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:268) at io.debezium.connector.postgresql.connection.PostgresReplicationConnection.createReplicationSlot(PostgresReplicationConnection.java:357) at io.debezium.connector.postgresql.PostgresConnectorTask.start(PostgresConnectorTask.java:127) ... 6 more

my table definition is in below.

CREATE TABLE emp_copy ( empno string NOT NULL, ename string, job string, mgr string, hiredate TIMESTAMP(6), sal decimal(7,2), comm decimal(7,2), deptno string, PRIMARY KEY (empno) not ENFORCED ) WITH ( 'connector' = 'postgres-cdc', 'hostname' = '127.0.0.1', 'port' = '5432', 'username' = 'postgres', 'password' = 'postgres', 'database-name' = 'testdb', 'schema-name' = 'public', 'table-name' = 'emp_copy' ,'debezium.slot.name' = 'xxx' -- ,'debezium.plugin.name' = 'pgoutput' );

uncomment this line -- ,'debezium.plugin.name' = 'pgoutput' the exception does disappear.

dk333 avatar Aug 15 '22 03:08 dk333

@dk333 What exception will be thrown? I tried with my envrionment, it works well without any exception.

image

submit job to flink is ok, but the job will restart again and again without ending.

dk333 avatar Aug 15 '22 03:08 dk333