abnormal behavior in batch mode

[AidenPTHuang]

A brief description of the bug.

In my case, I run openlog replicator in batch mode for each of archived redo logs. Openlog replicator would produce new chkpt-scnxxx.json file and the size of new file is bigger than last run, and it cause the size of chkpt-scnxxx.json might be up to GB scale, then the performance of OLR is upgraded due to big size of chkpt-scnxxx.json.

Is the bug present on the latest master branch. Yes

Describe steps required to reproduce the bug.

1. Build Openlog Replicator by https://github.com/bersler/OpenLogReplicator-tutorials
2. Run step 1~5 by https://github.com/bersler/OpenLogReplicator-tutorials offline example (Disable docker start ${OLR_CONTAINER} in step 4)
3. Run SQL to archive logs in ORA2 ALTER SESSION SET CONTAINER = CDB$ROOT; ALTER SYSTEM ARCHIVE LOG CURRENT;
4. Check the path of archived redo log(o1_mf_1_5_lyj59ron_.arc) and run Openlog Replicator with the following configuration { "version": "1.5.0", "log-level": 3, "trace": 0, "source": [ { "alias": "S1", "name": "ORA2", "reader": { "type": "batch", "redo-log": ["/opt/fra/XE/archivelog/2024_03_06/o1_mf_1_5_lyj59ron_.arc"] }, "format": { "type": "json" }, "filter": { "table": [ {"owner": "USR2", "table": "ADAM.*"} ] } } ], "target": [ { "alias": "T1", "source": "S1", "writer": { "type": "file", "output": "/opt/output/results.txt" } } ] }

Success to prase CDC records from redo logs

6. Insert some records to table
7. Run SQL to archive logs in ORA2
8. Check the path of archived redo log(o1_mf_1_6_lyj5g2x4_.arc) and run Openlog Replicator with the following configuration { "version": "1.5.0", "log-level": 3, "trace": 0, "source": [ { "alias": "S1", "name": "ORA2", "reader": { "type": "batch", "redo-log": ["/opt/fra/XE/archivelog/2024_03_06/o1_mf_1_6_lyj5g2x4_.arc"] }, "format": { "type": "json" }, "filter": { "table": [ {"owner": "USR2", "table": "ADAM.*"} ] } } ], "target": [ { "alias": "T1", "source": "S1", "writer": { "type": "file", "output": "/opt/output/results.txt" } } ] } At this step, got warning from logsWARN 60027 couldn't find archive log for seq: 5, found: 6, sleeping 10000000 us, and saw the value of seq in ORA2-chkpt-2625606.json be set to 4294967295 from 5
9. Re-run Openlog Replicator again At this step, got ERROR 50023 duplicate SYS.COL$ value: (rowid: AAAAACAABAAAACRAAl) and a new ORA2-chkpt-2627212.json is created with seq 6 and the values of online-redo and incarnations are bigger than last run. Ex: "online-redo":[ {"group":1,"path":["/opt/oracle/oradata/XE/redo01.log"]}, {"group":2,"path":["/opt/oracle/oradata/XE/redo02.log"]}, {"group":3,"path":["/opt/oracle/oradata/XE/redo03.log"]}, {"group":2,"path":["/opt/oracle/oradata/XE/redo02.log"]}, {"group":1,"path":["/opt/oracle/oradata/XE/redo01.log"]}, {"group":2,"path":["/opt/oracle/oradata/XE/redo02.log"]}, {"group":1,"path":["/opt/oracle/oradata/XE/redo01.log"]}, {"group":3,"path":["/opt/oracle/oradata/XE/redo03.log","/opt/oracle/oradata/XE/redo03.log"]}], "incarnations":[ {"incarnation":1,"resetlogs-scn":1,"prior-resetlogs-scn":0,"status":"PARENT","resetlogs":1080860703,"prior-incarnation":0}, {"incarnation":2,"resetlogs-scn":2571212,"prior-resetlogs-scn":1,"status":"CURRENT","resetlogs":1162882501,"prior-incarnation":1}, {"incarnation":1,"resetlogs-scn":1,"prior-resetlogs-scn":0,"status":"PARENT","resetlogs":1080860703,"prior-incarnation":0}, {"incarnation":2,"resetlogs-scn":2571212,"prior-resetlogs-scn":1,"status":"CURRENT","resetlogs":1162882501,"prior-incarnation":1}, {"incarnation":2,"resetlogs-scn":2571212,"prior-resetlogs-scn":1,"status":"CURRENT","resetlogs":1162882501,"prior-incarnation":1}, {"incarnation":1,"resetlogs-scn":1,"prior-resetlogs-scn":0,"status":"PARENT","resetlogs":1080860703,"prior-incarnation":0}], But it is success to prase CDC records from redo logs in this time

You can find full logs in OpenLogReplicator.log For bugs related to Redo Log parse error where reproduction is not possible. This is for cases, where the fault is related to actual redo log data parsing. There is some redo log file which is causing the error, but it is not known which actual combination of SQL commands caused the error. To make a fix possible, the redo log file is required (even if the schema file or checkpoint file is not available). Please provide the redo log file.

Provide the following information:

1. OLR configuration file (necessary)
2. Set of redo log files which contain error (necessary)
3. OLR checkpoint file set (helpful, but not crucial)
4. Information about expected error

Additional context Add any other context or screenshots about the feature request here.