br icon indicating copy to clipboard operation
br copied to clipboard
verified publisher icon pingcap

BR Restore incremental data encountered Error: [kv:1062]Duplicate entry

Open yilongrong opened this issue 3 years ago • 5 comments

Please answer these questions before submitting your issue. Thanks!

  1. What did you do? If possible, provide a recipe for reproducing the error.

Cluster TiDB1

  1. create table test( id int,name varchar(20));
  2. insert into test values(1,'a'),(3,'b'));
  3. tiup br:v5.4.0 backup table --pd "172.xx.x.138:16279" --db test --table test --storage "local:///tmp/backup" --log-file backup.log
  4. alter table test add age int default 1;
  5. update test set age=2 where id=1;
  6. update test set age=3 where id=3;
  7. alter table test add unique index indx(age);
  8. tiup br:v5.4.0 backup table --pd "172.xx.xx.138:16279" --db test --table test --storage "local:///tmp/backupincr" --log-file backup.log --lastbackupts 434262749356228611

Cluster TiDB2

  1. tiup br:v5.4.0 restore table --pd "172.xx.xx.136:16279" --db "test" --table "test" --storage "local:///tmp/backup"

  2. tiup br:v5.4.0 restore table --pd "172.xx.xx.136:16279" --db "test" --table "test" --storage "local:///tmp/backupincr"

  3. What did you expect to see?

restore incremental data success.

  1. What did you see instead?

alter table test add unique index indx(age); [2022/06/30 17:35:16.975 +08:00] [INFO] [collector.go:67] ["Table restore failed summary"] [total-ranges=0] [ranges-succeed=0] [ranges-failed=0] Error: [kv:1062]Duplicate entry '1' for key 'indx' Error: run /home/tidbryl/.tiup/components/br/v5.4.0/br (wd:/home/tidbryl/.tiup/data/TAES1t4) failed: exit status 1

  1. What version of BR and TiDB/TiKV/PD are you using?

BR version: Release Version: v5.4.0 Git Commit Hash: 55f3b24c1c9f506bd652ef1d162283541e428872 Git Branch: heads/refs/tags/v5.4.0 Go Version: go1.16.4 UTC Build Time: 2022-01-25 08:36:34

cluster version: v5.4.0

  1. Operation logs

    • Please upload br.log for BR if possible
    • Please upload tidb-lightning.log for TiDB-Lightning if possible
    • Please upload tikv-importer.log from TiKV-Importer if possible
    • Other interesting logs

  2. Configuration of the cluster and the task

    • tidb-lightning.toml for TiDB-Lightning if possible
    • tikv-importer.toml for TiKV-Importer if possible
    • topology.yml if deployed by TiUP

  3. Screenshot/exported-PDF of Grafana dashboard or metrics' graph in Prometheus if possible

yilongrong avatar Jun 30 '22 09:06 yilongrong

Cloud you please add the sql statement of adding column age?

WangLe1321 avatar Jul 05 '22 10:07 WangLe1321

Done，Please check.

yilongrong avatar Jul 05 '22 11:07 yilongrong

This problem seems to affect all 4.x and 5.x and I'am not sure whether it has been fixed in 6.x

yuqi1129 avatar Jul 06 '22 02:07 yuqi1129

The problem lies the logic: When restore, we always restore ddl first and then the table data, maybe we should rethink this logic again.

yuqi1129 avatar Jul 06 '22 02:07 yuqi1129

/assign @WangLe1321

WangLe1321 avatar Jul 06 '22 06:07 WangLe1321