br_log_restore fails with `start ts:421355834788020227 is greater than resolved ts:0`

[overvenus]

br_log_restore fails https://internal.pingcap.net/idc-jenkins/blue/organizations/jenkins/br_ghpr_unit_and_integration_test/detail/br_ghpr_unit_and_integration_test/4116/pipeline#step-1318-log-326

[2020-12-07T12:43:29.384Z] [2020/12/07 20:43:26.743 +08:00] [ERROR] [restore.go:37] ["failed to restore"] [error="start ts:421355834788020227 is greater than resolved ts:0: [BR:Restore:ErrRestoreResolvedTsConstrain]resolved ts constrain violation"] [errorVerbose="[BR:Restore:ErrRestoreResolvedTsConstrain]resolved ts constrain violation\nstart ts:421355834788020227 is greater than resolved ts:0\ngithub.com/pingcap/br/pkg/restore.(*LogClient).RestoreLogData\n\t/home/jenkins/agent/workspace/br_ghpr_unit_and_integration_test/go/src/github.com/pingcap/br/pkg/restore/log_client.go:938\ngithub.com/pingcap/br/pkg/task.RunLogRestore\n\t/home/jenkins/agent/workspace/br_ghpr_unit_and_integration_test/go/src/github.com/pingcap/br/pkg/task/restore_log.go:136\ngithub.com/pingcap/br/cmd.runLogRestoreCommand\n\t/home/jenkins/agent/workspace/br_ghpr_unit_and_integration_test/go/src/github.com/pingcap/br/cmd/restore.go:36\ngithub.com/pingcap/br/cmd.newLogRestoreCommand.func1\n\t/home/jenkins/agent/workspace/br_ghpr_unit_and_integration_test/go/src/github.com/pingcap/br/cmd/restore.go:132\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/[email protected]/command.go:842\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/[email protected]/command.go:950\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/[email protected]/command.go:887\ngithub.com/pingcap/br.main\n\t/home/jenkins/agent/workspace/br_ghpr_unit_and_integration_test/go/src/github.com/pingcap/br/main.go:58\ngithub.com/pingcap/br.TestRunMain.func1\n\t/home/jenkins/agent/workspace/br_ghpr_unit_and_integration_test/go/src/github.com/pingcap/br/main_test.go:39\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"] [stack="github.com/pingcap/br/cmd.runLogRestoreCommand\n\t/home/jenkins/agent/workspace/br_ghpr_unit_and_integration_test/go/src/github.com/pingcap/br/cmd/restore.go:37\ngithub.com/pingcap/br/cmd.newLogRestoreCommand.func1\n\t/home/jenkins/agent/workspace/br_ghpr_unit_and_integration_test/go/src/github.com/pingcap/br/cmd/restore.go:132\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/[email protected]/command.go:842\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/[email protected]/command.go:950\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/[email protected]/command.go:887\ngithub.com/pingcap/br.main\n\t/home/jenkins/agent/workspace/br_ghpr_unit_and_integration_test/go/src/github.com/pingcap/br/main.go:58\ngithub.com/pingcap/br.TestRunMain.func1\n\t/home/jenkins/agent/workspace/br_ghpr_unit_and_integration_test/go/src/github.com/pingcap/br/main_test.go:39"]

[2020-12-07T12:43:29.384Z] Error: start ts:421355834788020227 is greater than resolved ts:0: [BR:Restore:ErrRestoreResolvedTsConstrain]resolved ts constrain violation

[2020-12-07T12:43:29.384Z] [2020/12/07 20:43:26.743 +08:00] [ERROR] [main.go:59] ["br failed"] [error="start ts:421355834788020227 is greater than resolved ts:0: [BR:Restore:ErrRestoreResolvedTsConstrain]resolved ts constrain violation"] [errorVerbose="[BR:Restore:ErrRestoreResolvedTsConstrain]resolved ts constrain violation\nstart ts:421355834788020227 is greater than resolved ts:0\ngithub.com/pingcap/br/pkg/restore.(*LogClient).RestoreLogData\n\t/home/jenkins/agent/workspace/br_ghpr_unit_and_integration_test/go/src/github.com/pingcap/br/pkg/restore/log_client.go:938\ngithub.com/pingcap/br/pkg/task.RunLogRestore\n\t/home/jenkins/agent/workspace/br_ghpr_unit_and_integration_test/go/src/github.com/pingcap/br/pkg/task/restore_log.go:136\ngithub.com/pingcap/br/cmd.runLogRestoreCommand\n\t/home/jenkins/agent/workspace/br_ghpr_unit_and_integration_test/go/src/github.com/pingcap/br/cmd/restore.go:36\ngithub.com/pingcap/br/cmd.newLogRestoreCommand.func1\n\t/home/jenkins/agent/workspace/br_ghpr_unit_and_integration_test/go/src/github.com/pingcap/br/cmd/restore.go:132\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/[email protected]/command.go:842\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/[email protected]/command.go:950\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/[email protected]/command.go:887\ngithub.com/pingcap/br.main\n\t/home/jenkins/agent/workspace/br_ghpr_unit_and_integration_test/go/src/github.com/pingcap/br/main.go:58\ngithub.com/pingcap/br.TestRunMain.func1\n\t/home/jenkins/agent/workspace/br_ghpr_unit_and_integration_test/go/src/github.com/pingcap/br/main_test.go:39\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"] [stack="github.com/pingcap/br.main\n\t/home/jenkins/agent/workspace/br_ghpr_unit_and_integration_test/go/src/github.com/pingcap/br/main.go:59\ngithub.com/pingcap/br.TestRunMain.func1\n\t/home/jenkins/agent/workspace/br_ghpr_unit_and_integration_test/go/src/github.com/pingcap/br/main_test.go:39"]

[3pointer]

https://github.com/pingcap/br/pull/636#issuecomment-740344957 This error usually means cdc didn't flush meta to minio. it seems we need sleep more time here...